Infrastructure Monitoring

IP Service Level Agreement (IP SLA)

IP SLA allows you to probe different key performance indicators of the network such as reachability, latency, jitter, etc.

For the purpose of CCIE Datacenter, we are only going to enable feature sla sender. In particular, I am going to explain the ICMP echo configuration.

N9K01(config)# feature sla sender
N9K01(config)# ip sla 111
N9K01(config-ip-sla)# icmp-echo 1.1.1.1 source-interface e1/1
N9K01(config-ip-sla-echo)# threshold 40
N9K01(config-ip-sla-echo)# timeout 120
N9K01(config-ip-sla-echo)# frequency 2
N9K01(config-ip-sla-echo)# ip sla schedule 111 life forever start-time now

• Frequency: The seconds between each operation.
• Timeout: The number of milliseconds that operation waits for a response to report the IP SLA Up state.
• Threshold: Same as timeout but the milliseconds which generates syslog/trap. It needs to be less than or equal to the value of timeout.

After IP SLA configuration, you need to schedule it to start.

Object Tracker (Track)

You cannot employ IP SLA to do something for you based on it’s state with Cisco NX-OS. You need to “track” an SLA, then use that track object somewhere which supports track object. For example, you can use tracks in FHRPs, EEM, or a static route.

An object tracker monitors the state of an interface or an IP SLA probe. The snippet below shows how track object can track the state of an interface and the state of an IP SLA.

! An example that tracks the state of an interface
N9K01(config)# track 1 interface e1/1 line-protocol
! An example which shows how a track is leveraged to track the state of an SLA
N9K01(config)# track 111 ip sla 111 reachability

A track can also monitor another track. For example,

N9K01(config)# track 100
N9K01(config-track)# object 111

In the example above, track 100 tracks track 111. So, in the end of the day, they will have the same result. Well, that is not much useful for us. Let’s create a track object which tracks multiple other tracks.

N9K01(config)# feature sla sender
N9K01(config)# ip sla 111
N9K01(config-ip-sla)# icmp-echo 1.1.1.1 source-interface e1/1
N9K01(config-ip-sla-echo)# threshold 40
N9K01(config-ip-sla-echo)# timeout 120
N9K01(config-ip-sla-echo)# frequency 2
N9K01(config-ip-sla-echo)# ip sla schedule 111 life forever start-time now
N9K01(config)# track 111 ip sla 111 reachability
N9K01(config)# ip sla 112
N9K01(config-ip-sla)# icmp-echo 2.2.2.2 source-interface e1/1
N9K01(config-ip-sla-echo)# threshold 40
N9K01(config-ip-sla-echo)# timeout 120
N9K01(config-ip-sla-echo)# frequency 2
N9K01(config-ip-sla-echo)# ip sla schedule 112 life forever start-time now
N9K01(config)# track 112 ip sla 112 reachability
N9K01(config)# ip sla 113
N9K01(config-ip-sla)# icmp-echo 2.2.2.2 source-interface e1/1
N9K01(config-ip-sla-echo)# threshold 40
N9K01(config-ip-sla-echo)# timeout 120
N9K01(config-ip-sla-echo)# frequency 2
N9K01(config-ip-sla-echo)# ip sla schedule 113 life forever start-time now
N9K01(config)# track 113 ip sla 113 reachability
N9K01(config)# track 100 list threshold percentage
N9K01(config-track)# object 111
N9K01(config-track)# object 112
N9K01(config-track)# object 113
N9K01(config-track)# threshold percentage down 40 up 60
N9K01(config-track)# delay down 6

As an exercise, experiment with track NUMBER list boolean command.

Embedded Event Manager (EEM)

EEM monitors the events (event statement: for example, interface shuts down) that occur on your Nexus switch and then takes action (action statement: generate syslog/trap, send email, run a series of command, no shut the interface) upon that event.

To configure the EEM you need to go through the following steps:

• event manager applet NAME
• event event_statement
• action number action_statement

You can also define variables for EEM with command event manager environment variable_name variable_value

As an Example, Let’s ask NX-OS if the track 100 is down, it sends us a syslog message.

N9K01(config)# event manager applet TRACK_100_DOWN
N9K01(config-applet)# event track 100 state down
N9K01(config-applet)# action 01.0 snmp-trap strdata "SNMP: At least two of IP SLAs 111, 112, and 113 are not receiving ICMP echo replies on N9K01 - 91.91.91.9 anymore."
N9K01(config-applet)# action 02.0 syslog msg SYSLOG: At least two of IP SLAs 111, 112, and 113 are not receiving ICMP echo replies on N9K01 - 91.91.91.9 anymore.

Should you need more information about the above examples including EEM, I strongly suggest that you review the workshop in this post.

Workshop

In this workshop, we are going to simulate Automatic BGP switchover in case that not directly connected link to ISP fails but if some targets in the Internet fail as per IP SLA perspective. This can happen if the upstream service provider is DOSed for example. I implemented the original solution in production with IOS-XE to switchover the BGP back after one hour of stability. But with NX-OS since there was no event timer countdown command, I suggest you use the up delay which is restricted to maximum of 3 minutes.

N9K01:

hostname N9K01
feature bgp
feature sla sender

ip prefix-list AS_90 seq 5 permit 192.190.0.0/24
ip prefix-list DEFAULT_ROUTE seq 5 permit 0.0.0.0/0
route-map AS_PREPEND permit 10
  match ip address prefix-list AS_90
  
route-map LOCAL_PREF permit 10
  match ip address prefix-list DEFAULT_ROUTE
  set local-preference 150

interface Ethernet1/1
  ip address 91.91.91.9/24
  no shutdown

interface Ethernet1/2
  ip address 169.254.0.1/30
  no shutdown

interface loopback0
  ip address 192.190.0.1/24

router bgp 90
  address-family ipv4 unicast
    network 192.190.0.0/24
  neighbor 91.91.91.1
    remote-as 111
    address-family ipv4 unicast
      route-map LOCAL_PREF in
      route-map AS_PREPEND out
  neighbor 169.254.0.2
    remote-as 90
    address-family ipv4 unicast
      next-hop-self

ip access-list R1-DIRECT-INTERNET
        10 permit icmp 91.91.91.9/32 1.1.1.1/32
        20 permit icmp 91.91.91.9/32 2.2.2.2/32
        30 permit icmp 91.91.91.9/32 3.3.3.3/32
		
route-map R1-DIRECT-INTERNET permit 10
  match ip address R1-DIRECT-INTERNET
  set ip next-hop 91.91.91.1
ip local policy route-map R1-DIRECT-INTERNET
   
feature sla sender  
ip sla 111
 icmp-echo 1.1.1.1 source-interface e1/1
 threshold 40
 timeout 120
 frequency 2
ip sla schedule 111 life forever start-time now
track 111 ip sla 111 reachability

ip sla 112
 icmp-echo 2.2.2.2 source-interface e1/1
 threshold 40
 timeout 120
 frequency 2
ip sla schedule 112 life forever start-time now
track 112 ip sla 112 reachability

ip sla 113
 icmp-echo 3.3.3.3 source-interface e1/1
 threshold 40
 timeout 120
 frequency 2
ip sla schedule 113 life forever start-time now
track 113 ip sla 113 reachability

track 100 list threshold percentage
 object 111
 object 112
 object 113
 threshold percentage down 40 up 60
 delay down 6
 delay up 180

event manager applet TRACK_100_DOWN
event track 100 state down
action 01.0 snmp-trap strdata "SNMP: At least two of IP SLAs 111, 112, and 113 are not receiving ICMP echo replies on N9K01 - 91.91.91.9 anymore."
action 02.0 syslog msg SYSLOG: At least two of IP SLAs 111, 112, and 113 are not receiving ICMP echo replies on N9K01 - 91.91.91.9 anymore.

no event manager applet TRACK_ISP_UP
event manager applet TRACK_ISP_UP
  event track 100 state up
  action 0.9 event-default
  action 1.0 cli conf t
  action 2.4 cli router bgp 90
  action 2.5 cli neighbor 91.91.91.1
  action 2.6 cli no shutdown
  action 2.8 cli end
  action 2.9 cli syslog msg BGP neighbor 91.91.91.1 is UP by EEM script
  action 3.0 cli snmp-trap strdata "BGP neighbor 91.91.91.1 is UP by EEM"

no event manager applet eBGP-DOWN  
event manager applet eBGP-DOWN
  event track 100 state down
  action 0.9 event-default
  action 1.0 cli conf t
  action 2.0 cli no event manager applet eBGP_UP
  action 4.0 cli router bgp 90
  action 5.0 cli neighbor 91.91.91.1
  action 5.1 cli shutdown
  action 6.0 cli end
  action 7.0 syslog msg BGP neighbor 91.91.91.1 is shutdown by EEM script
  action 8.0 snmp-trap strdata "BGP neighbor 91.91.91.1 is shutdown by EEM"

N9K02:

hostname N9K02
feature bgp

ip prefix-list AS_90 seq 5 permit 192.190.0.0/24
ip prefix-list DEFAULT_ROUTE seq 5 permit 0.0.0.0/0
route-map AS_PREPEND permit 10
  match ip address prefix-list AS_90
  set as-path prepend 90 90
route-map LOCAL_PREF permit 10
  match ip address prefix-list DEFAULT_ROUTE
  set local-preference 90

interface Ethernet1/1
  ip address 92.92.92.9/24
  no shutdown

interface Ethernet1/2
  ip address 169.254.0.2/30
  no shutdown

interface loopback0
  ip address 192.190.0.2/24

router bgp 90
  address-family ipv4 unicast
    network 192.190.0.0/24
  neighbor 92.92.92.2
    remote-as 222
    address-family ipv4 unicast
      route-map LOCAL_PREF in
      route-map AS_PREPEND out
  neighbor 169.254.0.1
    remote-as 90
    address-family ipv4 unicast
      next-hop-self