EEM to send email when Interface goes down

Introduction about EEM:

Cisco IOS Embedded Event Manager (EEM) is a powerful and flexible subsystem that provides real-time network event detection and onboard automation. It gives you the ability to adapt the behavior of your network devices to align with your business needs.

Your business can benefit from the capabilities of IOS Embedded Event Manager without upgrading to a new version of Cisco IOS Software. It is available on a wide range of Cisco platforms.

IOS Embedded Event Manager supports more than 20 event detectors that are highly integrated with different Cisco IOS Software components to trigger actions in response to network events. Your business logic can be injected into network operations using IOS Embedded Event Manager policies. These policies are programmed using either simple command-line interface (CLi) or using a scripting language called Tool Command Language (Tcl).

Harnessing the significant intelligence within Cisco devices, IOS Embedded Event Manager helps enable creative solutions, including automated troubleshooting, fault detection, and device configuration.
Source: Cisco.com




Lets get to it:

I will create EEM applet called interface down, and then I will choose the pattern of which you will match a certain string of letters that will be showing in the syslog for example (interface up or interface down or OSPF neighbor aged out or anything you want to match against)
 R1#
 R1#event manager applet INTERFACE_DOWN
 R1 #event syslog pattern " Loopback123, changed state to up"
 R1 #action 1.0 cli command "enable"  \ \ optional
 R1#action 2.0 cli command "show ip interface br | in up"
  R1#action 3.0 mail server "mail.pbit.com" to "samer.rafid@pbit.com" from "net.admin@pbit.com"   subject "link is down" body "please check internet link"
R1#

then to see this in action, you can enable debugging for event manager to debug mail events
#debug event manager action mail

Apr 14 14:17:27: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write Date: Sun, 14 Apr 2019 14:17:27 +0300

Apr 14 14:17:27: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write Message-ID: <000A2019141727818@PBIT-VG2.PBIT.COM>

Apr 14 14:17:27: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write From: net.admin@pbit.com

Apr 14 14:17:27: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write To: samir.rafid@pbit.com

Apr 14 14:17:27: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write Subject: link is down

Apr 14 14:17:27: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write

Apr 14 14:17:27: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write please check internet link

Apr 14 14:17:27: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write .

Apr 14 14:17:27: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : fh_smtp_chk_reply looking for {250}

Apr 14 14:17:28: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : fh_smtp_read {250 2.6.0 <000A2019141727818@PBIT-VG2.PBIT.COM> [InternalId=14955076124730, Hostname=mail02.PBIT.com] 1526 bytes in 0.182, 8.147 KB/sec Queued mail for delivery

}

Apr 14 14:17:28: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write QUIT

Apr 14 14:17:28: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : fh_smtp_chk_reply looking for {221}

Apr 14 14:17:28: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : fh_smtp_read {250 2.1.5 Recipient OK

}

Apr 14 14:17:28: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write DATA

Apr 14 14:17:28: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : fh_smtp_chk_reply looking for {354}

Apr 14 14:17:28: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : fh_smtp_read {221 2.0.0 Service closing transmission channel

}

Apr 14 14:17:28: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : fh_smtp_read {354 Start mail input; end with <CRLF>.<CRLF>

}

Apr 14 14:17:28: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write Date: Sun, 14 Apr 2019 14:17:28 +0300

Apr 14 14:17:28: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write Message-ID: <000B2019141728818@PBIT-VG2.PBIT.COM>

Apr 14 14:17:28: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write From: net.admin@pbit.com

Apr 14 14:17:28: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write To: samir.rafid@pbit.com

Apr 14 14:17:28: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write Subject: link is down

Apr 14 14:17:28: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write

Apr 14 14:17:28: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write please check internet link

Apr 14 14:17:28: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write .

Apr 14 14:17:28: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : fh_smtp_chk_reply looking for {250}

Apr 14 14:17:29: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : fh_smtp_read {250 2.6.0 <000B2019141728818@PBIT-VG2.PBIT.COM> [InternalId=14955076124731, Hostname=mail02.PBIT.com] 1526 bytes in 0.169, 8.800 KB/sec Queued mail for delivery

}

Apr 14 14:17:29: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : smtp_write QUIT

Apr 14 14:17:29: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : fh_smtp_chk_reply looking for {221}

Apr 14 14:17:29: %HA_EM-6-LOG: INTERFACE_DOWN : DEBUG(smtp_lib) : fh_smtp_read {221 2.0.0 Service closing transmission channel

Samer R. Saleem

Configure basic Netflow to check top 10 talkers on Network

In this post I will be talking about how to configure Cisco router with Netflow to check top talkers in the network
so configuring the topology below:

R5 will be configured with NETFLOW and R6 will be sending ICMP packets to R5's loopback
configurations:

then we start sending packets from R6 then we use the command #show ip flow top-talkers
and the result will be showing gradually increasing as below:

you can see that I'm sourcing traffic from R6 loopback towards R5 loopback and the bytes are increasing while the ICMP ping increasing.

OSPF multi-area link configuration

What is OSPF Multi-area feature?

basically as the name indicates, it will make one interface work and form multiple OSPF adjacencies
over a single link

for example, consider the below topology:

as you can see, we need to make the link between R4-R3 to work in backbone area and another two area's ( Area 101, Area 999)

ok, how to configure this?
well, simply by going to interface level and use the command #ip ospf multi-area xx

where X is the new Area number.
ok lets configure it:

R3:
interface Ethernet0/0
 ip address 43.0.0.3 255.255.255.0
 ip ospf network point-to-point
 ip ospf multi-area 999
 ip ospf multi-area 101
 ip ospf 1 area 0
 ip ospf cost 1
end
R4:
 interface Ethernet0/0
 ip address 43.0.0.4 255.255.255.0
 ip ospf network point-to-point
 ip ospf multi-area 999
 ip ospf multi-area 101
 ip ospf 1 area 0
end

Note: IP OSPF NETWORK POINT-TO-POINT is a must.

ok, how to verify this is working or not?
well, the multi-area command should create a new virtual link that can be seen using:

R3#show ip ospf inter br
Interface    PID   Area            IP Address/Mask    Cost  State Nbrs F/C
Lo0               1     0               3.3.3.3/32         1     LOOP  0/0
Et0/0             1     0               43.0.0.3/24        1     P2P   1/1
MA1             1     101             Unnumbered Et0/0   1     P2P   1/1
Et0/1             1     101             73.0.0.3/24        1     P2P   1/1
MA2             1     999             Unnumbered Et0/0   1     P2P   1/1

also in OSPF neighbor command:

R3#show ip os nei

Neighbor ID     Pri   State           Dead Time   Address         Interface
4.4.4.4           0   FULL/  -        00:00:39    43.0.0.4        Ethernet0/0
4.4.4.4           0   FULL/  -        00:00:39    43.0.0.4        OSPF_MA1
7.7.7.7           0   FULL/  -        00:00:32    73.0.0.7        Ethernet0/1
4.4.4.4           0   FULL/  -        00:00:38    43.0.0.4        OSPF_MA2



 and in here:
#show ip ospf

OSPF_MA2 is up, line protocol is up
  Interface is unnumbered. Using address of Ethernet0/0 (43.0.0.3), Area 999, Attached via Multi-area
  Process ID 1, Router ID 3.3.3.3, Network Type POINT_TO_POINT, Cost: 1

OK, So now why we need to use this feature?

OK, one of the reasons is that, OSPF prefer routes learned internally over the routes learned from other area, for example: R2 will prefer the routes coming from R4 over the routes coming from R6 via R3 border router, even if the metric is much better, OSPF will use the routes learned from (O over O IA.)

So what to do in this case?
well, we can configure the OSPF Multi-area command on R4-R3 link to make it work in Area 101,
in this case R2 will compare the metric and choose the best one to reach R5 loopback

lets check the routing table:
 R2#show ip route 5.5.5.5
Routing entry for 5.5.5.5/32
  Known via "ospf 1", distance 110, metric 20
  Tag 111111, type extern 2, forward metric 4
  Last update from 26.0.0.6 on Ethernet0/2, 00:19:39 ago
  Routing Descriptor Blocks:
  * 26.0.0.6, from 4.4.4.4, 00:19:39 ago, via Ethernet0/2
      Route metric is 20, traffic share count is 1
      Route tag 111111
R2#



a trace route from R1:
R1#traceroute 5.5.5.5 num
Type escape sequence to abort.
Tracing the route to 5.5.5.5
VRF info: (vrf in name/id, vrf out name/id)
  1 12.0.0.2 0 msec 0 msec 1 msec
  2 26.0.0.6 1 msec 0 msec 0 msec  (R6)
  3 67.0.0.7 1 msec 1 msec 1 msec (R7)
  4 73.0.0.3 1 msec 1 msec 1 msec (R3)
  5 43.0.0.4 2 msec 1 msec 1 msec (R4)
  6 192.168.1.5 2 msec *  2 msec (R5)
R1#


 I hope this was useful!


Samer R. Saleem

OSPF MTU ignore feature

In this article we are going to check how MTU can affect OSPF adjacency establishment

first let us write what are the conditions for two neighbor routers to become OSPF adjacent routers

1. subnet mask ( they have to be on same network)
2.same area
3.authentication
4.MTU
5.hello time
6.stub flags (one router in area x which is stub and its neighbor in same area must be also configured to be stub)

ok, so we have the MTU that has to match on both routers, I'm going to configure two routers with different MTU's
simple lab here:

I'm going to configure one router to be using the default MTU (1500), and change the other router to 1400 MTU

R2 Configs:
!
router osp 1
router-id 200.200.200.200
interface Ethernet0/0
 ip address 192.168.1.2 255.255.255.0
 ip mtu 1400
 ip ospf 1 area 0
end

R1 Configs:
!
router ospf 1
router-id 100.100.100.100
interface Ethernet0/0
ip address 192.168.1.1 255.255.255.0
ip mtu 1500
ip ospf 1 area 0
end

========================================


Ok, first thing you will notice that you will not get log for OSPF status changing to FULL
and if you use the command
#show ip ospf neighbor

you will see the process is stuck at Exchange
R2#
R2#show ip os nei

Neighbor ID     Pri   State           Dead Time   Address         Interface
100.100.100.100   1   EXSTART/BDR     00:00:36    192.168.1.1     Ethernet0/0
R2#

Ok, so now let's see some debug on R1

#debug ip ospf adj
R1>
*Nov 14 06:04:37.604: OSPF-1 HELLO Et0/0: Send hello to 224.0.0.5 area 0 from 192.168.1.1
*Nov 14 06:04:38.184: OSPF-1 ADJ   Et0/0: Rcv DBD from 200.200.200.200 seq 0x2A2 opt 0x52 flag 0x7 len 32  mtu 1400 state EXCHANGE
*Nov 14 06:04:38.184: OSPF-1 ADJ   Et0/0: Nbr 200.200.200.200 has smaller interface MTU
*Nov 14 06:04:38.184: OSPF-1 ADJ   Et0/0: Send DBD to 200.200.200.200 seq 0x2A2 opt 0x52 flag 0x2 len 52
R1>

the output is very clear, we have smaller MTU coming from R2
ok, so what is our options to solve this?
1. make both values equal (set both to 1500 or 1400)
2. ignore this value, ok how?

Note: this must be done on both routers and under interface level

#ip ospf mtu-ignore

now let us see the logs change on R1
R1(config-if)#

*Nov 14 06:07:05.456: OSPF-1 ADJ   Et0/0: Rcv DBD from 200.200.200.200 seq 0x1E02 opt 0x52 flag 0x1 len 52  mtu 1400 state EXCHANGE
*Nov 14 06:07:05.456: OSPF-1 ADJ   Et0/0: Exchange Done with 200.200.200.200
*Nov 14 06:07:05.456: OSPF-1 ADJ   Et0/0: Send LS REQ to 200.200.200.200 length 36 LSA count 1
*Nov 14 06:07:05.456: OSPF-1 ADJ   Et0/0: Send DBD to 200.200.200.200 seq 0x1E02 opt 0x52 flag 0x0 len 32
*Nov 14 06:07:05.457: OSPF-1 ADJ   Et0/0: Rcv LS UPD from 200.200.200.200 length 76 LSA count 1
*Nov 14 06:07:05.457: OSPF-1 ADJ   Et0/0: Synchronized with 200.200.200.200, state FULL
*Nov 14 06:07:05.457: %OSPF-5-ADJCHG: Process 1, Nbr 200.200.200.200 on Ethernet0/0 from LOADING to FULL, Loading Done




R1#show ip os nei

Neighbor ID         Pri   State                 Dead Time   Address         Interface
200.200.200.200   1   FULL/DR         00:00:38    192.168.1.2     Ethernet0/0
R1#


I hope this was helpful

Redistribution and Optimal path selection

Hi,

So, today I'm going to create lab showing how redistribution may effect the optimal path selection, first lets understand what does optimal path means?
Optimal means the one best route from A to B

Ok, so I'm using the following topology:

here you can see  ip address used and X is the router number

R5,R7,R4 are in EIGRP domain, and R3,R1,R2,R7,R4(interfaces) in OSPF domain.

 on R7,R4 we will do mutual redistribution in order to get the routes to R5
after doing the above this is what routing table of R5 looks like:

R5 routing table has two paths to all destinations in OSPF domain, coming from both gateways (75.0.0.7 and 45.0.0.4)

ok, so what to do in order to reach R3 loopback (3.3.3.3) using only one best route (optimal path)?

Ok, I will create a route map on both R4 and R7 that matches OSPF metric cost and if it is equal to (21) I will redistribute it with bandwidth (10000) else it will be redistribute with bandwidth (1000)

Ok lets configure it:

R7,R4

#route-map METRIC permit 10
#match metric 10 +- 11
# set metric 10000 10 255 10 1500
#route-map METRIC permit 20
#set metric 1000 10 255 10 1500

#router eigrp 100
#redistribute OSPF 1 route-map METRIC

Now checking R5 routing table to R3 loopback:

R5#show ip route 3.3.3.3
Routing entry for 3.3.3.3/32
  Known via "eigrp 100", distance 170, metric 284160, type external
  Redistributing via eigrp 100
  Last update from 45.0.0.4 on Ethernet0/1, 00:00:43 ago
  Routing Descriptor Blocks:
  * 45.0.0.4, from 45.0.0.4, 00:00:43 ago, via Ethernet0/1
      Route metric is 284160, traffic share count is 1
      Total delay is 1100 microseconds, minimum bandwidth is 10000 Kbit
      Reliability 255/255, minimum MTU 1500 bytes
      Loading 10/255, Hops 1
R5#

R5 now is using the best route which is via R4 as its only 21 cost to R3 loopback!

Thanks, I hope this was useful!

Good Luck!

OSPF Path Selection types

Hi,

In this post I would like to give a simple note about OSPF path selection

some engineers wondering which one is correct?

RFC (1587)
Intra-Area(O) over Inter-Area(O IA) over External type (E1) over N1 over E2 over N2


OR


RFC (3101)
O over O IA over N1 over E1 over N2 over E2

well, both are correct, but according to your router ios version, because since (15.2S), Cisco started to use RFC3101 which uses O over O IA over N1 over E1 over N2 over E2 

while earlier software versions will use the Intra-Area(O) over Inter-Area(O IA) over External type (E1) over N1 over E2 over N2



Thanks

My first second and last CCIE attempt story

So this is my first attempt for CCIE, I didn't pass the exam, I would not call that was a fail, I learned something very important that day, which was I'm not ready yet!

so this is what happened, I booked for my exam in Dubai UAE, for July, the weather is really hot and since its a city near the sea, humidity was really high....

I booked in a hotel that is 2 minutes walking from Cisco exam center, yet these two minutes walking was like a marathon in Dubai weather.

anyhow, I arrived UAE three days before exam, which was really boring thing to do but this is what happens when you live in a country that doesnt have flights on daily basis to UAE.

Feelings before exam:
I was really clam and relaxed and that's what I thought before the exam, but after the exam I realized I was panicked but my brain managed to keep me clam somehow, until the moment the exam started.

so I arrived at Cisco building at 07:00 AM exactly, according to the recommendation, but we had to wait until 07:40AM until the proctor came, and the waiting is one of the factors that will start the panic process, the proctor took us to the exam room and it was a normal office, with normal temperature not like many people say that its cold.

the proctor didn't say that much, he seemed very quite.

btw, what I learned, you shouldnt need the proctor if you are really ready and really a CCIE, you should get into the exam, and you shouldnt have any question to be asked to proctor or tasks to be explained, you should sit and start solving the tasks until you finish it all.

anyhow, exam was really hard, the time was flying and I could solve almost nothing, I was surprised how people told me the exam is easy while what I saw was really different.

I learnt that you must be a CCIE before going into that room, passing the exam is just part of the process, its something that you have to do, the exam is really a test for your speed in
1. understand tasks
2. typing commands
3. understanding traffic flow and the idea behind the whole lab
4.time management
5. self-control

I passed one of the three sections only, which is (Diagnostic) while I failed the other two
the TSHOOT was the panic attack and I was all over the place, I forgot where to start and how to trace the problem which caused me failing this part

the Config section I remember wasting two hours in Layer2 config only.

then wasting another hour and a half on IGP


then I decided to give up and end the exam session, all I was thinking of was my wife and my kids, I missed them a lot, and it was really bad feeling watching all the time I did studying going like this in the exam
Ok, I walked out that building and I felt such satisfaction that I didnt understand why.
was is because of the pressure I put myself into? or was it because I did the exam and even if I failed I was ready that I will pass next time and the important thing is that I sat the exam and saw the exam?



I tried to enjoy Dubai until the flight date, I was really excited to go home and see my family.

the next thing to do was to set a new plan, I wanted to conquer CCIE and be a real one.

I started viewing my weak points, and set a plan to fill the gaps, until next attempt I must be 100% ready to pass the exam, no excuses and no panic

Next Attempt? February 2019.

 

update to this, I did not pass my second attempt in Feb. 2019, and I tried again few months later specifically in June 2019 and I passed the exam this time.

the feelings after passing the exam are unexplained, worked very hard to pass this monster exam.

it is time to move on now to learn more and prove myself as an expert and help others and payback the community.

Thanks

Samer R. Saleem