Basic but useful & set-and-forget Cisco stuff

Cisco Debug Stuff

1) Safe(r) Debugging

Any network engineer should be aware of the overhead debugging creates on the CPU as it demands packets are process switched, especially on a Cisco router which can cause it to stall or crash on busy systems.

However if you use buffered logging to a file instead, whilst still hard work for the device, the CPU will have a much better time and is less likely to crash;

no logging console debug   <- Exclude debug (7) logs from console
logging console 4          <- Enable log to console @syslog (0 - 4)
NB; Logging to the Console is dangerous and many crashes happen due to log write speed being far greater than the TTY console speed thus overflowing the TTY buffer.

logging buffered debug     <- Enable temp log to file buffer @syslog (0 - 7)
logging buffered 16384 debugging <- Make the rolling log longer

clear log                  <- Clear the log

debug dhcp detail          <- Enable some debugging

debug ip dhcp server event <- Enable some debugging

show log                   <- Show the log

show debug                 <- Show which debug functions are enabled

undebug all                <- Turn off all debugging

NB; debug's are syslog level 7 messages (the highest of the 8 levels).

Hint; The above example will show you the elusive DHCP Client ID / Fingerprint for a client, to allow you to define a static DHCP reservation on Cisco routers! ;)




2) Debugging packets to/from a specific Host

access-list 101 permit tcp host 172.16.0.10 gt 1024 10.10.0.0 0.0.255.255 eq 22   <- E.g. match src 172.16.0.10 accessing servers on 22
interface gigabitEthernet 0/0

no ip route-cache   <- Force "process switching" on

terminal monitor    <- If SSH session

logging console     <- If Console session

debug ip packet 101 detail

undebug all  <- Turn of all debugging processes

Cisco Logging and Tracking Stuff

Console Logging

[no] logging console  <- Log to the console
[no] logging buffer   <- Log to internal memory buffer
terminal monitor      <- Clone all logs to the Terminal TTY (SSH Connection)
logging monitor informational <- Show all logs on SSH TTY from "0 to 6"
line vty 0 15
(config-line)#logging synchronous <- Make logs and CLI input display nicely
logging 10.2.2.2      <- Send logs to remote syslog server

logging console 6     <- This shows all logs from 0 - 6 (excludes 7)




1) Persistent Syslog Logging (Crash/Assert Debugging)

 logging buffered 40960
 logging persistent url flash:/syslogs size 131072 filesize 32768

Logs should now be written to flash, making them available after a reload or assert fail etc. I successfully relied on this recently to debug an SSL-VPNv3 AIM which kept crashing a router when users sent lots of data over an SSL tunnels. Turned out to be an software issue with DTLS.

Often logs never reach the syslog server when assert fails are occurring, and so local logging may be the only way to find the root cause of the assert.


 2) Persistent Logging of all user CLI commands (Auditing)

Cisco supports tracking and logging of all CLI VTY commands.

archive
 log config
  record rc
  logging enable             <- Enable command logging
  notify syslog contenttype plaintext  <- Notify terminal

To see the command history typed by all admins;
rt2#show archive log config all
 idx   sess           user@line      Logged command
  125    16          admin@vty0     |interface GigabitEthernet0/1
  126    16          admin@vty0     | ip ospf 1 area 0
  127    16          admin@vty0     | ipv6 ospf 1 area 0
  128    16          admin@vty0     | router ospf 1
  129    16          admin@vty0     | no router-id
  130    16          admin@vty0     | router-id 172.16.31.254



3) Persistent Configuration History (Configuration Rollbacks)

Cisco supports being able to store a rolling log of up to 14 previous configurations, and also supports replace and rollback functionality;

archive  
 path flash:/startup-config  <- Backup location
 maximum 14                  <- Number of previous configs to remember
 rollback filter adaptive    <- Enable rollback
 write-memory                <- Enable auto config backup during 'wr'

Each time you save with "write memory" or "copy run start", the configuration will be saved with a time stamp to the path location.

To see all saved previous configurations;

 rt2#show archive
The maximum archive configurations allowed is 14.
There are currently 2 archive configurations saved.
The next archive file will be named flash:/startup-config-<timestamp>-2
 Archive #  Name
   1        flash:/startup-configOct-13-20-32-53.733-BST-0
   2        flash:/startup-configOct-13-20-40-49.870-BST-1 <- Most Recent

To load a specific previous configuration immediately; 

 configure replace flash:startup-config-Oct-20-22-56-13.799-BST-1

4) Timed Configuration Rollbacks

Do not use "reload in X" to protect from critical changes gone wrong, use automatic rollbacks instead with 'confirmed' changes (Introduced in 12.3(7)T IOS). revert is similar to replace however provides a transactional log style context over full config replace.

 

1) To rollback to a specific previous configuration automatically (WITHOUT RELOADING!); You will have 5 minutes to make your changes, and if the changes are a success, you must run "configure confirm" to accept your changes before the 5 minutes is up to cancel the rollback.

 configure replace flash:startup-config-Oct-20-22-56-13.799-BST-1 time 5

Make your risky changes ... When happy, cancel the rollback with;

 configure confirm

To rollback immediately;

 configure revert now

2) To automatically rollback only the most recent changes made within a 10 minute window, if "configure confirm" is not typed before the window is over;

 configure terminal revert timer 10
Make your risky changes
 configure confirm

When entering a change period/window like this, the CLI configuration is locked to stop other VTY admins making changes at the same time.
After confirming the commit, or after the rollback, the CLI lock is automatically released.

To see any locks;
 show configuration lock

NB; You will receive a "rolling back in 1 minute" warning in both examples.

For production deployments I recommend also installing "Rancid" (http://www.shrubbery.net/rancid/) or similar to maintain a config history with CVS/SVN/GIT etc, located on a different host for redundancy. For smaller networks, or networks with only a few devices, storing a history of configs on the flash: is often enough. 

General Helpful Cisco Things

1) Command aliases

Built in aliases;

• p for ping.
• h for help.
• lo for logout.
• u for undebug.
• w for where.

Create custom aliases to type less;
alias exec ...      <- Privileged Mode (router# prompt).
alias configure ... <- Global Configuration Mode (router(config)# prompt).
alias interface ... <- Interface Configuration Mode (router(config-if)# prompt).

Here are the aliases I use;
alias interface ns no shutdown

alias exec s show run
alias exec save copy running-config startup-config

alias exec siib show ip interface brief
alias exec sis show interface status
alias exec sir show ip route

alias exec si6ib show ipv6 interface brief
alias exec si6r show ipv6 route

alias exec c configure terminal
alias configure b router bgp 9999


2) Cisco Flash

Upgrading the flash memory

1) Backup the existing Flash to tftp server;

archive tar /create tftp://<tftp_server>/flashbackup.tar flash:

2) Insert new (hopefully larger) Flash card.

3) Erase compact flash memory card (Class B Format - Low End File System);
erase slot0:
erase flash:

NB; The Class B system is very basic and slow (r/w), but you can often use cards larger than the officially supported maximum, if you do not format. You should format if size is supported.

4) Format compact flash memory card (Class A/C Format - DOS File System);

format slot0:

format flash:

NB; slotX: is usually external, and flash: is usually internal. But on many models flash: also refers to an external slot. Yay cisco consistency.

5) Restore flash backup to new flash;

archive tar /xtract tftp://<tftp_server>/flashbackup.tar flash:

3) Working with the flash

Read/print any local text file in flash out directly to VTY console;

more flash:syslogs/log_20160928-211019

Show all storage systems;

dir all

Delete storage directories;

delete /f /r flash:/<directory>

4) Copying files to / from flash

a) Enable SCP server on Cisco; 

This assumes you already have SSH version 2 etc working. To allow SCP file copies to work, you need to be able to SSH in and be dropped directly into Privileged mode (enable is not needed). 

aaa new-model 

aaa authentication login default local

username <user> privilege 15 secret <secretpassword>

aaa authorization exec default local <- Drops you at '<host>#' without requiring 'enable' (negate this after upgrade complete)

ip scp server enable <- Enable SCP (negate this after works are complete)

b) Copy from Flash to Computer

scp admin@172.16.1.1:flash:/c3825-adventerprisek9-mz.150-1.M1.bin /var/tmp

c) Copy from Computer to Flash

scp /var/tmp/c3825-adventerprisek9-mz.151-4.M10.bin admin@172.16.1.1:flash:/c3825-adventerprisek9-mz.151-4.M10.bin

NB; Adding "flash:" in the commands may not be required on all systems.

d) Verify your uploaded firmware's MD5 matches the one shown on the Cisco site;

verify /md5 filesystem:filename [md5-hash]

http://www.cisco.com/c/en/us/td/docs/ios/fundamentals/configuration/guide/15_1s/cf_15_1s_book/cf_config-rollback.html