Cisco Debug Stuff
1) Safe(r) Debugging
Any network engineer should be aware of the overhead debugging creates on the CPU as it demands packets are process switched, especially on a Cisco router which can cause it to stall or crash on busy systems.
However if you use buffered logging to a file instead, whilst still hard work for the device, the CPU will have a much better time and is less likely to crash;
no logging console debug <- Exclude debug (7) logs from console
logging console 4 <- Enable log to console @syslog (0 - 4)
NB; Logging to the Console is dangerous and many crashes happen due to log write speed being far greater than the TTY console speed thus overflowing the TTY buffer.
logging buffered debug <- Enable temp log to file buffer @syslog (0 - 7)
logging buffered 16384 debugging <- Make the rolling log longer
clear log <- Clear the log
debug dhcp detail <- Enable some debugging
debug ip dhcp server event <- Enable some debugging
show log <- Show the log
show debug <- Show which debug functions are enabled
undebug all <- Turn off all debugging
NB; debug's are syslog level 7 messages (the highest of the 8 levels).
Hint; The above example will show you the elusive DHCP Client ID / Fingerprint for a client, to allow you to define a static DHCP reservation on Cisco routers! ;)
2) Debugging packets to/from a specific Host
access-list 101 permit tcp host 172.16.0.10 gt 1024 10.10.0.0 0.0.255.255 eq 22 <- E.g. match src 172.16.0.10 accessing servers on 22
interface gigabitEthernet 0/0
no ip route-cache <- Force "process switching" on
terminal monitor <- If SSH session
logging console <- If Console session
debug ip packet 101 detail
undebug all <- Turn of all debugging processes
Cisco Logging and Tracking Stuff
Console Logging
[no] logging console <- Log to the console
[no] logging buffer <- Log to internal memory buffer
terminal monitor <- Clone all logs to the Terminal TTY (SSH Connection)
logging monitor informational <- Show all logs on SSH TTY from "0 to 6"
line vty 0 15
(config-line)#logging synchronous <- Make logs and CLI input display nicely
logging 10.2.2.2 <- Send logs to remote syslog server
logging console 6 <- This shows all logs from 0 - 6 (excludes 7)
1) Persistent Syslog Logging (Crash/Assert Debugging)
logging buffered 40960
logging persistent url flash:/syslogs size 131072 filesize 32768
Logs should now be written to flash, making them available after a reload or assert fail etc. I successfully relied on this recently to debug an SSL-VPNv3 AIM which kept crashing a router when users sent lots of data over an SSL tunnels. Turned out to be an software issue with DTLS.
Often logs never reach the syslog server when assert fails are occurring, and so local logging may be the only way to find the root cause of the assert.
2) Persistent Logging of all user CLI commands (Auditing)
Cisco supports tracking and logging of all CLI VTY commands.
archive
log config
record rc
logging enable <- Enable command logging
notify syslog contenttype plaintext <- Notify terminal
To see the command history typed by all admins;
rt2#show archive log config all
idx sess user@line Logged command
125 16 admin@vty0 |interface GigabitEthernet0/1
126 16 admin@vty0 | ip ospf 1 area 0
127 16 admin@vty0 | ipv6 ospf 1 area 0
128 16 admin@vty0 | router ospf 1
129 16 admin@vty0 | no router-id
130 16 admin@vty0 | router-id 172.16.31.254
3) Persistent Configuration History (Configuration Rollbacks)
Cisco supports being able to store a rolling log of up to 14 previous configurations, and also supports replace and rollback functionality;
archive
path flash:/startup-config <- Backup location
maximum 14 <- Number of previous configs to remember
rollback filter adaptive <- Enable rollback
write-memory <- Enable auto config backup during 'wr'
Each time you save with "write memory" or "copy run start", the configuration will be saved with a time stamp to the path location.
To see all saved previous configurations;
rt2#show archive
The maximum archive configurations allowed is 14.
There are currently 2 archive configurations saved.
The next archive file will be named flash:/startup-config-<timestamp>-2
Archive # Name
1 flash:/startup-configOct-13-20-32-53.733-BST-0
2 flash:/startup-configOct-13-20-40-49.870-BST-1 <- Most Recent
To load a specific previous configuration immediately;
configure replace flash:startup-config-Oct-20-22-56-13.799-BST-1
4) Timed Configuration Rollbacks
Do not use "reload in X" to protect from critical changes gone wrong, use automatic rollbacks instead with 'confirmed' changes (Introduced in 12.3(7)T IOS). revert is similar to replace however provides a transactional log style context over full config replace.
1) To rollback to a specific previous configuration automatically (WITHOUT RELOADING!); You will have 5 minutes to make your changes, and if the changes are a success, you must run "configure confirm" to accept your changes before the 5 minutes is up to cancel the rollback.
configure replace flash:startup-config-Oct-20-22-56-13.799-BST-1 time 5
Make your risky changes ... When happy, cancel the rollback with;
configure confirm
To rollback immediately;
configure revert now
2) To automatically rollback only the most recent changes made within a 10 minute window, if "configure confirm" is not typed before the window is over;
configure terminal revert timer 10
Make your risky changes
configure confirm
When entering a change period/window like this, the CLI configuration is locked to stop other VTY admins making changes at the same time.
After confirming the commit, or after the rollback, the CLI lock is automatically released.
To see any locks;
show configuration lock
NB; You will receive a "rolling back in 1 minute" warning in both examples.
For production deployments I recommend also installing "Rancid" (http://www.shrubbery.net/rancid/) or similar to maintain a config history with CVS/SVN/GIT etc, located on a different host for redundancy. For smaller networks, or networks with only a few devices, storing a history of configs on the flash: is often enough.
General Helpful Cisco Things
1) Command aliases
Built in aliases;
- p for ping.
- h for help.
- lo for logout.
- u for undebug.
- w for where.
Create custom aliases to type less;
alias exec ... <- Privileged Mode (router# prompt).
alias configure ... <- Global Configuration Mode (router(config)# prompt).
alias interface ... <- Interface Configuration Mode (router(config-if)# prompt).
Here are the aliases I use;
alias interface ns no shutdown
alias exec s show run
alias exec save copy running-config startup-config
alias exec siib show ip interface brief
alias exec sis show interface status
alias exec sir show ip route
alias exec si6ib show ipv6 interface brief
alias exec si6r show ipv6 route
alias exec c configure terminal
alias configure b router bgp 9999
2) Cisco Flash
Upgrading the flash memory
1) Backup the existing Flash to tftp server;
archive tar /create tftp://<tftp_server>/flashbackup.tar flash:
2) Insert new (hopefully larger) Flash card.
3) Erase compact flash memory card (Class B Format - Low End File System);
erase slot0:
erase flash:
NB; The Class B system is very basic and slow (r/w), but you can often use cards larger than the officially supported maximum, if you do not format. You should format if size is supported.
4) Format compact flash memory card (Class A/C Format - DOS File System);
format slot0:
format flash:
NB; slotX: is usually external, and flash: is usually internal. But on many models flash: also refers to an external slot. Yay cisco consistency.
5) Restore flash backup to new flash;
archive tar /xtract tftp://<tftp_server>/flashbackup.tar flash:
3) Working with the flash
Read/print any local text file in flash out directly to VTY console;
more flash:syslogs/log_20160928-211019
Show all storage systems;
dir all
Delete storage directories;
delete /f /r flash:/<directory>
4) Copying files to / from flash
a) Enable SCP server on Cisco;
This assumes you already have SSH version 2 etc working. To allow SCP file copies to work, you need to be able to SSH in and be dropped directly into Privileged mode (enable is not needed).
aaa new-model
aaa authentication login default local
username <user> privilege 15 secret <secretpassword>
aaa authorization exec default local <- Drops you at '<host>#' without requiring 'enable' (negate this after upgrade complete)
ip scp server enable <- Enable SCP (negate this after works are complete)
b) Copy from Flash to Computer
scp admin@172.16.1.1:flash:/c3825-adventerprisek9-mz.150-1.M1.bin /var/tmp
c) Copy from Computer to Flash
scp /var/tmp/c3825-adventerprisek9-mz.151-4.M10.bin admin@172.16.1.1:flash:/c3825-adventerprisek9-mz.151-4.M10.bin
NB; Adding "flash:" in the commands may not be required on all systems.
d) Verify your uploaded firmware's MD5 matches the one shown on the Cisco site;
verify /md5 filesystem:filename [md5-hash]
http://www.cisco.com/c/en/us/td/docs/ios/fundamentals/configuration/guide/15_1s/cf_15_1s_book/cf_config-rollback.html
