4 October 2016

Modifying, Compiling and Running the OpenBSD Kernel

This guide uses 64bit OpenBSD 6.0 as an example.

Generally there two trains for compiling OpenBSD;
1) Making changes to a particular release
2) Making changes to the current code HEAD

If you want to make changes to a particular release, you must first install the .iso for that image, and then get the source for that release + Errata patches (Stable).

If you want to play with the current HEAD, you MUST install the .iso from the latest "Snapshot".

Do not try and get to "Current" from a release image. It might work, but you will also have to follow the same manual steps that the Devs use (see Flag Day) to uplift the environment to match the HEAD code.

When compiling OpenBSD please make sure that /usr is at least 4GB to ensure enough space and inodes.

Install required packages;
#export PKG_PATH="http://ftp.mirrorservice.org/pub/OpenBSD/snapshots/packages/amd64/"   <- Current Binaries for Snapshot install
export PKG_PATH="http://ftp.mirrorservice.org/pub/OpenBSD/6.0/packages/amd64/"   <- Stable (Patches) for Release install
 

pkg_add -i vim-7.4.1467p1-no_x11 gawk gcc gcc-libs g++ cmake automake-1.15p0 autoconf-2.69p2 git gtest libtool

Setup CVS;
export CVSROOT=anoncvs@anoncvs.spacehopper.org:/cvs
Use a CVS mirror closest to you (https://www.openbsd.org/anoncvs.html)

export MIRROR="http://mirror.exonetric.net"
Use a packages mirror closest to you (https://www.openbsd.org/ftp.html)

Setup the Directories for compiling;
cd /usr
mkdir -p src xenocara ports
chmod 775 src xenocara ports
chgrp wsrc src xenocara ports


You can pre-load the source tree using tar.gz downloads, and then run CVS to just update the tree. This is a lot quicker than trying to download the entire tree via CVS from scratch.

NB;
src = Kernal source & Built in User land source 
xenocara = X system source
ports = Userland "ports" source (https://www.openbsd.org/faq/faq15.html)

Pre-loading CVS Tree;
cd /var/tmp
wget ${MIRROR}/pub/OpenBSD/6.0/src.tar.gz
wget ${MIRROR}/pub/OpenBSD/6.0/sys.tar.gz
tar -zxvf /var/tmp/src.tar.gz -C /usr/src/
tar -zxvf /var/tmp/sys.tar.gz -C /usr/src/
cd /usr


NB; Both src and sys make up /usr/src (they are just divided into two downloads, one for the Kernel source and User land source).

Update the CVS tree;
#cvs -qd $CVSROOT get src               # Get Current HEAD (if you didnt preload)
#cvs -d $CVSROOT up -Pd -C              # Update to Current HEAD
cvs -d $CVSROOT up -rOPENBSD_6_0 -Pd    # Update Release Stable (Errata Patches)


NB; replace the label "OPENBSD_6_0" with the release you are working with.

If you want to checkout the entire source tree (-current) without preloading tree 
cvs -qd $CVSROOT checkout -P src
cvs -qd $CVSROOT get -P src
cvs -qd $CVSROOT get -P ports
cvs -qd $CVSROOT get -P xenocara

If you want to apply Patches to the Source;
# http://ftp.openbsd.org/pub/OpenBSD/patches/6.0.tar.gz <- (Provides All Errata patches - Also provided by updating the source tree as above with "
-rOPENBSD_6_0")
cd /usr/src
wget http://bulabula.org/diffs/hfsc64.diff
patch -p0 < /var/tmp/hfsc64.diff

 or (for signed patches)
cd /var/tmp
wget http://ftp.openbsd.org/pub/OpenBSD/patches/6.0.tar.gz
tar -zxvf 6.0.tar.gz
signify -Vep /etc/signify/openbsd-60-base.pub -x /var/tmp/6.0/common/001_uvmisavail.patch.sig -m - | (cd /usr/src && patch -p0)
NB; "/var/tmp/6.0/common/001_uvmisavail.patch.sig" is one of the extracted signed patches.


 I was recently debugging an issue with OpenBSD which was output dropping many VLAN packets. So I edited /usr/src/sys/net/if_vlan.c and added debug lines around all the places where if_oerrors is incremented.
E.g;

Index: if_vlan.c
===================================================================
RCS file: /cvs/src/sys/net/if_vlan.c,v
retrieving revision 1.165
diff -u -p -r1.165 if_vlan.c
--- if_vlan.c   18 May 2016 03:46:03 -0000      1.165
+++ if_vlan.c   4 Oct 2016 10:25:57 -0000
@@ -290,12 +290,14 @@ vlan_start(struct ifnet *ifp)
                            (prio << EVL_PRIO_BITS));

                        if (m == NULL) {

                                ifp->if_oerrors++;

+                               printf("Output Error due to NULL mbuff\n");

                                continue;

                        }

                }


                if (if_enqueue(ifp0, m)) {

                        ifp->if_oerrors++;
+                       printf("Output Error from if_enqueue\n");
                        continue;
                }
                ifp->if_opackets++;

I then recompiled the kernel, rebooted, ran some traffic through the firewall until the VLAN output drop counters incremented.

I then examined dmesg to see that all of my drops were being generated by the test (if_enqueue(ifp0, m)) being TRUE, and not because the mBuff was NULL.

From here keep tracing the code path back adding debugging lines each time until you get to the root cause :)


Building the Kernel;
Move old Config out the way
#mv /usr/src/sys/arch/i386/conf/GENERIC /usr/src/sys/arch/i386/conf/GENERIC.old
#mv /usr/src/sys/arch/amd64/conf/GENERIC.MP /usr/src/sys/arch/amd64/conf/GENERIC.MP.old


cd /usr/src/sys/arch/amd64/conf

Add any Kernal Options you want;
#echo "option  ART" >> /usr/src/sys/arch/amd64/conf/GENERIC.MP
config GENERIC.MP
cd ../compile/GENERIC.MP
make clean
make depend
make
cp /bsd /bsd.old
#install -o root -g wheel -m 644 bsd /
make install

You should now reboot to boot into your newly compiled Kernel

Building the Userland;
Clean up Userland first if built previously (else Objects will be all over the place)
cd /usr/src
find . -type l -name obj | xargs rm
make cleandir


Build Userland
rm -rf /usr/obj/*
cd /usr/src
make obj
cd /usr/src/etc && env DESTDIR=/ make distrib-dirs
cd /usr/src
make build
make install


You should now reboot again to boot into your newly compiled Kernel with new User land

Extra notes;
# Setup Ports Source
cd /var/tmp
wget ${MIRROR}/pub/OpenBSD/snapshots/ports.tar.gz   # Update to Current
#wget ${MIRROR}/pub/OpenBSD/5.9/ports.tar.gz        # Stable (Patches)
tar -zxvf /var/tmp/ports.tar.gz -C /usr/
cd /usr/ports
cvs -d $CVSROOT up -Pd -C               # Current
#cvs -d $CVSROOT up -rOPENBSD_5_9 -Pd   # Stable (Patches)


# Setup X11 Source
cd /var/tmp
wget ${MIRROR}/pub/OpenBSD/5.9/xenocara.tar.gz
tar -zxvf /var/tmp/xenocara.tar.gz -C /usr/
cd /usr/xenocara/
cvs -d $CVSROOT up -Pd -C               # Update to Current
#cvs -d $CVSROOT up -rOPENBSD_5_9 -P    # Stable (Patches)



Extra references;
man release
https://www.openbsd.org/faq/faq5.html
https://www.openbsd.org/faq/faq10.html#Patches

3 October 2016

Modifying / Upgrading / Reloading a Cisco Router or Switch

Their are many ways to place routers and switches into "maintenance mode", however unlike other systems, this "maintenance" state has little to do with the device itself and more to do with what all the other networking devices on the network think of the device that is being worked on.

Their is no one-size-fits-all. I.e. you have to do whatever is required, according to the rest of your infrastructure, and the protocols used, to divert all network traffic away from and around the device.

Other mechanisms exist including ISSU (In-Service Software Upgrade), NSF/SSO (Non-Stop Forwarding)/(Stateless Switch Over) with GR (Graceful Restart), and NSR (Non-Stop Routing). These licensed technologies allow a router to be upgraded/reloaded etc without downtime, and are single node HA (High Availability) techniques.

The method described here, does not require hardware and software licenses for these premium features, and instead simply relies on having a well designed network with redundancy, whereby you simply push the production traffic onto other routers, leaving you free to take the device offline for as long as desired.

This rather coarse technique is especially useful if you want to make changes to the topology and neighbour relationships that a router might have. 
 

        1) Backup Config and Firmware. (scp preferred)
        2) Place router into maintenance mode. (divert all traffic away from and around this router)
        3) Upgrade router firmware and Reload. (if upgrading)
        4) Make architecture/config changes. (service affecting changes)
        5) Checks; BFD peers, OSPF adjacencies, Tunnels, and MPLS neighbors etc.
        6) Take Router out of maintenance mode. (OSPF and BGP)


1) Backup Config and Firmware

a) copy running-config startup-config <- Saves config to NVRAM/Flash

b) Enable SCP server on Cisco; 
This assumes you already have SSH version 2 etc working. To allow SCP file copies to work, you need to be able to login and be dropped directly into Privileged mode (enable is not needed). 

aaa new-model 
aaa authentication login default local
username <user> privilege 15 secret <secretpassword>
aaa authorization exec default local <- Drops you at '<host>#' without requiring 'enable' (recommended to remove this after works are complete)
ip scp server enable <- Enable SCP (recommended to remove this after works are complete)

c) Backup existing Firmware, Configuration and any other important files

scp admin@172.16.1.1:flash:/c3825-adventerprisek9-mz.150-1.M1.bin /var/tmp
scp admin@172.16.1.1:flash:/vlan.dat /var/tmp
scp admin@172.16.1.1:nvram:/startup-config /var/tmp

NB; Many systems will have running-config and startup-config in flash: Use the command dir all to find yours.

d) Upload new firmware (if required)

scp /var/tmp/c3825-adventerprisek9-mz.151-4.M10.bin admin@172.16.1.1:flash:/c3825-adventerprisek9-mz.151-4.M10.bin

e) Verify your uploaded firmware's MD5 matches the one shown on the Cisco site;

verify /md5 filesystem:filename [md5-hash]

2) Place Cisco router into Maintenance mode

 The general idea behind preparing a router for maintenance work is to make the all of the router's routes less preferable at the control plane, whilst keeping the forwarding plane fully functional (thus minimising change impact and route flapping).

 For IGP's (OSPF in this example), this is done by increasing all the route costs/metrics, causing other routers to recalculate their SPF's and converge traffic gracefully and without loss via other paths if available, for BGP this is done by gracefully closing all BGP neighbours.

 If you have combined routers and switches (e.g. Cisco ME3600X's), and you are using layer 2 FHRP's like VRRP, you will also need to ensure to move the FHRP Layer 2 master roles away (and possibly any connected customer equipment roles too).

 Should the maintenance expose some priority traffic which is still flowing over the router, pause the works as it's ideal an opportunity to fix and analyse any broken redundancy, before doing any reloads and actually breaking the forwarding plane ;)


Before you begin:
 It is recommended to perform as many checks as possible on the router being worked on, and all the routers connected to it, to ensure that all IGP and EGP peering's are up, and that all routes-maps, prefix-lists, redistributions etc are correct, to ensure other routers can take over all the traffic when the router under maintenance goes down.

 It is best to connect to a routers loopback address for maintenance work.

 And it's also a good idea to use "configure terminal revert timer 10" before you start and "configure confirm" after before reloading, to automatically rollback any changes should something bad happen.


 a) BGP (Layer 3 EGP);
router bgp <ASNumber>
 bgp graceful-shutdown all neighbors 200 local-preference 10 <- Send GSHUT, set Local Preference on all routes to 10, and close neighbour sessions after 200 seconds
 bgp graceful-shutdown all neighbors activate <- Activate gracefull-shut config above allowing lossless re-convergence to other paths while this router is still active.


NB; Wait at least 200 seconds before making the router unavailable or unable to route packets to ensure that no neighbours are holding routes towards this router.


 b) OSPF (Layer 3 IGP); 
router ospf 1
 max-metric router-lsa <- Increase the Router LSA Metric
 auto-cost reference-bandwidth 4294967 <- Increase all the path costs
 no default-information originate metric-type 1
 no default-information originate <- Assuming you are originating a default in IGP on this router etc..

 c) VRRP (Layer 2 FHRP);
Verify all VRRP states are "Backup".
 show vrrp
 show vrrp brief

If router is "Master", you will need to reduce the VRRP priority;
E.g. for the VRRP IP on a Vlan 26 SVI
interface Vlan 26
 vrrp 26 priority 1

You may also need to check that customer devices which are using these VRRP Next Hops have also switched their master roles away off the switches being worked on.



3/4) Make configuration changes / Upgrade / Reload router

Do your thang.
 ... reload (etc)

5) Checks

show ip interface brief
show interface status
show bfd neighbors
show ip ospf neighbor
show ip interface tunnel 0
show mpls ldp neighbor
etc...

6) Take Router out of maintenance mode 

router ospf 1
 default-information originate metric-type 1
 auto-cost reference-bandwidth 10000
 no max-metric router-lsa
 max-metric router-lsa on-startup wait-for-bgp <- Set OSPF Router LSA Metric to Max until BGP has settled down and recieved all routes

router bgp 60868
 no bgp graceful-shutdown all neighbors activate <- Bring back all BGP sessions


Dont forget to restore your Layer 2 HSRP/VRRP priorities..



http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/sec_usr_ssh/configuration/15-s/sec-usr-ssh-15-s-book/sec-secure-copy.html 
http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_bgp/configuration/xe-3s/irg-xe-3s-book/configuring-bgp-graceful-shutdown.pdf
http://www.cisco.com/c/en/us/about/security-center/ios-image-verification.html


1 October 2016

Working with QEMU

Convert VM from RAW to QCOW2
qemu-img convert -O qcow2 /dev/main/nms2 /mnt/backups/nms2.qcow2

When using RAW based disks for VM's, you will need to backup the LVM directly with snapshots. Otherwise if using QCOW2 format on a filesystem, you will be able to perform snapshots and backups at the QEMU level.

Copy RAW image from RAW on one machine to RAW partition on another machine
dd if=/dev/main/lvm1 | ssh <remoteserver> dd of=/dev/zvol/rpool/data/vm-100-disk-1

General virsh Commands
virsh list --all
virsh start <vm>
virsh autostart <vm>
virsh shutdown <vm>

MD RAID Performance with LVMs and RAW
When you dont have a RAID controller and only have spinning disks, and you want to host some VMs for example, you will have to be careful with the very limited disk IO. Highly recomend getting SSDs for VM Hypervisors if you cannot afford a proper hardware RAID card.

In particular you want to look at using "far 2" layout instead of the default "near 2" layout with MD RAID 1.
This means you will be reading from both disks at the same time, obviously writes can only go to one disk at a time.

I.e. Build a new Linux server using only a small partition using the Linux installer or other install system like FAI etc.
Then create a new software MD raid volumne using "far 2", before setting up your LVMs on top.

ZFS is probably the most advanced storage type regarding snapshot and cloning.
https://www.howtoforge.com/tutorial/how-to-install-and-configure-zfs-on-debian-8-jessie/

zpool list
zpool status
zpool create -f pool0 /dev/sdb
zpool create -f pool1 mirror /dev/sdc /dev/sdd

Need a full ZFS notes write up here..