15 October 2016

Linux Tips

Network Transfers
Copying files from one machine to another machine using stdin/stdout tar -cf - /backup/dir | ssh <remotehost> "cat -> backupfile.tar"

LVM/Block devices
dd if=/dev/blkdevice | ssh <remotehost> dd of=/dev/blkdevice

Without SSH
Reciever;
nc -l -p 5000 > /path/backupfile.tar
Sender;
tar -cf - /backup/dir | nc remotehost 5000




Disk Performance
Test disk read / write performance

hdparm -Tt /dev/sda

dd if=/dev/urandom of=/var/tmp/testout conv=fdatasync bs=8k count=10k; rm -f /var/tmp/testout
/dev/urandom performs badly, but gives a better measure for SSD performance. If disk speed > urandom, pre-load a file (in RAM) with urandom.

dd if=/dev/zero of=/var/tmp/testout conv=fdatasync bs=8k count=10k; rm -f /var/tmp/testout

Monitor disk read/write utilisation

sar -dp 1 3
iotop





Oracle Java.. (Replace OpenJDK)

Easy Method
echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main
deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" > /etc/apt/sources.list.d/java-8-debian.list

 

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886
apt-get update
apt-get install oracle-java8-installer
apt-get install oracle-java8-set-default

Manual method
apt-get purge openjdk*

Download JRK/JDK onto the server
http://www.oracle.com/technetwork/java/javase/downloads/index.html

mkdir -p /usr/local/java

cp ./jdk-8u102-linux-x64.tar.gz /usr/local/java
cd /usr/local/java/
tar -zxvf jdk-8u102-linux-x64.tar.gz
chown root:root jdk1.8.0_102 * -R
cd jdk1.8.0_102

echo "JAVA_HOME=`pwd`" >> ~/.profile
echo "PATH=$PATH:$HOME/bin:$JAVA_HOME/bin" >> ~/.profile
echo "export JAVA_HOME" >> ~/.profile
echo "export PATH" >> ~/.profile
source ~/.profile

update-alternatives --install "/usr/bin/java" "java" "/usr/local/java/jdk1.8.0_102/bin/java" 1

update-alternatives --install "/usr/bin/javac" "javac" "/usr/local/java/jdk1.8.0_102/bin/javac" 1
update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/local/java/jdk1.8.0_102/bin/javaws" 1 <- NB; javaws does not exist in server java package

update-alternatives --set java /usr/local/java/jdk1.8.0_102/bin/java

update-alternatives --set javac /usr/local/java/jdk1.8.0_102/bin/javac
update-alternatives --set javaws /usr/local/java/jdk1.8.0_102/bin/javaws

reboot


Verify with 
java -version


Multiple Architectures (i386/i686 & 64)
Their is still a significant amount of software out there that is compiled as only 32bit! :(
This means you need to install 32bit libraries onto your 64bit Kernel and OS.

Debian/Ubuntu;
dpkg --print-architecture
dpkg --print-foreign-architectures

If NULL, enable multi-arch with;
dpkg --add-architecture i386
apt-get update





(To be finished)..


13 October 2016

HP Procurve 802.1X OpenLDAP and freeRADIUS - Dynamic VLANs

802.1X is one of those technologies you know you need, but the entry barrier can be quite high and RADIUS is anything but simple.

But once you have it working, the time it will save you on OpEx, and the number of security incidents you will reduce, makes it a no-brainer

802.1X works a lot better on many other switch Vendors, but HP has a massive foot print in SMB's and remote offices, and those are the locations where you really need it.

802.1X can be deployed in a few different ways depending on what you want to do.
This guide discusses setting up 802.1X for MAC Address Authentication to dynamically configure the Access port with the correct tagged and untagged VLANs, and port CoS priority based on the MAC address the switch receives.

Web/SSH Client Authentication, is not discussed here but the idea is to authenticate the User, where MAC Auth is for the hardware. If you want to get serious you can do both.. (recommended)

NB; With the advent of USB and Thunderbolt plugable NIC adaptors, MAC based auth is not a secure method, but it can still be used for managing VLAN assignment, and helping with general port level security.


  • Provides a centralised MAC<->VLAN management (OpenLDAP).
  • Requires communication with central store via RADIUS (FreeRADIUS).
  • Utilises standards based RADIUS ‘auth’ and implements the standards based VLAN assignment policies.
  • RFC3580 for untagged VLAN assignment and RFC4675 for tagged VLAN assignment.
  • Supports per-port CoS assignment via LDAP for QoS (RFC4675).
  • Supports many other features including configuring per-port ACLs and rate limiting.


Setup LDAP

LDAP is just one of many data storage back ends which RADIUS supports.


Install OpenLDAP;
aptitude install dkms
aptitude install slapd
aptitude install ldap-utils
We are going to backup and restore a production system into the lab for testing.

On new server move default /etc/ldap/slapd.d out of the way to /etc/ldap/slapd.d.orig

Copy LDAP schema's, certs and slapd.conf (/etc/ldap/*) from the existing OpenLDAP server (/etc/ldap/) to the new one. NB; set permissions to root:openldap on copied files


slapcat > /tmp/my.ldif (on existing LDAP server, dump backup)

slapadd -l /tmp/my.ldif (on new LDAP server LAB CLONE, dump import)



Test the new LDAP server
ldapsearch -x -b '' -s base '(objectclass=*)' namingContexts

  • You should now have a standalone Cloned Copy of the current LDAP tree
For production at least 2 LDAP servers need to be setup as replicated members of the existing LDAP infrastructure.

 



Setup RADIUS with LDAP

FreeRADIUS provides encryption, and a standardised interface for authentication. LDAP is just the back end, and so we will later connect RADIUS to LDAP, so that RADIUS is doing nothing more than providing the auth protocols for the Switches. RADIUS also has its own data stores, but these are not recommended unless you know what you are doing.

Install FreeRADIUS;
aptitude install freeradius
aptitude install freeradius-ldap

Configure OpenLDAP for FreeRADIUS;

cp /usr/share/doc/freeradius/examples/openldap.schema /etc/ldap/schema/radius-openldap.schema
Copy the default ‘freeradius’ LDAP schema into OpenLDAP for inclusion (to provide RADIUS elements to your LDAP objects);



Enable the new schema;
edit /etc/ldap/slapd.conf adding;
include /etc/ldap/schema/radius-openldap.schema

Now lets add some custom HP Specific attributes to the main LDAP schema with the correct data types, so that we can use tokens which fit more nicely with HP's interpretation of the RFC's.


Ensure the attributes ‘radiusHPCoS’ and ‘radiusMacAddress’ are defined and referenced in the local.schema file;
attributetype ( 1.3.6.1.4.1.37556.1.1.3.17 NAME 'radiusHPCoS'
       DESC '802.1p CoS mapping'
       EQUALITY integerMatch
       SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 SINGLE-VALUE )


attributetype ( 1.3.6.1.4.1.37556.1.1.3.16 NAME 'radiusMacAddress'

       DESC 'MAC Address'

       EQUALITY caseIgnoreIA5Match
       SUBSTR caseIgnoreSubstringsMatch
       SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 SINGLE-VALUE )
.
.
MAY ( radiusHPCoS $ radiusMacAddress $ …..


Notice that you need to addradiusHPCoS $ radiusMacAddress $not replace the line starting MAY.

service slapd restart
You should now have an LDAP schema which can support many new RADIUS related attributes/properties which can be defined in the LDAP objects.

Configure FreeRadius;

Edit /etc/freeradius/modules/ldap and configure FreeRadius to connect to OpenLDAP on localhost
server = "localhost"
identity = "cn=root,dc=base,dc=runtime-collective,dc=com"
password = <!LDAP Password!>
basedn = "dc=base,dc=runtime-collective,dc=com"
filter = "(radiusMacAddress=%{%{Stripped-User-Name}:-%{User-Name}})"
base_filter = "(objectclass=radiusprofile)"

ldap_connections_number = 10



Remove the default ldap connection if present;
rm /etc/freeradius/modules/ldap.dpkg-dist
NB; the "filter" statement ensures that the ‘User-Name’ RADIUS attribute maps to the new ‘radiusMacAddress’ LDAP attribute for the LDAP search.


Edit /etc/freeradius/sites-enabled/default uncomment ‘ldap
Edit /etc/freeradius/sites-enabled/inner-tunnel uncomment ‘ldap


Edit /etc/freeradius/clients.conf and define all the network clients (Wireless APs, Switches, any devices which need to authorise users/machines);

client 10.0.0.242 {

       ipaddr = 10.0.0.242
       secret = testing123
       shortname = 10.0.0.242
       nastype = "other"
}
Configure the RFC3580 Radius attributes, to map onto the new custom LDAP attributes as per local.schema (NB; When running MAC driven AAA port-based authentication, the MAC address of the conencting computer is used for both the User and Pass);

edit /etc/freeradius/ldap.attrmap and add the following lines;
checkItem User-Name radiusMacAddress
checkItem Cleartext-Password radiusMacAddress
replyItem HP-COS radiusHPCoS

service freeradius restart

 

Configure LDAP Objects


Configure the LDAP client object adding the new attributes with desired VLAN and CoS values;

objectClass: hardwareClass <- Required
objectClass: radiusprofile <- Required
dialupAccess: access_attr <- Required
radiusHPCoS: 3 <- Optional - CoS queue (see table below)
radiusMacAddress: 6805ca07e4ef <- Required - MAC Address (Must be lower-case without any special chars (if you want ":" formatted etc, add commands to the switch))
radiusTunnelMediumType: "IEEE-802" <- Required
radiusTunnelPrivateGroupId: 11 <- Required - Desired VLAN (Decimal) (VLAN must already exist on switches)
radiusTunnelType: VLAN <- Required

 Table 1: radiusHPCoS values (optional);
 
802.1p Value
Traffic Class
Queue (W firmware)
Queue (K firmware)
Priority
1
Background
1
1
Low
2
Spare
1
2
Low
0
Best Effort
2
3
Normal
3
Excellent Effort
2
4
Normal
4
Controlled Load
3
5
High
5
Video
3
6
High
6
Voice
4
7
Critical
7
Network Control
4
8
Critical

 


Setup HP Procurve Switches

Configure HP Procurve switches to use RADIUS for Dynamic VLAN assignment;

(DO NOT ENABLE MAC AUTH ON TRUNK PORTS, ONLY ACCESS PORTS)

max-vlans 256 <- Set as high as switch allows

crypto key generate cert 1024
crypto host-cert generate self-signed
radius-server host 10.0.0.100 <- Define multiple times for multiple Radius servers
radius-server key "testing123" <- Radius server encryption key (clients.conf)


aaa accounting update periodic 15 <- Optional - Radius Accounting

aaa accounting exec start-stop radius <- Optional - Radius Accounting

aaa accounting network start-stop radius <- Optional - Radius Accounting
aaa accounting system start-stop radius <- Optional - Radius Accounting

aaa port-access mac-based 1-48 <- Enable MAC based RADIUS auth on ports 1-48
aaa port-access mac-based 1-48 auth-vid 1 <- Set untagged VLAN=1 if auth success, but no VLAN ID provided by RADIUS.
aaa port-access mac-based 1-48 unauth-vid 1
aaa port-access mac-based 1-48 logoff-period 9999999
aaa port-access mac-based 1-48 quiet-period 30
aaa port-access mac-based 1-48 addr-limit 32
aaa port-access mac-based 1-48 addr-moves
aaa port-access 1-48 controlled-direction in
spanning-tree 1-48 bpdu-protection <- Shut port if BPDU received
spanning-tree 1-48 admin-edge-port <- Important to ensure port works properly, but enables immediate port forwarding (port-fast equivalent).


Config rules;
The RADIUS connection timeout must be less than the authentication server timeout for the switch to authenticate automatically when the RADIUS server is unavailable.


Ensure all VLANs set in LDAP are manually defined in switch running config, and north-bound trunk ports to spine/core allow tagged packets for all the VLANs.


  • Ports are now assigned to Untagged VLAN IDs by RADIUS, or defaults to Untagged VLAN 1

Configure IPMI and iLO2 hosts;


Reset to defaults and ensure DHCP enabled.
Ensure IPMI failover mode is set to dedicated in the BIOS (do not use IPMI failover with Supermicro to share the port as does not fail-back).
VLAN tagging with BCM systems normally does not work. We solved this by running VLAN tags on all servers, and running IPMI untagged. This way when a race occurs, the port is simply setup for both tagged X (for Linux) and untagged Y (for the BMC) thus resolving the race.


Configure Linux hosts (Optional);

By default Linux sends only a single GARP on Ethernet carrier up (link up), and so ifplugd is sometimes needed to send enough frames to ensure the switch port is fully configured by 802.1X (For example, if the host sends no frames, the host will be unreachable by other hosts until this host sends at least one frame). DHCP client would achieve the same effect.
install ifplugd

edit /etc/default/ifplugd;

INTERFACES="eth0 eth1"
ARGS="-q -f -u0 -d0 -w -I"
edit /etc/ifplugd/ifplugd.action
Change up and down actions to be simply ping <ip-of-gateway> -c 6

/etc/init.d/ifplugd start

Alternatively see my other post regarding configuring robust Debian networking.




Useful show / debug commands;

Procurve Commands;
show port-access mac-based <- MACAUTHED ports
show port-access mac-based clients
show port-access mac-based clients 1-48 detailed
show port-access config
show port-access mac-based 1 config


show vlans ports 1-48

show qos port-priority

show authentication

show port-access authenticator <- 802.11X Authed ports
show port-access authenticator clients <- 802.11X Authed ports








radiusx -D
radtest 0cc47a16da00 0cc47a16da00 127.0.0.1 0 testing123

Debugging;


Log into manager@coreswitch{0/1} to find access switch connected to MAC;
show mac-address [MAC]
show arp

DEBUG on ACCESS SWITCHES;
Enable debug
debug destination session
debug event
debug security
debug security port-access mac-based include port 1
or
debug security
debug security port-access mac-based include port all
Release port to stimulate new auth
aaa port-access mac-based 1 reauthenticate

no debug all

IF STILL ISSUE STILL NOT CLEAR - RESTART freeRADIUS in debug mode
service freeradius stop
freeradius -X
(Remember to stop freeRADIUS on other server)
 




Extras

LDAP based users logging into the switches can also be authenticated! However due to the LDAP search filter defined above, the "User-Name" RADIUS attribute is being mapped to "radiusMacAddress" and NOT "uid", therefore authenticating users via RADIUS/LDAP is mutulally exclusive to MAC based authenticatiuon.

Solved this previously by running multiple instances of freeRADIUS on different listening ports (each with appropraite LDAP mappings) all using the same LDAP back end. This allows Web based 802.1X, MAC based 802.1X and Also provides the framework for two factor authentication.

aaa authentication console login radius local    <- Enable RADIUS for console login
aaa authentication console enable radius local   <- Enable RADIUS for console enable
aaa authentication ssh login radius local        <- Enable RADIUS for ssh login
aaa authentication ssh enable radius local       <- Enable RADIUS for ssh enable
aaa authentication login privilege-mode          <- Force require radiusServiceType attribute for user privileges


User LDAP object;
objectClass: radiusprofile
radiusServiceType: Administrative-User    <- Give user enable privileges
radiusServiceType: NAS-Prompt-User        <- Give user login only privileges
WARNING; If switch config does not include ‘aaa authentication login privilege-mode’ ALL LDAP users have full enable privileges!




Alternative Solution

Assign the VLANs for each access-port dynamically.

Option 1 - Dynamic GVRP VLAN assignment (not recommended)

Overview

Whilst this is initially a very simple setup whereby Linux is responsible for negotiating required VLANs with the switch using GVRP, so the switch can assign the connected port to said VLAN, there are some caveats;
  • GVRP support is NOT widely deployed - severity = Minor
  • GVRP support has only just come into kernel 3.2 - severity = Minor
  • Enabling GVRP VLANs requires significant research and testing as GVRP has been warned against by HP when using advanced functions like active-active distributed trunks, VRRP, MSTP (all of which we use) - severity = Major
  • Due to the challenges with enabling GVRP it is not advisable to implement short term if a move to static VLANs is desired - severity = Critical
  • Can only implement using the ‘ip link’ command, not /etc/network/interfaces - severity = Major
  • GVRP is not widely used as it allows end-nodes to have control over the network configuration.
  • Buy its very nature of being dynamic, GVRP (1999) has various limitations and issues with scale and was superseded by MVRP (2011) to address these limitations - severity = Major
  • If there are problems, downtime will be experienced whilst GVRP re-converges - severity = Major
  • Will only work on Linux ports, so will not work for IPMI interfaces - severity = Blocker

Implementation

on switch;

gvrp

on Linux;

ip link add link eth1 name vlan6 type vlan id 6 gvrp on loose_binding on
ip link set vlan6 up
NB; no /etc/network/interfaces support




http://evilrouters.net/2008/11/19/configuring-freeradius-to-support-cisco-aaa-clients/