Binary Royale provide a network device management service for our clients. This entails not only initial configuration and installation of devices – firewalls, routers, switches and the like, but also updates to those configurations, and documentation of the configuration. Historically this has been a manual process – an engineer would build a config or make a change, and would manually export the new config, store it in a secure central location, and make notes on the changes made, and the reason for those changes. This process has been fine until recently, with the number of devices we’re looking after totalling less than 100. However a recent expansion is now straining this manual process, so I’ve been tasked with setting up a system to keep track of configurations in an automated fashion. The goals of the new system were:

• Track multiple versions of a device config, ideally in a version control repository.
• Reduce the admin burden on the engineer making the changes.
• Automate as much of the process of adding a new version as possible.
• Be low cost/free to impliment and license.

I initially set out to research a tool that could be used to accomplish the above goals, and quickly came across RANCID. From the RANCID website, it does the following:

• logs in to each configured device,
• runs various commands to get the information that will be saved,
• cooks the output; re-formats, removes oscillating or incrementing data,
• emails any differences from the previous collection to a mail list,
• and finally commits those changes to the version control system

With a little further investigation, I decided that it would be perfect for our needs, and with the addition of a little scripting, could be made into an (almost) fully automated system for editing configurations, and recording those edits. The following is a document of the process I went through to build the system that is now working for us:

RANCID Installation

Build the server

I started with a vanilla installation of Ubuntu as the OS to host the RANCID installation and repository. The instructions on the RANCID website are specific to CentOS, but I’m more familiar with Ubuntu, so I went with that. I won’t go into the details of the installation, save to say that you’ll want the server version – remember to update it after the install completes.

Check your networking

You’ll need to be able to access the devices that you intend to back up from the new server. It’s worth making sure that everything here is correct at the start, as it’ll have you headaches later. It’s easiest to add entries to your hosts file for all the devices that you intend to check. Open /etc/hosts, and you should see something like this:

127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters

Add a new entry for each device that you want RANCID to check:

1.1.1.1   firewall.nottingham.com   fw-not
1.1.1.2   router.nottingham.com   ro-not
1.1.1.3   switch.nottingham.com   sw-not
2.2.2.2   router.derby.com   ro-der
3.3.3.3   switch.leicester.com   sw-lei

Make sure that you can ping all the devices you add by both IP and by hostname. If you can’t, you’ll need to resolve the network issues before RANCID will be able to log into the devices. It’s worth checking that you can SSH into the devices at this stage too – you may well need to edit the device configs to allow this from your new server.

Get Sendmail working

The first job is to install Sendmail, so RANCID can email you with changes to configs as they’re made:

apt-get install sendmail

Next, configure sendmail to relay via your SMTP server, by adding the following to /etc/mail/sendmail.mc:

define(‘SMART_HOST’,’your-mail-server.com‘)dnl

Run the following commands to put your new config into Sendmail:

cd /etc/mail m4 sendmail.mc > sendmail.cf make service sendmail restart

Make sure that you authorise your RANCID server to relay on your SMTP server, otherwise your mail won’t get through. If you can telnet to the email server, you’re on the right track. Finally, you can test your sendmail config:

sendmail -v user@domainname < test.mail

test.mail contains the following:

Subject: test mail first line of your message [blank line]

If all is working, you will receive the test email. If not, Google is your friend. Until Sendmail is working, RANCID won’t be able to email you changes that it documents, but it won’t stop everything else working.

Install RANCID

RANCID is in the Ubuntu repository, so it can be installed with Aptitude:

sudo apt-get install rancid

Configure RANCID Groups

The installation creates a new user and group named “rancid” with a home directory of /var/lib/rancid. You must create at one least one group in RANCID to logically organize your devices. You can base the group names on any criteria you wish – geographic location, client, function etc. I used location. Edit the RANCID config file:

vim /etc/rancid/rancid.conf

It’s a blank file by default. Add a line of config with your chosen groups:

LIST_OF_GROUPS=”Nottingham Derby Leicester”

 Configure email notifications

RANCID needs to know who to notify about changes to devices in each group. This is done by creating email aliases in the MTA config. As we already know, this is Sendmail on Ubuntu, and the config file in question is /etc/aliases:

vim /etc/aliases

You need to add 2 aliases for each group that you created in the previous step; “rancid-<groupname>” and “rancid-admin-<groupname>”. You will need to add something like the following:

rancid-Nottingham: <your-email@address.com>
rancid-admin-Nottingham: <your-email@address.com>
rancid-Derby: <another-email@address.com>
rancid-admin-Derby: <another-email@address.com>
rancid-Leicester: <your-email@address.com>
rancid-admin-Leicester: <your-email@address.com>

Lastly, you’ll need to let Sendmail know about the new aliases:

sudo /usr/sbin/newaliases

Create the CVS Repository

By default, RANCID used Concurrent Versions System – CVS to store your device configs. This give you the ability to track changes over time, and compare versions to one another. RANCID can also use Subversion – SVN, but I didn’t see any reason not to go with CVS. RANCID will create the CVS repository and folder structure for you, based on the groups that you defined earlier. This command needs to be run as the ‘rancid’ user that was created automatically when you installed RANCID:

sudo su -c /var/lib/rancid/bin/rancid-cvs -s /bin/bash -l rancid

If that works without error, you should be able to see a bunch of new folders under the /var/lib/rancid folder, and within them, the router.bd config files that will contain details of the devices that you want to query:

[you@yourserver~]$ sudo find /var/lib/rancid -type f -name router.db
./Nottingham/router.db
./Derby/router.db
./Leicester/router.db

 Add the devices

The router.db files let RANCID know whaich devices exist in each group. Each device occupies a single line in the config file, and is in the format <device_name>:<device_type>:<state>. Device_name is the hostname or IP of the device in question. Device_type is the device manufacturer – see here for all the supported device types. State is up or down. Up devices will be queried, down ones will be ignored. In my example I might add the following to /var/lib/rancid/Nottingham/router.db:

fw-not:fortinet:up
ro-not:cisco:up
sw-not:hp:down

 Configure cloginrc

RANCID uses cloginrc to connect to, and authenticate against the devices that you have configured in the router.db files. To do this, it needs details of how to connecto to the device – SSH, Telnet etc, and the credentials to be passed through to each device. These details are in /var/lib/rancid/.cloginrc by default. I’m only going to cover connecting via SSH to devices that authenticate locally here – Telnet isn’t secure enough for this sort of thing really, and remote authentication via LDAP of the like can get complicated. For full details of all the protocols that cloginrc can use to connect to your remote devices, see the Man pages for cloginrc. Assuming the device authenticates locally, and we’re connecting via SSH, I might enter the following in /var/lib/rancid/.cloginrc:

#Nottingham Router add method ro-not {ssh}
add cyphertype ro-not {des}
add user ro-not {rancid}
add password ro-not {<user_password>} {<enable_password}

Add a set of config like the above, for each device. Obviously, you’ll need to make sure that the user you define exists on the remote device, and that you can log into that device, with that user, from your RANCID server.

Testing cloginrc

Once you’ve added the cloginrc config for all your configured devices, you can begin testing. firstly, we need to see if cloginrc can connect to each device, with the credentials you’ve provided for it:

/usr/lib/rancid/bin/clogin -f /var/lib/rancid/.cloginrc ro-not

If all is working, you should find yourself in Router Exec mode on you device – or the equivalent if it’s not a Cisco router. if you’re not so lucky, cloginrc should give you some helpful error messages to point you in the right direction.

Testing RANCID

With cloginrc working, we can now give RANCID a test:

sudo su -c /var/lib/rancid/bin/rancid-run -s /bin/bash -l rancid

Depending on the number of devices you’ve configured, this might take a while to run – be patient! If you’re having trouble with it, or if you’re testing an automation script – see below – it’s worth temporarily setting all but one of your devices to ‘down’ in the router.db files to speed this process up. Once it’s finished, check /var/log/rancid for details. You should also receive some emails with details of the new devices and their configs – assuming you got Sendmail working of course.

Viewing your backups

Once you’ve got your configs into CVS, you’re going to want to look at them. Of course you can open the most recent version with Vim on the RANCID server, but if you want to see older versions, you’ll need a client of some sort to access the repository.

ViewVC is a browser interface for CVS and Subversion version control repositories. It generates templatized HTML to present navigable directory, revision, and change log listings. It can display specific versions of files as well as diffs between those versions. See the ViewVC website for more details.

Installing ViewVC

ViewVC can be installed from the Ubuntu repository with Aptitude:

apt-get install viewvc

Configuring Apache

Apache is installed automatically as a dependency of ViewVC, but it is not configured out of the box. By default, ViewVC is installed in /usr/lib/viewvc/ and the main executable is in /usr/lib/cgi-bin/. In order to make Apache recognize ViewVC, you must add the following line to /etc/apache2/httpd.conf:

ScriptAlias /viewvc/ /usr/lib/cgi-bin/

Configuring ViewVC

ViewVC needs to know where your CVS repository is located, in order to view it. The location needs to be entered into /etc/viewvc/viewvc.conf under the[general] section of the config. You can either specify individual repositories or, as I did, specify a root parent location under which multiple repositories can be located, in case you have cause to add more in future:

root_parents = /var/lib/rancid/CVS : cvs

Restart Apache

service apache2 restart

Once Apache has restarted, you should be able to browse to http://[serverip]/viewvc/viewvc.cgi to see the contents of your RANCID repository.

Basic Automation

The most straightforward way to automate backups is to schedule RANCID to run periodically via cron. This will scan all the configured devices, and email you if there have been any changes. As RANCID needs to be run by the rancid user, we have to put the job in the crontab for that user:

sudo su -c “/usr/bin/crontab -e -u rancid”

Add the job, scheduling to meet your needs. the following would run RANCID every monday at 09:00:

# m h dom mon dow command 00 09 * * Mon /usr/bin/rancid-run

 Advanced Automation

The cron job was a little lacking for our implementation – we wanted a way to annotate the changes that were made to devices, and enter them into the repository immediately as they are made. I decided to write a wrapper script for RANCID that would do the following:

• Determine if the device a user was trying to edit was configured in RANCID
• Open a connection to that device, allowing the user to view and edit the config
• Perform a RANCID backup when the user logs out of the device
• Determine if the config for the device in question has been changed, and exit if it hasn’t
• Prompt the user to record their changes in a changelog file if the config has been changed
• Insert a blank log entry at the top of the device’s changelog, and populate with the device name, config version, user,  and date/time.
• Open the changelog so the user can add a description of the changes that they made
• When the user saves the changelog, commit it to the CVS repo, next to the device config

The script I have written, is pretty basic, but it does the job – it utilises some of the things you’ve already done to get to this point, and a bit of other basic bash scripting:

#!/bin/bash
####################################
#
# Custom Network Device Editing and backup script
# v0.1 01-05-2013 by Andy McCulloch of Binary Royale
# This script logs the user onto a remote network device, allows them to edit the config on it and, when they log out from it, it automatically initiatied a RANCID scan to docment the changes made, and insert them into the RANCID CVS repository.
#
####################################

## Check that a device has been passed in
if [[ $# -ne 1 ]]
then
 echo -ne "nUsage: $( basename $0 ) <device>nn"
 exit 4
fi

## Define Variables
device=$1
user=$USER
now=$(date +"%m-%d-%Y_%T")
templog="/home/$user/templog.txt"
newlog="/var/br/log_template.txt"

## Determine if the device being edited is configured in RANCID
if [[ $device == *-not ]]
then
 site="Nottingham"
else if [[ $device == *-der ]]
then
 site="Derby"
else if [[ $device == *-lei ]]
then
 site="Leicester"
else
 echo "The device you have specified has not been configured in RANCID. Aborting."
 exit 4
fi fi fi

## Set the Log file
log="/home/$user/devices/$site/configs/$device.log"

## Get the last revision of the device to be edited
lastrev=$(cvs log /home/$user/devices/$site/configs/$device | awk '/head:/ { print $2 }')

## Open a session to the chosen device
sudo su -c "/usr/lib/rancid/bin/clogin -f /var/lib/rancid/.cloginrc $device" -s /bin/bash -l rancid

## Start a RANCID backup
echo "Performing RANCID Backup"
sudo su -c /var/lib/rancid/bin/rancid-run -s /bin/bash -l rancid

## Update the local copy for this user
cvs update /home/$user/devices/$site

## Get the Head revision of the file just edited:
rev=$(cvs log /home/$user/devices/$site/configs/$device | awk '/head:/ { print $2 }')

## Check whether the device config has been changed
if [[ $lastrev == $rev ]]
then
 clear
 echo "The config of $device has not been changed; there is no need to alter the device changelog. Exiting."
 exit 0
else
 clear
 echo "The config of $device is now at revision $rev. Please now detail the changes you have just made."
 read -p "Press [enter] to continue to the changelog."

## Update the changelog file with a new section for the revision just made
 cat $newlog > $templog
 sed -i 's/DEVICE/'$device'/g' $templog
 sed -i 's/REV/'$rev'/g' $templog
 sed -i 's/USER/'$user'/g' $templog
 sed -i 's/DATE/'$now'/g' $templog
 cat $log >> $templog
 mv $templog $log

## Document the changes just made
 vim $log

## Commit the changes to the log file
 cvs add $log
 cvs commit -m "Notes for revision $rev of $device" /home/$user/devices/$site
fi
exit 0

The changelog template referred to in the above script looks like this:

Device: DEVICE
Revision: REV
Date: DATE
Name: USER
Change:

#############################################################################

That’s it. We are now using this system to edit configs, and record the changes at the same time, and it’s working pretty well so far.

Thanks for reading. These instructions have been gathered from numerous sources on the internet, and compiled into this guide. Most of the instructions above (except the script) are not my own – I’ve merely compiled and distilled several sets of instructions into a guide to achieve a particular goal. Trackbacks below. If you feel that you deserve credit, and have not been mentioned below, please let me know and I’ll be happy to include you.

http://www.shrubbery.net/rancid/#started
https://help.ubuntu.com/community/RANCID
http://evilrouters.net/2010/05/31/installing-rancid-on-ubuntu-10-04-lts/
http://archive09.linux.com/feature/154147