Introduction

These notes describe our cfengine-rsync setup, which currently is used to distribute the known hosts file for ssh. But it could be used for much more, and eventually it will be. Cfengine is described here. Cfengine's author presented an interesting paper about some of cfengine's internals at the 1997 LISA conference in San Diego.

The general idea is to use rsync to 'pull' the known hosts file from a server, instead of using rdist to 'push' the known hosts file from the server. This is necessary because a number of machines that need the known hosts file are dual-boot Linux/NT systems, and it is impossible to know at any given time which mode they will be in.

Though a pull model is also better for PPP-connected machines, or for machines aren't directly administered by the lab staff.

Cfengine has a builtin way of copying files from a server, but using rsync inside cfengine instead has several advantages:

rsync can keep files in step by copying only the parts that have changed, instead of dragging over a 100k file because 10 bytes are different
rsync can run over ssh, which has security advantages for distributing sensitive files
cfengine's internal copying method is hard to use to maintain the cfengine configuration file itself.

The ability to synchronize without having to copy entire files is particularly an advantage at boot time, when you don't want the system to stall while it updates a lot of stuff. A Linux/NT system rebooted in Linux after a month in NT mode might have a lot of catching up to do!

So this setup uses cfengine as a 'wrapper' around rsync; rsync handles the file copying chores, while cfengine manages things overall.

Cfengine

Cfengine runs at boot time and then once an hour while the system is up (or at least up in UNIX mode). Each system has a startup script that invokes cfengine in a special "boottime" mode:


	Solaris:	/etc/rc2.d/S72cfengine
	Linux:		/etc/rc.d/rc3.d/S31cfengine
	Digital UNIX:   /sbin/rc3.d/S17cfengine
	SunOS:		(part of) /etc/rc.local

In general the cfengine script runs as soon as possible after the networking setup is complete.

Cfengine is installed in the same location across all platforms (/usr/local/sbin/cfengine) and its configuration files live in the same directory across all platforms (/usr/local/lib/cfengine/inputs, with supporting programs in /usr/local/share/cfengine). This lets the boot time script be the same across all platforms:


#!/bin/sh

if [ -x /usr/local/sbin/cfengine -a -f /usr/local/lib/cfengine/inputs/cfengine.conf ]; then
	CFINPUTS=/usr/local/lib/cfengine/inputs; export CFINPUTS
	/usr/local/sbin/cfengine -f cfengine.conf --no-splay -Dboottime
fi

(download it here.)

The '--no-splay' option turns off a feature that's useful once the system has booted, but may cause the system to stall at boot time. Normally cfengine runs once an hour on each client, but that could mean a lot of machines beating on the server at the same time. To avoid that, cfengine staggers the time each client contacts the server. That range of times is controlled by the 'SplayTime' option:


	SplayTime = ( 5 ) # minutes

so that even though cfengine on both pe44 and pe45 is invoked at say 4pm, pe44 might not contact the server until 4:02pm while pe45 might wait until 4:05pm. Cfengine goes to sleep during this interval, which isn't something you want at boot time.

The other flag, '-Dboottime', defines a special class in the cfengine configuration file. Currently that's not used for anything, but it could be used to avoid long-running stuff that would be better run after the system has booted. The goal of the boot time cfengine run is to do the minimum needed, as quickly as possible.

cfengine.conf

The general cfengine configuration file is in /usr/local/lib/cfengine/inputs/cfengine.conf. It's the same across all platforms -- one of the goals of cfengine is to have one configuration file that spans all machines at a site. The configuration language uses a 'class' construction to isolate things that only should be done on a particular operating system or host.

The configuration file basically is


... various declarations like SplayTime ...

control:
	actionsequence = ( files editfiles shellcommands links )


files:
	... stuff involving creating files ...

editfiles:

	... stuff involving modifying files ...

shellcommands:

	... rsync runs here ...

links:

	... stuff for making symlinks ...

Cfengine does the 'actionsequence' steps in the order declared. So first it does the 'files:' actions, then the 'editfiles:' actions, then the 'shellcommands:' actions, and finally the 'links:' actions.

The 'files:' and 'editfiles:' actions are basically to set the system up for running cfengine at boot time and then once an hour. On a Linux system, for instance, the 'files:' action creates /etc/cron.hourly/cfengine, if it doesn't already exist, and sets the ownership and permissions appropriately. The 'editfiles:' action then edits /etc/cron.hourly/cfengine so that it contains


/usr/local/share/cfengine/cfwrap /usr/local/share/cfengine/cfhourly

Once that is in place, the cron program on Linux will run that command every hour.

The 'shellcommands:' step is a general hook for running shell scripts. In this case, it runs a script to use rsync to copy files from the directory /etc/distrib on the server (baskerville, for now) to /etc/distrib on the client. While rsync can run over ssh, we first need to check that the ssh config files are up-to-date. So the first part of the script handles copying the files in /etc/distrib/ssh, and then the script copies everything else in /etc/distrib over ssh:


#!/bin/sh

# 
# Script for pulling system administration files from a server via anonymous
# rsync over ssh
#

PATH=/bin:/usr/bin

RSYNC=/usr/local/bin/rsync
RSYNCPATH=/usr/local/bin/rsync
SERVER=baskerville
DIR=/etc/distrib
SSHPATH=/usr/local/bin/ssh

if [ -x $RSYNC ]; then

#
# First get the ssh config and known hosts files
#

	umask 022; mkdir -p $DIR/ssh 

	$RSYNC --archive --rsync-path $RSYNCPATH --timeout 30 $SERVER::ssh $DIR/ssh
	if [ $? = 0 ]; then
		if [ -s $DIR/ssh/ssh_config ]; then
			rm -f /etc/ssh_config
			ln -s /etc/distrib/ssh/ssh_config /etc/ssh_config
		fi
		if [ -s $DIR/ssh/sshd_config ]; then
			rm -f /etc/sshd_config
			ln -s /etc/distrib/ssh/sshd_config /etc/sshd_config
		fi
		if [ -s $DIR/ssh/ssh_known_hosts ]; then
			rm -f /etc/ssh_known_hosts
			ln -s /etc/distrib/ssh/ssh_known_hosts /etc/ssh_known_hosts
		fi

#
# Now we can use rsync over ssh for the rest
#

		$RSYNC --archive --exclude ssh --rsh $SSHPATH --rsync-path $RSYNCPATH --timeout 30 $SERVER::distrib $DIR
	fi
fi

(download it here.)

Note that the script dumps everything into /etc/distrib and then makes symlinks to the appropriate pathnames. So the ssh known hosts file is copied to /etc/distrib/ssh/ssh_known_hosts, and a symlink is made so /etc/ssh_known_hosts points to /etc/distrib/ssh/ssh_known_hosts.

The 'links:' action in the cfengine configuration file can do this too, but it seemed easier to handle the special case of linking the ssh config files in the script itself. For other things, though, the 'link:' action would be the place -- suppose /etc/distrib contained the canonical /etc/hosts for the department. Then the 'shellcommands:' step would copy /etc/distrib/hosts from the server to the client, and


	links:
		/etc/distrib/hosts -> /etc/hosts

would make /etc/hosts point to it.

rsyncd

The server supports clients copying files by running rsync in a 'daemon' mode that listens for incoming connection on tcp port 873; rsync allows anonymous connections, so it can be run as root on the client. (Before I was resorting to awful kludges involving root running su'd as 'nobody'.) Of course for many files we don't want connections from outside the department, and the rsync daemon configuration file, in /etc/rsyncd.conf, has a tcpwrappers-like facility to do that:


[ distrib ]
	comment = system administration files
	path = /etc/distrib
	read only = yes
	list = yes
	uid = nobody
	gid = nobody 
	hosts allow = *.CS.Arizona.EDU

[ ssh ]
	comment = ssh configuration files
	path = /etc/distrib/ssh
	read only = yes
	list = yes
	uid = nobody
	gid = nobody
	hosts allow = *.CS.Arizona.EDU

The 'read only' attribute should add some protection too. And we could block port 873 at our router, if need be.

Pulling on demand

Sometimes you want to update a file right now, instead of waiting for up to an hour for the clients to retrieve the new version. Cfengine comes with a program called cfd that runs in daemon mode on the clients and is used both for cfengine's internal way of pulling files, and for remotely executing cfengine on demand. Remote execution works by running another program, cfrun, on the server; cfrun contacts the cfd daemon on the clients which in turn invokes cfengine on the client. This lets you 'push' a message to the clients, telling them to pull the new file right now.

Since that's a little complicated, here's a picture:


 --------		 ------------------ 
|        |		|     		   |
| server | -- cfrun --> | cfd on the client|
|        |		|		   |
 --------                ------------------
                                  |         
                                  |
                                  V
                        cfengine -f cfengine.conf runs on the client

				shellcommands:

					rsync pulls files from the server

Unfortunately the information on configuring cfd is a little obscure, and there is a persistent bug in the access control setup cfd uses (more on that below). Still it's a valuable adjunct to cfengine.

cfd should start on the client at boot time, so modify the cfengine boot time startup file above to read


#!/bin/sh

if [ -x /usr/local/sbin/cfengine -a -f /usr/local/lib/cfengine/inputs/cfengine.conf ]; then
	CFINPUTS=/usr/local/lib/cfengine/inputs; export CFINPUTS
	/usr/local/sbin/cfengine -f cfengine.conf --no-splay -Dboottime
	if [ -x /usr/local/sbin/cfd -a -f /usr/local/lib/cfengine/inputs/cfd.conf ]; then
		/usr/local/sbin/cfd -f cfd.conf
	fi
fi

(download it here.)

Here's /usr/local/lib/cfengine/inputs/cfd.conf, the cfd configuration file:

control:

	domain = ( CS.Arizona.EDU ) # Bug alert!

	cfrunCommand = ("/usr/local/sbin/cfengine")

admit:

	/usr/local/sbin/cfengine	*	# Bug alert!!

The 'admit:' statement is supposed to control who can run what via cfrun; it's supposed to work like the /etc/hosts.allow facility of the tcpwrappers. Unfortunately the latest supported version of cfengine handles the 'domain' statement in a case-sensitive manner and this breaks access control for sites like us that use mixed-case fully qualified domain names. The '*' wildcard on the right-hand side works around that -- but then any host on the Internet can run /usr/local/sbin/cfengine via cfd.

However you can use the tcpwrappers to control access instead. The 'configure' step in building cfengine will pull in libwrap.a if it finds it in /usr/lib or /usr/local/lib; then you can do something like


cfd:  ALL

in /etc/hosts.deny on the clients, to block access by default, and then


cfd: baskerville.CS.Arizona.EDU

in /etc/hosts.allow on the clients. Then clients will reject connections to their cfd daemon that don't originate from baskerville.

And if your site doesn't use mixed-case fully qualified domain names then the 'admit:' statement should work just fine.

cfrun reads /usr/local/lib/cfengine/inputs/cfrun.hosts on the server to see which clients to contact. It's simply a list of client host names, one per line:

client-1.CS.Arizona.EDU
client-2.CS.Arizona.EDU

and so on.