Centralized Intrusion Detection Using Tripwire

Yunliang Yu <yu@math.duke.edu>

Summary:

A new setup for the popular filesystem integrity checking program, Tripwire, is described in this article. Our tripwire auditing system is easy to set up and manage, and it scales well to hundreds of clients. It makes the daily monitoring of our client systems much easier and we believe it's more secure than the default client based tripwire installation.

Introduction:

Like many .edu sites, we have been running tripwire on our systems since the earlier days of the program. It's a nice way to detect if anything has ever happened to those important files in your systems and is also a valuable assessment tool after a breaking-in. Over the years we've made some improvements to it to make it easier to run on hundreds of machines and make it more secure. The best idea we have come up so far is to start tripwire from a centrally secure host (I'll call it the secure host from now on) through ssh, and do not install anything on the client systems. Here is some excerpt of an email message I sent to tripwire@visualcomputing.com back in 1998 which described our basic setup:

In a local network, set up a secure UNIX machine without ANY network services running. The only access to it is at the console. This is going to be the tripwire monitor host; it hosts all tripwire binaries for different OSes on the network, and stores all tw.db_* files. A cron job is running on this machine to log on remote hosts on the network through ssh secure shell to run tripwire at randomly selected time and collect data when tripwire is done. RSA authentication is used and the private key never leaves this secure host. Before each tripwire run on a remote host, tripwire binary and database for that host are copied over to a randomly selected directory. Then the following command is issued immediately through the ssh connection:
    tripwire -tsig 2ChmQx2X43JfLdfRVKbvqo 
             -dsig 2t3FRZSoXbtjRx88DZQxgq 
             -csig 1Lq2wfGJJWTtIicS9wvJSj 
where -tsig is the MD5 signature of the tripwire binary itself, -dsig the signature of the database, -csig the signature of the config file. In tripwire source code, MD5 signature is calculated for every fgets and fprintf. If any of the signatures doesn't match, the program quits and reports to admin staff at once. The temporary database is written to this randomly selected directory instead of /tmp and has its MD5 signature reported back to the secure host through the secure connection. After each run, the secure host collects all the temporary databases, check their signatures, and generates report, etc. The temporary databases will be used to update the databases when needed.

We have changed the setup somewhat since then. For example, it now won't send the database to the client anymore, and a sampling scheme is used to reduce the number of reported changes. We'll describe it in more detail later in this article.

Advantages of our setup over the current tripwire-2.3.x default install include:

It doesn't implicitly trust the systems which run the tripwire binaries. In a default install, the tripwire binaries (run from crontab in most cases) can be easily modified by an attacker to cover up any fs changes, even though the config and policy files are protected by public key encryption.
No installation on client machines are required other than copying over the public key. The public key distribution can be automated in your post-install script or your CFengine script. To add a new client, you just need to copy over the public key and add it to the server's tw.hosts file.
The databases are compared, changes are filtered and the report is generated on the secure tripwire host. Not likely to be tainted.
It's hard to overlook any monitored systems. It'd be easy to overlook one client if you receive a hundred tripwire email reports a day. Our setup combines all the reports into one with all the systems account for, and the reports are archived at a secure web server.
Database management, such as updating the databases or changing the config files, is all done on the secure host.

How does it work:

A crontab entry was added by the tripwire user on the secure host which will start a run script for the tripwire system at a random time every night. The run script first creates a restricted ssh config file to protect the ssh connections to clients. It then starts ssh-agent and load in the pass phase for the secret key. It next spawns tripwire jobs for each hosts listed in the tw.hosts config file. In order not to overwhelm the network, it runs at most 15 concurrent jobs at a time. The tripwire job does the following:

figure out the OS type of the remote client,
compute md5 signatures for the right tripwire binary and config file,
scp the tripwire binary to a random directory on the remote client,
ssh to the client and start tripwire:
```
  cat tw.config | timer -7200 ssh -x -l root $rhost \
    "cd $dir && ./tripwire  -tsig md5_sig"
```
where timer is an alarm to kill the connection if the run takes more than 2 hours,
the md5 hash for the new database file is computed during the run and sent back to the secure host using the same ssh connection,
scp the new dabase file from the client after tripwire is done,
the md5 hash is recomputed to see if the database is altered; warn if it is,
it waits for the database files from all clients to come in or time out,
it now runs another script to compare changes for a group of selected sample hosts for each OS type. This is done so the final report'll be quiet about the same changes occuring on all the sampling hosts as part of package upgrading,
each new database is compared to the baseline database to find any changes,
parse the logs and generate reports. In the report, errors will be listed from any unsuccessful runs, followed by the critical changes and misc changes on all systems,
the report is sent to a secure web server for sysadmins to view.

On average, we spend about 5 minutes every morning to monitor about a hundred hosts.

How to set up:

The setup described here is based on the Tripwire 1.3.1 Academic Source Release which you can download from the tripwire web site. The same procedure should be easily adoptable for the older 1.2.x versions or the Open Source 2.3.x releases.

Find a machine to be your secure host. Nothing fancy, a 200MHz PC with 64MB ram and 4GB disk will do. We use a 140MHz SUN Sparc runing Red Hat linux.
Do a fresh install of the latest linux OS, apply all the updates. Remove and stop all network services from your init.d scripts. Remember there shouldn't be any network services running on this system; or you are just kidding yourself. Use nmap to scan it if you are not sure.
Create your ipchains rules to deny most outgoing and all incoming connections. Here is an example. Denied packets can be logged to your loghost by adding an entry to the /etc/syslog.conf file. This actually behaves like a cheap port scan detector ;)
Download and install SSH on the secure host. OpenSSH needs this simple patch to make it read pass phase from the stdin in the tripwire run script.
Create a tripwire user account, say "tw"; remove or lock all the other user accounts other than root.
As the tw user (from now on), Download this tar file and untar it in tw's home directory. It'll create a bin and a run directory.
Cd ro the run directory and run "ssh-keygen -C X@Y" to make an authentication key pair. Save the passphrase to the file .usercomment. The public key will be in ~tw/.ssh/identity.pub. Guard the private key ~tw/.ssh/identity with care; make sure it never leaves the secure host (even not show up in your backup tapes). The public key file identity.pub is going to be copied to the root's .ssh/authorized_keys file on all client hosts next.
Download the Tripwire 1.3.1 source and untar it to a temp directory,
Now apply the patch tripwire-1.3.1-1.yu1 in the run directory to the tripwire source tree you have downloaded eariler, and compile tripwire. Copy the compiled tripwire,siggen to your run/linux directory and ~tw/bin directory. Su root and setuid ~tw/bin/tripwire, which is necessary for tw to access all files on the secure host itself.
Copy the patched tripwire-1.3.1 source directory to machines running other OSes to make the corresponding tripwire,siggen binaries for those OSes. Copy them into the corresponding run/OS directories.
cd to the run directory and mkdir OStype for each OS type you want to run tripwire on, such as linux,solaris, irix,osf, etc.
Copy the sample config files in tripwire-1.3.1's configs directory to run/OSes directories. Modify if necessary. Note tw.config.hostname can also be created for each special hostname if needed.
Compile timer.c by "gcc -o timer timer.c" and move timer to ~tw/bin.
Modify the siggen=, tripwire=, mailto=, webhf= lines in run/runtw script.
Now you need to decide which hosts to run tripwire on. Add them to the file tw.hosts as in the example. The script "chkdns" can give you a list of all registered hosts in your domain.
Now scp the file ~tw/.ssh/identity.pub to root's .ssh/authorized_keys file on all the client hosts. (now you know why the private key file has to be secure)
Run "runtw -n host1" to try it out on host1. You might need to modify the scripts to make them work on your system; since we are running them on an sun box.
If tripwire works OK in host1 above, add a line to tw's crontab to start runtw every night.
The script "updatedb" can be used to view the complete logs and update any hosts' database files.

Remarks:

Backup for the system has to be secure. Either encrypt the data, or use a local tape drive attached to the secure host.
Tripwire should be used along side other measures to safe guard your systems. It won't stop any breaking-ins on its own.
As a side bonus, our patched tripwire also lists open network ports on the clients. Just add "/tcp:port_range" or "/tcp:port_range" to your tw.config. Adding "/localfsonly:" to the beginning of tw.config will limit the checking to local filesystems (linux/solaris only).
The public key encryption in the latest open source tripwire can be used to replace md5 above as a way to check the integrity of the database files sent back to the secure host.

Thanks:

Thanks for the Duke Sysadmin group (Andrew Schretter et al.) for interesting discussions.