Basic Intrusion Detection System with mtree

We'll call the machine that's being checked the 'remote' and the machine that's running the checks the 'local' machine (it might also be somewhere remote to you, but we need to call it something). The top level steps are:

1. From the local machine, connect to the remote and create the mtree specification with our secret seed value, and store this initial version as a 'master'.
2. Periodically re-run mtree to generate a new specification and compare it ot the master. If it doesn't match the master anymore, send an email to someone who cares.
3. If you update the remote machine in some way (install new programs, or update them) then you have to create a new 'master' which represents the new good state.

First create a new, unprivileged, user on the remote that the local machine can connect as via SSH. Call them whatever you like, but don't add them to the wheel group, they should not generally have any admin access rights, and so this account will be different to the default admin user account previously created. I'll refer to this user as <remote check user> later.

The main reason for running it as an unprivileged account is that it's an account that's accessible from another machine that, by its nature of being always on and not interacted with constantly like a computer you're sitting at, is probably more vulnerable to being hacked itself.

Setup SSH on your local machine to connect with a key file to the unprivileged user on the remote machine, and add these details to an alias in the /.ssh/config file on the local machine. We'll need this in a script later.

Next we have to choose which files and directories are actually worth checking? My list is the following, for both hosts and, should they exist, any jails running on the remote machine:

• /bin
• /sbin
• /etc
• /usr/bin
• /usr/sbin
• /usr/local/bin
• /usr/local/sbin
• /usr/local/etc

You can choose different directories based on your preferences. This should cover most executable and critical configuration files on FreeBSD, your system may vary.

This does lead to one small problem, and that is some of the files we want to checksum are themselves secret and only viewable by the root user or wheel group, such as /etc/passwd; but, we just created an unprivileged user…

With doas (and sudo) it's possible to give users, or groups, privileged rights to run only specific programs and nothing else.

The problem with mtree is that it's not read-only, you can use it to change files on disk to match a specification, and we don't want to allow an unprivileged user, accessible via an unsupervised server, to do that. It is also possible in doas to lock the command with specific arguments, but that requires specifying those arguments in /usr/local/etc/doas.conf, which is on the remote machine, and then we'd have to write the seed we want to keep secret, and off the remote machine, into it. So that's a no go.

The answer is to put the mtree command into a script - which we were going to have to do anyway to run across multiple directories - and set the permissions so this script can't be edited by the user. We'll add that script in a moment, first add the following line to /usr/local/etc/doas.conf:

permit nopass <remote check user> cmd /usr/local/bin/gen-mtrees

On the remote machine save the following script in the file 'gen-mtrees' and place it at /usr/local/bin/gen-mtrees, then make sure the permissions are set to read and execute only for all:

# chmod 755 /usr/local/bin/gen-mtrees

You should be able to run this script with the remote check user using doas, and it prints the resulting spec to stdout without complaining that it can't access any files:

$ doas /usr/local/bin/gen-mtrees 123456

This covers all the directories listed above, and then again for any jails found in /usr/local/jails/containers. (Jails can support security, but are not a solution by themselves. Even if a jail helps to keep the host safe, a compromised jail is just like a compromised machine by itself.)

#!/bin/sh

usage() {
    echo "Usage $0 [-h] seed"
    echo "Create an mtree spec for selected dirs on host and in jails"
    echo ""
    echo "Options:"
    echo " -h Display this help message"
    echo ""
}

case "$1" in
    "-h")
	usage
	exit 0
	;;
    *)
        # Carry on
	;;
esac
seed="$1"

gen_mtree_spec() {
    dir="$1"
    mtree -c -s "$seed" -p "$dir" -K sha1,cksum
}

# Dirs to check on the host, and potentially jails
check_dirs="/bin /sbin /etc /usr/bin /usr/sbin /usr/local/bin /usr/local/sbin /usr/local/etc"

for dir in $check_dirs; do
    gen_mtree_spec "$dir"
done

# Now repeat the check for any jails.
for jail in /usr/local/jails/containers/*/; do
    if [ -d "$jail" ]; then
	jail=${jail%*/}  # remove the trailing slash
	for dir in $check_dirs; do
	    gen_mtree_spec "$jail$dir"
	done
    fi
done

With the remote set up, we can now set up the local machine to run the check. Again we wrap what we need in a shell script, I've called this one check-mtrees.sh. The host value in the script below is the name given to the remote machine in your /.ssh/config file.

#!/bin/sh

usage() {
    echo "Usage $0 [-h -m] host seed [email]"
    echo "Run an mtree comparison on a remote host, or create a master reference."
    echo "Host must exist in /.ssh/config to connect."
    echo "Options:"
    echo " -h Display this help message"
    echo " -m Create master reference, don't run a comparison"
    echo "    (ignores the email argument)"
    echo ""
}

master=0

case "$1" in
    "-h")
      usage
      exit 0
      ;;
    "-m")
      master=1
      host="$2"
      seed="$3"
      ;;
    ,*)
      host="$1"
      seed="$2"
      email="$3"
      ;;
esac

# connect via SSH, run the local script to generate the mtrees
# which come as one stream, but then split them on their directories
# as mtree can't compare them all together if files and their links
# are in the same spec:
ssh "$host" "doas /usr/local/bin/gen-mtrees $seed" \
    | split -d -p '#[[:space:]]{4}user: .*' - "$host-"

if [ "$master" -eq 1 ]; then
    # Move all the outputs to a separate folder for later re-use
    mkdir -p "reference-$host"
    for file in "$host"-*; do
      mv "$file" "reference-$host/$file.mtree"
    done

    # Remove now out of date results file
    # if it exists
    if [ -f "mtree-results" ]; then
      rm mtree-results
    fi

    exit 0

else
    # Avoid using the shell's echo so we know it has -n,
    # at lest on FreeBSD,
    # so it doesn't print a newline
    # to make sure the file has size 0
    #shellcheck disable=3037
    /bin/echo -n "" > "mtree-results"
    for file in "$host"-*; do
      mtree -f "reference-$host/$file.mtree" -f "$file" >> "mtree-results"
    done
fi

if [ -f "mtree-results" ] && [ -s "mtree-results" ]; then
    echo "mtree found differences"
    mail -s "mtree failure on $host $(date)" \
       "$email" < mtree-results

elif [ -f "mtree-results" ] && [ ! -s "mtree-results" ]; then
    echo "mtree found no differences"
    rm mtree-results
fi

The first time you run it it should be run with the -m flag. This will create a set of master plan specifications stored in the directory '<remote name>-reference' against which subsequent runs are compared. If you update the remote machine (or its jails), install additional programs, add users etc, you will have to re-run the script with this option, otherwise you will get an alert that the files don't match - which is correct, but hopefully the change was intentional.

One detail to highlight is how the generated specifications are handled. The command on the remote just prints the results to stdout, which means they can be piped across SSH and not stored as files on the remote machine. On the local machine the command after the pipe (|):

split -d -p '#[[:space:]]{4}user: .*' - "$host-"

Uses split(1) to separate the incoming stream of text into one file per directory (named 'host-00', 'host-01' etc.) that mtree iterates through. The reason for doing this, and not just dumping it onto one file, is that mtree gets confused by finding the same file and it's link in the same specification when comparing them. This can happen in a number of places on a normal base FreeBSD install, such as /sbin/nologin that is a link to /usr/sbin/nologin. When encountering the second version mtree complains that this should be a link and not a file.

Usually, after creating the master reference, you can call this file from the local machine with:

$ ./check-mtrees.sh <remote host> <seed> someonewhocares@example.com

in a directory where you want to keep the resulting specifications.

In most cases you want to run this at a frequency in line with your level of paranoia, and set it as a cron job. Assuming the above script is saved to ~/scripts/check-mtrees.sh, then save the following as mtree-cron.sh and run it from your crontab. If you have more than one remote machine you're checking with this local machine you can just add them separated by spaces into the hosts variable below.

#!/bin/sh
# Add as many hosts as you want to check in a space separate list
hosts="<host>"

for host in $hosts; do
    mkdir -p ~/ids_checks/"$host"
    cd ~/ids_checks/"$host" || exit 1
    ~/scripts/check-mtrees.sh "$host" <seed> someonewhocares@example.com
done

That's it. I hope that was understandable. If you have any suggestions on how it can be done better please let me know.

And don't forget to re-run the generation of the master specification after an update - or don't, and then you can check it's working.