VNET Jails with DHCP on FreeBSD

I'm doing all of this FreeBSD 14.2-RELEASE-p1 GENERIC amd64 on an Thinkpad X220. For these experiments the important feature is that it has two interfaces:

• iwn0: <Intel Centrino Advanced-N 6205> (wireless network)
• em0: <Intel(R) 82579LM> (wired Ethernet)

In my rc.conf file I have the two following lines to set up the start the WiFi connection on boot. Normally the Ethernet port isn't connected, but we'll be using it for the jail(s).

wlans_iwn0="wlan0"
ifconfig_wlan0="WPA DHCP"

All commands are being run as root.

I'll create a test jail called 'jvnet' for this. Create a new ZFS filesystem for the jail, mine's zroot/jails/containers/jvnet, automatically mounted under /usr/local/jails/containers/jvnet:

Then I'll repeat the steps for creating a classic jail from the FreeBSD Handbook.

Download the userland:

# fetch https://download.freebsd.org/ftp/releases/amd64/amd64/14.2-RELEASE/base.txz -o \
              /usr/local/jails/media/14.2-RELEASE-base.txz

Extract and copy into our filesystem:

# tar -xf /usr/local/jails/media/14.2-RELEASE-base.txz -C /usr/local/jails/containers/jvnet --unlink

Copy resolv.conf and localtime:

# cp /etc/resolv.conf /etc/localtime /usr/local/jails/containers/jvnet/etc/.

The run freebsd-update to make sure you have the latest versions.

# freebsd-update -b /usr/local/jails/containers/jvnet/ fetch install

Then I'll just create a snapshot of that before we change anything, so we can also roll back if we manage to break anything:

# zfs snapshot zroot/jails/containers/jvnet@$(date -j "+%Y-%m-%d-post-install")

To work dhclient(8) needs access to /dev/bpf*, for this we need to add a rule into /etc/devfs.rules on the host as follows:

[devfsrules_jail_bpf=11]
add include $devfsrules_jail
add path 'bpf*' unhide

(The number in the square brackets isn't important, only that it's unique.)

The easiest example is just to use the 'spare' Ethernet interface on my laptop, em0, and let the jail have it. When running a vnet jail, any interface passed to the 'vnet.interface' parameter gets 'swallowed' by the jail and vanishes from the host system.

After plugging in the Ethernet cable (an important step, don't ask me how I know) you can launch a jail with:

# jail -c name=jvnet host.hostname=jvnet \
	 persist vnet \
	 vnet.interface=em0 \
	 path=/usr/local/jails/containers/jvnet \
	 mount.devfs devfs_ruleset=11

Then running jls you should see the jail running, and ifconfig on the host should reveal em0 gone.

Enter the jail with jexec jvnet and then run ifconfig to see the em0 interface. Then try to bring it up and get an IP address with:

# dhclient em0

That then send the DHCP discover request, and be given an IP address. You can test the connection by pinging your favourite host or IP address.

Back on the host, jls won't list an IP address, but you will be able to ping the jail at the address it got.

To remove this jail, created without a configuration file, run:

# jail -R jvnet

Checking ifconfig should show em0 being back 'on' the host. (That's what the -R, capital "R", does. If the interface isn't available again, perhaps -r was used.)

If you have a spare interface, and can donate it to a jail, that's great, but chances are either you want to share the interface with the host machine, or even with a spare interface, you want to share it between multiple jails.

We can solve this with a bridge(4), which acts like a switch on a network, except that it's all software. Then you can add a real interfaces, e.g. connecting two or more ports on a physical device, or virtual interfaces used with jails and virtual machines.

One important detail, which confused me for a few days, is that bridges don't normally work with wireless interfaces. (Thanks to Jim Pingle for pointing this out via Mastodon and the PF Sense documentation.) As I understand it: wireless packets already use up all the space in the header for IP addresses to deal with the access points, and so can't be used to navigate the bridge on their way to the final IP address¹.

The other item we need is epair(4), which is pair of virtual interfaces connected together. When we gave the jail access to em0 it disappeared from the host, epair allows passing one side through the veil into the jail, while leaving the other end available on the host machine, and in this case connected to the bridge. First make sure em0 is up and has an IP address, e.g. dhclient em0 has been run.

Then create the bridge and the epair:

# ifconfig bridge0 create
# ifconfig epair0 create

Checking the output of ifconfig should show the bridge, and both 'ends' of the epair, named epair0a and epair0b. Then add the 'a' end to the bridge, along with the em0 interface for onward connectivity, also set the bridge to 'up' so it's running.

# ifconfig bridge0 addm em0 addm epair0a up

Then start the epair0a with²:

# ifconfig epair0a up

Then we can run the same command as before to start our vnet jail, but this time using epair0b as the vnet.interface value.

# jail -c name=jvnet host.hostname=jvnet \
	 persist vnet \
	 vnet.interface=epair0b \
	 path=/usr/local/jails/containers/jvnet \
	 mount.devfs devfs_ruleset=11

Connect to the jail with jexec jvnet and run dhclient epair0b from within the jail, you should now get an IP address assigned in the jail, which you can then test with ping.

To tidy up, remove the jail, exit the jail, and then:

# jail -R jvnet

Then destroy/shut down the various interfaces

# ifconfig epair0a destroy # this also removes the other side
# ifconfig bridge0 destroy
# ifconfig em0 down

Running examples from the command line is OK, but I want something persistent that appears after each boot, so we should put these into a config file.

The other thing I'll add is specific MAC addresses, otherwise they're generated randomly. If I want my DHCP server to identify the hosts and assign specific IPs, I'm going to need a fixed identifier³. We'll create the bridge interface in /etc/rc.conf, but the epair can be done via the jail config file.

To create the bridge then add the following lines to rc.conf, I found the "SYNCDHCP" appeared to be mandatory, if I left it just as "DHCP" on either interface, the bridge would never get an IP.

# Setup network for vnet jails
ifconfig_em0="SYNCDHCP"  # Wait for an IP before creating the bridge
cloned_interfaces="bridge0"
ifconfig_bridge0="ether 58:9c:fc:10:84:67 addm em0 up"

This should then be enough for any jails on this machine. (The laptop is a bad example as usually the Ethernet connection wouldn't be used, but let's assume this represents a fixed machine.)

The creating of the epair0 etc. we can add into the jail's configuration file, on the assumption that you don't need them (virtually) lying around without the jail.

These steps have all been added into the config below, which also includes the following extras:

• Setting a known MAC address for the epair, which is the same, except for the last hex numbers being 0a and 0b for the 'a' and 'b' ends of the epair. This would have to different in each VNET jail you create on your network.
• The epair is renamed to carry the jail's name, in this example the jail is called "jvnet", and the epair0a is renamed to ep0_jvnet_a. This is useful if you're running multiple VNET jails on the same machine, and want to know which epair belongs to which.⁴
• Destroying the epair on jail shutdown, this also removes it from the bridge.

jvnet {

  exec.clean;
  exec.system_user = "root";
  exec.jail_user = "root";

  # PERMISSIONS
  allow.raw_sockets = 1;
  allow.set_hostname = 1;

  # Has to be before exec.start
  mount.devfs;
  devfs_ruleset = 11;
  enforce_statfs = 1;

  # NETWORK
  vnet;
  vnet.interface = "ep0_${name}_b";

  $mac="58:9c:fc:10:12:";

  exec.prestart = "ifconfig epair0 create ether ${mac}0a name ep0_${name}_a";
  exec.prestart += "ifconfig epair0b ether ${mac}0b name ep0_${name}_b";
  exec.prestart += "ifconfig ep0_${name}_a up";
  exec.prestart += "ifconfig bridge0 addm ep0_${name}_a";

  # STARTUP
  exec.start = "dhclient ep0_${name}_b";
  exec.start += "/bin/sh /etc/rc";

  # STOP
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.poststop = "ifconfig ep0_${name}_a destroy";

  exec.consolelog = "/var/log/jail_console_${name}.log";

  # HOSTNAME/PATH
  host.hostname = "${name}";
  path = "/usr/local/jails/containers/${name}";
}

I keep my jail configuration files as /etc/jail.conf.d/{name}.conf, and then load them via having the sole line in /etc/jail.conf:

.include "/etc/jail.conf.d/*.conf";

Then in addition with the following lines in /etc/rc.conf

jail_devfs_enable="YES"
jail_enable="YES"
jail_parallel_start="YES"

With that any jails configured like this should start on your machine after boot, and get their IP addresses automatically via DHCP.

Apart from the obvious sources like the FreeBSD Handbook and man pages, the following really helped me in getting this to work.

• FreeBSD jails and vnet from scratch and Motivation - amoradi.org
• Jail VNET by Examples [pdf]​ - Olivier Cochard-Labbé
• PFSense Documentation: Wireless and bridging
• /usr/share/examples/jails/jib - Script for creating VNET Jails, by Devin Teske