Using Warden to setup jails in FreeBSD
This is a guide on installing jails on FreeBSD using Warden. Warden is a new system for managing jails, originating from the PC-BSD project. I switched to Warden after using ezjail for some time. The reason for switch was that ezjail started showing some errors on FreeBSD 10.1, and Warden seems more streamlined.
Installation and initial setup
Installing Warden is as simple as:
cd /usr/ports/sysutils/pcbsd-utils && make install clean
Afterwards, I modified /usr/local/etc/warden.conf
to contain the
following:
#!/bin/sh
# Configuration options for the Warden
######################################################################
# Network Interface for the jails to use
NIC: lo1
# Directory to use for compressing / decompressing files
WTMP: /usr/jails
# Location of the jails
JDIR: /usr/jails
# When automatically creating jails with unspecified IPv4 addresses, use this
# address at the starting point for new addresses
IP4POOL: 10.10.10.0
The reason for using lo1
as the networking card is because I want to
have a greater control over the traffic to and from jails. Now, in order
to get lo1
on my system, I created a /etc/start_if.lo1
file:
cat >/etc/start_if.lo1 <<EOF
#!/bin/sh
/sbin/ifconfig \$1 create
EOF
Also, I included lo1
in the network_interfaces
setting in /etc/rc.conf
file.
Jail time!
The first jail I created on my FreeBSD NAS was for btsync, a file syncing tool that I use to sync many things in my household.
I had to decide first on the IP for this jail, and I chose 10.10.2.30
.
This is a is a private network,
RFC1918 address, local to the
machine only, as I don’t want to bind this jail to any address on my
home network. Thus, pf
on the host machine will do all the
NAT/redirecting.
warden create btsync --ipv4=10.10.2.30/32
warden start btsync
warden chroot btsync
# do the installation and setup of btsync
...
In addition, I wanted btsync
to store all synced data on a specific
ZFS filesystem. This can be achieved by calling:
warden fstab btsync
and adding the following in the editor:
# Device Mountpoint FStype Options Dump Pass
/mnt/btsync_data /usr/jails/btsync/btsync_data nullfs rw 0 0
Since ZFS filesystem is mounted on /mnt/btsync_data
on the host, this
line causes Warden to mount it to /usr/jails/btsync/btsync_data
. As
btsync
jail resides at /usr/jails/btsync
, this effectively means
that the jail will see this ZFS filesystem at /btsync_data
.
Finaly, one needs to set up pf
to NAT the traffic correctly. To do so,
first ensure that pf is enabled in /etc/rc.conf
, that is, these 2
lines are present:
pf_enable="YES"
pflog_enable="YES"
Then, add something like this to /etc/pf.conf
file:
ext_if="re0"
# btsync stuff
BTSYNC_IP="10.10.2.30"
BTSYNC_PORTS="{51551,50543}"
set skip on lo0
scrub in all
nat on $ext_if from $BTSYNC_IP to any -> ($ext_if)
rdr on $ext_if inet proto tcp from any to port $BTSYNC_PORTS -> $BTSYNC_IP
rdr on $ext_if inet proto udp from any to port $BTSYNC_PORTS -> $BTSYNC_IP
pass all
Et voilá! The jail is ready to get some data in :)
Additional tricks for easier handling of pf.conf
Modifying /etc/pf.conf
to accommodate all new jails can be tedious and
error-prone task. Thus, I created this little perl script that can take an input
file (I keep it at /usr/local/etc/jailnet.conf
) and produces a nice set of
settings that can be put in /etc/pf.conf
(brave souls can try to directly
overwrite it).
The script will take an input describing mappings for jails, in the form:
btsync;udp:51551,tcp:51551,tcp:50543
owncloud;tcp:8081/80,tcp:6000-6010,udp:2000-2010/3000-*,tcp:2011-2015/8112
and output two files, /etc/pf/jails.options.conf
and
/etc/pf/jails.translation.conf
. These two files should be included in
/etc/pf.conf
via the include
directive.
The input contains one line per jail, with two main fields separated by
‘;’: jail name and port settings. Port
settings are a ‘,’-separated list of ports,
in form protocol:portrange
. Protocol can be either ‘udp’ or ’tcp’.
Port range can be either a sole port number, or a range in the same
format pf
that understands, just with ‘:’ being replaced by
‘-’ here. An addition that the script
understands is useful when redirecting a different port on the host to a
port on the jail. The format is ‘rangehost/rangejail’. Again, just like
with pf.conf
, one can write ‘begin\_port-*’ in ‘rangejail’, and pf
will do a one-to-one mapping of port in ‘rangehost’ to ‘rangejail’.
Naturally, one can have a single port in ‘rangejail’ (and/or
‘rangehost’), and all ports from ‘rangehost’ will be redirected to a
single port defined in ‘rangejail’.
Finally, the script is as follows (an can be fetched from GitHub):
#!/usr/bin/env perl
use strict;
use warnings;
use v5.10; # because we use some new stuff
use feature qw(say);
my $card = 're0';
my $opt_file = "/etc/pf/jails.options.conf";
my $nat_file = "/etc/pf/jails.translation.conf";
my %ips = ();
my %ports = ();
# Parses an input string containing the port list the format of the
# list is:
# 'proto':port[-portend][/jail_port[-jail_port_end]]
#
# If portend is given, function will return all ports in range port to
# portend If jail_port is given, function will return values that
# notify the caller that translation should map host ports port to
# portend (on host) to jail_port to jail_port_end (on jail)
#
# jail_port_end might be omitted, in which case all input ports will
# map to a single port. If jail_port_end is '*', there will be a
# one-to-one mapping from the range of ports on host to the range of
# ports on jail, starting with 'port' and 'jail_port'.
#
# This function returns a reference to a hash whose key '$proto'
# contains a hash whose keys are the ports that translate to the same
# ports on the host. The returned hash reference might also contain
# another key, '${proto}_special' that describes port mapping through
# a hash whose keys represent host ports, while values represent host
# ports.
sub get_ports($$) {
my ($proto, $ports) = @_;
my @ports = map { s/$proto://; $_ }
grep { /$proto:/ }
split /,\s*/, $ports;
my %out = ();
foreach my $port (@ports) {
my ($rangeh, $rangej) = split /\//, $port;
$rangej = $rangej // $rangeh;
$rangeh =~ tr/-/:/;
$rangej =~ tr/-/:/;
say STDERR "comparing '$rangeh' to '$rangej'";
$out{$proto . ($rangeh eq $rangej?'':'_special')}->{$rangeh} = $rangej;
}
return \%out;
}
# Prints the option part of pf.conf
# This function creates variables 'jail_proto_ports' containing the
# list of ports that will be used verbatim on host, and a list of pairs of
# variables named 'jail_proto_ports_JX'/'jail_proto_ports_HX', where X
# is a number, which represents pairings of ports used on the jail and
# on the host, respectively.
sub print_opt($$$$) {
my ($fout, $JN, $proto, $ports) = @_;
my @normal = keys %{$ports->{$proto}};
my $special = $ports->{$proto . '_special'} // {};
say $fout qq(${JN}_\U$proto\E_PORTS="{) . (join ',', @normal) . qq(}") if 0+@normal;
my $cnt = 0;
foreach my $port (keys %$special) {
$cnt++;
say $fout qq(${JN}_\U$proto\E_PORTS_H${cnt}="$port");
say $fout qq(${JN}_\U$proto\E_PORTS_J${cnt}=") . $special->{$port} . qq(");
}
return (0+@normal, $cnt);
}
while (<>) {
# skip comments
next if /^#/;
chomp;
my ($jail, $ports) = split /;\s*/;
$ips{$jail} = `warden get ipv4 $jail`;
chomp($ips{$jail});
print STDERR "JAIL: $jail -> $ips{$jail}\n";
my $udp_ports = get_ports('udp', $ports);
my $tcp_ports = get_ports('tcp', $ports);
$ports{$jail} = { %$udp_ports, %$tcp_ports };
}
print STDERR "Writing data to output files\n";
open my $fopt, '>', $opt_file or die "Can't write to $opt_file: $!";
open my $fnat, '>', $nat_file or die "Can't write to $nat_file: $!";
foreach my $jail (keys %ips) {
my $JN = uc($jail);
say $fopt qq(${JN}_IP="$ips{$jail}");
my (%ok, %cnt);
($ok{udp}, $cnt{udp}) = print_opt($fopt, $JN, 'udp', $ports{$jail});
($ok{tcp}, $cnt{tcp}) = print_opt($fopt, $JN, 'tcp', $ports{$jail});
say $fnat qq,nat on $card from \$${JN}_IP to any -> ($card),;
foreach my $proto (qw(udp tcp)) {
say $fnat qq,rdr on $card inet proto $proto from any to port \$${JN}_\U$proto\E_PORTS -> \$${JN}_IP, if $ok{$proto};
foreach my $c (1..$cnt{$proto}) {
say $fnat qq,rdr on $card inet proto $proto from any to port \$${JN}_\U$proto\E_PORTS_H$c -> \$${JN}_IP port \$${JN}_\U$proto\E_PORTS_J$c,;
}
}
}
close $fopt or die "Can't close $fopt: $!\n";
close $fnat or die "Can't close $fnat: $!\n";