Chapter 17. Jails and Containers | FreeBSD Documentation Portal
Description: Jails improve on the concept of the traditional chroot environment in several ways
Keywords: jails,creating,managing,updating

# Chapter 17. Jails and Containers

## 17.1. Synopsis

Since system administration is a difficult task, many tools have been developed to make life easier for the administrator. These tools often enhance the way systems are installed, configured, and maintained. One of the tools which can be used to enhance the security of a FreeBSD system is jails. Jails have been available since FreeBSD 4.X and continue to be enhanced in their usefulness, performance, reliability, and security.

Jails build upon the chroot(2) concept, which is used to change the root directory of a set of processes. This creates a safe environment, separate from the rest of the system. Processes created in the chrooted environment can not access files or resources outside of it. For that reason, compromising a service running in a chrooted environment should not allow the attacker to compromise the entire system.

However, a chroot has several limitations. It is suited to easy tasks which do not require much flexibility or complex, advanced features. Over time, many ways have been found to escape from a chrooted environment, making it a less than ideal solution for securing services.

Jails improve on the concept of the traditional chroot environment in several ways.

In a traditional chroot environment, processes are only limited in the part of the file system they can access. The rest of the system resources, system users, running processes, and the networking subsystem are shared by the chrooted processes and the processes of the host system. Jails expand this model by virtualizing access to the file system, the set of users, and the networking subsystem. More fine-grained controls are available for tuning the access of a jailed environment. Jails can be considered as a type of operating system-level virtualization.

This chapter covers:

- What a jail is and what purpose it may serve in FreeBSD installations.

- The different types of jail.

- The different ways to configure the network for a jail.

- The jail configuration file.

- How to create the different types of jail.

- How to configure network address translation for jails.

- How to start, stop, and restart a jail.

- The basics of jail administration, both from inside and outside the jail.

- How to upgrade the different types of jail.

- How to limit the resources a jail can use.

- The additional per-jail security and feature options.

- How to delegate ZFS datasets to a jail.

- An incomplete list of the different FreeBSD jail managers.

## 17.2. Jail Types

Some administrators divide jails into different types, although the underlying technology is the same. Each administrator will have to assess what type of jail to create in each case depending on the problem they have to solve.

Below can be found a list of the different types, their characteristics, and considerations for use.

### 17.2.1. Thick Jails

A thick jail is a traditional form of FreeBSD Jail. In a thick jail, a complete copy of the base system is replicated within the jail’s environment. This means that the jail has its own separate instance of the FreeBSD base system, including libraries, executables, and configuration files. The jail can be thought of as an almost complete standalone FreeBSD installation, but running within the confines of the host system. This isolation ensures that the processes within the jail are kept separate from those on the host and other jails.

Advantages of Thick Jails:

- High degree of isolation: Processes within the jail are isolated from the host system and other jails.

- Independence: Thick jails can have different versions of libraries, configurations, and software than the host system or other jails.

- Security: Since the jail contains its own base system, vulnerabilities or issues affecting the jail environment will not directly impact the host or other jails.

Disadvantages of Thick Jails:

- Resource overhead: Because each jail maintains its own separate base system, thick jails consume more resources compared to thin jails.

- Maintenance: Each jail requires its own maintenance and updates for its base system components.

### 17.2.2. Thin Jails

A thin jail shares the base system from a template, using either OpenZFS snapshots or NullFS mounts. Only a minimal subset of the base system is duplicated for each thin jail, resulting in less resource consumption compared to a thick jail. The two mechanisms behave differently once the jails exist. With the OpenZFS method, each thin jail is created as a clone of a template snapshot: an independent copy-on-write copy that shares unchanged blocks with the template but diverges as it is modified. Later changes to the template do not propagate to existing clones, and each clone is maintained and updated individually, much like a thick jail. With the NullFS method, each thin jail mounts the template read-only at runtime, so the base system is shared live and updating the template updates every jail that mounts it. This shared, read-only base is also what gives NullFS thin jails less isolation and independence than thick jails, since changes in the shared components affect every jail using them simultaneously.

In summary, a FreeBSD Thin Jail is a type of FreeBSD Jail that replicates a substantial portion, but not all, of the base system within the isolated environment.

Advantages of Thin Jails:

- Resource Efficiency: Thin jails are more resource-efficient compared to thick jails. Since they share most of the base system, they consume less disk space and memory. This makes it possible to run more jails on the same hardware without consuming excessive resources.

- Faster Deployment: Creating and launching thin jails is generally faster compared to thick jails. This can be particularly advantageous when rapidly deploying multiple instances.

- Unified Maintenance: With the NullFS method, where the thin jails share a single read-only copy of the base system, updates and maintenance of common base system components (such as libraries and binaries) only need to be done once on the template. This simplifies the maintenance process compared to maintaining an individual base system for each thick jail. OpenZFS-clone thin jails do not share this benefit: because each clone is independent, updates must be applied to each jail individually, as with a thick jail.

- Shared Resources: Thin jails can more easily share common resources such as libraries and binaries with the host system. This can potentially lead to more efficient disk caching and improved performance for applications within the jail.

Disadvantages of Thin Jails:

- Reduced Isolation: The primary disadvantage of thin jails is that they offer less isolation compared to thick jails. With the NullFS method in particular, jails share a live copy of the template’s base system, so vulnerabilities or issues affecting those shared components could potentially impact multiple jails simultaneously.

- Security Concerns: The reduced isolation in thin jails could pose security risks, as a compromise in one jail might have a greater potential to affect other jails or the host system.

- Dependency Conflicts: If multiple thin jails require different versions of the same libraries or software, managing dependencies can become complex. In some cases, this might require additional effort to ensure compatibility.

- Compatibility Challenges: Applications within a thin jail might encounter compatibility issues if they assume a certain base system environment that differs from the shared components provided by the template.

### 17.2.3. Service Jails

A service jail shares the complete filesystem tree directly with the host (the jail root path is /) and as such can access and modify any file on the host, and shares the same user accounts with the host. By default it has no access to the network or other resources which are restricted in jails, but they can be configured to re-use the network of the host and to remove some of the jail-restrictions. The use case for service jails is automatic confinement of services/daemons inside a jail with minimal configuration, and without any knowledge of the files needed by such service/daemon. Service jails are available since FreeBSD 15.0.

Advantages of Service Jails:

- Zero Administration: A service jail ready service needs only one config line in /etc/rc.conf, a service which is not service jails ready needs two config lines.

- Resource Efficiency: Service jails are more resource efficient than thin jails, as they do not need any additional disk space or network resource.

- Faster Deployment: Creating and launching service jails is generally faster compared to thin jails if only distinct services/daemons shall be jailed and no parallel instances of the same service/daemon is needed.

- Shared Resources: Service jails share all resources such as libraries and binaries with the host system. This can potentially lead to more efficient disk caching and improved performance for applications within the jail.

- Process Isolation: Service jails isolate a particular service, it can not see processes which are not a child of the service jail, even if they run within the same user account.

Disadvantages of Service Jails:

- Reduced Isolation: The primary disadvantage of service jails is that they offer no filesystem isolation compared to thick or thin jails.

- Security Concerns: The reduced isolation in service jails could pose security risks, as a compromise in one jail might have a greater potential to affect everything on the host system.

Most of the configuration of jails which is discussed below is not needed for service jails. To understand how jails work, it is recommended to understand those configuration possibilities. The details about what is needed to configure a service jail are in the Service Jails section.

### 17.2.4. VNET Jails

A FreeBSD VNET jail is a virtualized environment that allows for the isolation and control of network resources for processes running within it. It provides a high level of network segmentation and security by creating a separate network stack for processes within the jail, ensuring that network traffic within the jail is isolated from the host system and other jails.

In essence, FreeBSD VNET jails add a network configuration mechanism. This means a VNET jail can be created as a Thick or Thin Jail.

### 17.2.5. Linux Jails

A FreeBSD Linux Jail is a feature in the FreeBSD operating system that enables the use of Linux binaries and applications within a FreeBSD jail. This functionality is achieved by incorporating a compatibility layer that allows certain Linux system calls and libraries to be translated and executed on the FreeBSD kernel. The purpose of a Linux Jail is to facilitate the execution of Linux software on a FreeBSD system without needing a separate Linux virtual machine or environment.

## 17.3. Host Configuration

Before creating any jail on the host system it is necessary to perform certain configuration and obtain some information from the host system.

It will be necessary to configure the jail(8) utility, create the necessary directories to configure and install jails, obtain information from the host’s network, and check whether the host uses OpenZFS or UFS as its file system.

| The FreeBSD version running in the jail can not be newer than the version running in the host.

### 17.3.1. Jail Utility

The jail(8) utility manages jails.

To start jails when the system boots, run the following commands:

    # sysrc jail_enable="YES"
    # sysrc jail_parallel_start="YES"

| With `jail_parallel_start`, all configured jails will be started in the background.

By default every jail with an entry in the configuration files is started at boot. To start only selected jails, list their names in `jail_list`:

    # sysrc jail_list="www db"

The names in `jail_list` are separated by spaces and must match the jail names defined in /etc/jail.conf or /etc/jail.conf.d/. Only the listed jails are started at boot, and they are started in the order given, so any jail left out of `jail_list` has to be started by hand. When `jail_list` is left empty, all jails defined in the configuration files are started.

To stop the jails in the reverse of their `jail_list` order at shutdown, set `jail_reverse_stop`:

    # sysrc jail_reverse_stop="YES"

| With `jail_parallel_start` enabled, jails start in the background and the `jail_list` order is no longer guaranteed. When one jail must be running before another starts, express that relationship with the `depend` parameter in /etc/jail.conf instead of relying on list order. A jail’s dependencies are created automatically before it starts, and stopped in the opposite order, regardless of `jail_list`. The `depend` parameter is described in Jail Configuration Files.

### 17.3.2. Networking

Networking for FreeBSD jails can be configured several different ways:

Host Networking Mode (IP Sharing)

  In host networking mode a jail shares the host’s network stack instead of receiving its own. Sharing the stack does not mean sharing the address: the common setup gives the jail its own dedicated `ip4.addr` (and `ip6.addr`) that jail(8) adds as an alias on the host interface, so the jail and the host answer on different addresses over the same stack. Only a jail configured with `ip4 = inherit` shares the host’s addresses wholesale and answers on the host’s own address, gaining unrestricted access to every address the host holds.

Virtual Networks (VNET)

  Virtual Networks are a feature of FreeBSD jails that offer more advanced and flexible networking solutions than a basic networking mode like host networking. VNET allows the creation of isolated network stacks for each jail, providing them with their own separate IP addresses, routing tables, and network interfaces. This offers a higher level of network isolation and allows jails to function as if they are running on separate virtual machines.

The netgraph system

  netgraph(4) is a versatile kernel framework for creating custom network configurations. It can be used to define how network traffic flows between jails and the host system and between different jails.

#### 17.3.2.1. Where a Jail’s IP Address Comes From

A shared-stack jail reaches the network through an address on a host interface. The `interface` parameter names that interface, and for every address in `ip4.addr` and `ip6.addr` jail(8) manages the alias itself: as jail(8) puts it, "an alias for each address will be added to the interface before the jail is created, and will be removed from the interface after the jail is removed". Nothing on the host has to pre-configure the address.

The interface can also be given per address, so a single line ties an address to the interface that should carry it:

    ip4.addr = "em0|192.168.1.151/24";

When an address must persist independently of the jail — because another service on the host relies on it, or because the `interface` parameter is not used — assign it on the host as an rc.conf alias and leave it out of jail(8)'s management:

    # sysrc ifconfig_em0_aliases="inet 192.168.1.151/32"

The older, numbered form `ifconfig_em0_alias0="inet 192.168.1.151/32"` still works, but the `_aliases` form is preferred for new configurations because it avoids the fragile requirement that the numbered aliases form an unbroken `alias0`, `alias1`, `alias2` sequence.

#### 17.3.2.2. Loopback Addresses in Shared-IP Jails

A shared-stack jail has no loopback interface of its own. The FreeBSD kernel compensates for this: in a non-VNET jail, whenever `127.0.0.1` or `::1` is not itself one of the jail’s own addresses, the kernel silently rewrites both `bind()` and `connect()` on the loopback address to the jail’s first `ip4.addr` or `ip6.addr`. This behavior is not documented in jail(8), but it matters, because it carries a security consequence.

| Do not assume that binding a service to `localhost` confines it to the jail. In a shared-IP jail a daemon told to listen on `127.0.0.1` actually binds the jail’s first address, which is frequently a LAN-facing one, and so becomes reachable from the network.

The standard mitigation is to give the jail a private loopback address and list it first. Clone a dedicated loopback interface on the host and assign it an address in the `127.0.0.0/8` range (or a private RFC 1918 address):

    # sysrc cloned_interfaces="lo1"
    # sysrc ifconfig_lo1="inet 127.0.1.1/32"

Then list that address first in the jail’s `ip4.addr`, ahead of any routable address, so the loopback remap lands on the private address instead of a LAN-facing one.

#### 17.3.2.3. Preparing Host Services

Each jail alias address shares the host’s ports. A host daemon bound to the wildcard address — shown as `*:port` in listener listings — is therefore listening on every jail alias too, and it occupies that port before a jail’s own service can claim it. Before starting any jails, audit what the host already listens on:

    # sockstat -4 -6 -l

Restrict each daemon that binds a wildcard address so it uses only the host’s own address and leaves the jail addresses free:

- syslogd(8): set `syslogd_flags="-ss"` in /etc/rc.conf. A single `-s` already runs syslogd in secure mode so it does not listen for remote log messages; per syslogd(8), if `-s` "is specified twice, no network socket will be opened at all", which also stops syslogd from sending to remote hosts.

- sshd(8): pin `ListenAddress` to the host’s address in /etc/ssh/sshd_config instead of leaving it on the default wildcard.

- ntpd(8) and sendmail(8): bind them to the host address as well through their own configuration.

For firewall-level control over which addresses a jail may use, see Firewalls.

### 17.3.3. Setting Up the Jail Directory Tree

There is no specific place to put the files for the jails.

Some administrators use /jail, others /usr/jail, and still others /usr/local/jails. In this chapter /usr/local/jails will be used.

Apart from /usr/local/jails other directories will be created:

- media will contain the compressed files of the downloaded userlands.

- templates will contain the templates when using Thin Jails.

- containers will contain the jails.

When using OpenZFS, execute the following commands to create datasets for these directories:

    # zfs create -o mountpoint=/usr/local/jails zroot/jails
    # zfs create zroot/jails/media
    # zfs create zroot/jails/templates
    # zfs create zroot/jails/containers

| In this case, `zroot` was used for the parent dataset, but other datasets could have been used.

Creating a separate child dataset per jail is what later enables per-jail snapshots, clones, quotas, and dataset delegation. Because each jail directory is then its own dataset, OpenZFS can both cap and guarantee its disk use with the `quota` and `reservation` properties; see Jail Resource Limits.

When using UFS, execute the following commands to create the directories:

    # mkdir /usr/local/jails/
    # mkdir /usr/local/jails/media
    # mkdir /usr/local/jails/templates
    # mkdir /usr/local/jails/containers

### 17.3.4. Jail Configuration Files

There are two ways to configure jails.

The first one is to add an entry for each jail to the file /etc/jail.conf. The other option is to create a file for each jail in the directory /etc/jail.conf.d/.

In case a host system has few jails, an entry for each jail can be added in the file /etc/jail.conf. If the host system has many jails, it is a good idea to have one configuration file for each jail in the /etc/jail.conf.d/ directory.

The files in /etc/jail.conf.d/ must have `.conf` as their extension and have to be included in /etc/jail.conf:

    .include "/etc/jail.conf.d/*.conf";

A typical jail entry would look like this:

    jailname { (1)
      # STARTUP/LOGGING
      exec.start = "/bin/sh /etc/rc"; (2)
      exec.stop = "/bin/sh /etc/rc.shutdown"; (3)
      exec.consolelog = "/var/log/jail_console_${name}.log"; (4)

      # PERMISSIONS
      allow.raw_sockets; (5)
      exec.clean; (6)
      mount.devfs; (7)

      # HOSTNAME/PATH
      host.hostname = "${name}"; (8)
      path = "/usr/local/jails/containers/${name}"; (9)

      # NETWORK
      ip4.addr = 192.168.1.151; (10)
      ip6.addr = "2001:db8::151"; (11)
      interface = em0; (12)
    }

1 | `jailname` - Name of the jail.
2 | `exec.start` - Command(s) to run in the jail environment when a jail is created. A typical command to run is "/bin/sh /etc/rc".
3 | `exec.stop` - Command(s) to run in the jail environment before a jail is removed. A typical command to run is "/bin/sh /etc/rc.shutdown".
4 | `exec.consolelog` - A file to direct command output (stdout and stderr) to.
5 | `allow.raw_sockets` - Allow creating raw sockets inside the jail. Setting this parameter allows utilities like ping(8) and traceroute(8) to operate inside the jail.
6 | `exec.clean` - Run commands in a clean environment.
7 | `mount.devfs` - Mount a devfs(5) filesystem on the chrooted /dev directory to restrict the devices visible inside the jail. It applies the ruleset named by the `devfs_ruleset` parameter, or, when that parameter is left at its default of `0`, the `mount.devfs` fallback of ruleset 4, `devfsrules_jail`, which hides every host device except a minimal, jail-safe set (see Devfs Rulesets).
8 | `host.hostname` - The hostname of the jail.
9 | `path` - The directory which is to be the root of the jail. Any commands that are run inside the jail, either by jail or from jexec(8), are run from this directory.
10 | `ip4.addr` - One IPv4 address, or a comma-separated list of addresses, assigned to the jail. Setting `ip4.addr` implies `ip4 = new`; `ip4` itself is not left at a standalone default. The `ip4` parameter accepts three values: `new` restricts the jail to the addresses listed in `ip4.addr`, `inherit` gives the jail unrestricted access to every IPv4 address of the host, and `disable` turns off IPv4 in the jail entirely. A jail with neither `ip4` nor `ip4.addr` set behaves like `inherit`, and `interface` and `ip4.addr` have no effect under `inherit`. In a jail with several addresses, `ip4.saddrsel` lets the kernel choose the source address for outgoing connections on unbound sockets, while `ip4.nosaddrsel` forces the first address in the list.
11 | `ip6.addr` - One IPv6 address, or a comma-separated list of addresses, assigned to the jail. The `ip6` parameter takes the same `new`, `inherit`, and `disable` values as `ip4`, with `ip6.saddrsel` and `ip6.nosaddrsel` controlling source-address selection. Use `ip6 = disable` for an IPv4-only jail, or `ip4 = disable` for an IPv6-only one.
12 | `interface` - A network interface to add the jail’s IP addresses. Usually the host interface.

| A jail has no virtual console: init(8) and getty(8) do not run inside it, so there is nothing to attach to interactively. The `exec.consolelog` file captures only the standard output and standard error of the `exec.*` commands, that is, the output of the /etc/rc startup and /etc/rc.shutdown scripts. It is not an interactive console; for that, use jexec(8) (see Access a Jail) or run sshd(8) inside the jail. The jail’s own daemons send their later output to the jail’s syslog.

| The `exec.consolelog` files grow without bound. Rotate them with newsyslog(8) by adding a single glob entry, for example in /etc/newsyslog.conf.d/jail:
    /var/log/jail_console_*.log            640  7     100  *     J

#### 17.3.4.1. Global Settings, Wildcards and Variables

Parameters set outside any jail block are wildcard defaults: jail.conf(5) applies them to every jail, exactly as if they were written in a `*` wildcard section. Setting the shared parameters once keeps each jail block short.

A name preceded by a dollar sign, and optionally enclosed in braces, is replaced with the value of that parameter or variable. The built-in `${name}` expands to the jail’s own name, and a custom variable is defined just like a parameter but with a leading dollar sign.

The following /etc/jail.conf defines a common header once and overrides only what differs per jail:

    # Shared defaults applied to every jail
    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
    exec.consolelog = "/var/log/jail_console_${name}.log";
    exec.clean;
    mount.devfs;
    host.hostname = "${name}";
    path = "/usr/local/jails/containers/${name}";
    interface = em0;

    www {
        ip4.addr = 192.168.1.151;
    }

    db {
        host.hostname = "database.example.com";
        ip4.addr = 192.168.1.152;
    }

Here `www` inherits everything from the header and sets only its address, while `db` also overrides `host.hostname`.

#### 17.3.4.2. Command Hooks

jail(8) runs a jail through a sequence of lifecycle hooks, and it matters whether each one runs in the host environment or inside the jail. In creation order:

`exec.prestart`

  Runs in the host environment before the jail is created.

`exec.created`

  Runs in the host environment after the jail exists but before any process starts inside it.

`exec.start`

  Runs inside the jail, typically `/bin/sh /etc/rc` to bring up its services.

`exec.poststart`

  Runs in the host environment after the jail has started.

Shutdown runs the mirror image:

`exec.prestop`

  Runs in the host environment before the jail is stopped.

`exec.stop`

  Runs inside the jail, typically `/bin/sh /etc/rc.shutdown`.

`exec.poststop`

  Runs in the host environment after the jail has stopped, to tear down whatever `exec.prestart` set up.

A hook may be assigned more than one command by appending with `+=`:

    exec.poststart += "logger jail ${name} started";

`exec.timeout` caps how long jail(8) waits for a command to finish, and `exec.consolelog` captures the output of these commands. The VNET section (VNET Jails) uses `exec.prestart` and `exec.poststop` to build and tear down each jail’s network plumbing on the host.

Where hooks order commands within a single jail, the `depend` parameter orders whole jails. Writing `depend = otherjail;` tells jail(8) that this jail requires `otherjail`: the dependency is created first, up to the completion of its last `exec.poststart` command, before this jail is created, and the jails are torn down in the reverse order.

#### 17.3.4.3. Devfs Rulesets

Every jail that sets `mount.devfs` gets a devfs ruleset that decides which device nodes appear under its /dev, either the one named by the `devfs_ruleset` parameter or, when that parameter is unset, the `mount.devfs` default of ruleset 4. devfs.rules(5) defines named rulesets in /etc/defaults/devfs.rules. Two of them are meant for jails: `devfsrules_jail`, ruleset 4, is the standard minimal device set for a jail, and `devfsrules_jail_vnet`, ruleset 5, adds /dev/pf so a VNET jail can run its own firewall. The VNET jail examples set `devfs_ruleset = 5` and the Linux jail example sets `devfs_ruleset = 4`; the classic and thin-jail examples set only `mount.devfs` and so use its default of ruleset 4.

To expose an extra device, write a custom ruleset in /etc/devfs.rules that includes one of the stock rulesets and unhides what it needs. The following ruleset starts from `devfsrules_jail` and additionally unhides the bpf(4) devices:

    [devfsrules_jail_bpf=100]
    add include $devfsrules_jail
    add path 'bpf*' unhide

Reload the rules, then reference the new ruleset by its number in the jail’s configuration:

    # service devfs restart

    devfs_ruleset = 100;

Rulesets in /etc/devfs.rules override same-numbered rulesets in /etc/defaults/devfs.rules; otherwise the two files are effectively merged.

More information about configuration variables can be found in jail(8) and jail.conf(5).

## 17.4. Classic Jail (Thick Jail)

These jails resemble a real FreeBSD system. They can be managed more or less like a normal host system and updated independently.

### 17.4.1. Creating a Classic Jail

In principle, a jail only needs a hostname, a root directory, an IP address, and a userland.

The userland for the jail can be obtained from the official FreeBSD download servers.

Execute the following command to download the userland:

    # fetch https://download.freebsd.org/releases/amd64/amd64/15.1-RELEASE/base.txz -o /usr/local/jails/media/15.1-RELEASE-base.txz

Once the download is complete, it will be necessary to extract the contents into the jail directory.

Execute the following commands to extract the userland into the jail’s directory:

    # mkdir -p /usr/local/jails/containers/classic
    # tar -xf /usr/local/jails/media/15.1-RELEASE-base.txz -C /usr/local/jails/containers/classic --unlink

With the userland extracted in the jail directory, it will be necessary to copy the timezone and DNS server files:

    # cp /etc/resolv.conf /usr/local/jails/containers/classic/etc/resolv.conf
    # cp /etc/localtime /usr/local/jails/containers/classic/etc/localtime

With the files copied, the next thing to do is update to the latest patch level by executing the following command:

    # freebsd-update -b /usr/local/jails/containers/classic/ fetch install

The last step is to configure the jail. It will be necessary to add an entry to the configuration file /etc/jail.conf or in jail.conf.d with the parameters of the jail.

An example would be the following:

    classic {
      # STARTUP/LOGGING
      exec.start = "/bin/sh /etc/rc";
      exec.stop = "/bin/sh /etc/rc.shutdown";
      exec.consolelog = "/var/log/jail_console_${name}.log";

      # PERMISSIONS
      allow.raw_sockets;
      exec.clean;
      mount.devfs;

      # HOSTNAME/PATH
      host.hostname = "${name}";
      path = "/usr/local/jails/containers/${name}";

      # NETWORK
      ip4.addr = 192.168.1.151;
      interface = em0;
    }

Execute the following command to start the jail:

    # service jail start classic

More information on how to manage jails can be found in the section Jail Management.

### 17.4.2. Obtaining the Jail Userland

The fetch-and-extract procedure shown above is the traditional way to populate a jail’s root directory, but it is not the only one. The same userland can be installed with bsdinstall(8), from base-system packages, or from a locally built source tree.

bsdinstall(8) provides a dedicated `jail` target that sets up a new userland at the given directory for use with jail(8). It fetches and extracts the distribution sets without installing a kernel, skipping the disk partitioning and network steps of a normal installation, which makes it a one-command, scriptable alternative to the manual fetch and tar(1) sequence:

    # bsdinstall jail /usr/local/jails/containers/classic

The `jail` target installs the base distribution (base.txz) by default. On FreeBSD 15.0 and later bsdinstall(8) can also populate the root from base-system packages with its `pkgbase` target; the pkg-based method is described next.

Starting with FreeBSD 15.0 the base system is also published as packages, so a jail userland can be installed entirely with pkg(8) from the `FreeBSD-base` repository. Point pkg(8) at the jail’s root directory with the global `--rootdir` option and install the jail variant of the base set:

    # pkg --rootdir /usr/local/jails/containers/classic install -r FreeBSD-base FreeBSD-set-base-jail

Use `FreeBSD-set-minimal-jail` instead of `FreeBSD-set-base-jail` for a minimal userland. A jail installed this way is afterwards patched and upgraded with pkg(8) rather than freebsd-update(8); see Upgrading pkgbase Jails.

| Base-system packages are a technology preview in FreeBSD 15.0. The `FreeBSD-base` repository is defined in /etc/pkg/FreeBSD.conf but disabled by default, so select it explicitly with `install -r FreeBSD-base`. The target root directory and its var/cache/pkg and var/db/pkg subdirectories may need to be created first. The traditional base.txz distribution is still published for FreeBSD 15.0, so the fetch-and-extract method continues to work there as well.

A userland built from a source tree can be installed into a jail’s root directory with the `DESTDIR` variable, which is the way to run a `-STABLE` or custom-built userland in a jail. After building the world, install it and the remaining distribution files into the jail directory:

    # cd /usr/src
    # make buildworld
    # make installworld DESTDIR=/usr/local/jails/containers/classic
    # make distribution DESTDIR=/usr/local/jails/containers/classic

`make distribution` installs the default configuration files under the jail’s /etc and is run only once, when populating the jail; repeating it during an update would overwrite the configuration inside the jail. To update such a jail from source, follow the same sequence as for the host: run `installworld` with the same `DESTDIR`, merge the configuration files with `etcupdate -D` pointed at the jail’s root directory, and remove obsolete files and libraries with `make delete-old delete-old-libs`, again with the same `DESTDIR`, rather than using freebsd-update(8). See Updating FreeBSD from Source for the complete source-update procedure.

### 17.4.3. Cloning a Jail as a Template

Once a thick jail is fully configured, with packages installed, services enabled, and local configuration in place, it can serve as a template image: a template from which many identical jails are deployed almost instantly.

On OpenZFS this is done by snapshotting the configured jail’s dataset and cloning it. This requires the source jail to live on its own dataset, so create it with `zfs create` rather than the `mkdir` used in the classic-jail procedure above:

    # zfs create -p zroot/jails/containers/classic

Stop the jail before snapshotting so the image is consistent, then snapshot and clone the dataset once for each new jail:

    # service jail stop classic
    # zfs snapshot zroot/jails/containers/classic@template
    # zfs clone zroot/jails/containers/classic@template zroot/jails/containers/web1

On a UFS host, copy the configured jail’s directory tree instead:

    # cp -a /usr/local/jails/containers/classic /usr/local/jails/containers/web1

Each clone needs its own entry in /etc/jail.conf, or in jail.conf.d, with a unique name, path, and IP address. Because the `path` and `host.hostname` parameters in the example above are derived from the block name with `${name}`, only the block name and the address differ:

    web1 {
      # STARTUP/LOGGING
      exec.start = "/bin/sh /etc/rc";
      exec.stop = "/bin/sh /etc/rc.shutdown";
      exec.consolelog = "/var/log/jail_console_${name}.log";

      # PERMISSIONS
      allow.raw_sockets;
      exec.clean;
      mount.devfs;

      # HOSTNAME/PATH
      host.hostname = "${name}";
      path = "/usr/local/jails/containers/${name}";

      # NETWORK
      ip4.addr = 192.168.1.152;
      interface = em0;
    }

A template-image clone differs from a thin jail (Thin Jails). A thin jail keeps sharing a read-only base and is maintained once at the template, whereas each template-image clone is a full, independently writable copy that is patched and upgraded on its own as described in Jail Upgrading. An OpenZFS clone starts as a space-efficient copy-on-write copy that shares blocks with the snapshot, and it keeps a dependency on that origin snapshot until `zfs promote` severs it. For a fully independent copy, or to replicate a jail to another pool or host, use `zfs send` piped to `zfs receive` instead of `zfs clone`.

## 17.5. Thin Jails

Although Thin Jails use the same technology as Thick Jails, the creation procedure is different. Thin jails can be created using OpenZFS snapshots or using templates and NullFS. The use of OpenZFS snapshots and templates using NullFS have certain advantages over classic jails, such as being able to create them faster from snapshots or being able to update multiple jails using NullFS.

### 17.5.1. Creating a Thin Jail Using OpenZFS Snapshots

Due to the good integration between FreeBSD and OpenZFS it is very easy to create new Thin Jails using OpenZFS Snapshots.

To create a Thin Jail using OpenZFS Snapshots the first step is to create the jail directory tree by following the instructions in Setting Up the Jail Directory Tree.

Next, create a template. Templates will only be used to create new jails. For this reason they are created in "read-only" mode so that jails are created with an immutable base.

To create the dataset for the template, execute the following command:

    # zfs create -p zroot/jails/templates/15.1-RELEASE

Then execute the following command to download the userland:

    # fetch https://download.freebsd.org/releases/amd64/amd64/15.1-RELEASE/base.txz -o /usr/local/jails/media/15.1-RELEASE-base.txz

Once the download is complete, it will be necessary to extract the contents in the template directory by executing the following command:

    # tar -xf /usr/local/jails/media/15.1-RELEASE-base.txz -C /usr/local/jails/templates/15.1-RELEASE --unlink

With the userland extracted in the templates directory, it will be necessary to copy the timezone and DNS server files to the template directory by executing the following command:

    # cp /etc/resolv.conf /usr/local/jails/templates/15.1-RELEASE/etc/resolv.conf
    # cp /etc/localtime /usr/local/jails/templates/15.1-RELEASE/etc/localtime

The next thing to do is update to the latest patch level by executing the following command:

    # freebsd-update -b /usr/local/jails/templates/15.1-RELEASE/ fetch install

Once the update is finished, the template is ready.

To create an OpenZFS Snapshot from the template, execute the following command:

    # zfs snapshot zroot/jails/templates/15.1-RELEASE@base

Once the OpenZFS Snapshot has been created, infinite jails can be created using the OpenZFS clone function.

To create a Thin Jail named `thinjail`, execute the following command:

    # zfs clone zroot/jails/templates/15.1-RELEASE@base zroot/jails/containers/thinjail

The last step is to configure the jail. It will be necessary to add an entry to the configuration file /etc/jail.conf or in jail.conf.d with the parameters of the jail.

An example would be the following:

    thinjail {
      # STARTUP/LOGGING
      exec.start = "/bin/sh /etc/rc";
      exec.stop = "/bin/sh /etc/rc.shutdown";
      exec.consolelog = "/var/log/jail_console_${name}.log";

      # PERMISSIONS
      allow.raw_sockets;
      exec.clean;
      mount.devfs;

      # HOSTNAME/PATH
      host.hostname = "${name}";
      path = "/usr/local/jails/containers/${name}";

      # NETWORK
      ip4 = inherit;
      interface = em0;
    }

Execute the following command to start the jail:

    # service jail start thinjail

More information on how to manage jails can be found in the section Jail Management.

### 17.5.2. Creating a Thin Jail Using NullFS

A jail can be created with reduced duplication of system files by using the Thin Jail technique and using NullFS to selectively share specific directories from the host system into the jail.

The first step is to create the dataset to save the template, execute the following command if using OpenZFS:

    # zfs create -p zroot/jails/templates/15.1-RELEASE-base

Or this one if using UFS:

    # mkdir /usr/local/jails/templates/15.1-RELEASE-base

Then execute the following command to download the userland:

    # fetch https://download.freebsd.org/releases/amd64/amd64/15.1-RELEASE/base.txz -o /usr/local/jails/media/15.1-RELEASE-base.txz

Once the download is complete, it will be necessary to extract the contents in the template directory by executing the following command:

    # tar -xf /usr/local/jails/media/15.1-RELEASE-base.txz -C /usr/local/jails/templates/15.1-RELEASE-base --unlink

Once the userland is extracted in the templates directory, it will be necessary to copy the timezone and DNS server files to the template directory by executing the following command:

    # cp /etc/resolv.conf /usr/local/jails/templates/15.1-RELEASE-base/etc/resolv.conf
    # cp /etc/localtime /usr/local/jails/templates/15.1-RELEASE-base/etc/localtime

With the files moved to the template, the next thing to do is update to the latest patch level by executing the following command:

    # freebsd-update -b /usr/local/jails/templates/15.1-RELEASE-base/ fetch install

In addition to the base template, it is also necessary to create a directory where the `skeleton` will be located. Some directories will be copied from the template to the `skeleton`.

Execute the following command to create the dataset for the `skeleton` in case of using OpenZFS:

    # zfs create -p zroot/jails/templates/15.1-RELEASE-skeleton

Or this one in case of using UFS:

    # mkdir /usr/local/jails/templates/15.1-RELEASE-skeleton

Then create the `skeleton` directories. The `skeleton` directories will hold the local directories of the jails.

Execute the following commands to create the directories:

    # mkdir -p /usr/local/jails/templates/15.1-RELEASE-skeleton/home
    # mkdir -p /usr/local/jails/templates/15.1-RELEASE-skeleton/usr
    # mv /usr/local/jails/templates/15.1-RELEASE-base/etc /usr/local/jails/templates/15.1-RELEASE-skeleton/etc
    # mv /usr/local/jails/templates/15.1-RELEASE-base/usr/local /usr/local/jails/templates/15.1-RELEASE-skeleton/usr/local
    # mv /usr/local/jails/templates/15.1-RELEASE-base/tmp /usr/local/jails/templates/15.1-RELEASE-skeleton/tmp
    # mv /usr/local/jails/templates/15.1-RELEASE-base/var /usr/local/jails/templates/15.1-RELEASE-skeleton/var
    # mv /usr/local/jails/templates/15.1-RELEASE-base/root /usr/local/jails/templates/15.1-RELEASE-skeleton/root

The next step is to create the symlinks to the `skeleton` by executing the following commands:

    # cd /usr/local/jails/templates/15.1-RELEASE-base/
    # mkdir skeleton
    # ln -s skeleton/etc etc
    # ln -s skeleton/home home
    # ln -s skeleton/root root
    # ln -s ../skeleton/usr/local usr/local
    # ln -s skeleton/tmp tmp
    # ln -s skeleton/var var

With the `skeleton` ready, it will be necessary to copy the data to the jail directory.

In case of using OpenZFS, OpenZFS snapshots can be used to easily create as many jails as necessary by executing the following commands:

    # zfs snapshot zroot/jails/templates/15.1-RELEASE-skeleton@base
    # zfs clone zroot/jails/templates/15.1-RELEASE-skeleton@base zroot/jails/containers/thinjail

In case of using UFS the cp(1) program can be used by executing the following command:

    # cp -R /usr/local/jails/templates/15.1-RELEASE-skeleton /usr/local/jails/containers/thinjail

Then create the directory in which the base template and the skeleton will be mounted:

    # mkdir -p /usr/local/jails/thinjail-nullfs-base

Add a jail entry in /etc/jail.conf or a file in jail.conf.d as follows:

    thinjail {
      # STARTUP/LOGGING
      exec.start = "/bin/sh /etc/rc";
      exec.stop = "/bin/sh /etc/rc.shutdown";
      exec.consolelog = "/var/log/jail_console_${name}.log";

      # PERMISSIONS
      allow.raw_sockets;
      exec.clean;
      mount.devfs;

      # HOSTNAME/PATH
      host.hostname = "${name}";
      path = "/usr/local/jails/${name}-nullfs-base";

      # NETWORK
      ip4.addr = 192.168.1.153;
      interface = em0;

      # MOUNT
      mount.fstab = "/usr/local/jails/${name}-nullfs-base.fstab";
    }

Then create the /usr/local/jails/thinjail-nullfs-base.fstab file as follows:

    /usr/local/jails/templates/15.1-RELEASE-base  /usr/local/jails/thinjail-nullfs-base/ nullfs   ro          0 0
    /usr/local/jails/containers/thinjail     /usr/local/jails/thinjail-nullfs-base/skeleton nullfs  rw  0 0

The fstab file controls how mount_nullfs(8) presents the base template and the skeleton inside the jail. The base template is mounted read-only (`ro`) so that every NullFS jail shares one immutable copy of the base system. The per-jail skeleton is mounted read-write (`rw`), which is where each jail keeps its own writable /etc, /var, /usr/local, and other local directories. Because all jails share that single read-only base template, updating the one template updates every NullFS jail at once, in contrast to independent OpenZFS clones which must each be updated separately.

Execute the following command to start the jail:

    # service jail start thinjail

## 17.6. VNET Jails

FreeBSD VNET Jails have their own distinct networking stack, including interfaces, IP addresses, routing tables, and firewall rules.

The first step to create a VNET jail is to create the bridge(4) by executing the following command:

    # ifconfig bridge create

The output should be similar to the following:

    bridge0

| Assigning an IP address to an interface that is a member of a bridge is deprecated. Assign the host’s address to `bridge0` itself rather than to `em0`, exactly as the /etc/rc.conf block below does. Such configurations keep working for now: FreeBSD 15.0 logs a kernel warning when an interface that carries an address is added to a bridge, and refuses it with `EINVAL` only when the `net.link.bridge.member_ifaddrs` sysctl is changed from its default of `1` to `0`. bridge(4) announces the removal of this sysctl in FreeBSD 16.0, after which addresses on bridge members will no longer be permitted.

With the `bridge` created, it will be necessary to attach it to the `em0` interface and bring both of them up by executing the following commands:

    # ifconfig bridge0 addm em0 up
    # ifconfig em0 up

To make this setting persist across reboots, add the following lines to /etc/rc.conf:

    defaultrouter="192.168.1.1"
    cloned_interfaces="bridge0"
    ifconfig_bridge0="inet 192.168.1.150/24 addm em0 up"
    ifconfig_em0="up"

For more information on bridging, see Network Bridging.

The next step is to create the jail as indicated above.

Either the Classic Jail (Thick Jail) procedure or the Thin Jails procedure can be used. The only thing that will change is the configuration in the /etc/jail.conf file.

The path /usr/local/jails/containers/vnet will be used as an example for the created jail.

The following is an example configuration for a VNET jail:

    vnet {
      # STARTUP/LOGGING
      exec.consolelog = "/var/log/jail_console_${name}.log";

      # PERMISSIONS
      allow.raw_sockets;
      exec.clean;
      mount.devfs;
      devfs_ruleset = 5;

      # PATH/HOSTNAME
      path = "/usr/local/jails/containers/${name}";
      host.hostname = "${name}";

      # VNET/VIMAGE
      vnet;
      vnet.interface = "${epair}b";

      # NETWORKS/INTERFACES
      $id = "154"; (1)
      $ip = "192.168.1.${id}/24";
      $gateway = "192.168.1.1";
      $bridge = "bridge0"; (2)
      $epair = "epair${id}";

      # ADD TO bridge INTERFACE
      exec.prestart  = "/sbin/ifconfig ${epair} create up";
      exec.prestart += "/sbin/ifconfig ${epair}a up descr jail:${name}";
      exec.prestart += "/sbin/ifconfig ${bridge} addm ${epair}a up";
      exec.start    += "/sbin/ifconfig ${epair}b ${ip} up";
      exec.start    += "/sbin/route add default ${gateway}";
      exec.start    += "/bin/sh /etc/rc";
      exec.stop      = "/bin/sh /etc/rc.shutdown";
      exec.poststop = "/sbin/ifconfig ${bridge} deletem ${epair}a";
      exec.poststop += "/sbin/ifconfig ${epair}a destroy";
    }

1 | A per-jail unique number used to derive both the epair device name (`epair${id}`) and the last octet of the jail’s address (`192.168.1.${id}`); give each jail on the bridge a different value.
2 | Refers to the bridge created previously.

The `vnet.interface` parameter names the interface that jail(8) moves into the jail’s network stack once the jail is created, here the jail side of the epair, `${epair}b`. That interface disappears from the host while the jail runs and is automatically released back to the host when the jail stops. A physical network card or an SR-IOV virtual function can be handed to a jail the same way, giving it dedicated networking without a bridge or an epair.

The `devfs_ruleset = 5` line applies the `devfsrules_jail_vnet` ruleset from /etc/defaults/devfs.rules, which extends the default jail ruleset by exposing /dev/pf. A VNET jail can therefore run its own pf(4) firewall with `pf_enable="YES"` and a private pf.conf, independent of the host. See Devfs Rulesets for how rulesets are defined and Firewalls for pf itself.

### 17.6.1. Automating VNET Networking with jib

Wiring up an epair and attaching it to the bridge by hand in `exec.prestart` and `exec.poststop`, as shown above, becomes tedious across many jails. The base system ships a helper script, /usr/share/examples/jails/jib, that performs the same plumbing automatically.

Reference it from the jail’s hooks instead of the five hand-rolled ifconfig(8) lines:

      exec.prestart += "jib addm ${name} em0";
      exec.poststop += "jib destroy ${name}";

      vnet.interface = "e0b_${name}";

`jib addm` creates an epair whose host side is named `e0a_<name>` and whose jail side is `e0b_<name>`, then attaches the host side to a bridge that it creates automatically, named after the member interface (for `em0` the bridge is `em0bridge`). Name that jail side in `vnet.interface`. When the jail stops, `jib destroy` removes the epair. The script also provides a `jib show` subcommand to list the interfaces it manages.

### 17.6.2. Netgraph VNET Networking with jng

The third networking mode mentioned in the Networking overview uses netgraph(4) instead of bridge(4) and epair. The base system ships a second helper, /usr/share/examples/jails/jng, that builds an `ng_bridge` connected to an `ng_eiface` for each jail:

      exec.prestart += "jng bridge ${name} em0";
      exec.poststop += "jng shutdown ${name}";

      vnet.interface = "ng0_${name}";

`jng bridge` creates the netgraph nodes and a jail-side interface named `ng0_<name>`, which is named in `vnet.interface`; `jng shutdown` tears the topology down when the jail stops. The result is functionally equivalent to the epair-and-bridge setup, but built on netgraph nodes rather than bridge(4).

### 17.6.3. Running a Firewall and DHCP inside a VNET Jail

Because `devfs_ruleset = 5` exposes /dev/pf, a VNET jail can run its own firewall. Enable it inside the jail exactly as on any host, with `pf_enable="YES"` in the jail’s /etc/rc.conf and its own /etc/pf.conf.

Ruleset 5 does not expose /dev/bpf, however. A VNET jail that obtains its address over DHCP, for example with `ifconfig_e0b_myjail="SYNCDHCP"` in its /etc/rc.conf, runs dhclient(8), which needs bpf and therefore fails under ruleset 5. Following the recipe in Devfs Rulesets, define a custom ruleset in /etc/devfs.rules that starts from `devfsrules_jail_vnet` and additionally unhides the bpf devices:

    [devfsrules_jail_vnet_dhcp=101]
    add include $devfsrules_jail_vnet
    add path 'bpf*' unhide

After running `service devfs restart` on the host, point the jail at the new ruleset with `devfs_ruleset = 101;`.

## 17.7. Linux Jails

FreeBSD can run a Linux userland inside a jail using Linux Binary Compatibility and sysutils/debootstrap. Jails do not have a kernel. They run on the host’s kernel. Therefore it is necessary to enable Linux Binary Compatibility in the host system.

To enable the Linux ABI at boot time, execute the following command:

    # sysrc linux_enable="YES"

Once enabled, it can be started without rebooting by executing the following command:

    # service linux start

The next step is to create a jail as indicated above, for example as in Creating a Thin Jail Using OpenZFS Snapshots, but without performing the configuration. FreeBSD Linux jails require a specific configuration that is detailed below.

If the host does not already have an /etc/jail.conf — for example on a system whose jails so far have been created only from the command line — create an empty one so that service(8) can stop the jail later:

    # touch /etc/jail.conf

The service(8) jail script reads /etc/jail.conf when stopping a jail, so the stop command fails if the file does not exist. If an /etc/jail.conf is already present from an earlier section, this step is not needed.

The following jail(8) command performs the required configuration for the jail and starts it directly, without a /etc/jail.conf entry. The `-c` flag creates a new jail and `-m` modifies an existing one; combined as `-cm`, jail(8) creates the jail if it does not yet exist and updates it if it does:

    # jail -cm \
        name=ubuntu \
        host.hostname="ubuntu.example.com" \
        path="/usr/local/jails/containers/ubuntu" \
        interface="em0" \
        ip4.addr="192.168.1.155" \
        exec.start="/bin/sh /etc/rc" \
        exec.stop="/bin/sh /etc/rc.shutdown" \
        mount.devfs \
        devfs_ruleset=4 \
        allow.mount \
        allow.mount.devfs \
        allow.mount.fdescfs \
        allow.mount.procfs \
        allow.mount.linprocfs \
        allow.mount.linsysfs \
        allow.mount.tmpfs \
        enforce_statfs=1

Access the newly created jail as `root` with jexec(8):

    # jexec -u root ubuntu sh

Inside the jail, install sysutils/debootstrap and use it to build the Ubuntu userland under /compat/ubuntu. sysutils/debootstrap is the tool that builds the Linux userland; it is not required to enter the jail afterward. Running pkg(7) for the first time bootstraps it automatically:

    # pkg install debootstrap
    # debootstrap jammy /compat/ubuntu

`jammy` selects Ubuntu 22.04 LTS. A newer LTS such as `noble` (Ubuntu 24.04) can be selected instead, provided it runs under the linuxulator.

When the process has finished and the message `Base system installed successfully` is displayed on the console, stop the jail from the host system. Because this jail is not enabled in /etc/rc.conf and not yet listed in /etc/jail.conf, use the `one`-prefixed service(8) action, which runs even for a jail that is not enabled:

    # service jail onestop ubuntu

The `onestart` and `onestop` actions are the counterparts of `start` and `stop` for jails that are not enabled through `jail_enable` or listed in `jail_list`.

Then add an entry in /etc/jail.conf for the Linux jail:

    ubuntu {
      # STARTUP/LOGGING
      exec.start = "/bin/sh /etc/rc";
      exec.stop = "/bin/sh /etc/rc.shutdown";
      exec.consolelog = "/var/log/jail_console_${name}.log";

      # PERMISSIONS
      allow.raw_sockets;
      exec.clean;
      mount.devfs;
      devfs_ruleset = 4;

      # HOSTNAME/PATH
      host.hostname = "${name}";
      path = "/usr/local/jails/containers/${name}";

      # NETWORK
      ip4.addr = 192.168.1.155;
      interface = em0;

      # MOUNT
      mount += "devfs     $path/compat/ubuntu/dev     devfs     rw  0 0";
      mount += "tmpfs     $path/compat/ubuntu/dev/shm tmpfs     rw,size=1g,mode=1777  0 0";
      mount += "fdescfs   $path/compat/ubuntu/dev/fd  fdescfs   rw,linrdlnk 0 0";
      mount += "linprocfs $path/compat/ubuntu/proc    linprocfs rw  0 0";
      mount += "linsysfs  $path/compat/ubuntu/sys     linsysfs  rw  0 0";
      mount += "/tmp      $path/compat/ubuntu/tmp     nullfs    rw  0 0";
      mount += "/home     $path/compat/ubuntu/home    nullfs    rw  0 0";
    }

| The jail can also present a specific Linux kernel identity to the programs running inside it. The `linux` parameter selects whether the jail inherits the host’s Linux emulation environment (`inherit`) or gets its own (`new`), and `linux.osname`, `linux.osrelease`, and `linux.oss_version` set the OS name, release, and OSS version that Linux binaries see. For example, `linux.osrelease=6.1.0` makes uname(1) and version-checking Linux programs inside the jail report that kernel release instead of the host-wide `compat.linux.osrelease` value.

Then the jail can be started as usual with the following command:

    # service jail start ubuntu

The Ubuntu environment can be accessed using the following command:

    # jexec ubuntu chroot /compat/ubuntu /bin/bash

More information can be found in the chapter Linux Binary Compatibility.

## 17.8. NAT and Port Forwarding for Jails

A jail that is given a private address — on a cloned lo(4) interface such as `lo1`, on a private bridge, or on a VNET epair on an internal network — has no route to the internet of its own. The host must translate the jail’s traffic to a routable address, and, to reach a service running inside such a jail from outside, redirect incoming connections to the jail.

First, allow the host to forward packets between its interfaces. Enable it at boot and immediately with the following commands:

    # sysrc gateway_enable="YES"
    # sysctl net.inet.ip.forwarding=1

For IPv6, use `ipv6_gateway_enable="YES"` and `net.inet6.ip6.forwarding=1`.

Then configure Network Address Translation on the host with pf(4). The following /etc/pf.conf gives every jail on the private `192.168.0.0/24` network outbound access through the host’s external interface `em0`, and publishes a jailed web server on the host’s port 80:

    ext_if = "em0"

    # Translate outbound jail traffic to the host address
    nat on $ext_if from 192.168.0.0/24 to any -> ($ext_if)

    # Forward inbound web traffic to the jailed server
    rdr on $ext_if proto tcp to port 80 -> 192.168.0.151

The `rdr` target must be the jail’s own address.

Enable and start pf(4) with the following commands:

    # sysrc pf_enable="YES"
    # service pf start

| FreeBSD 15.0 adds the OpenBSD-style inline `nat-to` and `rdr-to` syntax, which attaches translation to a `pass` or `match` rule, for example `match out on $ext_if from 192.168.0.0/24 to any nat-to ($ext_if)`. The classic `nat on` and `rdr on` rules shown above work on both FreeBSD 14.x and 15.x.

For the full treatment of pf(4) and ipfw(8), including complete rule sets and ipfw(8)-based NAT, see Firewalls.

## 17.9. Service Jails

Service jails are available since FreeBSD 15.0. A service jail is configured completely via /etc/rc.conf or sysrc(8). The base system services are service jails ready. They contain a config line which enables networking or lifts other restrictions of jails. Base system services which do not make sense to run inside jails are configured to not be started as a service jail, even if enabled in /etc/rc.conf. Some examples of such a service are services which want to mount or unmount something in the start or stop method, or only configure something like a route, or firewall, or the like.

Third party services may or may not be service jails ready. To check if a service is service jail ready, the following command can be used:

    # grep _svcj_options /path/to/rc.d/servicename

If there is no output, the service is not service jail ready, or does not need any additional privileges like for example, network access.

If the service is not service jail ready, and needs network access, it can be made ready by adding the necessary config to /etc/rc.conf:

    # sysrc servicename_svcj_options=net_basic

For all possible `_svcj_options` see the rc.conf(5) man-page.

To enable a service jail for a given service, the service needs to be stopped and the `servicename_svcj` variable needs to be set to YES. To put syslogd(8) into a service jail, use the following sequence of commands:

    # service syslogd stop
    # sysrc syslogd_svcj=YES
    # service syslogd start

If the `servicename_svcj` variable is changed, the service needs to be stopped before it is changed. If it is not stopped, the rc framework will not detect the correct state of the service and will not be able to do what is requested.

Service jails are managed only via rc.conf(5)/sysrc(8) and the service(8) command. The jail utilities, like jls(8) as described in Jail Management, can be used to investigate the operation, but the jail(8) command is not supposed to be used to manage them.

## 17.10. Jail Management

Once the jail is created, there are a number of operations that can be performed, like starting, rebooting or deleting the jail, installing software in it, etc. In this section the different actions that can be done with jails from the host will be described.

### 17.10.1. List Running Jails

To list the jails that are running on the host system, the command jls(8) can be used:

    # jls

The output should be similar to the following:

       JID  IP Address      Hostname                      Path
         1  192.168.1.151  classic                       /usr/local/jails/containers/classic

jls(8) supports the `--libxo` argument, which through the libxo(3) library allows other types of formats to be displayed, such as `JSON`, `HTML`, etc.

For example, execute the following command to get the `JSON` output:

    # jls --libxo=json

The output should be similar to the following:

    {"__version": "2", "jail-information": {"jail": [{"jid":1,"ipv4":"192.168.1.151","hostname":"classic","path":"/usr/local/jails/containers/classic"}]}}

The default listing is only a small part of what jls(8) can report. Add the `-v` flag for a verbose, multi-line summary of each jail that includes its state and its assigned cpuset ID:

    # jls -v

Use `-h` to print a header line with the field names above the values of the requested parameters:

    # jls -h jid name path

To query specific parameters of a single jail, name it with `-j` and list the parameters to print:

    # jls -j classic path host.hostname ip4.addr

The output should be similar to the following:

    /usr/local/jails/containers/classic classic 192.168.1.151

Add `-d` to include jails that are shutting down, which are otherwise hidden while in the dying state:

    # jls -d

### 17.10.2. Start, Restart, and Stop a Jail

service(8) is used to start, reboot, or stop a jail on the host.

For example, to start a jail, run the following command:

    # service jail start jailname

Change the `start` argument to `restart` or `stop` to perform other actions on the jail.

Stopping a jail is more than killing the processes inside it. When a jail is stopped, jail(8) first runs the commands in its `exec.stop` parameter — typically `/bin/sh /etc/rc.shutdown` — inside the jail so that its services shut down cleanly. It then sends `SIGTERM` to any processes still running in the jail and waits up to `stop.timeout` seconds (10 by default) for them to exit. Once that time has passed, the jail is removed, which kills whatever remains with `SIGKILL`. If `stop.timeout` is set to zero, no `SIGTERM` is sent and the jail is removed immediately. This is why a jail with a slow-stopping daemon can appear to hang for ten seconds while it stops; raise `stop.timeout` in /etc/jail.conf for a jail that runs a database or another service that needs longer to shut down gracefully.

### 17.10.3. Destroy a Jail

Destroying a jail is not as simple as stopping the jail using service(8) and removing the jail directory and /etc/jail.conf entry.

FreeBSD takes system security very seriously. For this reason there are certain files that not even the root user can delete. This functionality is known as File Flags. On a jail whose files live on a UFS file system, these flags must be cleared before the jail directory can be removed.

The first step is to stop the desired jail executing the following command:

    # service jail stop classic

The second step is to remove these flags with chflags(1) by executing the following command, in which `classic` is the name of the jail to remove:

    # chflags -R 0 /usr/local/jails/containers/classic

The third step is to delete the directory where the jail was:

    # rm -rf /usr/local/jails/containers/classic

The two steps above apply to a jail stored on a UFS file system. A jail backed by its own ZFS dataset — as created in Setting Up the Jail Directory Tree — is removed by destroying the dataset instead. Dataset destruction is not blocked by file flags, so the chflags(1) step is unnecessary in this case:

    # service jail stop classic
    # zfs destroy zroot/jails/containers/classic

Add `-r` to also destroy any snapshots the dataset holds.

| A ZFS thin jail created by cloning a template snapshot (see Thin Jails) depends on that snapshot. While any clone still exists, zfs(8) refuses to destroy the template snapshot and prints the list of dependent clones. To retire a template, first destroy every jail cloned from it, or run `zfs promote` on a clone so it no longer depends on the template.

Finally, in either case, remove the jail entry in /etc/jail.conf or in /etc/jail.conf.d. For a NullFS thin jail, also delete its .fstab file and its now-empty mountpoint directory.

### 17.10.4. Handle Packages in a Jail

The pkg(8) tool supports the `-j` argument in order to handle packages installed inside the jail.

For example, to install www/nginx-lite in the jail, the next command can be executed from the host:

    # pkg -j classic install nginx-lite

For more information on working with packages in FreeBSD, see Installing Applications: Packages and Ports.

### 17.10.5. Managing Services in a Jail from the Host

Services inside a jail can be enabled and controlled from the host without first entering the jail with jexec(8). Both service(8) and sysrc(8) accept a `-j` argument that names the target jail by ID or name and operates on that jail’s rc(8) configuration.

Continuing the package example above, enable and start nginx in the `classic` jail directly from the host:

    # sysrc -j classic nginx_enable=YES
    # service -j classic nginx start

These commands have the same effect as running sysrc(8) and service(8) through jexec(8), but they keep host-side administration and scripting uniform with an unjailed system.

### 17.10.6. Access a Jail

While it has been stated above that it is best to manage jails from the host system, a jail can be entered with jexec(8).

The jail can be entered by running jexec(8) from the host:

    # jexec -u root jailname

The `-u` flag selects the user to run as, resolving the name against the host password database. To resolve the user against the jail’s own password database instead, use `-U`. When no command is given, as here, jexec(8) starts that user’s login shell inside the jail. Add `-l` to run it in a clean login environment that keeps only `HOME`, `SHELL`, `TERM`, and `USER`, resets `PATH` to /bin:/usr/bin, and behaves like a fresh login; `-d` sets the working directory, which defaults to the jail root.

When gaining access to the jail, the message configured in motd(5) will be displayed.

### 17.10.7. Execute Commands in a Jail

To execute a command from the host system in a jail the jexec(8) can be used.

For example, to stop a service that is running inside a jail, the command will be executed:

    # jexec -l jailname service nginx stop

### 17.10.8. Viewing Jailed Processes and Sockets from the Host

From the host, the processes and sockets of every jail are visible and can be filtered by jail.

ps(1) accepts `-J` to restrict its output to a single jail, given by JID or name:

    # ps -J classic

The output should be similar to the following, where the `J` in the `STAT` column marks a process that is running in a jail:

     PID TT  STAT    TIME COMMAND
    2841  -  IsJ  0:00.02 /usr/sbin/syslogd -ss
    2903  -  IsJ  0:00.00 nginx: master process /usr/local/sbin/nginx

Use `ps -J 0` to show only the host’s own processes. The `jail` and `jid` keywords can also be added to the `-o` output format to show which jail each process belongs to. For example, to list every process on the host together with its jail:

    # ps ax -o pid,jail,command

top(1) takes `-J` to filter its live display to one jail, which implies `-j` and adds the `JID` column:

    # top -J classic

sockstat(1) lists the sockets belonging to a jail with `-j`, which is the natural way to check which ports a jail’s daemons have actually bound after starting a service in it:

    # sockstat -4 -6 -l -j classic

From inside a jail the picture is reversed: a jailed process sees only its own jail’s processes and sockets, never those of the host or of other jails.

### 17.10.9. Managing Jails Directly with the jail Utility

The service(8) wrapper shown above is the usual way to run jails at boot, but jail(8) can also create, modify, and remove jails directly, which is useful for scripting and for testing a configuration without the rc framework.

Create a configured jail by name:

    # jail -c classic

Remove a running jail; this also removes any child jails it created and kills the processes inside it:

    # jail -r classic

The `-m` flag modifies the parameters of an already-running jail without restarting it. For example, to add a second IPv4 address to a running jail:

    # jail -m name=classic ip4.addr+=192.168.250.71

jail(8) notes that "Some parameters may not be changed on a running jail"; in practice this includes `path` and `vnet`, which require the jail to be restarted.

Finally, `-e` exhibits all configured non-wildcard jails and their parameters without creating, modifying, or removing anything, which is handy for verifying a configuration file. It takes a string used to separate the parameters:

    # jail -e ,

### 17.10.10. Backing Up and Migrating a Jail

A jail is largely self-contained, so backing one up or moving it to another host is mostly a matter of copying its files and its configuration.

Stop the jail first for a consistent copy; a ZFS-backed jail can instead be snapshotted while running. For a jail on its own ZFS dataset, take a recursive snapshot and stream it to the destination host:

    # service jail stop classic
    # zfs snapshot -r zroot/jails/containers/classic@migrate
    # zfs send -R zroot/jails/containers/classic@migrate | ssh newhost zfs receive zroot/jails/containers/classic

The full mechanics of zfs-send(8) and zfs-receive(8), including incremental transfers, are covered in Replication.

For a jail on a UFS file system, use tar(1) with `-p` so that ownership, permissions, and file flags are preserved:

    # tar -cpf classic.tar -C /usr/local/jails/containers classic

Extract the archive as `root` on the destination host with `tar -xpf` so the flags survive. dump(8) and restore(8) are an alternative when a jail occupies a dedicated UFS file system of its own; they operate on whole file systems, so they do not apply to the layout used in this chapter, where all jails are directories within one shared file system.

Finally, copy the jail’s block from /etc/jail.conf — or its file in /etc/jail.conf.d — to the new host, adjust any host-specific parameters such as `interface`, `ip4.addr`, or the bridge used by a VNET jail, and start the jail there.

| The destination host must not run an older FreeBSD release than the jail’s userland, following the same rule that a jail can never run a newer version than its host. A NullFS thin jail is not self-contained: its template and skeleton must exist on the destination host too, so migrate those as well or convert the jail to a thick copy first. Any rctl(8) rules kept in /etc/rctl.conf must be carried over separately.

## 17.11. Administration from Inside a Jail

Most jail administration is done from the host, but some tasks are performed from within the jail itself, and a script running inside a jail sometimes needs to know that it is jailed. This section describes what an administrator sees and can do from inside a jail.

A process can detect whether it is running inside a jail by reading the `security.jail.jailed` sysctl, which is `1` inside a jail and `0` on the host:

    # sysctl security.jail.jailed
    security.jail.jailed: 1

Inside a jail the environment is deliberately restricted:

- The host’s view of the system is hidden. A jailed process sees only its own jail’s processes, sockets, and file systems, never those of the host or of other jails.

- Kernel state is largely read-only. Kernel tunables cannot be written from inside a jail, kldload(8) and kldunload(8) are not permitted, and dmesg(8) and the kernel message buffer are unavailable unless `allow.read_msgbuf` is set.

- Privileged operations that would cross the jail boundary are denied unless the host explicitly allows them. Creating raw sockets — as ping(8) and traceroute(8) require — needs `allow.raw_sockets` (see the parameter description in Jail Configuration Files), mounting file systems needs `allow.mount` and its per-filesystem children, and chflags(1) is disabled by default (see Jail Upgrading).

A handful of per-jail sysctl variables can be changed from inside the jail without affecting the host, including `kern.hostname`, `kern.domainname`, `kern.hostid`, `kern.hostuuid`, `kern.securelevel`, and `security.bsd.suser_enabled`. The current set of available jail parameters can be listed with sysctl(8):

    # sysctl -d security.jail.param

Managing the jail’s own services from inside works exactly as it does on an unjailed system: service(8) and sysrc(8) operate on the jail’s own rc(8) configuration. This is the in-jail counterpart to the host-side service -j and sysrc -j shown above.

## 17.12. Jail Upgrading

Upgrading FreeBSD Jails ensures that the isolated environments remain secure, up-to-date, and in line with the latest features and improvements available in the FreeBSD ecosystem.

The right tool depends on how the jail’s userland was installed. Jails installed from a distribution set (the base.txz extracted in the creation sections, or one produced by bsdinstall(8)) are updated from the host with freebsd-update(8) through its `-j` flag, on both FreeBSD 14.x and 15.x. Jails installed from base packages, available from FreeBSD 15.0, are updated with pkg(8) instead; freebsd-update(8) does not apply to them. Jails built from source are updated with the source procedure (`make installworld` and `make distribution`). Never mix these mechanisms on the same jail.

| Support for distribution sets is planned for removal in FreeBSD 16, but will continue, together with freebsd-update(8), for the lifetime of the FreeBSD 15 stable branch. Base packages are offered as a technology preview in FreeBSD 15.0 and are expected to become the standard method in future releases, so pkgbase is the forward-looking path.

| On OpenZFS, snapshot the jail’s dataset before upgrading, so a failed upgrade can be undone instantly:
    # zfs snapshot zroot/jails/containers/classic@preupgrade
If the upgrade fails, stop the jail and run `zfs rollback zroot/jails/containers/classic@preupgrade` to restore the previous state. This requires the jail to be its own dataset: the OpenZFS thin-jail clone already is, but a classic jail created with mkdir(1) is not, so create it with `zfs create` instead. For a NullFS thin jail, snapshot the template dataset instead. zfs-rollback(8) reverts only to the most recent snapshot unless `-r` is given to destroy any newer ones.

### 17.12.1. Upgrading a Classic Jail or a Thin Jail using OpenZFS Snapshots

Jails must be updated from the host operating system. The default behavior in FreeBSD is to disallow the use of chflags(1) in a jail. This will prevent the update of some files so updating from within the jail will fail.

To update the jail to the latest patch release of the version of FreeBSD it is running, execute the following commands on the host:

    # freebsd-update -j classic fetch install
    # service jail restart classic

To upgrade the jail to a new major or minor version, first upgrade the host system as described in Performing Major and Minor Version Upgrades. Once the host has been upgraded and rebooted, the jail can then be upgraded.

| In case of upgrade from one version to another, it is easier to create a new jail than to upgrade completely.

For example, to upgrade the `classic` jail to 15.1-RELEASE, execute the following commands on the host:

    # freebsd-update -j classic -r 15.1-RELEASE upgrade
    # freebsd-update -j classic install
    # service jail restart classic
    # freebsd-update -j classic install
    # service jail restart classic

| The `install` step is run twice on purpose. A jail has no kernel of its own (it runs the host’s kernel), so nothing here upgrades a kernel. The first `install` writes the new userland into the jail; after the jail restarts on the updated files, the second `install` finishes replacing the components that could not be changed while their previous versions were still in use.

Then, if it was a major version upgrade, reinstall all installed packages and restart the jail again. This is required because the ABI version changes when upgrading between major versions of FreeBSD.

From the host:

    # pkg -j classic upgrade -f
    # service jail restart classic

### 17.12.2. Upgrading OpenZFS Thin Jails

The thin jails created in Creating a Thin Jail Using OpenZFS Snapshots are OpenZFS clones of a template snapshot, but a clone is independent of that template once it exists. Refreshing the template does not update jails that were already cloned from it, and updating a clone does not update the template. There are therefore two ways to keep these jails current.

The first option is to update each clone in place with freebsd-update(8), exactly as for a classic jail, as described in Upgrading a Classic Jail or a Thin Jail using OpenZFS Snapshots. This is the simplest approach for a handful of jails, but every clone that is patched diverges from the template snapshot: the changed blocks are duplicated per jail, which erodes the space savings, and a major upgrade rewrites so much of the userland that most of the sharing is lost.

The second option keeps the sharing intact by refreshing the template and re-cloning. Update the template to the latest patch level on the host:

    # freebsd-update -b /usr/local/jails/templates/15.1-RELEASE/ fetch install

| freebsd-update(8) with `-b` reads the running release from the host’s `uname -r`, not from the template. If the template runs an older release than the host, add `--currently-running <template-release>` so it patches the template’s release instead of the host’s, as shown in Upgrading a Thin Jail Using NullFS.

Then take a new snapshot and clone future jails from it:

    # zfs snapshot zroot/jails/templates/15.1-RELEASE@base-p2
    # zfs clone zroot/jails/templates/15.1-RELEASE@base-p2 zroot/jails/containers/newjail

Existing clones keep the base they were created from until they are re-created from the new snapshot. For a major or minor version upgrade, create a fresh template dataset for the new release following Creating a Thin Jail Using OpenZFS Snapshots, rather than upgrading the existing template in place, and clone new jails from its `@base` snapshot.

| Keep each jail’s application data and configuration on separate datasets, or in the jail’s local storage, so that re-creating a jail from an updated template snapshot is cheap and preferable to patching every clone individually.

### 17.12.3. Upgrading a Thin Jail Using NullFS

Since Thin Jails that use NullFS share the majority of system directories, they are very easy to update. It is enough to update the template. This allows updating multiple jails at the same time.

To update the template to the latest patch release of the version of FreeBSD it is running, execute the following commands on the host:

    # freebsd-update -b /usr/local/jails/templates/15.1-RELEASE-base/ fetch install
    # service jail restart

To upgrade the template to a new major or minor version, first upgrade the host system as described in Performing Major and Minor Version Upgrades. Once the host has been upgraded and rebooted, the template can then be upgraded.

Because the host was upgraded first, freebsd-update(8) with `-b` would otherwise detect the host’s new release rather than the template’s older one: `-b` reads the running version from the host’s `uname -r`, unlike `-j`, which detects the jail’s own userland. Pass `--currently-running <template-release>` so the template is patched from its actual release. The following example upgrades a template running 14.2-RELEASE to 14.3-RELEASE; substitute the releases that apply:

    # freebsd-update -b /usr/local/jails/templates/14.2-RELEASE-base/ --currently-running 14.2-RELEASE -r 14.3-RELEASE upgrade
    # freebsd-update -b /usr/local/jails/templates/14.2-RELEASE-base/ install
    # service jail restart
    # freebsd-update -b /usr/local/jails/templates/14.2-RELEASE-base/ install
    # service jail restart

### 17.12.4. Upgrading pkgbase Jails

Jails installed from base packages, available from FreeBSD 15.0, are upgraded with pkg(8) against the `FreeBSD-base` repository rather than with freebsd-update(8).

The `FreeBSD-base` repository is defined in /etc/pkg/FreeBSD.conf but is disabled by default on FreeBSD 15.0, so enable it before upgrading.

From the host, point pkg(8) at the jail’s root directory with `-r` (`--rootdir`):

    # pkg -r /usr/local/jails/containers/pkgjail update
    # pkg -r /usr/local/jails/containers/pkgjail upgrade
    # service jail restart pkgjail

The same upgrade can be run from inside the jail with jexec(8) and a plain `pkg upgrade`:

    # jexec -l pkgjail pkg upgrade
    # service jail restart pkgjail

pkg(8) performs a three-way merge on tagged configuration files under /etc. When a merge cannot be completed automatically it leaves the new version as <file>.pkgnew, and when it takes over a file that was not previously packaged it saves the old copy as <file>.pkgsave. Review /etc after an upgrade and reconcile any such files by hand. Do not run etcupdate(8) on a pkgbase system; pkg(8) handles configuration merging itself. See Updating and Upgrading FreeBSD for the host-side base-package workflow.

## 17.13. Jail Resource Limits

Controlling the resources that a jail uses from the host system is a task to be taken into account by the system administrator.

Use rctl(8) to manage the resources that a jail can use from the host system.

| The `kern.racct.enable` tunable must be enabled at /boot/loader.conf.

The syntax to limit the resources of a jail is as follows:

    rctl -a jail:<jailname>:resource:action=amount[/per]

The optional `per` component is not a percentage; it is the accounting entity the limit is applied to, one of `process`, `user`, `loginclass`, or `jail`. For example, `deny=2G/jail` limits the sum for the whole jail, while `deny=2G/process` applies the same figure to each process individually. A percentage is only meaningful for the `pcpu` resource, whose amount is expressed as a percentage of a single CPU core.

For example, to limit the maximum RAM that a jail can access, run the following command:

    # rctl -a jail:classic:memoryuse:deny=2G

To make the limitation persistent across reboots of the host system, it will be necessary to add the rule to the /etc/rctl.conf file as follows:

    jail:classic:memoryuse:deny=2G/jail

rctl(8) can limit far more than memory. The resources most relevant to a jail include `pcpu` (percentage of a single CPU core), `maxproc` (number of processes), `openfiles` (open file descriptors), `vmemoryuse` (address space), and `swapuse` (swap space). Each of them is set the same way:

    # rctl -a jail:classic:pcpu:deny=80
    # rctl -a jail:classic:maxproc:deny=1000
    # rctl -a jail:classic:openfiles:deny=5000
    # rctl -a jail:classic:vmemoryuse:deny=4G

Disk throughput is limited with `readbps` and `writebps`, but these resources do not support the `deny` action; pair them with `throttle` instead:

    # rctl -a jail:classic:writebps:throttle=1m

The action after the resource selects what happens when the limit is reached: `deny` blocks the operation, `log` records it to the system log, and `devctl` sends a notification through devd(8).

Report the current resource usage of a jail with the `-u` flag, adding `-h` for human-readable units:

    # rctl -hu jail:classic

List the rules that apply to a jail:

    # rctl jail:classic

Remove every rule that matches a jail with `-r`:

    # rctl -r jail:classic

More information on resource limits can be found in the security chapter in the Resource Limits section.

### 17.13.1. CPU Pinning with cpuset

rctl(8) with `pcpu` caps how much CPU time a jail may consume, but it does not tie the jail to particular cores; cpuset(1) assigns specific cores instead.

Restrict a running jail to a set of CPUs by its name or JID:

    # cpuset -l 0-3 -j classic

Query the cores a jail is currently allowed to use with `-g`:

    # cpuset -g -j classic

The assigned set is also reported in the cpuset column of `jls -v`. There is no jail.conf parameter for CPU affinity, so pin the jail at start time by running cpuset(1) from an `exec.created` or `exec.poststart` hook, both of which run in the host environment:

    exec.poststart = "cpuset -l 0-3 -j classic";

### 17.13.2. Limiting Disk Space

rctl(8) has no resource for filesystem space. When each jail lives in its own ZFS dataset, cap and guarantee its disk usage with ZFS quotas and reservations instead:

    # zfs set quota=20G zroot/jails/containers/classic
    # zfs set reservation=5G zroot/jails/containers/classic

This requires the jail to be a dataset of its own. The ZFS thin jails created earlier already are, but the Classic Jail procedure creates the jail directory with `mkdir` inside the shared containers dataset, so create it with `zfs create` first for these commands to apply. See Setting Up the Jail Directory Tree for the dataset layout and zfs-set(8) for the `quota` and `reservation` properties.

## 17.14. More Jail Options

jail(8) exposes many more parameters than the examples in this chapter use. A jail is a strong isolation boundary, but it is not an absolute one. Every `allow.*` relaxation trades isolation for capability and moves the jail’s privileged users a step closer to the privileges of the host’s `root`. Grant a jail only the permissions the workload inside it genuinely needs, and leave everything else at its more restrictive default. The full list is in jail(8) and jail.conf(5); the parameters below are the ones an administrator reaches for most often.

### 17.14.1. The allow.* Permissions

The `allow.*` parameters are booleans that hand individual privileges to the jail. Per jail(8), "With the exception of allow.set_hostname and allow.reserved_ports, these boolean parameters are off by default", so a jail starts locked down and each capability must be granted explicitly.

Table 1. Common `allow.*` parameters
Parameter | Default | Effect
--- | --- | ---
`allow.set_hostname` | On | The jail may change its own hostname.
`allow.reserved_ports` | On | Jailed `root` may bind the reserved ports below 1024.
`allow.raw_sockets` | Off | Enables raw sockets, so ping(8) and traceroute(8) work inside the jail.
`allow.chflags` | Off | Jailed `root` may manipulate system file flags as it would on the host.
`allow.mount` | Off | The jail may mount and unmount filesystems, together with the `allow.mount.*` children.
`allow.quotas` | Off | The jail may administer filesystem quotas.
`allow.read_msgbuf` | Off | Jailed processes may read the kernel message buffer with dmesg(8).
`allow.socket_af` | Off | Allows socket address families other than IPv4, IPv6, local, and route.
`allow.mlock` | Off | Jailed processes may lock physical memory with mlock(2).
`allow.unprivileged_proc_debug` | Off | Unprivileged users may use debugging facilities on their own processes.
`allow.vmm` | Off | The jail may use the vmm(4) hypervisor to run bhyve(8) guests.
`allow.nfsd` | Off | The jail may run an NFS server.

Weigh each relaxation against what it exposes: `allow.raw_sockets` lets jailed processes forge packets, `allow.chflags` lets jailed `root` clear the immutable and append-only flags the host relies on, and `allow.mount` opens up the filesystem namespace.

Running an NFS server inside a jail, highlighted in the FreeBSD 14.0 release notes, needs `allow.nfsd` set, a VNET jail whose root directory is itself a filesystem mount point, and `enforce_statfs` not set to `0`.

### 17.14.2. System V IPC

Databases and other software that rely on System V shared memory, semaphores, and message queues are controlled per module with the `sysvmsg`, `sysvsem`, and `sysvshm` parameters. Each takes one of three values: `inherit` makes all of the host’s IPC objects visible to the jail, `new` gives the jail its own private key namespace, and `disable` makes the related system calls fail.

For the classic case of running PostgreSQL in a jail, `new` is the right choice, giving the database its own shared-memory namespace without exposing the host’s:

    sysvmsg = new;
    sysvsem = new;
    sysvshm = new;

The older `allow.sysvipc` parameter is deprecated. It is equivalent to setting all three module parameters to `inherit`, which shares the host-wide IPC namespace with the jail, so prefer the per-module parameters above.

### 17.14.3. Mounting Filesystems

`allow.mount` lets a jail mount filesystems, and a family of child parameters selects which types: `allow.mount.devfs`, `allow.mount.fdescfs`, `allow.mount.fusefs`, `allow.mount.nullfs`, `allow.mount.procfs`, `allow.mount.linprocfs`, `allow.mount.linsysfs`, `allow.mount.tmpfs`, and `allow.mount.zfs`. Per jail(8), each child "is effective only together with allow.mount and only when enforce_statfs is set to a value lower than 2". The convenience parameters `mount.devfs`, `mount.fdescfs`, and `mount.procfs` mount those filesystems for the jail automatically, and `mount.fstab` points at an fstab file of additional mounts.

The `enforce_statfs` parameter controls both which mount points a jail can see and, through the rule above, whether the `allow.mount.*` permissions take effect. It takes three values:

Table 2. `enforce_statfs` values
Value | Mount points visible to the jail
--- | ---
`0` | All mount points on the host, with no restrictions.
`1` | Only the mount points below the jail’s root directory.
`2` (default) | Only the single mount point where the jail’s root directory is located.

Because the `allow.mount.*` permissions require a value lower than `2`, a jail that must mount its own filesystems is typically configured with `enforce_statfs = 1`.

### 17.14.4. Per-Jail securelevel

The `securelevel` parameter sets the jail’s `kern.securelevel` and raises the kernel’s immutable-flag and device protections inside the jail. Per jail(8), "A jail never has a lower securelevel than its parent system, but by setting this parameter it may have a higher one". A network-facing jail can therefore run at a hardened `securelevel = 3` while the host stays at its default.

### 17.14.5. Hierarchical Jails

A jail may itself create child jails, which is how nested or hierarchical jail hierarchies are built. The `children.max` parameter sets how many child jails a jail may create; it defaults to `0`, meaning no child jails are allowed, and the read-only `children.cur` reports how many currently exist. A child can never exceed the privileges of its parent: per jail(8), "Jailed processes are not allowed to confer greater permissions than they themselves are given". This makes it possible to run a jail manager inside a jail.

### 17.14.6. Empty and Persistent Jails

By default a jail exists only while it has running processes; when its last process exits, the jail disappears. Setting the `persist` parameter, in the words of jail(8), "allows a jail to exist without any processes". This is what keeps a jail alive after it has been created empty and before anything is populated into it, and what anchors a VNET jail that only holds network state. A jail can be created empty and kept alive for experimentation:

    # jail -c name=test path=/rescue persist

Clear the flag with `nopersist` to let the jail be removed once idle again.

### 17.14.7. Presenting a Different OS Release

A jail’s userland may be older than the host kernel, but tools inside it still read the host’s version through uname(1) unless told otherwise. The `osrelease` parameter sets the string reported by the jail’s `kern.osrelease` and `uname -r`, and `osreldate` sets the number reported by `kern.osreldate` and `uname -K`. Setting them so that an older userland reports its own release keeps uname(1), sysctl(8) consumers, and version-sniffing build tooling behaving correctly, including freebsd-update(8) run inside the jail. Package management is unaffected either way, because pkg(8) derives its ABI from the ELF notes of the jail’s own binaries rather than from uname(1). This complements the rule that a jail’s userland can never be newer than the host kernel; see Jail Upgrading.

## 17.15. Jails and ZFS

A jail can be given control of one or more ZFS datasets, allowing the jailed root user to create, snapshot, clone, and roll back child datasets without any involvement from the host administrator. This is the recommended way to give a jail its own writable, snapshot-capable storage.

The host retains ownership of the dataset and only delegates management of the delegated tree and its descendants to the jail. The `jailed` property, the `zfs jail` subcommand, and the interaction between the two are described in ZFS and Jails.

### 17.15.1. Delegating a Dataset Manually

This method works on all supported releases of FreeBSD.

On the host, create the dataset to delegate and mark it as jailed:

    # zfs create zroot/jails/data
    # zfs set jailed=on zroot/jails/data

Setting the `jailed` property tells ZFS that the dataset will be managed from inside a jail, so the host stops mounting it automatically.

Next, allow the jail to mount ZFS filesystems and attach the dataset when the jail is created. Add the following to the jail’s section in /etc/jail.conf:

    allow.mount;
    allow.mount.zfs;
    enforce_statfs = 1;
    exec.created += "zfs jail ${name} zroot/jails/data";

The `allow.mount.zfs` permission takes effect only together with `allow.mount` and an `enforce_statfs` value below 2; see More Jail Options for the mount permissions and how they interact with `enforce_statfs`. Because a delegated dataset is detached when the jail stops, the `exec.created` hook re-attaches it on every start, where `${name}` expands to the jail name. The same attachment can be performed against an already running jail with `zfs jail <jid|name> zroot/jails/data`.

| The jail needs access to /dev/zfs. The default devfs ruleset for jails (`devfsrules_jail`, ruleset 4) already unhides it, so no custom devfs ruleset is required for ZFS delegation.

Inside the jail, mount the delegated datasets and manage them like any other:

    # zfs mount -a
    # zfs create zroot/jails/data/work
    # zfs snapshot zroot/jails/data/work@monday

The jailed root user can create, snapshot, clone, and roll back datasets under the delegated tree, but the `quota` property of the delegated dataset can only be set from the host.

### 17.15.2. Attaching Datasets Automatically

| The `zfs.dataset` parameter requires FreeBSD 15.0. It is not present in the jail(8) of FreeBSD 14.3.

Starting with FreeBSD 15.0, jail(8) can attach datasets on its own, removing the need for the manual `jailed` and `zfs jail` steps. List the datasets to delegate in the `zfs.dataset` parameter:

    allow.mount;
    allow.mount.zfs;
    enforce_statfs = 1;
    zfs.dataset = "zroot/jails/data";

jail(8) attaches each listed dataset when the jail starts, so the `exec.created` hook shown above is no longer needed. The parameter requires `allow.mount.zfs` to be set.

For unmaintained jail managers that do not yet understand `zfs.dataset`, the rc.d/jail startup script also honors the legacy variable `jail_<jailname>_zfs_dataset` in /etc/rc.conf.

### 17.15.3. Accessing Snapshots from Inside a Jail

The `zfs.mount_snapshot` parameter, available on FreeBSD 14.x and 15.x, lets jailed users reach snapshot contents without delegating a whole dataset. When set to `1`, jailed users may read the contents of a filesystem’s snapshots under its .zfs/snapshot directory:

    zfs.mount_snapshot = 1;

This is convenient for restoring individual files from host-taken snapshots. If `allow.mount.zfs` is also set, the snapshots may additionally be mounted inside the jail. The dataset’s `snapdir` property controls whether the .zfs directory is visible.

| A delegated dataset combined with `allow.mount.zfs` lets the jailed root user set mount-related ZFS properties and mount filesystems inside the jail. Grant these permissions deliberately, and only to jails that are trusted to manage their own storage.

## 17.16. Jail Managers and Containers

As previously explained, each type of FreeBSD Jail can be created and configured manually, but FreeBSD also has third-party utilities to make configuration and administration easier.

Below is an incomplete list of the different FreeBSD Jail managers:

Table 3. Jail Managers
Name | License | Package | Documentation
--- | --- | --- | ---
BastilleBSD | BSD-3 | sysutils/bastille | Documentation
pot | BSD-3 | sysutils/pot | Documentation
cbsd | BSD-2 | sysutils/cbsd | Documentation
AppJail | BSD-3 | sysutils/appjail, for devel sysutils/appjail-devel | Documentation
iocage | BSD-2 | sysutils/iocage | Documentation
ezjail | Beer Ware | sysutils/ezjail | Documentation

FreeBSD has published official OCI base-system images since FreeBSD 14.3-RELEASE, and OCI (Docker-style) containers (which are built on the same jail technology and managed with the Podman toolkit) are covered in the Containers chapter, which complements the native jails described here.

Last modified on: July 12, 2026 by Sergio Carlavilla Delgado

Prev
Home
Next