Simpler self-hosting with tmux and bubblewrap

A simpler way to self-host on Linux or FreeBSD, by running user services in tmux windows, secured by using bubblewrap or jail sandboxes, and started up on boot by cron. For motivations and rationale behind this repo, please see the blog post Simpler Linux self-hosting with tmux and bubblewrap

BWS is an alternative to using Docker/podman. Instead of containers (which contain a whole OS), it uses sandboxes (or jails on FreeBSD) that re-uses the host OS. The sandboxes get a read-only mount of the host OS (and hence cannot tamper with it), and all persistent state is written into a home directory that is specific to each sandbox. All the code and data reside in the host user's home directory, and hence it is easy to backup and/or move a BWS installation from one server to another simply by copying the home folder across.

BWS is a few shell scripts that help orchestrate the sandboxes by running them in tmux, allowing you to start, stop, and shell into your services. It also restarts services if they exit, starts them up on boot using a crontab entry, and collates their logs into ~/log. You can run BWS on a server even if you don't have root access (some caveats apply) since everything is contained in your home folder. Need to run 10 separate instances of Nextcloud? Simply create 10 separate users, copy (or symlink) the ~/services folder into their home folders, and start them all up!

What type of services can you run? Things like proxies (e.g. HAProxy), web servers (e.g. Caddy), databases (e.g. MySQL), backends (e.g. Python or PHP), your own custom web scraper, etc. See the BWS examples repo for some examples. You can even run conflicting versions of Python or MySQL (for example), in separate sandboxes. You can tie these services together to run Wordpress or Nextcloud or Email, or anything else you wish. This is the equivalent of using Docker compose to run all these services in separate containers, but much more lightweight, easier to debug, and easier to manage.

BWS is OS-agnostic and should run on any Linux or FreeBSD system. You can infact copy a BWS service from Linux to FreeBSD and it should just work! How does that work in terms of package management? There are two options: one option is to add a statically-linked binary for each platform into the service (e.g. Caddy) and pick the correct binary when starting the service; the other option is to create a nix.packages (Linux) and/or pkg.packages (FreeBSD) file containing the packages that the service needs, and BWS will install these packages into the sandbox before starting them. Nix package manager, which has the most packages, works on any Linux OS and manages packages independently of the host OS, so you can choose any distribution without worrying about it's package manager.

Installation

• Install a fresh Linux OS, e.g. Ubuntu Server, or Raspberry Pi OS (on a Raspberry Pi)
• Install tmux and bubblewrap if they aren't already installed.
• Log in as the user created during install.
• Download this repo to the user's home directory: download zip. You can alternatively use fossil clone https://fossil.tobykurien.com/bws to clone this repo, and you can keep it up to date via fossil update. See below for notes on Fossil.
• Add the line from crontab file to the user's crontab: crontab ./crontab

Usage

A service is just a sub-folder of ~/services containing a start.sh executable script to start the service and wait. There can also be an optional stop.sh script to stop the service gracefully (not guaranteed to be called e.g. on system shutdown). If the start.sh script exits for any reason, it will be restarted after 10 seconds, unless the exit code is 100. The service can also have optional configuration files (e.g. sandbox.args, config.env) described below.

• For help on BWS commands, run ~/services/bws
• To add a service, download the binaries into a subfolder of ~/services
  • e.g. ~/services/caddy/
• Make a script file called start.sh to start it, e.g. ~/services/caddy/start.sh. Don't forget to chmod +x start.sh to make it executable.
• Start the service: ~/services/bws start caddy
• The logs are written to ~/log/caddy.log so you can tail -f ~/log/caddy.log, or ~/services/bws attach caddy to attach to the tmux window. Use "Ctrl+b" followed by "d" to detach the window and go back to the command line.
• To auto-start the service on boot, add a line to ~/services/start, e.g. echo "~/services/bws start caddy" >> ~/services/start. Make sure to add a crontab entry to run this on boot (or every minute), e.g. @reboot cd ~/services/ && ./start
• A good way to explore the sandbox of a running service is to try running the ttyd service and poke around. You can also run ~/services/bws shell caddy to open a shell into the sandbox. This mode makes the service folder writable, which is useful if you need to install or configure the service.
• If you need to install some dependencies, e.g. python modules, you can open a service sandbox shell and install inside the sandbox, e.g.

  $ ~/services/bws shell myservice
  Starting a shell inside sandbox of service myservice
  Enabling write on app directory.
  $ pip3 install --user -r requirements.txt
  

• If you run your service on multiple platforms, you can manage the architecture-specific binaries by putting them into "bin/linux-{platform}/" (e.g. ~/services/gemini/bin/linux-x86_64/molly-brown) and the start.sh script can execute the correct binary for the platform using "uname" like so:

BIN="linux-`uname -m`"
bin/$BIN/molly-brown -c ./molly.conf 

The BWS Examples repo has real-world examples for reference.

Configuring the sandbox

Mounts and isolation: sandbox.args

You can configure the sandbox for each service by adding a text file called sandbox.args containing additional parameters to pass to the bubblewrap/jail sandbox, e.g.

--unshare-net
--bind-try /media/usb $HOME/usb
--ro-bind /etc/pki /etc/pki

The above will remove network access from the sandbox for the service, try to mount the /media/usb folder into the sandbox at ~/usb (writable), and read-only mount /etc/pki into the sandbox so that certain applications can verify SSL certificates. See the bubblewrap man page for more details.

On FreeBSD, the following are the supported options:

• --unshare-net: Creates a vnet jail for network isolation
• --bind [host path] [guest path]: read-write mount directory or file from host to guest
• --bind-try [host path] [guest path]: read-write mount directory or file from host to guest (ignore if host path doesn't exist)
• --ro-bind [host path] [guest path]: read-only mount directory or file from host to guest
• --ro-bind-try [host path] [guest path]: read-only mount directory or file from host to guest (ignore if host path doesn't exist)
• --tmpfs [guest path]: mount a read-write tmpfs filesystem into the guest (contents lost if service restarted)
• --devfs: mount the /dev filesystem into sandbox as read-write

Note that the sandbox.args file will be redacted inside the sandbox to avoid leaking information about the host system.

Environment variables: config.env

You can inject environment variables into the sandbox by creating a config.env that contains environment variables to export to the sandbox, e.g.

DB_NAME=wordpress
DB_USERNAME=wordpress
DB_PASSWORD=mypassword

Note that config.env is a shell script and will be executed when the sandbox starts, so be careful with it's contents. The contents of config.env will be redacted inside the sandbox to improve security.

Linux packages: nix.packages

Once you have installed the Nix package manager, you can configure the packages to be installed into the sandbox using a nix.packages (or packages.nix) file, which should contain the names of packages available in the nixpkgs channel, e.g. to install nixpkgs.haproxy and nixpkgs.thttpd, the file will contain:

haproxy
thttpd

You can find the package names by searching on https://search.nixos.org/packages.

FreeBSD packages: pkg.packages

On FreeBSD, simply add a pkg.packages file containing packages (one per line) to install into the sandbox, e.g.

haproxy
lighttpd

If the package already exists on the host system, it is not installed into the sandbox. This can save time and disk space if, for example, there are many services using the same large package.

Limitations

• On FreeBSD, passwordless root access is needed to mount the various filesystems, run chmod if needed, and to start the jails. This needs to be configured in doas.conf or sudoers file. Sudo and doas is disabled inside the sandbox.
• There is no network isolation between services (although it is possible to add it), it is preferred to use Unix domain sockets instead of TCP/UDP listening sockets. Typically services will create a socket in /run which should be mounted to ~/run on the host.
• There's no built-in log rotation or trimming, but it is easy to add one.
• No built-in mechanism to restrict the RAM/CPU/disk per service easily (on FreeBSD you could use rctl and zfs quotas)

Notes about Fossil

Fossil is a lot like git so it's familiar to use if you know git - see docs here. Why not just use git? I switched from using GitHub for my projects to self-hosting my repos after GitHub was bought by Microsoft. Fossil is much easier to self-host: it's a single binary file, stores it's repository into a single sqlite database, and provides similar functionality of the GitHub service (like issue tracking, wiki, and additionally a forum, etc.). All the data (including issues, etc.) are stored within the repo sqlite database file, making backups really simple. Type fossil ui in the checked out repo and you will get the full web interface, all served from one database file by a single binary executable! See Fossil vs Git for more.