OpenBSD With (Cloud)Flare

Engineering notes on setting up a cloudflared tunnel service on OpenBSD

Cloudflare’s reverse proxy tunnel setups are well documented. There are plenty of getting started guides out there, which anyone reading this should start with. However, setting up a tunnel service on a not-well-suported OS like OpenBSD takes a bit of extra time and effort.

Here we’ll run through compiling and setting up a cloudflared tunnel service on OpenBSD.

Prerequisites

This was performed successfully on OpenBSD 7.4. Before starting, we’ll need a few extra packages installed for this to work:

$ doas pkg_add git go gmake

Compiling cloudflared

Check out a local copy of cloudflared and compile:

$ git clone -q https://github.com/cloudflare/cloudflared.git
$ cd cloudflared
$ gmake install-go cloudflared cleanup-go
rm -rf /tmp/go
./.teamcity/install-cloudflare-go.sh
Building Go cmd/dist using /usr/local/go. (go1.21.1 openbsd/amd64)
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for openbsd/amd64.
---
Installed Go for openbsd/amd64 in /tmp/go
Installed commands in /tmp/go/bin
GOOS=openbsd GOARCH=amd64  go build -mod=vendor  -ldflags='-X "main.Version=2024.2.0-1-g56aeb6be" -X "main.BuildTime=2024-02-15-2334 UTC" ' github.com/cloudflare/cloudflared/cmd/cloudflared
rm -rf /tmp/go

The above commands may seem to hang for a long time, but it’s working. Patience is a virtue.

Note that cloudflared 2024.1 requires its own go toolchain for post-quantum cipher support. Otherwise, we could have skipped the custom go installation.

Once that is done, copy the executable to a suitable location:

$ doas cp cloudflared /usr/local/sbin/

Creating a Service

The cloudflared service install command does not work on OpenBSD, so we have to set everything up manually.

Normally on OpenBSD, daemons start as root, access any privileged files or setup privileged sockets as needed, make pledges, and drop privileges. The cloudflared daemon does not do things the OpenBSD way — it seems like its really intended to run in a container.

Since I still value privilege separation, things are going to be setup slightly differently than one might expect…

Daemon User

Since the cloudflared daemon does not require special privileges while running as a tunnel, we will setup a daemon service with reduced and separate privileges. This means setting up a dedicated user and group first:

$ doas group add _cflared
$ doas user add -g _cflared -c Cloudflared -s /sbin/nologin -d /var/empty _cflared

Initial Auth

We need to get credentials installed locally. Run cloudflared login as the root user and log in using the provided link in a browser:

$ doas cloudflared login
Please open the following URL and log in with your Cloudflare account:

https://dash.cloudflare.com/argotunnel?aud=&callback=<redacted>

Leave cloudflared running to download the cert automatically.
2024-02-15T23:50:56Z INF Waiting for login...
You have successfully logged in.
If you wish to copy your credentials to a server, they have been saved to:
/root/.cloudflared/cert.pem

Note: You may need to have already setup domains/websites within Cloudflare for this to work properly.

If you gave the _cflared user its own home directory, you could have used that instead of root.

We’re going to put the cert and configuration files in a new /etc/cloudflared/ directory owned by the new _cflared user. This directory is automatically checked by the daemon for certificates and configs, which will make things easier:

doas mkdir /etc/cloudflared
doas mv /root/.cloudflared/cert.pem /etc/cloudflared
doas chown -R _cflared:_cflared /etc/cloudflared

And create a tunnel as the _cflared user:

$ doas -u _cflared cloudflared tunnel create tunnel-mc-tunnel-face
Tunnel credentials written to /etc/cloudflared/a3eb939d-xxxx-yyyy-zzzz-b0b7d0c0c9ba.json. cloudflared chose this file based on where your origin certificate was found. Keep this file secret. To revoke these credentials, delete the tunnel.

Created tunnel tunnel-mc-tunnel-face with id a3eb939d-xxxx-yyyy-zzzz-b0b7d0c0c9ba

Note: As a reader pointed out, you might need to modify /etc/doas.conf to remove as <user> restrictions, or append the new _cflared user.

The newly created json file will have the right ownership and privileges this way.

$ ls -l /etc/cloudflared/
-rw-------   1 _cflared  _cflared  1930 Feb 13 21:39 cert.pem
-r--------   1 _cflared  _cflared   161 Feb 15 19:04 a3eb939d-xxxx-yyyy-zzzz-b0b7d0c0c9ba.json

Tunnel Config

There are two options:

1. Configure the tunnel in the Cloudflare dashboard, or
2. Configure the tunnel locally in a config.yaml file.

The juicy parts of the configuration can only exist in one location — no mixing and matching. In any case, one can create at least a token local config to specify the default tunnel to use.

Create /etc/cloudflared/config.yaml with:

---
tunnel: a3eb939d-xxxx-yyyy-zzzz-b0b7d0c0c9ba.json
credentials-file: /etc/cloudflared/a3eb939d-xxxx-yyyy-zzzz-b0b7d0c0c9ba.json

Add additional configuration in the same file, or set it up in the web dashboard. The rest of the instructions here are the same no matter which you choose.

Perform a quick test before continuing:

$ doas -u _cflared cloudflared tunnel run
2024-02-16T00:08:05Z INF Starting tunnel tunnelID=a3eb939d-xxxx-yyyy-zzzz-b0b7d0c0c9ba
2024-02-16T00:08:05Z INF Version 2024.2.0-1-g56aeb6be
2024-02-16T00:08:05Z INF GOOS: openbsd, GOVersion: go1.21.5-devel-cf, GoArch: amd64
2024-02-16T00:08:05Z INF Settings: map[cred-file:/etc/cloudflared/a3eb939d-xxxx-yyyy-zzzz-b0b7d0c0c9ba.json]
...

CTRL+C out and let’s setup the persistent service next.

Service Script

Next, create an rc script for the service at /etc/rc.d/cloudflared:

#!/bin/ksh

daemon="/usr/local/sbin/cloudflared"
daemon_flags="tunnel --no-autoupdate --log-directory /var/log/cloudflared run"
daemon_user="_cflared"

. /etc/rc.d/rc.subr

rc_bg=YES

rc_cmd $1

Note that rc_bg=YES is required since cloudflared has no option to daemonize on its own.

Enable the service with:

$ doas rcctl enable cloudflared

Logfile Setup

Using a log file directory lets cloudflared perform its own logfile rotation. We’ll create a directory and setup privileges appropriately:

$ doas mkdir /var/log/cloudflared
$ doas chmod 750 /var/log/cloudflared
$ doas chown _cflared:wheel /var/log/cloudflared

Trying to rotate log files for it with newsyslog would not work well — cloudflared does not support signals to refresh open handles, and we’d be forced to restart the daemon instead.

And Away We Go ☁️

With everything in place, the service should start:

$ doas rcctl start cloudflared
cloudflared(ok)

Check the log for success:

$ tail /var/log/cloudflared/cloudflared.log
...
{"level":"info","event":0,"connection":"<redacted>","connIndex":0,"location":"ord08","ip":"198.41.200.33","protocol":"quic","time":"2024-02-16T00:29:45Z","message":"Registered tunnel connection"}
{"level":"info","event":0,"connection":"<redacted>","connIndex":1,"location":"ord02","ip":"198.41.192.107","protocol":"quic","time":"2024-02-16T00:29:45Z","message":"Registered tunnel connection"}
{"level":"info","event":0,"connection":"<redacted>","connIndex":2,"location":"ord02","ip":"198.41.192.27","protocol":"quic","time":"2024-02-16T00:29:46Z","message":"Registered tunnel connection"}
{"level":"info","event":0,"connection":"<redacted>","connIndex":3,"location":"ord11","ip":"198.41.200.53","protocol":"quic","time":"2024-02-16T00:29:47Z","message":"Registered tunnel connection"}

And that the service is still running as expected:

$ ps -wwU _cflared
  PID TT  STAT        TIME COMMAND
37286 p0  S        0:00.35 /usr/local/sbin/cloudflared tunnel --no-autoupdate --log-directory /var/log/cloudflared run

Happy tunneling.

Addendum: Firewall Rules

If we want to restrict what internal services the cloudflared tunnel can actually reach, or have a fairly restrictive firewall in place already, we can setup some pf rules specifically for the _cflared user we created.

In general, the cloudflared tunnel can be taken over remotely. It’s by design and can be done through the web dashboard. Remotely, a user can reconfigure the tunnel to send traffic anywhere in the network.

Some users or use cases might desire something stricter, and not having a way to disable this potential takeover vector in software is quite odd and seems like a security oversight.

With that in mind, we’ll make a restrictive set of firewall rules to reduce the impact from any potential remote abuse. For the external side of things, Cloudflare has documented minimum requirements for network access.

All combined, my pf.conf file looks like this:

# General rules and setup (change to suit your needs)
main_if="rge0"
set block-policy return
block in
pass in on lo0
pass out

# Limit the cloudflared tunnel by default
block in proto {tcp udp} user _cflared
block out proto {tcp udp} user _cflared

# Allow outbound dns
pass out on $main_if proto udp from self to port 53 user _cflared

# Allow quic, http2 to the tunnel
table <cflared> const { "region1.v2.argotunnel.com" "region2.v2.argotunnel.com" }
pass out on $main_if proto { tcp udp } from self to <cflared> port 7844 user _cflared

# Allow access to tunneled services
pass out on lo0 proto tcp to lo0 port 443 user _cflared

Note that ICMP cannot be filtered by user, but ICMP tunnels are not supported by cloudflared on OpenBSD anyway.