Chronos Redux: On FreeBSD
Verbose engineering notes in creating a local, GPS synchronized stratum 1 time server with FreeBSD, BeagleBone Black, gpsd, and chrony.
Previously, I created a local time server using Linux and Buildroot, but after a while it felt little too mainstream for me. What’s even more niche? Let’s try FreeBSD.
Why make a local time server again in a less-supported OS? I’m probably a masochist.
Initial Setup
FreeBSD 13.0 on the BeagleBone Black runs out of the box. The ready-made armv7 GENERICSD release ISO image loads immediately on the board, without any customization. Kudos to everyone involved on putting that together.
The hardware will be identical to last time. So you don’t have to click on more links, details are below:
Parts
- BeagleBone Black: https://www.adafruit.com/product/1996
- Proto cape kit: https://www.adafruit.com/product/572
- GPS receiver: https://www.adafruit.com/product/746
- GPS antenna: https://www.adafruit.com/product/960
- Simple BeagleBone enclosure: https://www.adafruit.com/product/1555
Connectivity
- P9.26: UART Rx (connect to the GPS’s Tx)
- P9.24: UART Tx (connect to the GPS’s Rx)
- P9.42: PPS input
- P8.11: W1 bus master
Previously, we used Buildroot to generate a nearly-complete system image on an off-target build machine. For this round, we will manually setup FreeBSD as it is running on the device.
To get started, use dd to get the image on an SD card, pop it in, and boot. My total disk utilization is less than 5 GiB, so you won’t need a large SD card.
You can SSH in as freebsd:freebsd to get started, or use serial. Use su to switch to root, with the password of “root”. Good instructions are here for serial connection options, and general setup instructions here.
Also, change the default passwords to something secure.
Note: I had an older BeagleBone Black, with an older eMMC image. FreeBSD did not boot automatically from the SD card, but Linux did. FreeBSD also ran easily on a more recent board, so something was awry. Instead of battling with esoteric config values, I updated the board eMMC with one of the official BeagleBone “flasher” images. After update, the FreeBSD image boots right away from an SD card.
Packages
For an initial setup, run pkg and it will be bootstrapped. We’ll be installing things from the ports tree, so set it up with portsnap fetch extract (which will take a bit).
GPSD
For FreeBSD 13, gpsd is not available via pkg. Instead, we must compile it from source. I opted for the ports tree, which is simple (but slow) to setup and compile. Version 3.20 is available:
# cd /usr/ports/astro/gpsd
# make && make installDuring the gpsd build I enabled PPS support (feel free to skip, it won’t work) and unselected X11 and USB options, since I would never need them. You can be more diligent and remove other unnecessary support items.
You will get prompted a number of times to customize build configs, so be aware. Or, you can get the config out of the way with make config-recursive. I disabled a few options by default, running the following prior to vanilla make to make my life easier:
make OPTIONS_UNSET="DOCS EXAMPLES X11 USB" config-recursiveChrony
Chrony 4.0 is readily available via pkg install chrony. The config lives at /usr/local/etc/chrony.conf. Simple to install.
However, there are some key fixes in 4.1 that we’ll want (discussed later). Luckily, chrony 4.1 is in the ports tree to make our lives super easy.
# cd /usr/ports/net/chrony
# make && make installOthers
Customize to your liking. Some things I made sure to have:
vim tmux the_silver_searcher rrdtool python doas zsh wgetConfiguration
Device Tree
We need to tell FreeBSD about the hardware we have attached. Luckily, a lot of the work was previously done last time. Unluckily, there’s still a substantial amount of work and swearing to go from the old system to the new.
This time around we will use device tree overlays. This presents some interesting challenges: Figuring out the parts of the device tree to modify, and how to get FreeBSD to ingest them.
To start, we can dump the currently loaded device tree, in human-readable format, and check out what’s going on as a sanity check:
# sysctl -b hw.fdt.dtb | dtc -I dtb -O dts
/dts-v1/;
...
compatible = "ti,am335x-bone-black", "ti,am335x-bone", "ti,am33xx";
interrupt-parent = <0xce>;
#address-cells = <0x1>;
#size-cells = <0x1>;
model = "TI AM335x BeagleBone Black";
freebsd,dts-version = "5.9";
...Now that we have confirmed that it’s loading a sane default, we can check out the FreeBSD 13 source and examine the am335x related device tree files (which are a bit more readable, but the dump is more truthy). Luckily, it seems to be sourced directly from Linux, so nothing should be crazily new.
For overlays, I based my work on bb.org-overlays. I made my own my own fork, with a custom branch (chronos). The final overlays are below, after a lot of experimentation, failures, and crying.
bb-chronos-uart1.dts
/dts-v1/;
/plugin/;
#include <dt-bindings/board/am335x-bbw-bbb-base.h>
#include <dt-bindings/pinctrl/am33xx.h>
#include <dt-bindings/gpio/gpio.h>
/ {
fragment@0 {
target-path = "/";
__overlay__ {
chosen {
overlays {
bb-chronos-uart1 = __TIMESTAMP__;
};
};
};
};
fragment@1 {
target = <&am33xx_pinmux>;
__overlay__ {
uart1_pins: pinmux_uart1_pins {
pinctrl-single,pins = <
BONE_P9_24 (PIN_OUTPUT | MUX_MODE0)
BONE_P9_26 (PIN_INPUT | MUX_MODE0)
>;
};
};
};
// i2c2 uses the same pins -- disable
fragment@2 {
target = <&i2c2>;
__overlay__ {
status = "disabled";
};
};
fragment@3 {
target = <&uart1>;
__overlay__ {
status = "okay";
pinctrl-names = "default";
pinctrl-0 = <&uart1_pins>;
};
};
};bb-chronos-pps.dts
/dts-v1/;
/plugin/;
#include <dt-bindings/board/am335x-bbw-bbb-base.h>
#include <dt-bindings/pinctrl/am33xx.h>
#include <dt-bindings/gpio/gpio.h>/ {
fragment@0 {
target-path = "/";
__overlay__ {
chosen {
overlays {
bb-chronos-pps = __TIMESTAMP__;
};
};
};
};
fragment@1 {
target = <&am33xx_pinmux>;
__overlay__ {
pps_pins: pinmux_pps_pins {
pinctrl-single,pins = <
BONE_P9_42 (PIN_INPUT | MUX_MODE7)
>;
};
};
};
fragment@2 {
target-path = "/ocp/";
__overlay__ {
pps {
status = "okay";
compatible = "pps-gpio";
pinctrl-names = "default";
pinctrl-0 = <&pps_pins>;
gpios = <&gpio0 7 GPIO_ACTIVE_HIGH>;
};
};
};
};
bb-chronos-w1.dts (note: external pull-up required in FreeBSD)
/dts-v1/;
/plugin/;#include <dt-bindings/board/am335x-bbw-bbb-base.h>
#include <dt-bindings/pinctrl/am33xx.h>
#include <dt-bindings/gpio/gpio.h>/ {
fragment@0 {
target-path = "/";
__overlay__ {
chosen {
overlays {
bb-chronos-w1 = __TIMESTAMP__;
};
};
};
}; fragment@1 {
target = <&am33xx_pinmux>;
__overlay__ {
w1_pins: pinmux_w1_pins {
pinctrl-single,pins = <
BONE_P8_11 (PIN_INPUT_PULLUP | MUX_MODE7)
>;
};
};
}; fragment@2 {
target-path = "/ocp/";
__overlay__ {
onewire {
status = "okay";
compatible = "w1-gpio";
pinctrl-names = "default";
pinctrl-0 = <&w1_pins>;
gpios = <&gpio1 13 GPIO_PULL_UP>;
};
};
};
};
Some key points:
- The
/chosen/overlaysaddition is nice for debugging and knowing what’s loaded. Kudos to whoever thought that up. - Each device is in its own overlay. This lets us easily mix and match (and debug/isolate issues), at the cost of a slightly slower to boot.
- The new devices should be placed in the
/ocp/path. For some reason, thegpioppskernel module would not auto-load if placed at the root (/). And explicitly loading the modules inloader.confwithgpiopps_load="YES"causes the PPS device to attach too early, before thegpiocdevices are ready, and fail. More info here. - I disabled
i2c2which uses the same pins an UART1 CTS/RTS, but that might not have been necessary. - The macros are pretty nice and easy to read, compared to what I did last time.
Build and Install the Overlays
From the root of the forked repo, run make and wait for the dtbo files to be generated:
bb.org-overlays$ make
...
DTC src/arm/bb-chronos-uart1.dtbo
DTC src/arm/bb-chronos-w1.dtbo
DTC src/arm/bb-chronos-pps.dtbo
...Place these files on the system boot partition at /boot/dtb/overlays/ then modify loader.conf with the following line to load them:
fdt_overlays="bb-chronos-uart1,bb-chronos-pps,bb-chronos-w1"This applies each overlay one at a time during boot.
Note that errors and issues aren’t easily seen through the console (they fly by too fast, so perhaps use a logging serial console). If you need to, use ofwdump to see which overlays were successfully added to the device tree:
# ofwdump -p /chosen/overlays/
Node 0x170: overlays
bb-chronos-w1:
53 61 74 20 4a 75 6e 20 31 32 20 31 39 3a 31 33 3a 35 30 20
32 30 32 31 00
'Sat Jun 12 19:13:50 2021'
bb-chronos-pps:
53 61 74 20 4a 75 6e 20 31 32 20 31 39 3a 31 33 3a 32 34 20
32 30 32 31 00
'Sat Jun 12 19:13:24 2021'
bb-chronos-uart1:
53 61 74 20 4a 75 6e 20 31 32 20 31 39 3a 31 33 3a 33 35 20
32 30 32 31 00
'Sat Jun 12 19:13:35 2021'Checking dmesg we can verify things attached properly:
...
uart1: <TI UART (16550 compatible)> mem 0-0xfff irq 12 on ti_sysc9
...
owc0: <GPIO one-wire bus> on simplebus0
ow0: <1 Wire Bus> on owc0
ow0: Reset discovered bus wired wrong.
gpiopps0: <GPIO PPS> on simplebus0
gpiopps0: PPS input on gpio0 pin 7Success! …mostly. The 1-wire bus isn’t configured properly. There’s no hardware difference from the previous (working) system, and the pin config was even updated to specify the pullup requirement in more places. Hmm. Looking at the status in gpioctl reveals:
# gpioctl -f /dev/gpioc1 -l
...
pin 13: 0 gpio_13<IN>The pull-up isn’t enabled for some reason (no PU indicated), which is critical as I did not include a physical pullup. Looking at the kernel driver, I don’t see anything related to pullup configuration, unlike in Linux. In fact, it seems the FreeBSD driver is explicitly NOT using pull-ups when set for input:
#define INPIN(sc) gpio_pin_setflags((sc)->sc_pin, GPIO_PIN_INPUT)So, instead of modifying the kernel, I added a 10k pullup resistor, powered up, and was greeted with:
ow0: romid 28:f2:1c:a1:08:00:00:5c: no driverHmm. Seems we also need to manually load the ow_temp kernel module with kldload ow_temp. Afterward, dmesg greets us with:
ow_temp0: <Advanced One Wire Temperature> romid 28:f2:1c:a1:08:00:00:5c on ow0Success! Adding the line ow_temp_load="YES" to /boot/loader.conf will make this work on startup. I also changed the dts to remove the pullup configuration as it is no longer needed (see the repo for the final config).
Configure gpsd and chrony
So, I attempted to use the same config from last time for gpsd and chrony, where gpsd is connected to both the UART and the PPS sources. But… that didn’t work.
To start, there’s a requirement imposed by gpsd that the PPS device path needs to start with /dev/pps to be correctly identified, but we get /dev/gpiopps from the FreeBSD GPIO PPS driver. This is simple enough to fix with a symlink. We can set this up at boot by creating a custom devd config at /usr/local/etc/devd/pps.conf, with the following:
# Symlink /dev/pps0 to /dev/gpiopps0
attach 100 {
device-name "gpiopps0";
action "ln -s /dev/gpiopps0 /dev/pps0";
};However, after some tooling around, PPS in gpsd still wasn’t working. It would fail to open the PPS device. (Note: Fixed in 3.23.)
So, how about giving the PPS to chrony?
Tight PPS ref clock limits in chrony caused headaches in my previous setup. If the ref clock offset and dispersion was off by too much, it would just throw its hands up in the air and drop samples. However, looking at the change log, it seems that there was an improvement to the PPS lock behavior in chrony 4.1:
Increase PPS lock limit to 40% of pulse interval
Yyaaaass. Now that the PPS can be more easily synchronized to less stable time sources (like a jittery NMEA stream over slow serial), we can have chrony take on more of the work directly.
Side note: There’s an important bug fix in chrony 4.1 as well:
Fix responding to IPv4 command requests on FreeBSD
Running chronyc as root works fine as it uses the local UNIX socket interface. If you’re a non-root user, it’ll appear that chronyc commands hang and eventually error out as it attempts to connect to localhost using IPv4. A simple workaround is to connect via IPv6 with chronyc -6 <command>.
So, it’s best to install chrony 4.1 from ports instead of using 4.0 in pkg.
Add the following to /etc/rc.conf.local to enable services and configure gpsd:
hostname="chronos"
chronyd_enable="YES"
gpsd_enable="YES"
gpsd_devices="-n /dev/ttyu1"My new, working /usr/local/etc/chrony.conf file is below:
# Store (and load on boot) our clock drift info
driftfile /var/db/chrony/drift# Logging
logdir /var/log/chrony
log tracking measurements statistics# Step the system clock instead of slewing it if the adjustment is
# larger than 0.1 second, but only in the first few clock updates.
makestep 0.1 5# Make the service real time (SCHED_FIFO priority 50)
lock_all
sched_priority 50# PPS + gpsd shared memory for the ref, compensate for serial delay
refclock SHM 0 offset 0.5 refid SHM noselect
refclock PPS /dev/gpiopps0 refid PPS lock SHM# Just for (temporary) sanity checks
#server 0.us.pool.ntp.org noselect
#server 1.us.pool.ntp.org noselect
#server 2.us.pool.ntp.org noselect
#server 3.us.pool.ntp.org noselect# Share time with the world
allow all
Note that chrony’s support for real-time scheduling and memory locking in FreeBSD was added a few years ago by yours truly.
Others
I disabled almost all regularly scheduled tasks in /etc/crontab. The only task I left enabled was newsyslog. This will reduce hiccups as the system runs — gotta get those sweet, sweet microseconds of accuracy. I have no intention of setting up mail delivery to receive periodic results, and the system is intended to stay up indefinitely, so the maintenance was superfluous.
Performance
The numbers below look fantastic from a pure offset and standard deviation perspective, though it’s just a single snapshot in time. The “SHM” reference is the shared memory interface to gpsd, while “PPS” is what the system locks to. I added an external time server temporarily just for sanity checking.
Name/IP Address NP NR Span Frequency Freq Skew Offset Std Dev
====================================================================
SHM 15 8 234 +10.317 146.474 +41ms 11ms
PPS 64 34 1070 +0.000 0.002 +0ns 1182ns
clock.sjc.he.net 26 13 333m +0.024 0.105 -2780us 664usNotice that the shared memory interface alone provides awful performance compared to a normal internet time server. PPS synchronization is crucial.
For an indication of stability, I log the system frequency offset in parts per million, and throw it into rrdtool regularly. This number can change due to temperature fluctuations in the external oscillator (which actually changes the system clock frequency), but also due to errors in calculation caused by an unstable clock source.
Comparisons below show a generally smoother performance for the new setup.
A number of factors can contribute to system clock instability. One is actual instability due to temperature changes, another is perceived instability due to the system not responding to and time-stamping a GPIO edge transition in a timely manner (pun not intended). Note that system load can also increase temperature. A double whammy.
On Linux, I’ve run into issues with SD card interfaces where interrupts get disabled for some non-trivial duration somewhere in the IO path. This causes the GPIO edge-triggered interrupt to be delayed, sometimes quite significantly (on the order of a millisecond).
What about the larger blips? Below is a pathological case, with blips that are periodic, about one hour apart:
The graph above was captured when I had gpsd running at log level 8 (RAW), which was basically logging every single NMEA sentence. This caused a TON of log messages, which translates to a lot of log rollover and disk activity.
Here’s from another time when I was compiling something out of ports.
Reducing logging levels and disabling periodic tasks in FreeBSD brought this back to something more reasonable and less frequent, but that’s just a mitigation; the issue still remains. Something to investigate more another day.
Conclusion
With only a few bumps down the road, it was quite simple to setup a stratum 1 time server in FreeBSD. Only minor mental anguish along the way.
Comparison from my last attempt:
- chrony has much improved PPS handling, which reduces the reliance on gpsd.
- FreeBSD 13 appears to be more stable than before in the event of power loss (my Linux build had issues starting up cleanly).
- Device tree overlays can be quite elegant, if you know what you’re doing.
- Buildroot is nice to build off-target, but a fully running on-target system with dynamic package management and compilation options makes things nice for experimentation.
- FreeBSD isn’t dead.
