Troubleshooting

Where to look

cerberd status                     # hub, nodes, routers, links, last reconcile
cerberd status --json              # everything, including the last reconcile result
cerberd events --follow            # live events (see the events reference)
journalctl -u cerberd@wg0 -f       # daemon log (cerberd run --debug for more)
wg show wg0                        # what is actually on the interface

The first line of cerberd status summarises the hub:

hub wg0  overlay 10.99.0.0/16  on the interface 12  known 14 (blocked 2)  foreign 0  cerberd v1.4.0

on the interface counts peers on wg0, known peers in the database. Without foreign peers, on the interface = known − blocked.

Common problems

no cerberd instance running (no sockets in /run/cerberd) The daemon isn’t running, or you aren’t allowed to read /run/cerberd (root or the cerberd group). systemctl status cerberd@wg0.

several instances running (wg0, wg1): pick one with -i <interface> Pass -i wg0, or the interface as the argument of status / events.

The unit exits with status 78 and doesn’t restart A setup error: missing or invalid /etc/cerberd/wg0.yaml, no database (cerberd init or cerberd import), an unreadable token file, or a hub key mismatch (the live wg0 has a different private key than the database; cerberd never replaces the hub’s identity on its own). The journal has the exact message. wg0 keeps running meanwhile.

read config ...: permission denied The cerberd user needs read access: chgrp cerberd /etc/cerberd/wg0.yaml; chmod 640 /etc/cerberd/wg0.yaml.

hub.max_removals guard: reconcile held back A reconcile would have cut more than hub.max_removals peers or routes off the hub at once, so it changed nothing. Usually a wrong or old database. Check with cerberd peer ls against wg show wg0. If the change is intended, raise hub.max_removals temporarily and restart. A hub-guard event carries the details.

stored, but not applied to the hub yet: ... The database change succeeded but applying it to wg0 failed. The reconciler retries every timers.hub_reconcile; cerberd status shows the reconcile error until it succeeds.

A route is missing (hub-route-conflict event) Another device already has a route to that prefix (a LAN the hub is on, a blackhole). cerberd never takes such a route over; remove the other route or change the peer’s LAN.

Router unreachable / KEY MISMATCH in status unreachable: check the REST service, the user and password file, and that the router allows REST from the hub’s overlay address. KEY MISMATCH: the router’s WireGuard interface (wg_interface) has a different public key than the peer in the database, and no other interface has it: the key was changed on the router (cerberd peer rotate <name> --public-key K). A wrong interface name fixes itself: cerberd stores the interface that has the peer’s key (router-interface event). Offline nodes show not checked.

Link ends failed No direct handshake within timers.punch_timeout: a symmetric NAT on both sides, or an opener_ttl too low or too high for the site. Try routeros.defaults.opener_ttl (or the peer’s binding) 1 when the MikroTik holds the public IP, 2 behind an ISP router, more behind CGNAT.

Link ends no-gain The direct path worked but wasn’t verify.min_gain faster than the hub. cerberd link up a b --force keeps it anyway.

cerberd for wg0 is running and would bring it back (from cerberd down) Stop the daemon first: systemctl stop cerberd@wg0 && cerberd down wg0.