Seven ideas carry the whole design. Each one is checkable against the same question: is this still true if any single machine disappears?
Most monitoring is a star: every host reports to one collector, and a dashboard sits on the collector. The collector is the one machine that must never fail, and the day it does, you are blind. Photinus has no center. Every host runs a lantern, every lantern watches its peers, and any one of them can answer for the whole swarm.
Every couple of seconds each lantern runs its local checks, bundles what it currently believes into a flash, and gossips it to a few random peers. Those peers pass it on. Nobody talks to everyone; news still crosses the swarm in seconds, and the cost per lantern stays flat no matter how big the swarm grows.
Everything above and below hangs on one path: how a failing check becomes exactly one page. This is that path, the same on every lantern, with no step that needs a central machine.
One lantern failing to reach a host is a fact about that lantern as much as about the host. So a single opinion never pages. Every lantern votes with its own observation, and only when a majority of the swarm as last known agrees does the state become an outage.
A warning is never an outage: a full disk on a living host can only WARN, because the host is up and its own lantern is saying so. DOWN is reserved for things the swarm confirmed unreachable.
For a host's own facts, disk, cpu, memory, swap, uptime, network rate, no vote is taken. The host's own lantern is the sole authority on its own machine: its word decides, and hearsay from others never mixes in. Quorum is for the things no single machine can know alone, like reachability; authority is for the things only one machine can know at all.
When the swarm agrees something broke, fourteen lanterns must not send fourteen emails. There is no leader to do the sending, on purpose. Instead every lantern runs the same arithmetic: hash the alert, and the live lantern whose id sits closest to that hash is the sender. No election protocol, no coordination round. If the chosen lantern is itself the dead one, the same arithmetic picks the next, and the survivor pages.
A flapping subject, bouncing up and down, is damped: the swarm sends one page that says it is flapping, holds further pages, and closes with one page when it settles.
Some failures are things that did not happen. A backup that silently stopped running
has no port to dial. So the job reports in instead: its cron line ends by pinging
/pulse/backup-db on any lantern it can reach. The receipt gossips
like everything else. When a declared pulse stays silent past its window, every lantern
votes down, and quorum turns the silence into a page.
A partition is the nastiest case: both halves believe the other half died. Photinus counts quorum against the swarm as last known, not as currently reachable. The minority side can never reach a majority of that remembered number, so it goes quiet instead of screaming; the majority side still holds quorum and keeps working.
| check | what it tests | kind |
|---|---|---|
| tcp | dial a host:port | remote |
| http | fetch a url; 2xx or 3xx is up | remote |
| cert | TLS health; expiring soon warns, broken is down | remote |
| pulse | a heartbeat: a job pings any lantern, silence pages | remote |
| disk | filesystem usage against a threshold | local |
| cpu | processor utilization against a threshold | local |
| memory | RAM utilization against a threshold | local |
| swap | swap use, the early warning before OOM | local |
| uptime | flags a reboot since the last flash | local |
| net | traffic rate on the main interface, for spotting anomalies | local |
| skew | clock drift measured from peers' flash timestamps | relational |
| lantern | every member's liveness, the mesh watching itself | relational |
Local checks fall under the authority rule; remote checks under quorum; the relational ones are the swarm measuring itself. The standard local set runs by default: one binary and a seed address is a fully monitored host.