How a failure reaches a human
Public-safety software must never fail quietly. How does a problem reach a person before it reaches the field?
Three layers, each watching the one inside it: the app records every problem where a person can see it, monitoring outside the app watches the things that matter most and decides when something is bad enough to wake somebody, and a service fully independent of ours makes sure a person hears about it, because a watcher that goes down with the system cannot report the outage. Eight of the pieces below exist today; ten are planned, and are drawn as planned rather than as though they already run.
What the app records
- No problem is ignored Every error is either handled, reported, or recorded where somebody will see it. Automated checks refuse code that would quietly swallow one. Feeds: System Status, and the in-app banner.
- Proof that routine work happened Every scheduled task records whether it ran and did its job, ran with nothing to do, or could not run, so a task that silently stopped is noticed. Opens for signed-in staff Feeds: System Status, and the in-app banner.
- Every outside service tracked Each outside service we rely on keeps a record of what went wrong and how often, without copying any personal information into it. Opens for signed-in staff Feeds: System Status, and the in-app banner, One alert per condition, until it clears.
- Records that can be traced Everything one action caused can be followed from start to finish, with passwords and keys removed before anything is written down. Feeds: Watching for trends.
- The same rule in the browser The same “no problem is ignored” rule, applied to what runs in your browser as well as on our servers. Planned Feeds: System Status, and the in-app banner.
- A list of what must keep happening Every routine task and connection declares what proves it is working, so something that stops can be detected rather than stumbled on. Planned Feeds: System Status, and the in-app banner.
- What matters most, measured end to end A crew’s location arriving, staff seeing current status, a medical emergency response completing every step, signing in, logging an ICS-214 entry, and producing a document. Each measured by whether it worked and how fast. Planned Feeds: Watching for trends.
Where it shows in the app
- System Status, and the in-app banner The one place administrators look: accounts and settings, scheduled tasks, outside services, email delivery, backups, cost and speed. Anything that cannot be checked says so, rather than looking fine. Opens for signed-in staff Feeds: An email to the administrators.
- One alert per condition, until it clears A brief hiccup is shown but never emailed. A lasting problem reaches an administrator promptly, once, and again only if it comes back after being fixed. Opens for signed-in staff Feeds: An email to the administrators, A push to an admin’s phone.
Watching from outside the app
- Watching for trends Alerts are based on sustained trends rather than single errors, so a brief blip does not wake anybody, and they become more sensitive while an incident is active. Planned Feeds: The person on call.
- Uptime checks Is the service answering at all, asked from outside the service. Planned Feeds: The person on call.
- A heartbeat A signal sent on a timer whose only job is to keep arriving. Without it, monitoring that has stopped watching looks just like a system with nothing wrong. Planned Feeds: A dead-man’s switch.
Fully independent of our systems
- Synthetic checks from outside An independent service regularly tries the things that matter most, the way a user would, and reports what it found. Planned Feeds: The person on call, A status page the suite does not host.
- A dead-man’s switch The receiver that pages when the heartbeat stops. It is the only thing that can report the monitoring itself being down. Planned Feeds: The person on call.
How it reaches a person
- A status page the suite does not host So that “is it down?” has an answer that does not depend on it being up. Planned
- An email to the administrators What is wrong, what it affects in operational words, and what to check first — sent once per condition rather than on every blip, because an alert that fires constantly trains people to ignore alerts.
- A push to an admin’s phone For a problem that cannot wait. Today it is sent by the same system it is reporting on, which is exactly the gap the independent layer closes.
- The person on call A rotation, an escalation if nobody answers, and push, SMS or voice — run by a provider on none of this system’s infrastructure, credentials or billing. Planned