Initial commit: notifications, monitoring, on-call and incidents
One binary and one SQLite file. Topics with ntfy-style publish and subscribe, HTTP/TCP/DNS/TLS/heartbeat monitors with a public status page, schedules and escalation policies, and incidents with a timeline and post-mortem. The UI follows the Lightwell design language. Builds against an unreleased Reef (secretbox) through a local replace line. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
commit
0c5a57556a
22 files changed
+4953
No files matched your search
@@ -0,0 +1,144 @@
|
||||
# Sandbar
|
||||
|
||||
Self-hosted notifications, monitoring, on-call and incident response in one small program.
|
||||
One binary, one SQLite file, and everything is done in the web UI.
|
||||
|
||||
Sandbar covers the ground of four tools:
|
||||
|
||||
| Area | Like | What it does |
|
||||
|---|---|---|
|
||||
| **Topics** | ntfy | Publish a message to a topic with one HTTP request. Subscribe in the browser, with `curl`, or with the ntfy phone app |
|
||||
| **Monitors** | Tianji, Uptime Kuma | Website, port, DNS and certificate checks, heartbeats for cron jobs, and a public status page |
|
||||
| **On-call** | Opsgenie | Rotating schedules, escalation policies, and alerts that are acknowledged, snoozed or resolved |
|
||||
| **Incidents** | incident.io | Declare, assign, update and close an incident, with a timeline and a post-mortem |
|
||||
|
||||
It is built in the same style as Hammerhead, Blacktip and Bamboo and shares [Reef](https://git.veillax.com/Veillax/reef)
|
||||
with them (`dotenv`, `netguard`, `secretbox`).
|
||||
|
||||
## Quick start
|
||||
|
||||
```
|
||||
go build -o bin/ ./cmd/...
|
||||
|
||||
cp .env.example .env
|
||||
bin/sandbar -genkey # paste the output into .env as SANDBAR_MASTER_KEY=
|
||||
|
||||
bin/sandbar -listen :8090 -url https://sandbar.example.com
|
||||
```
|
||||
|
||||
Open `http://localhost:8090` and create the first administrator. Or set `SANDBAR_ADMIN_EMAIL` and
|
||||
`SANDBAR_ADMIN_PASSWORD` in `.env` to create it at startup.
|
||||
|
||||
Docker: `docker run -p 8090:8090 -v sandbar-data:/data -e SANDBAR_MASTER_KEY=... sandbar`.
|
||||
|
||||
## How the pieces fit
|
||||
|
||||
```
|
||||
monitor fails ─┐
|
||||
API request ───┼─> alert ─> escalation policy ─> person ─> topic + email
|
||||
Alertmanager ──┘ │ (repeat until acknowledged)
|
||||
└─> broadcast channels (Slack, Discord, ntfy, email)
|
||||
└─> optionally an incident, with a timeline
|
||||
```
|
||||
|
||||
Every alert is deduplicated by its key, so a monitor that stays down is one alert, not one per check.
|
||||
|
||||
## Topics
|
||||
|
||||
```
|
||||
# publish (ntfy-style headers)
|
||||
curl -H "Authorization: Bearer sb_..." -H "Title: Disk full" -H "Priority: urgent" -H "Tags: warning,disk" \
|
||||
-d "web-1 is at 98%" https://sandbar.example.com/ops
|
||||
|
||||
# subscribe: server-sent events, or JSON lines; ?since=all replays stored messages
|
||||
curl -N -H "Authorization: Bearer sb_..." https://sandbar.example.com/ops/sse
|
||||
curl -N -H "Authorization: Bearer sb_..." "https://sandbar.example.com/ops/json?since=all"
|
||||
```
|
||||
|
||||
A topic is 1 to 64 letters, digits, `-` or `_`. Publishing and subscribing both need a signed-in session or an
|
||||
API key (create one on the Account page). Clients that use basic auth can send the key as the password.
|
||||
Messages are kept for `SANDBAR_MESSAGE_DAYS` (default 30).
|
||||
|
||||
Two topics are filled for you: `alerts` gets every alert and its recovery, and `incidents` gets every incident
|
||||
update. Each person also has a personal topic (`u-<id>`, shown on the Account page). Subscribe to it in the ntfy
|
||||
app to be paged on your phone when an escalation policy reaches you.
|
||||
|
||||
## Monitors
|
||||
|
||||
| Type | Passes when | Notes |
|
||||
|---|---|---|
|
||||
| Website (HTTP) | The status is in the expected range (default 2xx) and the page contains the text you set | Redirects are not followed |
|
||||
| Port (TCP) | A connection opens | |
|
||||
| DNS lookup | The name resolves | |
|
||||
| Certificate expiry | The certificate has more than N days left (default 14) | |
|
||||
| Heartbeat | Your job requests `/api/v1/hb/<token>` within its interval plus the allowed delay | A request on a down heartbeat recovers it at once |
|
||||
|
||||
A monitor goes down after the set number of failures in a row (default 2), raises an alert with the monitor's
|
||||
escalation policy, and recovers on the first success. Monitors marked public appear on `/status`, together with
|
||||
incidents marked public. The status page never shows staff names or private monitors.
|
||||
|
||||
Monitors cannot probe private or internal addresses unless you tick **Allow targets on private networks** on that
|
||||
monitor, so a user who can add a monitor cannot use Sandbar to scan your network.
|
||||
|
||||
## On-call
|
||||
|
||||
- A **schedule** rotates through people, handing over every N hours from a start time. **Overrides** cover a
|
||||
holiday or swap a shift.
|
||||
- An **escalation policy** is a list of steps. Each step waits some minutes, then tells people and/or whoever is
|
||||
on call for a schedule. It can start over after the last step until someone acknowledges.
|
||||
- Alerts are stored with the time their next step is due, so restarting Sandbar loses nothing and repeats nothing.
|
||||
- **Acknowledge** stops escalation. **Snooze** pauses it and resumes at the same step. **Resolve** closes the alert.
|
||||
|
||||
## Incidents
|
||||
|
||||
Declare one from the UI or `POST /api/v1/incidents`, or from an open alert (which links them). An incident has a
|
||||
severity (sev1 to sev3), a status (investigating, identified, monitoring, resolved), a lead and a communications
|
||||
owner, a timeline of everything that happened, comments, and a post-mortem with a template filled from the
|
||||
timeline. Every change is announced on the `incidents` topic and on all channels.
|
||||
|
||||
## Sending alerts in
|
||||
|
||||
```
|
||||
# generic
|
||||
curl -H "Authorization: Bearer sb_..." -H "Content-Type: application/json" \
|
||||
-d '{"key":"cpu-web-1","summary":"CPU above 90%","severity":"warning","policy":"<policy id>"}' \
|
||||
https://sandbar.example.com/api/v1/alerts
|
||||
|
||||
# resolve it later with the same key
|
||||
curl ... -d '{"key":"cpu-web-1","status":"resolved"}' https://sandbar.example.com/api/v1/alerts
|
||||
```
|
||||
|
||||
Prometheus Alertmanager and Grafana can post their webhook to
|
||||
`/api/v1/integrations/alertmanager?policy=<policy id>` with an API key as the bearer token.
|
||||
|
||||
## Channels
|
||||
|
||||
A channel receives every alert and incident update: an email address list, or a webhook in Slack, Discord, ntfy or
|
||||
plain JSON format. The address is stored encrypted and is never sent back to the browser. Webhooks cannot point at
|
||||
internal addresses.
|
||||
|
||||
## Configuration and secrets
|
||||
|
||||
`.env.example` lists every setting. Secrets at rest use AES-256-GCM with `SANDBAR_MASTER_KEY`; passwords are bcrypt
|
||||
hashes and API keys and sessions are stored as SHA-256 hashes. With the wrong key, saved channel addresses cannot be
|
||||
read. If you do not set a key, one is generated into `<data>/master.key`; move it into `.env`, and **back it up
|
||||
separately**.
|
||||
|
||||
Prometheus metrics are served at `/metrics` when `SANDBAR_METRICS_TOKEN` is set.
|
||||
|
||||
## Database
|
||||
|
||||
SQLite (pure Go, no C compiler), in WAL mode, in `<data>/sandbar.db`. Messages, monitor results and alerts older
|
||||
than their retention are pruned hourly. Back up by copying the data folder while Sandbar is stopped, plus the key.
|
||||
|
||||
## Limits and notes
|
||||
|
||||
- One instance, one organization. SQLite means no clustering.
|
||||
- No ICMP ping monitor, no SMS or voice, and no two-factor or single sign-on yet.
|
||||
- Sandbar serves plain HTTP. Put it behind HTTPS and set `-url`, or keep it on a private network.
|
||||
- Sign-in attempts are limited to 10 per 10 minutes per address.
|
||||
- Tests: `go test ./...`. Style: `node scripts/lint-design.mjs` (see `docs/design.md`).
|
||||
|
||||
## License
|
||||
|
||||
AGPL-3.0. See `LICENSE`.
|
||||
Reference in new issue
Block a user