# Sandbar

Self-hosted notifications, monitoring, on-call and incident response in one small program.
One binary, one SQLite file, and everything is done in the web UI.

Sandbar covers the ground of four tools:

| Area | Like | What it does |
|---|---|---|
| **Topics** | ntfy | Publish a message to a topic with one HTTP request. Subscribe in the browser, with `curl`, or with the ntfy phone app |
| **Monitors** | Tianji, Uptime Kuma | Website, port, DNS and certificate checks, heartbeats for cron jobs, and a public status page |
| **On-call** | Opsgenie | Rotating schedules, escalation policies, and alerts that are acknowledged, snoozed or resolved |
| **Incidents** | incident.io | Declare, assign, update and close an incident, with a timeline and a post-mortem |

It is built in the same style as Hammerhead, Blacktip and Bamboo and shares [Reef](https://git.veillax.com/Veillax/reef)
with them (`dotenv`, `netguard`, `secretbox`).

## Quick start

```
go build -o bin/ ./cmd/...

cp .env.example .env
bin/sandbar -genkey               # paste the output into .env as SANDBAR_MASTER_KEY=

bin/sandbar -listen :8090 -url https://sandbar.example.com
```

Open `http://localhost:8090` and create the first administrator. Or set `SANDBAR_ADMIN_EMAIL` and
`SANDBAR_ADMIN_PASSWORD` in `.env` to create it at startup.

Docker: `docker run -p 8090:8090 -v sandbar-data:/data -e SANDBAR_MASTER_KEY=... sandbar`.

## How the pieces fit

```
monitor fails ─┐
API request ───┼─> alert ─> escalation policy ─> person ─> topic + email
Alertmanager ──┘        │                                 (repeat until acknowledged)
                        └─> broadcast channels (Slack, Discord, ntfy, email)
                        └─> optionally an incident, with a timeline
```

Every alert is deduplicated by its key, so a monitor that stays down is one alert, not one per check.

## Topics

```
# publish (ntfy-style headers)
curl -H "Authorization: Bearer sb_..." -H "Title: Disk full" -H "Priority: urgent" -H "Tags: warning,disk" \
     -d "web-1 is at 98%" https://sandbar.example.com/ops

# subscribe: server-sent events, or JSON lines; ?since=all replays stored messages
curl -N -H "Authorization: Bearer sb_..." https://sandbar.example.com/ops/sse
curl -N -H "Authorization: Bearer sb_..." "https://sandbar.example.com/ops/json?since=all"
```

A topic is 1 to 64 letters, digits, `-` or `_`. Publishing and subscribing both need a signed-in session or an
API key (create one on the Account page). Clients that use basic auth can send the key as the password.
Messages are kept for `SANDBAR_MESSAGE_DAYS` (default 30).

Two topics are filled for you: `alerts` gets every alert and its recovery, and `incidents` gets every incident
update. Each person also has a personal topic (`u-<id>`, shown on the Account page). Subscribe to it in the ntfy
app to be paged on your phone when an escalation policy reaches you.

## Monitors

| Type | Passes when | Notes |
|---|---|---|
| Website (HTTP) | The status is in the expected range (default 2xx) and the page contains the text you set | Redirects are not followed |
| Port (TCP) | A connection opens | |
| DNS lookup | The name resolves | |
| Certificate expiry | The certificate has more than N days left (default 14) | |
| Heartbeat | Your job requests `/api/v1/hb/<token>` within its interval plus the allowed delay | A request on a down heartbeat recovers it at once |

A monitor goes down after the set number of failures in a row (default 2), raises an alert with the monitor's
escalation policy, and recovers on the first success. Monitors marked public appear on `/status`, together with
incidents marked public. The status page never shows staff names or private monitors.

Monitors cannot probe private or internal addresses unless you tick **Allow targets on private networks** on that
monitor, so a user who can add a monitor cannot use Sandbar to scan your network.

## On-call

- A **schedule** rotates through people, handing over every N hours from a start time. **Overrides** cover a
  holiday or swap a shift.
- An **escalation policy** is a list of steps. Each step waits some minutes, then tells people and/or whoever is
  on call for a schedule. It can start over after the last step until someone acknowledges.
- Alerts are stored with the time their next step is due, so restarting Sandbar loses nothing and repeats nothing.
- **Acknowledge** stops escalation. **Snooze** pauses it and resumes at the same step. **Resolve** closes the alert.

## Incidents

Declare one from the UI or `POST /api/v1/incidents`, or from an open alert (which links them). An incident has a
severity (sev1 to sev3), a status (investigating, identified, monitoring, resolved), a lead and a communications
owner, a timeline of everything that happened, comments, and a post-mortem with a template filled from the
timeline. Every change is announced on the `incidents` topic and on all channels.

## Sending alerts in

```
# generic
curl -H "Authorization: Bearer sb_..." -H "Content-Type: application/json" \
     -d '{"key":"cpu-web-1","summary":"CPU above 90%","severity":"warning","policy":"<policy id>"}' \
     https://sandbar.example.com/api/v1/alerts

# resolve it later with the same key
curl ... -d '{"key":"cpu-web-1","status":"resolved"}' https://sandbar.example.com/api/v1/alerts
```

Prometheus Alertmanager and Grafana can post their webhook to
`/api/v1/integrations/alertmanager?policy=<policy id>` with an API key as the bearer token.

## Channels

A channel receives every alert and incident update: an email address list, or a webhook in Slack, Discord, ntfy or
plain JSON format. The address is stored encrypted and is never sent back to the browser. Webhooks cannot point at
internal addresses.

## Configuration and secrets

`.env.example` lists every setting. Secrets at rest use AES-256-GCM with `SANDBAR_MASTER_KEY`; passwords are bcrypt
hashes and API keys and sessions are stored as SHA-256 hashes. With the wrong key, saved channel addresses cannot be
read. If you do not set a key, one is generated into `<data>/master.key`; move it into `.env`, and **back it up
separately**.

Prometheus metrics are served at `/metrics` when `SANDBAR_METRICS_TOKEN` is set.

## Database

SQLite (pure Go, no C compiler), in WAL mode, in `<data>/sandbar.db`. Messages, monitor results and alerts older
than their retention are pruned hourly. Back up by copying the data folder while Sandbar is stopped, plus the key.

## Limits and notes

- One instance, one organization. SQLite means no clustering.
- No ICMP ping monitor, no SMS or voice, and no two-factor or single sign-on yet.
- Sandbar serves plain HTTP. Put it behind HTTPS and set `-url`, or keep it on a private network.
- Sign-in attempts are limited to 10 per 10 minutes per address.
- Tests: `go test ./...`. Style: `node scripts/lint-design.mjs` (see `docs/design.md`).

## License

AGPL-3.0. See `LICENSE`.