Sandbar/README.md

# Sandbar

Self-hosted notifications, monitoring, on-call and incident response in one small program.
One binary, one SQLite file, and everything is done in the web UI.

Sandbar covers the ground of four tools:

| Area | Like | What it does |
|---|---|---|
| **Topics** | ntfy | Publish a message to a topic with one HTTP request. Subscribe in the browser, with `curl`, or with the ntfy phone app |
| **Monitors** | Tianji, Uptime Kuma | Website, port, DNS and certificate checks, heartbeats for cron jobs, and a public status page |
| **On-call** | Opsgenie | Rotating schedules, escalation policies, and alerts that are acknowledged, snoozed or resolved |
| **Incidents** | incident.io | Declare, assign, update and close an incident, with a timeline and a post-mortem |

It is built in the same style as Hammerhead, Blacktip and Bamboo and shares [Reef](https://git.veillax.com/Veillax/reef)
with them (`dotenv`, `netguard`, `secretbox`).

## Quick start

```
go build -o bin/ ./cmd/...

cp .env.example .env
bin/sandbar -genkey               # paste the output into .env as SANDBAR_MASTER_KEY=

bin/sandbar -listen :8090 -url https://sandbar.example.com
```

Open `http://localhost:8090` and create the first administrator. Or set `SANDBAR_ADMIN_EMAIL` and
`SANDBAR_ADMIN_PASSWORD` in `.env` to create it at startup.

## Deploying

### Docker

Pushing a version tag publishes a multi-architecture (amd64 and arm64) image to the registry on the same server as
the repository, as `git.veillax.com/veillax/sandbar:<version>` and `:latest`. Generate a master key once with
`docker run --rm git.veillax.com/veillax/sandbar -genkey`, then keep it in a `.env` file next to this compose file:

```yaml
services:
  sandbar:
    image: git.veillax.com/veillax/sandbar:latest
    restart: unless-stopped
    ports:
      - "127.0.0.1:8090:8090"
    env_file: .env
    environment:
      SANDBAR_URL: https://sandbar.example.com
      SANDBAR_AUTO_UPDATE: "1"   # optional, see Updating below
    volumes:
      - sandbar-data:/data

volumes:
  sandbar-data:
```

```
docker compose up -d
```

- The container runs as user 65532 and keeps its database in `/data`. Use a named volume as above. If you bind-mount
  a folder instead, make it writable by 65532 (`chown 65532:65532 ./data`).
- Set `SANDBAR_MASTER_KEY` yourself. Without it a key is generated inside `/data`, next to the database it protects.
- To build the image locally, see the comment at the top of `Dockerfile`.
- Update by hand with `docker compose pull && docker compose up -d`, or let Sandbar do it (see Updating below).
  Your data stays in the volume. Copy the data folder before upgrading, since there is no migration tooling yet.

### Binary

Download an archive from the release page (Linux amd64 and arm64, Windows amd64) or build it (`go build -o bin/ ./cmd/sandbar`), copy it to the server with a `.env` file, and run it
under a service manager. A minimal systemd unit:

```ini
[Unit]
Description=Sandbar
After=network-online.target

[Service]
User=sandbar
WorkingDirectory=/var/lib/sandbar
ExecStart=/usr/local/bin/sandbar -listen 127.0.0.1:8090 -env /etc/sandbar/.env
Restart=on-failure

[Install]
WantedBy=multi-user.target
```

Put `SANDBAR_DATA=/var/lib/sandbar` in the env file and make `/etc/sandbar/.env` readable only by the `sandbar` user.

### Updating

Pushing a version tag publishes a release with archives and `SHA256SUMS` as well as the image. Sandbar can install
new releases itself, in Docker and as a plain binary, on Linux and macOS.

- **Automatic:** set `SANDBAR_AUTO_UPDATE=1` (or pass `-auto-update`). Sandbar checks a few minutes after start and
  then every few hours, downloads the archive for its platform, checks it against `SHA256SUMS`, checks that the new
  binary runs, replaces itself and restarts. Pre-releases (`v1.0.0-rc1`) are never installed.
- **On demand:** an administrator can see the newest release and click **Update** on the Account page, with or
  without automatic updates on.
- **In Docker:** the image is read-only to Sandbar, and a change inside the container would be lost when it is
  recreated. So the new binary is written to `<data>/bin/sandbar` on the data volume. At every start Sandbar runs
  that file when it is a newer release than the image, and deletes it once the image has caught up. The container
  keeps running through the update (it re-executes in place, so the process id does not change), and
  `docker compose pull` still works: a newer image takes over from the volume copy.
- **Pin a version:** leave automatic updates off and use an image tag such as `:v0.2.0` instead of `:latest`.
- **Requirements:** the release page must be readable without signing in, so the repository has to be public. A
  development build (`dev`) and Windows never update themselves. `SANDBAR_UPDATE_SERVER` and `SANDBAR_UPDATE_REPO`
  point it at a different Gitea server or fork. If a bad update leaves Sandbar unable to start, delete
  `<data>/bin/sandbar` (or set `SANDBAR_NO_OVERLAY=1`) and it starts from the image again.

### HTTPS and a reverse proxy

Sandbar serves plain HTTP, so put a reverse proxy in front of it and set `SANDBAR_URL` to the public address, since
links in notifications use it. Caddy needs one line:

```
sandbar.example.com {
    reverse_proxy 127.0.0.1:8090
}
```

With nginx, turn off response buffering for the live topic feeds (`/<topic>/sse` and `/<topic>/json`), or
subscribers see messages late: `proxy_buffering off; proxy_read_timeout 1h;`.

### After the first start

1. Open the address and create the administrator, or set `SANDBAR_ADMIN_EMAIL` and `SANDBAR_ADMIN_PASSWORD` first.
2. Add an email or webhook channel and use its test button, so you know alerts will reach you.
3. Create an API key on the Account page for scripts and monitoring agents.
4. Set up backups (see Database below), including the master key, which must be stored apart from the data.

## How the pieces fit

```
monitor fails ─┐
API request ───┼─> alert ─> escalation policy ─> person ─> topic + email
Alertmanager ──┘        │                                 (repeat until acknowledged)
                        └─> broadcast channels (Slack, Discord, ntfy, email)
                        └─> optionally an incident, with a timeline
```

Every alert is deduplicated by its key, so a monitor that stays down is one alert, not one per check.

## Topics

```
# publish (ntfy-style headers)
curl -H "Authorization: Bearer sb_..." -H "Title: Disk full" -H "Priority: urgent" -H "Tags: warning,disk" \
     -d "web-1 is at 98%" https://sandbar.example.com/ops

# subscribe: server-sent events, or JSON lines; ?since=all replays stored messages
curl -N -H "Authorization: Bearer sb_..." https://sandbar.example.com/ops/sse
curl -N -H "Authorization: Bearer sb_..." "https://sandbar.example.com/ops/json?since=all"
```

A topic is 1 to 64 letters, digits, `-` or `_`. Publishing and subscribing both need a signed-in session or an
API key (create one on the Account page). Clients that use basic auth can send the key as the password.
Messages are kept for `SANDBAR_MESSAGE_DAYS` (default 30).

`?since=` also takes unix seconds, a duration such as `10m`, or the id of the last message you have, and
`?poll=1` returns just that replay and closes, which is how the ntfy apps catch up.

### Phone app (ntfy)

1. Create an API key on the Account page and copy it.
2. In the ntfy app, open Settings, then Default server, and enter your Sandbar address
   (`https://sandbar.example.com`). Or tick "Use another server" when you add each subscription.
3. Add a subscription. For a page when an escalation reaches you, use your personal topic from the Account page
   (`u-<id>`). For everything, use `alerts` and `incidents`.
4. Under Manage users, add a user for the server. The username can be anything and the password is the API key.

The Android app keeps a connection open, so messages arrive at once. The iPhone app depends on ntfy.sh's push
relay, which Sandbar does not use, so there messages arrive only when the app polls.

Two topics are filled for you: `alerts` gets every alert and its recovery, and `incidents` gets every incident
update. Each person also has a personal topic (`u-<id>`, shown on the Account page). Subscribe to it in the ntfy
app to be paged on your phone when an escalation policy reaches you.

## Monitors

| Type | Passes when | Notes |
|---|---|---|
| Website (HTTP) | The status is in the expected range (default 2xx) and the page contains the text you set | Redirects are not followed |
| Port (TCP) | A connection opens | |
| DNS lookup | The name resolves | |
| Certificate expiry | The certificate has more than N days left (default 14) | |
| Heartbeat | Your job requests `/api/v1/hb/<token>` within its interval plus the allowed delay | A request on a down heartbeat recovers it at once |

A monitor goes down after the set number of failures in a row (default 2), raises an alert with the monitor's
escalation policy, and recovers on the first success. Monitors marked public appear on `/status`, together with
incidents marked public. The status page never shows staff names or private monitors.

The status page shows each public monitor as 90 daily uptime bars, like most hosted status pages. A day is green with
no failed checks, amber when at least 99% of checks passed, red below that, and grey with no data. Hovering a bar
(or tapping it on a phone) gives the date, the uptime and the number of failed checks. Days are UTC. Counts are kept
in a small per-day table, so the bars go back 90 days even though raw check results are kept for
`SANDBAR_CHECK_DAYS` only.

Monitors cannot probe private or internal addresses unless you tick **Allow targets on private networks** on that
monitor, so a user who can add a monitor cannot use Sandbar to scan your network.

## On-call

- A **schedule** rotates through people, handing over every N hours from a start time. **Overrides** cover a
  holiday or swap a shift.
- An **escalation policy** is a list of steps. Each step waits some minutes, then tells people and/or whoever is
  on call for a schedule. It can start over after the last step until someone acknowledges.
- Alerts are stored with the time their next step is due, so restarting Sandbar loses nothing and repeats nothing.
- **Acknowledge** stops escalation. **Snooze** pauses it and resumes at the same step. **Resolve** closes the alert.

## Incidents

Declare one from the UI or `POST /api/v1/incidents`, or from an open alert (which links them). An incident has a
severity (sev1 to sev3), a status (investigating, identified, monitoring, resolved), a lead and a communications
owner, a timeline of everything that happened, comments, and a post-mortem with a template filled from the
timeline. Every change is announced on the `incidents` topic and on all channels.

## Sending alerts in

```
# generic
curl -H "Authorization: Bearer sb_..." -H "Content-Type: application/json" \
     -d '{"key":"cpu-web-1","summary":"CPU above 90%","severity":"warning","policy":"<policy id>"}' \
     https://sandbar.example.com/api/v1/alerts

# resolve it later with the same key
curl ... -d '{"key":"cpu-web-1","status":"resolved"}' https://sandbar.example.com/api/v1/alerts
```

Prometheus Alertmanager and Grafana can post their webhook to
`/api/v1/integrations/alertmanager?policy=<policy id>` with an API key as the bearer token.

## Channels

A channel receives every alert and incident update: an email address list, or a webhook in Slack, Discord, ntfy or
plain JSON format. The address is stored encrypted and is never sent back to the browser. Webhooks cannot point at
internal addresses.

## Configuration and secrets

`.env.example` lists every setting. Secrets at rest use AES-256-GCM with `SANDBAR_MASTER_KEY`; passwords are bcrypt
hashes and API keys and sessions are stored as SHA-256 hashes. With the wrong key, saved channel addresses cannot be
read. If you do not set a key, one is generated into `<data>/master.key`; move it into `.env`, and **back it up
separately**.

Prometheus metrics are served at `/metrics` when `SANDBAR_METRICS_TOKEN` is set.

## Database

SQLite (pure Go, no C compiler), in WAL mode, in `<data>/sandbar.db`. Messages, monitor results and alerts older
than their retention are pruned hourly. Back up by copying the data folder while Sandbar is stopped, plus the key.

## Limits and notes

- One instance, one organization. SQLite means no clustering.
- No ICMP ping monitor, no SMS or voice, and no two-factor or single sign-on yet.
- Sandbar serves plain HTTP. Put it behind HTTPS and set `-url`, or keep it on a private network.
- Sign-in attempts are limited to 10 per 10 minutes per address.
- Tests: `go test ./...`. Style: `node scripts/lint-design.mjs` (see `docs/design.md`).

## License

AGPL-3.0. See `LICENSE`.