# Sandbar Self-hosted notifications, monitoring, on-call and incident response in one small program. One binary, one SQLite file, and everything is done in the web UI. Sandbar covers the ground of four tools: | Area | Like | What it does | |---|---|---| | **Topics** | ntfy | Publish a message to a topic with one HTTP request. Subscribe in the browser, with `curl`, or with the ntfy phone app | | **Monitors** | Tianji, Uptime Kuma | Website, port, DNS and certificate checks, heartbeats for cron jobs, and a public status page | | **On-call** | Opsgenie | Rotating schedules, escalation policies, and alerts that are acknowledged, snoozed or resolved | | **Incidents** | incident.io | Declare, assign, update and close an incident, with a timeline and a post-mortem | It is built in the same style as Hammerhead, Blacktip and Bamboo and shares [Reef](https://git.veillax.com/Veillax/reef) with them (`dotenv`, `netguard`, `secretbox`). ## Quick start ``` go build -o bin/ ./cmd/... cp .env.example .env bin/sandbar -genkey # paste the output into .env as SANDBAR_MASTER_KEY= bin/sandbar -listen :8090 -url https://sandbar.example.com ``` Open `http://localhost:8090` and create the first administrator. Or set `SANDBAR_ADMIN_EMAIL` and `SANDBAR_ADMIN_PASSWORD` in `.env` to create it at startup. ## Deploying ### Docker Pushing a version tag publishes a multi-architecture (amd64 and arm64) image to the registry on the same server as the repository, as `git.veillax.com/veillax/sandbar:` and `:latest`. Generate a master key once with `docker run --rm git.veillax.com/veillax/sandbar -genkey`, then keep it in a `.env` file next to this compose file: ```yaml services: sandbar: image: git.veillax.com/veillax/sandbar:latest restart: unless-stopped ports: - "127.0.0.1:8090:8090" env_file: .env environment: SANDBAR_URL: https://sandbar.example.com volumes: - sandbar-data:/data volumes: sandbar-data: ``` ``` docker compose up -d ``` - The container runs as user 65532 and keeps its database in `/data`. Use a named volume as above. If you bind-mount a folder instead, make it writable by 65532 (`chown 65532:65532 ./data`). - Set `SANDBAR_MASTER_KEY` yourself. Without it a key is generated inside `/data`, next to the database it protects. - To build the image locally, see the comment at the top of `Dockerfile`. - Update with `docker compose pull && docker compose up -d`. Your data stays in the volume. Copy the data folder before upgrading, since there is no migration tooling yet. ### Binary Build `sandbar` (`go build -o bin/ ./cmd/sandbar`; releases only publish the container image), copy it to the server with a `.env` file, and run it under a service manager. A minimal systemd unit: ```ini [Unit] Description=Sandbar After=network-online.target [Service] User=sandbar WorkingDirectory=/var/lib/sandbar ExecStart=/usr/local/bin/sandbar -listen 127.0.0.1:8090 -env /etc/sandbar/.env Restart=on-failure [Install] WantedBy=multi-user.target ``` Put `SANDBAR_DATA=/var/lib/sandbar` in the env file and make `/etc/sandbar/.env` readable only by the `sandbar` user. ### HTTPS and a reverse proxy Sandbar serves plain HTTP, so put a reverse proxy in front of it and set `SANDBAR_URL` to the public address, since links in notifications use it. Caddy needs one line: ``` sandbar.example.com { reverse_proxy 127.0.0.1:8090 } ``` With nginx, turn off response buffering for the live topic feeds (`//sse` and `//json`), or subscribers see messages late: `proxy_buffering off; proxy_read_timeout 1h;`. ### After the first start 1. Open the address and create the administrator, or set `SANDBAR_ADMIN_EMAIL` and `SANDBAR_ADMIN_PASSWORD` first. 2. Add an email or webhook channel and use its test button, so you know alerts will reach you. 3. Create an API key on the Account page for scripts and monitoring agents. 4. Set up backups (see Database below), including the master key, which must be stored apart from the data. ## How the pieces fit ``` monitor fails ─┐ API request ───┼─> alert ─> escalation policy ─> person ─> topic + email Alertmanager ──┘ │ (repeat until acknowledged) └─> broadcast channels (Slack, Discord, ntfy, email) └─> optionally an incident, with a timeline ``` Every alert is deduplicated by its key, so a monitor that stays down is one alert, not one per check. ## Topics ``` # publish (ntfy-style headers) curl -H "Authorization: Bearer sb_..." -H "Title: Disk full" -H "Priority: urgent" -H "Tags: warning,disk" \ -d "web-1 is at 98%" https://sandbar.example.com/ops # subscribe: server-sent events, or JSON lines; ?since=all replays stored messages curl -N -H "Authorization: Bearer sb_..." https://sandbar.example.com/ops/sse curl -N -H "Authorization: Bearer sb_..." "https://sandbar.example.com/ops/json?since=all" ``` A topic is 1 to 64 letters, digits, `-` or `_`. Publishing and subscribing both need a signed-in session or an API key (create one on the Account page). Clients that use basic auth can send the key as the password. Messages are kept for `SANDBAR_MESSAGE_DAYS` (default 30). `?since=` also takes unix seconds, a duration such as `10m`, or the id of the last message you have, and `?poll=1` returns just that replay and closes, which is how the ntfy apps catch up. ### Phone app (ntfy) 1. Create an API key on the Account page and copy it. 2. In the ntfy app, open Settings, then Default server, and enter your Sandbar address (`https://sandbar.example.com`). Or tick "Use another server" when you add each subscription. 3. Add a subscription. For a page when an escalation reaches you, use your personal topic from the Account page (`u-`). For everything, use `alerts` and `incidents`. 4. Under Manage users, add a user for the server. The username can be anything and the password is the API key. The Android app keeps a connection open, so messages arrive at once. The iPhone app depends on ntfy.sh's push relay, which Sandbar does not use, so there messages arrive only when the app polls. Two topics are filled for you: `alerts` gets every alert and its recovery, and `incidents` gets every incident update. Each person also has a personal topic (`u-`, shown on the Account page). Subscribe to it in the ntfy app to be paged on your phone when an escalation policy reaches you. ## Monitors | Type | Passes when | Notes | |---|---|---| | Website (HTTP) | The status is in the expected range (default 2xx) and the page contains the text you set | Redirects are not followed | | Port (TCP) | A connection opens | | | DNS lookup | The name resolves | | | Certificate expiry | The certificate has more than N days left (default 14) | | | Heartbeat | Your job requests `/api/v1/hb/` within its interval plus the allowed delay | A request on a down heartbeat recovers it at once | A monitor goes down after the set number of failures in a row (default 2), raises an alert with the monitor's escalation policy, and recovers on the first success. Monitors marked public appear on `/status`, together with incidents marked public. The status page never shows staff names or private monitors. Monitors cannot probe private or internal addresses unless you tick **Allow targets on private networks** on that monitor, so a user who can add a monitor cannot use Sandbar to scan your network. ## On-call - A **schedule** rotates through people, handing over every N hours from a start time. **Overrides** cover a holiday or swap a shift. - An **escalation policy** is a list of steps. Each step waits some minutes, then tells people and/or whoever is on call for a schedule. It can start over after the last step until someone acknowledges. - Alerts are stored with the time their next step is due, so restarting Sandbar loses nothing and repeats nothing. - **Acknowledge** stops escalation. **Snooze** pauses it and resumes at the same step. **Resolve** closes the alert. ## Incidents Declare one from the UI or `POST /api/v1/incidents`, or from an open alert (which links them). An incident has a severity (sev1 to sev3), a status (investigating, identified, monitoring, resolved), a lead and a communications owner, a timeline of everything that happened, comments, and a post-mortem with a template filled from the timeline. Every change is announced on the `incidents` topic and on all channels. ## Sending alerts in ``` # generic curl -H "Authorization: Bearer sb_..." -H "Content-Type: application/json" \ -d '{"key":"cpu-web-1","summary":"CPU above 90%","severity":"warning","policy":""}' \ https://sandbar.example.com/api/v1/alerts # resolve it later with the same key curl ... -d '{"key":"cpu-web-1","status":"resolved"}' https://sandbar.example.com/api/v1/alerts ``` Prometheus Alertmanager and Grafana can post their webhook to `/api/v1/integrations/alertmanager?policy=` with an API key as the bearer token. ## Channels A channel receives every alert and incident update: an email address list, or a webhook in Slack, Discord, ntfy or plain JSON format. The address is stored encrypted and is never sent back to the browser. Webhooks cannot point at internal addresses. ## Configuration and secrets `.env.example` lists every setting. Secrets at rest use AES-256-GCM with `SANDBAR_MASTER_KEY`; passwords are bcrypt hashes and API keys and sessions are stored as SHA-256 hashes. With the wrong key, saved channel addresses cannot be read. If you do not set a key, one is generated into `/master.key`; move it into `.env`, and **back it up separately**. Prometheus metrics are served at `/metrics` when `SANDBAR_METRICS_TOKEN` is set. ## Database SQLite (pure Go, no C compiler), in WAL mode, in `/sandbar.db`. Messages, monitor results and alerts older than their retention are pruned hourly. Back up by copying the data folder while Sandbar is stopped, plus the key. ## Limits and notes - One instance, one organization. SQLite means no clustering. - No ICMP ping monitor, no SMS or voice, and no two-factor or single sign-on yet. - Sandbar serves plain HTTP. Put it behind HTTPS and set `-url`, or keep it on a private network. - Sign-in attempts are limited to 10 per 10 minutes per address. - Tests: `go test ./...`. Style: `node scripts/lint-design.mjs` (see `docs/design.md`). ## License AGPL-3.0. See `LICENSE`.