# Sandbar Self-hosted notifications, monitoring, on-call and incident response in one small program. One binary, one SQLite file, and everything is done in the web UI. Sandbar covers the ground of five tools: | Area | Like | What it does | |---|---|---| | **Topics** | ntfy | Publish a message to a topic with one HTTP request. Subscribe in the browser, with `curl`, or with the ntfy phone app | | **Monitors** | Tianji, Uptime Kuma | Website, port, DNS and certificate checks, heartbeats for cron jobs, and a status page with groups, incident posts, maintenance and subscribers | | **Websites** | Tianji, Plausible, Umami | Page views, visitors, referrers, browsers, countries and custom events for your own sites, with no cookies and no personal data | | **On-call** | Opsgenie | Rotating schedules, escalation policies, and alerts that are acknowledged, snoozed or resolved | | **Incidents** | incident.io | Declare, assign, update and close an incident, with a timeline and a post-mortem | It is built in the same style as Hammerhead, Blacktip and Bamboo and shares [Reef](https://git.veillax.com/Veillax/reef) with them (`dotenv`, `netguard`, `secretbox`). ## Quick start ``` go build -o bin/ ./cmd/... cp .env.example .env bin/sandbar -genkey # paste the output into .env as SANDBAR_MASTER_KEY= bin/sandbar -listen :8090 -url https://sandbar.example.com ``` Open `http://localhost:8090` and create the first administrator. Or set `SANDBAR_ADMIN_EMAIL` and `SANDBAR_ADMIN_PASSWORD` in `.env` to create it at startup. ## Deploying ### Docker Pushing a version tag publishes a multi-architecture (amd64 and arm64) image to the registry on the same server as the repository, as `git.veillax.com/veillax/sandbar:` and `:latest`. Generate a master key once with `docker run --rm git.veillax.com/veillax/sandbar -genkey`, then keep it in a `.env` file next to this compose file: ```yaml services: sandbar: image: git.veillax.com/veillax/sandbar:latest restart: unless-stopped ports: - "127.0.0.1:8090:8090" env_file: .env environment: SANDBAR_URL: https://sandbar.example.com SANDBAR_AUTO_UPDATE: "1" # optional, see Updating below volumes: - sandbar-data:/data volumes: sandbar-data: ``` ``` docker compose up -d ``` - The container runs as user 65532 and keeps its database in `/data`. Use a named volume as above. If you bind-mount a folder instead, make it writable by 65532 (`chown 65532:65532 ./data`). - Set `SANDBAR_MASTER_KEY` yourself. Without it a key is generated inside `/data`, next to the database it protects. - To build the image locally, see the comment at the top of `Dockerfile`. - Update by hand with `docker compose pull && docker compose up -d`, or let Sandbar do it (see Updating below). Your data stays in the volume. Copy the data folder before upgrading, since there is no migration tooling yet. ### Binary Download an archive from the release page (Linux amd64 and arm64, Windows amd64) or build it (`go build -o bin/ ./cmd/sandbar`), copy it to the server with a `.env` file, and run it under a service manager. A minimal systemd unit: ```ini [Unit] Description=Sandbar After=network-online.target [Service] User=sandbar WorkingDirectory=/var/lib/sandbar ExecStart=/usr/local/bin/sandbar -listen 127.0.0.1:8090 -env /etc/sandbar/.env Restart=on-failure [Install] WantedBy=multi-user.target ``` Put `SANDBAR_DATA=/var/lib/sandbar` in the env file and make `/etc/sandbar/.env` readable only by the `sandbar` user. ### Updating Pushing a version tag publishes a release with archives and `SHA256SUMS` as well as the image. Sandbar can install new releases itself, in Docker and as a plain binary, on Linux and macOS. - **Automatic:** set `SANDBAR_AUTO_UPDATE=1` (or pass `-auto-update`). Sandbar checks a few minutes after start and then every few hours, downloads the archive for its platform, checks it against `SHA256SUMS`, checks that the new binary runs, replaces itself and restarts. Pre-releases (`v1.0.0-rc1`) are never installed. - **On demand:** an administrator can see the newest release and click **Update** on the Account page, with or without automatic updates on. - **In Docker:** the image is read-only to Sandbar, and a change inside the container would be lost when it is recreated. So the new binary is written to `/bin/sandbar` on the data volume. At every start Sandbar runs that file when it is a newer release than the image, and deletes it once the image has caught up. The container keeps running through the update (it re-executes in place, so the process id does not change), and `docker compose pull` still works: a newer image takes over from the volume copy. - **Pin a version:** leave automatic updates off and use an image tag such as `:v0.2.0` instead of `:latest`. - **Requirements:** the release page must be readable without signing in, so the repository has to be public. A development build (`dev`) and Windows never update themselves. `SANDBAR_UPDATE_SERVER` and `SANDBAR_UPDATE_REPO` point it at a different Gitea server or fork. If a bad update leaves Sandbar unable to start, delete `/bin/sandbar` (or set `SANDBAR_NO_OVERLAY=1`) and it starts from the image again. ### HTTPS and a reverse proxy Sandbar serves plain HTTP, so put a reverse proxy in front of it and set `SANDBAR_URL` to the public address, since links in notifications use it. Also set `SANDBAR_TRUST_PROXY=1`, so website analytics and the sign-in limit see the real client address instead of the proxy's. Caddy needs one line: ``` sandbar.example.com { reverse_proxy 127.0.0.1:8090 } ``` With nginx, turn off response buffering for the live topic feeds (`//sse` and `//json`), or subscribers see messages late: `proxy_buffering off; proxy_read_timeout 1h;`. ### After the first start 1. Open the address and create the administrator, or set `SANDBAR_ADMIN_EMAIL` and `SANDBAR_ADMIN_PASSWORD` first. 2. Add an email or webhook channel and use its test button, so you know alerts will reach you. 3. Create an API key on the Account page for scripts and monitoring agents. 4. Set up backups (see Database below), including the master key, which must be stored apart from the data. ## How the pieces fit ``` monitor fails ─┐ API request ───┼─> alert ─> escalation policy ─> person ─> topic + email Alertmanager ──┘ │ (repeat until acknowledged) └─> broadcast channels (Slack, Discord, ntfy, email) └─> optionally an incident, with a timeline ``` Every alert is deduplicated by its key, so a monitor that stays down is one alert, not one per check. ## Topics ``` # publish (ntfy-style headers) curl -H "Authorization: Bearer sb_..." -H "Title: Disk full" -H "Priority: urgent" -H "Tags: warning,disk" \ -d "web-1 is at 98%" https://sandbar.example.com/ops # subscribe: server-sent events, or JSON lines; ?since=all replays stored messages curl -N -H "Authorization: Bearer sb_..." https://sandbar.example.com/ops/sse curl -N -H "Authorization: Bearer sb_..." "https://sandbar.example.com/ops/json?since=all" ``` A topic is 1 to 64 letters, digits, `-` or `_`. Publishing and subscribing both need a signed-in session or an API key (create one on the Account page). Clients that use basic auth can send the key as the password. Messages are kept for `SANDBAR_MESSAGE_DAYS` (default 30). `?since=` also takes unix seconds, a duration such as `10m`, or the id of the last message you have, and `?poll=1` returns just that replay and closes, which is how the ntfy apps catch up. ### Phone app (ntfy) 1. Create an API key on the Account page and copy it. 2. In the ntfy app, open Settings, then Default server, and enter your Sandbar address (`https://sandbar.example.com`). Or tick "Use another server" when you add each subscription. 3. Add a subscription. For a page when an escalation reaches you, use your personal topic from the Account page (`u-`). For everything, use `alerts` and `incidents`. 4. Under Manage users, add a user for the server. The username can be anything and the password is the API key. The Android app keeps a connection open, so messages arrive at once. The iPhone app depends on ntfy.sh's push relay, which Sandbar does not use, so there messages arrive only when the app polls. Two topics are filled for you: `alerts` gets every alert and its recovery, and `incidents` gets every incident update. Each person also has a personal topic (`u-`, shown on the Account page). Subscribe to it in the ntfy app to be paged on your phone when an escalation policy reaches you. ## Monitors | Type | Passes when | Notes | |---|---|---| | Website (HTTP) | The status is in the expected range (default 2xx) and the page contains the text you set | Redirects are not followed | | Port (TCP) | A connection opens | | | DNS lookup | The name resolves | | | Certificate expiry | The certificate has more than N days left (default 14) | | | Heartbeat | Your job requests `/api/v1/hb/` within its interval plus the allowed delay | A request on a down heartbeat recovers it at once | A monitor goes down after the set number of failures in a row (default 2), raises an alert with the monitor's escalation policy, and recovers on the first success. Monitors marked public appear on the status page (see below), together with incidents marked public. The status page never shows staff names or private monitors. The status page shows each public monitor as 90 daily uptime bars, like most hosted status pages. A day is green with no failed checks, amber when at least 99% of checks passed, red below that, and grey with no data. Hovering a bar (or tapping it on a phone) gives the date, the uptime and the number of failed checks. Days are UTC. Counts are kept in a small per-day table, so the bars go back 90 days even though raw check results are kept for `SANDBAR_CHECK_DAYS` only. Monitors cannot probe private or internal addresses unless you tick **Allow targets on private networks** on that monitor, so a user who can add a monitor cannot use Sandbar to scan your network. ## Status page The public page is at `/status` (and at the root of its own domain, see below). Everything is set up under **Status page** in the sidebar. - **Components and groups.** Every monitor with **Show on the public status page** ticked is a component, shown as 90 daily uptime bars (see Monitors). Create groups on the Groups tab, then pick a group, and optionally a different public name, in the monitor's form. Groups are ordered by their order number; components without a group come first. - **Incidents.** An incident marked public is posted with its status, the components it affects, and the updates you write. Declare it as public, or switch it on later. **Post an update** (or changing the status) adds a post with your text, or standard wording if you leave it empty. Customers see only the posts. The internal timeline, comments, names and post-mortem never leave Sandbar. The page shows the last 14 days; everything older is on `/status/history`, and each incident has its own page with the full list of updates. - **What the page says.** A component is *Major outage* when its monitor is down or a sev1 incident names it, *Partial outage* for sev2, *Degraded performance* for sev3, and *Under maintenance* during a maintenance window for it. The banner shows the worst of them. - **Scheduled maintenance.** Give a window (your local time in the form, stored as UTC) and the affected components. It is announced when scheduled, listed on the page until it ends, and subscribers are told when it starts and when it ends. During the window its components show as under maintenance instead of as an outage. Windows can be moved or canceled, and subscribers are told. Alerts for failing checks still fire as usual. - **Branding.** Title, description, logo (an https address), accent color, a contact link and footer text. The accent is checked so that white text on it stays readable (WCAG 4.5 to 1). - **Custom domain.** Enter a host name such as `status.example.com`, point its DNS at your server and have the reverse proxy forward that host to Sandbar with the `Host` header unchanged (Caddy does this by default; with nginx add `proxy_set_header Host $host;`). Sandbar then serves the status page at `/` on that host, and nothing else: the app, API and topics are not reachable through it. Links in emails and feeds use `https://`. The proxy provides the certificate. With `SANDBAR_TRUST_PROXY=1` the host is read from `X-Forwarded-Host` when present. ### Following the page Visitors choose **Subscribe to updates** on the page: - **Email.** Needs `SANDBAR_SMTP_HOST`; the form is hidden without it. A visitor gets a confirmation link, and no mail is sent until it is opened. Every message has an unsubscribe link, which asks for one click on the page, so mail scanners cannot unsubscribe people. Unconfirmed addresses are forgotten after 7 days. - **Webhook.** The address must answer a test request with a 2xx status before it is subscribed. Requests to private networks are refused. Each update is a JSON `POST`: `{"page","event","title","status","message","url","time","unsubscribe"}`. The events are `incident.created`, `incident.updated`, `incident.resolved`, `maintenance.scheduled`, `maintenance.rescheduled`, `maintenance.started`, `maintenance.completed` and `maintenance.canceled`. The `unsubscribe` link works with a `POST` of `{"token": ...}` to `/status/unsubscribe`, or can be opened in a browser. - **RSS.** `/status/feed.rss` has every incident update and maintenance announcement, newest first. Subscribers can be seen and removed on the Subscribers tab (webhook addresses show only the host). Addresses are stored encrypted. Subscribing is limited to 6 requests per hour per address. ### Status page API | Request | | |---|---| | `GET /status.json` | Settings, groups, components with their 90 days, current incidents, upcoming and running maintenance | | `GET /status/api/history?before=