SSH access from the Sites page; containers and discovery dialog; dialogs fit small screens

Each site has an SSH button: it opens a TCP port on an edge whose target is
Bamboo's built-in SSH server, and shows the ssh command to use. An end-to-end
test signs in through the edge, the tunnel and Bamboo's server.

The Found by dialog now lists Docker containers and discovered services, wraps
long names and groups TCP and UDP ports. The dialog overlay no longer lets a
wide table push a dialog off a phone screen. Builds against Bamboo v0.1.8.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

.env.example

@@ -21,6 +21,14 @@ HAMMERHEAD_MASTER_KEY=
 # Leave empty and /metrics does not exist.
 #HAMMERHEAD_METRICS_TOKEN=
 
+# Serve the dashboard through a Blacktip edge on this same machine, at this name, with an
+# automatic HTTPS certificate: no Caddy or other web server in front of Hammerhead. Hammerhead
+# makes the edge itself and writes its token to <data>/self-edge.env (BLACKTIP_TOKEN=...).
+# deploy/setup.sh --domain sets this up for you. Needs HAMMERHEAD_URL set to the same name.
+#HAMMERHEAD_SELF_DOMAIN=hammerhead.example.com
+# The edge's public WireGuard address for sites; default <HAMMERHEAD_SELF_DOMAIN>:51820.
+#HAMMERHEAD_SELF_TUNNEL=hammerhead.example.com:51820
+
 # How long access logs are kept (days). Default 30.
 #HAMMERHEAD_LOG_DAYS=30
 
@@ -38,3 +46,7 @@ HAMMERHEAD_MASTER_KEY=
 # Token from Sites -> Show command.
 #BAMBOO_HAMMERHEAD=https://hammerhead.example.com
 #BAMBOO_TOKEN=
+
+# Check for new releases every few hours and update and restart by itself (Linux and macOS).
+# setup.sh turns this on; remove it to update by hand.
+#HAMMERHEAD_AUTO_UPDATE=1

.gitea/workflows/ci.yml

@@ -44,7 +44,7 @@ jobs:
       # Catches platform-specific breakage before a release is tagged.
       - name: Every release platform compiles
         run: |
-          for t in linux/amd64 linux/arm64 linux/arm windows/amd64 darwin/amd64 darwin/arm64; do
+          for t in linux/amd64 windows/amd64; do
             echo "building for $t"
             GOOS=${t%/*} GOARCH=${t#*/} GOARM=7 CGO_ENABLED=0 go build -o /dev/null ./cmd/hammerhead
           done

.gitea/workflows/release.yml

@@ -59,4 +59,4 @@ jobs:
             *-*) ;;                       # pre-releases (v1.0.0-rc1) don't move :latest
             *) tags="$tags -t $image:latest" ;;
           esac
-          docker buildx build --platform linux/amd64,linux/arm64 $tags --push .
+          docker buildx build --platform linux/amd64 $tags --push .

README.md

@@ -41,19 +41,19 @@ ports. Then open ports on the **Resources** page.
 
 ## Deploying
 
-Releases are built by Gitea Actions. Pushing a tag such as `v0.2.0` runs `.gitea/workflows/release.yml`, which tests the code, builds Hammerhead for Linux (amd64, arm64, armv7), Windows (amd64) and macOS (amd64, arm64), publishes the archives with a `SHA256SUMS` file on the Releases page, and pushes a multi-architecture container image to this server's registry (`git.veillax.com/veillax/hammerhead`). Every push and pull request runs `.gitea/workflows/ci.yml`: formatting, vet, tests, and a check that every release platform still compiles.
+Releases are built by Gitea Actions. Pushing a tag such as `v0.2.0` runs `.gitea/workflows/release.yml`, which tests the code, builds Hammerhead for Linux (amd64) and Windows (amd64), publishes the archives with a `SHA256SUMS` file on the Releases page, and pushes a container image (linux/amd64) to this server's registry (`git.veillax.com/veillax/hammerhead`). Every push and pull request runs `.gitea/workflows/ci.yml`: formatting, vet, tests, and a check that every release platform still compiles.
 
 **Fastest: one script on a fresh VPS**
 
-`deploy/setup.sh` (also inside the Linux archives) installs everything for you on a systemd Linux: the service, a generated master key, the firewall rules, and optionally HTTPS.
+`deploy/setup.sh` (also inside the Linux archives) installs everything for you on a systemd Linux: the services, a generated master key, the firewall rules and HTTPS.
 
 ```
-sudo sh setup.sh --domain hammerhead.example.com            # Hammerhead + Caddy for HTTPS
+sudo sh setup.sh --domain hammerhead.example.com            # Hammerhead and an edge on this machine
-sudo sh setup.sh edge  --hammerhead https://hammerhead.example.com --token <edge token> --ports 25565,30000-30100
+sudo sh setup.sh agent --hammerhead https://hammerhead.example.com --token <site token>   # on each site
-sudo sh setup.sh agent --hammerhead https://hammerhead.example.com --token <site token>
+sudo sh setup.sh edge  --hammerhead https://hammerhead.example.com --token <edge token>   # more public servers
 ```
 
-`edge` installs Blacktip (with its bundled Traefik) and lists every IP address on the machine that ports can be opened on; `agent` installs Bamboo on the machine that hosts your services. All three can run on one machine or on three.
+With `--domain`, Hammerhead listens only on `127.0.0.1:8080` and gets an edge of its own (`HAMMERHEAD_SELF_DOMAIN`): the Blacktip on the same machine serves the dashboard at `https://<domain>` with an automatic certificate, so no Caddy or other web server is needed. The route is built in, not a mapping you can edit or delete by mistake. DNS for the name must point at the server and ports 80 and 443 must be free. `--caddy` keeps the old style: Hammerhead alone, with Caddy providing HTTPS. Add `--ports 25565,30000-30100` to open game or mail ports in the firewall, and `--no-auto-update` to leave updates to you.
 
 **On a Linux server (systemd)**
 
@@ -67,7 +67,13 @@ sudoedit /etc/hammerhead/hammerhead.env   # your settings and secrets
 sudo systemctl start hammerhead
 ```
 
-Use `linux_arm64` or `linux_arm` for Raspberry Pi and other ARM machines. To upgrade, repeat the same steps: your settings and data are kept. `sudo ./install.sh --uninstall` removes the service and binary but leaves `/etc/hammerhead` and `/var/lib/hammerhead` alone. `hammerhead -version` tells you which build is running.
+Releases are published for x86-64 only; for ARM machines build it yourself (`TARGETS=linux/arm64 sh deploy/build.sh`). To upgrade, repeat the same steps: your settings and data are kept. `sudo ./install.sh --uninstall` removes the service and binary but leaves `/etc/hammerhead` and `/var/lib/hammerhead` alone. `hammerhead -version` tells you which build is running.
+
+**Updates**
+
+Set `HAMMERHEAD_AUTO_UPDATE=1` (or pass `-auto-update`) and Hammerhead checks this server for a newer release every few hours, verifies it against `SHA256SUMS`, checks that the new binary runs, swaps it in and restarts itself. It only follows full releases (never pre-releases), never downgrades, and keeps the old binary if anything looks wrong. `setup.sh` turns it on. It is not available on Windows, and it is for the systemd and plain-binary installs: with Docker, pull a new image instead.
+
+Edges and sites report their version, so Server → Edges and Sites show which release each one runs and flag a newer one. An **Update** button appears for any machine that runs with auto-update on (`BLACKTIP_AUTO_UPDATE=1` / `BAMBOO_AUTO_UPDATE=1`): it asks that machine to install the newest release now, and the result is shown in the table. Under Server → Settings, **Check for updates** does the same lookup for Hammerhead itself, with an **Update now** button when auto-update is on. A machine without auto-update never installs anything on request. Starting an update opens a progress bar with the live log of what the machine is doing (checking, downloading, verifying, installing, restarting). Hammerhead keeps the log of its own last update in `update.log` in its data folder, so it is still there after the restart.
 
 **With Docker**
 
@@ -165,7 +171,9 @@ for each provider (e.g. `CF_DNS_API_TOKEN` for `cloudflare`); `.env` is the plac
 - Several targets per port: round-robin load balancing with automatic failover.
 - Health checks (TCP connect or an HTTP path); unhealthy targets are skipped and shown in the UI.
 - Conflict detection (same ip/port/protocol, overlapping ranges, or `0.0.0.0` overlapping a specific IP).
-- Traffic totals and a 24-hour sparkline per port.
+- Traffic totals and a 24-hour sparkline per port, plus a **Traffic** chart per resource (and a
+  **Traffic overview** for the whole organization) with bytes in and out, connections and the
+  busiest visitor addresses. See *Traffic analytics* below.
 
 **HTTPS**: automatic Let's Encrypt certificates, optional port-80 redirect, `X-Forwarded-*`
 headers, Host rewrite, HTTPS to the backend, certificate status per port.
@@ -194,11 +202,26 @@ headers, Host rewrite, HTTPS to the backend, certificate status per port.
 **Alerts**
 - Email or webhook (JSON, Slack, Discord, ntfy) when an edge or site goes quiet, a port can't bind, a
   target fails its health check, a certificate errors, or a mail relay isn't deliverable (reverse DNS,
-  HELO, port 25, DKIM record missing).
+  HELO, port 25, DKIM record missing), a sending domain's SPF doesn't authorize the relay or its
+  DMARC record is missing, the relay's IP lands on a DNS blocklist, too much of its mail is being
+  refused, or too little of the mail it sends passes DMARC in the reports you upload. Each alert says
+  what to change.
 - A problem must last the **debounce time** (default 2 minutes, per organization) before you hear about
   it, so restarts and blips stay quiet. One message when it starts, one when it clears, optional
   reminders. Active alerts are stored, so restarting Hammerhead doesn't repeat them, and an alert is
   only marked resolved once the thing it describes has been re-checked.
+- **Sandbar** ([self-hosted notifications, monitors and on-call](https://git.veillax.com/Veillax/sandbar)) can
+  receive alerts in two ways. A *Sandbar* channel raises the alert through its API (and resolves it when the
+  problem clears), so the escalation policy you name pages whoever is on call. A *Sandbar topic* channel
+  publishes a plain message to a topic, which anyone subscribed (the web page, `curl`, the ntfy phone app) sees;
+  the API key is optional there, for servers that don't ask for one. Keys are sealed at rest and never sent back
+  to the browser.
+- **Sandbar monitors** (Server, Settings): turn this on with Sandbar's address and an administrator's API key,
+  and Hammerhead creates a Sandbar monitor for every enabled resource that has health checks: a website is
+  fetched at its public address (a sign-in page counts as up), a TCP port is connected to on the edge's public
+  IP. Each gets uptime history, an optional place on the status page, and the escalation policy you pick.
+  Redirects, UDP, wildcard hostnames and resources with health checks off get none. Hammerhead re-syncs every
+  minute, renames and removes the monitors it created, and never touches ones you made in Sandbar by hand.
 - A dead edge silences its own ports' alerts (the edge alert already says it). Channels are per
   organization; **server channels** get edge alerts.
 - Webhooks can't be pointed at internal addresses (no metadata service, no LAN) unless a server admin
@@ -206,6 +229,24 @@ headers, Host rewrite, HTTPS to the backend, certificate status per port.
 - **`/metrics`** (Prometheus) when `HAMMERHEAD_METRICS_TOKEN` is set: edge/site/port/target/mail-check
   gauges, traffic counters, alerts firing. Off (404) otherwise.
 
+**Traffic analytics**
+- Every edge counts connections and bytes in and out per port, plus the ten busiest client addresses
+  per port, and sends them with its regular report (every 20 seconds). Hammerhead adds them up at
+  three resolutions: per minute (kept 48 hours), per hour (31 days) and, for visitor addresses, per
+  hour (7 days). Older rows are deleted by the same hourly clean-up as the logs.
+- The charts are on the Resources page: **Traffic** on a row, or **Traffic overview** for everything.
+  Pick an hour or 6 hours (per minute), a day, a week or a month (per hour). Visitor addresses cover
+  at most the last 7 days, and very small visitors may be missing because each report names ten.
+- `GET /api/orgs/{org}/traffic/{resource id or "all"}?range=1h|6h|24h|7d|30d` returns the series,
+  totals and busiest addresses as JSON, for scripts or your own dashboards.
+- **Traffic alerts** (Alerts → Traffic, per organization, off by default), delivered like every other
+  alert, with the same debounce:
+  - *Spike*: a port carried at least N times its usual traffic in the last 5 minutes, compared with
+    the average over the previous 6 hours. A port needs an hour of history, and the burst must be over
+    10 MB, so quiet ports don't page you.
+  - *Daily amount*: a port carried more than the set number of MB (in plus out) since midnight UTC.
+  Neither blocks traffic; they only tell you.
+
 **Mail egress (send your mail server's outgoing mail from the edge's IP)**
 - Receiving servers judge mail by the IP it comes from: its reverse DNS, your HELO name and your SPF
   record all have to agree. A mail server at home usually can't satisfy that (residential IP, port 25
@@ -221,6 +262,20 @@ headers, Host rewrite, HTTPS to the backend, certificate status per port.
   record it shows. Mail is signed by From domain (a parent domain's key covers subdomains); a message
   your mail server already signed for that domain is left alone, and a signing problem sends the mail
   unsigned rather than dropping it. Hammerhead checks the DNS record and alerts if it's missing or wrong.
+- **Deliverability dashboard.** The Mail egress page shows a checklist per relay, each line with a fix:
+  reverse DNS, HELO, outbound port 25, every DKIM key, and for each domain you sign mail for, SPF
+  (evaluated against the relay's IP, including `include:`/`redirect=` and the 10-lookup limit), DMARC
+  (missing, monitoring-only, no `rua`, enforcing) and MTA-STS. The edge also looks the IP up on DNS
+  blocklists (Spamhaus ZEN, SpamCop, Barracuda, PSBL) and counts the last 24 hours of delivery outcomes
+  (delivered, deferred, refused, by recipient domain, with the latest refusal messages and what they
+  usually mean). Checks run every 15 minutes; failures raise alerts and clear when fixed.
+- **DMARC reports.** Put `rua=mailto:...` in a domain's DMARC record, and upload the daily aggregate
+  reports receivers send (the email itself, or its .xml/.gz/.zip attachment, several at once) on the same
+  page. Hammerhead parses them, keeps 90 days, and shows per domain how much mail passed, which sending
+  addresses fail (marking your own relays), and whether the policy can move from `none` to
+  `quarantine`/`reject`. `POST /api/orgs/<org>/dmarc` takes a report as the raw request body, so a mail
+  filter or script can forward reports automatically. Reports are only accepted for domains you have a
+  DKIM key for.
 - **MTA-STS enforcement.** When a recipient domain publishes an `enforce` policy, mail only goes to the
   servers it lists, over TLS with a verified certificate; otherwise it's a temporary failure so your
   server retries. (`testing` mode doesn't block.) Policies are cached for their `max_age`. The policy
@@ -253,6 +308,23 @@ Blacktip redirects an unauthenticated visitor to Hammerhead's login page. Hammer
 token with a per-edge key and sends the visitor back; Blacktip verifies it and sets a cookie. The same
 decision function runs in both proxy modes.
 
+## Ephemeral exposures (`shark expose`)
+
+A short-lived port or website for development, created and removed by an API key. [shark-expose](https://git.veillax.com/Veillax/shark-expose) is the command-line client; the API is plain JSON, with an API key (**Access**) as the bearer token. `-` can stand for the key's own organization in the path.
+
+| Call | Does |
+|---|---|
+| `POST /api/orgs/-/expose` | `{proto: "http"\|"https"\|"tcp", targetPort, targetHost?, name?, ttlSeconds?, port?, hostname?, edgeId?, auth?: {password?, pin?, sso?}}`. Makes a site and a port on an online edge, and returns the public address plus `siteToken`, the token Bamboo dials in with. |
+| `POST /api/orgs/-/expose/{id}/renew` | Pushes the deadline out by `ttlSeconds` (default 300) from now. |
+| `GET /api/orgs/-/expose[/{id}]` | Lists exposures, with whether the edge reports the port listening. |
+| `DELETE /api/orgs/-/expose/{id}` | Removes it now. Stopping twice is fine. |
+
+TCP gets a free port from 20000-29999. A website gets `<name>-<random>.<one of your domains>` (so a wildcard DNS record for that domain should point at the edge) on port 80/443, or on `port`. An exposure that isn't renewed within its TTL (30 seconds to 24 hours) is removed, along with its site and any login secrets. The dashboard shows them with the `ephemeral` tag; the deadlines live in memory, so a Hammerhead restart removes any that are still there. An organization can have 20 at a time.
+
+## Design
+
+The dashboard follows the Lightwell design language: warm neutrals, one teal accent, system fonts, status badges with text, and dialogs that name what they affect. See [docs/design.md](docs/design.md). `node scripts/lint-design.mjs` checks the rules that a machine can check.
+
 ## Limits and notes
 
 - No client-to-site VPN (Pangolin's Olm/private resources) and no blueprints/YAML import.

accounts.go

@@ -54,6 +54,9 @@ func (s *Server) principalFrom(r *http.Request) *principal {
 				if k.ID == parts[1] && hmac.Equal([]byte(k.Hash), []byte(sha(parts[2]))) {
 					k.LastUsed = time.Now()
 					kc := *k
+					if k.OrgID == "" { // a server key acts as a server administrator
+						return &principal{Key: &kc, User: User{Email: "apikey:" + k.Name, ServerAdmin: true}}
+					}
 					return &principal{Key: &kc}
 				}
 			}
@@ -66,21 +69,30 @@ func (s *Server) withUser(fn func(http.ResponseWriter, *http.Request, *principal
 	return func(w http.ResponseWriter, r *http.Request) {
 		p := s.principalFrom(r)
 		if p == nil || p.User.ID == "" {
-			fail(w, 401, errors.New("not logged in"))
+			fail(w, 401, errors.New("Sign in to continue."))
 			return
 		}
 		fn(w, r, p)
 	}
 }
 
+// serverKey reports whether the caller authenticated with a server-level API key.
+func (p *principal) serverKey() bool { return p.Key != nil && p.Key.OrgID == "" }
+
+// withServerAdmin lets signed-in server administrators and server API keys through.
 func (s *Server) withServerAdmin(fn func(http.ResponseWriter, *http.Request, *principal)) http.HandlerFunc {
-	return s.withUser(func(w http.ResponseWriter, r *http.Request, p *principal) {
+	return func(w http.ResponseWriter, r *http.Request) {
+		p := s.principalFrom(r)
+		if p == nil || (p.User.ID == "" && !p.serverKey()) {
+			fail(w, 401, errors.New("Sign in to continue."))
+			return
+		}
 		if !p.User.ServerAdmin {
 			fail(w, 403, errors.New("server administrators only"))
 			return
 		}
 		fn(w, r, p)
-	})
+	}
 }
 
 // orgCtx is the caller's standing inside one org.
@@ -96,10 +108,13 @@ func (s *Server) withOrg(needAdmin bool, fn func(http.ResponseWriter, *http.Requ
 	return func(w http.ResponseWriter, r *http.Request) {
 		p := s.principalFrom(r)
 		if p == nil {
-			fail(w, 401, errors.New("not logged in"))
+			fail(w, 401, errors.New("Sign in to continue."))
 			return
 		}
 		id := r.PathValue("org")
+		if id == "-" && p.Key != nil {
+			id = p.Key.OrgID // an API key belongs to one organization, so "-" can stand for it
+		}
 		s.mu.Lock()
 		org := s.findOrg(id)
 		c := &orgCtx{User: p.User, Key: p.Key}
@@ -107,6 +122,9 @@ func (s *Server) withOrg(needAdmin bool, fn func(http.ResponseWriter, *http.Requ
 		if org != nil {
 			c.Org = *org
 			switch {
+			case p.serverKey():
+				allowed, c.Admin = true, true
+				c.Actor = "apikey:" + p.Key.Name
 			case p.Key != nil:
 				allowed = p.Key.OrgID == org.ID
 				c.Admin = allowed
@@ -169,7 +187,7 @@ func (s *Server) createFirstAdmin(email, name, password, orgName string) error {
 	s.mu.Lock()
 	defer s.mu.Unlock()
 	if len(s.st.Users) != 0 {
-		return errors.New("setup already done")
+		return errors.New("Hammerhead is already set up. Sign in instead.")
 	}
 	u := &User{ID: newID(), Email: email, Name: name, PassHash: hash, ServerAdmin: true, Created: time.Now()}
 	s.st.Users = append(s.st.Users, u)
@@ -240,7 +258,7 @@ func (s *Server) login(w http.ResponseWriter, r *http.Request) {
 	}
 	email := strings.ToLower(strings.TrimSpace(in.Email))
 	if !s.allow("login|"+clientIP(r)+"|"+email, 8, 5*time.Minute) {
-		fail(w, 429, errors.New("too many attempts, wait a few minutes"))
+		fail(w, 429, errors.New("Too many attempts. Wait a few minutes and try again."))
 		return
 	}
 	s.mu.Lock()
@@ -254,7 +272,7 @@ func (s *Server) login(w http.ResponseWriter, r *http.Request) {
 	if u == nil || !good || u.PassHash == "" {
 		s.audit("", email, "login.failed", clientIP(r))
 		time.Sleep(300 * time.Millisecond)
-		fail(w, 401, errors.New("wrong email or password"))
+		fail(w, 401, errors.New("That email and password do not match. Check them and try again."))
 		return
 	}
 	if u.TOTPEnabled {
@@ -276,7 +294,7 @@ func (s *Server) loginTOTP(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 	if !s.allow("totp|"+clientIP(r), 10, 5*time.Minute) {
-		fail(w, 429, errors.New("too many attempts, wait a few minutes"))
+		fail(w, 429, errors.New("Too many attempts. Wait a few minutes and try again."))
 		return
 	}
 	s.mu.Lock()
@@ -307,7 +325,7 @@ func (s *Server) loginTOTP(w http.ResponseWriter, r *http.Request) {
 	}
 	s.mu.Unlock()
 	if !good {
-		fail(w, 401, errors.New("wrong code"))
+		fail(w, 401, errors.New("That code is not right. Check your authenticator and try again."))
 		return
 	}
 	s.startSession(w, r, u.ID)
@@ -413,7 +431,7 @@ func (s *Server) changePassword(w http.ResponseWriter, r *http.Request, p *princ
 		return
 	}
 	if p.User.PassHash != "" && bcrypt.CompareHashAndPassword([]byte(p.User.PassHash), []byte(in.Current)) != nil {
-		fail(w, 400, errors.New("current password is wrong"))
+		fail(w, 400, errors.New("Your current password is not right. Check it and try again."))
 		return
 	}
 	hash, err := hashPassword(in.New)
@@ -464,7 +482,7 @@ func (s *Server) totpEnable(w http.ResponseWriter, r *http.Request, p *principal
 	defer s.mu.Unlock()
 	u := s.findUser(p.User.ID)
 	if u.TOTPSecret == "" || !totpOK(u.TOTPSecret, strings.TrimSpace(in.Code)) {
-		fail(w, 400, errors.New("that code doesn't match; check your authenticator's clock"))
+		fail(w, 400, errors.New("That code does not match. Check that your authenticator's clock is right and try again."))
 		return
 	}
 	codes, hashes := make([]string, 8), make([]string, 8)
@@ -483,14 +501,14 @@ func (s *Server) totpDisable(w http.ResponseWriter, r *http.Request, p *principa
 		return
 	}
 	if p.User.PassHash != "" && bcrypt.CompareHashAndPassword([]byte(p.User.PassHash), []byte(in.Password)) != nil {
-		fail(w, 400, errors.New("password is wrong"))
+		fail(w, 400, errors.New("That password is not right. Check it and try again."))
 		return
 	}
 	s.mu.Lock()
 	defer s.mu.Unlock()
 	u := s.findUser(p.User.ID)
 	if !totpOK(u.TOTPSecret, strings.TrimSpace(in.Code)) {
-		fail(w, 400, errors.New("code is wrong"))
+		fail(w, 400, errors.New("That code is not right. Check your authenticator and try again."))
 		return
 	}
 	u.TOTPEnabled, u.TOTPSecret, u.BackupCodes = false, "", nil

agent.go

@@ -73,6 +73,10 @@ func (s *Server) edgeConfig(w http.ResponseWriter, r *http.Request) {
 			cfg.Mappings = append(cfg.Mappings, *m)
 		}
 	}
+	if m := s.selfMapping(e); m != nil {
+		cfg.Mappings = append(cfg.Mappings, *m)
+	}
+	cfg.UpdateID = e.UpdateID
 	cfg.MailEgress = []model.MailPolicy{}
 	for _, x := range s.st.MailEgress {
 		if x.EdgeID == e.ID && x.Enabled {
@@ -107,6 +111,9 @@ func (s *Server) edgeReport(w http.ResponseWriter, r *http.Request) {
 	}
 	e.LastSeen = time.Now()
 	e.IPs = in.IPs
+	oldVersion := e.Version
+	answered := noteUpdateReport(&e.Version, &e.AutoUpdate, &e.UpdateID, &e.UpdateMsg, in.Version, in.AutoUpdate, in.UpdateID, in.UpdateMsg)
+	s.recordUpdateProgress(e.ID, in.UpdateID, in.UpdateStage, in.UpdatePct, in.UpdateLog, oldVersion, e.Version)
 	for _, x := range s.st.MailEgress {
 		if x.EdgeID == e.ID {
 			if chk, found := in.MailChecks[x.ID]; found {
@@ -143,6 +150,8 @@ func (s *Server) edgeReport(w http.ResponseWriter, r *http.Request) {
 	if e.PublicKey != in.PublicKey {
 		e.PublicKey = in.PublicKey
 		s.bump() // sites need the edge's new key
+	} else if answered {
+		s.bump() // the edge's config no longer carries the update request
 	} else {
 		s.save()
 	}
@@ -202,7 +211,7 @@ func (s *Server) siteConfig(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 	st.LastSeen = time.Now()
-	cfg := model.SiteConfig{Version: s.st.Version, SiteID: st.ID, TunnelIP: st.TunnelIP, Edges: []model.EdgeRef{}, MailEgress: []model.MailSpec{}}
+	cfg := model.SiteConfig{Version: s.st.Version, SiteID: st.ID, TunnelIP: st.TunnelIP, Edges: []model.EdgeRef{}, MailEgress: []model.MailSpec{}, UpdateID: st.UpdateID}
 	seen := map[string]bool{}
 	addEdge := func(id string) {
 		if seen[id] {
@@ -246,7 +255,11 @@ func (s *Server) siteReport(w http.ResponseWriter, r *http.Request) {
 		in.Connected = []string{}
 	}
 	st.Connected = in.Connected
+	oldVersion := st.Version
+	answered := noteUpdateReport(&st.Version, &st.AutoUpdate, &st.UpdateID, &st.UpdateMsg, in.Version, in.AutoUpdate, in.UpdateID, in.UpdateMsg)
+	s.recordUpdateProgress(st.ID, in.UpdateID, in.UpdateStage, in.UpdatePct, in.UpdateLog, oldVersion, st.Version)
 	s.containers[st.ID] = in.Containers
+	s.discovered[st.ID] = cleanDiscovered(in.Discovered)
 	for _, x := range s.st.MailEgress {
 		if x.SiteID != st.ID {
 			continue
@@ -260,6 +273,8 @@ func (s *Server) siteReport(w http.ResponseWriter, r *http.Request) {
 	if in.PublicKey != "" && st.PublicKey != in.PublicKey {
 		st.PublicKey = in.PublicKey
 		s.bump() // every edge this site talks to needs the new key
+	} else if answered {
+		s.bump() // the site's config no longer carries the update request
 	}
 	ok(w)
 }

alerts.go

@@ -8,6 +8,7 @@ import (
 	"log"
 	"net/http"
 	"net/url"
+	"regexp"
 	"sort"
 	"strings"
 	"time"
@@ -23,19 +24,26 @@ import (
 // message when it starts and one when it clears, and optionally a reminder.
 // Active alerts are kept in the database so a restart doesn't repeat them.
 
+var topicName = regexp.MustCompile(`^[A-Za-z0-9_-]{1,64}$`)
+
 const (
 	defaultDebounce = 2 * time.Minute
 	defaultOffline  = 90 * time.Second
 )
 
-// AlertChannel is somewhere alerts are sent: an email address or a webhook.
+// AlertChannel is somewhere alerts are sent: an email address, a webhook, a
+// Sandbar server (which pages whoever is on call and tracks the alert), or a
+// Sandbar topic (a plain message for everyone subscribed to it).
 type AlertChannel struct {
 	ID           string `json:"id"`
 	OrgID        string `json:"orgId"` // "" = a server-level channel (for edges)
 	Name         string `json:"name"`
-	Kind         string `json:"kind"`   // "email" or "webhook"
+	Kind         string `json:"kind"`   // "email", "webhook", "sandbar" or "topic"
-	Target       string `json:"target"` // address(es), or a webhook URL (secret: URLs often carry tokens)
+	Target       string `json:"target"` // address(es), a webhook URL, or Sandbar's address (secret: URLs often carry tokens)
 	Format       string `json:"format"` // webhook body: "json", "slack", "discord" or "ntfy"
+	Token        string `json:"token"`  // Sandbar: API key (secret, never sent back to the browser; optional for a topic)
+	Topic        string `json:"topic"`  // topic: the topic to publish to
+	Policy       string `json:"policy"` // Sandbar: escalation policy id; blank sends the alert without paging anyone
 	AllowPrivate bool   `json:"allowPrivate"`
 	Enabled      bool   `json:"enabled"`
 }
@@ -50,6 +58,7 @@ type alertState struct {
 type problem struct{ org, kind, name, msg string }
 
 type alertNote struct {
+	Key     string    `json:"key"`    // stable per problem, so a recovery can find its alert
 	Status  string    `json:"status"` // "firing", "resolved" or "test"
 	Org     string    `json:"org"`
 	Kind    string    `json:"kind"`
@@ -123,6 +132,7 @@ func (s *Server) problemsLocked(now time.Time) map[string]problem {
 			}
 		}
 	}
+	s.trafficProblemsLocked(now, edgeDown, add)
 	for _, x := range s.st.MailEgress {
 		if !x.Enabled || edgeDown[x.EdgeID] || siteDown[x.SiteID] {
 			continue
@@ -142,6 +152,27 @@ func (s *Server) problemsLocked(now time.Time) map[string]problem {
 			}
 		}
 	}
+	for _, x := range s.st.MailEgress {
+		if !x.Enabled || edgeDown[x.EdgeID] || siteDown[x.SiteID] {
+			continue
+		}
+		chk, ok := s.mailChecks[x.ID]
+		if !ok {
+			continue
+		}
+		for _, f := range s.mailEgressFindingsLocked(x, chk) {
+			// ptr, helo, port 25 and dkim have alerts of their own above
+			if !f.alertWorthy() || f.ID == "ptr" || f.ID == "helo" || f.ID == "port25" || strings.HasPrefix(f.ID, "dkim:") {
+				continue
+			}
+			msg := fmt.Sprintf("%s: %s: %s", x.Name, f.Label, f.Detail)
+			if f.Hint != "" {
+				msg += ". " + f.Hint
+			}
+			add("mailfind:"+x.ID+":"+f.ID, x.OrgID, "mail", x.Name, msg)
+		}
+	}
+	s.dmarcProblemsLocked(add)
 	for _, k := range s.st.DKIM {
 		for _, x := range s.st.MailEgress {
 			if x.OrgID != k.OrgID || !x.Enabled || edgeDown[x.EdgeID] {
@@ -181,6 +212,13 @@ func (s *Server) judgeableLocked(st *alertState) bool {
 		}
 		_, known := s.mailStatus[rest]
 		return known
+	case "mailfind":
+		id, _, _ := strings.Cut(rest, ":")
+		if s.findMailEgress(id) == nil {
+			return true
+		}
+		_, known := s.mailChecks[id]
+		return known
 	case "mailptr", "mailhelo", "mail25":
 		if s.findMailEgress(rest) == nil {
 			return true
@@ -251,7 +289,7 @@ func (s *Server) evaluateAlerts(now time.Time) {
 				chans = append(chans, *c)
 			}
 		}
-		out = append(out, delivery{alertNote{Status: status, Org: orgName(st.OrgID), Kind: st.Kind, Name: st.Name, Message: st.Message, Since: st.Since, Time: now}, chans})
+		out = append(out, delivery{alertNote{Key: st.Key, Status: status, Org: orgName(st.OrgID), Kind: st.Kind, Name: st.Name, Message: st.Message, Since: st.Since, Time: now}, chans})
 		s.db.addAlertEvent(now, st.OrgID, st.Key, status, st.Name, st.Message)
 	}
 	keys := make([]string, 0, len(probs))
@@ -362,6 +400,10 @@ func (s *Server) send(ch AlertChannel, n alertNote) error {
 			}
 		}
 		return errors.Join(errs...)
+	case "sandbar":
+		return s.sendSandbar(ch, n)
+	case "topic":
+		return s.sendTopic(ch, n)
 	case "webhook":
 		body, ctype, hdr := webhookBody(ch.Format, n)
 		req, err := http.NewRequest("POST", ch.Target, bytes.NewReader(body))
@@ -385,6 +427,79 @@ func (s *Server) send(ch AlertChannel, n alertNote) error {
 	return errors.New("unknown channel kind")
 }
 
+// sendSandbar raises the alert on a Sandbar server, or resolves it there when the
+// problem clears. Sandbar de-duplicates by key, so a reminder updates the alert
+// that is already open instead of opening another. A test raises an info alert
+// with no policy (nobody is paged) and resolves it straight away.
+func (s *Server) sendSandbar(ch AlertChannel, n alertNote) error {
+	post := func(body map[string]string) error {
+		raw, _ := json.Marshal(body)
+		req, err := http.NewRequest("POST", strings.TrimRight(ch.Target, "/")+"/api/v1/alerts", bytes.NewReader(raw))
+		if err != nil {
+			return err
+		}
+		req.Header.Set("Content-Type", "application/json")
+		req.Header.Set("Authorization", "Bearer "+ch.Token)
+		resp, err := netguard.Client(ch.AllowPrivate).Do(req)
+		if err != nil {
+			return err
+		}
+		defer resp.Body.Close()
+		switch {
+		case resp.StatusCode == 401 || resp.StatusCode == 403:
+			return errors.New("Sandbar rejected the API key")
+		case resp.StatusCode >= 300:
+			return fmt.Errorf("Sandbar answered %s", resp.Status)
+		}
+		return nil
+	}
+	key := "hammerhead:" + n.Key
+	switch n.Status {
+	case "resolved":
+		return post(map[string]string{"key": key, "status": "resolved", "source": "hammerhead"})
+	case "test":
+		key = "hammerhead:test:" + ch.ID
+		if err := post(map[string]string{"key": key, "source": "hammerhead", "summary": "Hammerhead test alert", "detail": n.text(), "severity": "info"}); err != nil {
+			return err
+		}
+		return post(map[string]string{"key": key, "status": "resolved", "source": "hammerhead"})
+	}
+	return post(map[string]string{"key": key, "source": "hammerhead", "summary": n.Name + ": " + n.Message,
+		"detail": "Organization: " + n.Org + "\nSince: " + n.Since.Format(time.RFC1123), "severity": "critical", "policy": ch.Policy})
+}
+
+// sendTopic publishes the alert as a message on a Sandbar topic (ntfy's request
+// works too), so everyone subscribed, including the ntfy phone apps, sees it. A
+// topic has no alert to close, so a recovery is a second, calmer message.
+func (s *Server) sendTopic(ch AlertChannel, n alertNote) error {
+	prio, tags := "urgent", "rotating_light"
+	if n.Status != "firing" {
+		prio, tags = "default", "white_check_mark"
+	}
+	req, err := http.NewRequest("POST", strings.TrimRight(ch.Target, "/")+"/"+ch.Topic, strings.NewReader(n.text()+"\n\nOrganization: "+n.Org))
+	if err != nil {
+		return err
+	}
+	req.Header.Set("Title", n.subject())
+	req.Header.Set("Priority", prio)
+	req.Header.Set("Tags", tags)
+	if ch.Token != "" {
+		req.Header.Set("Authorization", "Bearer "+ch.Token)
+	}
+	resp, err := netguard.Client(ch.AllowPrivate).Do(req)
+	if err != nil {
+		return err
+	}
+	defer resp.Body.Close()
+	switch {
+	case resp.StatusCode == 401 || resp.StatusCode == 403:
+		return errors.New("Sandbar rejected the API key")
+	case resp.StatusCode >= 300:
+		return fmt.Errorf("Sandbar answered %s", resp.Status)
+	}
+	return nil
+}
+
 func webhookBody(format string, n alertNote) (body []byte, ctype string, hdr map[string]string) {
 	text := n.subject() + "\n" + n.text()
 	switch format {
@@ -415,12 +530,37 @@ func (s *Server) validateChannel(ch *AlertChannel, trusted bool) error {
 				return errors.New("enter one or more email addresses, separated by commas")
 			}
 		}
-		ch.Format, ch.AllowPrivate = "", false
+		ch.Format, ch.Token, ch.Policy, ch.Topic, ch.AllowPrivate = "", "", "", "", false
+	case "sandbar":
+		u, err := url.Parse(ch.Target)
+		if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
+			return errors.New("enter Sandbar's address, such as https://sandbar.example.com")
+		}
+		if ch.Token = strings.TrimSpace(ch.Token); ch.Token == "" {
+			return errors.New("enter an API key from Sandbar's Account page")
+		}
+		ch.Policy, ch.Format, ch.Topic = strings.TrimSpace(ch.Policy), "", ""
+		if ch.AllowPrivate && !trusted {
+			return errors.New("sending to private addresses (like a service on your home network) needs a server administrator to allow it for this organization")
+		}
+	case "topic":
+		u, err := url.Parse(ch.Target)
+		if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
+			return errors.New("enter Sandbar's address, such as https://sandbar.example.com")
+		}
+		if ch.Topic = strings.TrimSpace(ch.Topic); !topicName.MatchString(ch.Topic) {
+			return errors.New("a topic is 1 to 64 letters, digits, - or _")
+		}
+		ch.Token, ch.Policy, ch.Format = strings.TrimSpace(ch.Token), "", ""
+		if ch.AllowPrivate && !trusted {
+			return errors.New("sending to private addresses (like a service on your home network) needs a server administrator to allow it for this organization")
+		}
 	case "webhook":
 		u, err := url.Parse(ch.Target)
 		if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
 			return errors.New("the webhook must be an http:// or https:// URL")
 		}
+		ch.Token, ch.Policy, ch.Topic = "", "", ""
 		switch ch.Format {
 		case "json", "slack", "discord", "ntfy":
 		case "":
@@ -432,7 +572,7 @@ func (s *Server) validateChannel(ch *AlertChannel, trusted bool) error {
 			return errors.New("sending to private addresses (like a service on your home network) needs a server administrator to allow it for this organization")
 		}
 	default:
-		return errors.New("a channel is an email or a webhook")
+		return errors.New("a channel is an email, a webhook, a Sandbar server or a Sandbar topic")
 	}
 	if ch.Name == "" {
 		ch.Name = ch.Kind

alerts_api.go

@@ -23,9 +23,12 @@ type channelView struct {
 }
 
 func preview(ch AlertChannel) string {
-	if ch.Kind != "webhook" {
+	if ch.Kind == "email" {
 		return ch.Target
 	}
+	if u, err := url.Parse(ch.Target); err == nil && ch.Kind == "topic" {
+		return u.Scheme + "://" + u.Host + "/" + ch.Topic
+	}
 	if u, err := url.Parse(ch.Target); err == nil {
 		return u.Scheme + "://" + u.Host + "/…"
 	}
@@ -47,7 +50,7 @@ func (s *Server) alertOverview(w http.ResponseWriter, sc alertScope) {
 	for _, c := range s.st.Channels {
 		if c.OrgID == sc.org {
 			v := channelView{AlertChannel: *c, Preview: preview(*c)}
-			v.Target = "" // webhook URLs often contain tokens
+			v.Target, v.Token = "", "" // webhook URLs often contain tokens
 			if c.Kind == "email" {
 				v.Target = c.Target
 			}
@@ -61,12 +64,13 @@ func (s *Server) alertOverview(w http.ResponseWriter, sc alertScope) {
 		}
 	}
 	sort.Slice(active, func(i, j int) bool { return active[i].Since.Before(active[j].Since) })
-	settings := map[string]int{"debounceMin": 2, "repeatHours": 0}
+	settings := map[string]int{"debounceMin": 2, "repeatHours": 0, "spikeFactor": 0, "dailyMb": 0}
 	if o := s.findOrg(sc.org); o != nil {
 		if o.AlertDebounceMin > 0 {
 			settings["debounceMin"] = o.AlertDebounceMin
 		}
 		settings["repeatHours"] = o.AlertRepeatHours
+		settings["spikeFactor"], settings["dailyMb"] = o.TrafficSpikeFactor, o.TrafficDailyMB
 	}
 	mailOK := s.st.Settings.SMTP.Host != ""
 	s.mu.Unlock()
@@ -95,6 +99,9 @@ func (s *Server) saveChannel(w http.ResponseWriter, r *http.Request, sc alertSco
 		if in.Target == "" {
 			in.Target = existing.Target // blank keeps the saved URL/address
 		}
+		if id != "" && in.Token == "" && (in.Kind == "sandbar" || in.Kind == "topic") {
+			in.Token = existing.Token // blank keeps the saved API key
+		}
 	}
 	trusted := sc.trusted
 	if err := s.validateChannel(&in, trusted); err != nil {
@@ -111,7 +118,7 @@ func (s *Server) saveChannel(w http.ResponseWriter, r *http.Request, sc alertSco
 	}
 	s.save()
 	s.audit(sc.org, sc.actor, "alert.channel.save", in.Name)
-	ok(w)
+	writeJSON(w, 200, map[string]any{"ok": true, "id": in.ID})
 }
 
 func (s *Server) deleteChannel(w http.ResponseWriter, r *http.Request, sc alertScope) {
@@ -156,7 +163,7 @@ func (s *Server) alertSettings(w http.ResponseWriter, r *http.Request, sc alertS
 		return
 	}
 	if in.DebounceMin < 1 || in.DebounceMin > 120 || in.RepeatHours < 0 || in.RepeatHours > 168 {
-		fail(w, 400, errors.New("wait 1-120 minutes before alerting; remind every 0-168 hours (0 = never)"))
+		fail(w, 400, errors.New("Wait 1 to 120 minutes before alerting, and remind every 0 to 168 hours (0 means no reminders)."))
 		return
 	}
 	s.mu.Lock()
@@ -195,6 +202,7 @@ func (s *Server) registerAlertRoutes(mux *http.ServeMux) {
 	mux.HandleFunc("DELETE /api/orgs/{org}/alerts/channels/{id}", s.withOrgAlerts(s.deleteChannel))
 	mux.HandleFunc("POST /api/orgs/{org}/alerts/channels/{id}/test", s.withOrgAlerts(s.testChannel))
 	mux.HandleFunc("PUT /api/orgs/{org}/alerts/settings", s.withOrgAlerts(s.alertSettings))
+	mux.HandleFunc("PUT /api/orgs/{org}/alerts/traffic", s.withOrgAlerts(s.trafficAlertSettings))
 	mux.HandleFunc("GET /api/admin/alerts", s.withServerAlerts(overview))
 	mux.HandleFunc("POST /api/admin/alerts/channels", s.withServerAlerts(s.saveChannel))
 	mux.HandleFunc("PUT /api/admin/alerts/channels/{id}", s.withServerAlerts(s.saveChannel))

alerts_test.go

@@ -10,8 +10,8 @@ import (
 	"testing"
 	"time"
 
-	"git.veillax.com/Veillax/hammerhead/internal/secretbox"
 	"git.veillax.com/Veillax/reef/model"
+	"git.veillax.com/Veillax/reef/secretbox"
 )
 
 type hook struct {
@@ -487,3 +487,57 @@ func TestPromEscape(t *testing.T) {
 		t.Errorf("promEscape = %q", got)
 	}
 }
+
+func TestSandbarChannelRaisesAndResolves(t *testing.T) {
+	f := newFx(t)
+	h := newHook(t)
+	f.s.mu.Lock()
+	f.s.st.Channels = []*AlertChannel{{ID: "c2", OrgID: f.org, Name: "pager", Kind: "sandbar", Target: strings.TrimSuffix(h.URL, "/hook/secret-token"),
+		Token: "sb_secret", Policy: "pol1", AllowPrivate: true, Enabled: true}}
+	f.s.mu.Unlock()
+
+	f.eval(f.base)
+	f.setHealthy(false)
+	f.eval(f.base.Add(10 * time.Second))
+	f.eval(f.base.Add(3 * time.Minute))
+	var up map[string]string
+	if err := json.Unmarshal([]byte(h.next(2*time.Second)), &up); err != nil {
+		t.Fatal("no Sandbar request:", err)
+	}
+	if up["key"] != "hammerhead:target:m1:0" || up["policy"] != "pol1" || up["severity"] != "critical" || !strings.Contains(up["summary"], "Minecraft") {
+		t.Fatalf("unexpected alert sent to Sandbar: %v", up)
+	}
+	h.mu.Lock()
+	auth := h.hdrs[0].Get("Authorization")
+	h.mu.Unlock()
+	if auth != "Bearer sb_secret" {
+		t.Fatalf("API key not sent: %q", auth)
+	}
+
+	f.setHealthy(true)
+	f.eval(f.base.Add(4 * time.Minute))
+	var down map[string]string
+	if err := json.Unmarshal([]byte(h.next(2*time.Second)), &down); err != nil {
+		t.Fatal("no resolve request:", err)
+	}
+	if down["key"] != up["key"] || down["status"] != "resolved" {
+		t.Fatalf("recovery did not resolve the same alert: %v", down)
+	}
+}
+
+func TestSandbarChannelValidation(t *testing.T) {
+	f := newFx(t)
+	for _, ch := range []AlertChannel{
+		{Kind: "sandbar", Target: "not a url", Token: "k"},
+		{Kind: "sandbar", Target: "https://sandbar.example.com"}, // no API key
+	} {
+		ch := ch
+		if err := f.s.validateChannel(&ch, true); err == nil {
+			t.Fatalf("accepted %+v", ch)
+		}
+	}
+	ok := AlertChannel{Kind: "sandbar", Target: "https://sandbar.example.com", Token: " k ", Policy: " p "}
+	if err := f.s.validateChannel(&ok, true); err != nil || ok.Token != "k" || ok.Policy != "p" {
+		t.Fatalf("%v %+v", err, ok)
+	}
+}

apikeys_test.go

@@ -0,0 +1,83 @@
+package hammerhead
+
+import (
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+)
+
+func keyCall(t *testing.T, s *Server, key, method, path, body string) (int, map[string]any) {
+	t.Helper()
+	req := httptest.NewRequest(method, path, strings.NewReader(body))
+	if key != "" {
+		req.Header.Set("Authorization", "Bearer "+key)
+	}
+	rec := httptest.NewRecorder()
+	s.Handler().ServeHTTP(rec, req)
+	var out map[string]any
+	json.Unmarshal(rec.Body.Bytes(), &out)
+	return rec.Code, out
+}
+
+func mintKey(t *testing.T, s *Server, org string) string {
+	t.Helper()
+	k := "hhk_" + randHex(4) + "_" + randHex(24)
+	parts := strings.SplitN(k, "_", 3)
+	s.mu.Lock()
+	s.st.APIKeys = append(s.st.APIKeys, &APIKey{ID: parts[1], OrgID: org, Name: "t", Hash: sha(parts[2])})
+	s.mu.Unlock()
+	return k
+}
+
+func TestServerKeysManageEdgesAndOrgs(t *testing.T) {
+	f := newFx(t)
+	server := mintKey(t, f.s, "")
+	orgKey := mintKey(t, f.s, f.org)
+
+	if c, _ := keyCall(t, f.s, server, "POST", "/api/edges", `{"name":"e2","tunnelAddr":"1.2.3.4:51820"}`); c != 200 {
+		t.Fatalf("server key should create an edge, got %d", c)
+	}
+	if c, _ := keyCall(t, f.s, orgKey, "POST", "/api/edges", `{"name":"e3","tunnelAddr":"1.2.3.4:51820"}`); c != 401 {
+		t.Fatalf("org key must not create edges, got %d", c)
+	}
+	if c, _ := keyCall(t, f.s, server, "POST", "/api/orgs", `{"name":"Second"}`); c != 200 {
+		t.Fatalf("server key should create an org, got %d", c)
+	}
+	// server key reaches any org's endpoints; an org key only its own
+	if c, _ := keyCall(t, f.s, server, "POST", "/api/orgs/"+f.org+"/sites", `{"name":"x"}`); c != 200 {
+		t.Fatalf("server key on org: %d", c)
+	}
+	if c, _ := keyCall(t, f.s, orgKey, "POST", "/api/orgs/nope/sites", `{"name":"x"}`); c != 404 {
+		t.Fatalf("org key on other org: %d", c)
+	}
+	// server alert channels
+	if c, _ := keyCall(t, f.s, server, "GET", "/api/admin/alerts", ""); c != 200 {
+		t.Fatalf("server key alerts: %d", c)
+	}
+	// keys can't mint keys at server level
+	if c, _ := keyCall(t, f.s, server, "POST", "/api/admin/apikeys", `{"name":"k"}`); c != 401 {
+		t.Fatalf("server key must not mint server keys, got %d", c)
+	}
+	if c, _ := keyCall(t, f.s, "", "GET", "/api/orgs", ""); c != 401 {
+		t.Fatalf("anonymous list orgs: %d", c)
+	}
+	req := httptest.NewRequest("GET", "/api/orgs", nil)
+	req.Header.Set("Authorization", "Bearer "+orgKey)
+	rec := httptest.NewRecorder()
+	f.s.Handler().ServeHTTP(rec, req)
+	var orgs []Org
+	json.Unmarshal(rec.Body.Bytes(), &orgs)
+	if rec.Code != http.StatusOK || len(orgs) != 1 || orgs[0].ID != f.org {
+		t.Fatalf("org key should list only its org: %d %s", rec.Code, rec.Body)
+	}
+	// org key lists edges its org may use
+	req = httptest.NewRequest("GET", "/api/edges", nil)
+	req.Header.Set("Authorization", "Bearer "+orgKey)
+	rec = httptest.NewRecorder()
+	f.s.Handler().ServeHTTP(rec, req)
+	if rec.Code != 200 || !strings.Contains(rec.Body.String(), "edge-1") {
+		t.Fatalf("org key edges: %d %s", rec.Code, rec.Body)
+	}
+}

cmd/hammerhead/main.go

@@ -2,22 +2,38 @@
 package main
 
 import (
+	"context"
 	"flag"
 	"fmt"
 	"log"
+	"net"
 	"net/http"
 	"os"
 	"strconv"
 	"time"
 
 	"git.veillax.com/Veillax/hammerhead"
-	"git.veillax.com/Veillax/hammerhead/internal/secretbox"
 	"git.veillax.com/Veillax/reef/dotenv"
+	"git.veillax.com/Veillax/reef/secretbox"
+	"git.veillax.com/Veillax/reef/selfupdate"
 )
 
 // version is set at build time (deploy/build.sh).
 var version = "dev"
 
+// localAddr turns a listen address into one this machine can dial: ":8080" and "0.0.0.0:8080"
+// become "127.0.0.1:8080".
+func localAddr(listen string) string {
+	host, port, err := net.SplitHostPort(listen)
+	if err != nil {
+		return listen
+	}
+	if host == "" || host == "0.0.0.0" || host == "::" {
+		host = "127.0.0.1"
+	}
+	return net.JoinHostPort(host, port)
+}
+
 func env(k, def string) string {
 	if v := os.Getenv(k); v != "" {
 		return v
@@ -39,6 +55,7 @@ func main() {
 	url := flag.String("url", env("HAMMERHEAD_URL", ""), "public URL people reach hammerhead at (used for login redirects and SSO); can also be set in Settings")
 	flag.String("env", ".env", "file to read secrets and settings from")
 	genkey := flag.Bool("genkey", false, "print a new random HAMMERHEAD_MASTER_KEY and exit")
+	autoUpdate := flag.Bool("auto-update", os.Getenv("HAMMERHEAD_AUTO_UPDATE") == "1", "check for new releases every few hours and update and restart by itself (Linux and macOS)")
 	showVersion := flag.Bool("version", false, "print the version and exit")
 	flag.Parse()
 	if *showVersion {
@@ -58,11 +75,19 @@ func main() {
 		MasterKey:        os.Getenv("HAMMERHEAD_MASTER_KEY"),
 		MetricsToken:     os.Getenv("HAMMERHEAD_METRICS_TOKEN"),
 		LogRetentionDays: days,
+		SelfDomain:       os.Getenv("HAMMERHEAD_SELF_DOMAIN"),
+		SelfTarget:       localAddr(*listen),
+		SelfTunnel:       os.Getenv("HAMMERHEAD_SELF_TUNNEL"),
+		Version:          version,
+		AutoUpdate:       *autoUpdate,
 	})
 	if err != nil {
 		log.Fatal(err)
 	}
 	defer srv.Close()
+	if *autoUpdate {
+		go selfupdate.Run(context.Background(), selfupdate.Config{Server: "https://git.veillax.com", Repo: "Veillax/hammerhead", App: "hammerhead", Current: version, Logf: log.Printf, Restart: func(exe string) error { srv.Close(); return selfupdate.Restart(exe) }})
+	}
 	log.Printf("hammerhead %s listening on %s (open it in a browser to finish setup)", version, *listen)
 	hs := &http.Server{Addr: *listen, Handler: srv.Handler(), ReadHeaderTimeout: 10 * time.Second}
 	log.Fatal(hs.ListenAndServe())

db.go

@@ -9,8 +9,8 @@ import (
 	"sync"
 	"time"
 
-	"git.veillax.com/Veillax/hammerhead/internal/secretbox"
 	"git.veillax.com/Veillax/reef/model"
+	"git.veillax.com/Veillax/reef/secretbox"
 	_ "modernc.org/sqlite" // pure Go, so no C compiler is needed
 )
 
@@ -44,6 +44,16 @@ CREATE TABLE IF NOT EXISTS traffic_hourly (
 	bytes_in INTEGER NOT NULL DEFAULT 0, bytes_out INTEGER NOT NULL DEFAULT 0,
 	PRIMARY KEY (mapping, hour)
 );
+CREATE TABLE IF NOT EXISTS traffic_minutely (
+	mapping TEXT NOT NULL, minute INTEGER NOT NULL, conns INTEGER NOT NULL DEFAULT 0,
+	bytes_in INTEGER NOT NULL DEFAULT 0, bytes_out INTEGER NOT NULL DEFAULT 0,
+	PRIMARY KEY (mapping, minute)
+);
+CREATE TABLE IF NOT EXISTS traffic_sources (
+	mapping TEXT NOT NULL, hour INTEGER NOT NULL, ip TEXT NOT NULL, conns INTEGER NOT NULL DEFAULT 0,
+	bytes_in INTEGER NOT NULL DEFAULT 0, bytes_out INTEGER NOT NULL DEFAULT 0,
+	PRIMARY KEY (mapping, hour, ip)
+);
 CREATE TABLE IF NOT EXISTS alerts (
 	key TEXT PRIMARY KEY, org_id TEXT NOT NULL DEFAULT '', kind TEXT, name TEXT, message TEXT,
 	since INTEGER NOT NULL, fired INTEGER NOT NULL DEFAULT 0, notified INTEGER NOT NULL DEFAULT 0
@@ -53,7 +63,7 @@ CREATE TABLE IF NOT EXISTS alert_events (
 	key TEXT, status TEXT, name TEXT, message TEXT
 );
 CREATE INDEX IF NOT EXISTS alert_events_org ON alert_events (org_id, id);
-`
+` + dmarcSchema
 
 type DB struct {
 	sql *sql.DB
@@ -113,10 +123,11 @@ func (d *DB) sealed(v any) any {
 		return &c
 	case *AlertChannel:
 		c := *x
-		c.Target = b.Seal(c.Target)
+		c.Target, c.Token = b.Seal(c.Target), b.Seal(c.Token)
 		return &c
 	case Settings:
 		x.SMTP.Pass = b.Seal(x.SMTP.Pass)
+		x.Sandbar.Token = b.Seal(x.Sandbar.Token)
 		return x
 	}
 	return v
@@ -144,9 +155,9 @@ func (d *DB) unseal(v any) error {
 	case *model.DKIMKey:
 		return d.open1(&x.PrivateKey)
 	case *AlertChannel:
-		return d.open1(&x.Target)
+		return errors.Join(d.open1(&x.Target), d.open1(&x.Token))
 	case *Settings:
-		return d.open1(&x.SMTP.Pass)
+		return errors.Join(d.open1(&x.SMTP.Pass), d.open1(&x.Sandbar.Token))
 	}
 	return nil
 }
@@ -547,22 +558,167 @@ func (d *DB) queryAudit(org string, limit int, before int64) []AuditEntry {
 
 // purge deletes logs older than the retention periods.
 func (d *DB) purge(accessDays, auditDays int) {
+	d.purgeDMARC(time.Now().AddDate(0, 0, -dmarcKeepDays))
 	if accessDays > 0 {
 		d.sql.Exec(`DELETE FROM access_log WHERE ts < ?`, ms(time.Now().AddDate(0, 0, -accessDays)))
 	}
 	if auditDays > 0 {
 		d.sql.Exec(`DELETE FROM audit_log WHERE ts < ?`, ms(time.Now().AddDate(0, 0, -auditDays)))
 	}
-	d.sql.Exec(`DELETE FROM traffic_hourly WHERE hour < ?`, time.Now().Add(-31*24*time.Hour).Unix()/3600)
+	d.sql.Exec(`DELETE FROM traffic_hourly WHERE hour < ?`, time.Now().Add(-trafficHourlyKeep).Unix()/3600)
+	d.sql.Exec(`DELETE FROM traffic_minutely WHERE minute < ?`, time.Now().Add(-trafficMinutelyKeep).Unix()/60)
+	d.sql.Exec(`DELETE FROM traffic_sources WHERE hour < ?`, time.Now().Add(-trafficSourcesKeep).Unix()/3600)
 	d.sql.Exec(`DELETE FROM alert_events WHERE ts < ?`, ms(time.Now().AddDate(0, 0, -90)))
 }
 
 // ---------- traffic history ----------
 
-func (d *DB) addTraffic(mapping string, st model.Stat) {
+// How long each resolution of traffic history is kept. Minutes are for the last hours, hours for the
+// months, and the busiest client addresses sit in between.
+const (
+	trafficMinutelyKeep = 48 * time.Hour
+	trafficSourcesKeep  = 7 * 24 * time.Hour
+	trafficHourlyKeep   = 31 * 24 * time.Hour
+)
+
+func (d *DB) addTraffic(mapping string, st model.Stat) { d.addTrafficAt(mapping, st, time.Now()) }
+
+func (d *DB) addTrafficAt(mapping string, st model.Stat, at time.Time) {
 	d.sql.Exec(`INSERT INTO traffic_hourly (mapping, hour, conns, bytes_in, bytes_out) VALUES (?, ?, ?, ?, ?)
 		ON CONFLICT (mapping, hour) DO UPDATE SET conns = conns + excluded.conns, bytes_in = bytes_in + excluded.bytes_in, bytes_out = bytes_out + excluded.bytes_out`,
-		mapping, time.Now().Unix()/3600, st.Conns, st.BytesIn, st.BytesOut)
+		mapping, at.Unix()/3600, st.Conns, st.BytesIn, st.BytesOut)
+	d.sql.Exec(`INSERT INTO traffic_minutely (mapping, minute, conns, bytes_in, bytes_out) VALUES (?, ?, ?, ?, ?)
+		ON CONFLICT (mapping, minute) DO UPDATE SET conns = conns + excluded.conns, bytes_in = bytes_in + excluded.bytes_in, bytes_out = bytes_out + excluded.bytes_out`,
+		mapping, at.Unix()/60, st.Conns, st.BytesIn, st.BytesOut)
+	for _, src := range st.Sources {
+		d.sql.Exec(`INSERT INTO traffic_sources (mapping, hour, ip, conns, bytes_in, bytes_out) VALUES (?, ?, ?, ?, ?, ?)
+			ON CONFLICT (mapping, hour, ip) DO UPDATE SET conns = conns + excluded.conns, bytes_in = bytes_in + excluded.bytes_in, bytes_out = bytes_out + excluded.bytes_out`,
+			mapping, at.Unix()/3600, src.IP, src.Conns, src.BytesIn, src.BytesOut)
+	}
+}
+
+// TrafficPoint is the traffic of one time bucket.
+type TrafficPoint struct {
+	Time     int64 `json:"t"` // unix seconds at the start of the bucket
+	Conns    int64 `json:"conns"`
+	BytesIn  int64 `json:"bytesIn"`
+	BytesOut int64 `json:"bytesOut"`
+}
+
+// trafficSeries sums the given ports' traffic into count buckets of the given size ("minute" or
+// "hour") ending with the current one, oldest first. Empty buckets are included.
+func (d *DB) trafficSeries(ids []string, unit string, count int, now time.Time) []TrafficPoint {
+	table, col, secs := "traffic_hourly", "hour", int64(3600)
+	if unit == "minute" {
+		table, col, secs = "traffic_minutely", "minute", 60
+	}
+	cur := now.Unix() / secs
+	first := cur - int64(count) + 1
+	out := make([]TrafficPoint, count)
+	for i := range out {
+		out[i].Time = (first + int64(i)) * secs
+	}
+	if len(ids) == 0 {
+		return out
+	}
+	q := `SELECT ` + col + `, SUM(conns), SUM(bytes_in), SUM(bytes_out) FROM ` + table +
+		` WHERE ` + col + ` >= ? AND ` + col + ` <= ? AND mapping IN (` + placeholders(len(ids)) + `) GROUP BY ` + col
+	args := []any{first, cur}
+	for _, id := range ids {
+		args = append(args, id)
+	}
+	rows, err := d.sql.Query(q, args...)
+	if err != nil {
+		return out
+	}
+	defer rows.Close()
+	for rows.Next() {
+		var at int64
+		var p TrafficPoint
+		if rows.Scan(&at, &p.Conns, &p.BytesIn, &p.BytesOut) != nil {
+			continue
+		}
+		if i := int(at - first); i >= 0 && i < count {
+			out[i].Conns, out[i].BytesIn, out[i].BytesOut = p.Conns, p.BytesIn, p.BytesOut
+		}
+	}
+	return out
+}
+
+func placeholders(n int) string {
+	if n <= 0 {
+		return ""
+	}
+	return strings.TrimSuffix(strings.Repeat("?,", n), ",")
+}
+
+// TrafficSource is one client address's traffic over a period.
+type TrafficSource struct {
+	IP       string `json:"ip"`
+	Conns    int64  `json:"conns"`
+	BytesIn  int64  `json:"bytesIn"`
+	BytesOut int64  `json:"bytesOut"`
+}
+
+// topSources lists the busiest client addresses of the given ports over the last hours hours.
+func (d *DB) topSources(ids []string, hours, limit int, now time.Time) []TrafficSource {
+	out := []TrafficSource{}
+	if len(ids) == 0 {
+		return out
+	}
+	args := []any{now.Unix()/3600 - int64(hours) + 1}
+	for _, id := range ids {
+		args = append(args, id)
+	}
+	args = append(args, limit)
+	rows, err := d.sql.Query(`SELECT ip, SUM(conns), SUM(bytes_in), SUM(bytes_out) FROM traffic_sources WHERE hour >= ? AND mapping IN (`+placeholders(len(ids))+`)
+		GROUP BY ip ORDER BY SUM(bytes_in) + SUM(bytes_out) DESC, SUM(conns) DESC, ip LIMIT ?`, args...)
+	if err != nil {
+		return out
+	}
+	defer rows.Close()
+	for rows.Next() {
+		var s TrafficSource
+		if rows.Scan(&s.IP, &s.Conns, &s.BytesIn, &s.BytesOut) == nil {
+			out = append(out, s)
+		}
+	}
+	return out
+}
+
+// bytesSince is the total bytes (in + out) a port has carried from the start of the given hour (unix hours) until now.
+func (d *DB) bytesSince(mapping string, fromHour int64) int64 {
+	var n sql.NullInt64
+	d.sql.QueryRow(`SELECT SUM(bytes_in + bytes_out) FROM traffic_hourly WHERE mapping = ? AND hour >= ?`, mapping, fromHour).Scan(&n)
+	return n.Int64
+}
+
+// spikeWindow and spikeBaseline are the minutes a spike is judged over, and the minutes before it
+// that say what normal looks like.
+const (
+	spikeWindow   = 5
+	spikeBaseline = 6 * 60
+)
+
+// recentTraffic returns the bytes a port carried in the last spikeWindow minutes, the average for
+// the same span over the baseline before it, and how many minutes that baseline covers (up to spikeBaseline;
+// fewer for a port with a shorter history, which is averaged over what it has).
+func (d *DB) recentTraffic(mapping string, now time.Time) (recent int64, baselinePerWindow float64, historyMinutes int64) {
+	cur := now.Unix() / 60
+	var r, b, first sql.NullInt64
+	d.sql.QueryRow(`SELECT SUM(CASE WHEN minute > ? THEN bytes_in + bytes_out END), SUM(CASE WHEN minute <= ? THEN bytes_in + bytes_out END), MIN(minute)
+		FROM traffic_minutely WHERE mapping = ? AND minute > ?`, cur-spikeWindow, cur-spikeWindow, mapping, cur-spikeWindow-spikeBaseline).Scan(&r, &b, &first)
+	if !first.Valid {
+		return 0, 0, 0
+	}
+	span := cur - spikeWindow - first.Int64 + 1
+	if span > spikeBaseline {
+		span = spikeBaseline
+	}
+	if span <= 0 {
+		return r.Int64, 0, 0
+	}
+	return r.Int64, float64(b.Int64) / float64(span) * spikeWindow, span
 }
 
 // trafficHistory returns bytes (in + out) per hour for each mapping, oldest first.

db_test.go

@@ -13,8 +13,8 @@ import (
 	"testing"
 	"time"
 
-	"git.veillax.com/Veillax/hammerhead/internal/secretbox"
 	"git.veillax.com/Veillax/reef/model"
+	"git.veillax.com/Veillax/reef/secretbox"
 )
 
 func apiClient(t *testing.T, s *Server) (do func(method, path string, in any) (int, []byte)) {

deliverability.go

@@ -0,0 +1,143 @@
+package hammerhead
+
+import (
+	"fmt"
+	"sort"
+	"strings"
+
+	"git.veillax.com/Veillax/reef/model"
+)
+
+// A Finding is one line of a mail egress's deliverability checklist: what was
+// checked, how it came out, and what to do about it. The edge produces the raw
+// results; the findings put old and new checks in one shape for the dashboard
+// and the alerts.
+type Finding struct {
+	ID     string `json:"id"`               // stable, e.g. "ptr", "spf:example.com"
+	Label  string `json:"label"`            // "Reverse DNS", "SPF"
+	Domain string `json:"domain,omitempty"` // the sending domain, for per-domain checks
+	Status string `json:"status"`           // pass, warn, fail or none (nothing published)
+	Detail string `json:"detail"`
+	Hint   string `json:"hint,omitempty"`
+}
+
+// alertWorthy says whether a finding should raise an alert: failures do, and so
+// does a missing SPF or DMARC record (bulk senders are required to publish both).
+// Warnings and a missing MTA-STS policy are advice, not problems.
+func (f Finding) alertWorthy() bool {
+	if f.Status == model.FindingFail {
+		return true
+	}
+	return f.Status == model.FindingNone && (strings.HasPrefix(f.ID, "spf:") || strings.HasPrefix(f.ID, "dmarc:"))
+}
+
+// minOutcomeSample is how many messages a window needs before its failure rate says anything.
+const minOutcomeSample = 10
+
+func deliverabilityFindings(x *model.MailEgress, chk model.MailCheck, keys []*model.DKIMKey) []Finding {
+	var out []Finding
+	add := func(f Finding) { out = append(out, f) }
+
+	if chk.PTRMatchesHelo {
+		add(Finding{ID: "ptr", Label: "Reverse DNS", Status: model.FindingPass, Detail: x.SourceIP + " resolves back to " + x.HeloName})
+	} else {
+		found := "no reverse DNS record"
+		if len(chk.PTR) > 0 {
+			found = "reverse DNS is " + strings.Join(chk.PTR, ", ")
+		}
+		add(Finding{ID: "ptr", Label: "Reverse DNS", Status: model.FindingFail, Detail: found + ", not " + x.HeloName,
+			Hint: fmt.Sprintf("Reverse DNS is set at your hosting provider, not in your DNS zone: set the PTR record of %s to %s.", x.SourceIP, x.HeloName)})
+	}
+	if chk.HeloResolves {
+		add(Finding{ID: "helo", Label: "HELO name", Status: model.FindingPass, Detail: x.HeloName + " resolves to " + x.SourceIP})
+	} else {
+		add(Finding{ID: "helo", Label: "HELO name", Status: model.FindingFail, Detail: x.HeloName + " doesn't resolve to " + x.SourceIP,
+			Hint: fmt.Sprintf("Add an A record for %s pointing at %s (an AAAA record for an IPv6 address).", x.HeloName, x.SourceIP)})
+	}
+	if chk.Port25 == "open" {
+		add(Finding{ID: "port25", Label: "Outgoing port 25", Status: model.FindingPass, Detail: "connections to other mail servers work"})
+	} else if chk.Port25 != "" {
+		add(Finding{ID: "port25", Label: "Outgoing port 25", Status: model.FindingFail, Detail: chk.Port25,
+			Hint: "Ask your hosting provider to unblock outgoing port 25 for this address; many VPS hosts block it until you ask."})
+	}
+
+	for _, k := range keys {
+		if k.OrgID != x.OrgID {
+			continue
+		}
+		name := k.Selector + "._domainkey." + k.Domain
+		switch chk.DKIM[k.ID] {
+		case "published":
+			add(Finding{ID: "dkim:" + k.ID, Label: "DKIM", Domain: k.Domain, Status: model.FindingPass, Detail: name + " is published"})
+		case "missing":
+			add(Finding{ID: "dkim:" + k.ID, Label: "DKIM", Domain: k.Domain, Status: model.FindingFail, Detail: name + " isn't in DNS",
+				Hint: "Publish the TXT record shown under DKIM signing; signed mail fails verification until it exists."})
+		case "mismatch":
+			add(Finding{ID: "dkim:" + k.ID, Label: "DKIM", Domain: k.Domain, Status: model.FindingFail, Detail: name + " holds a different key",
+				Hint: "Replace the TXT record's value with the one shown under DKIM signing; it was probably left over from an older key."})
+		}
+	}
+
+	doms := make([]string, 0, len(chk.Domains))
+	for d := range chk.Domains {
+		doms = append(doms, d)
+	}
+	sort.Strings(doms)
+	for _, d := range doms {
+		c := chk.Domains[d]
+		add(Finding{ID: "spf:" + d, Label: "SPF", Domain: d, Status: c.SPF.Status, Detail: c.SPF.Detail, Hint: c.SPF.Hint})
+		add(Finding{ID: "dmarc:" + d, Label: "DMARC", Domain: d, Status: c.DMARC.Status, Detail: c.DMARC.Detail, Hint: c.DMARC.Hint})
+		add(Finding{ID: "mtasts:" + d, Label: "MTA-STS", Domain: d, Status: c.MTASTS.Status, Detail: c.MTASTS.Detail, Hint: c.MTASTS.Hint})
+	}
+
+	switch {
+	case len(chk.Blocklists) > 0:
+		add(Finding{ID: "blocklist", Label: "Blocklists", Status: model.FindingFail, Detail: x.SourceIP + " is listed on " + strings.Join(chk.Blocklists, ", "),
+			Hint: "Look the address up on each list's website to see why it was listed and request removal. Until then expect mail to be rejected or sent to spam; check that no one is relaying spam through this server."})
+	case chk.Listable > 0:
+		add(Finding{ID: "blocklist", Label: "Blocklists", Status: model.FindingPass, Detail: fmt.Sprintf("%s isn't on any of the %d blocklists checked", x.SourceIP, chk.Listable)})
+	case chk.Port25 != "" || chk.PTR != nil:
+		add(Finding{ID: "blocklist", Label: "Blocklists", Status: model.FindingWarn, Detail: "no blocklist answered, so the address couldn't be checked",
+			Hint: "Blocklists often refuse queries from public DNS resolvers; the edge needs a resolver of its own, or its provider's, to check them."})
+	}
+
+	if o := chk.Outcomes; o != nil {
+		if total := o.Delivered + o.Deferred + o.Failed; total >= minOutcomeSample {
+			bad := float64(o.Deferred+o.Failed) / float64(total)
+			detail := fmt.Sprintf("%d of %d messages in the last day were delivered (%d deferred, %d refused)", o.Delivered, total, o.Deferred, o.Failed)
+			hint := outcomeHint(o)
+			switch {
+			case bad > 0.3:
+				add(Finding{ID: "outcomes", Label: "Delivery", Status: model.FindingFail, Detail: detail, Hint: hint})
+			case bad > 0.1:
+				add(Finding{ID: "outcomes", Label: "Delivery", Status: model.FindingWarn, Detail: detail, Hint: hint})
+			default:
+				add(Finding{ID: "outcomes", Label: "Delivery", Status: model.FindingPass, Detail: detail})
+			}
+		}
+	}
+	return out
+}
+
+// outcomeHint explains the latest refusal in terms of what to fix.
+func outcomeHint(o *model.MailOutcomes) string {
+	if len(o.Recent) == 0 {
+		return ""
+	}
+	r := o.Recent[0]
+	low := strings.ToLower(r.Text)
+	advice := "Read the message, it usually says what the receiver objected to."
+	switch {
+	case strings.Contains(low, "spf") || strings.Contains(low, "dmarc") || strings.Contains(low, "dkim") || strings.Contains(low, "authenticat"):
+		advice = "The receiver wants authentication to pass: check the SPF, DKIM and DMARC lines above."
+	case strings.Contains(low, "blocklist") || strings.Contains(low, "blacklist") || strings.Contains(low, "rbl") || strings.Contains(low, "listed") || strings.Contains(low, "spamhaus"):
+		advice = "The receiver rejected the address as listed on a blocklist: see the Blocklists line and request delisting."
+	case strings.Contains(low, "greylist") || strings.Contains(low, "try again") || strings.Contains(low, "temporar"):
+		advice = "That receiver is delaying unfamiliar senders; these retry and normally get through."
+	case strings.Contains(low, "rate") || strings.Contains(low, "too many") || strings.Contains(low, "throttl"):
+		advice = "The receiver is limiting how fast you send to it; slow the sending or lower the relay's per-minute limit."
+	case strings.Contains(low, "ptr") || strings.Contains(low, "reverse"):
+		advice = "The receiver objects to the reverse DNS: see the Reverse DNS line."
+	}
+	return fmt.Sprintf("Latest from %s: %d %s. %s", r.Domain, r.Code, r.Text, advice)
+}

deliverability_test.go

@@ -0,0 +1,127 @@
+package hammerhead
+
+import (
+	"strings"
+	"testing"
+
+	"git.veillax.com/Veillax/reef/model"
+)
+
+func find(fs []Finding, id string) *Finding {
+	for i := range fs {
+		if fs[i].ID == id {
+			return &fs[i]
+		}
+	}
+	return nil
+}
+
+func TestFindingsCarryFixes(t *testing.T) {
+	x := &model.MailEgress{ID: "x1", OrgID: "o", SourceIP: "203.0.113.7", HeloName: "mail.example.com"}
+	keys := []*model.DKIMKey{{ID: "k1", OrgID: "o", Domain: "example.com", Selector: "s1"}, {ID: "k2", OrgID: "other", Domain: "x.test", Selector: "s"}}
+	chk := model.MailCheck{
+		PTR: []string{"host.vps.test"}, Port25: "can't connect",
+		DKIM: map[string]string{"k1": "mismatch", "k2": "missing"},
+		Domains: map[string]model.DomainCheck{"example.com": {
+			SPF:    model.MailFinding{Status: model.FindingFail, Detail: "no", Hint: "add ip4"},
+			DMARC:  model.MailFinding{Status: model.FindingNone, Detail: "none", Hint: "publish"},
+			MTASTS: model.MailFinding{Status: model.FindingNone, Detail: "none", Hint: "optional"},
+		}},
+		Blocklists: []string{"zen.spamhaus.org"},
+	}
+	fs := deliverabilityFindings(x, chk, keys)
+	for id, want := range map[string]string{"ptr": "fail", "helo": "fail", "port25": "fail", "dkim:k1": "fail", "spf:example.com": "fail", "blocklist": "fail"} {
+		f := find(fs, id)
+		if f == nil || f.Status != want {
+			t.Errorf("%s: %+v", id, f)
+		} else if want == "fail" && f.Hint == "" {
+			t.Errorf("%s fails without saying how to fix it", id)
+		}
+	}
+	if find(fs, "dkim:k2") != nil {
+		t.Error("another org's key leaked into the checklist")
+	}
+	if f := find(fs, "ptr"); !strings.Contains(f.Detail, "host.vps.test") || !strings.Contains(f.Hint, "mail.example.com") {
+		t.Errorf("ptr: %+v", f)
+	}
+	if find(fs, "mtasts:example.com").alertWorthy() {
+		t.Error("a missing MTA-STS policy is advice, not an alert")
+	}
+	if !find(fs, "dmarc:example.com").alertWorthy() {
+		t.Error("a missing DMARC record should alert")
+	}
+}
+
+func TestFindingsAllGood(t *testing.T) {
+	x := &model.MailEgress{OrgID: "o", SourceIP: "203.0.113.7", HeloName: "mail.example.com"}
+	chk := model.MailCheck{PTRMatchesHelo: true, HeloResolves: true, Port25: "open", Listable: 4,
+		Outcomes: &model.MailOutcomes{Delivered: 100}}
+	for _, f := range deliverabilityFindings(x, chk, nil) {
+		if f.Status != model.FindingPass || f.alertWorthy() {
+			t.Errorf("%+v", f)
+		}
+	}
+}
+
+func TestOutcomeFindings(t *testing.T) {
+	x := &model.MailEgress{OrgID: "o", SourceIP: "203.0.113.7", HeloName: "h"}
+	status := func(o *model.MailOutcomes) string {
+		f := find(deliverabilityFindings(x, model.MailCheck{Outcomes: o}, nil), "outcomes")
+		if f == nil {
+			return ""
+		}
+		return f.Status
+	}
+	for name, c := range map[string]struct {
+		o    *model.MailOutcomes
+		want string
+	}{
+		"too few to judge": {&model.MailOutcomes{Delivered: 2, Failed: 5}, ""},
+		"healthy":          {&model.MailOutcomes{Delivered: 95, Deferred: 5}, "pass"},
+		"some trouble":     {&model.MailOutcomes{Delivered: 80, Failed: 20}, "warn"},
+		"mostly refused":   {&model.MailOutcomes{Delivered: 20, Failed: 80}, "fail"},
+	} {
+		if got := status(c.o); got != c.want {
+			t.Errorf("%s: %q, want %q", name, got, c.want)
+		}
+	}
+	h := outcomeHint(&model.MailOutcomes{Recent: []model.OutcomeFailure{{Domain: "gmail.com", Code: 550, Text: "5.7.26 DMARC authentication failed"}}})
+	if !strings.Contains(h, "gmail.com") || !strings.Contains(h, "SPF, DKIM and DMARC") {
+		t.Errorf("hint: %q", h)
+	}
+	if outcomeHint(&model.MailOutcomes{}) != "" {
+		t.Error("no refusals, no hint")
+	}
+}
+
+func TestDeliverabilityRaisesAlertsWithHints(t *testing.T) {
+	f, _ := dmarcFx(t)
+	f.s.mu.Lock()
+	f.s.mailChecks["x1"] = model.MailCheck{PTRMatchesHelo: true, HeloResolves: true, Port25: "open", Listable: 4,
+		Blocklists: []string{"bl.spamcop.net"},
+		Domains: map[string]model.DomainCheck{"example.com": {
+			SPF:    model.MailFinding{Status: model.FindingFail, Detail: "SPF for example.com does not authorize 203.0.113.7", Hint: "Add ip4:203.0.113.7"},
+			DMARC:  model.MailFinding{Status: model.FindingWarn, Detail: "monitoring"},
+			MTASTS: model.MailFinding{Status: model.FindingNone},
+		}}}
+	probs := f.s.problemsLocked(f.base)
+	f.s.mu.Unlock()
+	spf, bl := probs["mailfind:x1:spf:example.com"], probs["mailfind:x1:blocklist"]
+	if !strings.Contains(spf.msg, "Add ip4:203.0.113.7") || !strings.Contains(bl.msg, "bl.spamcop.net") {
+		t.Fatalf("alerts should carry the fix: %+v %+v", spf, bl)
+	}
+	if _, noisy := probs["mailfind:x1:dmarc:example.com"]; noisy {
+		t.Error("a DMARC warning shouldn't alert")
+	}
+
+	// fixed: both clear
+	f.s.mu.Lock()
+	f.s.mailChecks["x1"] = model.MailCheck{PTRMatchesHelo: true, HeloResolves: true, Port25: "open", Listable: 4}
+	probs = f.s.problemsLocked(f.base)
+	f.s.mu.Unlock()
+	for k := range probs {
+		if strings.HasPrefix(k, "mailfind:") {
+			t.Errorf("%s should have cleared", k)
+		}
+	}
+}

deploy/build.sh

@@ -8,12 +8,13 @@
 #   dist/hammerhead_<version>_<os>_<arch>.tar.gz or .zip
 #   dist/SHA256SUMS
 #
-# Set TARGETS="linux/amd64 linux/arm64" to build fewer platforms.
+# Releases are published for linux/amd64 and windows/amd64 only. Set TARGETS="linux/arm64 darwin/arm64"
+# (any GOOS/GOARCH pair Go supports) to build something else by hand.
 set -eu
 NAME=hammerhead
 cd "$(dirname "$0")/.."
 VERSION=${1:-$(git describe --tags --always 2>/dev/null || echo dev)}
-TARGETS=${TARGETS:-"linux/amd64 linux/arm64 linux/arm windows/amd64 darwin/amd64 darwin/arm64"}
+TARGETS=${TARGETS:-"linux/amd64 windows/amd64"}
 
 rm -rf dist
 mkdir -p dist/bin dist/stage

deploy/hammerhead.service

@@ -8,7 +8,7 @@ Wants=network-online.target
 User=hammerhead
 Group=hammerhead
 EnvironmentFile=-/etc/hammerhead/hammerhead.env
-ExecStart=/usr/local/bin/hammerhead -listen :8080 -data /var/lib/hammerhead
+ExecStart=/usr/local/lib/hammerhead/hammerhead -listen :8080 -data /var/lib/hammerhead
 WorkingDirectory=/var/lib/hammerhead
 Restart=on-failure
 RestartSec=3
@@ -16,7 +16,7 @@ RestartSec=3
 # It only needs its own data folder and the network.
 NoNewPrivileges=true
 ProtectSystem=strict
-ReadWritePaths=/var/lib/hammerhead
+ReadWritePaths=/var/lib/hammerhead /usr/local/lib/hammerhead
 ProtectHome=true
 PrivateTmp=true
 ProtectKernelTunables=true

deploy/install.sh

@@ -5,7 +5,7 @@
 #   sudo ./install.sh --start      the same, then start (or restart) the service
 #   sudo ./install.sh --uninstall  remove the service and binary (keeps the settings and data)
 #
-# Files: /usr/local/bin/hammerhead, /etc/systemd/system/hammerhead.service,
+# Files: /usr/local/lib/hammerhead/hammerhead (linked from /usr/local/bin/hammerhead), /etc/systemd/system/hammerhead.service,
 #        /etc/hammerhead/hammerhead.env (your settings and secrets), /var/lib/hammerhead (data).
 # Set DESTDIR=/some/dir to stage everything there without touching the system.
 set -eu
@@ -17,6 +17,7 @@ UNIT=$ROOT/etc/systemd/system/$NAME.service
 CONF=$ROOT/etc/$NAME
 ENVFILE=$CONF/$NAME.env
 DATA=$ROOT/var/lib/$NAME
+LIB=$ROOT/usr/local/lib/$NAME # the program lives here, owned by the service, so auto-update can replace it
 
 say() { printf '%s\n' "$*"; }
 real() { [ -z "${DESTDIR:-}" ]; } # are we touching the real system?
@@ -30,7 +31,7 @@ if [ "${1:-}" = "--uninstall" ]; then
 	if real; then
 		systemctl disable --now "$NAME" > /dev/null 2>&1 || true
 	fi
-	rm -f "$UNIT" "$BIN"
+	rm -f "$UNIT" "$BIN" "$LIB/$NAME"
 	if real; then
 		systemctl daemon-reload || true
 	fi
@@ -46,15 +47,20 @@ if real && ! id "$NAME" > /dev/null 2>&1; then
 fi
 
 mkdir -p "$ROOT/usr/local/bin" "$ROOT/etc/systemd/system" "$CONF" "$DATA"
-install -m 0755 "$HERE/$NAME" "$BIN.new"
+mkdir -p "$LIB"
-mv -f "$BIN.new" "$BIN" # atomic, so upgrading a running service is safe
+install -m 0755 "$HERE/$NAME" "$LIB/$NAME.new"
+mv -f "$LIB/$NAME.new" "$LIB/$NAME" # atomic, so upgrading a running service is safe
+rm -f "$BIN"
+LINK=/usr/local/lib/$NAME/$NAME
+[ -z "${DESTDIR:-}" ] || LINK=$LIB/$NAME # staging: point inside the staging folder
+ln -s "$LINK" "$BIN"
 install -m 0644 "$HERE/$NAME.service" "$UNIT"
 if [ ! -f "$ENVFILE" ]; then
 	install -m 0600 "$HERE/.env.example" "$ENVFILE"
 	say "Created $ENVFILE. Edit it before starting."
 fi
 if real; then
-	chown -R "$NAME:$NAME" "$DATA"
+	chown -R "$NAME:$NAME" "$DATA" "$LIB"
 fi
 
 if real; then
@@ -66,7 +72,7 @@ if real; then
 	fi
 fi
 
-say "Installed $("$BIN" -version 2> /dev/null || echo "$NAME")."
+say "Installed $("$LIB/$NAME" -version 2> /dev/null || echo "$NAME")."
 say "Next:"
 say "  1. edit $ENVFILE"
 say "  2. systemctl start $NAME      (or run this again with --start)"

deploy/publish-release.sh

@@ -42,19 +42,30 @@ else
 	echo "release $TAG already exists (id $id): replacing its files"
 fi
 
-existing=$(curl -fsS -H "$AUTH" "$API/releases/$id/assets" | json 'import sys, json
+# the ids of the release's assets with a given name (a failed upload can leave a partial one behind)
+asset_ids() {
+	curl -fsS -H "$AUTH" "$API/releases/$id/assets" | "$PY" -c 'import sys, json
 for a in json.load(sys.stdin):
-    print(a["id"], a["name"])')
+    if a["name"] == sys.argv[1]:
+        print(a["id"])' "$1"
+}
 
 for f in dist/*.tar.gz dist/*.zip dist/SHA256SUMS; do
 	[ -f "$f" ] || continue
 	name=$(basename "$f")
-	echo "$existing" | while read -r aid aname; do
+	n=1
-		if [ "$aname" = "$name" ]; then
+	while :; do
+		for aid in $(asset_ids "$name"); do
 			curl -fsS -X DELETE -H "$AUTH" "$API/releases/$id/assets/$aid" > /dev/null
-		fi
 		done
-	echo "uploading $name"
+		echo "uploading $name (attempt $n)"
-	curl -fsS -X POST -H "$AUTH" -F "attachment=@$f" "$API/releases/$id/assets?name=$name" > /dev/null
+		if curl -fsS --max-time 1200 -X POST -H "$AUTH" -F "attachment=@$f" "$API/releases/$id/assets?name=$name" > /dev/null; then
+			break
+		fi
+		# large uploads can hit a proxy timeout (HTTP 502); try again
+		[ "$n" -lt 5 ] || { echo "giving up on $name after $n attempts" >&2; exit 1; }
+		n=$((n + 1))
+		sleep $((n * 10))
+	done
 done
 echo "published $SERVER/$REPO/releases/tag/$TAG"

deploy/setup.sh

@@ -1,34 +1,45 @@
 #!/bin/sh
 # One-shot setup for a VPS. Run as root.
 #
-#   sh setup.sh [--domain hh.example.com] [--version v0.1.0]     Hammerhead (dashboard and API)
+#   sh setup.sh --domain hh.example.com [options]     Hammerhead and a Blacktip edge on this machine
-#   sh setup.sh edge  --hammerhead URL --token TOKEN [--ports 25565,30000-30100] [--version V]
+#   sh setup.sh edge  --hammerhead URL --token TOKEN [options]    another edge
-#   sh setup.sh agent --hammerhead URL --token TOKEN [--version V]
+#   sh setup.sh agent --hammerhead URL --token TOKEN [options]    a site agent
 #
-# Hammerhead: installs it as a systemd service (with a generated master key, kept in
+# With --domain, this installs Hammerhead (the dashboard) and a Blacktip edge on the same
-#   /etc/hammerhead/hammerhead.env), and with --domain also Caddy, which gets and renews
+# machine. The edge serves the dashboard itself at https://<domain> with an automatic
-#   the HTTPS certificate for that name. DNS for the name must already point here.
+# certificate, so no separate web server is needed. DNS for the domain must already point here,
-# edge:  installs Blacktip, with its bundled Traefik, on this machine (or any other VPS),
+# and ports 80 and 443 must be free (stop Pangolin, Caddy, nginx... first).
-#   opens the firewall and lists every IP address it can open ports on.
-# agent: installs Bamboo, the site agent, on the machine that hosts your services.
 #
-# Edge and agent need a token, which only exists once Hammerhead is up: create the edge under
+# Options:
-# Server -> Edges (or the site under Sites) in the dashboard and copy the values from "Show command".
+#   --version V         release of Hammerhead (or of the edge/agent for those modes); default newest
+#   --edge-version V    release of the Blacktip installed next to Hammerhead; default newest
+#   --ports 25565,30000-30100   also open these (TCP and UDP) in the firewall
+#   --tunnel HOST:PORT  the edge's public WireGuard address; default <domain>:51820
+#   --caddy             old style: Hammerhead alone, with Caddy providing HTTPS (no edge here)
+#   --no-auto-update    don't let the programs update themselves
+#
+# Without --domain, Hammerhead alone is installed on port 8080 over plain HTTP.
+# Edge and agent need a token: create the edge under Server -> Edges (or the site under Sites)
+# in the dashboard and copy the values from "Show command".
 # Safe to run again: it upgrades and keeps your settings and data.
 set -eu
 
 BASE=${SHARKS_BASE:-https://git.veillax.com/Veillax}
 MODE=hammerhead
 case "${1:-}" in edge | agent) MODE=$1; shift ;; esac
-DOMAIN= VERSION= HH= TOKEN= PORTS=
+DOMAIN= VERSION= EDGE_VERSION= HH= TOKEN= PORTS= TUNNEL= CADDY=0 AUTOUP=1
 while [ $# -gt 0 ]; do
 	case "$1" in
 		--domain) DOMAIN=$2; shift 2 ;;
 		--version) VERSION=$2; shift 2 ;;
+		--edge-version) EDGE_VERSION=$2; shift 2 ;;
 		--hammerhead) HH=$2; shift 2 ;;
 		--token) TOKEN=$2; shift 2 ;;
 		--ports) PORTS=$2; shift 2 ;;
-		-h | --help) sed -n '2,16p' "$0"; exit 0 ;;
+		--tunnel) TUNNEL=$2; shift 2 ;;
+		--caddy) CADDY=1; shift ;;
+		--no-auto-update) AUTOUP=0; shift ;;
+		-h | --help) sed -n '2,24p' "$0"; exit 0 ;;
 		*) echo "unknown option $1 (see --help)" >&2; exit 2 ;;
 	esac
 done
@@ -40,9 +51,7 @@ command -v systemctl > /dev/null 2>&1 || die "this needs a systemd-based Linux (
 
 case "$(uname -m)" in
 	x86_64 | amd64) ARCH=amd64 ;;
-	aarch64 | arm64) ARCH=arm64 ;;
+	*) die "releases are published for x86-64 Linux only, not $(uname -m); build it yourself with: TARGETS=linux/arm64 sh deploy/build.sh" ;;
-	armv7* | armv6* | arm) ARCH=arm ;;
-	*) die "unsupported CPU: $(uname -m)" ;;
 esac
 
 pkg_install() { # installs packages with whichever package manager there is
@@ -68,10 +77,10 @@ latest() { # newest release tag of repo $1 (read from git, so no API token is ne
 	git ls-remote --tags --refs "$BASE/$1" 2> /dev/null | sed 's|.*refs/tags/||' | grep -E '^v[0-9]' | sort -V | tail -n 1
 }
 
-# fetch_install <app>: downloads that app's release for this CPU, verifies it and runs its install.sh
+# fetch_install <app> [version]: downloads that app's release for this CPU, verifies it and runs its install.sh
 fetch_install() {
 	app=$1
-	ver=${VERSION:-$(latest "$app")}
+	ver=${2:-$(latest "$app")}
 	[ -n "$ver" ] || die "couldn't find a release of $app at $BASE/$app (use --version, or SHARKS_BASE for a mirror)"
 	dir=$(mktemp -d)
 	say "downloading $app $ver ($ARCH)"
@@ -97,6 +106,12 @@ set_env() {
 	fi
 }
 
+# auto_update <file> <PREFIX>: lets the program update itself unless --no-auto-update was given
+auto_update() {
+	[ "$AUTOUP" = 1 ] && set_env "$1" "$2_AUTO_UPDATE" 1
+	return 0
+}
+
 firewall_open() { # firewall_open 80/tcp 443/tcp ...
 	if command -v ufw > /dev/null 2>&1 && ufw status 2> /dev/null | grep -q "Status: active"; then
 		for p in "$@"; do ufw allow "$p" > /dev/null; done
@@ -119,53 +134,18 @@ wait_active() { # wait_active <service>
 	systemctl is-active --quiet "$1" || { journalctl -u "$1" -n 20 --no-pager; die "$1 didn't start"; }
 }
 
-case $MODE in
+# listeners on 80/443 that would stop an edge serving websites
-hammerhead)
+port_users() {
-	fetch_install hammerhead
+	ss -H -ltnp '( sport = :80 or sport = :443 )' 2> /dev/null | grep -v '"blacktip"\|"traefik"' || true
-	ENVF=/etc/hammerhead/hammerhead.env
+}
-	if ! grep -q '^HAMMERHEAD_MASTER_KEY=.' "$ENVF"; then
+
-		say "generating the master key"
+# edge_setup <hammerhead url> <token> <version>: Blacktip with its bundled Traefik, configured but not started
-		set_env "$ENVF" HAMMERHEAD_MASTER_KEY "$(/usr/local/bin/hammerhead -genkey)"
+edge_setup() {
-		echo "Saved in $ENVF. Back that file up: without the key the encrypted secrets in the database can't be read."
+	fetch_install blacktip "$3"
-	fi
-	if [ -n "$DOMAIN" ]; then
-		set_env "$ENVF" HAMMERHEAD_URL "https://$DOMAIN"
-		# only Caddy needs to reach it
-		sed -i 's|-listen :8080|-listen 127.0.0.1:8080|' /etc/systemd/system/hammerhead.service
-		systemctl daemon-reload
-		if ! command -v caddy > /dev/null 2>&1; then
-			say "installing Caddy (HTTPS certificates for $DOMAIN)"
-			pkg_install caddy || die "couldn't install Caddy from this distribution; see https://caddyserver.com/docs/install and proxy $DOMAIN to 127.0.0.1:8080"
-		fi
-		mkdir -p /etc/caddy
-		printf '%s {\n\treverse_proxy 127.0.0.1:8080\n}\n' "$DOMAIN" > /etc/caddy/Caddyfile
-		systemctl enable caddy > /dev/null 2>&1 || true
-		systemctl restart caddy
-		firewall_open 80/tcp 443/tcp
-		URL="https://$DOMAIN"
-	else
-		firewall_open 8080/tcp
-		ip=$(public_ips | head -n 1)
-		URL="http://${ip:-this-server}:8080"
-		echo "No --domain given, so the dashboard is plain HTTP. Put it behind HTTPS before you create real logins (run this again with --domain)."
-	fi
-	systemctl restart hammerhead
-	wait_active hammerhead
-	say "Hammerhead is running"
-	echo "Open $URL to create the first administrator."
-	echo
-	echo "Next, in the dashboard: Server -> Edges -> add an edge, then on the machine that has your public IPs run"
-	echo "  sh setup.sh edge --hammerhead $URL --token <edge token>"
-	echo "and on each machine that hosts your services (home server, game server...) run"
-	echo "  sh setup.sh agent --hammerhead $URL --token <site token>"
-	echo "Either can be this same machine."
-	;;
-edge)
-	[ -n "$HH" ] && [ -n "$TOKEN" ] || die "edge needs --hammerhead URL and --token TOKEN (Server -> Edges -> Show command)"
-	fetch_install blacktip
 	ENVF=/etc/blacktip/blacktip.env
-	set_env "$ENVF" BLACKTIP_HAMMERHEAD "$HH"
+	set_env "$ENVF" BLACKTIP_HAMMERHEAD "$1"
-	set_env "$ENVF" BLACKTIP_TOKEN "$TOKEN"
+	set_env "$ENVF" BLACKTIP_TOKEN "$2"
+	auto_update "$ENVF" BLACKTIP
 	chmod 600 "$ENVF"
 	say "network"
 	# WireGuard moves a lot of UDP; the stock socket buffers are small
@@ -192,16 +172,98 @@ edge)
 	fi
 	echo "In the dashboard each port picks which of these addresses it listens on."
 	echo "Ports you open later in the dashboard must also be allowed in the firewall: run this again with --ports."
+}
+
+case $MODE in
+hammerhead)
+	fetch_install hammerhead "$VERSION"
+	ENVF=/etc/hammerhead/hammerhead.env
+	auto_update "$ENVF" HAMMERHEAD
+	if ! grep -q '^HAMMERHEAD_MASTER_KEY=.' "$ENVF"; then
+		say "generating the master key"
+		set_env "$ENVF" HAMMERHEAD_MASTER_KEY "$(/usr/local/lib/hammerhead/hammerhead -genkey)"
+		echo "Saved in $ENVF. Back that file up: without the key the encrypted secrets in the database can't be read."
+	fi
+	UNIT=/etc/systemd/system/hammerhead.service
+	if [ -n "$DOMAIN" ]; then
+		set_env "$ENVF" HAMMERHEAD_URL "https://$DOMAIN"
+		# only something on this machine needs to reach it
+		sed -i 's|-listen :8080|-listen 127.0.0.1:8080|' "$UNIT"
+		if [ "$CADDY" = 1 ]; then
+			systemctl daemon-reload
+			if ! command -v caddy > /dev/null 2>&1; then
+				say "installing Caddy (HTTPS certificates for $DOMAIN)"
+				pkg_install caddy || die "couldn't install Caddy from this distribution; see https://caddyserver.com/docs/install and proxy $DOMAIN to 127.0.0.1:8080"
+			fi
+			mkdir -p /etc/caddy
+			printf '%s {\n\treverse_proxy 127.0.0.1:8080\n}\n' "$DOMAIN" > /etc/caddy/Caddyfile
+			systemctl enable caddy > /dev/null 2>&1 || true
+			systemctl restart caddy
+			firewall_open 80/tcp 443/tcp
+			URL="https://$DOMAIN"
+		else
+			# the dashboard is served by the edge on this machine, so Hammerhead gets its own edge
+			set_env "$ENVF" HAMMERHEAD_SELF_DOMAIN "$DOMAIN"
+			[ -z "$TUNNEL" ] || set_env "$ENVF" HAMMERHEAD_SELF_TUNNEL "$TUNNEL"
+			URL="https://$DOMAIN"
+		fi
+	else
+		firewall_open 8080/tcp
+		ip=$(public_ips | head -n 1)
+		URL="http://${ip:-this-server}:8080"
+		echo "No --domain given, so the dashboard is plain HTTP. Put it behind HTTPS before you create real logins (run this again with --domain)."
+	fi
+	systemctl daemon-reload
+	systemctl restart hammerhead
+	wait_active hammerhead
+	say "Hammerhead is running"
+
+	if [ -n "$DOMAIN" ] && [ "$CADDY" = 0 ]; then
+		TOKENFILE=/var/lib/hammerhead/self-edge.env
+		for i in 1 2 3 4 5 6 7 8 9 10; do [ -f "$TOKENFILE" ] && break; sleep 1; done
+		[ -f "$TOKENFILE" ] || die "Hammerhead didn't create $TOKENFILE; check: journalctl -u hammerhead"
+		edge_token=$(sed -n 's/^BLACKTIP_TOKEN=//p' "$TOKENFILE")
+		edge_setup http://127.0.0.1:8080 "$edge_token" "$EDGE_VERSION"
+		# something else on 80/443 would stop the dashboard route from coming up
+		if systemctl is-active --quiet caddy 2> /dev/null; then
+			say "stopping Caddy"
+			echo "Caddy was serving this site and holds ports 80/443, which the edge needs now."
+			systemctl disable --now caddy > /dev/null 2>&1 || true
+		fi
+		others=$(port_users)
+		if [ -n "$others" ]; then
+			echo "$others" >&2
+			die "something else is listening on port 80 or 443 (above). Stop it (for Pangolin: docker compose stop), then run: systemctl restart blacktip"
+		fi
+		systemctl restart blacktip
+		wait_active blacktip
+		say "Blacktip is running"
+		echo "The dashboard is served by the edge on this machine at $URL once the certificate is issued"
+		echo "(a few seconds; $DOMAIN must point at this server). Until then, from your own computer:"
+		echo "  ssh -L 8080:127.0.0.1:8080 you@this-server      then open http://localhost:8080"
+	fi
+	echo
+	echo "Open $URL to create the first administrator."
+	echo
+	echo "Next: on each machine that hosts your services (home server, game server...) run"
+	echo "  sh setup.sh agent --hammerhead $URL --token <site token>"
+	echo "with the token from Sites -> add a site. For more public servers add edges under Server -> Edges and run"
+	echo "  sh setup.sh edge --hammerhead $URL --token <edge token>"
+	;;
+edge)
+	[ -n "$HH" ] && [ -n "$TOKEN" ] || die "edge needs --hammerhead URL and --token TOKEN (Server -> Edges -> Show command)"
+	edge_setup "$HH" "$TOKEN" "$VERSION"
 	systemctl restart blacktip
 	wait_active blacktip
 	say "Blacktip is running"
 	;;
 agent)
 	[ -n "$HH" ] && [ -n "$TOKEN" ] || die "agent needs --hammerhead URL and --token TOKEN (Sites -> Show command)"
-	fetch_install bamboo
+	fetch_install bamboo "$VERSION"
 	ENVF=/etc/bamboo/bamboo.env
 	set_env "$ENVF" BAMBOO_HAMMERHEAD "$HH"
 	set_env "$ENVF" BAMBOO_TOKEN "$TOKEN"
+	auto_update "$ENVF" BAMBOO
 	chmod 600 "$ENVF"
 	if command -v docker > /dev/null 2>&1 && getent group docker > /dev/null 2>&1; then
 		usermod -aG docker bamboo

discovered_test.go

@@ -0,0 +1,36 @@
+package hammerhead
+
+import (
+	"strings"
+	"testing"
+
+	"git.veillax.com/Veillax/reef/model"
+)
+
+func TestCleanDiscoveredDropsBadEntriesAndBoundsText(t *testing.T) {
+	in := []model.DiscoveredService{
+		{Host: "192.168.1.5", Port: 22, Proto: "udp", Service: "ssh", Detail: "SSH-2.0", Source: "scan"},
+		{Host: "127.0.0.1", Port: 8080, Service: strings.Repeat("x", 100), Detail: strings.Repeat("d", 1000), Source: "local"},
+		{Host: "<script>", Port: 80},
+		{Host: "10.0.0.1", Port: 0},
+		{Host: "10.0.0.1", Port: 70000},
+		{Host: "10.0.0.2", Port: 80, Source: "whatever"},
+	}
+	got := cleanDiscovered(in)
+	if len(got) != 3 {
+		t.Fatalf("want 3 valid entries, got %+v", got)
+	}
+	if got[0].Proto != "tcp" || got[1].Source != "local" || len(got[1].Service) != 20 || len(got[1].Detail) != 160 || got[2].Source != "scan" {
+		t.Errorf("not normalised: %+v", got)
+	}
+	many := make([]model.DiscoveredService, 900)
+	for i := range many {
+		many[i] = model.DiscoveredService{Host: "10.0.0.1", Port: i%60000 + 1}
+	}
+	if n := len(cleanDiscovered(many)); n != 500 {
+		t.Errorf("not capped: %d", n)
+	}
+	if cleanDiscovered(nil) == nil {
+		t.Error("should be an empty list, not nil, so the API returns []")
+	}
+}

dmarc.go

@@ -0,0 +1,349 @@
+package hammerhead
+
+import (
+	"database/sql"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"sort"
+	"strings"
+	"time"
+
+	"git.veillax.com/Veillax/reef/dmarc"
+	"git.veillax.com/Veillax/reef/model"
+)
+
+// DMARC aggregate reports. Receivers that handle mail claiming to be from your
+// domain mail a daily summary to the rua= address in its DMARC record. Those
+// emails are uploaded here (the raw .xml, .gz, .zip or the whole .eml), parsed,
+// and kept for 90 days; the dashboard shows how much of the mail sent as your
+// domain passed, and which senders are failing.
+
+const (
+	maxDMARCUpload   = 8 << 20
+	dmarcKeepDays    = 90
+	dmarcWindowDays  = 30
+	dmarcAlertMinMsg = 20  // messages from our own relays before their pass rate says anything
+	dmarcAlertBelow  = 0.9 // alert when less of our own mail than this passes DMARC
+)
+
+// ---------- storage ----------
+
+const dmarcSchema = `
+CREATE TABLE IF NOT EXISTS dmarc_reports (
+	org_id TEXT NOT NULL, domain TEXT NOT NULL, reporter TEXT NOT NULL, report_id TEXT NOT NULL,
+	begin_ts INTEGER NOT NULL, end_ts INTEGER NOT NULL, added INTEGER NOT NULL, data TEXT NOT NULL,
+	PRIMARY KEY (org_id, domain, reporter, report_id)
+);
+CREATE INDEX IF NOT EXISTS dmarc_end ON dmarc_reports (org_id, end_ts);
+`
+
+// addDMARC stores a report; it reports false if this exact report was already there.
+func (d *DB) addDMARC(org string, r *dmarc.Report) (bool, error) {
+	data, err := json.Marshal(r)
+	if err != nil {
+		return false, err
+	}
+	res, err := d.sql.Exec(`INSERT OR IGNORE INTO dmarc_reports (org_id, domain, reporter, report_id, begin_ts, end_ts, added, data) VALUES (?, ?, ?, ?, ?, ?, ?, ?)`,
+		org, r.Domain, strings.ToLower(r.Org), r.ReportID, ms(r.Begin), ms(r.End), ms(time.Now()), string(data))
+	if err != nil {
+		return false, err
+	}
+	n, _ := res.RowsAffected()
+	return n > 0, nil
+}
+
+// dmarcReports returns an org's reports that ended after since.
+func (d *DB) dmarcReports(org string, since time.Time) []*dmarc.Report {
+	rows, err := d.sql.Query(`SELECT data FROM dmarc_reports WHERE org_id = ? AND end_ts > ? ORDER BY end_ts`, org, ms(since))
+	if err != nil {
+		return nil
+	}
+	defer rows.Close()
+	var out []*dmarc.Report
+	for rows.Next() {
+		var data string
+		var r dmarc.Report
+		if rows.Scan(&data) == nil && json.Unmarshal([]byte(data), &r) == nil {
+			out = append(out, &r)
+		}
+	}
+	return out
+}
+
+func (d *DB) purgeDMARC(before time.Time) {
+	d.sql.Exec(`DELETE FROM dmarc_reports WHERE end_ts < ?`, ms(before))
+}
+
+func (d *DB) deleteDMARCDomain(org, domain string) (int64, error) {
+	var res sql.Result
+	res, err := d.sql.Exec(`DELETE FROM dmarc_reports WHERE org_id = ? AND domain = ?`, org, domain)
+	if err != nil {
+		return 0, err
+	}
+	return res.RowsAffected()
+}
+
+// ---------- summary ----------
+
+type dmarcSource struct {
+	IP        string `json:"ip"`
+	Messages  int    `json:"messages"`
+	Passed    int    `json:"passed"` // messages that passed DMARC
+	DKIMPass  int    `json:"dkimPass"`
+	SPFPass   int    `json:"spfPass"`
+	Ours      bool   `json:"ours"`      // the address is one of this org's mail egress IPs
+	Rejected  int    `json:"rejected"`  // the receiver quarantined or rejected these
+	Reporters int    `json:"reporters"` // how many receivers saw it
+	Hint      string `json:"hint,omitempty"`
+}
+
+type dmarcDomain struct {
+	Domain    string        `json:"domain"`
+	Reports   int           `json:"reports"`
+	Messages  int           `json:"messages"`
+	Passed    int           `json:"passed"`
+	OurMsgs   int           `json:"ourMessages"` // from our own relays
+	OurPassed int           `json:"ourPassed"`
+	Policy    string        `json:"policy"` // the latest published policy a receiver reported
+	Pct       int           `json:"pct"`
+	Reporters []string      `json:"reporters"`
+	First     time.Time     `json:"first"`
+	Last      time.Time     `json:"last"`
+	Sources   []dmarcSource `json:"sources"`
+	Advice    []string      `json:"advice,omitempty"`
+}
+
+// summarizeDMARC totals reports per domain. ours holds the org's egress source IPs.
+func summarizeDMARC(reports []*dmarc.Report, ours map[string]bool) []dmarcDomain {
+	type acc struct {
+		d         *dmarcDomain
+		src       map[string]*dmarcSource
+		srcRep    map[string]map[string]bool
+		reporters map[string]bool
+		lastEnd   time.Time
+	}
+	by := map[string]*acc{}
+	for _, r := range reports {
+		a := by[r.Domain]
+		if a == nil {
+			a = &acc{d: &dmarcDomain{Domain: r.Domain, First: r.Begin}, src: map[string]*dmarcSource{}, srcRep: map[string]map[string]bool{}, reporters: map[string]bool{}}
+			by[r.Domain] = a
+		}
+		d := a.d
+		d.Reports++
+		if r.Begin.Before(d.First) {
+			d.First = r.Begin
+		}
+		if r.End.After(a.lastEnd) { // the newest report's policy is the one in force
+			a.lastEnd, d.Last, d.Policy, d.Pct = r.End, r.End, r.Policy.P, r.Policy.Pct
+		}
+		a.reporters[r.Org] = true
+		for _, rec := range r.Records {
+			s := a.src[rec.SourceIP]
+			if s == nil {
+				s = &dmarcSource{IP: rec.SourceIP, Ours: ours[rec.SourceIP]}
+				a.src[rec.SourceIP] = s
+				a.srcRep[rec.SourceIP] = map[string]bool{}
+			}
+			a.srcRep[rec.SourceIP][r.Org] = true
+			s.Messages += rec.Count
+			d.Messages += rec.Count
+			if rec.Passed() {
+				s.Passed += rec.Count
+				d.Passed += rec.Count
+			}
+			if rec.DKIM == "pass" {
+				s.DKIMPass += rec.Count
+			}
+			if rec.SPF == "pass" {
+				s.SPFPass += rec.Count
+			}
+			if rec.Disposition == "quarantine" || rec.Disposition == "reject" {
+				s.Rejected += rec.Count
+			}
+			if s.Ours {
+				d.OurMsgs += rec.Count
+				if rec.Passed() {
+					d.OurPassed += rec.Count
+				}
+			}
+		}
+	}
+	var out []dmarcDomain
+	for _, a := range by {
+		d := a.d
+		for ip, s := range a.src {
+			s.Reporters = len(a.srcRep[ip])
+			s.Hint = sourceHint(s, d.Domain)
+			d.Sources = append(d.Sources, *s)
+		}
+		sort.Slice(d.Sources, func(i, j int) bool {
+			fi, fj := d.Sources[i].Messages-d.Sources[i].Passed, d.Sources[j].Messages-d.Sources[j].Passed
+			if fi != fj {
+				return fi > fj
+			}
+			return d.Sources[i].Messages > d.Sources[j].Messages
+		})
+		for r := range a.reporters {
+			d.Reporters = append(d.Reporters, r)
+		}
+		sort.Strings(d.Reporters)
+		d.Advice = dmarcAdvice(d)
+		out = append(out, *d)
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].Domain < out[j].Domain })
+	return out
+}
+
+func sourceHint(s *dmarcSource, domain string) string {
+	failed := s.Messages - s.Passed
+	if failed == 0 {
+		return ""
+	}
+	switch {
+	case s.Ours && s.DKIMPass == 0 && s.SPFPass == 0:
+		return "Your own relay's mail failed both SPF and DKIM alignment. Check that " + domain + " has a published DKIM key here and an SPF record authorizing " + s.IP + "."
+	case s.Ours && s.DKIMPass == 0:
+		return "Mail from your relay only passes through SPF. Create and publish a DKIM key for " + domain + " so it survives forwarding."
+	case s.Ours:
+		return "Some mail from your relay failed. It is probably mail signed or sent under a different domain than the From address."
+	case s.Rejected >= failed:
+		return "Receivers are already quarantining or rejecting this sender. If it isn't yours, your policy is working; if it is, add it to your SPF record and set up DKIM with it."
+	default:
+		return "A sender outside your relays is failing. If it's a service you use (a newsletter or helpdesk tool), add its SPF include and set up its DKIM; if you don't recognise it, someone may be spoofing your domain and a stricter policy would stop that."
+	}
+}
+
+func dmarcAdvice(d *dmarcDomain) []string {
+	var out []string
+	failedOther := 0
+	for _, s := range d.Sources {
+		if !s.Ours {
+			failedOther += s.Messages - s.Passed
+		}
+	}
+	if d.OurMsgs >= dmarcAlertMinMsg && float64(d.OurPassed)/float64(d.OurMsgs) < dmarcAlertBelow {
+		out = append(out, fmt.Sprintf("Only %d%% of the mail from your own relays passed DMARC. Fix this before tightening the policy.", 100*d.OurPassed/d.OurMsgs))
+	}
+	if d.Policy == "none" && d.Messages >= dmarcAlertMinMsg && failedOther == 0 && (d.OurMsgs == 0 || d.OurPassed == d.OurMsgs) {
+		out = append(out, "Everything that sent as "+d.Domain+" passed. You can move the policy from p=none to p=quarantine, then p=reject.")
+	}
+	if d.Policy == "none" && failedOther > 0 {
+		out = append(out, fmt.Sprintf("%d messages from other senders failed. Identify them below before you enforce a policy, or legitimate mail will be blocked.", failedOther))
+	}
+	if d.Policy != "none" && d.Pct > 0 && d.Pct < 100 {
+		out = append(out, fmt.Sprintf("The policy applies to only %d%% of mail (pct=%d). Raise it to 100 once reports look clean.", d.Pct, d.Pct))
+	}
+	return out
+}
+
+// ---------- API ----------
+
+func (s *Server) orgEgressIPsLocked(org string) map[string]bool {
+	ips := map[string]bool{}
+	for _, x := range s.st.MailEgress {
+		if x.OrgID == org {
+			ips[x.SourceIP] = true
+		}
+	}
+	return ips
+}
+
+func (s *Server) dmarcDomainsLocked(org string) map[string]bool {
+	doms := map[string]bool{}
+	for _, k := range s.st.DKIM {
+		if k.OrgID == org {
+			doms[strings.ToLower(k.Domain)] = true
+		}
+	}
+	return doms
+}
+
+// dmarcForOrg summarizes an org's recent reports. It reads the database without holding s.mu.
+func (s *Server) dmarcForOrg(org string, ours map[string]bool) []dmarcDomain {
+	return summarizeDMARC(s.db.dmarcReports(org, time.Now().AddDate(0, 0, -dmarcWindowDays)), ours)
+}
+
+func (s *Server) listDMARC(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	s.mu.Lock()
+	ours := s.orgEgressIPsLocked(c.Org.ID)
+	s.mu.Unlock()
+	out := s.dmarcForOrg(c.Org.ID, ours)
+	if out == nil {
+		out = []dmarcDomain{}
+	}
+	writeJSON(w, 200, out)
+}
+
+// uploadDMARC accepts one or more reports in the request body (a report, or a whole email with one attached).
+func (s *Server) uploadDMARC(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, maxDMARCUpload))
+	if err != nil {
+		fail(w, 413, errors.New("that file is too large to be a DMARC report"))
+		return
+	}
+	rep, err := dmarc.Parse(body)
+	if err != nil {
+		fail(w, 400, err)
+		return
+	}
+	s.mu.Lock()
+	known := s.dmarcDomainsLocked(c.Org.ID)[rep.Domain]
+	s.mu.Unlock()
+	if !known {
+		failf(w, 400, "this report is about %s, which isn't a domain you have a DKIM key for. Create the key first, or check the report is yours.", rep.Domain)
+		return
+	}
+	added, err := s.db.addDMARC(c.Org.ID, rep)
+	if err != nil {
+		fail(w, 500, err)
+		return
+	}
+	if added {
+		s.audit(c.Org.ID, c.Actor, "dmarc.upload", rep.Domain+" from "+rep.Org)
+	}
+	msgs := 0
+	for _, rec := range rep.Records {
+		msgs += rec.Count
+	}
+	writeJSON(w, 200, map[string]any{"domain": rep.Domain, "reporter": rep.Org, "messages": msgs, "duplicate": !added})
+}
+
+func (s *Server) deleteDMARC(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	domain := strings.ToLower(r.PathValue("domain"))
+	if _, err := s.db.deleteDMARCDomain(c.Org.ID, domain); err != nil {
+		fail(w, 500, err)
+		return
+	}
+	s.audit(c.Org.ID, c.Actor, "dmarc.delete", domain)
+	ok(w)
+}
+
+// dmarcProblems derives the alerts from reports: the mail our own relays send
+// as a domain is failing DMARC. Caller holds s.mu; the database is read inside,
+// which takes no server locks.
+func (s *Server) dmarcProblemsLocked(add func(key, org, kind, name, msg string)) {
+	orgs := map[string]bool{}
+	for _, k := range s.st.DKIM {
+		orgs[k.OrgID] = true
+	}
+	for org := range orgs {
+		for _, d := range s.dmarcForOrg(org, s.orgEgressIPsLocked(org)) {
+			if d.OurMsgs >= dmarcAlertMinMsg && float64(d.OurPassed)/float64(d.OurMsgs) < dmarcAlertBelow {
+				add("dmarcrate:"+org+":"+d.Domain, org, "mail", d.Domain,
+					fmt.Sprintf("Only %d%% of the mail your relays sent as %s passed DMARC in the last %d days (%d of %d messages). Open Mail → DMARC reports for the failing senders.",
+						100*d.OurPassed/d.OurMsgs, d.Domain, dmarcWindowDays, d.OurPassed, d.OurMsgs))
+			}
+		}
+	}
+}
+
+// ---------- findings for the dashboard ----------
+
+// mailEgressFindings lists the checklist for one egress. Caller holds s.mu.
+func (s *Server) mailEgressFindingsLocked(x *model.MailEgress, chk model.MailCheck) []Finding {
+	return deliverabilityFindings(x, chk, s.st.DKIM)
+}

dmarc_test.go

@@ -0,0 +1,158 @@
+package hammerhead
+
+import (
+	"bytes"
+	"compress/gzip"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strconv"
+	"strings"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/reef/dmarc"
+	"git.veillax.com/Veillax/reef/model"
+)
+
+func reportXML(domain, id, reporter string, rows ...[4]string) string { // ip, count, dkim, spf
+	var sb strings.Builder
+	sb.WriteString(`<feedback><report_metadata><org_name>` + reporter + `</org_name><report_id>` + id + `</report_id><date_range><begin>` +
+		strconv.FormatInt(time.Now().Add(-26*time.Hour).Unix(), 10) + `</begin><end>` + strconv.FormatInt(time.Now().Add(-2*time.Hour).Unix(), 10) + `</end></date_range></report_metadata><policy_published><domain>` + domain + `</domain><p>none</p><pct>100</pct></policy_published>`)
+	for _, r := range rows {
+		disp := "none"
+		if r[2] != "pass" && r[3] != "pass" {
+			disp = "quarantine"
+		}
+		sb.WriteString(`<record><row><source_ip>` + r[0] + `</source_ip><count>` + r[1] + `</count><policy_evaluated><disposition>` + disp + `</disposition><dkim>` + r[2] + `</dkim><spf>` + r[3] + `</spf></policy_evaluated></row><identifiers><header_from>` + domain + `</header_from></identifiers></record>`)
+	}
+	sb.WriteString(`</feedback>`)
+	return sb.String()
+}
+
+func gzipped(s string) []byte {
+	var b bytes.Buffer
+	w := gzip.NewWriter(&b)
+	w.Write([]byte(s))
+	w.Close()
+	return b.Bytes()
+}
+
+// dmarcFx is an org that sends as example.com from 203.0.113.7.
+func dmarcFx(t *testing.T) (*fx, *orgCtx) {
+	f := newFx(t)
+	f.s.mu.Lock()
+	f.s.st.DKIM = append(f.s.st.DKIM, &model.DKIMKey{ID: "k1", OrgID: f.org, Domain: "example.com", Selector: "s1"})
+	f.s.st.MailEgress = append(f.s.st.MailEgress, &model.MailEgress{ID: "x1", OrgID: f.org, Name: "relay", SiteID: "s1", EdgeID: "e1", SourceIP: "203.0.113.7", HeloName: "mail.example.com", Enabled: true})
+	org := f.s.findOrg(f.org)
+	f.s.mu.Unlock()
+	return f, &orgCtx{Org: *org, Actor: "test"}
+}
+
+func upload(t *testing.T, f *fx, c *orgCtx, body []byte) (int, map[string]any) {
+	w := httptest.NewRecorder()
+	f.s.uploadDMARC(w, httptest.NewRequest("POST", "/", bytes.NewReader(body)), c)
+	var out map[string]any
+	json.Unmarshal(w.Body.Bytes(), &out)
+	return w.Code, out
+}
+
+func TestUploadAndSummarize(t *testing.T) {
+	f, c := dmarcFx(t)
+	code, out := upload(t, f, c, gzipped(reportXML("example.com", "r1", "google.com", [4]string{"203.0.113.7", "30", "pass", "pass"}, [4]string{"198.51.100.9", "5", "fail", "fail"})))
+	if code != 200 || out["messages"] != float64(35) || out["duplicate"] != false {
+		t.Fatalf("%d %v", code, out)
+	}
+	if _, out = upload(t, f, c, []byte(reportXML("example.com", "r1", "google.com"))); out["duplicate"] != true {
+		t.Errorf("the same report twice should be a duplicate: %v", out)
+	}
+	upload(t, f, c, []byte(reportXML("example.com", "r2", "yahoo.com", [4]string{"203.0.113.7", "10", "pass", "fail"})))
+
+	w := httptest.NewRecorder()
+	f.s.listDMARC(w, httptest.NewRequest("GET", "/", nil), c)
+	var doms []dmarcDomain
+	if err := json.Unmarshal(w.Body.Bytes(), &doms); err != nil || len(doms) != 1 {
+		t.Fatalf("%v %s", err, w.Body.String())
+	}
+	d := doms[0]
+	if d.Reports != 2 || d.Messages != 45 || d.Passed != 40 || d.OurMsgs != 40 || d.OurPassed != 40 {
+		t.Errorf("totals: %+v", d)
+	}
+	if len(d.Sources) != 2 || d.Sources[0].IP != "198.51.100.9" || d.Sources[0].Ours || d.Sources[0].Hint == "" {
+		t.Errorf("failing sources should come first with a hint: %+v", d.Sources)
+	}
+	if !strings.Contains(strings.Join(d.Advice, " "), "failed") {
+		t.Errorf("advice should flag the unknown sender: %v", d.Advice)
+	}
+}
+
+func TestUploadRefusals(t *testing.T) {
+	f, c := dmarcFx(t)
+	if code, _ := upload(t, f, c, []byte("garbage")); code != 400 {
+		t.Errorf("garbage: %d", code)
+	}
+	code, out := upload(t, f, c, []byte(reportXML("someone-else.org", "r9", "google.com", [4]string{"1.2.3.4", "1", "pass", "pass"})))
+	if code != 400 || !strings.Contains(out["error"].(string), "someone-else.org") {
+		t.Errorf("a domain the org doesn't send from: %d %v", code, out)
+	}
+	big := make([]byte, maxDMARCUpload+1)
+	w := httptest.NewRecorder()
+	f.s.uploadDMARC(w, httptest.NewRequest("POST", "/", bytes.NewReader(big)), c)
+	if w.Code != http.StatusRequestEntityTooLarge {
+		t.Errorf("oversize: %d", w.Code)
+	}
+}
+
+func TestReportsStayInTheirOrg(t *testing.T) {
+	f, c := dmarcFx(t)
+	upload(t, f, c, []byte(reportXML("example.com", "r1", "google.com", [4]string{"203.0.113.7", "5", "pass", "pass"})))
+	if n := len(f.s.db.dmarcReports("another-org", time.Time{})); n != 0 {
+		t.Errorf("another org sees %d reports", n)
+	}
+	if n, _ := f.s.db.deleteDMARCDomain(f.org, "example.com"); n != 1 {
+		t.Errorf("delete removed %d", n)
+	}
+}
+
+func TestSummarizeAdvice(t *testing.T) {
+	rep := func(p string, pct int, rows ...dmarc.Record) *dmarc.Report {
+		return &dmarc.Report{Org: "g", ReportID: "1", Domain: "example.com", Begin: time.Unix(1, 0), End: time.Unix(2, 0), Policy: dmarc.Policy{P: p, Pct: pct}, Records: rows}
+	}
+	ours := map[string]bool{"203.0.113.7": true}
+	pass := dmarc.Record{SourceIP: "203.0.113.7", Count: 50, DKIM: "pass", SPF: "pass"}
+
+	got := summarizeDMARC([]*dmarc.Report{rep("none", 100, pass)}, ours)[0]
+	if !strings.Contains(strings.Join(got.Advice, " "), "p=quarantine") {
+		t.Errorf("a clean domain at p=none should be told to tighten: %v", got.Advice)
+	}
+	bad := dmarc.Record{SourceIP: "203.0.113.7", Count: 50, DKIM: "fail", SPF: "fail", Disposition: "quarantine"}
+	got = summarizeDMARC([]*dmarc.Report{rep("quarantine", 50, pass, bad)}, ours)[0]
+	joined := strings.Join(got.Advice, " ")
+	if !strings.Contains(joined, "50%") || !strings.Contains(joined, "pct=50") {
+		t.Errorf("advice: %v", got.Advice)
+	}
+	got = summarizeDMARC([]*dmarc.Report{rep("quarantine", 100, bad)}, ours)[0]
+	if got.Sources[0].Hint == "" || !strings.Contains(got.Sources[0].Hint, "SPF and DKIM") {
+		t.Errorf("own relay failing both: %q", got.Sources[0].Hint)
+	}
+}
+
+func TestOwnMailFailingDMARCRaisesAnAlert(t *testing.T) {
+	f, c := dmarcFx(t)
+	upload(t, f, c, []byte(reportXML("example.com", "r1", "google.com", [4]string{"203.0.113.7", "40", "fail", "fail"}, [4]string{"203.0.113.7", "10", "pass", "pass"})))
+	f.s.mu.Lock()
+	probs := f.s.problemsLocked(f.base)
+	f.s.mu.Unlock()
+	p, ok := probs["dmarcrate:"+f.org+":example.com"]
+	if !ok || !strings.Contains(p.msg, "20%") {
+		t.Fatalf("expected a DMARC pass-rate alert, got %v", probs)
+	}
+	// a healthy domain raises none
+	upload(t, f, c, []byte(reportXML("example.com", "r2", "yahoo.com", [4]string{"203.0.113.7", "500", "pass", "pass"})))
+	f.s.mu.Lock()
+	probs = f.s.problemsLocked(f.base)
+	f.s.mu.Unlock()
+	if _, ok := probs["dmarcrate:"+f.org+":example.com"]; ok {
+		t.Errorf("alert should clear once the pass rate recovers: %v", probs)
+	}
+}

docs/api-automation.md

@@ -0,0 +1,37 @@
+# Scripting Hammerhead: API audit
+
+Everything the dashboard does goes through the JSON API (`server.go` lists the routes). Callers
+authenticate with the `hh_session` cookie or `Authorization: Bearer hhk_<id>_<secret>`.
+
+## Credentials
+
+| Credential | Reaches | Made at |
+|---|---|---|
+| Organization key | that organization's `/api/orgs/{org}/...` routes (admin rights), `GET /api/orgs`, `GET /api/edges` (edges the org may use) | `POST /api/orgs/{org}/apikeys` |
+| Server key | every organization, plus the server-admin routes: edges, `/api/admin/alerts`, users, settings, `POST /api/orgs` | `POST /api/admin/apikeys` (signed-in server admin only) |
+
+The secret is shown once, in the `key` field of the create response. A key cannot mint keys at server
+level, and a server key has no user, so `/api/me` and other per-user routes refuse it.
+
+## Gaps found, and what was done
+
+| Gap | Resolution |
+|---|---|
+| Edges, server alerts, `POST /api/orgs` required a browser session (`withServerAdmin` refused keys) | server keys, accepted by `withServerAdmin` |
+| No way to mint an unattended server credential | `GET/POST/DELETE /api/admin/apikeys` |
+| `GET /api/edges` was session-only | accepts keys; an org key sees the edges its org may use |
+| No way to list orgs with a key | `GET /api/orgs` |
+| Key creation did not return the key's ID | create response now has `id` as well as `key` |
+| Creating an alert channel returned only `{ok:true}` | response now includes the channel `id` |
+| `createOrg` added the caller as a member | skipped for server keys (they have no user) |
+
+## Known remaining gaps (not blocking scripting)
+
+- No per-resource `GET`: sites, domains, ports and alert channels are read from `GET /api/orgs/{org}/state` and
+  `GET /api/orgs/{org}/alerts`. Clients filter by ID.
+- Alert channel secrets (webhook URL, Sandbar key) are never returned, so a client can't detect drift in them; a
+  blank value on `PUT` keeps the stored one.
+- Keys are all-powerful within their scope (no read-only or per-resource scopes), and have no expiry.
+- `PUT` on ports replaces the whole object; omitted fields reset.
+- Site tokens are returned on create and, for admins, in `state`.
+- Deleting an edge or site cascades silently (its ports and domains go with it, as do ports left with no target), so Terraform plans should not rely on dependent resources surviving.

docs/design.md

@@ -0,0 +1,64 @@
+# Hammerhead design language
+
+The dashboard and the visitor sign-in page follow the Lightwell design language. The reference is `docs/design.md` in the Lightwell repository. This page records how that language is applied here and where Hammerhead needs its own pieces. When the two disagree, Lightwell's document wins.
+
+## What is shared with Lightwell
+
+* **Tokens.** Colors, spacing (4px scale), radius (6px and 10px), type sizes (12 to 28px), the single shadow and the dark theme are the Lightwell values. The accent is Lightwell's teal, `#1f5c5c`, with white text on it. Everything derived from the accent uses `color-mix`.
+* **Principles.** Quiet neutrals with one accent, plain and specific wording, one way to do each thing, calm density, and WCAG 2.1 AA.
+* **Voice.** No emoji, no em or en dashes, no exclamation marks, sentence case, buttons that name the result, and errors that say what happened and what to do next. Errors that come from the server are capitalized in one place (`fail` in `store.go`).
+* **Status carries text.** A status is a badge with a label ("Online", "Listening", "Down"), never a colored dot on its own.
+
+## Where it lives
+
+| File | What it holds |
+| --- | --- |
+| `web/index.html` | The dashboard. The token block sits between `tokens:begin` and `tokens:end`; raw colors are allowed only there. |
+| `web/resource.html` | The sign-in page that visitors see in front of a protected resource. Same token block. |
+
+Both are single files with no build step, so the token block is repeated in each. Change a token in both.
+
+## Components
+
+Built from the helpers at the top of the script in `web/index.html`. Feature code uses these and does not style one-off.
+
+| Helper | Use |
+| --- | --- |
+| `card`, `pageHeader` | Surfaces and the page title with its one primary action |
+| `field`, `chk` | Label above the control, hint below |
+| `badge(text, tone)` | Status pill. Tones: success, warning, danger, accent, or none for neutral |
+| `emptyState(icon, title, text, action)` | Dashed outline, icon, a title that says "No X yet", one sentence |
+| `modal`, `dialogForm` | Dialogs. Focus moves in, Tab stays inside, Escape closes, focus returns to the opener |
+| `confirmDialog`, `ask` | Every destructive action. The title names the object, the body says what is lost and whether it can be undone |
+| `toast` | Confirmations at the bottom right. Errors stay longer and are announced as alerts |
+| `skeleton` | Placeholder rows while a page loads, not a spinner |
+| `updatePanel` | Progress bar and live log for an update |
+
+## Icons
+
+Lucide style, outline only, stroke 1.75, round caps, `currentColor`. Sizes are 16, 18 and 22 pixels. They are inline in the `ICONS` table. An icon goes with a text label, and icon-only buttons carry an `aria-label`.
+
+Approved names: `fin`, `mine`, `ports`, `sites`, `domains`, `access`, `logs`, `edges`, `orgs`, `users`, `idps`, `settings`, `audit`, `account`, `egress`, `alerts`, `out`, `menu`, `close`, `add`, `check`, `light`, `dark`, `alert`. Add a name here and in `ICONS` together.
+
+## Patterns
+
+* **One primary action per view.** It sits in the page header. Smaller forms inside a page use the default button.
+* **List pages** show the page header, then the table in a card with no second title.
+* **Dialogs and forms** end with Cancel and then the primary action, at the bottom right. Labels sit above fields, and rules such as "At least 8 characters" are hints below them, not part of the label.
+* **Dates** have the month written out ("Nov 2, 2026"), with `fmtDate` and `fmtTime`.
+* **Empty states** say why the list is empty: nothing yet, or nothing matching the filters.
+* **Long machine messages**, such as certificate errors, wrap and stop at three lines (`.clip`), with the full text in a tooltip.
+
+## Layout
+
+* Admin content is capped at 1120px. Below 860px the sidebar becomes a slide-in drawer with a top bar, and grids collapse. Phones keep a 16px side gutter.
+* On touch screens controls are at least 44px high and inputs are 16px so the page does not zoom.
+* Motion is minimal and turns off under `prefers-reduced-motion`. The theme follows the system and can be switched from the sidebar.
+
+## Checking a change
+
+```
+node scripts/lint-design.mjs
+```
+
+It fails on em and en dashes, emoji, and raw colors outside the token block, in the pages, the docs and the Go source. Then check the page in light and dark, at phone and desktop width, using only the keyboard.

e2e/e2e_test.go

@@ -241,6 +241,9 @@ func selfSigned(t *testing.T, host string) *tls.Certificate {
 
 // ---------- the environment ----------
 
+// siteCfgHook lets a test adjust the Bamboo that setup starts.
+var siteCfgHook func(*bamboo.Config)
+
 type world struct {
 	t       *testing.T
 	ts      *httptest.Server
@@ -297,7 +300,11 @@ func setup(t *testing.T, tr blacktip.TraefikConfig, opts ...func(*blacktip.Confi
 		t.Fatal(err)
 	}
 	go bt.Run(ctx)
-	bb, err := bamboo.New(bamboo.Config{Hammerhead: w.ts.URL, Token: site.Token, DataDir: t.TempDir()})
+	bbCfg := bamboo.Config{Hammerhead: w.ts.URL, Token: site.Token, DataDir: t.TempDir()}
+	if siteCfgHook != nil {
+		siteCfgHook(&bbCfg)
+	}
+	bb, err := bamboo.New(bbCfg)
 	if err != nil {
 		t.Fatal(err)
 	}

e2e/expose_test.go

@@ -0,0 +1,96 @@
+package e2e
+
+import (
+	"context"
+	"net"
+	"strconv"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/bamboo"
+	"git.veillax.com/Veillax/blacktip"
+)
+
+type exposed struct {
+	ID, Host, Hostname, URL, SiteToken string
+	Port                               int
+}
+
+// createExpose retries until an edge is online and the call succeeds.
+func (w *world) createExpose(c *client, in map[string]any) exposed {
+	w.t.Helper()
+	var out exposed
+	for i := 0; i < 40; i++ {
+		if code, _ := c.do("POST", "/api/orgs/-/expose", in, &out); code == 200 {
+			return out
+		}
+		time.Sleep(150 * time.Millisecond)
+	}
+	c.must("POST", "/api/orgs/-/expose", in, &out)
+	return out
+}
+
+func (w *world) agentFor(e exposed) context.CancelFunc {
+	ctx, cancel := context.WithCancel(context.Background())
+	w.t.Cleanup(cancel)
+	b, err := bamboo.New(bamboo.Config{Hammerhead: w.ts.URL, Token: e.SiteToken, DataDir: w.t.TempDir()})
+	if err != nil {
+		w.t.Fatal(err)
+	}
+	go b.Run(ctx)
+	return cancel
+}
+
+// An ephemeral exposure carries traffic through the tunnel, and is gone after DELETE.
+func TestEphemeralExpose(t *testing.T) {
+	w := setup(t, blacktip.TraefikConfig{})
+	var key struct{ Key string }
+	w.admin.must("POST", "/api/orgs/"+w.org+"/apikeys", map[string]string{"name": "shark"}, &key)
+	kc := newClient(t, w.ts.URL)
+	kc.key = key.Key
+	w.admin.must("POST", "/api/orgs/"+w.org+"/domains", map[string]any{"name": "example.test", "edgeId": w.edge}, nil)
+
+	// ----- tcp -----
+	svc := tcpBackend(t, "SVC")
+	e := w.createExpose(kc, map[string]any{"proto": "tcp", "targetPort": svc, "ttlSeconds": 60})
+	stop := w.agentFor(e)
+	addr := net.JoinHostPort("127.0.0.1", strconv.Itoa(e.Port))
+	if got := tcpAsk(t, addr, "hi"); got != "SVC:hi" {
+		t.Fatalf("through the tunnel: %q", got)
+	}
+	kc.must("POST", "/api/orgs/-/expose/"+e.ID+"/renew", nil, nil)
+	kc.must("DELETE", "/api/orgs/-/expose/"+e.ID, nil, nil)
+	stop()
+	closed := false
+	for i := 0; i < 40 && !closed; i++ {
+		c, err := net.DialTimeout("tcp", addr, 300*time.Millisecond)
+		if err != nil {
+			closed = true
+			break
+		}
+		c.Close()
+		time.Sleep(150 * time.Millisecond)
+	}
+	if !closed {
+		t.Error("the port is still open after DELETE")
+	}
+	kc.expect(404, "GET", "/api/orgs/-/expose/"+e.ID, nil)
+
+	// ----- http, with a password -----
+	web := webBackend(t, "app", 200)
+	port := freePort(t)
+	h := w.createExpose(kc, map[string]any{"proto": "http", "name": "demo", "port": port, "targetPort": web,
+		"auth": map[string]any{"password": "hunter22"}})
+	w.agentFor(h)
+	_, get := edgeClient("http", port)
+	var r webResp
+	for i := 0; i < 60; i++ {
+		if r = get("GET", h.Hostname, "/x"); r.status == 302 {
+			break
+		}
+		time.Sleep(150 * time.Millisecond)
+	}
+	if r.status != 302 {
+		t.Fatalf("protected exposure should send visitors to the login, got %+v", r)
+	}
+}

e2e/ssh_test.go

@@ -0,0 +1,108 @@
+package e2e
+
+import (
+	"context"
+	"crypto/ed25519"
+	"crypto/rand"
+	"fmt"
+	"net"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/bamboo/sshd"
+	"git.veillax.com/Veillax/blacktip"
+	"git.veillax.com/Veillax/reef/model"
+	"golang.org/x/crypto/ssh"
+)
+
+func newSSHSigner(t *testing.T) ssh.Signer {
+	t.Helper()
+	_, priv, err := ed25519.GenerateKey(rand.Reader)
+	if err != nil {
+		t.Fatal(err)
+	}
+	s, err := ssh.NewSignerFromKey(priv)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return s
+}
+
+// A TCP port on the edge whose target is Bamboo's SSH server at the site reaches that server
+// with an ordinary ssh client: the path is client, edge, WireGuard tunnel, Bamboo, sshd.
+func TestSSHThroughTheEdgeToBamboosServer(t *testing.T) {
+	w := setup(t, blacktip.TraefikConfig{})
+	dir := t.TempDir()
+	signer := newSSHSigner(t)
+	keys := filepath.Join(dir, "authorized_keys")
+	if err := os.WriteFile(keys, ssh.MarshalAuthorizedKey(signer.PublicKey()), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	srv, err := sshd.New(sshd.Config{AuthorizedKeys: keys, HostKey: filepath.Join(dir, "ssh_host_ed25519_key")})
+	if err != nil {
+		t.Fatal(err)
+	}
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		t.Fatal(err)
+	}
+	ctx, cancel := context.WithCancel(context.Background())
+	t.Cleanup(cancel)
+	go srv.Serve(ctx, ln)
+
+	edgePort := freePort(t)
+	w.open(model.Mapping{Proto: model.ProtoTCP, Port: edgePort, Name: "ssh", Targets: tgt(w.site, ln.Addr().(*net.TCPAddr).Port)})
+	addr := "127.0.0.1:" + strconv.Itoa(edgePort)
+
+	connect := func(auth ssh.AuthMethod) (*ssh.Client, error) {
+		return ssh.Dial("tcp", addr, &ssh.ClientConfig{User: "alex", Auth: []ssh.AuthMethod{auth}, Timeout: 5 * time.Second,
+			// the key the client sees through the tunnel must be Bamboo's, not something in between
+			HostKeyCallback: func(_ string, _ net.Addr, k ssh.PublicKey) error {
+				if got := ssh.FingerprintSHA256(k); got != srv.Fingerprint() {
+					return fmt.Errorf("host key %s is not the Bamboo server's %s", got, srv.Fingerprint())
+				}
+				return nil
+			}})
+	}
+	var client *ssh.Client
+	var last error
+	for i := 0; i < 60 && client == nil; i++ {
+		if client, last = connect(ssh.PublicKeys(signer)); client == nil {
+			time.Sleep(250 * time.Millisecond)
+		}
+	}
+	if client == nil {
+		t.Fatalf("could not sign in through the edge: %v", last)
+	}
+	defer client.Close()
+
+	if sshd.Supported {
+		sess, err := client.NewSession()
+		if err != nil {
+			t.Fatal(err)
+		}
+		out, err := sess.Output("echo through-the-tunnel")
+		sess.Close()
+		if err != nil || strings.TrimSpace(string(out)) != "through-the-tunnel" {
+			t.Fatalf("running a command: %q, %v", out, err)
+		}
+	}
+
+	// the same narrow rules hold through the tunnel
+	if c, err := connect(ssh.PublicKeys(newSSHSigner(t))); err == nil {
+		c.Close()
+		t.Error("a key that is not in authorized_keys signed in")
+	}
+	if c, err := connect(ssh.Password("hunter2")); err == nil {
+		c.Close()
+		t.Error("a password signed in")
+	}
+	if c, err := client.Dial("tcp", "127.0.0.1:1"); err == nil {
+		c.Close()
+		t.Error("port forwarding through the SSH server is allowed")
+	}
+}

e2e/traffic_test.go

@@ -0,0 +1,80 @@
+package e2e
+
+import (
+	"strconv"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/blacktip"
+	"git.veillax.com/Veillax/reef/model"
+)
+
+// Traffic through an edge shows up in Hammerhead's charts, with the client address named.
+func TestTrafficAnalyticsFlow(t *testing.T) {
+	w := setup(t, blacktip.TraefikConfig{})
+	target := freeRun(t, 1)
+	pub := freeRun(t, 1)
+	if pub == target {
+		pub = freeRun(t, 2) + 1
+	}
+	tcpOn(t, target)
+	var m model.Mapping
+	if code, raw := w.admin.do("POST", "/api/orgs/"+w.org+"/mappings", map[string]any{"proto": "tcp", "edgeId": w.edge, "bindIp": "127.0.0.1", "port": pub,
+		"targets": tgt(w.local, target), "enabled": true}, &m); code != 200 {
+		t.Fatalf("open: %d %s", code, raw)
+	}
+	for i := 0; i < 3; i++ {
+		if got := tcpAsk(t, "127.0.0.1:"+strconv.Itoa(pub), "hello"); got == "" {
+			t.Fatal("no answer through the edge")
+		}
+	}
+
+	type detail struct {
+		Total   struct{ Conns, BytesIn, BytesOut int64 } `json:"total"`
+		Points  []struct{ Conns int64 }                  `json:"points"`
+		Sources []struct {
+			IP    string
+			Conns int64
+		} `json:"sources"`
+	}
+	var d detail
+	for i := 0; ; i++ { // the edge reports every 20 seconds
+		d = detail{}
+		if code, _ := w.admin.do("GET", "/api/orgs/"+w.org+"/traffic/"+m.ID+"?range=1h", nil, &d); code == 200 && d.Total.Conns >= 3 {
+			break
+		}
+		if i >= 450 {
+			t.Fatal("timed out waiting for the edge to report the traffic")
+		}
+		time.Sleep(100 * time.Millisecond)
+	}
+	if d.Total.BytesIn < 15 || d.Total.BytesOut == 0 || len(d.Points) != 60 {
+		t.Errorf("totals: %+v with %d points", d.Total, len(d.Points))
+	}
+	if len(d.Sources) != 1 || d.Sources[0].IP != "127.0.0.1" || d.Sources[0].Conns < 3 {
+		t.Errorf("sources: %+v", d.Sources)
+	}
+
+	var all detail
+	if code, raw := w.admin.do("GET", "/api/orgs/"+w.org+"/traffic/all?range=24h", nil, &all); code != 200 || all.Total.Conns < 3 || len(all.Points) != 24 {
+		t.Errorf("organization total: %d %s", code, raw)
+	}
+	if code, _ := w.admin.do("GET", "/api/orgs/"+w.org+"/traffic/"+m.ID+"?range=5y", nil, nil); code != 400 {
+		t.Errorf("bad range: %d", code)
+	}
+	if code, _ := w.admin.do("GET", "/api/orgs/"+w.org+"/traffic/nope", nil, nil); code != 404 {
+		t.Errorf("unknown port: %d", code)
+	}
+
+	alerts := "/api/orgs/" + w.org + "/alerts"
+	if code, _ := w.admin.do("PUT", alerts+"/traffic", map[string]any{"spikeFactor": 1}, nil); code != 400 {
+		t.Errorf("a spike of 1x was accepted: %d", code)
+	}
+	if code, raw := w.admin.do("PUT", alerts+"/traffic", map[string]any{"spikeFactor": 5, "dailyMb": 2048}, nil); code != 200 {
+		t.Fatalf("save traffic alerts: %d %s", code, raw)
+	}
+	var ov struct{ Settings map[string]int }
+	if code, _ := w.admin.do("GET", alerts, nil, &ov); code != 200 || ov.Settings["spikeFactor"] != 5 || ov.Settings["dailyMb"] != 2048 || ov.Settings["debounceMin"] != 2 {
+		t.Errorf("settings after save: %d %+v", code, ov.Settings)
+	}
+}

e2e/updates_test.go

@@ -0,0 +1,132 @@
+package e2e
+
+import (
+	"context"
+	"strings"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/bamboo"
+	"git.veillax.com/Veillax/blacktip"
+	"git.veillax.com/Veillax/reef/model"
+)
+
+type edgeRow struct {
+	ID, Version, UpdateMsg string
+	AutoUpdate             bool
+	UpdatePending          bool
+	Online                 bool
+}
+
+func (w *world) edgeRow() edgeRow {
+	var edges []edgeRow
+	w.admin.must("GET", "/api/edges", nil, &edges)
+	for _, e := range edges {
+		if e.ID == w.edge {
+			return e
+		}
+	}
+	w.t.Fatal("edge not listed")
+	return edgeRow{}
+}
+
+func (w *world) siteRow() model.Site {
+	var st struct{ Sites []model.Site }
+	w.admin.must("GET", "/api/orgs/"+w.org+"/state", nil, &st)
+	for _, s := range st.Sites {
+		if s.ID == w.site {
+			return s
+		}
+	}
+	w.t.Fatal("site not listed")
+	return model.Site{}
+}
+
+func waitFor(t *testing.T, what string, ok func() bool) {
+	t.Helper()
+	for i := 0; i < 150; i++ {
+		if ok() {
+			return
+		}
+		time.Sleep(100 * time.Millisecond)
+	}
+	t.Fatalf("timed out waiting for %s", what)
+}
+
+func TestEdgesAndSitesReportVersionsAndUpdateOnRequest(t *testing.T) {
+	var edgeCalls, siteCalls atomic.Int32
+	siteCfgHook = func(c *bamboo.Config) {
+		c.Version, c.AutoUpdate = "v0.1.0", true
+		c.UpdateNow = func(ctx context.Context, progress func(string, int, string)) (string, error) {
+			siteCalls.Add(1)
+			progress("checking", 10, "Checking for the newest release")
+			progress("done", 100, "Already on the newest release")
+			return "", nil
+		}
+	}
+	defer func() { siteCfgHook = nil }()
+	w := setup(t, blacktip.TraefikConfig{}, func(c *blacktip.Config) {
+		c.Version, c.AutoUpdate = "v0.1.0", true
+		c.UpdateNow = func(ctx context.Context, progress func(string, int, string)) (string, error) {
+			edgeCalls.Add(1)
+			progress("downloading", 40, "Downloading blacktip_v0.2.0_linux_amd64.tar.gz")
+			progress("downloading", 70, "Downloaded 28.0 of 40.0 MB (70%)")
+			progress("restarting", 100, "Installed v0.2.0; restarting into it")
+			return "v0.2.0", nil
+		}
+	})
+
+	waitFor(t, "the edge to report its version", func() bool { e := w.edgeRow(); return e.Version == "v0.1.0" && e.AutoUpdate })
+	waitFor(t, "the site to report its version", func() bool { s := w.siteRow(); return s.Version == "v0.1.0" && s.AutoUpdate })
+
+	w.admin.must("POST", "/api/edges/"+w.edge+"/update", nil, nil)
+	waitFor(t, "the edge to answer the update request", func() bool {
+		e := w.edgeRow()
+		return !e.UpdatePending && e.UpdateMsg == "updated to v0.2.0, restarting"
+	})
+	if edgeCalls.Load() != 1 {
+		t.Fatalf("the edge ran its update %d times, want 1", edgeCalls.Load())
+	}
+	var log struct {
+		Running bool
+		Stage   string
+		Pct     int
+		Lines   []string
+	}
+	w.admin.must("GET", "/api/edges/"+w.edge+"/update/log", nil, &log)
+	text := strings.Join(log.Lines, "; ")
+	if log.Running || log.Stage != "restarting" || log.Pct != 100 || !strings.Contains(text, "Update requested") || !strings.Contains(text, "Downloaded 28.0 of 40.0 MB") || !strings.Contains(text, "restarting into it") {
+		t.Fatalf("edge update log: %+v (%s)", log, text)
+	}
+
+	w.admin.must("POST", "/api/orgs/"+w.org+"/sites/"+w.site+"/update", nil, nil)
+	waitFor(t, "the site to answer the update request", func() bool {
+		s := w.siteRow()
+		return s.UpdateID == "" && s.UpdateMsg == "already on the newest release"
+	})
+	if siteCalls.Load() != 1 {
+		t.Fatalf("the site ran its update %d times, want 1", siteCalls.Load())
+	}
+	w.admin.must("GET", "/api/orgs/"+w.org+"/sites/"+w.site+"/update/log", nil, &log)
+	if log.Running || log.Stage != "done" || !strings.Contains(strings.Join(log.Lines, "; "), "Already on the newest release") {
+		t.Fatalf("site update log: %+v", log)
+	}
+
+	// asking again works and is a fresh request, not a leftover answer
+	w.admin.must("POST", "/api/edges/"+w.edge+"/update", nil, nil)
+	waitFor(t, "a second update request to be answered", func() bool { return edgeCalls.Load() == 2 && !w.edgeRow().UpdatePending })
+}
+
+func TestUpdatesNeedAutoUpdateOnTheMachine(t *testing.T) {
+	w := setup(t, blacktip.TraefikConfig{}) // no version, auto-update off
+	waitFor(t, "the edge to be online", func() bool { return w.edgeRow().Online })
+	code, body := w.admin.do("POST", "/api/edges/"+w.edge+"/update", nil, nil)
+	if code != 400 || !strings.Contains(body, "BLACKTIP_AUTO_UPDATE") {
+		t.Fatalf("update on an edge without auto-update: %d %s", code, body)
+	}
+	code, body = w.admin.do("POST", "/api/orgs/"+w.org+"/sites/"+w.local+"/update", nil, nil)
+	if code != 400 {
+		t.Fatalf("update on a local site: %d %s", code, body)
+	}
+}

e2e/wildcard_test.go

@@ -0,0 +1,66 @@
+package e2e
+
+import (
+	"context"
+	"encoding/json"
+	"io"
+	"net"
+	"net/http"
+	"strconv"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/blacktip"
+	"git.veillax.com/Veillax/reef/model"
+)
+
+// A website on every address does not take the port from a website that names an address: each
+// listens where it should, and the named address answers only for its own site.
+func TestWildcardWebsiteStepsAsideForANamedAddress(t *testing.T) {
+	w := setup(t, blacktip.TraefikConfig{})
+	w.admin.must("POST", "/api/orgs/"+w.org+"/domains", map[string]any{"name": "example.test", "edgeId": w.edge}, nil)
+	wild, named := webBackend(t, "wild", 200), webBackend(t, "named", 200)
+	port := freePort(t)
+
+	w.open(model.Mapping{Proto: "http", BindIP: "0.0.0.0", Port: port, Hostname: "wild.example.test", Targets: tgt(w.site, wild)})
+	w.open(model.Mapping{Proto: "http", BindIP: "127.0.0.2", Port: port, Hostname: "named.example.test", Targets: tgt(w.site, named)})
+
+	ask := func(ip, host string) (int, string) {
+		c := &http.Client{Timeout: 3 * time.Second, Transport: &http.Transport{DisableKeepAlives: true,
+			DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
+				return (&net.Dialer{}).DialContext(ctx, "tcp", net.JoinHostPort(ip, strconv.Itoa(port)))
+			}}}
+		req, _ := http.NewRequest("GET", "http://"+host+"/", nil)
+		resp, err := c.Do(req)
+		if err != nil {
+			return 0, err.Error()
+		}
+		defer resp.Body.Close()
+		b, _ := io.ReadAll(resp.Body)
+		var body map[string]string
+		json.Unmarshal(b, &body)
+		return resp.StatusCode, body["backend"]
+	}
+	wait := func(ip, host, wantBackend string) {
+		t.Helper()
+		var code int
+		var got string
+		for i := 0; i < 60; i++ {
+			if code, got = ask(ip, host); code == 200 && got == wantBackend {
+				return
+			}
+			time.Sleep(150 * time.Millisecond)
+		}
+		t.Fatalf("%s via %s: status %d backend %q, want %q", host, ip, code, got, wantBackend)
+	}
+
+	wait("127.0.0.2", "named.example.test", "named") // the named address serves its own site
+	wait("127.0.0.1", "wild.example.test", "wild")   // the wildcard still covers the other addresses
+	// and the wildcard gave up the named address: its site is not served there
+	if code, _ := ask("127.0.0.2", "wild.example.test"); code == 200 {
+		t.Errorf("the wildcard website still answers on the address another website names")
+	}
+	if code, _ := ask("127.0.0.1", "named.example.test"); code == 200 {
+		t.Errorf("a website answers on an address it was not given")
+	}
+}

expose.go

@@ -0,0 +1,389 @@
+package hammerhead
+
+import (
+	"errors"
+	"fmt"
+	"math/rand/v2"
+	"net"
+	"net/http"
+	"regexp"
+	"strings"
+	"time"
+
+	"git.veillax.com/Veillax/reef/model"
+)
+
+// Ephemeral exposures back `shark expose`: a short-lived site plus one port or hostname that
+// vanish unless the client keeps renewing them. Both records carry exposeTag so they show up
+// in the dashboard as temporary, and so a restart (which forgets the deadlines) can sweep them.
+
+const (
+	exposeTag        = "ephemeral"
+	defaultExposeTTL = 5 * time.Minute
+	minExposeTTL     = 30 * time.Second
+	maxExposeTTL     = 24 * time.Hour
+	exposePortMin    = 20000
+	exposePortMax    = 29999
+	maxExposePerOrg  = 20
+)
+
+type exposeIn struct {
+	Proto      string `json:"proto"` // http, https or tcp
+	Name       string `json:"name"`  // label for the subdomain and the dashboard
+	EdgeID     string `json:"edgeId"`
+	Hostname   string `json:"hostname"` // http: full name under one of the org's domains; default <name>-<random>.<domain>
+	Port       int    `json:"port"`     // public port; default 80/443 for web, a free one in the ephemeral range for tcp
+	TargetHost string `json:"targetHost"`
+	TargetPort int    `json:"targetPort"`
+	TTL        int    `json:"ttlSeconds"`
+	Auth       *struct {
+		Password string `json:"password"`
+		Pin      string `json:"pin"`
+		SSO      bool   `json:"sso"`
+	} `json:"auth"`
+}
+
+type exposeOut struct {
+	ID         string    `json:"id"`
+	Proto      string    `json:"proto"`
+	URL        string    `json:"url,omitempty"` // http: the address to open
+	Host       string    `json:"host"`          // public host (tcp: where to connect)
+	Port       int       `json:"port"`
+	Hostname   string    `json:"hostname,omitempty"`
+	SiteToken  string    `json:"siteToken,omitempty"`  // only in the create response: what Bamboo dials in with
+	Hammerhead string    `json:"hammerhead,omitempty"` // only in the create response
+	Protected  bool      `json:"protected"`
+	ExpiresAt  time.Time `json:"expiresAt"`
+	Live       bool      `json:"live"` // the edge reports the port listening
+	Error      string    `json:"error,omitempty"`
+}
+
+var slugBad = regexp.MustCompile(`[^a-z0-9]+`)
+
+func exposeSlug(n string) string {
+	n = strings.Trim(slugBad.ReplaceAllString(strings.ToLower(n), "-"), "-")
+	if len(n) > 20 {
+		n = strings.Trim(n[:20], "-")
+	}
+	if n == "" {
+		n = "expose"
+	}
+	return n
+}
+
+func exposeTTL(sec int) (time.Duration, error) {
+	if sec == 0 {
+		return defaultExposeTTL, nil
+	}
+	d := time.Duration(sec) * time.Second
+	if d < minExposeTTL || d > maxExposeTTL {
+		return 0, fmt.Errorf("ttlSeconds must be between %d and %d", int(minExposeTTL.Seconds()), int(maxExposeTTL.Seconds()))
+	}
+	return d, nil
+}
+
+func isExposed(tags []string) bool {
+	for _, t := range tags {
+		if t == exposeTag {
+			return true
+		}
+	}
+	return false
+}
+
+// edgeHost is the public host of an edge's WireGuard endpoint, which is where its ports are reachable.
+func edgeHost(e *model.Edge) string {
+	h, _, err := net.SplitHostPort(e.TunnelAddr)
+	if err != nil {
+		return e.TunnelAddr
+	}
+	return h
+}
+
+// exposeEdges lists the edges an exposure may use: the named one, or every online edge the org may use.
+func (s *Server) exposeEdges(orgID, want string) []*model.Edge {
+	var out []*model.Edge
+	for _, e := range s.st.Edges {
+		if (want != "" && e.ID != want) || !s.edgeAllowedFor(e, orgID) {
+			continue
+		}
+		if want != "" || s.edgeOnline(e) {
+			out = append(out, e)
+		}
+	}
+	return out
+}
+
+func (s *Server) exposeView(m *model.Mapping, e *model.Edge) exposeOut {
+	st, known := s.status[m.ID]
+	o := exposeOut{ID: m.ID, Proto: m.Proto, Hostname: m.Hostname, Port: m.Port, Protected: m.Auth.Protected(),
+		ExpiresAt: s.expose[m.ID], URL: publicURL(m), Live: known && st.Error == "", Error: st.Error}
+	if e != nil {
+		o.Host = edgeHost(e)
+	}
+	if m.Hostname != "" {
+		o.Host = m.Hostname
+	}
+	return o
+}
+
+func (s *Server) createExpose(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	var in exposeIn
+	if !decode(w, r, &in) {
+		return
+	}
+	ttl, err := exposeTTL(in.TTL)
+	if err != nil {
+		fail(w, 400, err)
+		return
+	}
+	if in.Proto != model.ProtoHTTP && in.Proto != model.ProtoHTTPS && in.Proto != model.ProtoTCP {
+		fail(w, 400, errors.New("proto must be http, https or tcp"))
+		return
+	}
+	if in.TargetHost == "" {
+		in.TargetHost = "127.0.0.1"
+	}
+	if in.Auth != nil && !model.Web(in.Proto) {
+		fail(w, 400, errors.New("login protection works for http and https only"))
+		return
+	}
+	mi := mappingInput{}
+	if in.Auth != nil {
+		if in.Auth.Password != "" {
+			mi.NewPassword = &in.Auth.Password
+		}
+		if in.Auth.Pin != "" {
+			mi.NewPin = &in.Auth.Pin
+		}
+	}
+	pw, pin, err := hashSecrets(&mi)
+	if err != nil {
+		fail(w, 400, err)
+		return
+	}
+	slug := exposeSlug(in.Name)
+
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.sweepExposedLocked()
+	edges := s.exposeEdges(c.Org.ID, in.EdgeID)
+	if len(edges) == 0 {
+		fail(w, 409, errors.New("no edge is online for this organization"))
+		return
+	}
+	active := 0
+	for _, m := range s.st.Mappings {
+		if m.OrgID == c.Org.ID && isExposed(m.Tags) {
+			active++
+		}
+	}
+	if active >= maxExposePerOrg {
+		fail(w, 429, errors.New("too many active exposures; stop one first"))
+		return
+	}
+	site := &model.Site{ID: newID(), OrgID: c.Org.ID, Name: "expose " + slug, Token: randHex(24), Connected: []string{},
+		TunnelIP: s.allocIPLocked(), Tags: []string{exposeTag}}
+	if site.TunnelIP == "" {
+		fail(w, 507, errors.New("tunnel address space is full"))
+		return
+	}
+	// validateMapping looks the site up, so it is added first and taken out again if nothing fits
+	s.st.Sites = append(s.st.Sites, site)
+	lastErr := errors.New("no edge could take it")
+	for _, e := range edges {
+		m := model.Mapping{Name: "expose " + slug, Proto: in.Proto, EdgeID: e.ID, Enabled: true, Tags: []string{exposeTag},
+			Targets: []model.Target{{SiteID: site.ID, Host: in.TargetHost, Port: in.TargetPort}}, HealthSeconds: -1}
+		if model.Web(in.Proto) {
+			m.Port = in.Port
+			if m.Port == 0 {
+				m.Port = map[string]int{model.ProtoHTTP: 80, model.ProtoHTTPS: 443}[in.Proto]
+			}
+			m.Auth = model.Auth{Password: pw != nil, Pin: pin != nil, SSO: in.Auth != nil && in.Auth.SSO}
+			hosts := []string{strings.ToLower(in.Hostname)}
+			if in.Hostname == "" {
+				hosts = nil
+				for _, d := range s.st.Domains {
+					if d.OrgID == c.Org.ID && d.EdgeID == e.ID {
+						hosts = append(hosts, fmt.Sprintf("%s-%s.%s", slug, randHex(2), d.Name))
+					}
+				}
+				if len(hosts) == 0 {
+					lastErr = errors.New("no domain is set up for this edge; add one on the Domains tab, or give a hostname")
+				}
+			}
+			for _, h := range hosts {
+				m.Hostname = h
+				if lastErr = s.validateMapping(c, &m); lastErr == nil {
+					break
+				}
+			}
+		} else {
+			tries := 1
+			if in.Port == 0 {
+				tries = 60
+			}
+			for i := 0; i < tries; i++ {
+				m.Port = in.Port
+				if in.Port == 0 {
+					m.Port = exposePortMin + rand.IntN(exposePortMax-exposePortMin+1)
+				}
+				if lastErr = s.validateMapping(c, &m); lastErr == nil {
+					break
+				}
+			}
+		}
+		if lastErr != nil {
+			continue
+		}
+		m.ID = newID()
+		s.applySecrets(&m, pw, pin)
+		s.st.Mappings = append(s.st.Mappings, &m)
+		s.expose[m.ID] = time.Now().Add(ttl)
+		s.bump()
+		s.audit(c.Org.ID, c.Actor, "expose.create", m.Name+" on "+portLabel(m))
+		out := s.exposeView(&m, e)
+		out.SiteToken = site.Token
+		out.Hammerhead = s.dashboardURL()
+		writeJSON(w, 200, out)
+		return
+	}
+	s.st.Sites = filter(s.st.Sites, func(x *model.Site) bool { return x.ID != site.ID })
+	fail(w, 400, lastErr)
+}
+
+func (s *Server) findExposeLocked(c *orgCtx, id string) *model.Mapping {
+	m := s.findMapping(id)
+	if m == nil || m.OrgID != c.Org.ID || !isExposed(m.Tags) {
+		return nil
+	}
+	return m
+}
+
+func (s *Server) renewExpose(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	var in struct {
+		TTL int `json:"ttlSeconds"`
+	}
+	if r.ContentLength != 0 && !decode(w, r, &in) {
+		return
+	}
+	ttl, err := exposeTTL(in.TTL)
+	if err != nil {
+		fail(w, 400, err)
+		return
+	}
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.sweepExposedLocked()
+	m := s.findExposeLocked(c, r.PathValue("id"))
+	if m == nil {
+		fail(w, 404, errors.New("no such exposure (it may have expired)"))
+		return
+	}
+	s.expose[m.ID] = time.Now().Add(ttl)
+	writeJSON(w, 200, s.exposeView(m, s.findEdge(m.EdgeID)))
+}
+
+func (s *Server) getExpose(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.sweepExposedLocked()
+	m := s.findExposeLocked(c, r.PathValue("id"))
+	if m == nil {
+		fail(w, 404, errors.New("no such exposure (it may have expired)"))
+		return
+	}
+	writeJSON(w, 200, s.exposeView(m, s.findEdge(m.EdgeID)))
+}
+
+func (s *Server) listExpose(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.sweepExposedLocked()
+	out := []exposeOut{}
+	for _, m := range s.st.Mappings {
+		if m.OrgID == c.Org.ID && isExposed(m.Tags) {
+			out = append(out, s.exposeView(m, s.findEdge(m.EdgeID)))
+		}
+	}
+	writeJSON(w, 200, out)
+}
+
+func (s *Server) deleteExpose(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	m := s.findExposeLocked(c, r.PathValue("id"))
+	if m == nil {
+		ok(w) // already gone: stopping twice is fine
+		return
+	}
+	name := m.Name
+	s.dropExposeLocked(m.ID)
+	s.audit(c.Org.ID, c.Actor, "expose.delete", name)
+	ok(w)
+}
+
+// dropExposeLocked removes an ephemeral mapping and the site that served it.
+func (s *Server) dropExposeLocked(mappingID string) {
+	m := s.findMapping(mappingID)
+	if m == nil {
+		return
+	}
+	gone := map[string]bool{}
+	for _, t := range m.Targets {
+		if st := s.findSite(t.SiteID); st != nil && !st.Local && isExposed(st.Tags) {
+			gone[st.ID] = true
+		}
+	}
+	s.dropMappingLocked(mappingID)
+	delete(s.expose, mappingID)
+	s.st.Mappings = filter(s.st.Mappings, func(x *model.Mapping) bool { return x.ID != mappingID })
+	s.st.Sites = filter(s.st.Sites, func(x *model.Site) bool { return !gone[x.ID] })
+	s.bump()
+}
+
+// sweepExposedLocked removes every ephemeral mapping whose deadline has passed, or that has no
+// deadline at all (left over from before a restart), and any ephemeral site nothing uses.
+func (s *Server) sweepExposedLocked() {
+	now := time.Now()
+	var dead []string
+	for _, m := range s.st.Mappings {
+		if !isExposed(m.Tags) {
+			continue
+		}
+		if at, known := s.expose[m.ID]; !known || now.After(at) {
+			dead = append(dead, m.ID)
+		}
+	}
+	for _, id := range dead {
+		s.dropExposeLocked(id)
+	}
+	used := map[string]bool{}
+	for _, m := range s.st.Mappings {
+		for _, t := range m.Targets {
+			used[t.SiteID] = true
+		}
+	}
+	n := len(s.st.Sites)
+	s.st.Sites = filter(s.st.Sites, func(x *model.Site) bool { return !(isExposed(x.Tags) && !x.Local && !used[x.ID]) })
+	if n != len(s.st.Sites) {
+		s.bump()
+	}
+}
+
+// exposeLoop expires ephemeral exposures. Its first pass clears everything left from the last run,
+// since deadlines are kept in memory only.
+func (s *Server) exposeLoop() {
+	t := time.NewTicker(10 * time.Second)
+	defer t.Stop()
+	for {
+		s.mu.Lock()
+		s.sweepExposedLocked()
+		s.mu.Unlock()
+		select {
+		case <-s.stop:
+			return
+		case <-t.C:
+		}
+	}
+}

expose_test.go

@@ -0,0 +1,195 @@
+package hammerhead
+
+import (
+	"encoding/json"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/reef/model"
+)
+
+func exposeServer(t *testing.T) (*Server, func(method, path string, in any) (int, []byte), string) {
+	t.Helper()
+	dir := t.TempDir()
+	s, err := New(dir, Options{AdminEmail: "admin@test.local", AdminPassword: "password123", AlertInterval: time.Hour})
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(s.Close)
+	do := apiClient(t, s)
+	_, b := do("GET", "/api/me", nil)
+	var me struct{ Orgs []struct{ ID string } }
+	json.Unmarshal(b, &me)
+	code, raw := do("POST", "/api/edges", map[string]any{"name": "e", "tunnelAddr": "203.0.113.5:51820"})
+	if code != 200 {
+		t.Fatalf("edge: %d %s", code, raw)
+	}
+	var e model.Edge
+	json.Unmarshal(raw, &e)
+	s.mu.Lock()
+	s.findEdge(e.ID).LastSeen = time.Now()
+	s.mu.Unlock()
+	if code, raw := do("POST", "/api/orgs/"+me.Orgs[0].ID+"/domains", map[string]any{"name": "t.example", "edgeId": e.ID}); code != 200 {
+		t.Fatalf("domain: %d %s", code, raw)
+	}
+	return s, do, me.Orgs[0].ID
+}
+
+func TestExposeLifecycle(t *testing.T) {
+	s, do, org := exposeServer(t)
+	base := "/api/orgs/" + org + "/expose"
+
+	code, raw := do("POST", base, map[string]any{"proto": "tcp", "targetPort": 3000})
+	if code != 200 {
+		t.Fatalf("create tcp: %d %s", code, raw)
+	}
+	var tcp exposeOut
+	json.Unmarshal(raw, &tcp)
+	if tcp.Host != "203.0.113.5" || tcp.Port < exposePortMin || tcp.Port > exposePortMax || tcp.SiteToken == "" || tcp.ID == "" {
+		t.Fatalf("tcp exposure: %+v", tcp)
+	}
+
+	code, raw = do("POST", base, map[string]any{"proto": "http", "name": "My App!", "targetPort": 8080, "auth": map[string]any{"password": "hunter22"}})
+	if code != 200 {
+		t.Fatalf("create http: %d %s", code, raw)
+	}
+	var web exposeOut
+	json.Unmarshal(raw, &web)
+	if !strings.HasPrefix(web.Hostname, "my-app-") || !strings.HasSuffix(web.Hostname, ".t.example") || web.URL != "http://"+web.Hostname || !web.Protected {
+		t.Fatalf("http exposure: %+v", web)
+	}
+
+	// validation
+	for name, in := range map[string]map[string]any{
+		"bad proto":    {"proto": "udp", "targetPort": 1},
+		"bad ttl":      {"proto": "tcp", "targetPort": 1, "ttlSeconds": 5},
+		"bad target":   {"proto": "tcp", "targetPort": 0},
+		"tcp login":    {"proto": "tcp", "targetPort": 1, "auth": map[string]any{"password": "hunter22"}},
+		"short pass":   {"proto": "http", "targetPort": 1, "auth": map[string]any{"password": "x"}},
+		"foreign host": {"proto": "http", "targetPort": 1, "hostname": "evil.example.com"},
+		"unknown edge": {"proto": "tcp", "targetPort": 1, "edgeId": "nope"},
+		"bad pin":      {"proto": "http", "targetPort": 1, "auth": map[string]any{"pin": "12"}},
+	} {
+		if code, raw := do("POST", base, in); code < 400 {
+			t.Errorf("%s accepted: %d %s", name, code, raw)
+		}
+	}
+
+	_, raw = do("GET", base, nil)
+	var list []exposeOut
+	json.Unmarshal(raw, &list)
+	if len(list) != 2 {
+		t.Fatalf("list: %s", raw)
+	}
+	// failed creates must not leave sites behind
+	if sites := s.ephemeralSites(); sites != 2 {
+		t.Errorf("want 2 ephemeral sites, have %d", sites)
+	}
+
+	// renew pushes the deadline out; a deadline in the past expires it
+	s.mu.Lock()
+	s.expose[tcp.ID] = time.Now().Add(time.Second)
+	s.mu.Unlock()
+	if code, _ := do("POST", base+"/"+tcp.ID+"/renew", map[string]any{"ttlSeconds": 600}); code != 200 {
+		t.Fatalf("renew: %d", code)
+	}
+	s.mu.Lock()
+	left := time.Until(s.expose[tcp.ID])
+	s.expose[web.ID] = time.Now().Add(-time.Second)
+	s.mu.Unlock()
+	if left < 9*time.Minute {
+		t.Errorf("renew left %v", left)
+	}
+	if code, _ := do("GET", base+"/"+web.ID, nil); code != 404 {
+		t.Errorf("expired exposure still there: %d", code)
+	}
+	if code, _ := do("POST", base+"/"+web.ID+"/renew", nil); code != 404 {
+		t.Errorf("renewed an expired exposure: %d", code)
+	}
+	s.mu.Lock()
+	if s.ephemeralSitesLocked() != 1 || len(s.st.Mappings) != 1 || len(s.st.Secrets) != 0 {
+		t.Errorf("expiry left debris: %d sites, %d mappings, %d secrets", s.ephemeralSitesLocked(), len(s.st.Mappings), len(s.st.Secrets))
+	}
+	s.mu.Unlock()
+
+	// stopping is idempotent
+	for i := 0; i < 2; i++ {
+		if code, _ := do("DELETE", base+"/"+tcp.ID, nil); code != 200 {
+			t.Errorf("delete %d: %d", i, code)
+		}
+	}
+	s.mu.Lock()
+	if s.ephemeralSitesLocked() != 0 || len(s.st.Mappings) != 0 {
+		t.Errorf("delete left debris")
+	}
+	s.mu.Unlock()
+}
+
+func TestExposeSweptAfterRestart(t *testing.T) {
+	s, do, org := exposeServer(t)
+	if code, raw := do("POST", "/api/orgs/"+org+"/expose", map[string]any{"proto": "tcp", "targetPort": 3000}); code != 200 {
+		t.Fatalf("%d %s", code, raw)
+	}
+	// a regular mapping on a regular site must survive the sweep
+	_, raw := do("POST", "/api/orgs/"+org+"/sites", map[string]any{"name": "home"})
+	var site model.Site
+	json.Unmarshal(raw, &site)
+	s.mu.Lock()
+	s.expose = map[string]time.Time{} // what a restart does
+	s.sweepExposedLocked()
+	n, m := len(s.st.Mappings), 0
+	for _, x := range s.st.Sites {
+		if x.ID == site.ID {
+			m++
+		}
+	}
+	s.mu.Unlock()
+	if n != 0 || m != 1 {
+		t.Fatalf("mappings left %d, regular site kept %d", n, m)
+	}
+}
+
+func (s *Server) ephemeralSitesLocked() int {
+	n := 0
+	for _, x := range s.st.Sites {
+		if isExposed(x.Tags) {
+			n++
+		}
+	}
+	return n
+}
+
+func (s *Server) ephemeralSites() int {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	return s.ephemeralSitesLocked()
+}
+
+// An API key belongs to one organization, so "-" in the path stands for it.
+func TestExposeKeyOrgShorthand(t *testing.T) {
+	s, do, org := exposeServer(t)
+	_, raw := do("POST", "/api/orgs/"+org+"/apikeys", map[string]any{"name": "cli"})
+	var k struct{ Key string }
+	json.Unmarshal(raw, &k)
+	if !strings.HasPrefix(k.Key, "hhk_") {
+		t.Fatal(string(raw))
+	}
+	call := func(key, body string) int {
+		req := httptest.NewRequest("POST", "/api/orgs/-/expose", strings.NewReader(body))
+		req.Header.Set("Authorization", "Bearer "+key)
+		rec := httptest.NewRecorder()
+		s.Handler().ServeHTTP(rec, req)
+		return rec.Code
+	}
+	if got := call(k.Key, `{"proto":"tcp","targetPort":3000}`); got != 200 {
+		t.Errorf("key: %d", got)
+	}
+	if got := call("hhk_dead_beef", `{"proto":"tcp","targetPort":3000}`); got != 401 {
+		t.Errorf("bad key: %d", got)
+	}
+	if got := call("", `{"proto":"tcp","targetPort":3000}`); got != 401 {
+		t.Errorf("no key: %d", got)
+	}
+}

go.mod

@@ -3,9 +3,9 @@ module git.veillax.com/Veillax/hammerhead
 go 1.27.3
 
 require (
-	git.veillax.com/Veillax/bamboo v0.1.0
+	git.veillax.com/Veillax/bamboo v0.1.8
-	git.veillax.com/Veillax/blacktip v0.1.0
+	git.veillax.com/Veillax/blacktip v0.1.10
-	git.veillax.com/Veillax/reef v0.1.0
+	git.veillax.com/Veillax/reef v0.1.11
 	github.com/emersion/go-msgauth v0.7.0
 	golang.org/x/crypto v0.58.0
 	modernc.org/sqlite v1.61.0
@@ -13,6 +13,7 @@ require (
 
 require (
 	github.com/Microsoft/go-winio v0.6.4 // indirect
+	github.com/creack/pty v1.1.24 // indirect
 	github.com/dustin/go-humanize v1.1.0 // indirect
 	github.com/google/btree v1.1.2 // indirect
 	github.com/google/uuid v1.6.0 // indirect

go.sum

@@ -1,11 +1,13 @@
-git.veillax.com/Veillax/bamboo v0.1.0 h1:16BG1P8982dIkHA2l0LPS0xYQc8d86PruVAGxc8BP84=
+git.veillax.com/Veillax/bamboo v0.1.8 h1:mOdkGBzM8CjFVjR3784/WAYGt3U4bSdOaAJqFWMVh3I=
-git.veillax.com/Veillax/bamboo v0.1.0/go.mod h1:i4l4uJgG6Ymfwbjnlz0xeTt3qW/+1W1L2JBToMw50n0=
+git.veillax.com/Veillax/bamboo v0.1.8/go.mod h1:hUMWRFuDIiTNaqjpx2F7RxOg7ussrbVZAQIhzhPF780=
-git.veillax.com/Veillax/blacktip v0.1.0 h1:L8WpW56gqtvZ1IAapE95Ynl6NKjqTA98WwmEH01kFFk=
+git.veillax.com/Veillax/blacktip v0.1.10 h1:2e1wUbMCp4z/DvQjAJCRmLyqu4PX+qV3mo5WBliqGiE=
-git.veillax.com/Veillax/blacktip v0.1.0/go.mod h1:sj+mr8393rajzWbl0+RRZrstGyDtZ/GHsut6ebScVxg=
+git.veillax.com/Veillax/blacktip v0.1.10/go.mod h1:tarAhdKU9hx5pdufe9wG3dRWYkWgPYZvawKM5DWgt6o=
-git.veillax.com/Veillax/reef v0.1.0 h1:O0OhEKFx2CsvACJ20sfzJNjGsngomtfDdKdxu4ROpCs=
+git.veillax.com/Veillax/reef v0.1.11 h1:kbjgxJpmpp1HRckbW1tqJUm5/ftgvSlmUHAhFXugOlY=
-git.veillax.com/Veillax/reef v0.1.0/go.mod h1:ZBUd/BARO5kkpUNTP81TTY1aQXLHFR6bKDhCxckjHso=
+git.veillax.com/Veillax/reef v0.1.11/go.mod h1:ZBUd/BARO5kkpUNTP81TTY1aQXLHFR6bKDhCxckjHso=
 github.com/Microsoft/go-winio v0.6.4 h1:kJF2pamtYCWXf0xAT8sAkwKC0pMl2HYBrRPHAqyUc+g=
 github.com/Microsoft/go-winio v0.6.4/go.mod h1:1xl5JXG4ze/m74GH6Dc4Mwxfc3VL5siRVQjIAlHH07U=
+github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
+github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=
 github.com/dustin/go-humanize v1.1.0 h1:dbKTrvD0klcbBV/h4AWJdMuZogJACoMlvWIWZ5b2xWg=
 github.com/dustin/go-humanize v1.1.0/go.mod h1:hc1CvRkJMsgxqjmjMQF3QNRAZBwY8AXBAzKYoSX9sFI=
 github.com/emersion/go-msgauth v0.7.0 h1:vj2hMn6KhFtW41kshIBTXvp6KgYSqpA/ZN9Pv4g1INc=
@@ -34,6 +36,8 @@ golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
 golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
 golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
 golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
+golang.org/x/term v0.47.0 h1:cpyOwQkr2Nzxe2p6KnrTG7Ei2wtzjsuSjhpXXepZ/BM=
+golang.org/x/term v0.47.0/go.mod h1:efAfAuG6RKABvg1ArGebRkvX6LXul14SM67o2e4mUCY=
 golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
 golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
 golang.org/x/time v0.7.0 h1:ntUhktv3OPE6TgYxXWv9vKvUSJyIFJlyohwbkEwPrKQ=

infra.go

@@ -4,6 +4,7 @@ import (
 	"errors"
 	"net"
 	"net/http"
+	"net/netip"
 	"sort"
 	"strings"
 	"time"
@@ -24,6 +25,11 @@ type edgeView struct {
 	TunnelAddr string   `json:"tunnelAddr,omitempty"`
 	Token      string   `json:"token,omitempty"`
 	OrgIDs     []string `json:"orgIds,omitempty"`
+
+	Version       string `json:"version,omitempty"`
+	AutoUpdate    bool   `json:"autoUpdate,omitempty"`
+	UpdatePending bool   `json:"updatePending,omitempty"`
+	UpdateMsg     string `json:"updateMsg,omitempty"`
 }
 
 func (s *Server) viewEdge(e *model.Edge, admin bool) edgeView {
@@ -33,6 +39,7 @@ func (s *Server) viewEdge(e *model.Edge, admin bool) edgeView {
 	}
 	if admin {
 		v.TunnelAddr, v.Token, v.OrgIDs = e.TunnelAddr, e.Token, e.OrgIDs
+		v.Version, v.AutoUpdate, v.UpdatePending, v.UpdateMsg = e.Version, e.AutoUpdate, e.UpdateID != "", e.UpdateMsg
 		if v.OrgIDs == nil {
 			v.OrgIDs = []string{}
 		}
@@ -58,11 +65,22 @@ func validAddr(a string) bool {
 	return n > 0
 }
 
-func (s *Server) listEdges(w http.ResponseWriter, r *http.Request, p *principal) {
+func (s *Server) listEdges(w http.ResponseWriter, r *http.Request) {
+	p := s.principalFrom(r)
+	if p == nil {
+		fail(w, 401, errors.New("Sign in to continue."))
+		return
+	}
 	s.mu.Lock()
 	defer s.mu.Unlock()
 	out := []edgeView{}
 	for _, e := range s.st.Edges {
+		if p.Key != nil && !p.serverKey() { // an org key sees the edges its organization may use
+			if s.edgeAllowedFor(e, p.Key.OrgID) {
+				out = append(out, s.viewEdge(e, false))
+			}
+			continue
+		}
 		if p.User.ServerAdmin {
 			out = append(out, s.viewEdge(e, true))
 			continue
@@ -198,6 +216,43 @@ func (s *Server) addSite(w http.ResponseWriter, r *http.Request, c *orgCtx) {
 	writeJSON(w, 200, st)
 }
 
+// putSite renames a site and sets its tags. The built-in local site is shared between
+// organizations, so it can't be edited here.
+func (s *Server) putSite(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	var in struct {
+		Name string   `json:"name"`
+		Tags []string `json:"tags"`
+	}
+	if !decode(w, r, &in) {
+		return
+	}
+	in.Name = strings.TrimSpace(in.Name)
+	if in.Name == "" {
+		fail(w, 400, errors.New("name is required"))
+		return
+	}
+	tags, err := normalizeTags(in.Tags)
+	if err != nil {
+		fail(w, 400, err)
+		return
+	}
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	st := s.findSite(r.PathValue("id"))
+	if st == nil || st.OrgID != c.Org.ID {
+		fail(w, 404, errors.New("no such site"))
+		return
+	}
+	if st.Local {
+		fail(w, 400, errors.New("the built-in local site can't be edited"))
+		return
+	}
+	st.Name, st.Tags = in.Name, tags
+	s.bump()
+	s.audit(c.Org.ID, c.Actor, "site.update", st.Name)
+	ok(w)
+}
+
 func (s *Server) deleteSite(w http.ResponseWriter, r *http.Request, c *orgCtx) {
 	id := r.PathValue("id")
 	s.mu.Lock()
@@ -227,7 +282,7 @@ func (s *Server) deleteSite(w http.ResponseWriter, r *http.Request, c *orgCtx) {
 			}
 		}
 	}
-	s.st.Mappings = filter(s.st.Mappings, func(m *model.Mapping) bool { return len(m.Targets) > 0 })
+	s.st.Mappings = filter(s.st.Mappings, func(m *model.Mapping) bool { return len(m.Targets) > 0 || m.IsRedirect() })
 	s.bump()
 	s.audit(c.Org.ID, c.Actor, "site.delete", st.Name)
 	ok(w)
@@ -248,6 +303,47 @@ func (s *Server) siteContainers(w http.ResponseWriter, r *http.Request, c *orgCt
 	writeJSON(w, 200, out)
 }
 
+// cleanDiscovered keeps what an agent reports as suggestions only if it is well formed: it is shown in
+// the dashboard, so lengths are bounded and anything that is not a plain address is dropped.
+func cleanDiscovered(in []model.DiscoveredService) []model.DiscoveredService {
+	out := []model.DiscoveredService{}
+	clip := func(s string, n int) string {
+		if len(s) > n {
+			s = s[:n]
+		}
+		return strings.ToValidUTF8(s, "")
+	}
+	for _, d := range in {
+		if len(out) >= 500 {
+			break
+		}
+		if _, err := netip.ParseAddr(d.Host); err != nil || d.Port < 1 || d.Port > 65535 {
+			continue
+		}
+		src := "scan"
+		if d.Source == "local" {
+			src = "local"
+		}
+		out = append(out, model.DiscoveredService{Host: d.Host, Port: d.Port, Proto: "tcp", Service: clip(d.Service, 20), Detail: clip(d.Detail, 160), Source: src})
+	}
+	return out
+}
+
+func (s *Server) siteDiscovered(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	st := s.findSite(r.PathValue("id"))
+	if st == nil || !s.siteVisible(c, st) {
+		fail(w, 404, errors.New("no such site"))
+		return
+	}
+	out := s.discovered[st.ID]
+	if out == nil {
+		out = []model.DiscoveredService{}
+	}
+	writeJSON(w, 200, out)
+}
+
 // ---------- domains ----------
 
 func domainsOverlap(a, b string) bool {

internal/secretbox/secretbox.go

@@ -1,87 +0,0 @@
-// Package secretbox encrypts small secrets (tokens, passwords, private keys)
-// before they are written to the database, using AES-256-GCM.
-//
-// Sealed values look like "enc1:<base64>". Opening a value that isn't sealed
-// returns it unchanged, so data written before encryption was turned on is
-// still readable and gets sealed the next time it is saved.
-package secretbox
-
-import (
-	"crypto/aes"
-	"crypto/cipher"
-	"crypto/rand"
-	"crypto/sha256"
-	"encoding/base64"
-	"encoding/hex"
-	"errors"
-	"strings"
-)
-
-const prefix = "enc1:"
-
-type Box struct{ aead cipher.AEAD }
-
-// GenerateKey returns a new random key in the form New accepts.
-func GenerateKey() string {
-	b := make([]byte, 32)
-	rand.Read(b)
-	return base64.StdEncoding.EncodeToString(b)
-}
-
-// New builds a Box from a key given as base64 or hex of 32 bytes. Any other
-// text is treated as a passphrase and stretched with SHA-256.
-func New(key string) (*Box, error) {
-	key = strings.TrimSpace(key)
-	if key == "" {
-		return nil, errors.New("empty key")
-	}
-	raw := decodeKey(key)
-	block, err := aes.NewCipher(raw)
-	if err != nil {
-		return nil, err
-	}
-	aead, err := cipher.NewGCM(block)
-	if err != nil {
-		return nil, err
-	}
-	return &Box{aead: aead}, nil
-}
-
-func decodeKey(key string) []byte {
-	if b, err := base64.StdEncoding.DecodeString(key); err == nil && len(b) == 32 {
-		return b
-	}
-	if b, err := hex.DecodeString(key); err == nil && len(b) == 32 {
-		return b
-	}
-	sum := sha256.Sum256([]byte(key))
-	return sum[:]
-}
-
-// Seal encrypts s. The empty string stays empty so unset secrets stay visibly unset.
-func (b *Box) Seal(s string) string {
-	if s == "" || strings.HasPrefix(s, prefix) {
-		return s
-	}
-	nonce := make([]byte, b.aead.NonceSize())
-	rand.Read(nonce)
-	return prefix + base64.StdEncoding.EncodeToString(b.aead.Seal(nonce, nonce, []byte(s), nil))
-}
-
-// Open decrypts a sealed value. A wrong key or damaged data is an error, so a
-// bad key can never silently turn secrets into garbage.
-func (b *Box) Open(s string) (string, error) {
-	if !strings.HasPrefix(s, prefix) {
-		return s, nil
-	}
-	raw, err := base64.StdEncoding.DecodeString(s[len(prefix):])
-	if err != nil || len(raw) < b.aead.NonceSize() {
-		return "", errors.New("damaged secret")
-	}
-	n := b.aead.NonceSize()
-	plain, err := b.aead.Open(nil, raw[:n], raw[n:], nil)
-	if err != nil {
-		return "", errors.New("can't decrypt a stored secret: is HAMMERHEAD_MASTER_KEY the key it was saved with?")
-	}
-	return string(plain), nil
-}

internal/secretbox/secretbox_test.go

@@ -1,51 +0,0 @@
-package secretbox
-
-import (
-	"strings"
-	"testing"
-)
-
-func TestSealOpenRoundTripAndKeys(t *testing.T) {
-	key := GenerateKey()
-	b, err := New(key)
-	if err != nil {
-		t.Fatal(err)
-	}
-	sealed := b.Seal("hunter2")
-	if !strings.HasPrefix(sealed, "enc1:") || strings.Contains(sealed, "hunter2") {
-		t.Fatalf("not sealed: %q", sealed)
-	}
-	if again := b.Seal("hunter2"); again == sealed {
-		t.Error("two seals of the same value must differ (random nonce)")
-	}
-	if b.Seal(sealed) != sealed {
-		t.Error("sealing an already sealed value must not double-encrypt")
-	}
-	if got, err := b.Open(sealed); err != nil || got != "hunter2" {
-		t.Errorf("Open = %q, %v", got, err)
-	}
-	if got, _ := b.Open("legacy plaintext"); got != "legacy plaintext" {
-		t.Errorf("unsealed legacy data should pass through, got %q", got)
-	}
-	if b.Seal("") != "" {
-		t.Error("empty stays empty")
-	}
-
-	other, _ := New(GenerateKey())
-	if _, err := other.Open(sealed); err == nil {
-		t.Error("the wrong key must fail loudly, not return garbage")
-	}
-	if _, err := b.Open(sealed[:len(sealed)-4] + "AAAA"); err == nil {
-		t.Error("tampered data must fail")
-	}
-
-	// a passphrase works as a key, and is stable
-	p1, _ := New("correct horse battery staple")
-	p2, _ := New("correct horse battery staple")
-	if got, err := p2.Open(p1.Seal("x")); err != nil || got != "x" {
-		t.Errorf("passphrase keys must be reproducible: %q %v", got, err)
-	}
-	if _, err := New("  "); err == nil {
-		t.Error("an empty key must be rejected")
-	}
-}

mailegress.go

@@ -149,6 +149,7 @@ type mailEgressView struct {
 	Status      string           `json:"status"`
 	StatusKnown bool             `json:"statusKnown"`
 	Check       *model.MailCheck `json:"check,omitempty"`
+	Findings    []Finding        `json:"findings"` // the deliverability checklist, with fixes
 }
 
 // mailEgressViewsLocked lists an org's mail egress without secrets. Caller holds s.mu.
@@ -164,6 +165,10 @@ func (s *Server) mailEgressViewsLocked(orgID string) []mailEgressView {
 		if chk, ok := s.mailChecks[e.ID]; ok {
 			c := chk
 			v.Check = &c
+			v.Findings = s.mailEgressFindingsLocked(e, chk)
+		}
+		if v.Findings == nil {
+			v.Findings = []Finding{}
 		}
 		out = append(out, v)
 	}

mappings.go

@@ -6,6 +6,8 @@ import (
 	"fmt"
 	"net"
 	"net/http"
+	"net/url"
+	"strconv"
 	"strings"
 	"time"
 
@@ -53,9 +55,39 @@ func listFromCSV(in []string) []string {
 	return out
 }
 
+// normalizeTags trims tags, drops blanks and repeats (ignoring case; the first spelling wins)
+// and keeps the list short enough to stay readable in the dashboard.
+func normalizeTags(in []string) ([]string, error) {
+	var out []string
+	seen := map[string]bool{}
+	for _, t := range in {
+		t = strings.Join(strings.Fields(t), " ")
+		if t == "" || seen[strings.ToLower(t)] {
+			continue
+		}
+		if len([]rune(t)) > 32 {
+			return nil, errors.New("A tag can be at most 32 characters.")
+		}
+		if strings.ContainsAny(t, ",<>\"") {
+			return nil, errors.New("Tags can't contain commas, quotes or angle brackets.")
+		}
+		seen[strings.ToLower(t)] = true
+		out = append(out, t)
+	}
+	if len(out) > 10 {
+		return nil, errors.New("Use at most 10 tags.")
+	}
+	return out, nil
+}
+
 // validateMapping checks m against org c and normalises it. Caller holds s.mu.
 func (s *Server) validateMapping(c *orgCtx, m *model.Mapping) error {
 	m.Name = strings.TrimSpace(m.Name)
+	tags, err := normalizeTags(m.Tags)
+	if err != nil {
+		return err
+	}
+	m.Tags = tags
 	m.Hostname = strings.ToLower(strings.TrimSpace(m.Hostname))
 	m.OrgID = c.Org.ID
 	switch m.Proto {
@@ -67,7 +99,16 @@ func (s *Server) validateMapping(c *orgCtx, m *model.Mapping) error {
 	if edge == nil || !s.edgeAllowedFor(edge, c.Org.ID) {
 		return errors.New("choose an edge")
 	}
-	if len(m.Targets) == 0 {
+	redirect := strings.TrimSpace(m.RedirectTo) != ""
+	if redirect {
+		if !model.Web(m.Proto) {
+			return errors.New("only websites (http or https) can redirect")
+		}
+		m.Targets = []model.Target{} // a redirect has nothing behind it
+	} else {
+		m.RedirectTo, m.RedirectPermanent, m.RedirectKeepPath = "", false, false
+	}
+	if len(m.Targets) == 0 && !redirect {
 		return errors.New("add at least one target")
 	}
 	if m.Proto == model.ProtoUDP && len(m.Targets) > 1 {
@@ -87,7 +128,7 @@ func (s *Server) validateMapping(c *orgCtx, m *model.Mapping) error {
 			return errors.New("every target needs a host (e.g. 127.0.0.1 or 192.168.1.20)")
 		}
 		if t.Port < 1 || t.Port > 65535 {
-			return errors.New("target ports must be 1-65535")
+			return errors.New("Target ports must be between 1 and 65535.")
 		}
 	}
 	if m.BindIP == "" {
@@ -99,7 +140,7 @@ func (s *Server) validateMapping(c *orgCtx, m *model.Mapping) error {
 	}
 	m.BindIP = ip.String()
 	if m.Port < 1 || m.Port > 65535 {
-		return errors.New("public port must be 1-65535")
+		return errors.New("The public port must be between 1 and 65535.")
 	}
 	switch {
 	case m.PortEnd == 0 || m.PortEnd == m.Port:
@@ -107,7 +148,7 @@ func (s *Server) validateMapping(c *orgCtx, m *model.Mapping) error {
 	case m.PortEnd < m.Port:
 		return errors.New("the end of a port range can't be before its start")
 	case m.PortEnd > 65535:
-		return errors.New("public ports must be 1-65535")
+		return errors.New("Public ports must be between 1 and 65535.")
 	case m.Proto != model.ProtoTCP && m.Proto != model.ProtoUDP:
 		return errors.New("port ranges work for tcp and udp only")
 	case m.PortEnd-m.Port+1 > 1000:
@@ -143,9 +184,19 @@ func (s *Server) validateMapping(c *orgCtx, m *model.Mapping) error {
 		return errors.New("PROXY protocol must be v1, v2 or off")
 	}
 	// web-only options
-	if !model.Web(m.Proto) {
+	if redirect {
+		to, err := cleanRedirect(m)
+		if err != nil {
+			return err
+		}
+		m.RedirectTo = to
+	}
+	if !model.Web(m.Proto) || redirect {
 		m.Auth, m.Rules, m.RewriteHost = model.Auth{}, nil, ""
 		m.RedirectHTTP, m.BackendHTTPS, m.HealthPath = false, false, ""
+		if redirect {
+			m.HealthSeconds = -1 // nothing to check
+		}
 	} else {
 		a := &m.Auth
 		a.AllowedRoles, a.AllowedUsers, a.AllowedMails = listFromCSV(a.AllowedRoles), listFromCSV(a.AllowedUsers), listFromCSV(a.AllowedMails)
@@ -193,7 +244,7 @@ func (s *Server) validateMapping(c *orgCtx, m *model.Mapping) error {
 		}
 	}
 	if m.HealthSeconds != 0 && m.HealthSeconds != -1 && (m.HealthSeconds < 5 || m.HealthSeconds > 3600) {
-		return errors.New("health check interval must be 5-3600 seconds, or off")
+		return errors.New("The health check interval must be 5 to 3600 seconds, or off.")
 	}
 
 	for _, o := range s.st.Mappings {
@@ -204,7 +255,23 @@ func (s *Server) validateMapping(c *orgCtx, m *model.Mapping) error {
 		if m.Proto == o.Proto && model.HostBased(m.Proto) && m.BindIP == o.BindIP && m.Hostname != o.Hostname {
 			continue // same listener, different hostname
 		}
-		return fmt.Errorf("%s port %s on %s is already used by %q", netClass(m.Proto), portLabel(*m), o.BindIP, o.Name)
+		if m.Proto == o.Proto && model.Web(m.Proto) && m.BindIP != o.BindIP && isWild(m.BindIP) != isWild(o.BindIP) {
+			continue // a website on every address steps aside for one that names an address
+		}
+		return fmt.Errorf("%s port %s on %s is already used by %q. Choose another port or address.", strings.ToUpper(netClass(m.Proto)), portLabel(*m), o.BindIP, o.Name)
+	}
+	// an https resource with the redirect on also answers on port 80 at its address, so a raw port
+	// there would clash with it (and the other way round)
+	for _, o := range s.st.Mappings {
+		if o.ID == m.ID || o.EdgeID != m.EdgeID || !overlap(o.BindIP, m.BindIP) {
+			continue
+		}
+		switch {
+		case m.Proto == model.ProtoHTTPS && m.RedirectHTTP && netClass(o.Proto) == "tcp" && !model.Web(o.Proto) && o.Port <= 80 && 80 <= o.LastPort():
+			return fmt.Errorf("Redirecting HTTP to HTTPS also uses TCP port 80 on %s, which %q uses. Turn the redirect off or choose another port for %q.", o.BindIP, o.Name, o.Name)
+		case o.Proto == model.ProtoHTTPS && o.RedirectHTTP && netClass(m.Proto) == "tcp" && !model.Web(m.Proto) && m.Port <= 80 && 80 <= m.LastPort():
+			return fmt.Errorf("TCP port 80 on %s is also used by %q to redirect HTTP to HTTPS. Choose another port, or turn that redirect off.", o.BindIP, o.Name)
+		}
 	}
 	if m.Name == "" {
 		if m.Hostname != "" {
@@ -216,6 +283,28 @@ func (s *Server) validateMapping(c *orgCtx, m *model.Mapping) error {
 	return nil
 }
 
+// cleanRedirect checks where a redirect sends visitors and refuses a destination that would
+// send them straight back to the same address.
+func cleanRedirect(m *model.Mapping) (string, error) {
+	to := strings.TrimSpace(m.RedirectTo)
+	u, err := url.Parse(to)
+	if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Hostname() == "" || u.User != nil {
+		return "", errors.New("Redirect to a full address such as https://www.example.com.")
+	}
+	if len(to) > 2048 || strings.ContainsAny(to, " \t\r\n$") {
+		return "", errors.New("The redirect address is too long or contains spaces or a dollar sign.")
+	}
+	port := u.Port()
+	if port == "" {
+		port = map[string]string{"http": "80", "https": "443"}[u.Scheme]
+	}
+	same := u.Scheme == m.Proto && strings.EqualFold(u.Hostname(), m.Hostname) && port == strconv.Itoa(m.Port)
+	if same && (m.RedirectKeepPath || u.Path == "" || u.Path == "/") {
+		return "", errors.New("That redirect would send visitors back to the same address.")
+	}
+	return to, nil
+}
+
 // portLabel is "25565" or, for a range, "2456-2458".
 func portLabel(m model.Mapping) string {
 	if m.LastPort() > m.Port {
@@ -226,7 +315,7 @@ func portLabel(m model.Mapping) string {
 
 func checkPIN(p string) error {
 	if len(p) < 4 || len(p) > 12 {
-		return errors.New("a PIN is 4-12 digits")
+		return errors.New("A PIN is 4 to 12 digits.")
 	}
 	for _, c := range p {
 		if c < '0' || c > '9' {

mappings_test.go

@@ -0,0 +1,76 @@
+package hammerhead
+
+import (
+	"encoding/json"
+	"strings"
+	"testing"
+)
+
+// websites share a port by hostname, and a website on every address steps aside for one that names an address
+func TestWebsitesShareAPortAndWildcardsStepAside(t *testing.T) {
+	s, err := New(t.TempDir(), Options{AdminEmail: "admin@test.local", AdminPassword: "password123", MasterKey: key})
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	do := apiClient(t, s)
+	var me struct{ Orgs []struct{ ID string } }
+	_, b := do("GET", "/api/me", nil)
+	json.Unmarshal(b, &me)
+	org := me.Orgs[0].ID
+	var edge, site struct{ ID string }
+	_, b = do("POST", "/api/edges", map[string]any{"name": "e", "tunnelAddr": "203.0.113.5:51820"})
+	json.Unmarshal(b, &edge)
+	_, b = do("POST", "/api/orgs/"+org+"/sites", map[string]any{"name": "home"})
+	json.Unmarshal(b, &site)
+	if code, b := do("POST", "/api/orgs/"+org+"/domains", map[string]any{"name": "example.test", "edgeId": edge.ID, "ip": "203.0.113.10"}); code != 200 {
+		t.Fatalf("domain: %d %s", code, b)
+	}
+	base := "/api/orgs/" + org + "/mappings"
+	open := func(m map[string]any) (int, string) {
+		m["edgeId"] = edge.ID
+		m["enabled"] = true
+		m["targets"] = []any{map[string]any{"siteId": site.ID, "host": "127.0.0.1", "port": 8080}}
+		code, b := do("POST", base, m)
+		return code, string(b)
+	}
+	want := func(code int, got int, body, what string) {
+		t.Helper()
+		if code != got {
+			t.Errorf("%s: got %d, want %d (%s)", what, got, code, body)
+		}
+	}
+
+	c, body := open(map[string]any{"proto": "https", "bindIp": "0.0.0.0", "port": 443, "hostname": "a.example.test"})
+	want(200, c, body, "first website on every address")
+	c, body = open(map[string]any{"proto": "https", "bindIp": "203.0.113.10", "port": 443, "hostname": "b.example.test"})
+	want(200, c, body, "a website naming an address shares the port with the wildcard one")
+	c, body = open(map[string]any{"proto": "https", "bindIp": "203.0.113.11", "port": 443, "hostname": "c.example.test"})
+	want(200, c, body, "a second named address")
+	c, body = open(map[string]any{"proto": "https", "bindIp": "203.0.113.10", "port": 443, "hostname": "d.example.test"})
+	want(200, c, body, "another hostname on a named address")
+	// still refused: the same hostname twice on one listener, a raw port on a website's port, and another protocol
+	c, body = open(map[string]any{"proto": "https", "bindIp": "203.0.113.10", "port": 443, "hostname": "b.example.test"})
+	want(400, c, body, "the same hostname twice on one address")
+	c, body = open(map[string]any{"proto": "tcp", "bindIp": "0.0.0.0", "port": 443})
+	want(400, c, body, "a raw port on top of websites")
+	c, body = open(map[string]any{"proto": "http", "bindIp": "203.0.113.10", "port": 443, "hostname": "e.example.test"})
+	want(400, c, body, "http on an https port")
+
+	// the HTTP to HTTPS redirect also uses port 80 at its address
+	c, body = open(map[string]any{"proto": "tcp", "bindIp": "203.0.113.20", "port": 80})
+	want(200, c, body, "a raw port 80 on its own address")
+	c, body = open(map[string]any{"proto": "https", "bindIp": "203.0.113.20", "port": 8443, "hostname": "f.example.test", "redirectHttp": true})
+	want(400, c, body, "a redirect onto a raw port 80")
+	if !strings.Contains(body, "port 80") {
+		t.Errorf("the error should name port 80: %s", body)
+	}
+	c, body = open(map[string]any{"proto": "https", "bindIp": "203.0.113.20", "port": 8443, "hostname": "f.example.test"})
+	want(200, c, body, "the same website without the redirect")
+	c, body = open(map[string]any{"proto": "https", "bindIp": "203.0.113.21", "port": 443, "hostname": "g.example.test", "redirectHttp": true})
+	want(200, c, body, "a redirect where port 80 is free")
+	c, body = open(map[string]any{"proto": "tcp", "bindIp": "203.0.113.21", "port": 80})
+	want(400, c, body, "a raw port 80 where a redirect lives")
+	c, body = open(map[string]any{"proto": "http", "bindIp": "203.0.113.21", "port": 80, "hostname": "h.example.test"})
+	want(200, c, body, "an http website beside a redirect on port 80")
+}

metrics.go

@@ -84,6 +84,11 @@ func (s *Server) metrics(w http.ResponseWriter, r *http.Request) {
 			line("hammerhead_mail_check_ok", "gauge", "1 if the deliverability check passes", b2f(c.PTRMatchesHelo), "relay", x.Name, "org", org, "check", "reverse_dns")
 			line("hammerhead_mail_check_ok", "gauge", "1 if the deliverability check passes", b2f(c.HeloResolves), "relay", x.Name, "org", org, "check", "helo_resolves")
 			line("hammerhead_mail_check_ok", "gauge", "1 if the deliverability check passes", b2f(c.Port25 == "open"), "relay", x.Name, "org", org, "check", "port25")
+			for _, f := range s.mailEgressFindingsLocked(x, c) {
+				if f.Domain != "" || f.ID == "blocklist" || f.ID == "outcomes" {
+					line("hammerhead_mail_finding_ok", "gauge", "1 if a per-domain, blocklist or delivery check passes", b2f(f.Status == "pass"), "relay", x.Name, "org", org, "check", f.Label, "domain", f.Domain)
+				}
+			}
 		}
 	}
 	firing := map[string]int{}

orgs.go

@@ -29,7 +29,9 @@ func (s *Server) createOrg(w http.ResponseWriter, r *http.Request, p *principal)
 	}
 	s.mu.Lock()
 	org, adminRole := s.newOrgLocked(in.Name, in.AllowLocal)
+	if p.User.ID != "" { // a server key belongs to no user
 		s.st.Members = append(s.st.Members, &Member{UserID: p.User.ID, OrgID: org.ID, RoleID: adminRole.ID})
+	}
 	s.save()
 	o := *org
 	s.mu.Unlock()
@@ -349,7 +351,7 @@ func (s *Server) acceptInvite(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 	if !s.allow("invite|"+clientIP(r), 20, 10*time.Minute) {
-		fail(w, 429, errors.New("too many attempts"))
+		fail(w, 429, errors.New("Too many attempts. Wait a few minutes and try again."))
 		return
 	}
 	h := sha(in.Token)
@@ -423,7 +425,8 @@ func (s *Server) listKeys(w http.ResponseWriter, r *http.Request, c *orgCtx) {
 	writeJSON(w, 200, out)
 }
 
-func (s *Server) createKey(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+// issueKey stores a new key for the org ("" = a server key) and returns its ID and full token.
+func (s *Server) issueKey(w http.ResponseWriter, r *http.Request, org, actor string) {
 	var in struct{ Name string }
 	if !decode(w, r, &in) {
 		return
@@ -435,11 +438,82 @@ func (s *Server) createKey(w http.ResponseWriter, r *http.Request, c *orgCtx) {
 	}
 	id, secret := randHex(4), randHex(24)
 	s.mu.Lock()
-	s.st.APIKeys = append(s.st.APIKeys, &APIKey{ID: id, OrgID: c.Org.ID, Name: in.Name, Hash: sha(secret), Created: time.Now()})
+	s.st.APIKeys = append(s.st.APIKeys, &APIKey{ID: id, OrgID: org, Name: in.Name, Hash: sha(secret), Created: time.Now()})
 	s.save()
 	s.mu.Unlock()
-	s.audit(c.Org.ID, c.Actor, "apikey.create", in.Name)
+	s.audit(org, actor, "apikey.create", in.Name)
-	writeJSON(w, 200, map[string]string{"key": "hhk_" + id + "_" + secret})
+	writeJSON(w, 200, map[string]string{"id": id, "key": "hhk_" + id + "_" + secret})
+}
+
+func (s *Server) createKey(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	s.issueKey(w, r, c.Org.ID, c.Actor)
+}
+
+// Server keys manage edges, server-level alerts and organizations, so only a signed-in
+// server administrator may mint, list or revoke them (a key can't make more keys).
+
+func (s *Server) withServerAdminUser(fn func(http.ResponseWriter, *http.Request, *principal)) http.HandlerFunc {
+	return s.withUser(func(w http.ResponseWriter, r *http.Request, p *principal) {
+		if !p.User.ServerAdmin {
+			fail(w, 403, errors.New("server administrators only"))
+			return
+		}
+		fn(w, r, p)
+	})
+}
+
+func (s *Server) listServerKeys(w http.ResponseWriter, r *http.Request, p *principal) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	out := []APIKey{}
+	for _, k := range s.st.APIKeys {
+		if k.OrgID == "" {
+			cp := *k
+			cp.Hash = ""
+			out = append(out, cp)
+		}
+	}
+	writeJSON(w, 200, out)
+}
+
+func (s *Server) createServerKey(w http.ResponseWriter, r *http.Request, p *principal) {
+	s.issueKey(w, r, "", p.User.Email)
+}
+
+func (s *Server) deleteServerKey(w http.ResponseWriter, r *http.Request, p *principal) {
+	id := r.PathValue("id")
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.st.APIKeys = filter(s.st.APIKeys, func(k *APIKey) bool { return !(k.ID == id && k.OrgID == "") })
+	s.save()
+	s.audit("", p.User.Email, "apikey.delete", id)
+	ok(w)
+}
+
+// listOrgs shows the organizations the caller can reach: a server key sees all of them,
+// an org key its own.
+func (s *Server) listOrgs(w http.ResponseWriter, r *http.Request) {
+	p := s.principalFrom(r)
+	if p == nil {
+		fail(w, 401, errors.New("Sign in to continue."))
+		return
+	}
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	out := []Org{}
+	for _, o := range s.st.Orgs {
+		switch {
+		case p.serverKey() || p.User.ServerAdmin:
+			out = append(out, *o)
+		case p.Key != nil:
+			if p.Key.OrgID == o.ID {
+				out = append(out, *o)
+			}
+		case s.member(p.User.ID, o.ID) != nil:
+			out = append(out, *o)
+		}
+	}
+	writeJSON(w, 200, out)
 }
 
 func (s *Server) deleteKey(w http.ResponseWriter, r *http.Request, c *orgCtx) {

paths_test.go

@@ -0,0 +1,34 @@
+package hammerhead
+
+import (
+	"os"
+	"regexp"
+	"testing"
+)
+
+// Every address the dashboard JavaScript navigates to must be served by the server, or a
+// reload on that page would 404.
+func TestDashboardPathsMatchThePage(t *testing.T) {
+	src, err := os.ReadFile("web/index.html")
+	if err != nil {
+		t.Fatal(err)
+	}
+	m := regexp.MustCompile(`TAB_PATH = \{([^}]*)\}`).FindSubmatch(src)
+	if m == nil {
+		t.Fatal("TAB_PATH not found in web/index.html")
+	}
+	paths := regexp.MustCompile(`'(/[a-z/-]+)'`).FindAllSubmatch(m[1], -1)
+	if len(paths) < 10 {
+		t.Fatalf("found only %d page paths", len(paths))
+	}
+	for _, p := range paths {
+		if !isDashboardPath(string(p[1])) {
+			t.Errorf("%s is a dashboard page but the server doesn't serve it", p[1])
+		}
+	}
+	for _, p := range []string{"/api/me", "/nope", "/server", "/server/nope"} {
+		if isDashboardPath(p) {
+			t.Errorf("%s should not serve the dashboard", p)
+		}
+	}
+}

redirects_test.go

@@ -0,0 +1,109 @@
+package hammerhead
+
+import (
+	"encoding/json"
+	"strings"
+	"testing"
+)
+
+// a redirect is a website with a destination instead of a target
+func TestDomainRedirects(t *testing.T) {
+	s, err := New(t.TempDir(), Options{AdminEmail: "admin@test.local", AdminPassword: "password123", MasterKey: key})
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	do := apiClient(t, s)
+	var me struct{ Orgs []struct{ ID string } }
+	_, b := do("GET", "/api/me", nil)
+	json.Unmarshal(b, &me)
+	org := me.Orgs[0].ID
+	var edge, site struct{ ID string }
+	_, b = do("POST", "/api/edges", map[string]any{"name": "e", "tunnelAddr": "203.0.113.5:51820"})
+	json.Unmarshal(b, &edge)
+	_, b = do("POST", "/api/orgs/"+org+"/sites", map[string]any{"name": "home"})
+	json.Unmarshal(b, &site)
+	if code, b := do("POST", "/api/orgs/"+org+"/domains", map[string]any{"name": "example.test", "edgeId": edge.ID, "ip": "203.0.113.10"}); code != 200 {
+		t.Fatalf("domain: %d %s", code, b)
+	}
+	base := "/api/orgs/" + org + "/mappings"
+	open := func(m map[string]any) (int, []byte) {
+		m["edgeId"], m["enabled"], m["bindIp"] = edge.ID, true, "0.0.0.0"
+		return do("POST", base, m)
+	}
+
+	// the redirect needs no target, and anything that only makes sense with one is dropped
+	code, raw := open(map[string]any{"proto": "https", "port": 443, "hostname": "old.example.test", "redirectTo": " https://new.example.test/home ", "redirectPermanent": true, "redirectKeepPath": true,
+		"auth": map[string]any{"password": true}, "rewriteHost": "x", "healthPath": "/h"})
+	if code != 200 {
+		t.Fatalf("create redirect: %d %s", code, raw)
+	}
+	var m struct {
+		ID, RedirectTo, RewriteHost, HealthPath string
+		RedirectPermanent, RedirectKeepPath     bool
+		Targets                                 []any
+		HealthSeconds                           int
+		Auth                                    struct{ Password bool }
+	}
+	json.Unmarshal(raw, &m)
+	if m.RedirectTo != "https://new.example.test/home" || !m.RedirectPermanent || !m.RedirectKeepPath || len(m.Targets) != 0 ||
+		m.Auth.Password || m.RewriteHost != "" || m.HealthPath != "" || m.HealthSeconds != -1 {
+		t.Fatalf("redirect was not normalised: %s", raw)
+	}
+
+	// it shares the port with other websites by hostname, like any website
+	if code, raw := open(map[string]any{"proto": "https", "port": 443, "hostname": "app.example.test", "targets": []any{map[string]any{"siteId": site.ID, "host": "127.0.0.1", "port": 8080}}}); code != 200 {
+		t.Fatalf("a website beside the redirect: %d %s", code, raw)
+	}
+	// and its plain-HTTP twin on port 80 is its own resource
+	if code, raw := open(map[string]any{"proto": "http", "port": 80, "hostname": "old.example.test", "redirectTo": "https://new.example.test/home"}); code != 200 {
+		t.Fatalf("http side of the redirect: %d %s", code, raw)
+	}
+
+	for name, in := range map[string]map[string]any{
+		"not a web port":      {"proto": "tcp", "port": 9000, "redirectTo": "https://new.example.test"},
+		"no scheme":           {"proto": "https", "port": 443, "hostname": "a.example.test", "redirectTo": "new.example.test"},
+		"other scheme":        {"proto": "https", "port": 443, "hostname": "a.example.test", "redirectTo": "ftp://new.example.test"},
+		"credentials":         {"proto": "https", "port": 443, "hostname": "a.example.test", "redirectTo": "https://u:p@new.example.test"},
+		"a dollar sign":       {"proto": "https", "port": 443, "hostname": "a.example.test", "redirectTo": "https://new.example.test/$1"},
+		"back to itself":      {"proto": "https", "port": 443, "hostname": "a.example.test", "redirectTo": "https://a.example.test/"},
+		"itself with a path":  {"proto": "https", "port": 443, "hostname": "a.example.test", "redirectTo": "https://a.example.test/x", "redirectKeepPath": true},
+		"host not in domains": {"proto": "https", "port": 443, "hostname": "a.elsewhere.test", "redirectTo": "https://new.example.test"},
+	} {
+		if code, raw := open(in); code != 400 {
+			t.Errorf("%s: expected 400, got %d %s", name, code, raw)
+		}
+	}
+	// http to https on the same name is the normal use, and is allowed
+	if code, raw := open(map[string]any{"proto": "http", "port": 8080, "hostname": "a.example.test", "redirectTo": "https://a.example.test/"}); code != 200 {
+		t.Errorf("an http to https redirect: %d %s", code, raw)
+	}
+
+	// turning a redirect into a normal website needs a target again
+	code, raw = do("PUT", base+"/"+m.ID, map[string]any{"proto": "https", "edgeId": edge.ID, "bindIp": "0.0.0.0", "port": 443, "hostname": "old.example.test", "enabled": true})
+	if code != 400 || !strings.Contains(string(raw), "target") {
+		t.Errorf("a website with no target and no redirect: %d %s", code, raw)
+	}
+
+	// deleting a site must not sweep away redirects, which have no targets
+	if code, raw := do("DELETE", "/api/orgs/"+org+"/sites/"+site.ID, nil); code != 200 {
+		t.Fatalf("delete site: %d %s", code, raw)
+	}
+	_, raw = do("GET", "/api/orgs/"+org+"/state", nil)
+	var st struct {
+		Mappings []struct{ Hostname, RedirectTo string }
+	}
+	json.Unmarshal(raw, &st)
+	var redirects, apps int
+	for _, x := range st.Mappings {
+		if x.RedirectTo != "" {
+			redirects++
+		}
+		if x.Hostname == "app.example.test" {
+			apps++
+		}
+	}
+	if redirects != 3 || apps != 0 {
+		t.Errorf("after deleting the site: %d redirects (want 3), %d apps (want 0): %s", redirects, apps, raw)
+	}
+}

resauth.go

@@ -54,10 +54,10 @@ func (s *Server) protectedMapping(id string) *model.Mapping {
 func checkReturn(m *model.Mapping, raw string) (*url.URL, error) {
 	u, err := url.Parse(raw)
 	if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
-		return nil, errors.New("invalid return address")
+		return nil, errors.New("This sign-in link is incomplete. Open the page you wanted again and you will be brought back here.")
 	}
 	if (m.Proto == model.ProtoHTTPS) != (u.Scheme == "https") {
-		return nil, errors.New("return address uses the wrong scheme")
+		return nil, errors.New("This sign-in link is not valid. Open the page you wanted again and you will be brought back here.")
 	}
 	host := u.Hostname()
 	port := u.Port()
@@ -68,7 +68,7 @@ func checkReturn(m *model.Mapping, raw string) (*url.URL, error) {
 		}
 	}
 	if !model.HostMatches(m.Hostname, strings.ToLower(host)) || port != fmt.Sprint(m.Port) {
-		return nil, errors.New("return address doesn't belong to this resource")
+		return nil, errors.New("This sign-in link points somewhere else. Open the page you wanted again and you will be brought back here.")
 	}
 	return u, nil
 }
@@ -108,7 +108,7 @@ func (s *Server) resourceInfo(w http.ResponseWriter, r *http.Request) {
 	defer s.mu.Unlock()
 	m := s.protectedMapping(r.PathValue("id"))
 	if m == nil {
-		fail(w, 404, errors.New("this resource doesn't need a login, or doesn't exist"))
+		fail(w, 404, errors.New("This page does not need a sign-in, or it no longer exists. Check the address, or ask the person who shared it."))
 		return
 	}
 	if _, err := checkReturn(m, r.URL.Query().Get("return")); err != nil {
@@ -141,7 +141,7 @@ func (s *Server) grant(w http.ResponseWriter, r *http.Request, m *model.Mapping,
 	}
 	s.mu.Unlock()
 	if key == "" {
-		fail(w, 500, errors.New("edge is not set up for logins"))
+		fail(w, 500, errors.New("This server is not set up for sign-ins yet. Ask the administrator to finish setting it up."))
 		return
 	}
 	tok := authtoken.Sign(key, authtoken.Token{Mapping: m.ID, Sub: sub, Kind: authtoken.KindCallback, Epoch: m.AuthEpoch, Exp: time.Now().Add(2 * time.Minute).Unix()})
@@ -158,7 +158,7 @@ func (s *Server) resourceAuth(method string, fn func(w http.ResponseWriter, r *h
 			return
 		}
 		if !s.allow("res|"+method+"|"+clientIP(r)+"|"+r.PathValue("id"), 12, 5*time.Minute) {
-			fail(w, 429, errors.New("too many attempts, wait a few minutes"))
+			fail(w, 429, errors.New("Too many attempts. Wait a few minutes and try again."))
 			return
 		}
 		s.mu.Lock()
@@ -169,7 +169,7 @@ func (s *Server) resourceAuth(method string, fn func(w http.ResponseWriter, r *h
 		}
 		s.mu.Unlock()
 		if pm == nil {
-			fail(w, 404, errors.New("no such resource"))
+			fail(w, 404, errors.New("That page does not exist. Check the address."))
 			return
 		}
 		back, err := checkReturn(&m, body["return"])
@@ -191,7 +191,7 @@ func (s *Server) resourcePassword(w http.ResponseWriter, r *http.Request, m *mod
 	}
 	if bcrypt.CompareHashAndPassword([]byte(hash), []byte(body["password"])) != nil || hash == string(dummyHash) {
 		time.Sleep(300 * time.Millisecond)
-		fail(w, 401, errors.New("wrong password"))
+		fail(w, 401, errors.New("That password is not right. Check it and try again."))
 		return
 	}
 	s.grant(w, r, m, back, "password")
@@ -207,7 +207,7 @@ func (s *Server) resourcePin(w http.ResponseWriter, r *http.Request, m *model.Ma
 	}
 	if bcrypt.CompareHashAndPassword([]byte(hash), []byte(body["pin"])) != nil || hash == string(dummyHash) {
 		time.Sleep(300 * time.Millisecond)
-		fail(w, 401, errors.New("wrong PIN"))
+		fail(w, 401, errors.New("That PIN is not right. Check it and try again."))
 		return
 	}
 	s.grant(w, r, m, back, "pin")
@@ -216,7 +216,7 @@ func (s *Server) resourcePin(w http.ResponseWriter, r *http.Request, m *model.Ma
 func (s *Server) resourceSSO(w http.ResponseWriter, r *http.Request, m *model.Mapping, back *url.URL, body map[string]string) {
 	p := s.principalFrom(r)
 	if p == nil || p.User.ID == "" || !m.Auth.SSO {
-		fail(w, 401, errors.New("sign in first"))
+		fail(w, 401, errors.New("Sign in first, then try again."))
 		return
 	}
 	s.mu.Lock()
@@ -224,7 +224,7 @@ func (s *Server) resourceSSO(w http.ResponseWriter, r *http.Request, m *model.Ma
 	allowed := u != nil && s.canAccess(u, m)
 	s.mu.Unlock()
 	if !allowed {
-		fail(w, 403, errors.New("your account doesn't have access to this resource"))
+		fail(w, 403, errors.New("Your account does not have access to this page. Ask the person who shared it to add you."))
 		return
 	}
 	s.grant(w, r, m, back, p.User.Email)
@@ -243,7 +243,7 @@ func mailAllowed(list []string, email string) bool {
 func (s *Server) resourceEmailStart(w http.ResponseWriter, r *http.Request, m *model.Mapping, back *url.URL, body map[string]string) {
 	email := strings.ToLower(strings.TrimSpace(body["email"]))
 	if !m.Auth.Email || !s.mailConfigured() {
-		fail(w, 400, errors.New("email codes are not available for this resource"))
+		fail(w, 400, errors.New("Email codes are not turned on for this page. Use another sign-in method."))
 		return
 	}
 	// always answer the same way so addresses can't be probed
@@ -279,7 +279,7 @@ func (s *Server) resourceEmailVerify(w http.ResponseWriter, r *http.Request, m *
 	}
 	s.mu.Unlock()
 	if !good || !mailAllowed(m.Auth.AllowedMails, email) {
-		fail(w, 401, errors.New("wrong or expired code"))
+		fail(w, 401, errors.New("That code is wrong or has expired. Request a new one and try again."))
 		return
 	}
 	s.grant(w, r, m, back, email)
@@ -303,7 +303,7 @@ func (s *Server) resourceShare(w http.ResponseWriter, r *http.Request, m *model.
 	}
 	s.mu.Unlock()
 	if !valid {
-		fail(w, 401, errors.New("this share link is invalid, expired or used up"))
+		fail(w, 401, errors.New("This share link is invalid, expired or already used up. Ask for a new one."))
 		return
 	}
 	if name == "" {

sandbar.go

@@ -0,0 +1,343 @@
+package hammerhead
+
+import (
+	"bytes"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"log"
+	"net"
+	"net/http"
+	"net/url"
+	"sort"
+	"strconv"
+	"strings"
+	"time"
+
+	"git.veillax.com/Veillax/reef/model"
+	"git.veillax.com/Veillax/reef/netguard"
+)
+
+// Sandbar monitors: with this switched on under Server, Settings, every enabled
+// resource that has health checks gets a monitor on a Sandbar server, which then
+// tracks its uptime, shows it on the status page and pages whoever is on call.
+// Sandbar probes the public address (so it sees what visitors see), where
+// Hammerhead's own health checks look at the targets behind it. Hammerhead only
+// ever changes the monitors it created: the link from resource to monitor is
+// kept in the settings, so a monitor you add in Sandbar by hand is never touched.
+
+// SandbarSync is the connection to Sandbar for creating monitors.
+type SandbarSync struct {
+	Enabled      bool              `json:"enabled"`
+	URL          string            `json:"url"`
+	Token        string            `json:"token"`              // an administrator's API key (secret, never sent back to the browser)
+	PolicyID     string            `json:"policyId"`           // escalation policy for the alerts the monitors raise; blank pages nobody
+	Public       bool              `json:"public"`             // show the monitors on Sandbar's status page
+	AllowPrivate bool              `json:"allowPrivate"`       // Sandbar is on a private network, and so are the edge addresses it will probe
+	Monitors     map[string]string `json:"monitors,omitempty"` // resource id -> Sandbar monitor id
+}
+
+// sandbarSyncStatus is how the last sync went, for the settings page.
+type sandbarSyncStatus struct {
+	At       time.Time `json:"at"`
+	Error    string    `json:"error"`
+	Monitors int       `json:"monitors"`
+}
+
+const sandbarSyncEvery = time.Minute
+
+type monitorSpec struct {
+	Name      string
+	Kind      string
+	Target    string
+	IntervalS int
+	Expect    string
+}
+
+// sandbarEdgeHost is the address visitors reach a resource's edge on: the address it is
+// bound to, else the first public address the edge reported, else the host of its
+// tunnel endpoint.
+func sandbarEdgeHost(e *model.Edge, bind string) string {
+	if ip := net.ParseIP(bind); ip != nil && !ip.IsUnspecified() {
+		return bind
+	}
+	var first string
+	for _, a := range e.IPs {
+		ip := net.ParseIP(a)
+		if ip == nil || ip.IsLoopback() || ip.IsLinkLocalUnicast() {
+			continue
+		}
+		if !ip.IsPrivate() {
+			return a
+		}
+		if first == "" {
+			first = a
+		}
+	}
+	if first != "" {
+		return first
+	}
+	host, _, _ := net.SplitHostPort(e.TunnelAddr)
+	return host
+}
+
+// monitorFor describes the Sandbar monitor a resource should have, or false when
+// it has nothing worth probing (a redirect, UDP, health checks off, a wildcard).
+func (s *Server) monitorFor(m *model.Mapping) (monitorSpec, bool) {
+	if !m.Enabled || m.HealthSeconds == -1 || m.RedirectTo != "" || m.Proto == model.ProtoUDP {
+		return monitorSpec{}, false
+	}
+	e := s.findEdge(m.EdgeID)
+	if e == nil {
+		return monitorSpec{}, false
+	}
+	spec := monitorSpec{Name: m.Name, IntervalS: 30}
+	if m.HealthSeconds > 0 {
+		spec.IntervalS = max(m.HealthSeconds, 10)
+	}
+	host := m.Hostname
+	if strings.HasPrefix(host, "*") {
+		return monitorSpec{}, false
+	}
+	if host == "" {
+		if host = sandbarEdgeHost(e, m.BindIP); host == "" {
+			return monitorSpec{}, false
+		}
+	}
+	addr := net.JoinHostPort(host, strconv.Itoa(m.Port))
+	switch m.Proto {
+	case model.ProtoHTTP, model.ProtoHTTPS:
+		u := url.URL{Scheme: m.Proto, Host: addr, Path: m.HealthPath}
+		if (m.Proto == model.ProtoHTTP && m.Port == 80) || (m.Proto == model.ProtoHTTPS && m.Port == 443) {
+			u.Host = strings.Trim(host, "[]") // a bare IPv6 address needs its brackets back
+			if strings.Contains(host, ":") {
+				u.Host = "[" + u.Host + "]"
+			}
+		}
+		spec.Kind, spec.Target = "http", u.String()
+		if m.Auth.Protected() {
+			spec.Expect = "200-399" // visitors are sent to a sign-in page, which is the site working
+		}
+	default:
+		spec.Kind, spec.Target = "tcp", addr
+	}
+	return spec, true
+}
+
+// desiredMonitorsLocked is the monitor every qualifying resource should have, by resource id.
+func (s *Server) desiredMonitorsLocked() map[string]monitorSpec {
+	out := map[string]monitorSpec{}
+	for _, m := range s.st.Mappings {
+		if spec, ok := s.monitorFor(m); ok {
+			out[m.ID] = spec
+		}
+	}
+	return out
+}
+
+func sandbarCall(cfg SandbarSync, method, path string, body any, out any) error {
+	var rd io.Reader
+	if body != nil {
+		raw, _ := json.Marshal(body)
+		rd = bytes.NewReader(raw)
+	}
+	req, err := http.NewRequest(method, strings.TrimRight(cfg.URL, "/")+path, rd)
+	if err != nil {
+		return err
+	}
+	req.Header.Set("Content-Type", "application/json")
+	req.Header.Set("Authorization", "Bearer "+cfg.Token)
+	resp, err := netguard.Client(cfg.AllowPrivate).Do(req)
+	if err != nil {
+		return err
+	}
+	defer resp.Body.Close()
+	switch {
+	case resp.StatusCode == 401:
+		return errors.New("Sandbar rejected the API key")
+	case resp.StatusCode == 403:
+		return errors.New("Sandbar needs an administrator's API key to manage monitors")
+	case resp.StatusCode == 404 && method == "DELETE":
+		return nil // already gone
+	case resp.StatusCode >= 300:
+		msg, _ := io.ReadAll(io.LimitReader(resp.Body, 300))
+		return fmt.Errorf("Sandbar answered %s %s", resp.Status, strings.TrimSpace(string(msg)))
+	}
+	if out != nil {
+		return json.NewDecoder(resp.Body).Decode(out)
+	}
+	return nil
+}
+
+// syncSandbar makes Sandbar's monitors match the resources: it creates the ones
+// that are missing, corrects the ones that drifted, and removes those whose
+// resource is gone or no longer has health checks.
+func (s *Server) syncSandbar() (err error) {
+	s.sandbarMu.Lock()
+	defer s.sandbarMu.Unlock()
+	s.mu.Lock()
+	cfg := s.st.Settings.Sandbar
+	if !cfg.Enabled {
+		s.mu.Unlock()
+		return nil
+	}
+	want := s.desiredMonitorsLocked()
+	s.mu.Unlock()
+	links := map[string]string{}
+	for k, v := range cfg.Monitors {
+		links[k] = v
+	}
+	defer func() {
+		st := sandbarSyncStatus{At: time.Now(), Monitors: len(links)}
+		if err != nil {
+			st.Error = err.Error()
+		}
+		s.mu.Lock()
+		s.sandbarStatus = st
+		if now := &s.st.Settings.Sandbar; now.URL == cfg.URL && !sameLinks(now.Monitors, links) {
+			now.Monitors = links // kept even after a failure: a monitor already created must not be created twice
+			s.save()
+		}
+		s.mu.Unlock()
+	}()
+
+	var remote []map[string]any
+	if err := sandbarCall(cfg, "GET", "/api/v1/monitors", nil, &remote); err != nil {
+		return err
+	}
+	byID := map[string]map[string]any{}
+	for _, m := range remote {
+		if id, _ := m["id"].(string); id != "" {
+			byID[id] = m
+		}
+	}
+
+	ids := make([]string, 0, len(want))
+	for id := range want {
+		ids = append(ids, id)
+	}
+	sort.Strings(ids)
+	var errs []error
+	for _, id := range ids {
+		spec := want[id]
+		fields := map[string]any{"name": spec.Name, "kind": spec.Kind, "target": spec.Target, "intervalS": spec.IntervalS, "expect": spec.Expect,
+			"policyId": cfg.PolicyID, "public": cfg.Public, "allowPrivate": cfg.AllowPrivate, "enabled": true}
+		cur, exists := byID[links[id]]
+		if !exists {
+			var made struct{ ID string }
+			if err := sandbarCall(cfg, "POST", "/api/v1/monitors", fields, &made); err != nil {
+				errs = append(errs, fmt.Errorf("%s: %w", spec.Name, err))
+				continue
+			}
+			links[id] = made.ID
+			continue
+		}
+		drift := false
+		for k, v := range fields {
+			if !sameJSON(cur[k], v) {
+				cur[k], drift = v, true
+			}
+		}
+		if drift { // the whole document is replaced, so what Sandbar added (token, retries) rides along
+			if err := sandbarCall(cfg, "PUT", "/api/v1/monitors/"+links[id], cur, nil); err != nil {
+				errs = append(errs, fmt.Errorf("%s: %w", spec.Name, err))
+			}
+		}
+	}
+	for id, rid := range links {
+		if _, still := want[id]; still {
+			continue
+		}
+		if err := sandbarCall(cfg, "DELETE", "/api/v1/monitors/"+rid, nil, nil); err != nil {
+			errs = append(errs, err)
+			continue
+		}
+		delete(links, id)
+	}
+	return errors.Join(errs...)
+}
+
+func sameJSON(a, b any) bool {
+	x, _ := json.Marshal(a)
+	y, _ := json.Marshal(b)
+	if string(x) == "null" { // a field Sandbar left out is its zero value
+		x = []byte(`""`)
+		if _, isNum := b.(int); isNum {
+			x = []byte("0")
+		} else if _, isBool := b.(bool); isBool {
+			x = []byte("false")
+		}
+	}
+	return bytes.Equal(x, y)
+}
+
+func sameLinks(a, b map[string]string) bool {
+	if len(a) != len(b) {
+		return false
+	}
+	for k, v := range a {
+		if b[k] != v {
+			return false
+		}
+	}
+	return true
+}
+
+func (s *Server) sandbarLoop() {
+	t := time.NewTicker(sandbarSyncEvery)
+	defer t.Stop()
+	for {
+		select {
+		case <-s.stop:
+			return
+		case <-t.C:
+			if err := s.syncSandbar(); err != nil {
+				log.Printf("sandbar monitors: %v", err)
+			}
+		}
+	}
+}
+
+// syncSandbarNow is the "Sync now" button: the same sync, with its outcome shown.
+func (s *Server) syncSandbarNow(w http.ResponseWriter, r *http.Request, p *principal) {
+	s.mu.Lock()
+	on := s.st.Settings.Sandbar.Enabled
+	s.mu.Unlock()
+	if !on {
+		fail(w, 400, errors.New("Turn on Sandbar monitors and save the settings first."))
+		return
+	}
+	if err := s.syncSandbar(); err != nil {
+		fail(w, 400, err)
+		return
+	}
+	s.mu.Lock()
+	st := s.sandbarStatus
+	s.mu.Unlock()
+	writeJSON(w, 200, st)
+}
+
+// cleanSandbar validates the settings in `in`, filling in what a blank field means.
+// Caller holds s.mu.
+func (s *Server) cleanSandbar(in *SandbarSync) error {
+	old := s.st.Settings.Sandbar
+	in.URL, in.PolicyID = strings.TrimRight(strings.TrimSpace(in.URL), "/"), strings.TrimSpace(in.PolicyID)
+	if in.Token = strings.TrimSpace(in.Token); in.Token == "" {
+		in.Token = old.Token // blank keeps the saved API key
+	}
+	in.Monitors = old.Monitors // not editable here
+	if in.URL != old.URL {
+		in.Monitors = nil // those belong to the previous server
+	}
+	if !in.Enabled {
+		return nil
+	}
+	if u, err := url.Parse(in.URL); err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
+		return errors.New("enter Sandbar's address, such as https://sandbar.example.com")
+	}
+	if in.Token == "" {
+		return errors.New("enter an administrator's API key from Sandbar's Account page")
+	}
+	return nil
+}

sandbar_test.go

@@ -0,0 +1,350 @@
+package hammerhead
+
+import (
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strconv"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/reef/model"
+)
+
+// fakeSandbar keeps monitors in memory the way Sandbar's /api/v1/monitors does.
+type fakeSandbar struct {
+	URL string
+	mu  sync.Mutex
+	mon map[string]map[string]any
+	n   int
+	// requests that carried the wrong key are counted, not served
+	badAuth int
+	calls   []string
+}
+
+func newFakeSandbar(t *testing.T) *fakeSandbar {
+	f := &fakeSandbar{mon: map[string]map[string]any{}}
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		f.mu.Lock()
+		defer f.mu.Unlock()
+		if r.Header.Get("Authorization") != "Bearer sb_admin" {
+			f.badAuth++
+			http.Error(w, "no", 401)
+			return
+		}
+		f.calls = append(f.calls, r.Method)
+		id := strings.TrimPrefix(r.URL.Path, "/api/v1/monitors/")
+		switch {
+		case r.Method == "GET" && r.URL.Path == "/api/v1/monitors":
+			out := []any{}
+			for _, m := range f.mon {
+				out = append(out, m)
+			}
+			json.NewEncoder(w).Encode(out)
+		case r.Method == "POST" && r.URL.Path == "/api/v1/monitors":
+			var m map[string]any
+			json.NewDecoder(r.Body).Decode(&m)
+			f.n++
+			m["id"], m["retries"] = "mon"+strconv.Itoa(f.n), 2 // Sandbar adds its own fields
+			f.mon[m["id"].(string)] = m
+			json.NewEncoder(w).Encode(m)
+		case r.Method == "PUT" && f.mon[id] != nil:
+			var m map[string]any
+			json.NewDecoder(r.Body).Decode(&m)
+			f.mon[id] = m
+			json.NewEncoder(w).Encode(m)
+		case r.Method == "DELETE":
+			delete(f.mon, id)
+			w.Write([]byte(`{"ok":true}`))
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	t.Cleanup(ts.Close)
+	f.URL = ts.URL
+	return f
+}
+
+func (f *fakeSandbar) names() map[string]string { // monitor id -> "kind target"
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	out := map[string]string{}
+	for id, m := range f.mon {
+		out[id] = m["kind"].(string) + " " + m["target"].(string)
+	}
+	return out
+}
+
+func (f *fakeSandbar) count(method string) int {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	n := 0
+	for _, c := range f.calls {
+		if c == method {
+			n++
+		}
+	}
+	return n
+}
+
+func enableSandbarSync(f *fx, sb *fakeSandbar) {
+	f.s.mu.Lock()
+	f.s.st.Edges[0].IPs = []string{"10.0.0.5", "203.0.113.7"}
+	f.s.st.Settings.Sandbar = SandbarSync{Enabled: true, URL: sb.URL, Token: "sb_admin", PolicyID: "pol1", Public: true, AllowPrivate: true}
+	f.s.mu.Unlock()
+}
+
+func TestSandbarMonitorsFollowTheResources(t *testing.T) {
+	f := newFx(t)
+	sb := newFakeSandbar(t)
+	enableSandbarSync(f, sb)
+	f.s.mu.Lock()
+	f.s.st.Mappings = append(f.s.st.Mappings,
+		&model.Mapping{ID: "m2", OrgID: f.org, Name: "Blog", EdgeID: "e1", Proto: "https", Port: 443, Hostname: "blog.example.com", HealthPath: "/ping", Enabled: true,
+			Targets: []model.Target{{SiteID: "s1", Host: "192.168.1.31", Port: 80}}},
+		&model.Mapping{ID: "m3", OrgID: f.org, Name: "Docs", EdgeID: "e1", Proto: "https", Port: 8443, Hostname: "docs.example.com", Enabled: true, Auth: model.Auth{Password: true},
+			Targets: []model.Target{{SiteID: "s1", Host: "192.168.1.32", Port: 80}}},
+		&model.Mapping{ID: "m4", OrgID: f.org, Name: "Unchecked", EdgeID: "e1", Proto: "tcp", Port: 2222, HealthSeconds: -1, Enabled: true},
+		&model.Mapping{ID: "m5", OrgID: f.org, Name: "Voice", EdgeID: "e1", Proto: "udp", Port: 9987, Enabled: true},
+		&model.Mapping{ID: "m6", OrgID: f.org, Name: "Wildcard", EdgeID: "e1", Proto: "https", Port: 443, Hostname: "*.example.com", Enabled: true},
+		&model.Mapping{ID: "m7", OrgID: f.org, Name: "Off", EdgeID: "e1", Proto: "tcp", Port: 3333},
+	)
+	f.s.mu.Unlock()
+
+	if err := f.s.syncSandbar(); err != nil {
+		t.Fatal(err)
+	}
+	got := sb.names()
+	want := map[string]bool{"tcp 203.0.113.7:25565": true, "http https://blog.example.com/ping": true, "http https://docs.example.com:8443": true}
+	if len(got) != len(want) {
+		t.Fatalf("monitors created: %v", got)
+	}
+	for _, v := range got {
+		if !want[v] {
+			t.Errorf("unexpected monitor %q", v)
+		}
+	}
+	for _, m := range sb.mon {
+		if m["policyId"] != "pol1" || m["public"] != true || m["enabled"] != true {
+			t.Errorf("monitor settings not applied: %v", m)
+		}
+		if m["name"] == "Docs" && m["expect"] != "200-399" {
+			t.Errorf("a protected site should accept its sign-in redirect: %v", m)
+		}
+	}
+
+	// A second pass changes nothing.
+	posts := sb.count("POST")
+	if err := f.s.syncSandbar(); err != nil {
+		t.Fatal(err)
+	}
+	if sb.count("POST") != posts || sb.count("PUT") != 0 || sb.count("DELETE") != 0 {
+		t.Fatalf("a settled sync still wrote: %v", sb.calls)
+	}
+
+	// A rename and a changed policy are pushed, and Sandbar's own fields survive.
+	f.s.mu.Lock()
+	f.s.findMapping("m1").Name = "Minecraft Java"
+	f.s.st.Settings.Sandbar.PolicyID = "pol2"
+	f.s.mu.Unlock()
+	if err := f.s.syncSandbar(); err != nil {
+		t.Fatal(err)
+	}
+	var renamed bool
+	for _, m := range sb.mon {
+		if m["policyId"] != "pol2" || m["retries"] != float64(2) {
+			t.Errorf("update lost data or missed the policy: %v", m)
+		}
+		renamed = renamed || m["name"] == "Minecraft Java"
+	}
+	if !renamed {
+		t.Error("the rename was not pushed")
+	}
+
+	// Deleting a resource, or turning its checks off, removes its monitor; one added by hand stays.
+	sb.mu.Lock()
+	sb.mon["hand"] = map[string]any{"id": "hand", "name": "mine", "kind": "tcp", "target": "x:1"}
+	sb.mu.Unlock()
+	f.s.mu.Lock()
+	f.s.findMapping("m2").HealthSeconds = -1
+	f.s.st.Mappings = filter(f.s.st.Mappings, func(m *model.Mapping) bool { return m.ID != "m3" })
+	f.s.mu.Unlock()
+	if err := f.s.syncSandbar(); err != nil {
+		t.Fatal(err)
+	}
+	got = sb.names()
+	if len(got) != 2 || got["hand"] == "" {
+		t.Fatalf("after removing resources: %v", got)
+	}
+	f.s.mu.Lock()
+	links := len(f.s.st.Settings.Sandbar.Monitors)
+	f.s.mu.Unlock()
+	if links != 1 {
+		t.Errorf("links kept for %d resources, want 1", links)
+	}
+}
+
+func TestSandbarSyncRecreatesAMonitorDeletedInSandbarAndSurvivesARestart(t *testing.T) {
+	f := newFx(t)
+	sb := newFakeSandbar(t)
+	enableSandbarSync(f, sb)
+	if err := f.s.syncSandbar(); err != nil {
+		t.Fatal(err)
+	}
+	sb.mu.Lock()
+	sb.mon = map[string]map[string]any{}
+	sb.mu.Unlock()
+	if err := f.s.syncSandbar(); err != nil {
+		t.Fatal(err)
+	}
+	if len(sb.names()) != 1 {
+		t.Fatalf("not recreated: %v", sb.names())
+	}
+
+	// the links are stored, so a restart does not create a second monitor
+	f.s.mu.Lock()
+	err := f.s.save()
+	f.s.mu.Unlock()
+	if err != nil {
+		t.Fatal(err)
+	}
+	dir, key := f.s.dir, f.s.opts.MasterKey
+	f.s.Close()
+	s2, err := New(dir, Options{MasterKey: key, AlertInterval: time.Hour})
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s2.Close()
+	if err := s2.syncSandbar(); err != nil {
+		t.Fatal(err)
+	}
+	if len(sb.names()) != 1 {
+		t.Fatalf("a restart duplicated the monitor: %v", sb.names())
+	}
+}
+
+func TestSandbarSyncReportsABadKeyAndIsOffByDefault(t *testing.T) {
+	f := newFx(t)
+	sb := newFakeSandbar(t)
+	if err := f.s.syncSandbar(); err != nil || sb.badAuth != 0 || len(sb.calls) != 0 {
+		t.Fatal("sync ran while switched off")
+	}
+	enableSandbarSync(f, sb)
+	f.s.mu.Lock()
+	f.s.st.Settings.Sandbar.Token = "wrong"
+	f.s.mu.Unlock()
+	if err := f.s.syncSandbar(); err == nil || !strings.Contains(err.Error(), "API key") {
+		t.Fatalf("error: %v", err)
+	}
+	f.s.mu.Lock()
+	msg := f.s.sandbarStatus.Error
+	f.s.mu.Unlock()
+	if msg == "" {
+		t.Error("the failure was not recorded for the settings page")
+	}
+}
+
+func TestSandbarSettingsKeepTheKeySealedAndHidden(t *testing.T) {
+	f := newFx(t)
+	f.s.mu.Lock()
+	in := SandbarSync{Enabled: true, URL: " https://sandbar.example.com/ ", Token: " sb_admin "}
+	err := f.s.cleanSandbar(&in)
+	f.s.st.Settings.Sandbar = in
+	f.s.save()
+	f.s.mu.Unlock()
+	if err != nil || in.URL != "https://sandbar.example.com" || in.Token != "sb_admin" {
+		t.Fatalf("%v %+v", err, in)
+	}
+
+	// a blank key in a later save keeps the saved one; a new address forgets the old server's links
+	f.s.mu.Lock()
+	f.s.st.Settings.Sandbar.Monitors = map[string]string{"m1": "mon1"}
+	next := SandbarSync{Enabled: true, URL: "https://other.example.com"}
+	err = f.s.cleanSandbar(&next)
+	f.s.mu.Unlock()
+	if err != nil || next.Token != "sb_admin" || next.Monitors != nil {
+		t.Fatalf("%v %+v", err, next)
+	}
+	for _, bad := range []SandbarSync{{Enabled: true, URL: "not a url", Token: "k"}, {Enabled: true, URL: "https://x.example.com", Token: ""}} {
+		bad := bad
+		f.s.mu.Lock()
+		f.s.st.Settings.Sandbar.Token = ""
+		err := f.s.cleanSandbar(&bad)
+		f.s.mu.Unlock()
+		if err == nil {
+			t.Errorf("accepted %+v", bad)
+		}
+	}
+
+	// at rest the key is encrypted, and the settings API never returns it
+	raw, _ := json.Marshal(f.s.db.sealed(f.s.st.Settings))
+	if strings.Contains(string(raw), "sb_admin") {
+		t.Error("the API key is stored in the clear")
+	}
+	f.s.mu.Lock()
+	f.s.st.Settings.Sandbar.Token = "sb_admin"
+	f.s.mu.Unlock()
+	rec := httptest.NewRecorder()
+	f.s.getSettings(rec, httptest.NewRequest("GET", "/api/admin/settings", nil), nil)
+	if strings.Contains(rec.Body.String(), "sb_admin") || !strings.Contains(rec.Body.String(), `"sandbarTokenSet":true`) {
+		t.Errorf("settings response: %s", rec.Body.String())
+	}
+}
+
+func TestTopicChannelPublishes(t *testing.T) {
+	f := newFx(t)
+	var gotPath, gotTitle, gotPrio, gotAuth, gotBody string
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		b := make([]byte, 512)
+		n, _ := r.Body.Read(b)
+		gotPath, gotTitle, gotPrio, gotAuth, gotBody = r.URL.Path, r.Header.Get("Title"), r.Header.Get("Priority"), r.Header.Get("Authorization"), string(b[:n])
+	}))
+	defer ts.Close()
+	ch := AlertChannel{ID: "t1", Kind: "topic", Target: ts.URL + "/", Topic: "ops", Token: "sb_k", AllowPrivate: true, Enabled: true}
+	n := alertNote{Status: "firing", Org: "Acme", Name: "Minecraft", Message: "Minecraft can't listen", Since: time.Now()}
+	if err := f.s.send(ch, n); err != nil {
+		t.Fatal(err)
+	}
+	if gotPath != "/ops" || gotTitle != "[Hammerhead] Problem: Minecraft" || gotPrio != "urgent" || gotAuth != "Bearer sb_k" || !strings.Contains(gotBody, "can't listen") {
+		t.Errorf("published %q %q %q %q %q", gotPath, gotTitle, gotPrio, gotAuth, gotBody)
+	}
+	n.Status, ch.Token = "resolved", ""
+	if err := f.s.send(ch, n); err != nil {
+		t.Fatal(err)
+	}
+	if gotPrio != "default" || gotAuth != "" || !strings.HasPrefix(gotBody, "Resolved") {
+		t.Errorf("recovery %q %q %q", gotPrio, gotAuth, gotBody)
+	}
+
+	// a refusal surfaces as an error
+	deny := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { http.Error(w, "no", 401) }))
+	defer deny.Close()
+	ch.Target = deny.URL
+	if err := f.s.send(ch, n); err == nil || !strings.Contains(err.Error(), "API key") {
+		t.Errorf("error: %v", err)
+	}
+}
+
+func TestTopicChannelValidation(t *testing.T) {
+	f := newFx(t)
+	for _, ch := range []AlertChannel{
+		{Kind: "topic", Target: "not a url", Topic: "ops"},
+		{Kind: "topic", Target: "https://sandbar.example.com", Topic: ""},
+		{Kind: "topic", Target: "https://sandbar.example.com", Topic: "has space"},
+		{Kind: "topic", Target: "https://sandbar.example.com", Topic: "ops", AllowPrivate: true}, // needs a trusted caller
+	} {
+		ch := ch
+		if err := f.s.validateChannel(&ch, false); err == nil {
+			t.Fatalf("accepted %+v", ch)
+		}
+	}
+	ok := AlertChannel{Kind: "topic", Target: "https://sandbar.example.com", Topic: " ops ", Token: " ", Policy: "x", Format: "slack"}
+	if err := f.s.validateChannel(&ok, false); err != nil || ok.Topic != "ops" || ok.Token != "" || ok.Policy != "" || ok.Format != "" {
+		t.Fatalf("%v %+v", err, ok)
+	}
+	if p := preview(ok); p != "https://sandbar.example.com/ops" {
+		t.Errorf("preview %q", p)
+	}
+}

scripts/lint-design.mjs

@@ -0,0 +1,57 @@
+// Style gate for the dashboard, following the Lightwell design language (see docs/design.md).
+// Fails on em or en dashes, emoji, exclamation marks in interface copy, and raw colors outside the
+// token block of web/index.html. Run with: node scripts/lint-design.mjs
+import { readdirSync, readFileSync, statSync } from "node:fs";
+import { extname, join, relative, sep } from "node:path";
+
+const root = new URL("..", import.meta.url).pathname.replace(/^\/([A-Za-z]:)/, "$1");
+const targets = ["web", "docs", "README.md", "deploy", "scripts", "cmd", "."];
+const skipDirs = new Set(["node_modules", ".git", "e2e", "internal"]);
+const exts = new Set([".html", ".md", ".go", ".sh", ".mjs", ".example"]);
+
+const dash = /[–—−]/;
+const emoji = /[\u{1F300}-\u{1FAFF}\u{2600}-\u{27BF}\u{1F900}-\u{1F9FF}\u{FE0F}\u{2B50}\u{2705}\u{274C}]/u;
+const rawColor = /#[0-9a-fA-F]{3,8}\b|\brgba?\(|\bhsla?\(/;
+
+const seen = new Set();
+function* walk(p, top) {
+  const st = statSync(p, { throwIfNoEntry: false });
+  if (!st || seen.has(p)) return;
+  if (st.isFile()) { seen.add(p); return yield p; }
+  for (const name of readdirSync(p)) {
+    if (skipDirs.has(name) || (top && name.startsWith("."))) continue;
+    const full = join(p, name);
+    // the "." target only contributes the files at the top level
+    if (p === root && statSync(full).isDirectory()) continue;
+    yield* walk(full, false);
+  }
+}
+
+const problems = [];
+for (const t of targets) {
+  for (const file of walk(join(root, t), true)) {
+    if (!exts.has(extname(file)) && !file.endsWith(".env.example")) continue;
+    const rel = relative(root, file).split(sep).join("/");
+    if (rel.endsWith("go.sum") || rel === "scripts/lint-design.mjs") continue;
+    const isPage = rel.startsWith("web/") && rel.endsWith(".html");
+    let inTokens = false;
+    readFileSync(file, "utf8").split("\n").forEach((line, i) => {
+      const at = (msg) => problems.push(`${rel}:${i + 1}: ${msg}`);
+      if (line.includes("tokens:begin")) inTokens = true;
+      if (dash.test(line)) at("em dash, en dash or minus sign. Use a comma, colon, period or the word to.");
+      if (emoji.test(line)) at("emoji or pictographic symbol. Use an approved outline icon.");
+      if (isPage && !inTokens && rawColor.test(line.replace(/\/\*.*?\*\//g, "")) && !/placeholder|href="#|&#/.test(line)) {
+        at("raw color outside the token block. Use a token.");
+      }
+      if (isPage && /(?:toast|textContent\s*=|class:'(?:empty|notice)')[^;\n]*'[^']*[A-Za-z]![^']*'/.test(line)) at("exclamation mark in interface copy.");
+      if (line.includes("tokens:end")) inTokens = false;
+    });
+  }
+}
+
+if (problems.length) {
+  console.error(problems.join("\n"));
+  console.error(`\n${problems.length} style problem(s).`);
+  process.exit(1);
+}
+console.log("Style checks passed.");

selfedge.go

@@ -0,0 +1,71 @@
+package hammerhead
+
+import (
+	"errors"
+	"fmt"
+	"net"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+
+	"git.veillax.com/Veillax/reef/model"
+)
+
+const selfMappingID = "hammerhead-dashboard"
+
+// ensureSelfEdge makes sure this server has an edge of its own, for a Blacktip on the same
+// machine, and writes that edge's token to <data>/self-edge.env for the installer. The edge
+// also carries a built-in https route for the dashboard (see selfMapping), so no separate
+// reverse proxy is needed in front of Hammerhead.
+func (s *Server) ensureSelfEdge() error {
+	o := s.opts
+	if o.SelfDomain == "" {
+		return nil
+	}
+	host, port, err := net.SplitHostPort(o.SelfTarget)
+	if err != nil {
+		return fmt.Errorf("HAMMERHEAD_SELF_DOMAIN needs a listen address like 127.0.0.1:8080: %w", err)
+	}
+	if n, _ := strconv.Atoi(port); n < 1 || host == "" {
+		return errors.New("the dashboard's listen address needs a host and a port")
+	}
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	e := s.findEdge(s.st.Settings.SelfEdgeID)
+	if e == nil {
+		e = &model.Edge{ID: newID(), Name: "Hammerhead server", Token: randHex(24), AuthKey: randHex(32),
+			TunnelAddr: o.SelfTunnel, TunnelIP: s.allocIPLocked(), OrgIDs: []string{}, IPs: []string{}}
+		s.st.Edges = append(s.st.Edges, e)
+		s.st.Sites = append(s.st.Sites, &model.Site{ID: newID(), Name: e.Name + " (this machine)",
+			Local: true, EdgeID: e.ID, Connected: []string{}})
+		s.st.Settings.SelfEdgeID = e.ID
+		s.bump()
+	}
+	body := "# written by Hammerhead for the Blacktip on this machine; keep it private\nBLACKTIP_TOKEN=" + e.Token + "\n"
+	return os.WriteFile(filepath.Join(s.dir, "self-edge.env"), []byte(body), 0o600)
+}
+
+// selfMapping is the dashboard's own route on the self edge: https on the configured name,
+// forwarded to Hammerhead's listen address through the edge's local site. It is added to that
+// edge's config and never stored, so it can't be edited or deleted by mistake in the UI.
+// Caller holds s.mu.
+func (s *Server) selfMapping(e *model.Edge) *model.Mapping {
+	o := s.opts
+	if o.SelfDomain == "" || e.ID != s.st.Settings.SelfEdgeID {
+		return nil
+	}
+	host, port, err := net.SplitHostPort(o.SelfTarget)
+	if err != nil {
+		return nil
+	}
+	n, _ := strconv.Atoi(port)
+	for _, st := range s.st.Sites {
+		if st.Local && st.EdgeID == e.ID {
+			return &model.Mapping{ID: selfMappingID, Name: "Hammerhead dashboard", Proto: model.ProtoHTTPS, EdgeID: e.ID,
+				BindIP: "0.0.0.0", Port: 443, Hostname: strings.ToLower(o.SelfDomain), Enabled: true, RedirectHTTP: true,
+				HealthSeconds: -1, Targets: []model.Target{{SiteID: st.ID, Host: host, Port: n}}}
+		}
+	}
+	return nil
+}

selfedge_test.go

@@ -0,0 +1,121 @@
+package hammerhead
+
+import (
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"git.veillax.com/Veillax/reef/model"
+	"git.veillax.com/Veillax/reef/secretbox"
+)
+
+func edgeConfigOf(t *testing.T, s *Server, token string) model.EdgeConfig {
+	t.Helper()
+	ts := httptest.NewServer(s.Handler())
+	defer ts.Close()
+	req, _ := http.NewRequest("GET", ts.URL+"/edge/config?v=0", nil)
+	req.Header.Set("Authorization", "Bearer "+token)
+	resp, err := http.DefaultClient.Do(req)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer resp.Body.Close()
+	var cfg model.EdgeConfig
+	if resp.StatusCode != 200 {
+		t.Fatalf("edge config: %s", resp.Status)
+	}
+	json.NewDecoder(resp.Body).Decode(&cfg)
+	return cfg
+}
+
+func tokenFile(t *testing.T, dir string) string {
+	b, err := os.ReadFile(filepath.Join(dir, "self-edge.env"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	return strings.TrimSpace(strings.SplitN(strings.SplitN(string(b), "BLACKTIP_TOKEN=", 2)[1], "\n", 2)[0])
+}
+
+var key = secretbox.GenerateKey()
+
+func TestSelfEdgeServesTheDashboardWithoutAProxy(t *testing.T) {
+	dir := t.TempDir()
+	opts := Options{MasterKey: key, SelfDomain: "Hammerhead.Example.com", SelfTarget: "127.0.0.1:8080"}
+	s, err := New(dir, opts)
+	if err != nil {
+		t.Fatal(err)
+	}
+	token := tokenFile(t, dir)
+	cfg := edgeConfigOf(t, s, token)
+	if len(cfg.Mappings) != 1 || len(cfg.Local) != 1 {
+		t.Fatalf("mappings=%d local=%d", len(cfg.Mappings), len(cfg.Local))
+	}
+	m := cfg.Mappings[0]
+	if m.Proto != model.ProtoHTTPS || m.Hostname != "hammerhead.example.com" || m.Port != 443 || !m.RedirectHTTP || !m.Enabled ||
+		len(m.Targets) != 1 || m.Targets[0].Host != "127.0.0.1" || m.Targets[0].Port != 8080 || m.Targets[0].SiteID != cfg.Local[0] {
+		t.Fatalf("unexpected dashboard route: %+v", m)
+	}
+	if m.Auth.Protected() {
+		t.Fatal("the dashboard has its own login and must not sit behind resource auth")
+	}
+	edgeID := s.st.Settings.SelfEdgeID
+	if info, _ := os.Stat(filepath.Join(dir, "self-edge.env")); info.Mode().Perm()&0o077 != 0 && os.PathSeparator == '/' {
+		t.Fatalf("token file is readable by others: %v", info.Mode())
+	}
+	s.Close()
+
+	// a restart keeps the same edge and token instead of making a new one
+	s2, err := New(dir, opts)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s2.Close()
+	if s2.st.Settings.SelfEdgeID != edgeID || tokenFile(t, dir) != token || len(s2.st.Edges) != 1 {
+		t.Fatal("restart changed the self edge")
+	}
+	if len(edgeConfigOf(t, s2, token).Mappings) != 1 {
+		t.Fatal("route missing after restart")
+	}
+}
+
+func TestNoSelfEdgeUnlessAsked(t *testing.T) {
+	dir := t.TempDir()
+	s, err := New(dir, Options{MasterKey: key})
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	if len(s.st.Edges) != 0 {
+		t.Fatal("an edge appeared without HAMMERHEAD_SELF_DOMAIN")
+	}
+	if _, err := os.Stat(filepath.Join(dir, "self-edge.env")); err == nil {
+		t.Fatal("token file written without HAMMERHEAD_SELF_DOMAIN")
+	}
+}
+
+func TestSavingSettingsKeepsTheSelfEdge(t *testing.T) {
+	dir := t.TempDir()
+	opts := Options{AdminEmail: "admin@test.local", AdminPassword: "password123", MasterKey: key, SelfDomain: "hh.example.com", SelfTarget: "127.0.0.1:8080"}
+	s, err := New(dir, opts)
+	if err != nil {
+		t.Fatal(err)
+	}
+	token := tokenFile(t, dir)
+	do := apiClient(t, s)
+	if code, b := do("PUT", "/api/admin/settings", map[string]any{"dashboardUrl": "https://hh.example.com", "acmeEmail": "me@example.com"}); code != 200 {
+		t.Fatalf("settings: %d %s", code, b)
+	}
+	s.Close()
+	s2, err := New(dir, opts)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s2.Close()
+	if len(s2.st.Edges) != 1 || tokenFile(t, dir) != token {
+		t.Fatalf("saving settings lost the self edge: %d edges", len(s2.st.Edges))
+	}
+}

server.go

@@ -21,11 +21,22 @@ func page(html []byte) http.HandlerFunc {
 	}
 }
 
+// dashboardPaths are the addresses the dashboard shows pages at. They all serve the same app,
+// which reads the path to decide what to show. Keep in step with TAB_PATH in web/index.html.
+var dashboardPaths = map[string]bool{
+	"/": true, "/apps": true, "/resources": true, "/sites": true, "/domains": true, "/mail-egress": true,
+	"/alerts": true, "/access": true, "/logs": true, "/account": true,
+	"/server/edges": true, "/server/organizations": true, "/server/users": true, "/server/sso": true,
+	"/server/settings": true, "/server/alerts": true, "/server/audit": true,
+}
+
+func isDashboardPath(p string) bool { return dashboardPaths[p] }
+
 func (s *Server) Handler() http.Handler {
 	mux := http.NewServeMux()
 	index := page(indexHTML)
 	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
-		if r.URL.Path != "/" {
+		if !isDashboardPath(r.URL.Path) {
 			http.NotFound(w, r)
 			return
 		}
@@ -68,13 +79,17 @@ func (s *Server) Handler() http.Handler {
 	mux.HandleFunc("GET /api/me/sessions", s.withUser(s.mySessions))
 	mux.HandleFunc("DELETE /api/me/sessions/{id}", s.withUser(s.endSession))
 	mux.HandleFunc("POST /api/me/sessions/revoke-others", s.withUser(s.endOtherSessions))
-	mux.HandleFunc("GET /api/edges", s.withUser(s.listEdges))
+	mux.HandleFunc("GET /api/edges", s.listEdges)
 
 	// server administrators
 	mux.HandleFunc("POST /api/edges", s.withServerAdmin(s.saveEdge))
 	mux.HandleFunc("PUT /api/edges/{id}", s.withServerAdmin(s.saveEdge))
 	mux.HandleFunc("DELETE /api/edges/{id}", s.withServerAdmin(s.deleteEdge))
+	mux.HandleFunc("GET /api/orgs", s.listOrgs)
 	mux.HandleFunc("POST /api/orgs", s.withServerAdmin(s.createOrg))
+	mux.HandleFunc("GET /api/admin/apikeys", s.withServerAdminUser(s.listServerKeys))
+	mux.HandleFunc("POST /api/admin/apikeys", s.withServerAdminUser(s.createServerKey))
+	mux.HandleFunc("DELETE /api/admin/apikeys/{id}", s.withServerAdminUser(s.deleteServerKey))
 	mux.HandleFunc("GET /api/admin/users", s.withServerAdmin(s.adminUsers))
 	mux.HandleFunc("POST /api/admin/users", s.withServerAdmin(s.adminCreateUser))
 	mux.HandleFunc("PUT /api/admin/users/{id}", s.withServerAdmin(s.adminUpdateUser))
@@ -86,6 +101,15 @@ func (s *Server) Handler() http.Handler {
 	mux.HandleFunc("GET /api/admin/settings", s.withServerAdmin(s.getSettings))
 	mux.HandleFunc("PUT /api/admin/settings", s.withServerAdmin(s.putSettings))
 	mux.HandleFunc("POST /api/admin/settings/test-mail", s.withServerAdmin(s.testMail))
+	mux.HandleFunc("POST /api/admin/settings/sandbar-sync", s.withServerAdmin(s.syncSandbarNow))
+	mux.HandleFunc("GET /api/admin/update", s.withServerAdmin(s.checkUpdate))
+	mux.HandleFunc("POST /api/admin/update/apply", s.withServerAdmin(s.applyUpdate))
+	mux.HandleFunc("GET /api/admin/update/status", s.withServerAdmin(s.updateStatus))
+	mux.HandleFunc("GET /api/versions", s.withUser(s.latestVersions))
+	mux.HandleFunc("GET /api/edges/{id}/update/log", s.withServerAdmin(s.edgeUpdateLog))
+	mux.HandleFunc("GET /api/orgs/{org}/sites/{id}/update/log", s.withOrg(true, s.siteUpdateLog))
+	mux.HandleFunc("POST /api/edges/{id}/update", s.withServerAdmin(s.updateEdge))
+	mux.HandleFunc("POST /api/orgs/{org}/sites/{id}/update", s.withOrg(true, s.updateSite))
 	mux.HandleFunc("GET /api/admin/audit", s.withServerAdmin(func(w http.ResponseWriter, r *http.Request, p *principal) {
 		writeJSON(w, 200, s.logs.queryAudit("", 300, 0))
 	}))
@@ -108,8 +132,10 @@ func (s *Server) Handler() http.Handler {
 	mux.HandleFunc("POST /api/orgs/{org}/apikeys", s.withOrg(true, s.createKey))
 	mux.HandleFunc("DELETE /api/orgs/{org}/apikeys/{id}", s.withOrg(true, s.deleteKey))
 	mux.HandleFunc("POST /api/orgs/{org}/sites", s.withOrg(true, s.addSite))
+	mux.HandleFunc("PUT /api/orgs/{org}/sites/{id}", s.withOrg(true, s.putSite))
 	mux.HandleFunc("DELETE /api/orgs/{org}/sites/{id}", s.withOrg(true, s.deleteSite))
 	mux.HandleFunc("GET /api/orgs/{org}/sites/{id}/containers", s.withOrg(true, s.siteContainers))
+	mux.HandleFunc("GET /api/orgs/{org}/sites/{id}/discovered", s.withOrg(true, s.siteDiscovered))
 	mux.HandleFunc("POST /api/orgs/{org}/domains", s.withOrg(true, s.addDomain))
 	mux.HandleFunc("DELETE /api/orgs/{org}/domains/{id}", s.withOrg(true, s.deleteDomain))
 	mux.HandleFunc("POST /api/orgs/{org}/domains/{id}/check", s.withOrg(true, s.checkDomain))
@@ -120,15 +146,24 @@ func (s *Server) Handler() http.Handler {
 	mux.HandleFunc("POST /api/orgs/{org}/mappings/{id}/shares", s.withOrg(true, s.createShare))
 	mux.HandleFunc("POST /api/orgs/{org}/mappings/{id}/revoke", s.withOrg(true, s.revokeVisitors))
 	mux.HandleFunc("DELETE /api/orgs/{org}/mappings/{id}/shares/{share}", s.withOrg(true, s.deleteShare))
+	mux.HandleFunc("POST /api/orgs/{org}/expose", s.withOrg(true, s.createExpose))
+	mux.HandleFunc("GET /api/orgs/{org}/expose", s.withOrg(true, s.listExpose))
+	mux.HandleFunc("GET /api/orgs/{org}/expose/{id}", s.withOrg(true, s.getExpose))
+	mux.HandleFunc("POST /api/orgs/{org}/expose/{id}/renew", s.withOrg(true, s.renewExpose))
+	mux.HandleFunc("DELETE /api/orgs/{org}/expose/{id}", s.withOrg(true, s.deleteExpose))
 	mux.HandleFunc("POST /api/orgs/{org}/mail-egress", s.withOrg(true, s.saveMailEgress))
 	mux.HandleFunc("PUT /api/orgs/{org}/mail-egress/{id}", s.withOrg(true, s.saveMailEgress))
 	mux.HandleFunc("DELETE /api/orgs/{org}/mail-egress/{id}", s.withOrg(true, s.deleteMailEgress))
 	mux.HandleFunc("GET /api/orgs/{org}/dkim", s.withOrg(true, s.listDKIM))
 	mux.HandleFunc("POST /api/orgs/{org}/dkim", s.withOrg(true, s.createDKIM))
 	mux.HandleFunc("DELETE /api/orgs/{org}/dkim/{id}", s.withOrg(true, s.deleteDKIM))
+	mux.HandleFunc("GET /api/orgs/{org}/dmarc", s.withOrg(true, s.listDMARC))
+	mux.HandleFunc("POST /api/orgs/{org}/dmarc", s.withOrg(true, s.uploadDMARC))
+	mux.HandleFunc("DELETE /api/orgs/{org}/dmarc/{domain}", s.withOrg(true, s.deleteDMARC))
 	mux.HandleFunc("GET /api/orgs/{org}/logs", s.withOrg(true, s.orgLogs))
 	mux.HandleFunc("GET /api/orgs/{org}/stats", s.withOrg(false, s.orgStats))
 	mux.HandleFunc("GET /api/orgs/{org}/traffic", s.withOrg(false, s.orgTraffic))
+	mux.HandleFunc("GET /api/orgs/{org}/traffic/{id}", s.withOrg(false, s.orgTrafficDetail))
 
 	// agents
 	mux.HandleFunc("GET /edge/config", s.edgeConfig)

settings.go

@@ -17,7 +17,8 @@ func (s *Server) getSettings(w http.ResponseWriter, r *http.Request, p *principa
 	defer s.mu.Unlock()
 	st := s.st.Settings
 	st.SMTP.Pass = ""                               // write-only
-	writeJSON(w, 200, map[string]any{"settings": st, "smtpPassSet": s.st.Settings.SMTP.Pass != ""})
+	st.Sandbar.Token, st.Sandbar.Monitors = "", nil // write-only
+	writeJSON(w, 200, map[string]any{"settings": st, "smtpPassSet": s.st.Settings.SMTP.Pass != "", "sandbarTokenSet": s.st.Settings.Sandbar.Token != "", "sandbarSync": s.sandbarStatus})
 }
 
 func (s *Server) putSettings(w http.ResponseWriter, r *http.Request, p *principal) {
@@ -38,9 +39,15 @@ func (s *Server) putSettings(w http.ResponseWriter, r *http.Request, p *principa
 	if in.SMTP.Pass == "" {
 		in.SMTP.Pass = s.st.Settings.SMTP.Pass // blank keeps the saved password
 	}
+	if err := s.cleanSandbar(&in.Sandbar); err != nil {
+		s.mu.Unlock()
+		fail(w, 400, err)
+		return
+	}
 	if s.opts.DashboardURL != "" {
 		in.DashboardURL = s.st.Settings.DashboardURL // pinned by flag/env
 	}
+	in.SelfEdgeID = s.st.Settings.SelfEdgeID // not editable here; losing it would make the next start create a second self edge
 	s.st.Settings = in
 	s.bump() // edges receive the dashboard URL and ACME settings
 	s.mu.Unlock()
@@ -68,7 +75,7 @@ func (s *Server) smtpSend(to, subject, body string) error {
 	c := s.st.Settings.SMTP
 	s.mu.Unlock()
 	if c.Host == "" {
-		return errors.New("email is not configured (Settings -> Email)")
+		return errors.New("Email is not set up yet. Set it up under Server, Settings.")
 	}
 	if c.Port == 0 {
 		c.Port = 587

store.go

@@ -16,8 +16,10 @@ import (
 	"sync"
 	"time"
 
-	"git.veillax.com/Veillax/hammerhead/internal/secretbox"
 	"git.veillax.com/Veillax/reef/model"
+	"git.veillax.com/Veillax/reef/secretbox"
+	"unicode"
+	"unicode/utf8"
 )
 
 type Org struct {
@@ -27,6 +29,10 @@ type Org struct {
 
 	AlertDebounceMin int `json:"alertDebounceMin,omitempty"` // minutes a problem must last before alerting; 0 = 2
 	AlertRepeatHours int `json:"alertRepeatHours,omitempty"` // remind about an unresolved problem this often; 0 = never
+
+	// Traffic alerts, applied to every port of the organization. 0 = off.
+	TrafficSpikeFactor int `json:"trafficSpikeFactor,omitempty"` // alert when a port carries this many times its usual traffic
+	TrafficDailyMB     int `json:"trafficDailyMb,omitempty"`     // alert when a port carries more than this many MB in one UTC day
 }
 
 type Role struct {
@@ -111,6 +117,9 @@ type Settings struct {
 	AcmeEmail    string      `json:"acmeEmail"`
 	AcmeStaging  bool        `json:"acmeStaging"`
 	SMTP         SMTP        `json:"smtp"`
+	Sandbar      SandbarSync `json:"sandbar"`
+	// SelfEdgeID is this server's own edge (see HAMMERHEAD_SELF_DOMAIN).
+	SelfEdgeID string `json:"selfEdgeId,omitempty"`
 }
 
 type Secret struct {
@@ -170,6 +179,22 @@ type Options struct {
 	AlertInterval    time.Duration // how often alerts are evaluated; 0 = 15s
 	AlertDebounce    time.Duration // overrides the per-organization delay before alerting (tests)
 	OfflineAfter     time.Duration // how long an edge or site may stay silent before it counts as down; 0 = 90s
+
+	// SelfDomain (HAMMERHEAD_SELF_DOMAIN) gives this server an edge of its own, with a built-in
+	// https route for the dashboard at that name, so no reverse proxy is needed in front of it.
+	// SelfTarget is the address Hammerhead listens on (e.g. 127.0.0.1:8080); SelfTunnel is the
+	// edge's public WireGuard endpoint (default <SelfDomain>:51820).
+	SelfDomain string
+	SelfTarget string
+	SelfTunnel string
+
+	// Version is the running release (e.g. v0.1.5), shown under Settings with a check for updates.
+	Version string
+	// AutoUpdate says whether this process updates itself (HAMMERHEAD_AUTO_UPDATE), for the same screen.
+	AutoUpdate bool
+	// UpdateServer and UpdateRepo say where releases are published; the defaults are this project's own.
+	UpdateServer string
+	UpdateRepo   string
 }
 
 type Server struct {
@@ -182,6 +207,7 @@ type Server struct {
 	mailStatus map[string]string                    // mail egress ID -> "" (listening) or an error
 	mailChecks map[string]model.MailCheck           // mail egress ID -> the edge's deliverability report
 	containers map[string][]model.Container         // by site ID
+	discovered map[string][]model.DiscoveredService // by site ID: suggested targets the agent found
 	changed    chan struct{}
 
 	tickets map[string]ticket
@@ -191,8 +217,18 @@ type Server struct {
 	oidc    map[string]*oidcDoc
 
 	alerts   map[string]*alertState // active problems by key (see alerts.go)
+	updRuns  map[string]*updRun     // progress of updates running on edges and sites, by ID
+	selfUpd  selfUpdate             // progress of this server's own update
+	verMu    sync.Mutex             // guards verCache and verAt, the latest releases seen by /api/versions
+	verCache map[string]string
+	verAt    time.Time
 	mtaSent  map[string]time.Time // reminder throttle for repeat notifications
 
+	expose map[string]time.Time // deadline of each ephemeral mapping, by ID (see expose.go)
+
+	sandbarMu     sync.Mutex        // one monitor sync at a time (see sandbar.go)
+	sandbarStatus sandbarSyncStatus // how the last one went
+
 	logs *logStore
 	db   *DB
 	stop chan struct{}
@@ -224,6 +260,7 @@ func New(dir string, opts Options) (*Server, error) {
 		mailStatus: map[string]string{},
 		mailChecks: map[string]model.MailCheck{},
 		containers: map[string][]model.Container{},
+		discovered: map[string][]model.DiscoveredService{},
 		changed:    make(chan struct{}),
 		tickets:    map[string]ticket{},
 		oauth:      map[string]oauthState{},
@@ -232,6 +269,8 @@ func New(dir string, opts Options) (*Server, error) {
 		oidc:       map[string]*oidcDoc{},
 		alerts:     map[string]*alertState{},
 		mtaSent:    map[string]time.Time{},
+		updRuns:    map[string]*updRun{},
+		expose:     map[string]time.Time{},
 	}
 	s.st.Sessions = map[string]*Session{}
 	s.st.Secrets = map[string]*Secret{}
@@ -258,13 +297,23 @@ func New(dir string, opts Options) (*Server, error) {
 			return nil, err
 		}
 	}
+	if opts.SelfDomain != "" && opts.SelfTunnel == "" {
+		s.opts.SelfTunnel = opts.SelfDomain + ":51820"
+	}
+	s.finishSelfUpdateLog()
+	if err := s.ensureSelfEdge(); err != nil {
+		db.Close()
+		return nil, err
+	}
 	if err := s.save(); err != nil {
 		db.Close()
 		return nil, err
 	}
 	s.loadAlerts()
 	go s.housekeeping()
+	go s.exposeLoop()
 	go s.alertLoop()
+	go s.sandbarLoop()
 	return s, nil
 }
 
@@ -401,8 +450,13 @@ func writeJSON(w http.ResponseWriter, code int, v any) {
 	json.NewEncoder(w).Encode(v)
 }
 
+// fail sends an error to the browser. Messages are shown to people, so they start with a capital letter.
 func fail(w http.ResponseWriter, code int, err error) {
-	writeJSON(w, code, map[string]string{"error": err.Error()})
+	msg := err.Error()
+	if r, n := utf8.DecodeRuneInString(msg); n > 0 {
+		msg = string(unicode.ToUpper(r)) + msg[n:]
+	}
+	writeJSON(w, code, map[string]string{"error": msg})
 }
 
 func failf(w http.ResponseWriter, code int, format string, a ...any) {

tags_test.go

@@ -0,0 +1,122 @@
+package hammerhead
+
+import (
+	"encoding/json"
+	"reflect"
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestNormalizeTags(t *testing.T) {
+	got, err := normalizeTags([]string{" Minecraft ", "minecraft", "", "prod  east", "PROD east"})
+	if err != nil || !reflect.DeepEqual(got, []string{"Minecraft", "prod east"}) {
+		t.Fatalf("got %v, %v", got, err)
+	}
+	if got, err := normalizeTags(nil); err != nil || got != nil {
+		t.Fatalf("nil in should give nil out: %v, %v", got, err)
+	}
+	tooMany := make([]string, 11)
+	for i := range tooMany {
+		tooMany[i] = "t" + strings.Repeat("x", i)
+	}
+	for _, bad := range [][]string{{strings.Repeat("a", 33)}, {"a,b"}, {"<b>"}, tooMany} {
+		if _, err := normalizeTags(bad); err == nil {
+			t.Errorf("accepted %v", bad)
+		}
+	}
+}
+
+func TestSiteTagsAndRenameThroughTheAPI(t *testing.T) {
+	s, err := New(t.TempDir(), Options{AdminEmail: "admin@test.local", AdminPassword: "password123", AlertInterval: time.Hour})
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	do := apiClient(t, s)
+	_, b := do("GET", "/api/me", nil)
+	var me struct{ Orgs []struct{ ID string } }
+	json.Unmarshal(b, &me)
+	org := me.Orgs[0].ID
+
+	code, raw := do("POST", "/api/orgs/"+org+"/sites", map[string]any{"name": "Home"})
+	if code != 200 {
+		t.Fatalf("add site: %d %s", code, raw)
+	}
+	var site struct{ ID string }
+	json.Unmarshal(raw, &site)
+
+	if code, raw := do("PUT", "/api/orgs/"+org+"/sites/"+site.ID, map[string]any{"name": " Home lab ", "tags": []string{"Lab", "lab", "nas"}}); code != 200 {
+		t.Fatalf("edit site: %d %s", code, raw)
+	}
+	_, raw = do("GET", "/api/orgs/"+org+"/state", nil)
+	var st struct {
+		Sites []struct {
+			ID, Name string
+			Tags     []string
+			Local    bool
+		}
+	}
+	json.Unmarshal(raw, &st)
+	found := false
+	for _, x := range st.Sites {
+		if x.ID == site.ID {
+			found = true
+			if x.Name != "Home lab" || !reflect.DeepEqual(x.Tags, []string{"Lab", "nas"}) {
+				t.Fatalf("site after edit: %+v", x)
+			}
+		}
+	}
+	if !found {
+		t.Fatalf("site missing from state: %s", raw)
+	}
+
+	for name, body := range map[string]map[string]any{"blank name": {"name": " "}, "bad tag": {"name": "x", "tags": []string{"a,b"}}} {
+		if code, _ := do("PUT", "/api/orgs/"+org+"/sites/"+site.ID, body); code != 400 {
+			t.Errorf("%s: expected 400, got %d", name, code)
+		}
+	}
+	if code, _ := do("PUT", "/api/orgs/"+org+"/sites/nope", map[string]any{"name": "x"}); code != 404 {
+		t.Errorf("unknown site: expected 404, got %d", code)
+	}
+}
+
+func TestSandbarKeyIsNeverSentBackAndBlankKeepsIt(t *testing.T) {
+	s, err := New(t.TempDir(), Options{AdminEmail: "admin@test.local", AdminPassword: "password123", AlertInterval: time.Hour})
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	do := apiClient(t, s)
+	_, b := do("GET", "/api/me", nil)
+	var me struct{ Orgs []struct{ ID string } }
+	json.Unmarshal(b, &me)
+	org := me.Orgs[0].ID
+
+	code, raw := do("POST", "/api/orgs/"+org+"/alerts/channels", map[string]any{"name": "pager", "kind": "sandbar", "target": "https://sandbar.example.com", "token": "sb_topsecret", "policy": "p1", "enabled": true})
+	if code != 200 {
+		t.Fatalf("create channel: %d %s", code, raw)
+	}
+	_, raw = do("GET", "/api/orgs/"+org+"/alerts", nil)
+	if strings.Contains(string(raw), "sb_topsecret") || strings.Contains(string(raw), "sandbar.example.com/api") {
+		t.Fatalf("the overview leaks Sandbar details: %s", raw)
+	}
+	var ov struct {
+		Channels []struct{ ID, Kind, Policy string }
+	}
+	json.Unmarshal(raw, &ov)
+	if len(ov.Channels) != 1 || ov.Channels[0].Kind != "sandbar" || ov.Channels[0].Policy != "p1" {
+		t.Fatalf("overview: %s", raw)
+	}
+	// editing without re-entering the address or key keeps both
+	if code, raw := do("PUT", "/api/orgs/"+org+"/alerts/channels/"+ov.Channels[0].ID, map[string]any{"name": "pager 2", "kind": "sandbar", "policy": "p2", "enabled": true}); code != 200 {
+		t.Fatalf("edit channel: %d %s", code, raw)
+	}
+	s.mu.Lock()
+	ch := s.st.Channels[0]
+	kept := ch.Token == "sb_topsecret" && ch.Target == "https://sandbar.example.com" && ch.Policy == "p2"
+	s.mu.Unlock()
+	if !kept {
+		t.Fatalf("an edit lost the saved key or address: %+v", ch)
+	}
+}

traffic.go

@@ -0,0 +1,161 @@
+package hammerhead
+
+import (
+	"errors"
+	"fmt"
+	"net/http"
+	"time"
+)
+
+// Traffic analytics: edges report per-port bytes, connections and their busiest client addresses with
+// every report. Hammerhead keeps them at three resolutions (see the *Keep constants in db.go) and draws
+// charts from them. Organizations can also be alerted when a port suddenly carries far more traffic
+// than usual, or more than a daily amount.
+
+// Spikes are judged on the last few minutes against the hours before them. A port has to have an
+// hour of history before it can spike, and the burst has to be worth noticing in absolute terms, so
+// a quiet port going from 1 KB to 50 KB doesn't page anyone.
+const (
+	spikeMinHistory = 60               // minutes of baseline needed before a spike can be called
+	spikeMinBytes   = 10 * 1024 * 1024 // bytes in the window below which nothing counts as a spike
+)
+
+// spiking says whether recent traffic is a spike over the baseline for a port.
+func spiking(recent int64, baseline float64, history int64, factor int) bool {
+	if factor <= 0 || history < spikeMinHistory || recent < spikeMinBytes {
+		return false
+	}
+	return float64(recent) > baseline*float64(factor)
+}
+
+// trafficProblemsLocked adds an alert problem for each port of an organization that is spiking or has
+// gone over its daily amount. Caller holds s.mu.
+func (s *Server) trafficProblemsLocked(now time.Time, edgeDown map[string]bool, add func(key, org, kind, name, msg string)) {
+	dayStart := now.UTC().Truncate(24*time.Hour).Unix() / 3600
+	for _, m := range s.st.Mappings {
+		if !m.Enabled || edgeDown[m.EdgeID] {
+			continue
+		}
+		o := s.findOrg(m.OrgID)
+		if o == nil || (o.TrafficSpikeFactor <= 0 && o.TrafficDailyMB <= 0) {
+			continue
+		}
+		if o.TrafficSpikeFactor > 0 {
+			recent, base, history := s.db.recentTraffic(m.ID, now)
+			if spiking(recent, base, history, o.TrafficSpikeFactor) {
+				add("spike:"+m.ID, m.OrgID, "traffic", m.Name, fmt.Sprintf("%s: %s in the last %d minutes, about %.0f times its usual %s",
+					m.Name, byteText(recent), spikeWindow, float64(recent)/maxFloat(base, 1), byteText(int64(base))))
+			}
+		}
+		if limit := int64(o.TrafficDailyMB) << 20; limit > 0 {
+			if used := s.db.bytesSince(m.ID, dayStart); used > limit {
+				add("quota:"+m.ID, m.OrgID, "traffic", m.Name, fmt.Sprintf("%s has carried %s today, over the %s a day you set", m.Name, byteText(used), byteText(limit)))
+			}
+		}
+	}
+}
+
+func maxFloat(a, b float64) float64 {
+	if a > b {
+		return a
+	}
+	return b
+}
+
+func byteText(n int64) string {
+	f, units := float64(n), []string{"B", "KB", "MB", "GB", "TB"}
+	i := 0
+	for f >= 1024 && i < len(units)-1 {
+		f /= 1024
+		i++
+	}
+	if i == 0 {
+		return fmt.Sprintf("%d B", n)
+	}
+	return fmt.Sprintf("%.1f %s", f, units[i])
+}
+
+// trafficRanges maps the ranges the dashboard offers to a bucket size, a bucket count and how many hours of top sources to include.
+var trafficRanges = map[string]struct {
+	unit   string
+	count  int
+	hours  int
+	bucket time.Duration
+}{
+	"1h":  {"minute", 60, 1, time.Minute},
+	"6h":  {"minute", 360, 6, time.Minute},
+	"24h": {"hour", 24, 24, time.Hour},
+	"7d":  {"hour", 7 * 24, 7 * 24, time.Hour},
+	"30d": {"hour", 30 * 24, 7 * 24, time.Hour}, // client addresses are only kept for a week
+}
+
+// orgTrafficDetail serves one port's traffic chart and busiest clients, or the organization's total
+// for id "all". ?range= is one of 1h, 6h, 24h (default), 7d or 30d.
+func (s *Server) orgTrafficDetail(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	rng := r.URL.Query().Get("range")
+	if rng == "" {
+		rng = "24h"
+	}
+	spec, found := trafficRanges[rng]
+	if !found {
+		fail(w, 400, errors.New("range must be 1h, 6h, 24h, 7d or 30d"))
+		return
+	}
+	id := r.PathValue("id")
+	s.mu.Lock()
+	var ids []string
+	name := "All resources"
+	for _, m := range s.st.Mappings {
+		if m.OrgID != c.Org.ID {
+			continue
+		}
+		if id == "all" {
+			ids = append(ids, m.ID)
+		} else if m.ID == id {
+			ids, name = []string{m.ID}, m.Name
+		}
+	}
+	s.mu.Unlock()
+	if id != "all" && len(ids) == 0 {
+		fail(w, 404, errors.New("no such resource"))
+		return
+	}
+	now := time.Now()
+	points := s.db.trafficSeries(ids, spec.unit, spec.count, now)
+	var tot TrafficPoint
+	var peak int64
+	for _, p := range points {
+		tot.Conns += p.Conns
+		tot.BytesIn += p.BytesIn
+		tot.BytesOut += p.BytesOut
+		if b := p.BytesIn + p.BytesOut; b > peak {
+			peak = b
+		}
+	}
+	writeJSON(w, 200, map[string]any{
+		"name": name, "range": rng, "bucketSeconds": int(spec.bucket / time.Second), "points": points,
+		"total": tot, "peakBucket": peak, "sources": s.db.topSources(ids, spec.hours, 10, now), "sourcesHours": spec.hours,
+	})
+}
+
+// trafficAlertSettings sets when an organization is told about unusual traffic.
+func (s *Server) trafficAlertSettings(w http.ResponseWriter, r *http.Request, sc alertScope) {
+	var in struct{ SpikeFactor, DailyMB int }
+	if !decode(w, r, &in) {
+		return
+	}
+	if (in.SpikeFactor != 0 && (in.SpikeFactor < 2 || in.SpikeFactor > 1000)) || in.DailyMB < 0 || in.DailyMB > 100_000_000 {
+		fail(w, 400, errors.New("A spike is 2 to 1000 times the usual traffic (0 turns it off), and the daily amount can't be negative (0 turns it off)."))
+		return
+	}
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	o := s.findOrg(sc.org)
+	if o == nil {
+		fail(w, 404, errors.New("organization settings only"))
+		return
+	}
+	o.TrafficSpikeFactor, o.TrafficDailyMB = in.SpikeFactor, in.DailyMB
+	s.save()
+	ok(w)
+}

traffic_test.go

@@ -0,0 +1,159 @@
+package hammerhead
+
+import (
+	"strings"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/reef/model"
+)
+
+func TestSpiking(t *testing.T) {
+	const mb = 1 << 20
+	cases := []struct {
+		name    string
+		recent  int64
+		base    float64
+		history int64
+		factor  int
+		want    bool
+	}{
+		{"ten times usual", 200 * mb, 10 * mb, 180, 5, true},
+		{"just under the factor", 49 * mb, 10 * mb, 180, 5, false},
+		{"off", 500 * mb, 10 * mb, 180, 0, false},
+		{"too little history", 500 * mb, 10 * mb, 30, 5, false},
+		{"tiny in absolute terms", 5 * mb, 0.01 * mb, 180, 5, false},
+		{"from nothing to a lot", 50 * mb, 0, 180, 5, true},
+	}
+	for _, c := range cases {
+		if got := spiking(c.recent, c.base, c.history, c.factor); got != c.want {
+			t.Errorf("%s: spiking = %v, want %v", c.name, got, c.want)
+		}
+	}
+}
+
+func TestTrafficStorage(t *testing.T) {
+	s := newFx(t).s
+	now := time.Now()
+	st := func(in, out int64, src ...model.SourceStat) model.Stat {
+		return model.Stat{Conns: 1, BytesIn: in, BytesOut: out, Sources: src}
+	}
+	s.db.addTrafficAt("m0", st(100, 900, model.SourceStat{IP: "203.0.113.1", Conns: 1, BytesIn: 100, BytesOut: 900}), now)
+	s.db.addTrafficAt("m0", st(10, 10, model.SourceStat{IP: "203.0.113.1", Conns: 1, BytesIn: 10, BytesOut: 10}, model.SourceStat{IP: "203.0.113.2", Conns: 1, BytesIn: 1}), now)
+	s.db.addTrafficAt("m0", st(1, 1), now.Add(-3*time.Minute))
+	s.db.addTrafficAt("m1", st(7, 7), now)
+
+	pts := s.db.trafficSeries([]string{"m0"}, "minute", 60, now)
+	if len(pts) != 60 || pts[59].BytesIn != 110 || pts[59].BytesOut != 910 || pts[59].Conns != 2 || pts[56].BytesIn != 1 {
+		t.Fatalf("minute series: last=%+v 3 back=%+v", pts[59], pts[56])
+	}
+	if pts[59].Time != now.Unix()/60*60 || pts[58].Time != pts[59].Time-60 {
+		t.Errorf("bucket times: %d %d", pts[58].Time, pts[59].Time)
+	}
+	all := s.db.trafficSeries([]string{"m0", "m1"}, "hour", 24, now)
+	if all[23].BytesIn+all[22].BytesIn < 118 || len(all) != 24 {
+		t.Errorf("hour series over two ports: %+v", all[22:])
+	}
+	if got := s.db.trafficSeries(nil, "hour", 3, now); len(got) != 3 || got[2].BytesIn != 0 {
+		t.Errorf("no ports: %+v", got)
+	}
+
+	top := s.db.topSources([]string{"m0"}, 24, 10, now)
+	if len(top) != 2 || top[0].IP != "203.0.113.1" || top[0].BytesOut != 910 || top[0].Conns != 2 {
+		t.Fatalf("top sources: %+v", top)
+	}
+	if got := s.db.topSources([]string{"m0"}, 24, 1, now); len(got) != 1 {
+		t.Errorf("limit: %+v", got)
+	}
+	if got := s.db.topSources([]string{"m1"}, 24, 10, now); len(got) != 0 {
+		t.Errorf("another port's sources leaked: %+v", got)
+	}
+}
+
+func TestTrafficRetention(t *testing.T) {
+	s := newFx(t).s
+	now := time.Now()
+	old := model.Stat{Conns: 1, BytesIn: 5, Sources: []model.SourceStat{{IP: "203.0.113.9", Conns: 1, BytesIn: 5}}}
+	s.db.addTrafficAt("m0", old, now.Add(-trafficMinutelyKeep-time.Hour)) // too old for minutes, fine for hours and sources
+	s.db.addTrafficAt("m0", old, now.Add(-trafficSourcesKeep-time.Hour))  // too old for sources too
+	s.db.addTrafficAt("m0", old, now.Add(-trafficHourlyKeep-time.Hour))   // gone everywhere
+	s.db.addTrafficAt("m0", old, now)
+	s.db.purge(30, 365)
+	count := func(table string) (n int) {
+		s.db.sql.QueryRow(`SELECT COUNT(*) FROM ` + table).Scan(&n)
+		return
+	}
+	if n := count("traffic_minutely"); n != 1 {
+		t.Errorf("minutely rows after purge: %d, want 1", n)
+	}
+	if n := count("traffic_sources"); n != 2 {
+		t.Errorf("source rows after purge: %d, want 2", n)
+	}
+	if n := count("traffic_hourly"); n != 3 {
+		t.Errorf("hourly rows after purge: %d, want 3", n)
+	}
+}
+
+func TestRecentTraffic(t *testing.T) {
+	s := newFx(t).s
+	now := time.Now()
+	const mb = 1 << 20
+	for h := 3 * 60; h > spikeWindow; h-- { // three hours of 1 MB a minute
+		s.db.addTrafficAt("m0", model.Stat{BytesIn: mb}, now.Add(-time.Duration(h)*time.Minute))
+	}
+	s.db.addTrafficAt("m0", model.Stat{BytesIn: 100 * mb}, now)
+	recent, base, hist := s.db.recentTraffic("m0", now)
+	if recent != 100*mb || hist < 170 || base < 4.9*mb || base > 5.1*mb { // 1 MB a minute is 5 MB per window
+		t.Fatalf("recent=%d base=%.0f history=%d", recent, base, hist)
+	}
+	if !spiking(recent, base, hist, 5) {
+		t.Error("100 MB over a 5 MB norm should spike")
+	}
+	if r, b, h := s.db.recentTraffic("nothing", now); r != 0 || b != 0 || h != 0 {
+		t.Errorf("unknown port: %d %f %d", r, b, h)
+	}
+}
+
+func TestTrafficAlertsFireAndClear(t *testing.T) {
+	f := newFx(t)
+	s := f.s
+	const mb = 1 << 20
+	now := time.Now()
+	s.mu.Lock()
+	m := s.st.Mappings[0]
+	o := s.findOrg(m.OrgID)
+	o.TrafficSpikeFactor, o.TrafficDailyMB = 4, 500
+	id := m.ID
+	s.mu.Unlock()
+	for min := 3 * 60; min > spikeWindow; min-- {
+		s.db.addTrafficAt(id, model.Stat{BytesIn: mb}, now.Add(-time.Duration(min)*time.Minute))
+	}
+
+	problems := func() map[string]problem {
+		s.mu.Lock()
+		defer s.mu.Unlock()
+		return s.problemsLocked(now)
+	}
+	if p := problems(); p["spike:"+id].msg != "" || p["quota:"+id].msg != "" {
+		t.Fatalf("steady traffic raised a problem: %+v", p)
+	}
+	s.db.addTrafficAt(id, model.Stat{BytesOut: 300 * mb}, now)
+	p := problems()
+	if !strings.Contains(p["spike:"+id].msg, "usual") || p["spike:"+id].kind != "traffic" {
+		t.Fatalf("no spike problem: %+v", p)
+	}
+	if p["quota:"+id].msg != "" {
+		t.Errorf("under the daily amount, yet: %+v", p["quota:"+id])
+	}
+	s.db.addTrafficAt(id, model.Stat{BytesOut: 300 * mb}, now)
+	if q := problems()["quota:"+id]; !strings.Contains(q.msg, "over the") {
+		t.Fatalf("no quota problem: %+v", q)
+	}
+
+	s.mu.Lock()
+	o.TrafficSpikeFactor, o.TrafficDailyMB = 0, 0
+	s.mu.Unlock()
+	if p := problems(); p["spike:"+id].msg != "" || p["quota:"+id].msg != "" {
+		t.Errorf("alerts off, yet: %+v", p)
+	}
+}

update.go

@@ -0,0 +1,183 @@
+package hammerhead
+
+import (
+	"context"
+	"errors"
+	"net/http"
+	"os"
+	"strings"
+	"sync"
+	"time"
+
+	"git.veillax.com/Veillax/reef/selfupdate"
+)
+
+func (s *Server) updateConfig(app, current string) selfupdate.Config {
+	o := s.opts
+	server := o.UpdateServer
+	if server == "" {
+		server = "https://git.veillax.com"
+	}
+	owner := "Veillax"
+	if o.UpdateRepo != "" {
+		owner = strings.SplitN(o.UpdateRepo, "/", 2)[0]
+	}
+	return selfupdate.Config{Server: server, Repo: owner + "/" + app, App: app, Current: current}
+}
+
+// checkUpdate asks the release server whether a newer Hammerhead exists. It only looks.
+func (s *Server) checkUpdate(w http.ResponseWriter, r *http.Request, p *principal) {
+	o := s.opts
+	ctx, cancel := context.WithTimeout(r.Context(), 20*time.Second)
+	defer cancel()
+	out := map[string]any{"current": o.Version, "autoUpdate": o.AutoUpdate, "checked": time.Now()}
+	latest, newer, err := selfupdate.Check(ctx, s.updateConfig("hammerhead", o.Version))
+	out["latest"], out["newer"] = latest, newer
+	if err != nil {
+		out["error"] = err.Error()
+	}
+	writeJSON(w, 200, out)
+}
+
+// applyUpdate installs the newest release of Hammerhead itself and restarts into it. Like the
+// scheduled check it needs auto-update to be on, which is also what makes the binary replaceable.
+func (s *Server) applyUpdate(w http.ResponseWriter, r *http.Request, p *principal) {
+	if !s.opts.AutoUpdate {
+		fail(w, 400, errors.New("auto-update is off: set HAMMERHEAD_AUTO_UPDATE=1 and restart Hammerhead, or update it by hand"))
+		return
+	}
+	u := &s.selfUpd
+	u.mu.Lock()
+	if u.running {
+		u.mu.Unlock()
+		fail(w, 409, errors.New("an update is already running"))
+		return
+	}
+	u.running, u.stage, u.pct, u.lines = true, "starting", 0, nil
+	u.mu.Unlock()
+	os.WriteFile(s.updateLogPath(), nil, 0o600) // a new run starts a new log
+	cfg := s.updateConfig("hammerhead", s.opts.Version)
+	cfg.Progress = func(e selfupdate.Event) { s.selfNote(e.Stage, e.Pct, e.Line) }
+	cfg.Restart = func(exe string) error {
+		s.selfNote("restarting", 100, "Closing the database and restarting")
+		s.Close()
+		return selfupdate.Restart(exe)
+	}
+	s.selfNote("starting", 0, "Update requested by "+p.User.Email)
+	s.audit("", p.User.Email, "server.update", "")
+	go func() {
+		ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
+		defer cancel()
+		selfupdate.CheckOnce(ctx, cfg) // on success the process is replaced and this never returns
+		u.mu.Lock()
+		u.running = false
+		u.mu.Unlock()
+	}()
+	writeJSON(w, 200, map[string]any{"started": true})
+}
+
+// latestVersions returns the newest release of each program, for "update available" badges. Results
+// are cached for ten minutes so a busy dashboard doesn't hammer the release server.
+func (s *Server) latestVersions(w http.ResponseWriter, r *http.Request, p *principal) {
+	s.verMu.Lock()
+	if time.Since(s.verAt) > 10*time.Minute || s.verCache == nil {
+		ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
+		out := map[string]string{}
+		var mu sync.Mutex
+		var wg sync.WaitGroup
+		for _, app := range []string{"hammerhead", "blacktip", "bamboo"} {
+			wg.Add(1)
+			go func(app string) {
+				defer wg.Done()
+				if tag, _, err := selfupdate.Check(ctx, s.updateConfig(app, "")); err == nil && tag != "" {
+					mu.Lock()
+					out[app] = tag
+					mu.Unlock()
+				}
+			}(app)
+		}
+		wg.Wait()
+		cancel()
+		s.verCache, s.verAt = out, time.Now()
+	}
+	out := s.verCache
+	s.verMu.Unlock()
+	writeJSON(w, 200, out)
+}
+
+// noteUpdateReport records what an edge or site says about its version and any update that was
+// requested of it. It reports whether a pending request was answered, which means the config the
+// agent polls has changed (it no longer carries the request).
+func noteUpdateReport(version *string, auto *bool, updateID, updateMsg *string, inVersion string, inAuto bool, inID, inMsg string) bool {
+	upgraded := *version != "" && inVersion != "" && inVersion != *version
+	if inVersion != "" {
+		*version = inVersion
+	}
+	*auto = inAuto
+	if *updateID == "" {
+		return false
+	}
+	switch {
+	case inID == *updateID && inMsg != "":
+		*updateMsg = inMsg
+	case upgraded:
+		*updateMsg = "updated to " + inVersion
+	default:
+		return false
+	}
+	*updateID = ""
+	return true
+}
+
+func (s *Server) updateEdge(w http.ResponseWriter, r *http.Request, p *principal) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	e := s.findEdge(r.PathValue("id"))
+	if e == nil {
+		fail(w, 404, errors.New("no such edge"))
+		return
+	}
+	if err := requestUpdate(&e.AutoUpdate, &e.UpdateID, &e.UpdateMsg, s.edgeOnline(e), "BLACKTIP"); err != nil {
+		fail(w, 400, err)
+		return
+	}
+	delete(s.updRuns, e.ID) // a new request starts a new log
+	s.bump()
+	s.audit("", p.User.Email, "edge.update", e.Name)
+	ok(w)
+}
+
+func (s *Server) updateSite(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	st := s.findSite(r.PathValue("id"))
+	if st == nil || !s.siteVisible(c, st) {
+		fail(w, 404, errors.New("no such site"))
+		return
+	}
+	if st.Local {
+		fail(w, 400, errors.New("this site is the edge's own machine; update the edge instead"))
+		return
+	}
+	if err := requestUpdate(&st.AutoUpdate, &st.UpdateID, &st.UpdateMsg, s.siteOnline(st), "BAMBOO"); err != nil {
+		fail(w, 400, err)
+		return
+	}
+	delete(s.updRuns, st.ID)
+	s.bump()
+	s.audit(c.Org.ID, c.Actor, "site.update", st.Name)
+	ok(w)
+}
+
+// requestUpdate marks an update as wanted. The agent installs it only if it has auto-update on, and
+// the request stays until the agent answers (or reports a new version). Asking again replaces it.
+func requestUpdate(auto *bool, updateID, updateMsg *string, online bool, prefix string) error {
+	if !online {
+		return errors.New("it is offline right now")
+	}
+	if !*auto {
+		return errors.New("it isn't running with auto-update on, so it won't install updates from here (set " + prefix + "_AUTO_UPDATE=1 and restart it)")
+	}
+	*updateID, *updateMsg = randHex(8), ""
+	return nil
+}

update_test.go

@@ -0,0 +1,90 @@
+package hammerhead
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func TestCheckForUpdates(t *testing.T) {
+	gitea := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path != "/api/v1/repos/Veillax/hammerhead/releases/latest" {
+			http.NotFound(w, r)
+			return
+		}
+		fmt.Fprint(w, `{"tag_name":"v0.3.0","prerelease":false,"assets":[]}`)
+	}))
+	defer gitea.Close()
+	for _, c := range []struct {
+		version string
+		newer   bool
+	}{{"v0.2.0", true}, {"v0.3.0", false}, {"dev", false}} {
+		s, err := New(t.TempDir(), Options{AdminEmail: "admin@test.local", AdminPassword: "password123", MasterKey: key,
+			Version: c.version, UpdateServer: gitea.URL, UpdateRepo: "Veillax/hammerhead", AutoUpdate: true})
+		if err != nil {
+			t.Fatal(err)
+		}
+		do := apiClient(t, s)
+		code, b := do("GET", "/api/admin/update", nil)
+		var out struct {
+			Current, Latest, Error string
+			Newer, AutoUpdate      bool
+		}
+		json.Unmarshal(b, &out)
+		if code != 200 || out.Current != c.version || out.Latest != "v0.3.0" || out.Newer != c.newer || !out.AutoUpdate || out.Error != "" {
+			t.Errorf("%s: %d %s", c.version, code, b)
+		}
+		s.Close()
+	}
+
+	// an unreachable release server is reported, not fatal
+	s, err := New(t.TempDir(), Options{AdminEmail: "admin@test.local", AdminPassword: "password123", MasterKey: key, Version: "v0.1.0", UpdateServer: "http://127.0.0.1:1"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	code, b := apiClient(t, s)("GET", "/api/admin/update", nil)
+	var out struct{ Error string }
+	json.Unmarshal(b, &out)
+	if code != 200 || out.Error == "" {
+		t.Errorf("unreachable server: %d %s", code, b)
+	}
+}
+
+func TestSelfUpdateLogSurvivesTheRestartItEndsWith(t *testing.T) {
+	dir := t.TempDir()
+	if err := os.WriteFile(filepath.Join(dir, "update.log"), []byte("12:00:00 Checking\n12:00:09 Installed v0.3.0; restarting into it\n"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	s, err := New(dir, Options{AdminEmail: "admin@test.local", AdminPassword: "password123", MasterKey: key, Version: "v0.3.0"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	do := apiClient(t, s)
+	code, b := do("GET", "/api/admin/update/status", nil)
+	var out struct {
+		Running bool
+		Stage   string
+		Pct     int
+		Lines   []string
+		Version string
+	}
+	json.Unmarshal(b, &out)
+	last := ""
+	if len(out.Lines) > 0 {
+		last = out.Lines[len(out.Lines)-1]
+	}
+	if code != 200 || out.Running || out.Stage != "done" || out.Pct != 100 || len(out.Lines) != 3 || !strings.Contains(last, "now running v0.3.0") {
+		t.Fatalf("status: %d %s", code, b)
+	}
+	// starting an update needs auto-update on
+	if code, b := do("POST", "/api/admin/update/apply", nil); code != 400 || !strings.Contains(string(b), "HAMMERHEAD_AUTO_UPDATE") {
+		t.Fatalf("apply without auto-update: %d %s", code, b)
+	}
+}

updatelog.go

@@ -0,0 +1,152 @@
+package hammerhead
+
+import (
+	"errors"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"time"
+)
+
+const maxUpdateLines = 200
+
+// updRun is the progress of one update an edge or site is carrying out, as that agent reported it.
+type updRun struct {
+	ID    string // the update request it belongs to
+	Stage string
+	Pct   int
+	Lines []string
+}
+
+// recordUpdateProgress keeps what an agent says about the update it is running. When the agent
+// reports a new version, the update is finished: its log gets a closing line and is then left alone,
+// so the freshly restarted program (which starts a new, empty log) can't overwrite it.
+// Caller holds s.mu.
+func (s *Server) recordUpdateProgress(agentID, reqID, stage string, pct int, lines []string, oldVersion, newVersion string) {
+	run := s.updRuns[agentID]
+	if run != nil && run.Stage != "done" && oldVersion != "" && newVersion != "" && newVersion != oldVersion {
+		run.Lines = append(run.Lines, time.Now().Format("15:04:05")+" Now running "+newVersion)
+		run.Stage, run.Pct = "done", 100
+		return
+	}
+	if reqID == "" || (stage == "" && len(lines) == 0) {
+		return
+	}
+	if run != nil && run.ID == reqID && run.Stage == "done" {
+		return
+	}
+	s.updRuns[agentID] = &updRun{ID: reqID, Stage: stage, Pct: pct, Lines: lines}
+}
+
+func (s *Server) updateLogView(agentID string, pending bool, msg, version string) map[string]any {
+	out := map[string]any{"running": pending, "msg": msg, "version": version, "stage": "", "pct": 0, "lines": []string{}}
+	if run := s.updRuns[agentID]; run != nil {
+		out["stage"], out["pct"], out["lines"] = run.Stage, run.Pct, run.Lines
+	}
+	return out
+}
+
+func (s *Server) edgeUpdateLog(w http.ResponseWriter, r *http.Request, p *principal) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	e := s.findEdge(r.PathValue("id"))
+	if e == nil {
+		fail(w, 404, errors.New("no such edge"))
+		return
+	}
+	writeJSON(w, 200, s.updateLogView(e.ID, e.UpdateID != "", e.UpdateMsg, e.Version))
+}
+
+func (s *Server) siteUpdateLog(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	st := s.findSite(r.PathValue("id"))
+	if st == nil || !s.siteVisible(c, st) {
+		fail(w, 404, errors.New("no such site"))
+		return
+	}
+	writeJSON(w, 200, s.updateLogView(st.ID, st.UpdateID != "", st.UpdateMsg, st.Version))
+}
+
+// selfUpdate is the progress of this server's own update. It is kept in memory and in <data>/update.log,
+// so the page can still show it after the restart that ends a successful update.
+type selfUpdate struct {
+	mu      sync.Mutex
+	running bool
+	stage   string
+	pct     int
+	lines   []string
+}
+
+func (s *Server) updateLogPath() string { return filepath.Join(s.dir, "update.log") }
+
+func (s *Server) selfNote(stage string, pct int, line string) {
+	u := &s.selfUpd
+	u.mu.Lock()
+	defer u.mu.Unlock()
+	if stage != "" {
+		u.stage = stage
+	}
+	if pct >= 0 {
+		u.pct = pct
+	}
+	if line == "" {
+		return
+	}
+	line = time.Now().Format("15:04:05") + " " + line
+	u.lines = append(u.lines, line)
+	if len(u.lines) > maxUpdateLines {
+		u.lines = u.lines[len(u.lines)-maxUpdateLines:]
+	}
+	if f, err := os.OpenFile(s.updateLogPath(), os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o600); err == nil {
+		f.WriteString(line + "\n")
+		f.Close()
+	}
+}
+
+func readLines(path string) []string {
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return nil
+	}
+	lines := strings.Split(strings.TrimRight(string(b), "\n"), "\n")
+	if len(lines) == 1 && lines[0] == "" {
+		return nil
+	}
+	if len(lines) > maxUpdateLines {
+		lines = lines[len(lines)-maxUpdateLines:]
+	}
+	return lines
+}
+
+// finishSelfUpdateLog runs at startup: if the last run ended by restarting into this process, say so.
+func (s *Server) finishSelfUpdateLog() {
+	lines := readLines(s.updateLogPath())
+	if n := len(lines); n > 0 && strings.Contains(lines[n-1], "restarting") && s.opts.Version != "" {
+		if f, err := os.OpenFile(s.updateLogPath(), os.O_APPEND|os.O_WRONLY, 0o600); err == nil {
+			f.WriteString(time.Now().Format("15:04:05") + " Hammerhead restarted: now running " + s.opts.Version + "\n")
+			f.Close()
+		}
+	}
+}
+
+// updateStatus returns the progress and log of this server's own update, for the Settings page.
+func (s *Server) updateStatus(w http.ResponseWriter, r *http.Request, p *principal) {
+	u := &s.selfUpd
+	u.mu.Lock()
+	out := map[string]any{"running": u.running, "stage": u.stage, "pct": u.pct, "lines": append([]string(nil), u.lines...), "version": s.opts.Version}
+	u.mu.Unlock()
+	if lines, _ := out["lines"].([]string); len(lines) == 0 {
+		// nothing in memory: this is a fresh process, so show the last update's log from disk
+		if saved := readLines(s.updateLogPath()); len(saved) > 0 {
+			stage := "done"
+			if strings.Contains(saved[len(saved)-1], "Failed:") {
+				stage = "failed"
+			}
+			out["lines"], out["stage"], out["pct"] = saved, stage, 100
+		}
+	}
+	writeJSON(w, 200, out)
+}

web/resource.html

@@ -3,34 +3,81 @@
 <head>
 <meta charset="utf-8">
 <meta name="viewport" content="width=device-width, initial-scale=1">
+<meta name="color-scheme" content="light dark">
 <title>Sign in</title>
 <style>
-  :root { --bg:#0e1621; --panel:#16212f; --line:#25344a; --text:#e6edf5; --dim:#8da2bb; --accent:#06b6d4; --bad:#ef5350; }
+  /* tokens:begin  Design tokens, the Lightwell design language (docs/design.md). This is the only place raw colors and sizes are defined. */
-  @media (prefers-color-scheme: light) { :root { --bg:#f3f6fa; --panel:#fff; --line:#d5dde8; --text:#18222f; --dim:#5b6c80; --accent:#0891b2; } }
+  :root {
-  * { box-sizing:border-box; }
+    --accent:#1f5c5c; --accent-contrast:#ffffff;
-  body { margin:0; background:var(--bg); color:var(--text); font:15px/1.45 system-ui,sans-serif; display:grid; place-items:center; min-height:100vh; padding:16px; }
+    --bg:#f6f5f3; --surface:#ffffff; --surface-2:#efede9; --border:#e0ddd7; --border-strong:#c4c0b8;
-  .card { background:var(--panel); border:1px solid var(--line); border-radius:12px; padding:24px; width:100%; max-width:380px; }
+    --text:#1d1c1a; --text-muted:#5f5b54; --text-faint:#7a766e;
-  h1 { font-size:18px; margin:0 0 4px; }
+    --accent-text:var(--accent);
-  .sub { color:var(--dim); margin:0 0 16px; font-size:13px; }
+    --accent-soft:color-mix(in srgb, var(--accent) 9%, var(--surface));
-  .sec { border-top:1px solid var(--line); padding-top:14px; margin-top:14px; display:flex; flex-direction:column; gap:8px; }
+    --accent-hover:color-mix(in srgb, var(--accent) 88%, #000000);
+    --focus:color-mix(in srgb, var(--accent) 70%, #ffffff);
+    --success:#1c6b3a; --success-bg:#e3f1e8; --warning:#85560a; --warning-bg:#f9eed6; --danger:#a3271e; --danger-bg:#f9e6e3; --neutral-bg:#eceae6;
+    --overlay:rgb(20 19 17 / 0.55); --shadow:0 8px 24px rgb(20 19 17 / 0.14);
+    --s1:4px; --s2:8px; --s3:12px; --s4:16px; --s5:24px; --s6:32px; --s7:48px; --s8:64px;
+    --radius:6px; --radius-lg:10px; --border-w:1px;
+    --font:system-ui,-apple-system,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif;
+    --font-mono:ui-monospace,"SF Mono",Menlo,Consolas,monospace;
+    --text-xs:12px; --text-sm:13px; --text-base:15px; --text-lg:18px; --text-xl:22px; --text-2xl:28px; --leading:1.5;
+    --sidebar-w:232px; --content-max:1120px; --touch:44px; --control-h:38px;
+    --z-sticky:20; --z-overlay:100; --z-toast:200; --ease:140ms ease;
+    color-scheme:light;
+  }
+  @media (prefers-color-scheme: dark) { :root:not([data-theme="light"]) {
+    --bg:#131312; --surface:#1b1b19; --surface-2:#252523; --border:#33322e; --border-strong:#4a4943;
+    --text:#ecebe7; --text-muted:#adaaa2; --text-faint:#8f8c84;
+    --accent-text:color-mix(in srgb, var(--accent) 45%, #ffffff);
+    --accent-soft:color-mix(in srgb, var(--accent) 22%, var(--surface));
+    --accent-hover:color-mix(in srgb, var(--accent) 85%, #ffffff);
+    --success:#86d3a0; --success-bg:#17301f; --warning:#e8bd6b; --warning-bg:#33280f; --danger:#f19a92; --danger-bg:#3a1b18; --neutral-bg:#2b2a27;
+    --overlay:rgb(0 0 0 / 0.65); --shadow:0 8px 24px rgb(0 0 0 / 0.5); color-scheme:dark;
+  } }
+  :root[data-theme="dark"] {
+    --bg:#131312; --surface:#1b1b19; --surface-2:#252523; --border:#33322e; --border-strong:#4a4943;
+    --text:#ecebe7; --text-muted:#adaaa2; --text-faint:#8f8c84;
+    --accent-text:color-mix(in srgb, var(--accent) 45%, #ffffff);
+    --accent-soft:color-mix(in srgb, var(--accent) 22%, var(--surface));
+    --accent-hover:color-mix(in srgb, var(--accent) 85%, #ffffff);
+    --success:#86d3a0; --success-bg:#17301f; --warning:#e8bd6b; --warning-bg:#33280f; --danger:#f19a92; --danger-bg:#3a1b18; --neutral-bg:#2b2a27;
+    --overlay:rgb(0 0 0 / 0.65); --shadow:0 8px 24px rgb(0 0 0 / 0.5); color-scheme:dark;
+  }
+  /* tokens:end */
+
+  *, *::before, *::after { box-sizing:border-box; }
+  html { -webkit-text-size-adjust:100%; }
+  body { margin:0; background:var(--bg); color:var(--text); font:var(--text-base)/var(--leading) var(--font); display:grid; place-items:center; min-height:100vh; padding:var(--s4); -webkit-font-smoothing:antialiased; }
+  .card { background:var(--surface); border:var(--border-w) solid var(--border); border-radius:var(--radius-lg); padding:var(--s5); width:100%; max-width:440px; }
+  h1 { font-size:var(--text-xl); font-weight:600; letter-spacing:-.01em; line-height:1.25; margin:0 0 var(--s1); }
+  h2 { font-size:var(--text-base); font-weight:600; margin:0; }
+  .lead { color:var(--text-muted); margin:0 0 var(--s4); }
+  .sec { border-top:var(--border-w) solid var(--border); padding-top:var(--s4); margin-top:var(--s4); display:flex; flex-direction:column; gap:var(--s3); }
   .sec:first-of-type { border-top:0; padding-top:0; margin-top:0; }
-  h2 { font-size:12px; text-transform:uppercase; letter-spacing:.6px; color:var(--dim); margin:0; }
+  label { display:flex; flex-direction:column; gap:var(--s1); font-size:var(--text-sm); font-weight:600; }
-  input { background:var(--bg); color:var(--text); border:1px solid var(--line); border-radius:6px; padding:9px 10px; font:inherit; width:100%; }
+  input { background:var(--surface); color:var(--text); border:var(--border-w) solid var(--border-strong); border-radius:var(--radius); padding:var(--s2) var(--s3); min-height:var(--control-h); font:inherit; font-weight:400; width:100%; }
-  input:focus { outline:2px solid var(--accent); outline-offset:-1px; }
+  input:focus { outline:2px solid var(--focus); outline-offset:0; border-color:var(--accent); }
-  button, a.btn { background:var(--accent); color:#fff; border:0; border-radius:6px; padding:9px 12px; font:inherit; font-weight:600; cursor:pointer; text-align:center; text-decoration:none; display:block; }
+  :focus-visible { outline:2px solid var(--focus); outline-offset:2px; border-radius:var(--radius); }
-  button.ghost, a.ghost { background:none; border:1px solid var(--line); color:var(--text); }
+  button, a.btn { display:flex; align-items:center; justify-content:center; min-height:var(--control-h); padding:0 var(--s4); background:var(--accent); color:var(--accent-contrast); border:var(--border-w) solid var(--accent); border-radius:var(--radius); font:inherit; font-weight:500; cursor:pointer; text-decoration:none; transition:background var(--ease), border-color var(--ease); }
-  #err { color:var(--bad); min-height:18px; font-size:13px; margin-top:10px; }
+  button:hover, a.btn:hover { background:var(--accent-hover); border-color:var(--accent-hover); }
-  .ok { color:var(--dim); font-size:13px; }
+  button.ghost, a.ghost { background:var(--surface); border-color:var(--border-strong); color:var(--text); }
+  button.ghost:hover, a.ghost:hover { background:var(--surface-2); border-color:var(--border-strong); }
+  #err { color:var(--danger); font-size:var(--text-sm); margin-top:var(--s3); }
+  #err:empty { display:none; }
+  .hint { color:var(--text-muted); font-size:var(--text-sm); margin:0; }
   [hidden] { display:none !important; }
+  @media (prefers-reduced-motion: reduce) { * { transition:none !important; } }
+  @media (pointer: coarse) { button, a.btn, input { min-height:var(--touch); } input { font-size:16px; } }
 </style>
 </head>
 <body>
-<div class="card">
+<main class="card">
   <h1 id="title">Sign in</h1>
-  <p class="sub" id="sub"></p>
+  <p class="lead" id="sub"></p>
   <div id="body"></div>
-  <div id="err"></div>
+  <div id="err" role="alert"></div>
-</div>
+</main>
 <script>
 const id = location.pathname.split('/')[3];
 const params = new URLSearchParams(location.search);
@@ -49,6 +96,9 @@ function h(tag, props, ...kids) {
   return el;
 }
 const err = m => $('#err').textContent = m || '';
+let n = 0;
+// a visible label above the control, as the design language requires
+const field = (label, input) => { input.id = input.id || 'f' + (++n); return h('label', {for:input.id}, label, input); };
 
 async function post(path, data) {
   err('');
@@ -67,54 +117,57 @@ async function finish(path, data) {
 function section(title, ...kids) { return h('div', {class:'sec'}, h('h2', {}, title), ...kids); }
 
 function passwordForm(kind, label, type) {
-  const i = h('input', {type, placeholder:label, autocomplete: kind === 'password' ? 'current-password' : 'off', inputmode: kind === 'pin' ? 'numeric' : null});
+  const i = h('input', {type, autocomplete: kind === 'password' ? 'current-password' : 'off', inputmode: kind === 'pin' ? 'numeric' : null});
   const go = () => finish(kind, { [kind]: i.value });
   i.addEventListener('keydown', e => e.key === 'Enter' && go());
-  return section(label, i, h('button', {onclick:go}, 'Continue'));
+  return h('div', {class:'sec'}, field(label, i), h('button', {onclick:go}, 'Continue'));
 }
 
 function emailForm() {
   const email = h('input', {type:'email', placeholder:'you@example.com', autocomplete:'email'});
-  const code = h('input', {placeholder:'6-digit code', inputmode:'numeric', hidden:true});
+  const code = h('input', {inputmode:'numeric', autocomplete:'one-time-code'});
-  const msg = h('div', {class:'ok'});
+  const codeField = field('6-digit code', code); codeField.hidden = true;
+  const msg = h('p', {class:'hint'});
   const go = h('button', {}, 'Email me a code');
   go.onclick = async () => {
     try {
-      if (code.hidden) {
+      if (codeField.hidden) {
         const j = await post(`/auth/r/${id}/email/start`, { email: email.value, return: ret });
-        msg.textContent = j.status; code.hidden = false; go.textContent = 'Verify code'; code.focus();
+        msg.textContent = j.status; codeField.hidden = false; go.textContent = 'Verify code'; code.focus();
       } else await finish('email/verify', { email: email.value, code: code.value });
     } catch (e) { err(e.message); }
   };
-  return section('Email code', email, code, msg, go);
+  return section('Email code', field('Email address', email), codeField, msg, go);
 }
 
 function ssoForm(info) {
   const kids = [];
   if (info.user) {
     if (info.user.allowed) {
-      kids.push(h('div', {class:'ok'}, `Signed in as ${info.user.email}`), h('button', {onclick:() => finish('sso', {})}, 'Continue'));
+      kids.push(h('p', {class:'hint'}, `Signed in as ${info.user.email}.`), h('button', {onclick:() => finish('sso', {})}, 'Continue'));
     } else {
-      kids.push(h('div', {class:'ok'}, `${info.user.email} doesn't have access to this resource.`));
+      kids.push(h('p', {class:'hint'}, `${info.user.email} does not have access to this resource. Ask the person who shared it to add you.`));
     }
     kids.push(h('button', {class:'ghost', onclick: async () => { await fetch('/api/logout', {method:'POST'}); location.reload(); }}, 'Use a different account'));
     return section('Account', ...kids);
   }
-  const email = h('input', {type:'email', placeholder:'Email', autocomplete:'username'});
+  const email = h('input', {type:'email', autocomplete:'username'});
-  const pw = h('input', {type:'password', placeholder:'Password', autocomplete:'current-password'});
+  const pw = h('input', {type:'password', autocomplete:'current-password'});
-  const code = h('input', {placeholder:'Authenticator code', inputmode:'numeric', autocomplete:'one-time-code', hidden:true});
+  const code = h('input', {inputmode:'numeric', autocomplete:'one-time-code'});
+  const emailField = field('Email address', email), pwField = field('Password', pw), codeField = field('Authenticator code', code);
+  codeField.hidden = true;
   let ticket = null;
   const go = async () => {
     try {
       let j;
       if (ticket) j = await post('/api/login/totp', { ticket, code: code.value });
       else j = await post('/api/login', { email: email.value, password: pw.value });
-      if (j.needTotp) { ticket = j.ticket; code.hidden = false; email.hidden = pw.hidden = true; code.focus(); return; }
+      if (j.needTotp) { ticket = j.ticket; codeField.hidden = false; emailField.hidden = pwField.hidden = true; code.focus(); return; }
       location.reload();
     } catch (e) { err(e.message); }
   };
   [email, pw, code].forEach(i => i.addEventListener('keydown', e => e.key === 'Enter' && go()));
-  kids.push(email, pw, code, h('button', {onclick:go}, 'Sign in'));
+  kids.push(emailField, pwField, codeField, h('button', {onclick:go}, 'Sign in'));
   for (const p of info.providers || []) {
     const back = `/auth/r/${id}?return=${encodeURIComponent(ret)}`;
     kids.push(h('a', {class:'btn ghost', href:`/auth/oidc/${p.id}/start?return=${encodeURIComponent(back)}`}, `Continue with ${p.name}`));
@@ -125,19 +178,20 @@ function ssoForm(info) {
 async function main() {
   // share links skip the form
   const share = params.get('share');
-  if (share) { $('#title').textContent = 'Checking link…'; return finish('share', { token: share }); }
+  if (share) { $('#title').textContent = 'Checking your link'; return finish('share', { token: share }); }
   try {
     const r = await fetch(`/auth/r/${id}/info?return=${encodeURIComponent(ret)}`);
     const info = await r.json();
     if (!r.ok) throw new Error(info.error);
     $('#title').textContent = info.name;
-    $('#sub').textContent = `${info.host} is protected. Sign in to continue.`;
+    document.title = 'Sign in to ' + info.name;
+    $('#sub').textContent = `Sign in to open ${info.host}.`;
     if (info.sso) body.append(ssoForm(info));
     if (info.password) body.append(passwordForm('password', 'Password', 'password'));
     if (info.pin) body.append(passwordForm('pin', 'PIN', 'password'));
     if (info.email) body.append(emailForm());
-    if (!body.children.length) err('No sign-in method is available.');
+    if (!body.children.length) err('No sign-in method is set up for this page. Ask the person who shared it with you.');
-  } catch (e) { $('#title').textContent = 'Unavailable'; err(e.message); }
+  } catch (e) { $('#title').textContent = 'This page is unavailable'; err(e.message); }
 }
 main();
 </script>