Terminal: show the shell's error output, so a missing shell is explained

The shell's stderr was discarded, so a shell that could not start looked like an
immediate disconnect. It now goes to the terminal like other output, followed by
the exit code. An end-to-end test covers a machine with no shell. Builds against
Bamboo v0.1.10, which says why when there is no shell to run.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

README.md

@@ -163,6 +163,17 @@ for each provider (e.g. `CF_DNS_API_TOKEN` for `cloudflare`); `.env` is the plac
 - **Port ranges** for tcp and udp: `2456-2458` opens all three, each sent to the target port plus the
   same offset (up to 1000 ports per entry; a busy port inside a range is named in the status).
 - TCP, UDP, "TCP + UDP" in one click, HTTP and HTTPS reverse proxy, and TLS passthrough by SNI.
+- **SSH into a site, and a terminal in the dashboard.** Start Bamboo with `-ssh` and list the allowed public
+  keys in its `authorized_keys`. The **SSH** button on the Sites page shows Hammerhead's own public key: add
+  it to `authorized_keys` and **Open terminal** gives an administrator a shell in the browser. No port has to
+  be open anywhere: Bamboo reports its SSH server, and when a terminal is opened it calls back to Hammerhead
+  and joins that connection to the server. Hammerhead is the SSH client, saves each site's host key the
+  first time and refuses a different one later (with a button to forget it), allows a terminal only from the
+  dashboard's own origin, logs opening and closing in the audit log, and closes an idle one after 30
+  minutes. The shell runs as the user Bamboo runs as, so treat Hammerhead's key like a root login if Bamboo
+  runs as root. To connect from your own computer with an ordinary `ssh` client, the same dialog can open a
+  TCP port on an edge to that server and shows the command to use. An older Bamboo that does not report its
+  SSH server can still be reached through such a port.
 - **PROXY protocol (v1/v2)** on tcp and tls ports: the target is told the real client address, so IP
   bans, rate limits, spam checks and logs see the visitor instead of the tunnel. Your server must be
   set to expect it and to trust only the edge or site: Velocity/BungeeCord `haproxy-protocol`, Postfix
@@ -171,7 +182,9 @@ for each provider (e.g. `CF_DNS_API_TOKEN` for `cloudflare`); `.env` is the plac
 - Several targets per port: round-robin load balancing with automatic failover.
 - Health checks (TCP connect or an HTTP path); unhealthy targets are skipped and shown in the UI.
 - Conflict detection (same ip/port/protocol, overlapping ranges, or `0.0.0.0` overlapping a specific IP).
-- Traffic totals and a 24-hour sparkline per port.
+- Traffic totals and a 24-hour sparkline per port, plus a **Traffic** chart per resource (and a
+  **Traffic overview** for the whole organization) with bytes in and out, connections and the
+  busiest visitor addresses. See *Traffic analytics* below.
 
 **HTTPS**: automatic Let's Encrypt certificates, optional port-80 redirect, `X-Forwarded-*`
 headers, Host rewrite, HTTPS to the backend, certificate status per port.
@@ -200,11 +213,26 @@ headers, Host rewrite, HTTPS to the backend, certificate status per port.
 **Alerts**
 - Email or webhook (JSON, Slack, Discord, ntfy) when an edge or site goes quiet, a port can't bind, a
   target fails its health check, a certificate errors, or a mail relay isn't deliverable (reverse DNS,
-  HELO, port 25, DKIM record missing).
+  HELO, port 25, DKIM record missing), a sending domain's SPF doesn't authorize the relay or its
+  DMARC record is missing, the relay's IP lands on a DNS blocklist, too much of its mail is being
+  refused, or too little of the mail it sends passes DMARC in the reports you upload. Each alert says
+  what to change.
 - A problem must last the **debounce time** (default 2 minutes, per organization) before you hear about
   it, so restarts and blips stay quiet. One message when it starts, one when it clears, optional
   reminders. Active alerts are stored, so restarting Hammerhead doesn't repeat them, and an alert is
   only marked resolved once the thing it describes has been re-checked.
+- **Sandbar** ([self-hosted notifications, monitors and on-call](https://git.veillax.com/Veillax/sandbar)) can
+  receive alerts in two ways. A *Sandbar* channel raises the alert through its API (and resolves it when the
+  problem clears), so the escalation policy you name pages whoever is on call. A *Sandbar topic* channel
+  publishes a plain message to a topic, which anyone subscribed (the web page, `curl`, the ntfy phone app) sees;
+  the API key is optional there, for servers that don't ask for one. Keys are sealed at rest and never sent back
+  to the browser.
+- **Sandbar monitors** (Server, Settings): turn this on with Sandbar's address and an administrator's API key,
+  and Hammerhead creates a Sandbar monitor for every enabled resource that has health checks: a website is
+  fetched at its public address (a sign-in page counts as up), a TCP port is connected to on the edge's public
+  IP. Each gets uptime history, an optional place on the status page, and the escalation policy you pick.
+  Redirects, UDP, wildcard hostnames and resources with health checks off get none. Hammerhead re-syncs every
+  minute, renames and removes the monitors it created, and never touches ones you made in Sandbar by hand.
 - A dead edge silences its own ports' alerts (the edge alert already says it). Channels are per
   organization; **server channels** get edge alerts.
 - Webhooks can't be pointed at internal addresses (no metadata service, no LAN) unless a server admin
@@ -212,6 +240,24 @@ headers, Host rewrite, HTTPS to the backend, certificate status per port.
 - **`/metrics`** (Prometheus) when `HAMMERHEAD_METRICS_TOKEN` is set: edge/site/port/target/mail-check
   gauges, traffic counters, alerts firing. Off (404) otherwise.
 
+**Traffic analytics**
+- Every edge counts connections and bytes in and out per port, plus the ten busiest client addresses
+  per port, and sends them with its regular report (every 20 seconds). Hammerhead adds them up at
+  three resolutions: per minute (kept 48 hours), per hour (31 days) and, for visitor addresses, per
+  hour (7 days). Older rows are deleted by the same hourly clean-up as the logs.
+- The charts are on the Resources page: **Traffic** on a row, or **Traffic overview** for everything.
+  Pick an hour or 6 hours (per minute), a day, a week or a month (per hour). Visitor addresses cover
+  at most the last 7 days, and very small visitors may be missing because each report names ten.
+- `GET /api/orgs/{org}/traffic/{resource id or "all"}?range=1h|6h|24h|7d|30d` returns the series,
+  totals and busiest addresses as JSON, for scripts or your own dashboards.
+- **Traffic alerts** (Alerts → Traffic, per organization, off by default), delivered like every other
+  alert, with the same debounce:
+  - *Spike*: a port carried at least N times its usual traffic in the last 5 minutes, compared with
+    the average over the previous 6 hours. A port needs an hour of history, and the burst must be over
+    10 MB, so quiet ports don't page you.
+  - *Daily amount*: a port carried more than the set number of MB (in plus out) since midnight UTC.
+  Neither blocks traffic; they only tell you.
+
 **Mail egress (send your mail server's outgoing mail from the edge's IP)**
 - Receiving servers judge mail by the IP it comes from: its reverse DNS, your HELO name and your SPF
   record all have to agree. A mail server at home usually can't satisfy that (residential IP, port 25
@@ -227,6 +273,20 @@ headers, Host rewrite, HTTPS to the backend, certificate status per port.
   record it shows. Mail is signed by From domain (a parent domain's key covers subdomains); a message
   your mail server already signed for that domain is left alone, and a signing problem sends the mail
   unsigned rather than dropping it. Hammerhead checks the DNS record and alerts if it's missing or wrong.
+- **Deliverability dashboard.** The Mail egress page shows a checklist per relay, each line with a fix:
+  reverse DNS, HELO, outbound port 25, every DKIM key, and for each domain you sign mail for, SPF
+  (evaluated against the relay's IP, including `include:`/`redirect=` and the 10-lookup limit), DMARC
+  (missing, monitoring-only, no `rua`, enforcing) and MTA-STS. The edge also looks the IP up on DNS
+  blocklists (Spamhaus ZEN, SpamCop, Barracuda, PSBL) and counts the last 24 hours of delivery outcomes
+  (delivered, deferred, refused, by recipient domain, with the latest refusal messages and what they
+  usually mean). Checks run every 15 minutes; failures raise alerts and clear when fixed.
+- **DMARC reports.** Put `rua=mailto:...` in a domain's DMARC record, and upload the daily aggregate
+  reports receivers send (the email itself, or its .xml/.gz/.zip attachment, several at once) on the same
+  page. Hammerhead parses them, keeps 90 days, and shows per domain how much mail passed, which sending
+  addresses fail (marking your own relays), and whether the policy can move from `none` to
+  `quarantine`/`reject`. `POST /api/orgs/<org>/dmarc` takes a report as the raw request body, so a mail
+  filter or script can forward reports automatically. Reports are only accepted for domains you have a
+  DKIM key for.
 - **MTA-STS enforcement.** When a recipient domain publishes an `enforce` policy, mail only goes to the
   servers it lists, over TLS with a verified certificate; otherwise it's a temporary failure so your
   server retries. (`testing` mode doesn't block.) Policies are cached for their `max_age`. The policy
@@ -259,6 +319,19 @@ Blacktip redirects an unauthenticated visitor to Hammerhead's login page. Hammer
 token with a per-edge key and sends the visitor back; Blacktip verifies it and sets a cookie. The same
 decision function runs in both proxy modes.
 
+## Ephemeral exposures (`shark expose`)
+
+A short-lived port or website for development, created and removed by an API key. [shark-expose](https://git.veillax.com/Veillax/shark-expose) is the command-line client; the API is plain JSON, with an API key (**Access**) as the bearer token. `-` can stand for the key's own organization in the path.
+
+| Call | Does |
+|---|---|
+| `POST /api/orgs/-/expose` | `{proto: "http"\|"https"\|"tcp", targetPort, targetHost?, name?, ttlSeconds?, port?, hostname?, edgeId?, auth?: {password?, pin?, sso?}}`. Makes a site and a port on an online edge, and returns the public address plus `siteToken`, the token Bamboo dials in with. |
+| `POST /api/orgs/-/expose/{id}/renew` | Pushes the deadline out by `ttlSeconds` (default 300) from now. |
+| `GET /api/orgs/-/expose[/{id}]` | Lists exposures, with whether the edge reports the port listening. |
+| `DELETE /api/orgs/-/expose/{id}` | Removes it now. Stopping twice is fine. |
+
+TCP gets a free port from 20000-29999. A website gets `<name>-<random>.<one of your domains>` (so a wildcard DNS record for that domain should point at the edge) on port 80/443, or on `port`. An exposure that isn't renewed within its TTL (30 seconds to 24 hours) is removed, along with its site and any login secrets. The dashboard shows them with the `ephemeral` tag; the deadlines live in memory, so a Hammerhead restart removes any that are still there. An organization can have 20 at a time.
+
 ## Design
 
 The dashboard follows the Lightwell design language: warm neutrals, one teal accent, system fonts, status badges with text, and dialogs that name what they affect. See [docs/design.md](docs/design.md). `node scripts/lint-design.mjs` checks the rules that a machine can check.

accounts.go

@@ -54,6 +54,9 @@ func (s *Server) principalFrom(r *http.Request) *principal {
 				if k.ID == parts[1] && hmac.Equal([]byte(k.Hash), []byte(sha(parts[2]))) {
 					k.LastUsed = time.Now()
 					kc := *k
+					if k.OrgID == "" { // a server key acts as a server administrator
+						return &principal{Key: &kc, User: User{Email: "apikey:" + k.Name, ServerAdmin: true}}
+					}
 					return &principal{Key: &kc}
 				}
 			}
@@ -73,14 +76,23 @@ func (s *Server) withUser(fn func(http.ResponseWriter, *http.Request, *principal
 	}
 }
 
+// serverKey reports whether the caller authenticated with a server-level API key.
+func (p *principal) serverKey() bool { return p.Key != nil && p.Key.OrgID == "" }
+
+// withServerAdmin lets signed-in server administrators and server API keys through.
 func (s *Server) withServerAdmin(fn func(http.ResponseWriter, *http.Request, *principal)) http.HandlerFunc {
-	return s.withUser(func(w http.ResponseWriter, r *http.Request, p *principal) {
+	return func(w http.ResponseWriter, r *http.Request) {
+		p := s.principalFrom(r)
+		if p == nil || (p.User.ID == "" && !p.serverKey()) {
+			fail(w, 401, errors.New("Sign in to continue."))
+			return
+		}
 		if !p.User.ServerAdmin {
 			fail(w, 403, errors.New("server administrators only"))
 			return
 		}
 		fn(w, r, p)
-	})
+	}
 }
 
 // orgCtx is the caller's standing inside one org.
@@ -100,6 +112,9 @@ func (s *Server) withOrg(needAdmin bool, fn func(http.ResponseWriter, *http.Requ
 			return
 		}
 		id := r.PathValue("org")
+		if id == "-" && p.Key != nil {
+			id = p.Key.OrgID // an API key belongs to one organization, so "-" can stand for it
+		}
 		s.mu.Lock()
 		org := s.findOrg(id)
 		c := &orgCtx{User: p.User, Key: p.Key}
@@ -107,6 +122,9 @@ func (s *Server) withOrg(needAdmin bool, fn func(http.ResponseWriter, *http.Requ
 		if org != nil {
 			c.Org = *org
 			switch {
+			case p.serverKey():
+				allowed, c.Admin = true, true
+				c.Actor = "apikey:" + p.Key.Name
 			case p.Key != nil:
 				allowed = p.Key.OrgID == org.ID
 				c.Admin = allowed

agent.go

@@ -195,7 +195,7 @@ func (s *Server) siteConfig(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 	have, _ := strconv.ParseInt(r.URL.Query().Get("v"), 10, 64)
-	if !s.wait(r, have) {
+	if !s.waitUntil(r, have, func() bool { return s.newTerminalsLocked(st.ID) }) {
 		s.mu.Lock()
 		if st := s.siteByToken(r); st != nil {
 			st.LastSeen = time.Now()
@@ -235,6 +235,7 @@ func (s *Server) siteConfig(w http.ResponseWriter, r *http.Request) {
 			cfg.MailEgress = append(cfg.MailEgress, model.MailSpec{ID: x.ID, EdgeID: x.EdgeID, Listen: x.Listen, Username: x.Username, Password: x.Password})
 		}
 	}
+	cfg.Terminals = s.waitingTerminalsLocked(st.ID)
 	writeJSON(w, 200, cfg)
 }
 
@@ -259,6 +260,8 @@ func (s *Server) siteReport(w http.ResponseWriter, r *http.Request) {
 	answered := noteUpdateReport(&st.Version, &st.AutoUpdate, &st.UpdateID, &st.UpdateMsg, in.Version, in.AutoUpdate, in.UpdateID, in.UpdateMsg)
 	s.recordUpdateProgress(st.ID, in.UpdateID, in.UpdateStage, in.UpdatePct, in.UpdateLog, oldVersion, st.Version)
 	s.containers[st.ID] = in.Containers
+	s.discovered[st.ID] = cleanDiscovered(in.Discovered)
+	st.SSH, st.SSHRelay = clipText(in.SSH, 100), in.SSHRelay
 	for _, x := range s.st.MailEgress {
 		if x.SiteID != st.ID {
 			continue

alerts.go

@@ -8,6 +8,7 @@ import (
 	"log"
 	"net/http"
 	"net/url"
+	"regexp"
 	"sort"
 	"strings"
 	"time"
@@ -23,21 +24,25 @@ import (
 // message when it starts and one when it clears, and optionally a reminder.
 // Active alerts are kept in the database so a restart doesn't repeat them.
 
+var topicName = regexp.MustCompile(`^[A-Za-z0-9_-]{1,64}$`)
+
 const (
 	defaultDebounce = 2 * time.Minute
 	defaultOffline  = 90 * time.Second
 )
 
-// AlertChannel is somewhere alerts are sent: an email address, a webhook, or a
-// Sandbar server (which pages whoever is on call and tracks the alert).
+// AlertChannel is somewhere alerts are sent: an email address, a webhook, a
+// Sandbar server (which pages whoever is on call and tracks the alert), or a
+// Sandbar topic (a plain message for everyone subscribed to it).
 type AlertChannel struct {
 	ID           string `json:"id"`
 	OrgID        string `json:"orgId"` // "" = a server-level channel (for edges)
 	Name         string `json:"name"`
-	Kind         string `json:"kind"`   // "email", "webhook" or "sandbar"
+	Kind         string `json:"kind"`   // "email", "webhook", "sandbar" or "topic"
 	Target       string `json:"target"` // address(es), a webhook URL, or Sandbar's address (secret: URLs often carry tokens)
 	Format       string `json:"format"` // webhook body: "json", "slack", "discord" or "ntfy"
-	Token        string `json:"token"`  // Sandbar: API key (secret, never sent back to the browser)
+	Token        string `json:"token"`  // Sandbar: API key (secret, never sent back to the browser; optional for a topic)
+	Topic        string `json:"topic"`  // topic: the topic to publish to
 	Policy       string `json:"policy"` // Sandbar: escalation policy id; blank sends the alert without paging anyone
 	AllowPrivate bool   `json:"allowPrivate"`
 	Enabled      bool   `json:"enabled"`
@@ -127,6 +132,7 @@ func (s *Server) problemsLocked(now time.Time) map[string]problem {
 			}
 		}
 	}
+	s.trafficProblemsLocked(now, edgeDown, add)
 	for _, x := range s.st.MailEgress {
 		if !x.Enabled || edgeDown[x.EdgeID] || siteDown[x.SiteID] {
 			continue
@@ -146,6 +152,27 @@ func (s *Server) problemsLocked(now time.Time) map[string]problem {
 			}
 		}
 	}
+	for _, x := range s.st.MailEgress {
+		if !x.Enabled || edgeDown[x.EdgeID] || siteDown[x.SiteID] {
+			continue
+		}
+		chk, ok := s.mailChecks[x.ID]
+		if !ok {
+			continue
+		}
+		for _, f := range s.mailEgressFindingsLocked(x, chk) {
+			// ptr, helo, port 25 and dkim have alerts of their own above
+			if !f.alertWorthy() || f.ID == "ptr" || f.ID == "helo" || f.ID == "port25" || strings.HasPrefix(f.ID, "dkim:") {
+				continue
+			}
+			msg := fmt.Sprintf("%s: %s: %s", x.Name, f.Label, f.Detail)
+			if f.Hint != "" {
+				msg += ". " + f.Hint
+			}
+			add("mailfind:"+x.ID+":"+f.ID, x.OrgID, "mail", x.Name, msg)
+		}
+	}
+	s.dmarcProblemsLocked(add)
 	for _, k := range s.st.DKIM {
 		for _, x := range s.st.MailEgress {
 			if x.OrgID != k.OrgID || !x.Enabled || edgeDown[x.EdgeID] {
@@ -185,6 +212,13 @@ func (s *Server) judgeableLocked(st *alertState) bool {
 		}
 		_, known := s.mailStatus[rest]
 		return known
+	case "mailfind":
+		id, _, _ := strings.Cut(rest, ":")
+		if s.findMailEgress(id) == nil {
+			return true
+		}
+		_, known := s.mailChecks[id]
+		return known
 	case "mailptr", "mailhelo", "mail25":
 		if s.findMailEgress(rest) == nil {
 			return true
@@ -368,6 +402,8 @@ func (s *Server) send(ch AlertChannel, n alertNote) error {
 		return errors.Join(errs...)
 	case "sandbar":
 		return s.sendSandbar(ch, n)
+	case "topic":
+		return s.sendTopic(ch, n)
 	case "webhook":
 		body, ctype, hdr := webhookBody(ch.Format, n)
 		req, err := http.NewRequest("POST", ch.Target, bytes.NewReader(body))
@@ -432,6 +468,38 @@ func (s *Server) sendSandbar(ch AlertChannel, n alertNote) error {
 		"detail": "Organization: " + n.Org + "\nSince: " + n.Since.Format(time.RFC1123), "severity": "critical", "policy": ch.Policy})
 }
 
+// sendTopic publishes the alert as a message on a Sandbar topic (ntfy's request
+// works too), so everyone subscribed, including the ntfy phone apps, sees it. A
+// topic has no alert to close, so a recovery is a second, calmer message.
+func (s *Server) sendTopic(ch AlertChannel, n alertNote) error {
+	prio, tags := "urgent", "rotating_light"
+	if n.Status != "firing" {
+		prio, tags = "default", "white_check_mark"
+	}
+	req, err := http.NewRequest("POST", strings.TrimRight(ch.Target, "/")+"/"+ch.Topic, strings.NewReader(n.text()+"\n\nOrganization: "+n.Org))
+	if err != nil {
+		return err
+	}
+	req.Header.Set("Title", n.subject())
+	req.Header.Set("Priority", prio)
+	req.Header.Set("Tags", tags)
+	if ch.Token != "" {
+		req.Header.Set("Authorization", "Bearer "+ch.Token)
+	}
+	resp, err := netguard.Client(ch.AllowPrivate).Do(req)
+	if err != nil {
+		return err
+	}
+	defer resp.Body.Close()
+	switch {
+	case resp.StatusCode == 401 || resp.StatusCode == 403:
+		return errors.New("Sandbar rejected the API key")
+	case resp.StatusCode >= 300:
+		return fmt.Errorf("Sandbar answered %s", resp.Status)
+	}
+	return nil
+}
+
 func webhookBody(format string, n alertNote) (body []byte, ctype string, hdr map[string]string) {
 	text := n.subject() + "\n" + n.text()
 	switch format {
@@ -462,7 +530,7 @@ func (s *Server) validateChannel(ch *AlertChannel, trusted bool) error {
 				return errors.New("enter one or more email addresses, separated by commas")
 			}
 		}
-		ch.Format, ch.Token, ch.Policy, ch.AllowPrivate = "", "", "", false
+		ch.Format, ch.Token, ch.Policy, ch.Topic, ch.AllowPrivate = "", "", "", "", false
 	case "sandbar":
 		u, err := url.Parse(ch.Target)
 		if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
@@ -471,7 +539,19 @@ func (s *Server) validateChannel(ch *AlertChannel, trusted bool) error {
 		if ch.Token = strings.TrimSpace(ch.Token); ch.Token == "" {
 			return errors.New("enter an API key from Sandbar's Account page")
 		}
-		ch.Policy, ch.Format = strings.TrimSpace(ch.Policy), ""
+		ch.Policy, ch.Format, ch.Topic = strings.TrimSpace(ch.Policy), "", ""
+		if ch.AllowPrivate && !trusted {
+			return errors.New("sending to private addresses (like a service on your home network) needs a server administrator to allow it for this organization")
+		}
+	case "topic":
+		u, err := url.Parse(ch.Target)
+		if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
+			return errors.New("enter Sandbar's address, such as https://sandbar.example.com")
+		}
+		if ch.Topic = strings.TrimSpace(ch.Topic); !topicName.MatchString(ch.Topic) {
+			return errors.New("a topic is 1 to 64 letters, digits, - or _")
+		}
+		ch.Token, ch.Policy, ch.Format = strings.TrimSpace(ch.Token), "", ""
 		if ch.AllowPrivate && !trusted {
 			return errors.New("sending to private addresses (like a service on your home network) needs a server administrator to allow it for this organization")
 		}
@@ -480,7 +560,7 @@ func (s *Server) validateChannel(ch *AlertChannel, trusted bool) error {
 		if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
 			return errors.New("the webhook must be an http:// or https:// URL")
 		}
-		ch.Token, ch.Policy = "", ""
+		ch.Token, ch.Policy, ch.Topic = "", "", ""
 		switch ch.Format {
 		case "json", "slack", "discord", "ntfy":
 		case "":
@@ -492,7 +572,7 @@ func (s *Server) validateChannel(ch *AlertChannel, trusted bool) error {
 			return errors.New("sending to private addresses (like a service on your home network) needs a server administrator to allow it for this organization")
 		}
 	default:
-		return errors.New("a channel is an email, a webhook or a Sandbar server")
+		return errors.New("a channel is an email, a webhook, a Sandbar server or a Sandbar topic")
 	}
 	if ch.Name == "" {
 		ch.Name = ch.Kind

alerts_api.go

@@ -26,6 +26,9 @@ func preview(ch AlertChannel) string {
 	if ch.Kind == "email" {
 		return ch.Target
 	}
+	if u, err := url.Parse(ch.Target); err == nil && ch.Kind == "topic" {
+		return u.Scheme + "://" + u.Host + "/" + ch.Topic
+	}
 	if u, err := url.Parse(ch.Target); err == nil {
 		return u.Scheme + "://" + u.Host + "/…"
 	}
@@ -61,12 +64,13 @@ func (s *Server) alertOverview(w http.ResponseWriter, sc alertScope) {
 		}
 	}
 	sort.Slice(active, func(i, j int) bool { return active[i].Since.Before(active[j].Since) })
-	settings := map[string]int{"debounceMin": 2, "repeatHours": 0}
+	settings := map[string]int{"debounceMin": 2, "repeatHours": 0, "spikeFactor": 0, "dailyMb": 0}
 	if o := s.findOrg(sc.org); o != nil {
 		if o.AlertDebounceMin > 0 {
 			settings["debounceMin"] = o.AlertDebounceMin
 		}
 		settings["repeatHours"] = o.AlertRepeatHours
+		settings["spikeFactor"], settings["dailyMb"] = o.TrafficSpikeFactor, o.TrafficDailyMB
 	}
 	mailOK := s.st.Settings.SMTP.Host != ""
 	s.mu.Unlock()
@@ -95,7 +99,7 @@ func (s *Server) saveChannel(w http.ResponseWriter, r *http.Request, sc alertSco
 		if in.Target == "" {
 			in.Target = existing.Target // blank keeps the saved URL/address
 		}
-		if id != "" && in.Token == "" && in.Kind == "sandbar" {
+		if id != "" && in.Token == "" && (in.Kind == "sandbar" || in.Kind == "topic") {
 			in.Token = existing.Token // blank keeps the saved API key
 		}
 	}
@@ -114,7 +118,7 @@ func (s *Server) saveChannel(w http.ResponseWriter, r *http.Request, sc alertSco
 	}
 	s.save()
 	s.audit(sc.org, sc.actor, "alert.channel.save", in.Name)
-	ok(w)
+	writeJSON(w, 200, map[string]any{"ok": true, "id": in.ID})
 }
 
 func (s *Server) deleteChannel(w http.ResponseWriter, r *http.Request, sc alertScope) {
@@ -198,6 +202,7 @@ func (s *Server) registerAlertRoutes(mux *http.ServeMux) {
 	mux.HandleFunc("DELETE /api/orgs/{org}/alerts/channels/{id}", s.withOrgAlerts(s.deleteChannel))
 	mux.HandleFunc("POST /api/orgs/{org}/alerts/channels/{id}/test", s.withOrgAlerts(s.testChannel))
 	mux.HandleFunc("PUT /api/orgs/{org}/alerts/settings", s.withOrgAlerts(s.alertSettings))
+	mux.HandleFunc("PUT /api/orgs/{org}/alerts/traffic", s.withOrgAlerts(s.trafficAlertSettings))
 	mux.HandleFunc("GET /api/admin/alerts", s.withServerAlerts(overview))
 	mux.HandleFunc("POST /api/admin/alerts/channels", s.withServerAlerts(s.saveChannel))
 	mux.HandleFunc("PUT /api/admin/alerts/channels/{id}", s.withServerAlerts(s.saveChannel))

apikeys_test.go

@@ -0,0 +1,83 @@
+package hammerhead
+
+import (
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+)
+
+func keyCall(t *testing.T, s *Server, key, method, path, body string) (int, map[string]any) {
+	t.Helper()
+	req := httptest.NewRequest(method, path, strings.NewReader(body))
+	if key != "" {
+		req.Header.Set("Authorization", "Bearer "+key)
+	}
+	rec := httptest.NewRecorder()
+	s.Handler().ServeHTTP(rec, req)
+	var out map[string]any
+	json.Unmarshal(rec.Body.Bytes(), &out)
+	return rec.Code, out
+}
+
+func mintKey(t *testing.T, s *Server, org string) string {
+	t.Helper()
+	k := "hhk_" + randHex(4) + "_" + randHex(24)
+	parts := strings.SplitN(k, "_", 3)
+	s.mu.Lock()
+	s.st.APIKeys = append(s.st.APIKeys, &APIKey{ID: parts[1], OrgID: org, Name: "t", Hash: sha(parts[2])})
+	s.mu.Unlock()
+	return k
+}
+
+func TestServerKeysManageEdgesAndOrgs(t *testing.T) {
+	f := newFx(t)
+	server := mintKey(t, f.s, "")
+	orgKey := mintKey(t, f.s, f.org)
+
+	if c, _ := keyCall(t, f.s, server, "POST", "/api/edges", `{"name":"e2","tunnelAddr":"1.2.3.4:51820"}`); c != 200 {
+		t.Fatalf("server key should create an edge, got %d", c)
+	}
+	if c, _ := keyCall(t, f.s, orgKey, "POST", "/api/edges", `{"name":"e3","tunnelAddr":"1.2.3.4:51820"}`); c != 401 {
+		t.Fatalf("org key must not create edges, got %d", c)
+	}
+	if c, _ := keyCall(t, f.s, server, "POST", "/api/orgs", `{"name":"Second"}`); c != 200 {
+		t.Fatalf("server key should create an org, got %d", c)
+	}
+	// server key reaches any org's endpoints; an org key only its own
+	if c, _ := keyCall(t, f.s, server, "POST", "/api/orgs/"+f.org+"/sites", `{"name":"x"}`); c != 200 {
+		t.Fatalf("server key on org: %d", c)
+	}
+	if c, _ := keyCall(t, f.s, orgKey, "POST", "/api/orgs/nope/sites", `{"name":"x"}`); c != 404 {
+		t.Fatalf("org key on other org: %d", c)
+	}
+	// server alert channels
+	if c, _ := keyCall(t, f.s, server, "GET", "/api/admin/alerts", ""); c != 200 {
+		t.Fatalf("server key alerts: %d", c)
+	}
+	// keys can't mint keys at server level
+	if c, _ := keyCall(t, f.s, server, "POST", "/api/admin/apikeys", `{"name":"k"}`); c != 401 {
+		t.Fatalf("server key must not mint server keys, got %d", c)
+	}
+	if c, _ := keyCall(t, f.s, "", "GET", "/api/orgs", ""); c != 401 {
+		t.Fatalf("anonymous list orgs: %d", c)
+	}
+	req := httptest.NewRequest("GET", "/api/orgs", nil)
+	req.Header.Set("Authorization", "Bearer "+orgKey)
+	rec := httptest.NewRecorder()
+	f.s.Handler().ServeHTTP(rec, req)
+	var orgs []Org
+	json.Unmarshal(rec.Body.Bytes(), &orgs)
+	if rec.Code != http.StatusOK || len(orgs) != 1 || orgs[0].ID != f.org {
+		t.Fatalf("org key should list only its org: %d %s", rec.Code, rec.Body)
+	}
+	// org key lists edges its org may use
+	req = httptest.NewRequest("GET", "/api/edges", nil)
+	req.Header.Set("Authorization", "Bearer "+orgKey)
+	rec = httptest.NewRecorder()
+	f.s.Handler().ServeHTTP(rec, req)
+	if rec.Code != 200 || !strings.Contains(rec.Body.String(), "edge-1") {
+		t.Fatalf("org key edges: %d %s", rec.Code, rec.Body)
+	}
+}

db.go

@@ -44,6 +44,16 @@ CREATE TABLE IF NOT EXISTS traffic_hourly (
 	bytes_in INTEGER NOT NULL DEFAULT 0, bytes_out INTEGER NOT NULL DEFAULT 0,
 	PRIMARY KEY (mapping, hour)
 );
+CREATE TABLE IF NOT EXISTS traffic_minutely (
+	mapping TEXT NOT NULL, minute INTEGER NOT NULL, conns INTEGER NOT NULL DEFAULT 0,
+	bytes_in INTEGER NOT NULL DEFAULT 0, bytes_out INTEGER NOT NULL DEFAULT 0,
+	PRIMARY KEY (mapping, minute)
+);
+CREATE TABLE IF NOT EXISTS traffic_sources (
+	mapping TEXT NOT NULL, hour INTEGER NOT NULL, ip TEXT NOT NULL, conns INTEGER NOT NULL DEFAULT 0,
+	bytes_in INTEGER NOT NULL DEFAULT 0, bytes_out INTEGER NOT NULL DEFAULT 0,
+	PRIMARY KEY (mapping, hour, ip)
+);
 CREATE TABLE IF NOT EXISTS alerts (
 	key TEXT PRIMARY KEY, org_id TEXT NOT NULL DEFAULT '', kind TEXT, name TEXT, message TEXT,
 	since INTEGER NOT NULL, fired INTEGER NOT NULL DEFAULT 0, notified INTEGER NOT NULL DEFAULT 0
@@ -53,7 +63,7 @@ CREATE TABLE IF NOT EXISTS alert_events (
 	key TEXT, status TEXT, name TEXT, message TEXT
 );
 CREATE INDEX IF NOT EXISTS alert_events_org ON alert_events (org_id, id);
-`
+` + dmarcSchema
 
 type DB struct {
 	sql *sql.DB
@@ -117,6 +127,8 @@ func (d *DB) sealed(v any) any {
 		return &c
 	case Settings:
 		x.SMTP.Pass = b.Seal(x.SMTP.Pass)
+		x.Sandbar.Token = b.Seal(x.Sandbar.Token)
+		x.Terminal.Key = b.Seal(x.Terminal.Key)
 		return x
 	}
 	return v
@@ -146,7 +158,7 @@ func (d *DB) unseal(v any) error {
 	case *AlertChannel:
 		return errors.Join(d.open1(&x.Target), d.open1(&x.Token))
 	case *Settings:
-		return d.open1(&x.SMTP.Pass)
+		return errors.Join(d.open1(&x.SMTP.Pass), d.open1(&x.Sandbar.Token), d.open1(&x.Terminal.Key))
 	}
 	return nil
 }
@@ -547,22 +559,167 @@ func (d *DB) queryAudit(org string, limit int, before int64) []AuditEntry {
 
 // purge deletes logs older than the retention periods.
 func (d *DB) purge(accessDays, auditDays int) {
+	d.purgeDMARC(time.Now().AddDate(0, 0, -dmarcKeepDays))
 	if accessDays > 0 {
 		d.sql.Exec(`DELETE FROM access_log WHERE ts < ?`, ms(time.Now().AddDate(0, 0, -accessDays)))
 	}
 	if auditDays > 0 {
 		d.sql.Exec(`DELETE FROM audit_log WHERE ts < ?`, ms(time.Now().AddDate(0, 0, -auditDays)))
 	}
-	d.sql.Exec(`DELETE FROM traffic_hourly WHERE hour < ?`, time.Now().Add(-31*24*time.Hour).Unix()/3600)
+	d.sql.Exec(`DELETE FROM traffic_hourly WHERE hour < ?`, time.Now().Add(-trafficHourlyKeep).Unix()/3600)
+	d.sql.Exec(`DELETE FROM traffic_minutely WHERE minute < ?`, time.Now().Add(-trafficMinutelyKeep).Unix()/60)
+	d.sql.Exec(`DELETE FROM traffic_sources WHERE hour < ?`, time.Now().Add(-trafficSourcesKeep).Unix()/3600)
 	d.sql.Exec(`DELETE FROM alert_events WHERE ts < ?`, ms(time.Now().AddDate(0, 0, -90)))
 }
 
 // ---------- traffic history ----------
 
-func (d *DB) addTraffic(mapping string, st model.Stat) {
+// How long each resolution of traffic history is kept. Minutes are for the last hours, hours for the
+// months, and the busiest client addresses sit in between.
+const (
+	trafficMinutelyKeep = 48 * time.Hour
+	trafficSourcesKeep  = 7 * 24 * time.Hour
+	trafficHourlyKeep   = 31 * 24 * time.Hour
+)
+
+func (d *DB) addTraffic(mapping string, st model.Stat) { d.addTrafficAt(mapping, st, time.Now()) }
+
+func (d *DB) addTrafficAt(mapping string, st model.Stat, at time.Time) {
 	d.sql.Exec(`INSERT INTO traffic_hourly (mapping, hour, conns, bytes_in, bytes_out) VALUES (?, ?, ?, ?, ?)
 		ON CONFLICT (mapping, hour) DO UPDATE SET conns = conns + excluded.conns, bytes_in = bytes_in + excluded.bytes_in, bytes_out = bytes_out + excluded.bytes_out`,
-		mapping, time.Now().Unix()/3600, st.Conns, st.BytesIn, st.BytesOut)
+		mapping, at.Unix()/3600, st.Conns, st.BytesIn, st.BytesOut)
+	d.sql.Exec(`INSERT INTO traffic_minutely (mapping, minute, conns, bytes_in, bytes_out) VALUES (?, ?, ?, ?, ?)
+		ON CONFLICT (mapping, minute) DO UPDATE SET conns = conns + excluded.conns, bytes_in = bytes_in + excluded.bytes_in, bytes_out = bytes_out + excluded.bytes_out`,
+		mapping, at.Unix()/60, st.Conns, st.BytesIn, st.BytesOut)
+	for _, src := range st.Sources {
+		d.sql.Exec(`INSERT INTO traffic_sources (mapping, hour, ip, conns, bytes_in, bytes_out) VALUES (?, ?, ?, ?, ?, ?)
+			ON CONFLICT (mapping, hour, ip) DO UPDATE SET conns = conns + excluded.conns, bytes_in = bytes_in + excluded.bytes_in, bytes_out = bytes_out + excluded.bytes_out`,
+			mapping, at.Unix()/3600, src.IP, src.Conns, src.BytesIn, src.BytesOut)
+	}
+}
+
+// TrafficPoint is the traffic of one time bucket.
+type TrafficPoint struct {
+	Time     int64 `json:"t"` // unix seconds at the start of the bucket
+	Conns    int64 `json:"conns"`
+	BytesIn  int64 `json:"bytesIn"`
+	BytesOut int64 `json:"bytesOut"`
+}
+
+// trafficSeries sums the given ports' traffic into count buckets of the given size ("minute" or
+// "hour") ending with the current one, oldest first. Empty buckets are included.
+func (d *DB) trafficSeries(ids []string, unit string, count int, now time.Time) []TrafficPoint {
+	table, col, secs := "traffic_hourly", "hour", int64(3600)
+	if unit == "minute" {
+		table, col, secs = "traffic_minutely", "minute", 60
+	}
+	cur := now.Unix() / secs
+	first := cur - int64(count) + 1
+	out := make([]TrafficPoint, count)
+	for i := range out {
+		out[i].Time = (first + int64(i)) * secs
+	}
+	if len(ids) == 0 {
+		return out
+	}
+	q := `SELECT ` + col + `, SUM(conns), SUM(bytes_in), SUM(bytes_out) FROM ` + table +
+		` WHERE ` + col + ` >= ? AND ` + col + ` <= ? AND mapping IN (` + placeholders(len(ids)) + `) GROUP BY ` + col
+	args := []any{first, cur}
+	for _, id := range ids {
+		args = append(args, id)
+	}
+	rows, err := d.sql.Query(q, args...)
+	if err != nil {
+		return out
+	}
+	defer rows.Close()
+	for rows.Next() {
+		var at int64
+		var p TrafficPoint
+		if rows.Scan(&at, &p.Conns, &p.BytesIn, &p.BytesOut) != nil {
+			continue
+		}
+		if i := int(at - first); i >= 0 && i < count {
+			out[i].Conns, out[i].BytesIn, out[i].BytesOut = p.Conns, p.BytesIn, p.BytesOut
+		}
+	}
+	return out
+}
+
+func placeholders(n int) string {
+	if n <= 0 {
+		return ""
+	}
+	return strings.TrimSuffix(strings.Repeat("?,", n), ",")
+}
+
+// TrafficSource is one client address's traffic over a period.
+type TrafficSource struct {
+	IP       string `json:"ip"`
+	Conns    int64  `json:"conns"`
+	BytesIn  int64  `json:"bytesIn"`
+	BytesOut int64  `json:"bytesOut"`
+}
+
+// topSources lists the busiest client addresses of the given ports over the last hours hours.
+func (d *DB) topSources(ids []string, hours, limit int, now time.Time) []TrafficSource {
+	out := []TrafficSource{}
+	if len(ids) == 0 {
+		return out
+	}
+	args := []any{now.Unix()/3600 - int64(hours) + 1}
+	for _, id := range ids {
+		args = append(args, id)
+	}
+	args = append(args, limit)
+	rows, err := d.sql.Query(`SELECT ip, SUM(conns), SUM(bytes_in), SUM(bytes_out) FROM traffic_sources WHERE hour >= ? AND mapping IN (`+placeholders(len(ids))+`)
+		GROUP BY ip ORDER BY SUM(bytes_in) + SUM(bytes_out) DESC, SUM(conns) DESC, ip LIMIT ?`, args...)
+	if err != nil {
+		return out
+	}
+	defer rows.Close()
+	for rows.Next() {
+		var s TrafficSource
+		if rows.Scan(&s.IP, &s.Conns, &s.BytesIn, &s.BytesOut) == nil {
+			out = append(out, s)
+		}
+	}
+	return out
+}
+
+// bytesSince is the total bytes (in + out) a port has carried from the start of the given hour (unix hours) until now.
+func (d *DB) bytesSince(mapping string, fromHour int64) int64 {
+	var n sql.NullInt64
+	d.sql.QueryRow(`SELECT SUM(bytes_in + bytes_out) FROM traffic_hourly WHERE mapping = ? AND hour >= ?`, mapping, fromHour).Scan(&n)
+	return n.Int64
+}
+
+// spikeWindow and spikeBaseline are the minutes a spike is judged over, and the minutes before it
+// that say what normal looks like.
+const (
+	spikeWindow   = 5
+	spikeBaseline = 6 * 60
+)
+
+// recentTraffic returns the bytes a port carried in the last spikeWindow minutes, the average for
+// the same span over the baseline before it, and how many minutes that baseline covers (up to spikeBaseline;
+// fewer for a port with a shorter history, which is averaged over what it has).
+func (d *DB) recentTraffic(mapping string, now time.Time) (recent int64, baselinePerWindow float64, historyMinutes int64) {
+	cur := now.Unix() / 60
+	var r, b, first sql.NullInt64
+	d.sql.QueryRow(`SELECT SUM(CASE WHEN minute > ? THEN bytes_in + bytes_out END), SUM(CASE WHEN minute <= ? THEN bytes_in + bytes_out END), MIN(minute)
+		FROM traffic_minutely WHERE mapping = ? AND minute > ?`, cur-spikeWindow, cur-spikeWindow, mapping, cur-spikeWindow-spikeBaseline).Scan(&r, &b, &first)
+	if !first.Valid {
+		return 0, 0, 0
+	}
+	span := cur - spikeWindow - first.Int64 + 1
+	if span > spikeBaseline {
+		span = spikeBaseline
+	}
+	if span <= 0 {
+		return r.Int64, 0, 0
+	}
+	return r.Int64, float64(b.Int64) / float64(span) * spikeWindow, span
 }
 
 // trafficHistory returns bytes (in + out) per hour for each mapping, oldest first.

deliverability.go

@@ -0,0 +1,143 @@
+package hammerhead
+
+import (
+	"fmt"
+	"sort"
+	"strings"
+
+	"git.veillax.com/Veillax/reef/model"
+)
+
+// A Finding is one line of a mail egress's deliverability checklist: what was
+// checked, how it came out, and what to do about it. The edge produces the raw
+// results; the findings put old and new checks in one shape for the dashboard
+// and the alerts.
+type Finding struct {
+	ID     string `json:"id"`               // stable, e.g. "ptr", "spf:example.com"
+	Label  string `json:"label"`            // "Reverse DNS", "SPF"
+	Domain string `json:"domain,omitempty"` // the sending domain, for per-domain checks
+	Status string `json:"status"`           // pass, warn, fail or none (nothing published)
+	Detail string `json:"detail"`
+	Hint   string `json:"hint,omitempty"`
+}
+
+// alertWorthy says whether a finding should raise an alert: failures do, and so
+// does a missing SPF or DMARC record (bulk senders are required to publish both).
+// Warnings and a missing MTA-STS policy are advice, not problems.
+func (f Finding) alertWorthy() bool {
+	if f.Status == model.FindingFail {
+		return true
+	}
+	return f.Status == model.FindingNone && (strings.HasPrefix(f.ID, "spf:") || strings.HasPrefix(f.ID, "dmarc:"))
+}
+
+// minOutcomeSample is how many messages a window needs before its failure rate says anything.
+const minOutcomeSample = 10
+
+func deliverabilityFindings(x *model.MailEgress, chk model.MailCheck, keys []*model.DKIMKey) []Finding {
+	var out []Finding
+	add := func(f Finding) { out = append(out, f) }
+
+	if chk.PTRMatchesHelo {
+		add(Finding{ID: "ptr", Label: "Reverse DNS", Status: model.FindingPass, Detail: x.SourceIP + " resolves back to " + x.HeloName})
+	} else {
+		found := "no reverse DNS record"
+		if len(chk.PTR) > 0 {
+			found = "reverse DNS is " + strings.Join(chk.PTR, ", ")
+		}
+		add(Finding{ID: "ptr", Label: "Reverse DNS", Status: model.FindingFail, Detail: found + ", not " + x.HeloName,
+			Hint: fmt.Sprintf("Reverse DNS is set at your hosting provider, not in your DNS zone: set the PTR record of %s to %s.", x.SourceIP, x.HeloName)})
+	}
+	if chk.HeloResolves {
+		add(Finding{ID: "helo", Label: "HELO name", Status: model.FindingPass, Detail: x.HeloName + " resolves to " + x.SourceIP})
+	} else {
+		add(Finding{ID: "helo", Label: "HELO name", Status: model.FindingFail, Detail: x.HeloName + " doesn't resolve to " + x.SourceIP,
+			Hint: fmt.Sprintf("Add an A record for %s pointing at %s (an AAAA record for an IPv6 address).", x.HeloName, x.SourceIP)})
+	}
+	if chk.Port25 == "open" {
+		add(Finding{ID: "port25", Label: "Outgoing port 25", Status: model.FindingPass, Detail: "connections to other mail servers work"})
+	} else if chk.Port25 != "" {
+		add(Finding{ID: "port25", Label: "Outgoing port 25", Status: model.FindingFail, Detail: chk.Port25,
+			Hint: "Ask your hosting provider to unblock outgoing port 25 for this address; many VPS hosts block it until you ask."})
+	}
+
+	for _, k := range keys {
+		if k.OrgID != x.OrgID {
+			continue
+		}
+		name := k.Selector + "._domainkey." + k.Domain
+		switch chk.DKIM[k.ID] {
+		case "published":
+			add(Finding{ID: "dkim:" + k.ID, Label: "DKIM", Domain: k.Domain, Status: model.FindingPass, Detail: name + " is published"})
+		case "missing":
+			add(Finding{ID: "dkim:" + k.ID, Label: "DKIM", Domain: k.Domain, Status: model.FindingFail, Detail: name + " isn't in DNS",
+				Hint: "Publish the TXT record shown under DKIM signing; signed mail fails verification until it exists."})
+		case "mismatch":
+			add(Finding{ID: "dkim:" + k.ID, Label: "DKIM", Domain: k.Domain, Status: model.FindingFail, Detail: name + " holds a different key",
+				Hint: "Replace the TXT record's value with the one shown under DKIM signing; it was probably left over from an older key."})
+		}
+	}
+
+	doms := make([]string, 0, len(chk.Domains))
+	for d := range chk.Domains {
+		doms = append(doms, d)
+	}
+	sort.Strings(doms)
+	for _, d := range doms {
+		c := chk.Domains[d]
+		add(Finding{ID: "spf:" + d, Label: "SPF", Domain: d, Status: c.SPF.Status, Detail: c.SPF.Detail, Hint: c.SPF.Hint})
+		add(Finding{ID: "dmarc:" + d, Label: "DMARC", Domain: d, Status: c.DMARC.Status, Detail: c.DMARC.Detail, Hint: c.DMARC.Hint})
+		add(Finding{ID: "mtasts:" + d, Label: "MTA-STS", Domain: d, Status: c.MTASTS.Status, Detail: c.MTASTS.Detail, Hint: c.MTASTS.Hint})
+	}
+
+	switch {
+	case len(chk.Blocklists) > 0:
+		add(Finding{ID: "blocklist", Label: "Blocklists", Status: model.FindingFail, Detail: x.SourceIP + " is listed on " + strings.Join(chk.Blocklists, ", "),
+			Hint: "Look the address up on each list's website to see why it was listed and request removal. Until then expect mail to be rejected or sent to spam; check that no one is relaying spam through this server."})
+	case chk.Listable > 0:
+		add(Finding{ID: "blocklist", Label: "Blocklists", Status: model.FindingPass, Detail: fmt.Sprintf("%s isn't on any of the %d blocklists checked", x.SourceIP, chk.Listable)})
+	case chk.Port25 != "" || chk.PTR != nil:
+		add(Finding{ID: "blocklist", Label: "Blocklists", Status: model.FindingWarn, Detail: "no blocklist answered, so the address couldn't be checked",
+			Hint: "Blocklists often refuse queries from public DNS resolvers; the edge needs a resolver of its own, or its provider's, to check them."})
+	}
+
+	if o := chk.Outcomes; o != nil {
+		if total := o.Delivered + o.Deferred + o.Failed; total >= minOutcomeSample {
+			bad := float64(o.Deferred+o.Failed) / float64(total)
+			detail := fmt.Sprintf("%d of %d messages in the last day were delivered (%d deferred, %d refused)", o.Delivered, total, o.Deferred, o.Failed)
+			hint := outcomeHint(o)
+			switch {
+			case bad > 0.3:
+				add(Finding{ID: "outcomes", Label: "Delivery", Status: model.FindingFail, Detail: detail, Hint: hint})
+			case bad > 0.1:
+				add(Finding{ID: "outcomes", Label: "Delivery", Status: model.FindingWarn, Detail: detail, Hint: hint})
+			default:
+				add(Finding{ID: "outcomes", Label: "Delivery", Status: model.FindingPass, Detail: detail})
+			}
+		}
+	}
+	return out
+}
+
+// outcomeHint explains the latest refusal in terms of what to fix.
+func outcomeHint(o *model.MailOutcomes) string {
+	if len(o.Recent) == 0 {
+		return ""
+	}
+	r := o.Recent[0]
+	low := strings.ToLower(r.Text)
+	advice := "Read the message, it usually says what the receiver objected to."
+	switch {
+	case strings.Contains(low, "spf") || strings.Contains(low, "dmarc") || strings.Contains(low, "dkim") || strings.Contains(low, "authenticat"):
+		advice = "The receiver wants authentication to pass: check the SPF, DKIM and DMARC lines above."
+	case strings.Contains(low, "blocklist") || strings.Contains(low, "blacklist") || strings.Contains(low, "rbl") || strings.Contains(low, "listed") || strings.Contains(low, "spamhaus"):
+		advice = "The receiver rejected the address as listed on a blocklist: see the Blocklists line and request delisting."
+	case strings.Contains(low, "greylist") || strings.Contains(low, "try again") || strings.Contains(low, "temporar"):
+		advice = "That receiver is delaying unfamiliar senders; these retry and normally get through."
+	case strings.Contains(low, "rate") || strings.Contains(low, "too many") || strings.Contains(low, "throttl"):
+		advice = "The receiver is limiting how fast you send to it; slow the sending or lower the relay's per-minute limit."
+	case strings.Contains(low, "ptr") || strings.Contains(low, "reverse"):
+		advice = "The receiver objects to the reverse DNS: see the Reverse DNS line."
+	}
+	return fmt.Sprintf("Latest from %s: %d %s. %s", r.Domain, r.Code, r.Text, advice)
+}

deliverability_test.go

@@ -0,0 +1,127 @@
+package hammerhead
+
+import (
+	"strings"
+	"testing"
+
+	"git.veillax.com/Veillax/reef/model"
+)
+
+func find(fs []Finding, id string) *Finding {
+	for i := range fs {
+		if fs[i].ID == id {
+			return &fs[i]
+		}
+	}
+	return nil
+}
+
+func TestFindingsCarryFixes(t *testing.T) {
+	x := &model.MailEgress{ID: "x1", OrgID: "o", SourceIP: "203.0.113.7", HeloName: "mail.example.com"}
+	keys := []*model.DKIMKey{{ID: "k1", OrgID: "o", Domain: "example.com", Selector: "s1"}, {ID: "k2", OrgID: "other", Domain: "x.test", Selector: "s"}}
+	chk := model.MailCheck{
+		PTR: []string{"host.vps.test"}, Port25: "can't connect",
+		DKIM: map[string]string{"k1": "mismatch", "k2": "missing"},
+		Domains: map[string]model.DomainCheck{"example.com": {
+			SPF:    model.MailFinding{Status: model.FindingFail, Detail: "no", Hint: "add ip4"},
+			DMARC:  model.MailFinding{Status: model.FindingNone, Detail: "none", Hint: "publish"},
+			MTASTS: model.MailFinding{Status: model.FindingNone, Detail: "none", Hint: "optional"},
+		}},
+		Blocklists: []string{"zen.spamhaus.org"},
+	}
+	fs := deliverabilityFindings(x, chk, keys)
+	for id, want := range map[string]string{"ptr": "fail", "helo": "fail", "port25": "fail", "dkim:k1": "fail", "spf:example.com": "fail", "blocklist": "fail"} {
+		f := find(fs, id)
+		if f == nil || f.Status != want {
+			t.Errorf("%s: %+v", id, f)
+		} else if want == "fail" && f.Hint == "" {
+			t.Errorf("%s fails without saying how to fix it", id)
+		}
+	}
+	if find(fs, "dkim:k2") != nil {
+		t.Error("another org's key leaked into the checklist")
+	}
+	if f := find(fs, "ptr"); !strings.Contains(f.Detail, "host.vps.test") || !strings.Contains(f.Hint, "mail.example.com") {
+		t.Errorf("ptr: %+v", f)
+	}
+	if find(fs, "mtasts:example.com").alertWorthy() {
+		t.Error("a missing MTA-STS policy is advice, not an alert")
+	}
+	if !find(fs, "dmarc:example.com").alertWorthy() {
+		t.Error("a missing DMARC record should alert")
+	}
+}
+
+func TestFindingsAllGood(t *testing.T) {
+	x := &model.MailEgress{OrgID: "o", SourceIP: "203.0.113.7", HeloName: "mail.example.com"}
+	chk := model.MailCheck{PTRMatchesHelo: true, HeloResolves: true, Port25: "open", Listable: 4,
+		Outcomes: &model.MailOutcomes{Delivered: 100}}
+	for _, f := range deliverabilityFindings(x, chk, nil) {
+		if f.Status != model.FindingPass || f.alertWorthy() {
+			t.Errorf("%+v", f)
+		}
+	}
+}
+
+func TestOutcomeFindings(t *testing.T) {
+	x := &model.MailEgress{OrgID: "o", SourceIP: "203.0.113.7", HeloName: "h"}
+	status := func(o *model.MailOutcomes) string {
+		f := find(deliverabilityFindings(x, model.MailCheck{Outcomes: o}, nil), "outcomes")
+		if f == nil {
+			return ""
+		}
+		return f.Status
+	}
+	for name, c := range map[string]struct {
+		o    *model.MailOutcomes
+		want string
+	}{
+		"too few to judge": {&model.MailOutcomes{Delivered: 2, Failed: 5}, ""},
+		"healthy":          {&model.MailOutcomes{Delivered: 95, Deferred: 5}, "pass"},
+		"some trouble":     {&model.MailOutcomes{Delivered: 80, Failed: 20}, "warn"},
+		"mostly refused":   {&model.MailOutcomes{Delivered: 20, Failed: 80}, "fail"},
+	} {
+		if got := status(c.o); got != c.want {
+			t.Errorf("%s: %q, want %q", name, got, c.want)
+		}
+	}
+	h := outcomeHint(&model.MailOutcomes{Recent: []model.OutcomeFailure{{Domain: "gmail.com", Code: 550, Text: "5.7.26 DMARC authentication failed"}}})
+	if !strings.Contains(h, "gmail.com") || !strings.Contains(h, "SPF, DKIM and DMARC") {
+		t.Errorf("hint: %q", h)
+	}
+	if outcomeHint(&model.MailOutcomes{}) != "" {
+		t.Error("no refusals, no hint")
+	}
+}
+
+func TestDeliverabilityRaisesAlertsWithHints(t *testing.T) {
+	f, _ := dmarcFx(t)
+	f.s.mu.Lock()
+	f.s.mailChecks["x1"] = model.MailCheck{PTRMatchesHelo: true, HeloResolves: true, Port25: "open", Listable: 4,
+		Blocklists: []string{"bl.spamcop.net"},
+		Domains: map[string]model.DomainCheck{"example.com": {
+			SPF:    model.MailFinding{Status: model.FindingFail, Detail: "SPF for example.com does not authorize 203.0.113.7", Hint: "Add ip4:203.0.113.7"},
+			DMARC:  model.MailFinding{Status: model.FindingWarn, Detail: "monitoring"},
+			MTASTS: model.MailFinding{Status: model.FindingNone},
+		}}}
+	probs := f.s.problemsLocked(f.base)
+	f.s.mu.Unlock()
+	spf, bl := probs["mailfind:x1:spf:example.com"], probs["mailfind:x1:blocklist"]
+	if !strings.Contains(spf.msg, "Add ip4:203.0.113.7") || !strings.Contains(bl.msg, "bl.spamcop.net") {
+		t.Fatalf("alerts should carry the fix: %+v %+v", spf, bl)
+	}
+	if _, noisy := probs["mailfind:x1:dmarc:example.com"]; noisy {
+		t.Error("a DMARC warning shouldn't alert")
+	}
+
+	// fixed: both clear
+	f.s.mu.Lock()
+	f.s.mailChecks["x1"] = model.MailCheck{PTRMatchesHelo: true, HeloResolves: true, Port25: "open", Listable: 4}
+	probs = f.s.problemsLocked(f.base)
+	f.s.mu.Unlock()
+	for k := range probs {
+		if strings.HasPrefix(k, "mailfind:") {
+			t.Errorf("%s should have cleared", k)
+		}
+	}
+}

discovered_test.go

@@ -0,0 +1,36 @@
+package hammerhead
+
+import (
+	"strings"
+	"testing"
+
+	"git.veillax.com/Veillax/reef/model"
+)
+
+func TestCleanDiscoveredDropsBadEntriesAndBoundsText(t *testing.T) {
+	in := []model.DiscoveredService{
+		{Host: "192.168.1.5", Port: 22, Proto: "udp", Service: "ssh", Detail: "SSH-2.0", Source: "scan"},
+		{Host: "127.0.0.1", Port: 8080, Service: strings.Repeat("x", 100), Detail: strings.Repeat("d", 1000), Source: "local"},
+		{Host: "<script>", Port: 80},
+		{Host: "10.0.0.1", Port: 0},
+		{Host: "10.0.0.1", Port: 70000},
+		{Host: "10.0.0.2", Port: 80, Source: "whatever"},
+	}
+	got := cleanDiscovered(in)
+	if len(got) != 3 {
+		t.Fatalf("want 3 valid entries, got %+v", got)
+	}
+	if got[0].Proto != "tcp" || got[1].Source != "local" || len(got[1].Service) != 20 || len(got[1].Detail) != 160 || got[2].Source != "scan" {
+		t.Errorf("not normalised: %+v", got)
+	}
+	many := make([]model.DiscoveredService, 900)
+	for i := range many {
+		many[i] = model.DiscoveredService{Host: "10.0.0.1", Port: i%60000 + 1}
+	}
+	if n := len(cleanDiscovered(many)); n != 500 {
+		t.Errorf("not capped: %d", n)
+	}
+	if cleanDiscovered(nil) == nil {
+		t.Error("should be an empty list, not nil, so the API returns []")
+	}
+}

dmarc.go

@@ -0,0 +1,349 @@
+package hammerhead
+
+import (
+	"database/sql"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"sort"
+	"strings"
+	"time"
+
+	"git.veillax.com/Veillax/reef/dmarc"
+	"git.veillax.com/Veillax/reef/model"
+)
+
+// DMARC aggregate reports. Receivers that handle mail claiming to be from your
+// domain mail a daily summary to the rua= address in its DMARC record. Those
+// emails are uploaded here (the raw .xml, .gz, .zip or the whole .eml), parsed,
+// and kept for 90 days; the dashboard shows how much of the mail sent as your
+// domain passed, and which senders are failing.
+
+const (
+	maxDMARCUpload   = 8 << 20
+	dmarcKeepDays    = 90
+	dmarcWindowDays  = 30
+	dmarcAlertMinMsg = 20  // messages from our own relays before their pass rate says anything
+	dmarcAlertBelow  = 0.9 // alert when less of our own mail than this passes DMARC
+)
+
+// ---------- storage ----------
+
+const dmarcSchema = `
+CREATE TABLE IF NOT EXISTS dmarc_reports (
+	org_id TEXT NOT NULL, domain TEXT NOT NULL, reporter TEXT NOT NULL, report_id TEXT NOT NULL,
+	begin_ts INTEGER NOT NULL, end_ts INTEGER NOT NULL, added INTEGER NOT NULL, data TEXT NOT NULL,
+	PRIMARY KEY (org_id, domain, reporter, report_id)
+);
+CREATE INDEX IF NOT EXISTS dmarc_end ON dmarc_reports (org_id, end_ts);
+`
+
+// addDMARC stores a report; it reports false if this exact report was already there.
+func (d *DB) addDMARC(org string, r *dmarc.Report) (bool, error) {
+	data, err := json.Marshal(r)
+	if err != nil {
+		return false, err
+	}
+	res, err := d.sql.Exec(`INSERT OR IGNORE INTO dmarc_reports (org_id, domain, reporter, report_id, begin_ts, end_ts, added, data) VALUES (?, ?, ?, ?, ?, ?, ?, ?)`,
+		org, r.Domain, strings.ToLower(r.Org), r.ReportID, ms(r.Begin), ms(r.End), ms(time.Now()), string(data))
+	if err != nil {
+		return false, err
+	}
+	n, _ := res.RowsAffected()
+	return n > 0, nil
+}
+
+// dmarcReports returns an org's reports that ended after since.
+func (d *DB) dmarcReports(org string, since time.Time) []*dmarc.Report {
+	rows, err := d.sql.Query(`SELECT data FROM dmarc_reports WHERE org_id = ? AND end_ts > ? ORDER BY end_ts`, org, ms(since))
+	if err != nil {
+		return nil
+	}
+	defer rows.Close()
+	var out []*dmarc.Report
+	for rows.Next() {
+		var data string
+		var r dmarc.Report
+		if rows.Scan(&data) == nil && json.Unmarshal([]byte(data), &r) == nil {
+			out = append(out, &r)
+		}
+	}
+	return out
+}
+
+func (d *DB) purgeDMARC(before time.Time) {
+	d.sql.Exec(`DELETE FROM dmarc_reports WHERE end_ts < ?`, ms(before))
+}
+
+func (d *DB) deleteDMARCDomain(org, domain string) (int64, error) {
+	var res sql.Result
+	res, err := d.sql.Exec(`DELETE FROM dmarc_reports WHERE org_id = ? AND domain = ?`, org, domain)
+	if err != nil {
+		return 0, err
+	}
+	return res.RowsAffected()
+}
+
+// ---------- summary ----------
+
+type dmarcSource struct {
+	IP        string `json:"ip"`
+	Messages  int    `json:"messages"`
+	Passed    int    `json:"passed"` // messages that passed DMARC
+	DKIMPass  int    `json:"dkimPass"`
+	SPFPass   int    `json:"spfPass"`
+	Ours      bool   `json:"ours"`      // the address is one of this org's mail egress IPs
+	Rejected  int    `json:"rejected"`  // the receiver quarantined or rejected these
+	Reporters int    `json:"reporters"` // how many receivers saw it
+	Hint      string `json:"hint,omitempty"`
+}
+
+type dmarcDomain struct {
+	Domain    string        `json:"domain"`
+	Reports   int           `json:"reports"`
+	Messages  int           `json:"messages"`
+	Passed    int           `json:"passed"`
+	OurMsgs   int           `json:"ourMessages"` // from our own relays
+	OurPassed int           `json:"ourPassed"`
+	Policy    string        `json:"policy"` // the latest published policy a receiver reported
+	Pct       int           `json:"pct"`
+	Reporters []string      `json:"reporters"`
+	First     time.Time     `json:"first"`
+	Last      time.Time     `json:"last"`
+	Sources   []dmarcSource `json:"sources"`
+	Advice    []string      `json:"advice,omitempty"`
+}
+
+// summarizeDMARC totals reports per domain. ours holds the org's egress source IPs.
+func summarizeDMARC(reports []*dmarc.Report, ours map[string]bool) []dmarcDomain {
+	type acc struct {
+		d         *dmarcDomain
+		src       map[string]*dmarcSource
+		srcRep    map[string]map[string]bool
+		reporters map[string]bool
+		lastEnd   time.Time
+	}
+	by := map[string]*acc{}
+	for _, r := range reports {
+		a := by[r.Domain]
+		if a == nil {
+			a = &acc{d: &dmarcDomain{Domain: r.Domain, First: r.Begin}, src: map[string]*dmarcSource{}, srcRep: map[string]map[string]bool{}, reporters: map[string]bool{}}
+			by[r.Domain] = a
+		}
+		d := a.d
+		d.Reports++
+		if r.Begin.Before(d.First) {
+			d.First = r.Begin
+		}
+		if r.End.After(a.lastEnd) { // the newest report's policy is the one in force
+			a.lastEnd, d.Last, d.Policy, d.Pct = r.End, r.End, r.Policy.P, r.Policy.Pct
+		}
+		a.reporters[r.Org] = true
+		for _, rec := range r.Records {
+			s := a.src[rec.SourceIP]
+			if s == nil {
+				s = &dmarcSource{IP: rec.SourceIP, Ours: ours[rec.SourceIP]}
+				a.src[rec.SourceIP] = s
+				a.srcRep[rec.SourceIP] = map[string]bool{}
+			}
+			a.srcRep[rec.SourceIP][r.Org] = true
+			s.Messages += rec.Count
+			d.Messages += rec.Count
+			if rec.Passed() {
+				s.Passed += rec.Count
+				d.Passed += rec.Count
+			}
+			if rec.DKIM == "pass" {
+				s.DKIMPass += rec.Count
+			}
+			if rec.SPF == "pass" {
+				s.SPFPass += rec.Count
+			}
+			if rec.Disposition == "quarantine" || rec.Disposition == "reject" {
+				s.Rejected += rec.Count
+			}
+			if s.Ours {
+				d.OurMsgs += rec.Count
+				if rec.Passed() {
+					d.OurPassed += rec.Count
+				}
+			}
+		}
+	}
+	var out []dmarcDomain
+	for _, a := range by {
+		d := a.d
+		for ip, s := range a.src {
+			s.Reporters = len(a.srcRep[ip])
+			s.Hint = sourceHint(s, d.Domain)
+			d.Sources = append(d.Sources, *s)
+		}
+		sort.Slice(d.Sources, func(i, j int) bool {
+			fi, fj := d.Sources[i].Messages-d.Sources[i].Passed, d.Sources[j].Messages-d.Sources[j].Passed
+			if fi != fj {
+				return fi > fj
+			}
+			return d.Sources[i].Messages > d.Sources[j].Messages
+		})
+		for r := range a.reporters {
+			d.Reporters = append(d.Reporters, r)
+		}
+		sort.Strings(d.Reporters)
+		d.Advice = dmarcAdvice(d)
+		out = append(out, *d)
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].Domain < out[j].Domain })
+	return out
+}
+
+func sourceHint(s *dmarcSource, domain string) string {
+	failed := s.Messages - s.Passed
+	if failed == 0 {
+		return ""
+	}
+	switch {
+	case s.Ours && s.DKIMPass == 0 && s.SPFPass == 0:
+		return "Your own relay's mail failed both SPF and DKIM alignment. Check that " + domain + " has a published DKIM key here and an SPF record authorizing " + s.IP + "."
+	case s.Ours && s.DKIMPass == 0:
+		return "Mail from your relay only passes through SPF. Create and publish a DKIM key for " + domain + " so it survives forwarding."
+	case s.Ours:
+		return "Some mail from your relay failed. It is probably mail signed or sent under a different domain than the From address."
+	case s.Rejected >= failed:
+		return "Receivers are already quarantining or rejecting this sender. If it isn't yours, your policy is working; if it is, add it to your SPF record and set up DKIM with it."
+	default:
+		return "A sender outside your relays is failing. If it's a service you use (a newsletter or helpdesk tool), add its SPF include and set up its DKIM; if you don't recognise it, someone may be spoofing your domain and a stricter policy would stop that."
+	}
+}
+
+func dmarcAdvice(d *dmarcDomain) []string {
+	var out []string
+	failedOther := 0
+	for _, s := range d.Sources {
+		if !s.Ours {
+			failedOther += s.Messages - s.Passed
+		}
+	}
+	if d.OurMsgs >= dmarcAlertMinMsg && float64(d.OurPassed)/float64(d.OurMsgs) < dmarcAlertBelow {
+		out = append(out, fmt.Sprintf("Only %d%% of the mail from your own relays passed DMARC. Fix this before tightening the policy.", 100*d.OurPassed/d.OurMsgs))
+	}
+	if d.Policy == "none" && d.Messages >= dmarcAlertMinMsg && failedOther == 0 && (d.OurMsgs == 0 || d.OurPassed == d.OurMsgs) {
+		out = append(out, "Everything that sent as "+d.Domain+" passed. You can move the policy from p=none to p=quarantine, then p=reject.")
+	}
+	if d.Policy == "none" && failedOther > 0 {
+		out = append(out, fmt.Sprintf("%d messages from other senders failed. Identify them below before you enforce a policy, or legitimate mail will be blocked.", failedOther))
+	}
+	if d.Policy != "none" && d.Pct > 0 && d.Pct < 100 {
+		out = append(out, fmt.Sprintf("The policy applies to only %d%% of mail (pct=%d). Raise it to 100 once reports look clean.", d.Pct, d.Pct))
+	}
+	return out
+}
+
+// ---------- API ----------
+
+func (s *Server) orgEgressIPsLocked(org string) map[string]bool {
+	ips := map[string]bool{}
+	for _, x := range s.st.MailEgress {
+		if x.OrgID == org {
+			ips[x.SourceIP] = true
+		}
+	}
+	return ips
+}
+
+func (s *Server) dmarcDomainsLocked(org string) map[string]bool {
+	doms := map[string]bool{}
+	for _, k := range s.st.DKIM {
+		if k.OrgID == org {
+			doms[strings.ToLower(k.Domain)] = true
+		}
+	}
+	return doms
+}
+
+// dmarcForOrg summarizes an org's recent reports. It reads the database without holding s.mu.
+func (s *Server) dmarcForOrg(org string, ours map[string]bool) []dmarcDomain {
+	return summarizeDMARC(s.db.dmarcReports(org, time.Now().AddDate(0, 0, -dmarcWindowDays)), ours)
+}
+
+func (s *Server) listDMARC(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	s.mu.Lock()
+	ours := s.orgEgressIPsLocked(c.Org.ID)
+	s.mu.Unlock()
+	out := s.dmarcForOrg(c.Org.ID, ours)
+	if out == nil {
+		out = []dmarcDomain{}
+	}
+	writeJSON(w, 200, out)
+}
+
+// uploadDMARC accepts one or more reports in the request body (a report, or a whole email with one attached).
+func (s *Server) uploadDMARC(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, maxDMARCUpload))
+	if err != nil {
+		fail(w, 413, errors.New("that file is too large to be a DMARC report"))
+		return
+	}
+	rep, err := dmarc.Parse(body)
+	if err != nil {
+		fail(w, 400, err)
+		return
+	}
+	s.mu.Lock()
+	known := s.dmarcDomainsLocked(c.Org.ID)[rep.Domain]
+	s.mu.Unlock()
+	if !known {
+		failf(w, 400, "this report is about %s, which isn't a domain you have a DKIM key for. Create the key first, or check the report is yours.", rep.Domain)
+		return
+	}
+	added, err := s.db.addDMARC(c.Org.ID, rep)
+	if err != nil {
+		fail(w, 500, err)
+		return
+	}
+	if added {
+		s.audit(c.Org.ID, c.Actor, "dmarc.upload", rep.Domain+" from "+rep.Org)
+	}
+	msgs := 0
+	for _, rec := range rep.Records {
+		msgs += rec.Count
+	}
+	writeJSON(w, 200, map[string]any{"domain": rep.Domain, "reporter": rep.Org, "messages": msgs, "duplicate": !added})
+}
+
+func (s *Server) deleteDMARC(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	domain := strings.ToLower(r.PathValue("domain"))
+	if _, err := s.db.deleteDMARCDomain(c.Org.ID, domain); err != nil {
+		fail(w, 500, err)
+		return
+	}
+	s.audit(c.Org.ID, c.Actor, "dmarc.delete", domain)
+	ok(w)
+}
+
+// dmarcProblems derives the alerts from reports: the mail our own relays send
+// as a domain is failing DMARC. Caller holds s.mu; the database is read inside,
+// which takes no server locks.
+func (s *Server) dmarcProblemsLocked(add func(key, org, kind, name, msg string)) {
+	orgs := map[string]bool{}
+	for _, k := range s.st.DKIM {
+		orgs[k.OrgID] = true
+	}
+	for org := range orgs {
+		for _, d := range s.dmarcForOrg(org, s.orgEgressIPsLocked(org)) {
+			if d.OurMsgs >= dmarcAlertMinMsg && float64(d.OurPassed)/float64(d.OurMsgs) < dmarcAlertBelow {
+				add("dmarcrate:"+org+":"+d.Domain, org, "mail", d.Domain,
+					fmt.Sprintf("Only %d%% of the mail your relays sent as %s passed DMARC in the last %d days (%d of %d messages). Open Mail → DMARC reports for the failing senders.",
+						100*d.OurPassed/d.OurMsgs, d.Domain, dmarcWindowDays, d.OurPassed, d.OurMsgs))
+			}
+		}
+	}
+}
+
+// ---------- findings for the dashboard ----------
+
+// mailEgressFindings lists the checklist for one egress. Caller holds s.mu.
+func (s *Server) mailEgressFindingsLocked(x *model.MailEgress, chk model.MailCheck) []Finding {
+	return deliverabilityFindings(x, chk, s.st.DKIM)
+}

dmarc_test.go

@@ -0,0 +1,158 @@
+package hammerhead
+
+import (
+	"bytes"
+	"compress/gzip"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strconv"
+	"strings"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/reef/dmarc"
+	"git.veillax.com/Veillax/reef/model"
+)
+
+func reportXML(domain, id, reporter string, rows ...[4]string) string { // ip, count, dkim, spf
+	var sb strings.Builder
+	sb.WriteString(`<feedback><report_metadata><org_name>` + reporter + `</org_name><report_id>` + id + `</report_id><date_range><begin>` +
+		strconv.FormatInt(time.Now().Add(-26*time.Hour).Unix(), 10) + `</begin><end>` + strconv.FormatInt(time.Now().Add(-2*time.Hour).Unix(), 10) + `</end></date_range></report_metadata><policy_published><domain>` + domain + `</domain><p>none</p><pct>100</pct></policy_published>`)
+	for _, r := range rows {
+		disp := "none"
+		if r[2] != "pass" && r[3] != "pass" {
+			disp = "quarantine"
+		}
+		sb.WriteString(`<record><row><source_ip>` + r[0] + `</source_ip><count>` + r[1] + `</count><policy_evaluated><disposition>` + disp + `</disposition><dkim>` + r[2] + `</dkim><spf>` + r[3] + `</spf></policy_evaluated></row><identifiers><header_from>` + domain + `</header_from></identifiers></record>`)
+	}
+	sb.WriteString(`</feedback>`)
+	return sb.String()
+}
+
+func gzipped(s string) []byte {
+	var b bytes.Buffer
+	w := gzip.NewWriter(&b)
+	w.Write([]byte(s))
+	w.Close()
+	return b.Bytes()
+}
+
+// dmarcFx is an org that sends as example.com from 203.0.113.7.
+func dmarcFx(t *testing.T) (*fx, *orgCtx) {
+	f := newFx(t)
+	f.s.mu.Lock()
+	f.s.st.DKIM = append(f.s.st.DKIM, &model.DKIMKey{ID: "k1", OrgID: f.org, Domain: "example.com", Selector: "s1"})
+	f.s.st.MailEgress = append(f.s.st.MailEgress, &model.MailEgress{ID: "x1", OrgID: f.org, Name: "relay", SiteID: "s1", EdgeID: "e1", SourceIP: "203.0.113.7", HeloName: "mail.example.com", Enabled: true})
+	org := f.s.findOrg(f.org)
+	f.s.mu.Unlock()
+	return f, &orgCtx{Org: *org, Actor: "test"}
+}
+
+func upload(t *testing.T, f *fx, c *orgCtx, body []byte) (int, map[string]any) {
+	w := httptest.NewRecorder()
+	f.s.uploadDMARC(w, httptest.NewRequest("POST", "/", bytes.NewReader(body)), c)
+	var out map[string]any
+	json.Unmarshal(w.Body.Bytes(), &out)
+	return w.Code, out
+}
+
+func TestUploadAndSummarize(t *testing.T) {
+	f, c := dmarcFx(t)
+	code, out := upload(t, f, c, gzipped(reportXML("example.com", "r1", "google.com", [4]string{"203.0.113.7", "30", "pass", "pass"}, [4]string{"198.51.100.9", "5", "fail", "fail"})))
+	if code != 200 || out["messages"] != float64(35) || out["duplicate"] != false {
+		t.Fatalf("%d %v", code, out)
+	}
+	if _, out = upload(t, f, c, []byte(reportXML("example.com", "r1", "google.com"))); out["duplicate"] != true {
+		t.Errorf("the same report twice should be a duplicate: %v", out)
+	}
+	upload(t, f, c, []byte(reportXML("example.com", "r2", "yahoo.com", [4]string{"203.0.113.7", "10", "pass", "fail"})))
+
+	w := httptest.NewRecorder()
+	f.s.listDMARC(w, httptest.NewRequest("GET", "/", nil), c)
+	var doms []dmarcDomain
+	if err := json.Unmarshal(w.Body.Bytes(), &doms); err != nil || len(doms) != 1 {
+		t.Fatalf("%v %s", err, w.Body.String())
+	}
+	d := doms[0]
+	if d.Reports != 2 || d.Messages != 45 || d.Passed != 40 || d.OurMsgs != 40 || d.OurPassed != 40 {
+		t.Errorf("totals: %+v", d)
+	}
+	if len(d.Sources) != 2 || d.Sources[0].IP != "198.51.100.9" || d.Sources[0].Ours || d.Sources[0].Hint == "" {
+		t.Errorf("failing sources should come first with a hint: %+v", d.Sources)
+	}
+	if !strings.Contains(strings.Join(d.Advice, " "), "failed") {
+		t.Errorf("advice should flag the unknown sender: %v", d.Advice)
+	}
+}
+
+func TestUploadRefusals(t *testing.T) {
+	f, c := dmarcFx(t)
+	if code, _ := upload(t, f, c, []byte("garbage")); code != 400 {
+		t.Errorf("garbage: %d", code)
+	}
+	code, out := upload(t, f, c, []byte(reportXML("someone-else.org", "r9", "google.com", [4]string{"1.2.3.4", "1", "pass", "pass"})))
+	if code != 400 || !strings.Contains(out["error"].(string), "someone-else.org") {
+		t.Errorf("a domain the org doesn't send from: %d %v", code, out)
+	}
+	big := make([]byte, maxDMARCUpload+1)
+	w := httptest.NewRecorder()
+	f.s.uploadDMARC(w, httptest.NewRequest("POST", "/", bytes.NewReader(big)), c)
+	if w.Code != http.StatusRequestEntityTooLarge {
+		t.Errorf("oversize: %d", w.Code)
+	}
+}
+
+func TestReportsStayInTheirOrg(t *testing.T) {
+	f, c := dmarcFx(t)
+	upload(t, f, c, []byte(reportXML("example.com", "r1", "google.com", [4]string{"203.0.113.7", "5", "pass", "pass"})))
+	if n := len(f.s.db.dmarcReports("another-org", time.Time{})); n != 0 {
+		t.Errorf("another org sees %d reports", n)
+	}
+	if n, _ := f.s.db.deleteDMARCDomain(f.org, "example.com"); n != 1 {
+		t.Errorf("delete removed %d", n)
+	}
+}
+
+func TestSummarizeAdvice(t *testing.T) {
+	rep := func(p string, pct int, rows ...dmarc.Record) *dmarc.Report {
+		return &dmarc.Report{Org: "g", ReportID: "1", Domain: "example.com", Begin: time.Unix(1, 0), End: time.Unix(2, 0), Policy: dmarc.Policy{P: p, Pct: pct}, Records: rows}
+	}
+	ours := map[string]bool{"203.0.113.7": true}
+	pass := dmarc.Record{SourceIP: "203.0.113.7", Count: 50, DKIM: "pass", SPF: "pass"}
+
+	got := summarizeDMARC([]*dmarc.Report{rep("none", 100, pass)}, ours)[0]
+	if !strings.Contains(strings.Join(got.Advice, " "), "p=quarantine") {
+		t.Errorf("a clean domain at p=none should be told to tighten: %v", got.Advice)
+	}
+	bad := dmarc.Record{SourceIP: "203.0.113.7", Count: 50, DKIM: "fail", SPF: "fail", Disposition: "quarantine"}
+	got = summarizeDMARC([]*dmarc.Report{rep("quarantine", 50, pass, bad)}, ours)[0]
+	joined := strings.Join(got.Advice, " ")
+	if !strings.Contains(joined, "50%") || !strings.Contains(joined, "pct=50") {
+		t.Errorf("advice: %v", got.Advice)
+	}
+	got = summarizeDMARC([]*dmarc.Report{rep("quarantine", 100, bad)}, ours)[0]
+	if got.Sources[0].Hint == "" || !strings.Contains(got.Sources[0].Hint, "SPF and DKIM") {
+		t.Errorf("own relay failing both: %q", got.Sources[0].Hint)
+	}
+}
+
+func TestOwnMailFailingDMARCRaisesAnAlert(t *testing.T) {
+	f, c := dmarcFx(t)
+	upload(t, f, c, []byte(reportXML("example.com", "r1", "google.com", [4]string{"203.0.113.7", "40", "fail", "fail"}, [4]string{"203.0.113.7", "10", "pass", "pass"})))
+	f.s.mu.Lock()
+	probs := f.s.problemsLocked(f.base)
+	f.s.mu.Unlock()
+	p, ok := probs["dmarcrate:"+f.org+":example.com"]
+	if !ok || !strings.Contains(p.msg, "20%") {
+		t.Fatalf("expected a DMARC pass-rate alert, got %v", probs)
+	}
+	// a healthy domain raises none
+	upload(t, f, c, []byte(reportXML("example.com", "r2", "yahoo.com", [4]string{"203.0.113.7", "500", "pass", "pass"})))
+	f.s.mu.Lock()
+	probs = f.s.problemsLocked(f.base)
+	f.s.mu.Unlock()
+	if _, ok := probs["dmarcrate:"+f.org+":example.com"]; ok {
+		t.Errorf("alert should clear once the pass rate recovers: %v", probs)
+	}
+}

docs/api-automation.md

@@ -0,0 +1,37 @@
+# Scripting Hammerhead: API audit
+
+Everything the dashboard does goes through the JSON API (`server.go` lists the routes). Callers
+authenticate with the `hh_session` cookie or `Authorization: Bearer hhk_<id>_<secret>`.
+
+## Credentials
+
+| Credential | Reaches | Made at |
+|---|---|---|
+| Organization key | that organization's `/api/orgs/{org}/...` routes (admin rights), `GET /api/orgs`, `GET /api/edges` (edges the org may use) | `POST /api/orgs/{org}/apikeys` |
+| Server key | every organization, plus the server-admin routes: edges, `/api/admin/alerts`, users, settings, `POST /api/orgs` | `POST /api/admin/apikeys` (signed-in server admin only) |
+
+The secret is shown once, in the `key` field of the create response. A key cannot mint keys at server
+level, and a server key has no user, so `/api/me` and other per-user routes refuse it.
+
+## Gaps found, and what was done
+
+| Gap | Resolution |
+|---|---|
+| Edges, server alerts, `POST /api/orgs` required a browser session (`withServerAdmin` refused keys) | server keys, accepted by `withServerAdmin` |
+| No way to mint an unattended server credential | `GET/POST/DELETE /api/admin/apikeys` |
+| `GET /api/edges` was session-only | accepts keys; an org key sees the edges its org may use |
+| No way to list orgs with a key | `GET /api/orgs` |
+| Key creation did not return the key's ID | create response now has `id` as well as `key` |
+| Creating an alert channel returned only `{ok:true}` | response now includes the channel `id` |
+| `createOrg` added the caller as a member | skipped for server keys (they have no user) |
+
+## Known remaining gaps (not blocking scripting)
+
+- No per-resource `GET`: sites, domains, ports and alert channels are read from `GET /api/orgs/{org}/state` and
+  `GET /api/orgs/{org}/alerts`. Clients filter by ID.
+- Alert channel secrets (webhook URL, Sandbar key) are never returned, so a client can't detect drift in them; a
+  blank value on `PUT` keeps the stored one.
+- Keys are all-powerful within their scope (no read-only or per-resource scopes), and have no expiry.
+- `PUT` on ports replaces the whole object; omitted fields reset.
+- Site tokens are returned on create and, for admins, in `state`.
+- Deleting an edge or site cascades silently (its ports and domains go with it, as do ports left with no target), so Terraform plans should not rely on dependent resources surviving.

e2e/expose_test.go

@@ -0,0 +1,96 @@
+package e2e
+
+import (
+	"context"
+	"net"
+	"strconv"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/bamboo"
+	"git.veillax.com/Veillax/blacktip"
+)
+
+type exposed struct {
+	ID, Host, Hostname, URL, SiteToken string
+	Port                               int
+}
+
+// createExpose retries until an edge is online and the call succeeds.
+func (w *world) createExpose(c *client, in map[string]any) exposed {
+	w.t.Helper()
+	var out exposed
+	for i := 0; i < 40; i++ {
+		if code, _ := c.do("POST", "/api/orgs/-/expose", in, &out); code == 200 {
+			return out
+		}
+		time.Sleep(150 * time.Millisecond)
+	}
+	c.must("POST", "/api/orgs/-/expose", in, &out)
+	return out
+}
+
+func (w *world) agentFor(e exposed) context.CancelFunc {
+	ctx, cancel := context.WithCancel(context.Background())
+	w.t.Cleanup(cancel)
+	b, err := bamboo.New(bamboo.Config{Hammerhead: w.ts.URL, Token: e.SiteToken, DataDir: w.t.TempDir()})
+	if err != nil {
+		w.t.Fatal(err)
+	}
+	go b.Run(ctx)
+	return cancel
+}
+
+// An ephemeral exposure carries traffic through the tunnel, and is gone after DELETE.
+func TestEphemeralExpose(t *testing.T) {
+	w := setup(t, blacktip.TraefikConfig{})
+	var key struct{ Key string }
+	w.admin.must("POST", "/api/orgs/"+w.org+"/apikeys", map[string]string{"name": "shark"}, &key)
+	kc := newClient(t, w.ts.URL)
+	kc.key = key.Key
+	w.admin.must("POST", "/api/orgs/"+w.org+"/domains", map[string]any{"name": "example.test", "edgeId": w.edge}, nil)
+
+	// ----- tcp -----
+	svc := tcpBackend(t, "SVC")
+	e := w.createExpose(kc, map[string]any{"proto": "tcp", "targetPort": svc, "ttlSeconds": 60})
+	stop := w.agentFor(e)
+	addr := net.JoinHostPort("127.0.0.1", strconv.Itoa(e.Port))
+	if got := tcpAsk(t, addr, "hi"); got != "SVC:hi" {
+		t.Fatalf("through the tunnel: %q", got)
+	}
+	kc.must("POST", "/api/orgs/-/expose/"+e.ID+"/renew", nil, nil)
+	kc.must("DELETE", "/api/orgs/-/expose/"+e.ID, nil, nil)
+	stop()
+	closed := false
+	for i := 0; i < 40 && !closed; i++ {
+		c, err := net.DialTimeout("tcp", addr, 300*time.Millisecond)
+		if err != nil {
+			closed = true
+			break
+		}
+		c.Close()
+		time.Sleep(150 * time.Millisecond)
+	}
+	if !closed {
+		t.Error("the port is still open after DELETE")
+	}
+	kc.expect(404, "GET", "/api/orgs/-/expose/"+e.ID, nil)
+
+	// ----- http, with a password -----
+	web := webBackend(t, "app", 200)
+	port := freePort(t)
+	h := w.createExpose(kc, map[string]any{"proto": "http", "name": "demo", "port": port, "targetPort": web,
+		"auth": map[string]any{"password": "hunter22"}})
+	w.agentFor(h)
+	_, get := edgeClient("http", port)
+	var r webResp
+	for i := 0; i < 60; i++ {
+		if r = get("GET", h.Hostname, "/x"); r.status == 302 {
+			break
+		}
+		time.Sleep(150 * time.Millisecond)
+	}
+	if r.status != 302 {
+		t.Fatalf("protected exposure should send visitors to the login, got %+v", r)
+	}
+}

e2e/ssh_test.go

@@ -0,0 +1,108 @@
+package e2e
+
+import (
+	"context"
+	"crypto/ed25519"
+	"crypto/rand"
+	"fmt"
+	"net"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/bamboo/sshd"
+	"git.veillax.com/Veillax/blacktip"
+	"git.veillax.com/Veillax/reef/model"
+	"golang.org/x/crypto/ssh"
+)
+
+func newSSHSigner(t *testing.T) ssh.Signer {
+	t.Helper()
+	_, priv, err := ed25519.GenerateKey(rand.Reader)
+	if err != nil {
+		t.Fatal(err)
+	}
+	s, err := ssh.NewSignerFromKey(priv)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return s
+}
+
+// A TCP port on the edge whose target is Bamboo's SSH server at the site reaches that server
+// with an ordinary ssh client: the path is client, edge, WireGuard tunnel, Bamboo, sshd.
+func TestSSHThroughTheEdgeToBamboosServer(t *testing.T) {
+	w := setup(t, blacktip.TraefikConfig{})
+	dir := t.TempDir()
+	signer := newSSHSigner(t)
+	keys := filepath.Join(dir, "authorized_keys")
+	if err := os.WriteFile(keys, ssh.MarshalAuthorizedKey(signer.PublicKey()), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	srv, err := sshd.New(sshd.Config{AuthorizedKeys: keys, HostKey: filepath.Join(dir, "ssh_host_ed25519_key")})
+	if err != nil {
+		t.Fatal(err)
+	}
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		t.Fatal(err)
+	}
+	ctx, cancel := context.WithCancel(context.Background())
+	t.Cleanup(cancel)
+	go srv.Serve(ctx, ln)
+
+	edgePort := freePort(t)
+	w.open(model.Mapping{Proto: model.ProtoTCP, Port: edgePort, Name: "ssh", Targets: tgt(w.site, ln.Addr().(*net.TCPAddr).Port)})
+	addr := "127.0.0.1:" + strconv.Itoa(edgePort)
+
+	connect := func(auth ssh.AuthMethod) (*ssh.Client, error) {
+		return ssh.Dial("tcp", addr, &ssh.ClientConfig{User: "alex", Auth: []ssh.AuthMethod{auth}, Timeout: 5 * time.Second,
+			// the key the client sees through the tunnel must be Bamboo's, not something in between
+			HostKeyCallback: func(_ string, _ net.Addr, k ssh.PublicKey) error {
+				if got := ssh.FingerprintSHA256(k); got != srv.Fingerprint() {
+					return fmt.Errorf("host key %s is not the Bamboo server's %s", got, srv.Fingerprint())
+				}
+				return nil
+			}})
+	}
+	var client *ssh.Client
+	var last error
+	for i := 0; i < 60 && client == nil; i++ {
+		if client, last = connect(ssh.PublicKeys(signer)); client == nil {
+			time.Sleep(250 * time.Millisecond)
+		}
+	}
+	if client == nil {
+		t.Fatalf("could not sign in through the edge: %v", last)
+	}
+	defer client.Close()
+
+	if sshd.Supported {
+		sess, err := client.NewSession()
+		if err != nil {
+			t.Fatal(err)
+		}
+		out, err := sess.Output("echo through-the-tunnel")
+		sess.Close()
+		if err != nil || strings.TrimSpace(string(out)) != "through-the-tunnel" {
+			t.Fatalf("running a command: %q, %v", out, err)
+		}
+	}
+
+	// the same narrow rules hold through the tunnel
+	if c, err := connect(ssh.PublicKeys(newSSHSigner(t))); err == nil {
+		c.Close()
+		t.Error("a key that is not in authorized_keys signed in")
+	}
+	if c, err := connect(ssh.Password("hunter2")); err == nil {
+		c.Close()
+		t.Error("a password signed in")
+	}
+	if c, err := client.Dial("tcp", "127.0.0.1:1"); err == nil {
+		c.Close()
+		t.Error("port forwarding through the SSH server is allowed")
+	}
+}

e2e/terminal_relay_test.go

@@ -0,0 +1,193 @@
+package e2e
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"net"
+	"net/url"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/bamboo"
+	"git.veillax.com/Veillax/bamboo/sshd"
+	"git.veillax.com/Veillax/blacktip"
+	"git.veillax.com/Veillax/reef/model"
+	"golang.org/x/crypto/ssh"
+	"golang.org/x/net/websocket"
+)
+
+// With Bamboo reporting its SSH server, the terminal works with no SSH port anywhere: Hammerhead
+// asks the agent to call back and signs in over that connection.
+func TestWebTerminalWithoutAnyOpenPort(t *testing.T) {
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		t.Fatal(err)
+	}
+	siteCfgHook = func(c *bamboo.Config) { c.SSH = ln.Addr().String() } // what bamboo -ssh tells the agent
+	w := setup(t, blacktip.TraefikConfig{})
+	siteCfgHook = nil
+	base := "/api/orgs/" + w.org
+	u, _ := url.Parse(w.ts.URL)
+
+	var key struct{ PublicKey string }
+	w.admin.must("GET", base+"/terminal", nil, &key)
+	dir := t.TempDir()
+	keys := filepath.Join(dir, "authorized_keys")
+	if err := os.WriteFile(keys, []byte(key.PublicKey+"\n"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	srv, err := sshd.New(sshd.Config{AuthorizedKeys: keys, HostKey: filepath.Join(dir, "ssh_host_ed25519_key")})
+	if err != nil {
+		t.Fatal(err)
+	}
+	ctx, cancel := context.WithCancel(context.Background())
+	t.Cleanup(cancel)
+	go srv.Serve(ctx, ln)
+
+	// the agent's next report tells Hammerhead where its SSH server is
+	var st struct{ Sites []model.Site }
+	for i := 0; i < 60; i++ {
+		w.admin.must("GET", base+"/state", nil, &st)
+		for _, s := range st.Sites {
+			if s.ID == w.site && s.SSH != "" {
+				i = 60
+			}
+		}
+		if i < 60 {
+			time.Sleep(200 * time.Millisecond)
+		}
+	}
+	var site model.Site
+	for _, s := range st.Sites {
+		if s.ID == w.site {
+			site = s
+		}
+	}
+	if site.SSH != ln.Addr().String() || !site.SSHRelay {
+		t.Fatalf("the site did not report its SSH server: %+v", site)
+	}
+
+	wsURL := "ws" + strings.TrimPrefix(w.ts.URL, "http") + base + "/sites/" + w.site + "/terminal?cols=100&rows=30"
+	dial := func() *websocket.Conn {
+		t.Helper()
+		cfg, err := websocket.NewConfig(wsURL, w.ts.URL)
+		if err != nil {
+			t.Fatal(err)
+		}
+		var ck []string
+		for _, c := range w.admin.c.Jar.Cookies(u) {
+			ck = append(ck, c.Name+"="+c.Value)
+		}
+		cfg.Header.Set("Cookie", strings.Join(ck, "; "))
+		ws, err := websocket.DialConfig(cfg)
+		if err != nil {
+			t.Fatal(err)
+		}
+		return ws
+	}
+	first := func() (m termMsg) {
+		t.Helper()
+		ws := dial()
+		defer ws.Close()
+		for {
+			ws.SetReadDeadline(time.Now().Add(20 * time.Second))
+			var f []byte
+			if err := websocket.Message.Receive(ws, &f); err != nil {
+				t.Fatalf("waiting for the terminal: %v", err)
+			}
+			if len(f) > 0 && f[0] == '{' {
+				return parseTermMsg(f)
+			}
+		}
+	}
+
+	m := first()
+	if sshd.Supported {
+		if m.Type != "ready" {
+			t.Fatalf("opening the terminal with no port: %+v", m)
+		}
+		ws := dial()
+		defer ws.Close()
+		// $((40+2)) is expanded by the shell, so "relay-42" can only come from the command running
+		if err := websocket.Message.Send(ws, []byte("echo relay-$((40+2))\n")); err != nil {
+			t.Fatal(err)
+		}
+		var out bytes.Buffer
+		for !strings.Contains(out.String(), "relay-42\r\n") {
+			ws.SetReadDeadline(time.Now().Add(15 * time.Second))
+			var f []byte
+			if err := websocket.Message.Receive(ws, &f); err != nil {
+				t.Fatalf("waiting for the command's output (got %q): %v", out.String(), err)
+			}
+			out.Write(f)
+		}
+	} else if m.Type != "error" || !strings.Contains(m.Message, "terminal") {
+		// this platform can't run shells, but asking for one proves the call-back and Hammerhead's key worked
+		t.Fatalf("with no port, on a platform without shells: %+v", m)
+	}
+
+	// it really did not need a port
+	var maps struct{ Mappings []model.Mapping }
+	w.admin.must("GET", base+"/state", nil, &maps)
+	if len(maps.Mappings) != 0 {
+		t.Errorf("the test opened a port after all: %+v", maps.Mappings)
+	}
+
+	// a machine with no shell (the published container image has none) says why, instead of the
+	// terminal just disconnecting: the shell's error output reaches the browser, then the exit code
+	if sshd.Supported {
+		t.Setenv("SHELL", "/nonexistent/shell")
+		ws := dial()
+		defer ws.Close()
+		var out bytes.Buffer
+		var exit termMsg
+		for exit.Type != "exit" {
+			ws.SetReadDeadline(time.Now().Add(15 * time.Second))
+			var f []byte
+			if err := websocket.Message.Receive(ws, &f); err != nil {
+				t.Fatalf("waiting for the shell to end (got %q): %v", out.String(), err)
+			}
+			if len(f) > 0 && f[0] == '{' {
+				if m := parseTermMsg(f); m.Type == "exit" {
+					exit = m
+				}
+				continue
+			}
+			out.Write(f)
+		}
+		if exit.Code != 127 || !strings.Contains(out.String(), "no shell to run") {
+			t.Errorf("a missing shell: exit %d, output %q", exit.Code, out.String())
+		}
+	}
+
+	// the SSH server still decides who gets in
+	other := newSSHSigner(t)
+	if err := os.WriteFile(keys, ssh.MarshalAuthorizedKey(other.PublicKey()), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	if m := first(); m.Type != "error" || !strings.Contains(m.Message, "authorized_keys") {
+		t.Errorf("after removing Hammerhead's key: %+v", m)
+	}
+
+	// and a stopped SSH server is reported at once rather than after a timeout
+	cancel()
+	ln.Close()
+	time.Sleep(300 * time.Millisecond)
+	start := time.Now()
+	if m := first(); m.Type != "error" || !strings.Contains(m.Message, "SSH server") {
+		t.Errorf("with the SSH server stopped: %+v", m)
+	}
+	if time.Since(start) > 8*time.Second {
+		t.Errorf("a stopped SSH server took %s to be reported", time.Since(start))
+	}
+}
+
+func parseTermMsg(b []byte) termMsg {
+	var m termMsg
+	json.Unmarshal(b, &m)
+	return m
+}

e2e/terminal_test.go

@@ -0,0 +1,196 @@
+package e2e
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"net"
+	"net/url"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/bamboo/sshd"
+	"git.veillax.com/Veillax/blacktip"
+	"golang.org/x/crypto/ssh"
+	"golang.org/x/net/websocket"
+)
+
+type termMsg struct {
+	Type, Message  string
+	Code           int
+	HostKeyChanged bool
+}
+
+// The dashboard's terminal: browser, Hammerhead (as the SSH client), the SSH port on the edge,
+// the WireGuard tunnel, Bamboo, and Bamboo's SSH server.
+func TestWebTerminalThroughTheEdgeToBamboosServer(t *testing.T) {
+	w := setup(t, blacktip.TraefikConfig{})
+	base := "/api/orgs/" + w.org
+	u, _ := url.Parse(w.ts.URL)
+	wsURL := "ws" + strings.TrimPrefix(w.ts.URL, "http") + base + "/sites/" + w.site + "/terminal?cols=100&rows=30"
+
+	// dial opens the terminal's WebSocket as the administrator, or as nobody
+	dial := func(origin string, signedIn bool) (*websocket.Conn, error) {
+		cfg, err := websocket.NewConfig(wsURL, origin)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if signedIn {
+			var ck []string
+			for _, c := range w.admin.c.Jar.Cookies(u) {
+				ck = append(ck, c.Name+"="+c.Value)
+			}
+			cfg.Header.Set("Cookie", strings.Join(ck, "; "))
+		}
+		return websocket.DialConfig(cfg)
+	}
+	// control reads until the next text frame: the terminal's own messages ("ready", "error", "exit")
+	control := func(ws *websocket.Conn) termMsg {
+		t.Helper()
+		for {
+			ws.SetReadDeadline(time.Now().Add(15 * time.Second))
+			var f []byte
+			if err := websocket.Message.Receive(ws, &f); err != nil {
+				t.Fatalf("waiting for the terminal: %v", err)
+			}
+			var m termMsg
+			if json.Unmarshal(f, &m) == nil && m.Type != "" && f[0] == '{' {
+				return m
+			}
+		}
+	}
+	// first says how a new terminal begins
+	first := func() termMsg {
+		t.Helper()
+		ws, err := dial(w.ts.URL, true)
+		if err != nil {
+			t.Fatal(err)
+		}
+		defer ws.Close()
+		return control(ws)
+	}
+
+	// nobody signed in, or a page on another site, cannot open it
+	if ws, err := dial(w.ts.URL, false); err == nil {
+		ws.Close()
+		t.Error("the terminal opened for someone who is not signed in")
+	}
+	if ws, err := dial("https://evil.example", true); err == nil {
+		ws.Close()
+		t.Error("a page on another site opened the terminal with the administrator's cookie")
+	}
+	anon := newClient(t, w.ts.URL)
+	anon.expect(401, "GET", base+"/terminal", nil)
+
+	// this Bamboo has no SSH server of its own to call back to, so before a port exists the message says to start it
+	if m := first(); m.Type != "error" || !strings.Contains(m.Message, "-ssh") {
+		t.Errorf("before opening an SSH port: %+v", m)
+	}
+
+	// Hammerhead's public key, which goes in the site's authorized_keys
+	var key struct{ PublicKey, Fingerprint string }
+	w.admin.must("GET", base+"/terminal", nil, &key)
+	if !strings.HasPrefix(key.PublicKey, "ssh-ed25519 ") || strings.Contains(key.PublicKey, "PRIVATE") {
+		t.Fatalf("public key: %q", key.PublicKey)
+	}
+	var again struct{ PublicKey string }
+	w.admin.must("GET", base+"/terminal", nil, &again)
+	if again.PublicKey != key.PublicKey {
+		t.Fatal("the key changed between requests")
+	}
+	dir := t.TempDir()
+	keys := filepath.Join(dir, "authorized_keys")
+	if err := os.WriteFile(keys, []byte(key.PublicKey+"\n"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	srv, err := sshd.New(sshd.Config{AuthorizedKeys: keys, HostKey: filepath.Join(dir, "ssh_host_ed25519_key")})
+	if err != nil {
+		t.Fatal(err)
+	}
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		t.Fatal(err)
+	}
+	ctx, cancel := context.WithCancel(context.Background())
+	t.Cleanup(cancel)
+	go srv.Serve(ctx, ln)
+
+	edgePort := freePort(t)
+	w.admin.must("POST", base+"/mappings", map[string]any{"name": "ssh", "proto": "tcp", "edgeId": w.edge, "bindIp": "127.0.0.1", "port": edgePort,
+		"enabled": true, "tags": []string{"ssh"}, "targets": tgt(w.site, ln.Addr().(*net.TCPAddr).Port)}, nil)
+
+	// wait for the path through the tunnel to come up
+	var m termMsg
+	for i := 0; i < 60; i++ {
+		if m = first(); !strings.Contains(m.Message, "reach the site") {
+			break
+		}
+		time.Sleep(250 * time.Millisecond)
+	}
+	if sshd.Supported {
+		if m.Type != "ready" {
+			t.Fatalf("opening the terminal: %+v", m)
+		}
+	} else if m.Type != "error" || !strings.Contains(m.Message, "terminal") {
+		// this platform can't run shells, but asking for one proves Hammerhead's key was accepted
+		t.Fatalf("on a platform without shells: %+v", m)
+	}
+
+	if sshd.Supported {
+		ws, err := dial(w.ts.URL, true)
+		if err != nil {
+			t.Fatal(err)
+		}
+		defer ws.Close()
+		if m := control(ws); m.Type != "ready" {
+			t.Fatalf("second terminal: %+v", m)
+		}
+		// $((40+2)) is expanded by the shell, so "terminal-42" can only come from the command running
+		if err := websocket.Message.Send(ws, []byte("echo terminal-$((40+2))\n")); err != nil {
+			t.Fatal(err)
+		}
+		var out strings.Builder
+		for !strings.Contains(out.String(), "terminal-42\r\n") {
+			ws.SetReadDeadline(time.Now().Add(10 * time.Second))
+			var f []byte
+			if err := websocket.Message.Receive(ws, &f); err != nil {
+				t.Fatalf("waiting for the command's output (got %q): %v", out.String(), err)
+			}
+			out.Write(f)
+		}
+		websocket.Message.Send(ws, `{"type":"resize","cols":120,"rows":40}`)
+		websocket.Message.Send(ws, []byte("exit\n"))
+		var sawExit bool
+		for !sawExit {
+			ws.SetReadDeadline(time.Now().Add(10 * time.Second))
+			var f []byte
+			if err := websocket.Message.Receive(ws, &f); err != nil {
+				break
+			}
+			sawExit = bytes.Contains(f, []byte(`"type":"exit"`))
+		}
+		if !sawExit {
+			t.Error("the browser was not told the shell ended")
+		}
+	}
+
+	// taking Hammerhead's key out of authorized_keys is refused with a message that says what to fix
+	other := newSSHSigner(t)
+	if err := os.WriteFile(keys, ssh.MarshalAuthorizedKey(other.PublicKey()), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	if m := first(); m.Type != "error" || !strings.Contains(m.Message, "authorized_keys") {
+		t.Errorf("after removing the key: %+v", m)
+	}
+
+	// a site whose SSH server has stopped is reported, not left hanging
+	cancel()
+	ln.Close()
+	time.Sleep(300 * time.Millisecond)
+	if m := first(); m.Type != "error" || !strings.Contains(m.Message, "reach the site") {
+		t.Errorf("with the SSH server stopped: %+v", m)
+	}
+}

e2e/traffic_test.go

@@ -0,0 +1,80 @@
+package e2e
+
+import (
+	"strconv"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/blacktip"
+	"git.veillax.com/Veillax/reef/model"
+)
+
+// Traffic through an edge shows up in Hammerhead's charts, with the client address named.
+func TestTrafficAnalyticsFlow(t *testing.T) {
+	w := setup(t, blacktip.TraefikConfig{})
+	target := freeRun(t, 1)
+	pub := freeRun(t, 1)
+	if pub == target {
+		pub = freeRun(t, 2) + 1
+	}
+	tcpOn(t, target)
+	var m model.Mapping
+	if code, raw := w.admin.do("POST", "/api/orgs/"+w.org+"/mappings", map[string]any{"proto": "tcp", "edgeId": w.edge, "bindIp": "127.0.0.1", "port": pub,
+		"targets": tgt(w.local, target), "enabled": true}, &m); code != 200 {
+		t.Fatalf("open: %d %s", code, raw)
+	}
+	for i := 0; i < 3; i++ {
+		if got := tcpAsk(t, "127.0.0.1:"+strconv.Itoa(pub), "hello"); got == "" {
+			t.Fatal("no answer through the edge")
+		}
+	}
+
+	type detail struct {
+		Total   struct{ Conns, BytesIn, BytesOut int64 } `json:"total"`
+		Points  []struct{ Conns int64 }                  `json:"points"`
+		Sources []struct {
+			IP    string
+			Conns int64
+		} `json:"sources"`
+	}
+	var d detail
+	for i := 0; ; i++ { // the edge reports every 20 seconds
+		d = detail{}
+		if code, _ := w.admin.do("GET", "/api/orgs/"+w.org+"/traffic/"+m.ID+"?range=1h", nil, &d); code == 200 && d.Total.Conns >= 3 {
+			break
+		}
+		if i >= 450 {
+			t.Fatal("timed out waiting for the edge to report the traffic")
+		}
+		time.Sleep(100 * time.Millisecond)
+	}
+	if d.Total.BytesIn < 15 || d.Total.BytesOut == 0 || len(d.Points) != 60 {
+		t.Errorf("totals: %+v with %d points", d.Total, len(d.Points))
+	}
+	if len(d.Sources) != 1 || d.Sources[0].IP != "127.0.0.1" || d.Sources[0].Conns < 3 {
+		t.Errorf("sources: %+v", d.Sources)
+	}
+
+	var all detail
+	if code, raw := w.admin.do("GET", "/api/orgs/"+w.org+"/traffic/all?range=24h", nil, &all); code != 200 || all.Total.Conns < 3 || len(all.Points) != 24 {
+		t.Errorf("organization total: %d %s", code, raw)
+	}
+	if code, _ := w.admin.do("GET", "/api/orgs/"+w.org+"/traffic/"+m.ID+"?range=5y", nil, nil); code != 400 {
+		t.Errorf("bad range: %d", code)
+	}
+	if code, _ := w.admin.do("GET", "/api/orgs/"+w.org+"/traffic/nope", nil, nil); code != 404 {
+		t.Errorf("unknown port: %d", code)
+	}
+
+	alerts := "/api/orgs/" + w.org + "/alerts"
+	if code, _ := w.admin.do("PUT", alerts+"/traffic", map[string]any{"spikeFactor": 1}, nil); code != 400 {
+		t.Errorf("a spike of 1x was accepted: %d", code)
+	}
+	if code, raw := w.admin.do("PUT", alerts+"/traffic", map[string]any{"spikeFactor": 5, "dailyMb": 2048}, nil); code != 200 {
+		t.Fatalf("save traffic alerts: %d %s", code, raw)
+	}
+	var ov struct{ Settings map[string]int }
+	if code, _ := w.admin.do("GET", alerts, nil, &ov); code != 200 || ov.Settings["spikeFactor"] != 5 || ov.Settings["dailyMb"] != 2048 || ov.Settings["debounceMin"] != 2 {
+		t.Errorf("settings after save: %d %+v", code, ov.Settings)
+	}
+}

expose.go

@@ -0,0 +1,389 @@
+package hammerhead
+
+import (
+	"errors"
+	"fmt"
+	"math/rand/v2"
+	"net"
+	"net/http"
+	"regexp"
+	"strings"
+	"time"
+
+	"git.veillax.com/Veillax/reef/model"
+)
+
+// Ephemeral exposures back `shark expose`: a short-lived site plus one port or hostname that
+// vanish unless the client keeps renewing them. Both records carry exposeTag so they show up
+// in the dashboard as temporary, and so a restart (which forgets the deadlines) can sweep them.
+
+const (
+	exposeTag        = "ephemeral"
+	defaultExposeTTL = 5 * time.Minute
+	minExposeTTL     = 30 * time.Second
+	maxExposeTTL     = 24 * time.Hour
+	exposePortMin    = 20000
+	exposePortMax    = 29999
+	maxExposePerOrg  = 20
+)
+
+type exposeIn struct {
+	Proto      string `json:"proto"` // http, https or tcp
+	Name       string `json:"name"`  // label for the subdomain and the dashboard
+	EdgeID     string `json:"edgeId"`
+	Hostname   string `json:"hostname"` // http: full name under one of the org's domains; default <name>-<random>.<domain>
+	Port       int    `json:"port"`     // public port; default 80/443 for web, a free one in the ephemeral range for tcp
+	TargetHost string `json:"targetHost"`
+	TargetPort int    `json:"targetPort"`
+	TTL        int    `json:"ttlSeconds"`
+	Auth       *struct {
+		Password string `json:"password"`
+		Pin      string `json:"pin"`
+		SSO      bool   `json:"sso"`
+	} `json:"auth"`
+}
+
+type exposeOut struct {
+	ID         string    `json:"id"`
+	Proto      string    `json:"proto"`
+	URL        string    `json:"url,omitempty"` // http: the address to open
+	Host       string    `json:"host"`          // public host (tcp: where to connect)
+	Port       int       `json:"port"`
+	Hostname   string    `json:"hostname,omitempty"`
+	SiteToken  string    `json:"siteToken,omitempty"`  // only in the create response: what Bamboo dials in with
+	Hammerhead string    `json:"hammerhead,omitempty"` // only in the create response
+	Protected  bool      `json:"protected"`
+	ExpiresAt  time.Time `json:"expiresAt"`
+	Live       bool      `json:"live"` // the edge reports the port listening
+	Error      string    `json:"error,omitempty"`
+}
+
+var slugBad = regexp.MustCompile(`[^a-z0-9]+`)
+
+func exposeSlug(n string) string {
+	n = strings.Trim(slugBad.ReplaceAllString(strings.ToLower(n), "-"), "-")
+	if len(n) > 20 {
+		n = strings.Trim(n[:20], "-")
+	}
+	if n == "" {
+		n = "expose"
+	}
+	return n
+}
+
+func exposeTTL(sec int) (time.Duration, error) {
+	if sec == 0 {
+		return defaultExposeTTL, nil
+	}
+	d := time.Duration(sec) * time.Second
+	if d < minExposeTTL || d > maxExposeTTL {
+		return 0, fmt.Errorf("ttlSeconds must be between %d and %d", int(minExposeTTL.Seconds()), int(maxExposeTTL.Seconds()))
+	}
+	return d, nil
+}
+
+func isExposed(tags []string) bool {
+	for _, t := range tags {
+		if t == exposeTag {
+			return true
+		}
+	}
+	return false
+}
+
+// edgeHost is the public host of an edge's WireGuard endpoint, which is where its ports are reachable.
+func edgeHost(e *model.Edge) string {
+	h, _, err := net.SplitHostPort(e.TunnelAddr)
+	if err != nil {
+		return e.TunnelAddr
+	}
+	return h
+}
+
+// exposeEdges lists the edges an exposure may use: the named one, or every online edge the org may use.
+func (s *Server) exposeEdges(orgID, want string) []*model.Edge {
+	var out []*model.Edge
+	for _, e := range s.st.Edges {
+		if (want != "" && e.ID != want) || !s.edgeAllowedFor(e, orgID) {
+			continue
+		}
+		if want != "" || s.edgeOnline(e) {
+			out = append(out, e)
+		}
+	}
+	return out
+}
+
+func (s *Server) exposeView(m *model.Mapping, e *model.Edge) exposeOut {
+	st, known := s.status[m.ID]
+	o := exposeOut{ID: m.ID, Proto: m.Proto, Hostname: m.Hostname, Port: m.Port, Protected: m.Auth.Protected(),
+		ExpiresAt: s.expose[m.ID], URL: publicURL(m), Live: known && st.Error == "", Error: st.Error}
+	if e != nil {
+		o.Host = edgeHost(e)
+	}
+	if m.Hostname != "" {
+		o.Host = m.Hostname
+	}
+	return o
+}
+
+func (s *Server) createExpose(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	var in exposeIn
+	if !decode(w, r, &in) {
+		return
+	}
+	ttl, err := exposeTTL(in.TTL)
+	if err != nil {
+		fail(w, 400, err)
+		return
+	}
+	if in.Proto != model.ProtoHTTP && in.Proto != model.ProtoHTTPS && in.Proto != model.ProtoTCP {
+		fail(w, 400, errors.New("proto must be http, https or tcp"))
+		return
+	}
+	if in.TargetHost == "" {
+		in.TargetHost = "127.0.0.1"
+	}
+	if in.Auth != nil && !model.Web(in.Proto) {
+		fail(w, 400, errors.New("login protection works for http and https only"))
+		return
+	}
+	mi := mappingInput{}
+	if in.Auth != nil {
+		if in.Auth.Password != "" {
+			mi.NewPassword = &in.Auth.Password
+		}
+		if in.Auth.Pin != "" {
+			mi.NewPin = &in.Auth.Pin
+		}
+	}
+	pw, pin, err := hashSecrets(&mi)
+	if err != nil {
+		fail(w, 400, err)
+		return
+	}
+	slug := exposeSlug(in.Name)
+
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.sweepExposedLocked()
+	edges := s.exposeEdges(c.Org.ID, in.EdgeID)
+	if len(edges) == 0 {
+		fail(w, 409, errors.New("no edge is online for this organization"))
+		return
+	}
+	active := 0
+	for _, m := range s.st.Mappings {
+		if m.OrgID == c.Org.ID && isExposed(m.Tags) {
+			active++
+		}
+	}
+	if active >= maxExposePerOrg {
+		fail(w, 429, errors.New("too many active exposures; stop one first"))
+		return
+	}
+	site := &model.Site{ID: newID(), OrgID: c.Org.ID, Name: "expose " + slug, Token: randHex(24), Connected: []string{},
+		TunnelIP: s.allocIPLocked(), Tags: []string{exposeTag}}
+	if site.TunnelIP == "" {
+		fail(w, 507, errors.New("tunnel address space is full"))
+		return
+	}
+	// validateMapping looks the site up, so it is added first and taken out again if nothing fits
+	s.st.Sites = append(s.st.Sites, site)
+	lastErr := errors.New("no edge could take it")
+	for _, e := range edges {
+		m := model.Mapping{Name: "expose " + slug, Proto: in.Proto, EdgeID: e.ID, Enabled: true, Tags: []string{exposeTag},
+			Targets: []model.Target{{SiteID: site.ID, Host: in.TargetHost, Port: in.TargetPort}}, HealthSeconds: -1}
+		if model.Web(in.Proto) {
+			m.Port = in.Port
+			if m.Port == 0 {
+				m.Port = map[string]int{model.ProtoHTTP: 80, model.ProtoHTTPS: 443}[in.Proto]
+			}
+			m.Auth = model.Auth{Password: pw != nil, Pin: pin != nil, SSO: in.Auth != nil && in.Auth.SSO}
+			hosts := []string{strings.ToLower(in.Hostname)}
+			if in.Hostname == "" {
+				hosts = nil
+				for _, d := range s.st.Domains {
+					if d.OrgID == c.Org.ID && d.EdgeID == e.ID {
+						hosts = append(hosts, fmt.Sprintf("%s-%s.%s", slug, randHex(2), d.Name))
+					}
+				}
+				if len(hosts) == 0 {
+					lastErr = errors.New("no domain is set up for this edge; add one on the Domains tab, or give a hostname")
+				}
+			}
+			for _, h := range hosts {
+				m.Hostname = h
+				if lastErr = s.validateMapping(c, &m); lastErr == nil {
+					break
+				}
+			}
+		} else {
+			tries := 1
+			if in.Port == 0 {
+				tries = 60
+			}
+			for i := 0; i < tries; i++ {
+				m.Port = in.Port
+				if in.Port == 0 {
+					m.Port = exposePortMin + rand.IntN(exposePortMax-exposePortMin+1)
+				}
+				if lastErr = s.validateMapping(c, &m); lastErr == nil {
+					break
+				}
+			}
+		}
+		if lastErr != nil {
+			continue
+		}
+		m.ID = newID()
+		s.applySecrets(&m, pw, pin)
+		s.st.Mappings = append(s.st.Mappings, &m)
+		s.expose[m.ID] = time.Now().Add(ttl)
+		s.bump()
+		s.audit(c.Org.ID, c.Actor, "expose.create", m.Name+" on "+portLabel(m))
+		out := s.exposeView(&m, e)
+		out.SiteToken = site.Token
+		out.Hammerhead = s.dashboardURL()
+		writeJSON(w, 200, out)
+		return
+	}
+	s.st.Sites = filter(s.st.Sites, func(x *model.Site) bool { return x.ID != site.ID })
+	fail(w, 400, lastErr)
+}
+
+func (s *Server) findExposeLocked(c *orgCtx, id string) *model.Mapping {
+	m := s.findMapping(id)
+	if m == nil || m.OrgID != c.Org.ID || !isExposed(m.Tags) {
+		return nil
+	}
+	return m
+}
+
+func (s *Server) renewExpose(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	var in struct {
+		TTL int `json:"ttlSeconds"`
+	}
+	if r.ContentLength != 0 && !decode(w, r, &in) {
+		return
+	}
+	ttl, err := exposeTTL(in.TTL)
+	if err != nil {
+		fail(w, 400, err)
+		return
+	}
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.sweepExposedLocked()
+	m := s.findExposeLocked(c, r.PathValue("id"))
+	if m == nil {
+		fail(w, 404, errors.New("no such exposure (it may have expired)"))
+		return
+	}
+	s.expose[m.ID] = time.Now().Add(ttl)
+	writeJSON(w, 200, s.exposeView(m, s.findEdge(m.EdgeID)))
+}
+
+func (s *Server) getExpose(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.sweepExposedLocked()
+	m := s.findExposeLocked(c, r.PathValue("id"))
+	if m == nil {
+		fail(w, 404, errors.New("no such exposure (it may have expired)"))
+		return
+	}
+	writeJSON(w, 200, s.exposeView(m, s.findEdge(m.EdgeID)))
+}
+
+func (s *Server) listExpose(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.sweepExposedLocked()
+	out := []exposeOut{}
+	for _, m := range s.st.Mappings {
+		if m.OrgID == c.Org.ID && isExposed(m.Tags) {
+			out = append(out, s.exposeView(m, s.findEdge(m.EdgeID)))
+		}
+	}
+	writeJSON(w, 200, out)
+}
+
+func (s *Server) deleteExpose(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	m := s.findExposeLocked(c, r.PathValue("id"))
+	if m == nil {
+		ok(w) // already gone: stopping twice is fine
+		return
+	}
+	name := m.Name
+	s.dropExposeLocked(m.ID)
+	s.audit(c.Org.ID, c.Actor, "expose.delete", name)
+	ok(w)
+}
+
+// dropExposeLocked removes an ephemeral mapping and the site that served it.
+func (s *Server) dropExposeLocked(mappingID string) {
+	m := s.findMapping(mappingID)
+	if m == nil {
+		return
+	}
+	gone := map[string]bool{}
+	for _, t := range m.Targets {
+		if st := s.findSite(t.SiteID); st != nil && !st.Local && isExposed(st.Tags) {
+			gone[st.ID] = true
+		}
+	}
+	s.dropMappingLocked(mappingID)
+	delete(s.expose, mappingID)
+	s.st.Mappings = filter(s.st.Mappings, func(x *model.Mapping) bool { return x.ID != mappingID })
+	s.st.Sites = filter(s.st.Sites, func(x *model.Site) bool { return !gone[x.ID] })
+	s.bump()
+}
+
+// sweepExposedLocked removes every ephemeral mapping whose deadline has passed, or that has no
+// deadline at all (left over from before a restart), and any ephemeral site nothing uses.
+func (s *Server) sweepExposedLocked() {
+	now := time.Now()
+	var dead []string
+	for _, m := range s.st.Mappings {
+		if !isExposed(m.Tags) {
+			continue
+		}
+		if at, known := s.expose[m.ID]; !known || now.After(at) {
+			dead = append(dead, m.ID)
+		}
+	}
+	for _, id := range dead {
+		s.dropExposeLocked(id)
+	}
+	used := map[string]bool{}
+	for _, m := range s.st.Mappings {
+		for _, t := range m.Targets {
+			used[t.SiteID] = true
+		}
+	}
+	n := len(s.st.Sites)
+	s.st.Sites = filter(s.st.Sites, func(x *model.Site) bool { return !(isExposed(x.Tags) && !x.Local && !used[x.ID]) })
+	if n != len(s.st.Sites) {
+		s.bump()
+	}
+}
+
+// exposeLoop expires ephemeral exposures. Its first pass clears everything left from the last run,
+// since deadlines are kept in memory only.
+func (s *Server) exposeLoop() {
+	t := time.NewTicker(10 * time.Second)
+	defer t.Stop()
+	for {
+		s.mu.Lock()
+		s.sweepExposedLocked()
+		s.mu.Unlock()
+		select {
+		case <-s.stop:
+			return
+		case <-t.C:
+		}
+	}
+}

expose_test.go

@@ -0,0 +1,195 @@
+package hammerhead
+
+import (
+	"encoding/json"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/reef/model"
+)
+
+func exposeServer(t *testing.T) (*Server, func(method, path string, in any) (int, []byte), string) {
+	t.Helper()
+	dir := t.TempDir()
+	s, err := New(dir, Options{AdminEmail: "admin@test.local", AdminPassword: "password123", AlertInterval: time.Hour})
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(s.Close)
+	do := apiClient(t, s)
+	_, b := do("GET", "/api/me", nil)
+	var me struct{ Orgs []struct{ ID string } }
+	json.Unmarshal(b, &me)
+	code, raw := do("POST", "/api/edges", map[string]any{"name": "e", "tunnelAddr": "203.0.113.5:51820"})
+	if code != 200 {
+		t.Fatalf("edge: %d %s", code, raw)
+	}
+	var e model.Edge
+	json.Unmarshal(raw, &e)
+	s.mu.Lock()
+	s.findEdge(e.ID).LastSeen = time.Now()
+	s.mu.Unlock()
+	if code, raw := do("POST", "/api/orgs/"+me.Orgs[0].ID+"/domains", map[string]any{"name": "t.example", "edgeId": e.ID}); code != 200 {
+		t.Fatalf("domain: %d %s", code, raw)
+	}
+	return s, do, me.Orgs[0].ID
+}
+
+func TestExposeLifecycle(t *testing.T) {
+	s, do, org := exposeServer(t)
+	base := "/api/orgs/" + org + "/expose"
+
+	code, raw := do("POST", base, map[string]any{"proto": "tcp", "targetPort": 3000})
+	if code != 200 {
+		t.Fatalf("create tcp: %d %s", code, raw)
+	}
+	var tcp exposeOut
+	json.Unmarshal(raw, &tcp)
+	if tcp.Host != "203.0.113.5" || tcp.Port < exposePortMin || tcp.Port > exposePortMax || tcp.SiteToken == "" || tcp.ID == "" {
+		t.Fatalf("tcp exposure: %+v", tcp)
+	}
+
+	code, raw = do("POST", base, map[string]any{"proto": "http", "name": "My App!", "targetPort": 8080, "auth": map[string]any{"password": "hunter22"}})
+	if code != 200 {
+		t.Fatalf("create http: %d %s", code, raw)
+	}
+	var web exposeOut
+	json.Unmarshal(raw, &web)
+	if !strings.HasPrefix(web.Hostname, "my-app-") || !strings.HasSuffix(web.Hostname, ".t.example") || web.URL != "http://"+web.Hostname || !web.Protected {
+		t.Fatalf("http exposure: %+v", web)
+	}
+
+	// validation
+	for name, in := range map[string]map[string]any{
+		"bad proto":    {"proto": "udp", "targetPort": 1},
+		"bad ttl":      {"proto": "tcp", "targetPort": 1, "ttlSeconds": 5},
+		"bad target":   {"proto": "tcp", "targetPort": 0},
+		"tcp login":    {"proto": "tcp", "targetPort": 1, "auth": map[string]any{"password": "hunter22"}},
+		"short pass":   {"proto": "http", "targetPort": 1, "auth": map[string]any{"password": "x"}},
+		"foreign host": {"proto": "http", "targetPort": 1, "hostname": "evil.example.com"},
+		"unknown edge": {"proto": "tcp", "targetPort": 1, "edgeId": "nope"},
+		"bad pin":      {"proto": "http", "targetPort": 1, "auth": map[string]any{"pin": "12"}},
+	} {
+		if code, raw := do("POST", base, in); code < 400 {
+			t.Errorf("%s accepted: %d %s", name, code, raw)
+		}
+	}
+
+	_, raw = do("GET", base, nil)
+	var list []exposeOut
+	json.Unmarshal(raw, &list)
+	if len(list) != 2 {
+		t.Fatalf("list: %s", raw)
+	}
+	// failed creates must not leave sites behind
+	if sites := s.ephemeralSites(); sites != 2 {
+		t.Errorf("want 2 ephemeral sites, have %d", sites)
+	}
+
+	// renew pushes the deadline out; a deadline in the past expires it
+	s.mu.Lock()
+	s.expose[tcp.ID] = time.Now().Add(time.Second)
+	s.mu.Unlock()
+	if code, _ := do("POST", base+"/"+tcp.ID+"/renew", map[string]any{"ttlSeconds": 600}); code != 200 {
+		t.Fatalf("renew: %d", code)
+	}
+	s.mu.Lock()
+	left := time.Until(s.expose[tcp.ID])
+	s.expose[web.ID] = time.Now().Add(-time.Second)
+	s.mu.Unlock()
+	if left < 9*time.Minute {
+		t.Errorf("renew left %v", left)
+	}
+	if code, _ := do("GET", base+"/"+web.ID, nil); code != 404 {
+		t.Errorf("expired exposure still there: %d", code)
+	}
+	if code, _ := do("POST", base+"/"+web.ID+"/renew", nil); code != 404 {
+		t.Errorf("renewed an expired exposure: %d", code)
+	}
+	s.mu.Lock()
+	if s.ephemeralSitesLocked() != 1 || len(s.st.Mappings) != 1 || len(s.st.Secrets) != 0 {
+		t.Errorf("expiry left debris: %d sites, %d mappings, %d secrets", s.ephemeralSitesLocked(), len(s.st.Mappings), len(s.st.Secrets))
+	}
+	s.mu.Unlock()
+
+	// stopping is idempotent
+	for i := 0; i < 2; i++ {
+		if code, _ := do("DELETE", base+"/"+tcp.ID, nil); code != 200 {
+			t.Errorf("delete %d: %d", i, code)
+		}
+	}
+	s.mu.Lock()
+	if s.ephemeralSitesLocked() != 0 || len(s.st.Mappings) != 0 {
+		t.Errorf("delete left debris")
+	}
+	s.mu.Unlock()
+}
+
+func TestExposeSweptAfterRestart(t *testing.T) {
+	s, do, org := exposeServer(t)
+	if code, raw := do("POST", "/api/orgs/"+org+"/expose", map[string]any{"proto": "tcp", "targetPort": 3000}); code != 200 {
+		t.Fatalf("%d %s", code, raw)
+	}
+	// a regular mapping on a regular site must survive the sweep
+	_, raw := do("POST", "/api/orgs/"+org+"/sites", map[string]any{"name": "home"})
+	var site model.Site
+	json.Unmarshal(raw, &site)
+	s.mu.Lock()
+	s.expose = map[string]time.Time{} // what a restart does
+	s.sweepExposedLocked()
+	n, m := len(s.st.Mappings), 0
+	for _, x := range s.st.Sites {
+		if x.ID == site.ID {
+			m++
+		}
+	}
+	s.mu.Unlock()
+	if n != 0 || m != 1 {
+		t.Fatalf("mappings left %d, regular site kept %d", n, m)
+	}
+}
+
+func (s *Server) ephemeralSitesLocked() int {
+	n := 0
+	for _, x := range s.st.Sites {
+		if isExposed(x.Tags) {
+			n++
+		}
+	}
+	return n
+}
+
+func (s *Server) ephemeralSites() int {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	return s.ephemeralSitesLocked()
+}
+
+// An API key belongs to one organization, so "-" in the path stands for it.
+func TestExposeKeyOrgShorthand(t *testing.T) {
+	s, do, org := exposeServer(t)
+	_, raw := do("POST", "/api/orgs/"+org+"/apikeys", map[string]any{"name": "cli"})
+	var k struct{ Key string }
+	json.Unmarshal(raw, &k)
+	if !strings.HasPrefix(k.Key, "hhk_") {
+		t.Fatal(string(raw))
+	}
+	call := func(key, body string) int {
+		req := httptest.NewRequest("POST", "/api/orgs/-/expose", strings.NewReader(body))
+		req.Header.Set("Authorization", "Bearer "+key)
+		rec := httptest.NewRecorder()
+		s.Handler().ServeHTTP(rec, req)
+		return rec.Code
+	}
+	if got := call(k.Key, `{"proto":"tcp","targetPort":3000}`); got != 200 {
+		t.Errorf("key: %d", got)
+	}
+	if got := call("hhk_dead_beef", `{"proto":"tcp","targetPort":3000}`); got != 401 {
+		t.Errorf("bad key: %d", got)
+	}
+	if got := call("", `{"proto":"tcp","targetPort":3000}`); got != 401 {
+		t.Errorf("no key: %d", got)
+	}
+}

go.mod

@@ -3,23 +3,24 @@ module git.veillax.com/Veillax/hammerhead
 go 1.27.3
 
 require (
-	git.veillax.com/Veillax/bamboo v0.1.5
-	git.veillax.com/Veillax/blacktip v0.1.8
-	git.veillax.com/Veillax/reef v0.1.9
+	git.veillax.com/Veillax/bamboo v0.1.10
+	git.veillax.com/Veillax/blacktip v0.1.10
+	git.veillax.com/Veillax/reef v0.1.12
 	github.com/emersion/go-msgauth v0.7.0
 	golang.org/x/crypto v0.58.0
+	golang.org/x/net v0.61.0
 	modernc.org/sqlite v1.61.0
 )
 
 require (
 	github.com/Microsoft/go-winio v0.6.4 // indirect
+	github.com/creack/pty v1.1.24 // indirect
 	github.com/dustin/go-humanize v1.1.0 // indirect
 	github.com/google/btree v1.1.2 // indirect
 	github.com/google/uuid v1.6.0 // indirect
 	github.com/mattn/go-isatty v0.0.24 // indirect
 	github.com/ncruces/go-strftime v1.1.0 // indirect
 	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
-	golang.org/x/net v0.61.0 // indirect
 	golang.org/x/sys v0.49.0 // indirect
 	golang.org/x/text v0.43.0 // indirect
 	golang.org/x/time v0.7.0 // indirect

go.sum

@@ -1,11 +1,13 @@
-git.veillax.com/Veillax/bamboo v0.1.5 h1:i81c75/7wjayP9tOQ3tSnJWTBhvbjAwNBEPhT+3P2Xk=
-git.veillax.com/Veillax/bamboo v0.1.5/go.mod h1:se4kkcP6c3uCGD4HQd+nDLoblS0lN8qa+eFWLo5Horw=
-git.veillax.com/Veillax/blacktip v0.1.8 h1:TR4HfXlGwrNjCqOY+xUnJCJwESRTwFerb2wVhq/zbUk=
-git.veillax.com/Veillax/blacktip v0.1.8/go.mod h1:dVm4wC6wufPjFzK8pvHnqNJJlxroRYmnq8mdkCqOKGc=
-git.veillax.com/Veillax/reef v0.1.9 h1:dmC+oPmRQIOVfx2kgS95QvRDn2wudwpZw7H9p0m6W8Y=
-git.veillax.com/Veillax/reef v0.1.9/go.mod h1:ZBUd/BARO5kkpUNTP81TTY1aQXLHFR6bKDhCxckjHso=
+git.veillax.com/Veillax/bamboo v0.1.10 h1:R21+L54cS87VXmOS4eAsfRwp6vL+ORLqJUBaRHBIlJE=
+git.veillax.com/Veillax/bamboo v0.1.10/go.mod h1:Z3O7xoa52mFaADs7RxaBQl0a0K+5HlHPB1eOYQjM15k=
+git.veillax.com/Veillax/blacktip v0.1.10 h1:2e1wUbMCp4z/DvQjAJCRmLyqu4PX+qV3mo5WBliqGiE=
+git.veillax.com/Veillax/blacktip v0.1.10/go.mod h1:tarAhdKU9hx5pdufe9wG3dRWYkWgPYZvawKM5DWgt6o=
+git.veillax.com/Veillax/reef v0.1.12 h1:Yizgcbh/TUz/dkO/4F+MOZkXlN7FU3/et/5uq1dH72s=
+git.veillax.com/Veillax/reef v0.1.12/go.mod h1:ZBUd/BARO5kkpUNTP81TTY1aQXLHFR6bKDhCxckjHso=
 github.com/Microsoft/go-winio v0.6.4 h1:kJF2pamtYCWXf0xAT8sAkwKC0pMl2HYBrRPHAqyUc+g=
 github.com/Microsoft/go-winio v0.6.4/go.mod h1:1xl5JXG4ze/m74GH6Dc4Mwxfc3VL5siRVQjIAlHH07U=
+github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
+github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=
 github.com/dustin/go-humanize v1.1.0 h1:dbKTrvD0klcbBV/h4AWJdMuZogJACoMlvWIWZ5b2xWg=
 github.com/dustin/go-humanize v1.1.0/go.mod h1:hc1CvRkJMsgxqjmjMQF3QNRAZBwY8AXBAzKYoSX9sFI=
 github.com/emersion/go-msgauth v0.7.0 h1:vj2hMn6KhFtW41kshIBTXvp6KgYSqpA/ZN9Pv4g1INc=
@@ -34,6 +36,8 @@ golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
 golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
 golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
 golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
+golang.org/x/term v0.47.0 h1:cpyOwQkr2Nzxe2p6KnrTG7Ei2wtzjsuSjhpXXepZ/BM=
+golang.org/x/term v0.47.0/go.mod h1:efAfAuG6RKABvg1ArGebRkvX6LXul14SM67o2e4mUCY=
 golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
 golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
 golang.org/x/time v0.7.0 h1:ntUhktv3OPE6TgYxXWv9vKvUSJyIFJlyohwbkEwPrKQ=

infra.go

@@ -4,6 +4,7 @@ import (
 	"errors"
 	"net"
 	"net/http"
+	"net/netip"
 	"sort"
 	"strings"
 	"time"
@@ -64,11 +65,22 @@ func validAddr(a string) bool {
 	return n > 0
 }
 
-func (s *Server) listEdges(w http.ResponseWriter, r *http.Request, p *principal) {
+func (s *Server) listEdges(w http.ResponseWriter, r *http.Request) {
+	p := s.principalFrom(r)
+	if p == nil {
+		fail(w, 401, errors.New("Sign in to continue."))
+		return
+	}
 	s.mu.Lock()
 	defer s.mu.Unlock()
 	out := []edgeView{}
 	for _, e := range s.st.Edges {
+		if p.Key != nil && !p.serverKey() { // an org key sees the edges its organization may use
+			if s.edgeAllowedFor(e, p.Key.OrgID) {
+				out = append(out, s.viewEdge(e, false))
+			}
+			continue
+		}
 		if p.User.ServerAdmin {
 			out = append(out, s.viewEdge(e, true))
 			continue
@@ -270,7 +282,7 @@ func (s *Server) deleteSite(w http.ResponseWriter, r *http.Request, c *orgCtx) {
 			}
 		}
 	}
-	s.st.Mappings = filter(s.st.Mappings, func(m *model.Mapping) bool { return len(m.Targets) > 0 })
+	s.st.Mappings = filter(s.st.Mappings, func(m *model.Mapping) bool { return len(m.Targets) > 0 || m.IsRedirect() })
 	s.bump()
 	s.audit(c.Org.ID, c.Actor, "site.delete", st.Name)
 	ok(w)
@@ -291,6 +303,47 @@ func (s *Server) siteContainers(w http.ResponseWriter, r *http.Request, c *orgCt
 	writeJSON(w, 200, out)
 }
 
+// cleanDiscovered keeps what an agent reports as suggestions only if it is well formed: it is shown in
+// the dashboard, so lengths are bounded and anything that is not a plain address is dropped.
+func cleanDiscovered(in []model.DiscoveredService) []model.DiscoveredService {
+	out := []model.DiscoveredService{}
+	clip := func(s string, n int) string {
+		if len(s) > n {
+			s = s[:n]
+		}
+		return strings.ToValidUTF8(s, "")
+	}
+	for _, d := range in {
+		if len(out) >= 500 {
+			break
+		}
+		if _, err := netip.ParseAddr(d.Host); err != nil || d.Port < 1 || d.Port > 65535 {
+			continue
+		}
+		src := "scan"
+		if d.Source == "local" {
+			src = "local"
+		}
+		out = append(out, model.DiscoveredService{Host: d.Host, Port: d.Port, Proto: "tcp", Service: clip(d.Service, 20), Detail: clip(d.Detail, 160), Source: src})
+	}
+	return out
+}
+
+func (s *Server) siteDiscovered(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	st := s.findSite(r.PathValue("id"))
+	if st == nil || !s.siteVisible(c, st) {
+		fail(w, 404, errors.New("no such site"))
+		return
+	}
+	out := s.discovered[st.ID]
+	if out == nil {
+		out = []model.DiscoveredService{}
+	}
+	writeJSON(w, 200, out)
+}
+
 // ---------- domains ----------
 
 func domainsOverlap(a, b string) bool {

mailegress.go

@@ -149,6 +149,7 @@ type mailEgressView struct {
 	Status      string           `json:"status"`
 	StatusKnown bool             `json:"statusKnown"`
 	Check       *model.MailCheck `json:"check,omitempty"`
+	Findings    []Finding        `json:"findings"` // the deliverability checklist, with fixes
 }
 
 // mailEgressViewsLocked lists an org's mail egress without secrets. Caller holds s.mu.
@@ -164,6 +165,10 @@ func (s *Server) mailEgressViewsLocked(orgID string) []mailEgressView {
 		if chk, ok := s.mailChecks[e.ID]; ok {
 			c := chk
 			v.Check = &c
+			v.Findings = s.mailEgressFindingsLocked(e, chk)
+		}
+		if v.Findings == nil {
+			v.Findings = []Finding{}
 		}
 		out = append(out, v)
 	}

mappings.go

@@ -6,6 +6,8 @@ import (
 	"fmt"
 	"net"
 	"net/http"
+	"net/url"
+	"strconv"
 	"strings"
 	"time"
 
@@ -97,7 +99,16 @@ func (s *Server) validateMapping(c *orgCtx, m *model.Mapping) error {
 	if edge == nil || !s.edgeAllowedFor(edge, c.Org.ID) {
 		return errors.New("choose an edge")
 	}
-	if len(m.Targets) == 0 {
+	redirect := strings.TrimSpace(m.RedirectTo) != ""
+	if redirect {
+		if !model.Web(m.Proto) {
+			return errors.New("only websites (http or https) can redirect")
+		}
+		m.Targets = []model.Target{} // a redirect has nothing behind it
+	} else {
+		m.RedirectTo, m.RedirectPermanent, m.RedirectKeepPath = "", false, false
+	}
+	if len(m.Targets) == 0 && !redirect {
 		return errors.New("add at least one target")
 	}
 	if m.Proto == model.ProtoUDP && len(m.Targets) > 1 {
@@ -173,9 +184,19 @@ func (s *Server) validateMapping(c *orgCtx, m *model.Mapping) error {
 		return errors.New("PROXY protocol must be v1, v2 or off")
 	}
 	// web-only options
-	if !model.Web(m.Proto) {
+	if redirect {
+		to, err := cleanRedirect(m)
+		if err != nil {
+			return err
+		}
+		m.RedirectTo = to
+	}
+	if !model.Web(m.Proto) || redirect {
 		m.Auth, m.Rules, m.RewriteHost = model.Auth{}, nil, ""
 		m.RedirectHTTP, m.BackendHTTPS, m.HealthPath = false, false, ""
+		if redirect {
+			m.HealthSeconds = -1 // nothing to check
+		}
 	} else {
 		a := &m.Auth
 		a.AllowedRoles, a.AllowedUsers, a.AllowedMails = listFromCSV(a.AllowedRoles), listFromCSV(a.AllowedUsers), listFromCSV(a.AllowedMails)
@@ -262,6 +283,28 @@ func (s *Server) validateMapping(c *orgCtx, m *model.Mapping) error {
 	return nil
 }
 
+// cleanRedirect checks where a redirect sends visitors and refuses a destination that would
+// send them straight back to the same address.
+func cleanRedirect(m *model.Mapping) (string, error) {
+	to := strings.TrimSpace(m.RedirectTo)
+	u, err := url.Parse(to)
+	if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Hostname() == "" || u.User != nil {
+		return "", errors.New("Redirect to a full address such as https://www.example.com.")
+	}
+	if len(to) > 2048 || strings.ContainsAny(to, " \t\r\n$") {
+		return "", errors.New("The redirect address is too long or contains spaces or a dollar sign.")
+	}
+	port := u.Port()
+	if port == "" {
+		port = map[string]string{"http": "80", "https": "443"}[u.Scheme]
+	}
+	same := u.Scheme == m.Proto && strings.EqualFold(u.Hostname(), m.Hostname) && port == strconv.Itoa(m.Port)
+	if same && (m.RedirectKeepPath || u.Path == "" || u.Path == "/") {
+		return "", errors.New("That redirect would send visitors back to the same address.")
+	}
+	return to, nil
+}
+
 // portLabel is "25565" or, for a range, "2456-2458".
 func portLabel(m model.Mapping) string {
 	if m.LastPort() > m.Port {

metrics.go

@@ -84,6 +84,11 @@ func (s *Server) metrics(w http.ResponseWriter, r *http.Request) {
 			line("hammerhead_mail_check_ok", "gauge", "1 if the deliverability check passes", b2f(c.PTRMatchesHelo), "relay", x.Name, "org", org, "check", "reverse_dns")
 			line("hammerhead_mail_check_ok", "gauge", "1 if the deliverability check passes", b2f(c.HeloResolves), "relay", x.Name, "org", org, "check", "helo_resolves")
 			line("hammerhead_mail_check_ok", "gauge", "1 if the deliverability check passes", b2f(c.Port25 == "open"), "relay", x.Name, "org", org, "check", "port25")
+			for _, f := range s.mailEgressFindingsLocked(x, c) {
+				if f.Domain != "" || f.ID == "blocklist" || f.ID == "outcomes" {
+					line("hammerhead_mail_finding_ok", "gauge", "1 if a per-domain, blocklist or delivery check passes", b2f(f.Status == "pass"), "relay", x.Name, "org", org, "check", f.Label, "domain", f.Domain)
+				}
+			}
 		}
 	}
 	firing := map[string]int{}

orgs.go

@@ -29,7 +29,9 @@ func (s *Server) createOrg(w http.ResponseWriter, r *http.Request, p *principal)
 	}
 	s.mu.Lock()
 	org, adminRole := s.newOrgLocked(in.Name, in.AllowLocal)
+	if p.User.ID != "" { // a server key belongs to no user
 		s.st.Members = append(s.st.Members, &Member{UserID: p.User.ID, OrgID: org.ID, RoleID: adminRole.ID})
+	}
 	s.save()
 	o := *org
 	s.mu.Unlock()
@@ -423,7 +425,8 @@ func (s *Server) listKeys(w http.ResponseWriter, r *http.Request, c *orgCtx) {
 	writeJSON(w, 200, out)
 }
 
-func (s *Server) createKey(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+// issueKey stores a new key for the org ("" = a server key) and returns its ID and full token.
+func (s *Server) issueKey(w http.ResponseWriter, r *http.Request, org, actor string) {
 	var in struct{ Name string }
 	if !decode(w, r, &in) {
 		return
@@ -435,11 +438,82 @@ func (s *Server) createKey(w http.ResponseWriter, r *http.Request, c *orgCtx) {
 	}
 	id, secret := randHex(4), randHex(24)
 	s.mu.Lock()
-	s.st.APIKeys = append(s.st.APIKeys, &APIKey{ID: id, OrgID: c.Org.ID, Name: in.Name, Hash: sha(secret), Created: time.Now()})
+	s.st.APIKeys = append(s.st.APIKeys, &APIKey{ID: id, OrgID: org, Name: in.Name, Hash: sha(secret), Created: time.Now()})
 	s.save()
 	s.mu.Unlock()
-	s.audit(c.Org.ID, c.Actor, "apikey.create", in.Name)
-	writeJSON(w, 200, map[string]string{"key": "hhk_" + id + "_" + secret})
+	s.audit(org, actor, "apikey.create", in.Name)
+	writeJSON(w, 200, map[string]string{"id": id, "key": "hhk_" + id + "_" + secret})
+}
+
+func (s *Server) createKey(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	s.issueKey(w, r, c.Org.ID, c.Actor)
+}
+
+// Server keys manage edges, server-level alerts and organizations, so only a signed-in
+// server administrator may mint, list or revoke them (a key can't make more keys).
+
+func (s *Server) withServerAdminUser(fn func(http.ResponseWriter, *http.Request, *principal)) http.HandlerFunc {
+	return s.withUser(func(w http.ResponseWriter, r *http.Request, p *principal) {
+		if !p.User.ServerAdmin {
+			fail(w, 403, errors.New("server administrators only"))
+			return
+		}
+		fn(w, r, p)
+	})
+}
+
+func (s *Server) listServerKeys(w http.ResponseWriter, r *http.Request, p *principal) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	out := []APIKey{}
+	for _, k := range s.st.APIKeys {
+		if k.OrgID == "" {
+			cp := *k
+			cp.Hash = ""
+			out = append(out, cp)
+		}
+	}
+	writeJSON(w, 200, out)
+}
+
+func (s *Server) createServerKey(w http.ResponseWriter, r *http.Request, p *principal) {
+	s.issueKey(w, r, "", p.User.Email)
+}
+
+func (s *Server) deleteServerKey(w http.ResponseWriter, r *http.Request, p *principal) {
+	id := r.PathValue("id")
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.st.APIKeys = filter(s.st.APIKeys, func(k *APIKey) bool { return !(k.ID == id && k.OrgID == "") })
+	s.save()
+	s.audit("", p.User.Email, "apikey.delete", id)
+	ok(w)
+}
+
+// listOrgs shows the organizations the caller can reach: a server key sees all of them,
+// an org key its own.
+func (s *Server) listOrgs(w http.ResponseWriter, r *http.Request) {
+	p := s.principalFrom(r)
+	if p == nil {
+		fail(w, 401, errors.New("Sign in to continue."))
+		return
+	}
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	out := []Org{}
+	for _, o := range s.st.Orgs {
+		switch {
+		case p.serverKey() || p.User.ServerAdmin:
+			out = append(out, *o)
+		case p.Key != nil:
+			if p.Key.OrgID == o.ID {
+				out = append(out, *o)
+			}
+		case s.member(p.User.ID, o.ID) != nil:
+			out = append(out, *o)
+		}
+	}
+	writeJSON(w, 200, out)
 }
 
 func (s *Server) deleteKey(w http.ResponseWriter, r *http.Request, c *orgCtx) {

redirects_test.go

@@ -0,0 +1,109 @@
+package hammerhead
+
+import (
+	"encoding/json"
+	"strings"
+	"testing"
+)
+
+// a redirect is a website with a destination instead of a target
+func TestDomainRedirects(t *testing.T) {
+	s, err := New(t.TempDir(), Options{AdminEmail: "admin@test.local", AdminPassword: "password123", MasterKey: key})
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	do := apiClient(t, s)
+	var me struct{ Orgs []struct{ ID string } }
+	_, b := do("GET", "/api/me", nil)
+	json.Unmarshal(b, &me)
+	org := me.Orgs[0].ID
+	var edge, site struct{ ID string }
+	_, b = do("POST", "/api/edges", map[string]any{"name": "e", "tunnelAddr": "203.0.113.5:51820"})
+	json.Unmarshal(b, &edge)
+	_, b = do("POST", "/api/orgs/"+org+"/sites", map[string]any{"name": "home"})
+	json.Unmarshal(b, &site)
+	if code, b := do("POST", "/api/orgs/"+org+"/domains", map[string]any{"name": "example.test", "edgeId": edge.ID, "ip": "203.0.113.10"}); code != 200 {
+		t.Fatalf("domain: %d %s", code, b)
+	}
+	base := "/api/orgs/" + org + "/mappings"
+	open := func(m map[string]any) (int, []byte) {
+		m["edgeId"], m["enabled"], m["bindIp"] = edge.ID, true, "0.0.0.0"
+		return do("POST", base, m)
+	}
+
+	// the redirect needs no target, and anything that only makes sense with one is dropped
+	code, raw := open(map[string]any{"proto": "https", "port": 443, "hostname": "old.example.test", "redirectTo": " https://new.example.test/home ", "redirectPermanent": true, "redirectKeepPath": true,
+		"auth": map[string]any{"password": true}, "rewriteHost": "x", "healthPath": "/h"})
+	if code != 200 {
+		t.Fatalf("create redirect: %d %s", code, raw)
+	}
+	var m struct {
+		ID, RedirectTo, RewriteHost, HealthPath string
+		RedirectPermanent, RedirectKeepPath     bool
+		Targets                                 []any
+		HealthSeconds                           int
+		Auth                                    struct{ Password bool }
+	}
+	json.Unmarshal(raw, &m)
+	if m.RedirectTo != "https://new.example.test/home" || !m.RedirectPermanent || !m.RedirectKeepPath || len(m.Targets) != 0 ||
+		m.Auth.Password || m.RewriteHost != "" || m.HealthPath != "" || m.HealthSeconds != -1 {
+		t.Fatalf("redirect was not normalised: %s", raw)
+	}
+
+	// it shares the port with other websites by hostname, like any website
+	if code, raw := open(map[string]any{"proto": "https", "port": 443, "hostname": "app.example.test", "targets": []any{map[string]any{"siteId": site.ID, "host": "127.0.0.1", "port": 8080}}}); code != 200 {
+		t.Fatalf("a website beside the redirect: %d %s", code, raw)
+	}
+	// and its plain-HTTP twin on port 80 is its own resource
+	if code, raw := open(map[string]any{"proto": "http", "port": 80, "hostname": "old.example.test", "redirectTo": "https://new.example.test/home"}); code != 200 {
+		t.Fatalf("http side of the redirect: %d %s", code, raw)
+	}
+
+	for name, in := range map[string]map[string]any{
+		"not a web port":      {"proto": "tcp", "port": 9000, "redirectTo": "https://new.example.test"},
+		"no scheme":           {"proto": "https", "port": 443, "hostname": "a.example.test", "redirectTo": "new.example.test"},
+		"other scheme":        {"proto": "https", "port": 443, "hostname": "a.example.test", "redirectTo": "ftp://new.example.test"},
+		"credentials":         {"proto": "https", "port": 443, "hostname": "a.example.test", "redirectTo": "https://u:p@new.example.test"},
+		"a dollar sign":       {"proto": "https", "port": 443, "hostname": "a.example.test", "redirectTo": "https://new.example.test/$1"},
+		"back to itself":      {"proto": "https", "port": 443, "hostname": "a.example.test", "redirectTo": "https://a.example.test/"},
+		"itself with a path":  {"proto": "https", "port": 443, "hostname": "a.example.test", "redirectTo": "https://a.example.test/x", "redirectKeepPath": true},
+		"host not in domains": {"proto": "https", "port": 443, "hostname": "a.elsewhere.test", "redirectTo": "https://new.example.test"},
+	} {
+		if code, raw := open(in); code != 400 {
+			t.Errorf("%s: expected 400, got %d %s", name, code, raw)
+		}
+	}
+	// http to https on the same name is the normal use, and is allowed
+	if code, raw := open(map[string]any{"proto": "http", "port": 8080, "hostname": "a.example.test", "redirectTo": "https://a.example.test/"}); code != 200 {
+		t.Errorf("an http to https redirect: %d %s", code, raw)
+	}
+
+	// turning a redirect into a normal website needs a target again
+	code, raw = do("PUT", base+"/"+m.ID, map[string]any{"proto": "https", "edgeId": edge.ID, "bindIp": "0.0.0.0", "port": 443, "hostname": "old.example.test", "enabled": true})
+	if code != 400 || !strings.Contains(string(raw), "target") {
+		t.Errorf("a website with no target and no redirect: %d %s", code, raw)
+	}
+
+	// deleting a site must not sweep away redirects, which have no targets
+	if code, raw := do("DELETE", "/api/orgs/"+org+"/sites/"+site.ID, nil); code != 200 {
+		t.Fatalf("delete site: %d %s", code, raw)
+	}
+	_, raw = do("GET", "/api/orgs/"+org+"/state", nil)
+	var st struct {
+		Mappings []struct{ Hostname, RedirectTo string }
+	}
+	json.Unmarshal(raw, &st)
+	var redirects, apps int
+	for _, x := range st.Mappings {
+		if x.RedirectTo != "" {
+			redirects++
+		}
+		if x.Hostname == "app.example.test" {
+			apps++
+		}
+	}
+	if redirects != 3 || apps != 0 {
+		t.Errorf("after deleting the site: %d redirects (want 3), %d apps (want 0): %s", redirects, apps, raw)
+	}
+}

sandbar.go

@@ -0,0 +1,343 @@
+package hammerhead
+
+import (
+	"bytes"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"log"
+	"net"
+	"net/http"
+	"net/url"
+	"sort"
+	"strconv"
+	"strings"
+	"time"
+
+	"git.veillax.com/Veillax/reef/model"
+	"git.veillax.com/Veillax/reef/netguard"
+)
+
+// Sandbar monitors: with this switched on under Server, Settings, every enabled
+// resource that has health checks gets a monitor on a Sandbar server, which then
+// tracks its uptime, shows it on the status page and pages whoever is on call.
+// Sandbar probes the public address (so it sees what visitors see), where
+// Hammerhead's own health checks look at the targets behind it. Hammerhead only
+// ever changes the monitors it created: the link from resource to monitor is
+// kept in the settings, so a monitor you add in Sandbar by hand is never touched.
+
+// SandbarSync is the connection to Sandbar for creating monitors.
+type SandbarSync struct {
+	Enabled      bool              `json:"enabled"`
+	URL          string            `json:"url"`
+	Token        string            `json:"token"`              // an administrator's API key (secret, never sent back to the browser)
+	PolicyID     string            `json:"policyId"`           // escalation policy for the alerts the monitors raise; blank pages nobody
+	Public       bool              `json:"public"`             // show the monitors on Sandbar's status page
+	AllowPrivate bool              `json:"allowPrivate"`       // Sandbar is on a private network, and so are the edge addresses it will probe
+	Monitors     map[string]string `json:"monitors,omitempty"` // resource id -> Sandbar monitor id
+}
+
+// sandbarSyncStatus is how the last sync went, for the settings page.
+type sandbarSyncStatus struct {
+	At       time.Time `json:"at"`
+	Error    string    `json:"error"`
+	Monitors int       `json:"monitors"`
+}
+
+const sandbarSyncEvery = time.Minute
+
+type monitorSpec struct {
+	Name      string
+	Kind      string
+	Target    string
+	IntervalS int
+	Expect    string
+}
+
+// sandbarEdgeHost is the address visitors reach a resource's edge on: the address it is
+// bound to, else the first public address the edge reported, else the host of its
+// tunnel endpoint.
+func sandbarEdgeHost(e *model.Edge, bind string) string {
+	if ip := net.ParseIP(bind); ip != nil && !ip.IsUnspecified() {
+		return bind
+	}
+	var first string
+	for _, a := range e.IPs {
+		ip := net.ParseIP(a)
+		if ip == nil || ip.IsLoopback() || ip.IsLinkLocalUnicast() {
+			continue
+		}
+		if !ip.IsPrivate() {
+			return a
+		}
+		if first == "" {
+			first = a
+		}
+	}
+	if first != "" {
+		return first
+	}
+	host, _, _ := net.SplitHostPort(e.TunnelAddr)
+	return host
+}
+
+// monitorFor describes the Sandbar monitor a resource should have, or false when
+// it has nothing worth probing (a redirect, UDP, health checks off, a wildcard).
+func (s *Server) monitorFor(m *model.Mapping) (monitorSpec, bool) {
+	if !m.Enabled || m.HealthSeconds == -1 || m.RedirectTo != "" || m.Proto == model.ProtoUDP {
+		return monitorSpec{}, false
+	}
+	e := s.findEdge(m.EdgeID)
+	if e == nil {
+		return monitorSpec{}, false
+	}
+	spec := monitorSpec{Name: m.Name, IntervalS: 30}
+	if m.HealthSeconds > 0 {
+		spec.IntervalS = max(m.HealthSeconds, 10)
+	}
+	host := m.Hostname
+	if strings.HasPrefix(host, "*") {
+		return monitorSpec{}, false
+	}
+	if host == "" {
+		if host = sandbarEdgeHost(e, m.BindIP); host == "" {
+			return monitorSpec{}, false
+		}
+	}
+	addr := net.JoinHostPort(host, strconv.Itoa(m.Port))
+	switch m.Proto {
+	case model.ProtoHTTP, model.ProtoHTTPS:
+		u := url.URL{Scheme: m.Proto, Host: addr, Path: m.HealthPath}
+		if (m.Proto == model.ProtoHTTP && m.Port == 80) || (m.Proto == model.ProtoHTTPS && m.Port == 443) {
+			u.Host = strings.Trim(host, "[]") // a bare IPv6 address needs its brackets back
+			if strings.Contains(host, ":") {
+				u.Host = "[" + u.Host + "]"
+			}
+		}
+		spec.Kind, spec.Target = "http", u.String()
+		if m.Auth.Protected() {
+			spec.Expect = "200-399" // visitors are sent to a sign-in page, which is the site working
+		}
+	default:
+		spec.Kind, spec.Target = "tcp", addr
+	}
+	return spec, true
+}
+
+// desiredMonitorsLocked is the monitor every qualifying resource should have, by resource id.
+func (s *Server) desiredMonitorsLocked() map[string]monitorSpec {
+	out := map[string]monitorSpec{}
+	for _, m := range s.st.Mappings {
+		if spec, ok := s.monitorFor(m); ok {
+			out[m.ID] = spec
+		}
+	}
+	return out
+}
+
+func sandbarCall(cfg SandbarSync, method, path string, body any, out any) error {
+	var rd io.Reader
+	if body != nil {
+		raw, _ := json.Marshal(body)
+		rd = bytes.NewReader(raw)
+	}
+	req, err := http.NewRequest(method, strings.TrimRight(cfg.URL, "/")+path, rd)
+	if err != nil {
+		return err
+	}
+	req.Header.Set("Content-Type", "application/json")
+	req.Header.Set("Authorization", "Bearer "+cfg.Token)
+	resp, err := netguard.Client(cfg.AllowPrivate).Do(req)
+	if err != nil {
+		return err
+	}
+	defer resp.Body.Close()
+	switch {
+	case resp.StatusCode == 401:
+		return errors.New("Sandbar rejected the API key")
+	case resp.StatusCode == 403:
+		return errors.New("Sandbar needs an administrator's API key to manage monitors")
+	case resp.StatusCode == 404 && method == "DELETE":
+		return nil // already gone
+	case resp.StatusCode >= 300:
+		msg, _ := io.ReadAll(io.LimitReader(resp.Body, 300))
+		return fmt.Errorf("Sandbar answered %s %s", resp.Status, strings.TrimSpace(string(msg)))
+	}
+	if out != nil {
+		return json.NewDecoder(resp.Body).Decode(out)
+	}
+	return nil
+}
+
+// syncSandbar makes Sandbar's monitors match the resources: it creates the ones
+// that are missing, corrects the ones that drifted, and removes those whose
+// resource is gone or no longer has health checks.
+func (s *Server) syncSandbar() (err error) {
+	s.sandbarMu.Lock()
+	defer s.sandbarMu.Unlock()
+	s.mu.Lock()
+	cfg := s.st.Settings.Sandbar
+	if !cfg.Enabled {
+		s.mu.Unlock()
+		return nil
+	}
+	want := s.desiredMonitorsLocked()
+	s.mu.Unlock()
+	links := map[string]string{}
+	for k, v := range cfg.Monitors {
+		links[k] = v
+	}
+	defer func() {
+		st := sandbarSyncStatus{At: time.Now(), Monitors: len(links)}
+		if err != nil {
+			st.Error = err.Error()
+		}
+		s.mu.Lock()
+		s.sandbarStatus = st
+		if now := &s.st.Settings.Sandbar; now.URL == cfg.URL && !sameLinks(now.Monitors, links) {
+			now.Monitors = links // kept even after a failure: a monitor already created must not be created twice
+			s.save()
+		}
+		s.mu.Unlock()
+	}()
+
+	var remote []map[string]any
+	if err := sandbarCall(cfg, "GET", "/api/v1/monitors", nil, &remote); err != nil {
+		return err
+	}
+	byID := map[string]map[string]any{}
+	for _, m := range remote {
+		if id, _ := m["id"].(string); id != "" {
+			byID[id] = m
+		}
+	}
+
+	ids := make([]string, 0, len(want))
+	for id := range want {
+		ids = append(ids, id)
+	}
+	sort.Strings(ids)
+	var errs []error
+	for _, id := range ids {
+		spec := want[id]
+		fields := map[string]any{"name": spec.Name, "kind": spec.Kind, "target": spec.Target, "intervalS": spec.IntervalS, "expect": spec.Expect,
+			"policyId": cfg.PolicyID, "public": cfg.Public, "allowPrivate": cfg.AllowPrivate, "enabled": true}
+		cur, exists := byID[links[id]]
+		if !exists {
+			var made struct{ ID string }
+			if err := sandbarCall(cfg, "POST", "/api/v1/monitors", fields, &made); err != nil {
+				errs = append(errs, fmt.Errorf("%s: %w", spec.Name, err))
+				continue
+			}
+			links[id] = made.ID
+			continue
+		}
+		drift := false
+		for k, v := range fields {
+			if !sameJSON(cur[k], v) {
+				cur[k], drift = v, true
+			}
+		}
+		if drift { // the whole document is replaced, so what Sandbar added (token, retries) rides along
+			if err := sandbarCall(cfg, "PUT", "/api/v1/monitors/"+links[id], cur, nil); err != nil {
+				errs = append(errs, fmt.Errorf("%s: %w", spec.Name, err))
+			}
+		}
+	}
+	for id, rid := range links {
+		if _, still := want[id]; still {
+			continue
+		}
+		if err := sandbarCall(cfg, "DELETE", "/api/v1/monitors/"+rid, nil, nil); err != nil {
+			errs = append(errs, err)
+			continue
+		}
+		delete(links, id)
+	}
+	return errors.Join(errs...)
+}
+
+func sameJSON(a, b any) bool {
+	x, _ := json.Marshal(a)
+	y, _ := json.Marshal(b)
+	if string(x) == "null" { // a field Sandbar left out is its zero value
+		x = []byte(`""`)
+		if _, isNum := b.(int); isNum {
+			x = []byte("0")
+		} else if _, isBool := b.(bool); isBool {
+			x = []byte("false")
+		}
+	}
+	return bytes.Equal(x, y)
+}
+
+func sameLinks(a, b map[string]string) bool {
+	if len(a) != len(b) {
+		return false
+	}
+	for k, v := range a {
+		if b[k] != v {
+			return false
+		}
+	}
+	return true
+}
+
+func (s *Server) sandbarLoop() {
+	t := time.NewTicker(sandbarSyncEvery)
+	defer t.Stop()
+	for {
+		select {
+		case <-s.stop:
+			return
+		case <-t.C:
+			if err := s.syncSandbar(); err != nil {
+				log.Printf("sandbar monitors: %v", err)
+			}
+		}
+	}
+}
+
+// syncSandbarNow is the "Sync now" button: the same sync, with its outcome shown.
+func (s *Server) syncSandbarNow(w http.ResponseWriter, r *http.Request, p *principal) {
+	s.mu.Lock()
+	on := s.st.Settings.Sandbar.Enabled
+	s.mu.Unlock()
+	if !on {
+		fail(w, 400, errors.New("Turn on Sandbar monitors and save the settings first."))
+		return
+	}
+	if err := s.syncSandbar(); err != nil {
+		fail(w, 400, err)
+		return
+	}
+	s.mu.Lock()
+	st := s.sandbarStatus
+	s.mu.Unlock()
+	writeJSON(w, 200, st)
+}
+
+// cleanSandbar validates the settings in `in`, filling in what a blank field means.
+// Caller holds s.mu.
+func (s *Server) cleanSandbar(in *SandbarSync) error {
+	old := s.st.Settings.Sandbar
+	in.URL, in.PolicyID = strings.TrimRight(strings.TrimSpace(in.URL), "/"), strings.TrimSpace(in.PolicyID)
+	if in.Token = strings.TrimSpace(in.Token); in.Token == "" {
+		in.Token = old.Token // blank keeps the saved API key
+	}
+	in.Monitors = old.Monitors // not editable here
+	if in.URL != old.URL {
+		in.Monitors = nil // those belong to the previous server
+	}
+	if !in.Enabled {
+		return nil
+	}
+	if u, err := url.Parse(in.URL); err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
+		return errors.New("enter Sandbar's address, such as https://sandbar.example.com")
+	}
+	if in.Token == "" {
+		return errors.New("enter an administrator's API key from Sandbar's Account page")
+	}
+	return nil
+}

sandbar_test.go

@@ -0,0 +1,350 @@
+package hammerhead
+
+import (
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strconv"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/reef/model"
+)
+
+// fakeSandbar keeps monitors in memory the way Sandbar's /api/v1/monitors does.
+type fakeSandbar struct {
+	URL string
+	mu  sync.Mutex
+	mon map[string]map[string]any
+	n   int
+	// requests that carried the wrong key are counted, not served
+	badAuth int
+	calls   []string
+}
+
+func newFakeSandbar(t *testing.T) *fakeSandbar {
+	f := &fakeSandbar{mon: map[string]map[string]any{}}
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		f.mu.Lock()
+		defer f.mu.Unlock()
+		if r.Header.Get("Authorization") != "Bearer sb_admin" {
+			f.badAuth++
+			http.Error(w, "no", 401)
+			return
+		}
+		f.calls = append(f.calls, r.Method)
+		id := strings.TrimPrefix(r.URL.Path, "/api/v1/monitors/")
+		switch {
+		case r.Method == "GET" && r.URL.Path == "/api/v1/monitors":
+			out := []any{}
+			for _, m := range f.mon {
+				out = append(out, m)
+			}
+			json.NewEncoder(w).Encode(out)
+		case r.Method == "POST" && r.URL.Path == "/api/v1/monitors":
+			var m map[string]any
+			json.NewDecoder(r.Body).Decode(&m)
+			f.n++
+			m["id"], m["retries"] = "mon"+strconv.Itoa(f.n), 2 // Sandbar adds its own fields
+			f.mon[m["id"].(string)] = m
+			json.NewEncoder(w).Encode(m)
+		case r.Method == "PUT" && f.mon[id] != nil:
+			var m map[string]any
+			json.NewDecoder(r.Body).Decode(&m)
+			f.mon[id] = m
+			json.NewEncoder(w).Encode(m)
+		case r.Method == "DELETE":
+			delete(f.mon, id)
+			w.Write([]byte(`{"ok":true}`))
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	t.Cleanup(ts.Close)
+	f.URL = ts.URL
+	return f
+}
+
+func (f *fakeSandbar) names() map[string]string { // monitor id -> "kind target"
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	out := map[string]string{}
+	for id, m := range f.mon {
+		out[id] = m["kind"].(string) + " " + m["target"].(string)
+	}
+	return out
+}
+
+func (f *fakeSandbar) count(method string) int {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	n := 0
+	for _, c := range f.calls {
+		if c == method {
+			n++
+		}
+	}
+	return n
+}
+
+func enableSandbarSync(f *fx, sb *fakeSandbar) {
+	f.s.mu.Lock()
+	f.s.st.Edges[0].IPs = []string{"10.0.0.5", "203.0.113.7"}
+	f.s.st.Settings.Sandbar = SandbarSync{Enabled: true, URL: sb.URL, Token: "sb_admin", PolicyID: "pol1", Public: true, AllowPrivate: true}
+	f.s.mu.Unlock()
+}
+
+func TestSandbarMonitorsFollowTheResources(t *testing.T) {
+	f := newFx(t)
+	sb := newFakeSandbar(t)
+	enableSandbarSync(f, sb)
+	f.s.mu.Lock()
+	f.s.st.Mappings = append(f.s.st.Mappings,
+		&model.Mapping{ID: "m2", OrgID: f.org, Name: "Blog", EdgeID: "e1", Proto: "https", Port: 443, Hostname: "blog.example.com", HealthPath: "/ping", Enabled: true,
+			Targets: []model.Target{{SiteID: "s1", Host: "192.168.1.31", Port: 80}}},
+		&model.Mapping{ID: "m3", OrgID: f.org, Name: "Docs", EdgeID: "e1", Proto: "https", Port: 8443, Hostname: "docs.example.com", Enabled: true, Auth: model.Auth{Password: true},
+			Targets: []model.Target{{SiteID: "s1", Host: "192.168.1.32", Port: 80}}},
+		&model.Mapping{ID: "m4", OrgID: f.org, Name: "Unchecked", EdgeID: "e1", Proto: "tcp", Port: 2222, HealthSeconds: -1, Enabled: true},
+		&model.Mapping{ID: "m5", OrgID: f.org, Name: "Voice", EdgeID: "e1", Proto: "udp", Port: 9987, Enabled: true},
+		&model.Mapping{ID: "m6", OrgID: f.org, Name: "Wildcard", EdgeID: "e1", Proto: "https", Port: 443, Hostname: "*.example.com", Enabled: true},
+		&model.Mapping{ID: "m7", OrgID: f.org, Name: "Off", EdgeID: "e1", Proto: "tcp", Port: 3333},
+	)
+	f.s.mu.Unlock()
+
+	if err := f.s.syncSandbar(); err != nil {
+		t.Fatal(err)
+	}
+	got := sb.names()
+	want := map[string]bool{"tcp 203.0.113.7:25565": true, "http https://blog.example.com/ping": true, "http https://docs.example.com:8443": true}
+	if len(got) != len(want) {
+		t.Fatalf("monitors created: %v", got)
+	}
+	for _, v := range got {
+		if !want[v] {
+			t.Errorf("unexpected monitor %q", v)
+		}
+	}
+	for _, m := range sb.mon {
+		if m["policyId"] != "pol1" || m["public"] != true || m["enabled"] != true {
+			t.Errorf("monitor settings not applied: %v", m)
+		}
+		if m["name"] == "Docs" && m["expect"] != "200-399" {
+			t.Errorf("a protected site should accept its sign-in redirect: %v", m)
+		}
+	}
+
+	// A second pass changes nothing.
+	posts := sb.count("POST")
+	if err := f.s.syncSandbar(); err != nil {
+		t.Fatal(err)
+	}
+	if sb.count("POST") != posts || sb.count("PUT") != 0 || sb.count("DELETE") != 0 {
+		t.Fatalf("a settled sync still wrote: %v", sb.calls)
+	}
+
+	// A rename and a changed policy are pushed, and Sandbar's own fields survive.
+	f.s.mu.Lock()
+	f.s.findMapping("m1").Name = "Minecraft Java"
+	f.s.st.Settings.Sandbar.PolicyID = "pol2"
+	f.s.mu.Unlock()
+	if err := f.s.syncSandbar(); err != nil {
+		t.Fatal(err)
+	}
+	var renamed bool
+	for _, m := range sb.mon {
+		if m["policyId"] != "pol2" || m["retries"] != float64(2) {
+			t.Errorf("update lost data or missed the policy: %v", m)
+		}
+		renamed = renamed || m["name"] == "Minecraft Java"
+	}
+	if !renamed {
+		t.Error("the rename was not pushed")
+	}
+
+	// Deleting a resource, or turning its checks off, removes its monitor; one added by hand stays.
+	sb.mu.Lock()
+	sb.mon["hand"] = map[string]any{"id": "hand", "name": "mine", "kind": "tcp", "target": "x:1"}
+	sb.mu.Unlock()
+	f.s.mu.Lock()
+	f.s.findMapping("m2").HealthSeconds = -1
+	f.s.st.Mappings = filter(f.s.st.Mappings, func(m *model.Mapping) bool { return m.ID != "m3" })
+	f.s.mu.Unlock()
+	if err := f.s.syncSandbar(); err != nil {
+		t.Fatal(err)
+	}
+	got = sb.names()
+	if len(got) != 2 || got["hand"] == "" {
+		t.Fatalf("after removing resources: %v", got)
+	}
+	f.s.mu.Lock()
+	links := len(f.s.st.Settings.Sandbar.Monitors)
+	f.s.mu.Unlock()
+	if links != 1 {
+		t.Errorf("links kept for %d resources, want 1", links)
+	}
+}
+
+func TestSandbarSyncRecreatesAMonitorDeletedInSandbarAndSurvivesARestart(t *testing.T) {
+	f := newFx(t)
+	sb := newFakeSandbar(t)
+	enableSandbarSync(f, sb)
+	if err := f.s.syncSandbar(); err != nil {
+		t.Fatal(err)
+	}
+	sb.mu.Lock()
+	sb.mon = map[string]map[string]any{}
+	sb.mu.Unlock()
+	if err := f.s.syncSandbar(); err != nil {
+		t.Fatal(err)
+	}
+	if len(sb.names()) != 1 {
+		t.Fatalf("not recreated: %v", sb.names())
+	}
+
+	// the links are stored, so a restart does not create a second monitor
+	f.s.mu.Lock()
+	err := f.s.save()
+	f.s.mu.Unlock()
+	if err != nil {
+		t.Fatal(err)
+	}
+	dir, key := f.s.dir, f.s.opts.MasterKey
+	f.s.Close()
+	s2, err := New(dir, Options{MasterKey: key, AlertInterval: time.Hour})
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s2.Close()
+	if err := s2.syncSandbar(); err != nil {
+		t.Fatal(err)
+	}
+	if len(sb.names()) != 1 {
+		t.Fatalf("a restart duplicated the monitor: %v", sb.names())
+	}
+}
+
+func TestSandbarSyncReportsABadKeyAndIsOffByDefault(t *testing.T) {
+	f := newFx(t)
+	sb := newFakeSandbar(t)
+	if err := f.s.syncSandbar(); err != nil || sb.badAuth != 0 || len(sb.calls) != 0 {
+		t.Fatal("sync ran while switched off")
+	}
+	enableSandbarSync(f, sb)
+	f.s.mu.Lock()
+	f.s.st.Settings.Sandbar.Token = "wrong"
+	f.s.mu.Unlock()
+	if err := f.s.syncSandbar(); err == nil || !strings.Contains(err.Error(), "API key") {
+		t.Fatalf("error: %v", err)
+	}
+	f.s.mu.Lock()
+	msg := f.s.sandbarStatus.Error
+	f.s.mu.Unlock()
+	if msg == "" {
+		t.Error("the failure was not recorded for the settings page")
+	}
+}
+
+func TestSandbarSettingsKeepTheKeySealedAndHidden(t *testing.T) {
+	f := newFx(t)
+	f.s.mu.Lock()
+	in := SandbarSync{Enabled: true, URL: " https://sandbar.example.com/ ", Token: " sb_admin "}
+	err := f.s.cleanSandbar(&in)
+	f.s.st.Settings.Sandbar = in
+	f.s.save()
+	f.s.mu.Unlock()
+	if err != nil || in.URL != "https://sandbar.example.com" || in.Token != "sb_admin" {
+		t.Fatalf("%v %+v", err, in)
+	}
+
+	// a blank key in a later save keeps the saved one; a new address forgets the old server's links
+	f.s.mu.Lock()
+	f.s.st.Settings.Sandbar.Monitors = map[string]string{"m1": "mon1"}
+	next := SandbarSync{Enabled: true, URL: "https://other.example.com"}
+	err = f.s.cleanSandbar(&next)
+	f.s.mu.Unlock()
+	if err != nil || next.Token != "sb_admin" || next.Monitors != nil {
+		t.Fatalf("%v %+v", err, next)
+	}
+	for _, bad := range []SandbarSync{{Enabled: true, URL: "not a url", Token: "k"}, {Enabled: true, URL: "https://x.example.com", Token: ""}} {
+		bad := bad
+		f.s.mu.Lock()
+		f.s.st.Settings.Sandbar.Token = ""
+		err := f.s.cleanSandbar(&bad)
+		f.s.mu.Unlock()
+		if err == nil {
+			t.Errorf("accepted %+v", bad)
+		}
+	}
+
+	// at rest the key is encrypted, and the settings API never returns it
+	raw, _ := json.Marshal(f.s.db.sealed(f.s.st.Settings))
+	if strings.Contains(string(raw), "sb_admin") {
+		t.Error("the API key is stored in the clear")
+	}
+	f.s.mu.Lock()
+	f.s.st.Settings.Sandbar.Token = "sb_admin"
+	f.s.mu.Unlock()
+	rec := httptest.NewRecorder()
+	f.s.getSettings(rec, httptest.NewRequest("GET", "/api/admin/settings", nil), nil)
+	if strings.Contains(rec.Body.String(), "sb_admin") || !strings.Contains(rec.Body.String(), `"sandbarTokenSet":true`) {
+		t.Errorf("settings response: %s", rec.Body.String())
+	}
+}
+
+func TestTopicChannelPublishes(t *testing.T) {
+	f := newFx(t)
+	var gotPath, gotTitle, gotPrio, gotAuth, gotBody string
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		b := make([]byte, 512)
+		n, _ := r.Body.Read(b)
+		gotPath, gotTitle, gotPrio, gotAuth, gotBody = r.URL.Path, r.Header.Get("Title"), r.Header.Get("Priority"), r.Header.Get("Authorization"), string(b[:n])
+	}))
+	defer ts.Close()
+	ch := AlertChannel{ID: "t1", Kind: "topic", Target: ts.URL + "/", Topic: "ops", Token: "sb_k", AllowPrivate: true, Enabled: true}
+	n := alertNote{Status: "firing", Org: "Acme", Name: "Minecraft", Message: "Minecraft can't listen", Since: time.Now()}
+	if err := f.s.send(ch, n); err != nil {
+		t.Fatal(err)
+	}
+	if gotPath != "/ops" || gotTitle != "[Hammerhead] Problem: Minecraft" || gotPrio != "urgent" || gotAuth != "Bearer sb_k" || !strings.Contains(gotBody, "can't listen") {
+		t.Errorf("published %q %q %q %q %q", gotPath, gotTitle, gotPrio, gotAuth, gotBody)
+	}
+	n.Status, ch.Token = "resolved", ""
+	if err := f.s.send(ch, n); err != nil {
+		t.Fatal(err)
+	}
+	if gotPrio != "default" || gotAuth != "" || !strings.HasPrefix(gotBody, "Resolved") {
+		t.Errorf("recovery %q %q %q", gotPrio, gotAuth, gotBody)
+	}
+
+	// a refusal surfaces as an error
+	deny := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { http.Error(w, "no", 401) }))
+	defer deny.Close()
+	ch.Target = deny.URL
+	if err := f.s.send(ch, n); err == nil || !strings.Contains(err.Error(), "API key") {
+		t.Errorf("error: %v", err)
+	}
+}
+
+func TestTopicChannelValidation(t *testing.T) {
+	f := newFx(t)
+	for _, ch := range []AlertChannel{
+		{Kind: "topic", Target: "not a url", Topic: "ops"},
+		{Kind: "topic", Target: "https://sandbar.example.com", Topic: ""},
+		{Kind: "topic", Target: "https://sandbar.example.com", Topic: "has space"},
+		{Kind: "topic", Target: "https://sandbar.example.com", Topic: "ops", AllowPrivate: true}, // needs a trusted caller
+	} {
+		ch := ch
+		if err := f.s.validateChannel(&ch, false); err == nil {
+			t.Fatalf("accepted %+v", ch)
+		}
+	}
+	ok := AlertChannel{Kind: "topic", Target: "https://sandbar.example.com", Topic: " ops ", Token: " ", Policy: "x", Format: "slack"}
+	if err := f.s.validateChannel(&ok, false); err != nil || ok.Topic != "ops" || ok.Token != "" || ok.Policy != "" || ok.Format != "" {
+		t.Fatalf("%v %+v", err, ok)
+	}
+	if p := preview(ok); p != "https://sandbar.example.com/ops" {
+		t.Errorf("preview %q", p)
+	}
+}

server.go

@@ -45,6 +45,10 @@ func (s *Server) Handler() http.Handler {
 
 	s.registerAlertRoutes(mux)
 	mux.HandleFunc("GET /metrics", s.metrics)
+	mux.Handle("GET /static/term/", terminalStatic())
+	mux.HandleFunc("GET /api/orgs/{org}/terminal", s.withOrg(true, s.terminalKey))
+	mux.HandleFunc("GET /api/orgs/{org}/sites/{id}/terminal", s.withOrg(true, s.siteTerminal))
+	mux.HandleFunc("DELETE /api/orgs/{org}/sites/{id}/terminal/hostkey", s.withOrg(true, s.forgetHostKey))
 
 	// public
 	mux.HandleFunc("GET /api/public", s.publicInfo)
@@ -79,13 +83,17 @@ func (s *Server) Handler() http.Handler {
 	mux.HandleFunc("GET /api/me/sessions", s.withUser(s.mySessions))
 	mux.HandleFunc("DELETE /api/me/sessions/{id}", s.withUser(s.endSession))
 	mux.HandleFunc("POST /api/me/sessions/revoke-others", s.withUser(s.endOtherSessions))
-	mux.HandleFunc("GET /api/edges", s.withUser(s.listEdges))
+	mux.HandleFunc("GET /api/edges", s.listEdges)
 
 	// server administrators
 	mux.HandleFunc("POST /api/edges", s.withServerAdmin(s.saveEdge))
 	mux.HandleFunc("PUT /api/edges/{id}", s.withServerAdmin(s.saveEdge))
 	mux.HandleFunc("DELETE /api/edges/{id}", s.withServerAdmin(s.deleteEdge))
+	mux.HandleFunc("GET /api/orgs", s.listOrgs)
 	mux.HandleFunc("POST /api/orgs", s.withServerAdmin(s.createOrg))
+	mux.HandleFunc("GET /api/admin/apikeys", s.withServerAdminUser(s.listServerKeys))
+	mux.HandleFunc("POST /api/admin/apikeys", s.withServerAdminUser(s.createServerKey))
+	mux.HandleFunc("DELETE /api/admin/apikeys/{id}", s.withServerAdminUser(s.deleteServerKey))
 	mux.HandleFunc("GET /api/admin/users", s.withServerAdmin(s.adminUsers))
 	mux.HandleFunc("POST /api/admin/users", s.withServerAdmin(s.adminCreateUser))
 	mux.HandleFunc("PUT /api/admin/users/{id}", s.withServerAdmin(s.adminUpdateUser))
@@ -97,6 +105,7 @@ func (s *Server) Handler() http.Handler {
 	mux.HandleFunc("GET /api/admin/settings", s.withServerAdmin(s.getSettings))
 	mux.HandleFunc("PUT /api/admin/settings", s.withServerAdmin(s.putSettings))
 	mux.HandleFunc("POST /api/admin/settings/test-mail", s.withServerAdmin(s.testMail))
+	mux.HandleFunc("POST /api/admin/settings/sandbar-sync", s.withServerAdmin(s.syncSandbarNow))
 	mux.HandleFunc("GET /api/admin/update", s.withServerAdmin(s.checkUpdate))
 	mux.HandleFunc("POST /api/admin/update/apply", s.withServerAdmin(s.applyUpdate))
 	mux.HandleFunc("GET /api/admin/update/status", s.withServerAdmin(s.updateStatus))
@@ -130,6 +139,7 @@ func (s *Server) Handler() http.Handler {
 	mux.HandleFunc("PUT /api/orgs/{org}/sites/{id}", s.withOrg(true, s.putSite))
 	mux.HandleFunc("DELETE /api/orgs/{org}/sites/{id}", s.withOrg(true, s.deleteSite))
 	mux.HandleFunc("GET /api/orgs/{org}/sites/{id}/containers", s.withOrg(true, s.siteContainers))
+	mux.HandleFunc("GET /api/orgs/{org}/sites/{id}/discovered", s.withOrg(true, s.siteDiscovered))
 	mux.HandleFunc("POST /api/orgs/{org}/domains", s.withOrg(true, s.addDomain))
 	mux.HandleFunc("DELETE /api/orgs/{org}/domains/{id}", s.withOrg(true, s.deleteDomain))
 	mux.HandleFunc("POST /api/orgs/{org}/domains/{id}/check", s.withOrg(true, s.checkDomain))
@@ -140,15 +150,24 @@ func (s *Server) Handler() http.Handler {
 	mux.HandleFunc("POST /api/orgs/{org}/mappings/{id}/shares", s.withOrg(true, s.createShare))
 	mux.HandleFunc("POST /api/orgs/{org}/mappings/{id}/revoke", s.withOrg(true, s.revokeVisitors))
 	mux.HandleFunc("DELETE /api/orgs/{org}/mappings/{id}/shares/{share}", s.withOrg(true, s.deleteShare))
+	mux.HandleFunc("POST /api/orgs/{org}/expose", s.withOrg(true, s.createExpose))
+	mux.HandleFunc("GET /api/orgs/{org}/expose", s.withOrg(true, s.listExpose))
+	mux.HandleFunc("GET /api/orgs/{org}/expose/{id}", s.withOrg(true, s.getExpose))
+	mux.HandleFunc("POST /api/orgs/{org}/expose/{id}/renew", s.withOrg(true, s.renewExpose))
+	mux.HandleFunc("DELETE /api/orgs/{org}/expose/{id}", s.withOrg(true, s.deleteExpose))
 	mux.HandleFunc("POST /api/orgs/{org}/mail-egress", s.withOrg(true, s.saveMailEgress))
 	mux.HandleFunc("PUT /api/orgs/{org}/mail-egress/{id}", s.withOrg(true, s.saveMailEgress))
 	mux.HandleFunc("DELETE /api/orgs/{org}/mail-egress/{id}", s.withOrg(true, s.deleteMailEgress))
 	mux.HandleFunc("GET /api/orgs/{org}/dkim", s.withOrg(true, s.listDKIM))
 	mux.HandleFunc("POST /api/orgs/{org}/dkim", s.withOrg(true, s.createDKIM))
 	mux.HandleFunc("DELETE /api/orgs/{org}/dkim/{id}", s.withOrg(true, s.deleteDKIM))
+	mux.HandleFunc("GET /api/orgs/{org}/dmarc", s.withOrg(true, s.listDMARC))
+	mux.HandleFunc("POST /api/orgs/{org}/dmarc", s.withOrg(true, s.uploadDMARC))
+	mux.HandleFunc("DELETE /api/orgs/{org}/dmarc/{domain}", s.withOrg(true, s.deleteDMARC))
 	mux.HandleFunc("GET /api/orgs/{org}/logs", s.withOrg(true, s.orgLogs))
 	mux.HandleFunc("GET /api/orgs/{org}/stats", s.withOrg(false, s.orgStats))
 	mux.HandleFunc("GET /api/orgs/{org}/traffic", s.withOrg(false, s.orgTraffic))
+	mux.HandleFunc("GET /api/orgs/{org}/traffic/{id}", s.withOrg(false, s.orgTrafficDetail))
 
 	// agents
 	mux.HandleFunc("GET /edge/config", s.edgeConfig)
@@ -156,5 +175,6 @@ func (s *Server) Handler() http.Handler {
 	mux.HandleFunc("POST /edge/logs", s.edgeLogs)
 	mux.HandleFunc("GET /site/config", s.siteConfig)
 	mux.HandleFunc("POST /site/report", s.siteReport)
+	mux.HandleFunc("GET /site/terminal/{id}", s.terminalCallBack)
 	return mux
 }

settings.go

@@ -17,7 +17,9 @@ func (s *Server) getSettings(w http.ResponseWriter, r *http.Request, p *principa
 	defer s.mu.Unlock()
 	st := s.st.Settings
 	st.SMTP.Pass = ""                               // write-only
-	writeJSON(w, 200, map[string]any{"settings": st, "smtpPassSet": s.st.Settings.SMTP.Pass != ""})
+	st.Sandbar.Token, st.Sandbar.Monitors = "", nil // write-only
+	st.Terminal = TerminalSettings{}                // the private key never leaves the server
+	writeJSON(w, 200, map[string]any{"settings": st, "smtpPassSet": s.st.Settings.SMTP.Pass != "", "sandbarTokenSet": s.st.Settings.Sandbar.Token != "", "sandbarSync": s.sandbarStatus})
 }
 
 func (s *Server) putSettings(w http.ResponseWriter, r *http.Request, p *principal) {
@@ -38,10 +40,16 @@ func (s *Server) putSettings(w http.ResponseWriter, r *http.Request, p *principa
 	if in.SMTP.Pass == "" {
 		in.SMTP.Pass = s.st.Settings.SMTP.Pass // blank keeps the saved password
 	}
+	if err := s.cleanSandbar(&in.Sandbar); err != nil {
+		s.mu.Unlock()
+		fail(w, 400, err)
+		return
+	}
 	if s.opts.DashboardURL != "" {
 		in.DashboardURL = s.st.Settings.DashboardURL // pinned by flag/env
 	}
 	in.SelfEdgeID = s.st.Settings.SelfEdgeID // not editable here; losing it would make the next start create a second self edge
+	in.Terminal = s.st.Settings.Terminal     // the terminal key and trusted host keys are not editable here either
 	s.st.Settings = in
 	s.bump() // edges receive the dashboard URL and ACME settings
 	s.mu.Unlock()

store.go

@@ -29,6 +29,10 @@ type Org struct {
 
 	AlertDebounceMin int `json:"alertDebounceMin,omitempty"` // minutes a problem must last before alerting; 0 = 2
 	AlertRepeatHours int `json:"alertRepeatHours,omitempty"` // remind about an unresolved problem this often; 0 = never
+
+	// Traffic alerts, applied to every port of the organization. 0 = off.
+	TrafficSpikeFactor int `json:"trafficSpikeFactor,omitempty"` // alert when a port carries this many times its usual traffic
+	TrafficDailyMB     int `json:"trafficDailyMb,omitempty"`     // alert when a port carries more than this many MB in one UTC day
 }
 
 type Role struct {
@@ -113,8 +117,11 @@ type Settings struct {
 	AcmeEmail    string      `json:"acmeEmail"`
 	AcmeStaging  bool        `json:"acmeStaging"`
 	SMTP         SMTP        `json:"smtp"`
+	Sandbar      SandbarSync `json:"sandbar"`
 	// SelfEdgeID is this server's own edge (see HAMMERHEAD_SELF_DOMAIN).
 	SelfEdgeID string `json:"selfEdgeId,omitempty"`
+	// Terminal holds the web terminal's SSH key and the host keys it has seen (see terminal.go).
+	Terminal TerminalSettings `json:"terminal"`
 }
 
 type Secret struct {
@@ -202,6 +209,8 @@ type Server struct {
 	mailStatus map[string]string                    // mail egress ID -> "" (listening) or an error
 	mailChecks map[string]model.MailCheck           // mail egress ID -> the edge's deliverability report
 	containers map[string][]model.Container         // by site ID
+	discovered map[string][]model.DiscoveredService // by site ID: suggested targets the agent found
+	terms      map[string]*pendingTerm              // terminal sessions waiting for an agent to call back, by session ID
 	changed    chan struct{}
 
 	tickets map[string]ticket
@@ -218,6 +227,11 @@ type Server struct {
 	verAt    time.Time
 	mtaSent  map[string]time.Time // reminder throttle for repeat notifications
 
+	expose map[string]time.Time // deadline of each ephemeral mapping, by ID (see expose.go)
+
+	sandbarMu     sync.Mutex        // one monitor sync at a time (see sandbar.go)
+	sandbarStatus sandbarSyncStatus // how the last one went
+
 	logs *logStore
 	db   *DB
 	stop chan struct{}
@@ -249,6 +263,8 @@ func New(dir string, opts Options) (*Server, error) {
 		mailStatus: map[string]string{},
 		mailChecks: map[string]model.MailCheck{},
 		containers: map[string][]model.Container{},
+		discovered: map[string][]model.DiscoveredService{},
+		terms:      map[string]*pendingTerm{},
 		changed:    make(chan struct{}),
 		tickets:    map[string]ticket{},
 		oauth:      map[string]oauthState{},
@@ -258,6 +274,7 @@ func New(dir string, opts Options) (*Server, error) {
 		alerts:     map[string]*alertState{},
 		mtaSent:    map[string]time.Time{},
 		updRuns:    map[string]*updRun{},
+		expose:     map[string]time.Time{},
 	}
 	s.st.Sessions = map[string]*Session{}
 	s.st.Secrets = map[string]*Secret{}
@@ -298,7 +315,9 @@ func New(dir string, opts Options) (*Server, error) {
 	}
 	s.loadAlerts()
 	go s.housekeeping()
+	go s.exposeLoop()
 	go s.alertLoop()
+	go s.sandbarLoop()
 	return s, nil
 }
 
@@ -395,9 +414,15 @@ func (s *Server) bump() {
 }
 
 // wait blocks until the version moves past have, or 25s pass. It returns false on timeout.
-func (s *Server) wait(r *http.Request, have int64) bool {
+func (s *Server) wait(r *http.Request, have int64) bool { return s.waitUntil(r, have, nil) }
+
+// waitUntil is wait that also returns at once when ready (called with s.mu held) says there is
+// something to deliver that does not move the version, such as a terminal waiting for an agent.
+// Checking it under the same lock that takes the change channel is what keeps a request that
+// arrives between two polls from being missed.
+func (s *Server) waitUntil(r *http.Request, have int64, ready func() bool) bool {
 	s.mu.Lock()
-	if s.st.Version > have {
+	if s.st.Version > have || (ready != nil && ready()) {
 		s.mu.Unlock()
 		return true
 	}

terminal.go

@@ -0,0 +1,581 @@
+package hammerhead
+
+import (
+	"crypto/ed25519"
+	"crypto/rand"
+	"crypto/x509"
+	"embed"
+	"encoding/json"
+	"encoding/pem"
+	"errors"
+	"fmt"
+	"io"
+	"io/fs"
+	"net"
+	"net/http"
+	"net/url"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"git.veillax.com/Veillax/reef/model"
+	"golang.org/x/crypto/ssh"
+	"golang.org/x/net/websocket"
+)
+
+// The web terminal opens a shell on a site from the dashboard. Hammerhead is the SSH client: it
+// signs in to the SSH port that an edge exposes for the site (a TCP resource whose target is
+// Bamboo's built-in SSH server) with a key of its own, and relays the shell to the browser over a
+// WebSocket. Bamboo only lets in keys listed in its authorized_keys, so a site is reachable this
+// way only after someone puts Hammerhead's public key there.
+
+//go:embed web/term
+var termFiles embed.FS
+
+const (
+	sshTargetPort    = 2222 // where Bamboo's SSH server listens by default
+	termIdle         = 30 * time.Minute
+	termMaxSessions  = 8
+	termMaxFrame     = 64 << 10
+	termDialTimeout  = 10 * time.Second
+	termDefaultTerm  = "xterm-256color"
+	termResizeMaxCol = 500
+	termResizeMaxRow = 200
+)
+
+// TerminalSettings is what the web terminal remembers: its own SSH key, and the host key each
+// site showed the first time (trusted on first use, like an ordinary ssh client).
+type TerminalSettings struct {
+	Key   string            `json:"key,omitempty"`   // PKCS#8 PEM private key; sealed at rest, never sent to the browser
+	Hosts map[string]string `json:"hosts,omitempty"` // site ID -> host key fingerprint
+}
+
+// terminalStatic serves the embedded xterm.js files.
+func terminalStatic() http.Handler {
+	sub, _ := fs.Sub(termFiles, "web/term")
+	fileServer := http.StripPrefix("/static/term/", http.FileServer(http.FS(sub)))
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Cache-Control", "public, max-age=86400")
+		w.Header().Set("X-Content-Type-Options", "nosniff")
+		fileServer.ServeHTTP(w, r)
+	})
+}
+
+// terminalSigner returns Hammerhead's SSH key, making it on first use. Caller holds s.mu.
+func (s *Server) terminalSignerLocked() (ssh.Signer, error) {
+	if s.st.Settings.Terminal.Key == "" {
+		_, priv, err := ed25519.GenerateKey(rand.Reader)
+		if err != nil {
+			return nil, err
+		}
+		der, err := x509.MarshalPKCS8PrivateKey(priv)
+		if err != nil {
+			return nil, err
+		}
+		s.st.Settings.Terminal.Key = string(pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: der}))
+		s.save()
+	}
+	return ssh.ParsePrivateKey([]byte(s.st.Settings.Terminal.Key))
+}
+
+// terminalKey tells an administrator which public key to put in a site's authorized_keys.
+func (s *Server) terminalKey(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	s.mu.Lock()
+	signer, err := s.terminalSignerLocked()
+	s.mu.Unlock()
+	if err != nil {
+		fail(w, 500, err)
+		return
+	}
+	line := strings.TrimSpace(string(ssh.MarshalAuthorizedKey(signer.PublicKey()))) + " hammerhead"
+	writeJSON(w, 200, map[string]any{"publicKey": line, "fingerprint": ssh.FingerprintSHA256(signer.PublicKey())})
+}
+
+// isSSHResource reports whether m is a TCP port that leads to the SSH server of the given site.
+func isSSHResource(m *model.Mapping, siteID string) bool {
+	if m.Proto != model.ProtoTCP || m.IsRedirect() || !m.Enabled {
+		return false
+	}
+	for _, t := range m.Targets {
+		if t.SiteID != siteID {
+			continue
+		}
+		if t.Port == sshTargetPort {
+			return true
+		}
+		for _, tag := range m.Tags {
+			if tag == "ssh" {
+				return true
+			}
+		}
+	}
+	return false
+}
+
+// terminalAddr finds where to connect for a site: the public address of the SSH port that
+// leads to it. A port bound to every address is reached through the edge's own address.
+func (s *Server) terminalAddr(c *orgCtx, siteID string) (string, error) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	st := s.findSite(siteID)
+	if st == nil || st.OrgID != c.Org.ID || st.Local {
+		return "", errors.New("no such site")
+	}
+	for _, m := range s.st.Mappings {
+		if m.OrgID != c.Org.ID || !isSSHResource(m, siteID) {
+			continue
+		}
+		host := m.BindIP
+		if ip := net.ParseIP(host); host == "" || (ip != nil && ip.IsUnspecified()) {
+			e := s.findEdge(m.EdgeID)
+			if e == nil {
+				continue
+			}
+			h, _, err := net.SplitHostPort(e.TunnelAddr)
+			if err != nil || h == "" {
+				continue
+			}
+			host = h
+		}
+		return net.JoinHostPort(host, strconv.Itoa(m.Port)), nil
+	}
+	return "", errors.New("This site has no SSH port yet. Open one from the SSH button on the Sites page.")
+}
+
+type hostKeyChanged struct{ site, was, now string }
+
+func (e *hostKeyChanged) Error() string {
+	return fmt.Sprintf("The host key of %s has changed (was %s, now %s). If you reinstalled Bamboo on it, forget the saved key and try again.", e.site, e.was, e.now)
+}
+
+// trustHostKey remembers a site's host key the first time and refuses a different one afterwards.
+func (s *Server) trustHostKey(siteID, siteName string) ssh.HostKeyCallback {
+	return func(_ string, _ net.Addr, key ssh.PublicKey) error {
+		fp := ssh.FingerprintSHA256(key)
+		s.mu.Lock()
+		defer s.mu.Unlock()
+		t := &s.st.Settings.Terminal
+		switch known := t.Hosts[siteID]; {
+		case known == "":
+			if t.Hosts == nil {
+				t.Hosts = map[string]string{}
+			}
+			t.Hosts[siteID] = fp
+			s.save()
+			return nil
+		case known != fp:
+			return &hostKeyChanged{siteName, known, fp}
+		}
+		return nil
+	}
+}
+
+func (s *Server) forgetHostKey(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	id := r.PathValue("id")
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if st := s.findSite(id); st == nil || st.OrgID != c.Org.ID {
+		fail(w, 404, errors.New("no such site"))
+		return
+	}
+	delete(s.st.Settings.Terminal.Hosts, id)
+	s.save()
+	s.audit(c.Org.ID, c.Actor, "terminal.forget-host-key", id)
+	ok(w)
+}
+
+// sameOrigin lets a browser open the terminal only from this dashboard: a page on another site
+// cannot use a signed-in administrator's cookie to start a shell. Requests without an Origin
+// header do not come from a browser page.
+func sameOrigin(r *http.Request) bool {
+	o := r.Header.Get("Origin")
+	if o == "" {
+		return true
+	}
+	u, err := url.Parse(o)
+	return err == nil && strings.EqualFold(u.Host, r.Host)
+}
+
+type termFrame struct {
+	text bool
+	data []byte
+}
+
+// frames reads and writes WebSocket frames keeping their type: binary frames carry the terminal's
+// bytes, text frames carry control messages (resize, exit).
+var frames = websocket.Codec{
+	Marshal: func(v any) ([]byte, byte, error) {
+		f := v.(termFrame)
+		if f.text {
+			return f.data, websocket.TextFrame, nil
+		}
+		return f.data, websocket.BinaryFrame, nil
+	},
+	Unmarshal: func(data []byte, pt byte, v any) error {
+		*(v.(*termFrame)) = termFrame{text: pt == websocket.TextFrame, data: data}
+		return nil
+	},
+}
+
+type termControl struct {
+	Type string `json:"type"`
+	Cols int    `json:"cols,omitempty"`
+	Rows int    `json:"rows,omitempty"`
+	Code int    `json:"code,omitempty"`
+
+	Message        string `json:"message,omitempty"`        // type "error": what went wrong, in words for the person at the keyboard
+	HostKeyChanged bool   `json:"hostKeyChanged,omitempty"` // type "error": the site's host key is not the one saved, so offer to forget it
+}
+
+func clamp(v, lo, hi int) int {
+	if v < lo {
+		return lo
+	}
+	if v > hi {
+		return hi
+	}
+	return v
+}
+
+var termSlots = make(chan struct{}, termMaxSessions)
+
+// siteTerminal is the WebSocket that carries one shell session. The connection is accepted first
+// and any problem (no SSH port, a refused key, a changed host key) is sent as an "error" message,
+// because a browser cannot read the body of a failed WebSocket handshake.
+func (s *Server) siteTerminal(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	if !sameOrigin(r) {
+		fail(w, 403, errors.New("The terminal can only be opened from the dashboard itself."))
+		return
+	}
+	siteID := r.PathValue("id")
+	cols, _ := strconv.Atoi(r.URL.Query().Get("cols"))
+	rows, _ := strconv.Atoi(r.URL.Query().Get("rows"))
+	cols, rows = clamp(cols, 20, termResizeMaxCol), clamp(rows, 5, termResizeMaxRow)
+
+	srv := websocket.Server{
+		Handshake: func(*websocket.Config, *http.Request) error { return nil }, // the origin was checked above
+		Handler: func(ws *websocket.Conn) {
+			ws.MaxPayloadBytes = termMaxFrame
+			s.runTerminal(ws, c, siteID, cols, rows)
+		},
+	}
+	srv.ServeHTTP(w, r)
+}
+
+func (s *Server) runTerminal(ws *websocket.Conn, c *orgCtx, siteID string, cols, rows int) {
+	var wmu sync.Mutex
+	send := func(f termFrame) error {
+		wmu.Lock()
+		defer wmu.Unlock()
+		return frames.Send(ws, f)
+	}
+	control := func(v termControl) {
+		msg, _ := json.Marshal(v)
+		send(termFrame{text: true, data: msg})
+	}
+	problem := func(msg string) {
+		control(termControl{Type: "error", Message: msg})
+		time.Sleep(200 * time.Millisecond) // let it reach the browser before the connection closes
+	}
+	defer ws.Close()
+
+	s.mu.Lock()
+	signer, err := s.terminalSignerLocked()
+	siteName, viaAgent, agentCanRelay := "", false, false
+	if st := s.findSite(siteID); st != nil {
+		siteName, viaAgent, agentCanRelay = st.Name, st.SSH != "" && st.SSHRelay, st.SSHRelay
+	}
+	s.mu.Unlock()
+	if err != nil {
+		problem(err.Error())
+		return
+	}
+	select {
+	case termSlots <- struct{}{}:
+		defer func() { <-termSlots }()
+	default:
+		problem("Too many terminals are open. Close one and try again.")
+		return
+	}
+
+	cfg := &ssh.ClientConfig{User: "hammerhead", Auth: []ssh.AuthMethod{ssh.PublicKeys(signer)},
+		HostKeyCallback: s.trustHostKey(siteID, siteName), Timeout: termDialTimeout}
+	var client *ssh.Client
+	if viaAgent {
+		// Bamboo reports its SSH server, so the agent calls back and no port has to be open anywhere.
+		var release func()
+		client, release, err = s.dialViaAgent(siteID, cfg)
+		defer func() {
+			if release != nil {
+				release()
+			}
+		}()
+	} else {
+		addr, aerr := s.terminalAddr(c, siteID)
+		if aerr != nil {
+			switch {
+			case agentCanRelay: // a current Bamboo whose SSH server is not running
+				problem("That machine's SSH server isn't running. Start Bamboo there with -ssh, and add Hammerhead's public key to authorized_keys.")
+			default:
+				problem(aerr.Error() + " Updating Bamboo on that machine lets the terminal work without any port.")
+			}
+			return
+		}
+		client, err = ssh.Dial("tcp", addr, cfg)
+	}
+	if err != nil {
+		var hk *hostKeyChanged
+		switch {
+		case errors.As(err, &hk):
+			control(termControl{Type: "error", Message: hk.Error(), HostKeyChanged: true})
+			time.Sleep(200 * time.Millisecond)
+		case errors.Is(err, errAgentSilent):
+			problem("The site didn't answer. Is Bamboo running there and connected to Hammerhead?")
+		case strings.Contains(err.Error(), "unable to authenticate"):
+			problem("The site refused Hammerhead's key. Add Hammerhead's public key to authorized_keys on that machine.")
+		default:
+			problem(fmt.Sprintf("Couldn't reach the site's SSH server: is Bamboo running with -ssh? (%v)", err))
+		}
+		return
+	}
+	defer client.Close()
+	sess, err := client.NewSession()
+	if err != nil {
+		problem(err.Error())
+		return
+	}
+	defer sess.Close()
+	if err := sess.RequestPty(termDefaultTerm, rows, cols, ssh.TerminalModes{ssh.ECHO: 1, ssh.TTY_OP_ISPEED: 14400, ssh.TTY_OP_OSPEED: 14400}); err != nil {
+		problem(fmt.Sprintf("The site wouldn't give a terminal: %v", err))
+		return
+	}
+	stdin, err := sess.StdinPipe()
+	if err != nil {
+		problem(err.Error())
+		return
+	}
+	stdout, err := sess.StdoutPipe()
+	if err != nil {
+		problem(err.Error())
+		return
+	}
+	// What the shell writes to its error output (including why it could not start) goes to the
+	// terminal like everything else, instead of being thrown away.
+	sess.Stderr = termWriter(func(p []byte) error { return send(termFrame{data: append([]byte(nil), p...)}) })
+	if err := sess.Shell(); err != nil {
+		problem(fmt.Sprintf("Couldn't start a shell: %v", err))
+		return
+	}
+	s.audit(c.Org.ID, c.Actor, "terminal.open", siteName)
+	started := time.Now()
+	defer func() {
+		s.audit(c.Org.ID, c.Actor, "terminal.close", fmt.Sprintf("%s after %s", siteName, time.Since(started).Round(time.Second)))
+	}()
+	control(termControl{Type: "ready"})
+
+	done := make(chan struct{})
+	var once sync.Once
+	stop := func() { once.Do(func() { close(done); ws.Close() }) }
+	defer stop()
+
+	waited := make(chan struct{}) // closed once the browser has been told how the shell ended
+	go func() {                   // shell output to the browser
+		defer func() {
+			select { // the output ends just before the shell is reaped: let the exit message go out first
+			case <-waited:
+			case <-time.After(2 * time.Second):
+			}
+			stop()
+		}()
+		buf := make([]byte, 16<<10)
+		for {
+			n, err := stdout.Read(buf)
+			if n > 0 {
+				if send(termFrame{data: append([]byte(nil), buf[:n]...)}) != nil {
+					return
+				}
+			}
+			if err != nil {
+				return
+			}
+		}
+	}()
+	go func() { // the shell ended: say how, then close
+		code := 0
+		if err := sess.Wait(); err != nil {
+			var ee *ssh.ExitError
+			if errors.As(err, &ee) {
+				code = ee.ExitStatus()
+			} else if !errors.Is(err, io.EOF) {
+				code = -1
+			}
+		}
+		control(termControl{Type: "exit", Code: code})
+		close(waited)
+		time.Sleep(200 * time.Millisecond) // let the exit message reach the browser first
+		stop()
+	}()
+	idle := time.AfterFunc(termIdle, func() {
+		control(termControl{Type: "idle"})
+		stop()
+	})
+	defer idle.Stop()
+	for { // browser to shell
+		var f termFrame
+		if err := frames.Receive(ws, &f); err != nil {
+			return
+		}
+		idle.Reset(termIdle)
+		if !f.text {
+			if _, err := stdin.Write(f.data); err != nil {
+				return
+			}
+			continue
+		}
+		var ctl termControl
+		if json.Unmarshal(f.data, &ctl) == nil && ctl.Type == "resize" {
+			sess.WindowChange(clamp(ctl.Rows, 5, termResizeMaxRow), clamp(ctl.Cols, 20, termResizeMaxCol))
+		}
+	}
+}
+
+// ---------- the call-back: a terminal without any open port ----------
+
+// termCallBack is how long Hammerhead waits for an agent to answer a terminal request. The agent
+// notices the request within a moment, because it is long-polling for its configuration.
+const termCallBack = 15 * time.Second
+
+var errAgentSilent = errors.New("the agent did not call back")
+
+// pendingTerm is a terminal session waiting for the site's agent to call back.
+type pendingTerm struct {
+	site    string
+	created time.Time
+	claimed bool
+	offered bool          // an agent has been sent this session in its configuration
+	conn    chan net.Conn // the agent's call-back connection, once it arrives
+	done    chan struct{} // closed when Hammerhead has finished with that connection
+}
+
+// wakeAgentsLocked ends every long poll now, so an agent hears about a change at once.
+// The configuration version does not move, so nothing is saved or re-applied.
+func (s *Server) wakeAgentsLocked() {
+	close(s.changed)
+	s.changed = make(chan struct{})
+}
+
+// waitingTerminalsLocked lists the sessions an agent should answer, and forgets stale ones.
+func (s *Server) waitingTerminalsLocked(siteID string) []string {
+	var ids []string
+	for id, p := range s.terms {
+		if time.Since(p.created) > 2*termCallBack {
+			delete(s.terms, id)
+			continue
+		}
+		if p.site == siteID && !p.claimed {
+			ids = append(ids, id)
+			p.offered = true
+		}
+	}
+	return ids
+}
+
+// newTerminalsLocked reports whether the site has a session its agent has not been told about yet,
+// which makes that agent's long poll return straight away instead of waiting for a change.
+func (s *Server) newTerminalsLocked(siteID string) bool {
+	for _, p := range s.terms {
+		if p.site == siteID && !p.claimed && !p.offered {
+			return true
+		}
+	}
+	return false
+}
+
+// dialViaAgent asks the site's agent to call back and signs in to its SSH server over that
+// connection. release must be called when the session is over.
+func (s *Server) dialViaAgent(siteID string, cfg *ssh.ClientConfig) (*ssh.Client, func(), error) {
+	id := randHex(12)
+	p := &pendingTerm{site: siteID, created: time.Now(), conn: make(chan net.Conn, 1), done: make(chan struct{})}
+	s.mu.Lock()
+	s.terms[id] = p
+	s.wakeAgentsLocked()
+	s.mu.Unlock()
+	var once sync.Once
+	release := func() {
+		once.Do(func() {
+			s.mu.Lock()
+			delete(s.terms, id)
+			s.mu.Unlock()
+			close(p.done)
+		})
+	}
+	var conn net.Conn
+	select {
+	case conn = <-p.conn:
+	case <-time.After(termCallBack):
+		release()
+		return nil, nil, errAgentSilent
+	}
+	conn.SetDeadline(time.Now().Add(termDialTimeout))
+	cc, chans, reqs, err := ssh.NewClientConn(conn, "site", cfg)
+	if err != nil {
+		conn.Close()
+		release()
+		return nil, nil, err
+	}
+	conn.SetDeadline(time.Time{})
+	return ssh.NewClient(cc, chans, reqs), release, nil
+}
+
+// terminalCallBack is where an agent connects to answer a terminal request. The connection is
+// handed to the session that asked for it and carries that session's SSH traffic.
+func (s *Server) terminalCallBack(w http.ResponseWriter, r *http.Request) {
+	s.mu.Lock()
+	st := s.siteByToken(r)
+	var p *pendingTerm
+	if st != nil {
+		if q := s.terms[r.PathValue("id")]; q != nil && q.site == st.ID && !q.claimed {
+			q.claimed, p = true, q
+		}
+	}
+	s.mu.Unlock()
+	switch {
+	case st == nil:
+		fail(w, 401, errors.New("bad site token"))
+		return
+	case p == nil:
+		fail(w, 404, errors.New("no such terminal"))
+		return
+	}
+	websocket.Server{
+		Handshake: func(*websocket.Config, *http.Request) error { return nil }, // authenticated by the site token
+		Handler: func(ws *websocket.Conn) {
+			ws.PayloadType = websocket.BinaryFrame
+			ws.MaxPayloadBytes = termMaxFrame
+			select {
+			case p.conn <- ws:
+				<-p.done // the websocket closes when this handler returns
+			case <-time.After(termCallBack):
+			}
+		},
+	}.ServeHTTP(w, r)
+}
+
+// clipText cuts s to at most n bytes.
+func clipText(s string, n int) string {
+	if len(s) > n {
+		return s[:n]
+	}
+	return s
+}
+
+// termWriter sends what is written to it to the browser as terminal output.
+type termWriter func(p []byte) error
+
+func (w termWriter) Write(p []byte) (int, error) {
+	if err := w(p); err != nil {
+		return 0, err
+	}
+	return len(p), nil
+}

traffic.go

@@ -0,0 +1,161 @@
+package hammerhead
+
+import (
+	"errors"
+	"fmt"
+	"net/http"
+	"time"
+)
+
+// Traffic analytics: edges report per-port bytes, connections and their busiest client addresses with
+// every report. Hammerhead keeps them at three resolutions (see the *Keep constants in db.go) and draws
+// charts from them. Organizations can also be alerted when a port suddenly carries far more traffic
+// than usual, or more than a daily amount.
+
+// Spikes are judged on the last few minutes against the hours before them. A port has to have an
+// hour of history before it can spike, and the burst has to be worth noticing in absolute terms, so
+// a quiet port going from 1 KB to 50 KB doesn't page anyone.
+const (
+	spikeMinHistory = 60               // minutes of baseline needed before a spike can be called
+	spikeMinBytes   = 10 * 1024 * 1024 // bytes in the window below which nothing counts as a spike
+)
+
+// spiking says whether recent traffic is a spike over the baseline for a port.
+func spiking(recent int64, baseline float64, history int64, factor int) bool {
+	if factor <= 0 || history < spikeMinHistory || recent < spikeMinBytes {
+		return false
+	}
+	return float64(recent) > baseline*float64(factor)
+}
+
+// trafficProblemsLocked adds an alert problem for each port of an organization that is spiking or has
+// gone over its daily amount. Caller holds s.mu.
+func (s *Server) trafficProblemsLocked(now time.Time, edgeDown map[string]bool, add func(key, org, kind, name, msg string)) {
+	dayStart := now.UTC().Truncate(24*time.Hour).Unix() / 3600
+	for _, m := range s.st.Mappings {
+		if !m.Enabled || edgeDown[m.EdgeID] {
+			continue
+		}
+		o := s.findOrg(m.OrgID)
+		if o == nil || (o.TrafficSpikeFactor <= 0 && o.TrafficDailyMB <= 0) {
+			continue
+		}
+		if o.TrafficSpikeFactor > 0 {
+			recent, base, history := s.db.recentTraffic(m.ID, now)
+			if spiking(recent, base, history, o.TrafficSpikeFactor) {
+				add("spike:"+m.ID, m.OrgID, "traffic", m.Name, fmt.Sprintf("%s: %s in the last %d minutes, about %.0f times its usual %s",
+					m.Name, byteText(recent), spikeWindow, float64(recent)/maxFloat(base, 1), byteText(int64(base))))
+			}
+		}
+		if limit := int64(o.TrafficDailyMB) << 20; limit > 0 {
+			if used := s.db.bytesSince(m.ID, dayStart); used > limit {
+				add("quota:"+m.ID, m.OrgID, "traffic", m.Name, fmt.Sprintf("%s has carried %s today, over the %s a day you set", m.Name, byteText(used), byteText(limit)))
+			}
+		}
+	}
+}
+
+func maxFloat(a, b float64) float64 {
+	if a > b {
+		return a
+	}
+	return b
+}
+
+func byteText(n int64) string {
+	f, units := float64(n), []string{"B", "KB", "MB", "GB", "TB"}
+	i := 0
+	for f >= 1024 && i < len(units)-1 {
+		f /= 1024
+		i++
+	}
+	if i == 0 {
+		return fmt.Sprintf("%d B", n)
+	}
+	return fmt.Sprintf("%.1f %s", f, units[i])
+}
+
+// trafficRanges maps the ranges the dashboard offers to a bucket size, a bucket count and how many hours of top sources to include.
+var trafficRanges = map[string]struct {
+	unit   string
+	count  int
+	hours  int
+	bucket time.Duration
+}{
+	"1h":  {"minute", 60, 1, time.Minute},
+	"6h":  {"minute", 360, 6, time.Minute},
+	"24h": {"hour", 24, 24, time.Hour},
+	"7d":  {"hour", 7 * 24, 7 * 24, time.Hour},
+	"30d": {"hour", 30 * 24, 7 * 24, time.Hour}, // client addresses are only kept for a week
+}
+
+// orgTrafficDetail serves one port's traffic chart and busiest clients, or the organization's total
+// for id "all". ?range= is one of 1h, 6h, 24h (default), 7d or 30d.
+func (s *Server) orgTrafficDetail(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	rng := r.URL.Query().Get("range")
+	if rng == "" {
+		rng = "24h"
+	}
+	spec, found := trafficRanges[rng]
+	if !found {
+		fail(w, 400, errors.New("range must be 1h, 6h, 24h, 7d or 30d"))
+		return
+	}
+	id := r.PathValue("id")
+	s.mu.Lock()
+	var ids []string
+	name := "All resources"
+	for _, m := range s.st.Mappings {
+		if m.OrgID != c.Org.ID {
+			continue
+		}
+		if id == "all" {
+			ids = append(ids, m.ID)
+		} else if m.ID == id {
+			ids, name = []string{m.ID}, m.Name
+		}
+	}
+	s.mu.Unlock()
+	if id != "all" && len(ids) == 0 {
+		fail(w, 404, errors.New("no such resource"))
+		return
+	}
+	now := time.Now()
+	points := s.db.trafficSeries(ids, spec.unit, spec.count, now)
+	var tot TrafficPoint
+	var peak int64
+	for _, p := range points {
+		tot.Conns += p.Conns
+		tot.BytesIn += p.BytesIn
+		tot.BytesOut += p.BytesOut
+		if b := p.BytesIn + p.BytesOut; b > peak {
+			peak = b
+		}
+	}
+	writeJSON(w, 200, map[string]any{
+		"name": name, "range": rng, "bucketSeconds": int(spec.bucket / time.Second), "points": points,
+		"total": tot, "peakBucket": peak, "sources": s.db.topSources(ids, spec.hours, 10, now), "sourcesHours": spec.hours,
+	})
+}
+
+// trafficAlertSettings sets when an organization is told about unusual traffic.
+func (s *Server) trafficAlertSettings(w http.ResponseWriter, r *http.Request, sc alertScope) {
+	var in struct{ SpikeFactor, DailyMB int }
+	if !decode(w, r, &in) {
+		return
+	}
+	if (in.SpikeFactor != 0 && (in.SpikeFactor < 2 || in.SpikeFactor > 1000)) || in.DailyMB < 0 || in.DailyMB > 100_000_000 {
+		fail(w, 400, errors.New("A spike is 2 to 1000 times the usual traffic (0 turns it off), and the daily amount can't be negative (0 turns it off)."))
+		return
+	}
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	o := s.findOrg(sc.org)
+	if o == nil {
+		fail(w, 404, errors.New("organization settings only"))
+		return
+	}
+	o.TrafficSpikeFactor, o.TrafficDailyMB = in.SpikeFactor, in.DailyMB
+	s.save()
+	ok(w)
+}

traffic_test.go

@@ -0,0 +1,159 @@
+package hammerhead
+
+import (
+	"strings"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/reef/model"
+)
+
+func TestSpiking(t *testing.T) {
+	const mb = 1 << 20
+	cases := []struct {
+		name    string
+		recent  int64
+		base    float64
+		history int64
+		factor  int
+		want    bool
+	}{
+		{"ten times usual", 200 * mb, 10 * mb, 180, 5, true},
+		{"just under the factor", 49 * mb, 10 * mb, 180, 5, false},
+		{"off", 500 * mb, 10 * mb, 180, 0, false},
+		{"too little history", 500 * mb, 10 * mb, 30, 5, false},
+		{"tiny in absolute terms", 5 * mb, 0.01 * mb, 180, 5, false},
+		{"from nothing to a lot", 50 * mb, 0, 180, 5, true},
+	}
+	for _, c := range cases {
+		if got := spiking(c.recent, c.base, c.history, c.factor); got != c.want {
+			t.Errorf("%s: spiking = %v, want %v", c.name, got, c.want)
+		}
+	}
+}
+
+func TestTrafficStorage(t *testing.T) {
+	s := newFx(t).s
+	now := time.Now()
+	st := func(in, out int64, src ...model.SourceStat) model.Stat {
+		return model.Stat{Conns: 1, BytesIn: in, BytesOut: out, Sources: src}
+	}
+	s.db.addTrafficAt("m0", st(100, 900, model.SourceStat{IP: "203.0.113.1", Conns: 1, BytesIn: 100, BytesOut: 900}), now)
+	s.db.addTrafficAt("m0", st(10, 10, model.SourceStat{IP: "203.0.113.1", Conns: 1, BytesIn: 10, BytesOut: 10}, model.SourceStat{IP: "203.0.113.2", Conns: 1, BytesIn: 1}), now)
+	s.db.addTrafficAt("m0", st(1, 1), now.Add(-3*time.Minute))
+	s.db.addTrafficAt("m1", st(7, 7), now)
+
+	pts := s.db.trafficSeries([]string{"m0"}, "minute", 60, now)
+	if len(pts) != 60 || pts[59].BytesIn != 110 || pts[59].BytesOut != 910 || pts[59].Conns != 2 || pts[56].BytesIn != 1 {
+		t.Fatalf("minute series: last=%+v 3 back=%+v", pts[59], pts[56])
+	}
+	if pts[59].Time != now.Unix()/60*60 || pts[58].Time != pts[59].Time-60 {
+		t.Errorf("bucket times: %d %d", pts[58].Time, pts[59].Time)
+	}
+	all := s.db.trafficSeries([]string{"m0", "m1"}, "hour", 24, now)
+	if all[23].BytesIn+all[22].BytesIn < 118 || len(all) != 24 {
+		t.Errorf("hour series over two ports: %+v", all[22:])
+	}
+	if got := s.db.trafficSeries(nil, "hour", 3, now); len(got) != 3 || got[2].BytesIn != 0 {
+		t.Errorf("no ports: %+v", got)
+	}
+
+	top := s.db.topSources([]string{"m0"}, 24, 10, now)
+	if len(top) != 2 || top[0].IP != "203.0.113.1" || top[0].BytesOut != 910 || top[0].Conns != 2 {
+		t.Fatalf("top sources: %+v", top)
+	}
+	if got := s.db.topSources([]string{"m0"}, 24, 1, now); len(got) != 1 {
+		t.Errorf("limit: %+v", got)
+	}
+	if got := s.db.topSources([]string{"m1"}, 24, 10, now); len(got) != 0 {
+		t.Errorf("another port's sources leaked: %+v", got)
+	}
+}
+
+func TestTrafficRetention(t *testing.T) {
+	s := newFx(t).s
+	now := time.Now()
+	old := model.Stat{Conns: 1, BytesIn: 5, Sources: []model.SourceStat{{IP: "203.0.113.9", Conns: 1, BytesIn: 5}}}
+	s.db.addTrafficAt("m0", old, now.Add(-trafficMinutelyKeep-time.Hour)) // too old for minutes, fine for hours and sources
+	s.db.addTrafficAt("m0", old, now.Add(-trafficSourcesKeep-time.Hour))  // too old for sources too
+	s.db.addTrafficAt("m0", old, now.Add(-trafficHourlyKeep-time.Hour))   // gone everywhere
+	s.db.addTrafficAt("m0", old, now)
+	s.db.purge(30, 365)
+	count := func(table string) (n int) {
+		s.db.sql.QueryRow(`SELECT COUNT(*) FROM ` + table).Scan(&n)
+		return
+	}
+	if n := count("traffic_minutely"); n != 1 {
+		t.Errorf("minutely rows after purge: %d, want 1", n)
+	}
+	if n := count("traffic_sources"); n != 2 {
+		t.Errorf("source rows after purge: %d, want 2", n)
+	}
+	if n := count("traffic_hourly"); n != 3 {
+		t.Errorf("hourly rows after purge: %d, want 3", n)
+	}
+}
+
+func TestRecentTraffic(t *testing.T) {
+	s := newFx(t).s
+	now := time.Now()
+	const mb = 1 << 20
+	for h := 3 * 60; h > spikeWindow; h-- { // three hours of 1 MB a minute
+		s.db.addTrafficAt("m0", model.Stat{BytesIn: mb}, now.Add(-time.Duration(h)*time.Minute))
+	}
+	s.db.addTrafficAt("m0", model.Stat{BytesIn: 100 * mb}, now)
+	recent, base, hist := s.db.recentTraffic("m0", now)
+	if recent != 100*mb || hist < 170 || base < 4.9*mb || base > 5.1*mb { // 1 MB a minute is 5 MB per window
+		t.Fatalf("recent=%d base=%.0f history=%d", recent, base, hist)
+	}
+	if !spiking(recent, base, hist, 5) {
+		t.Error("100 MB over a 5 MB norm should spike")
+	}
+	if r, b, h := s.db.recentTraffic("nothing", now); r != 0 || b != 0 || h != 0 {
+		t.Errorf("unknown port: %d %f %d", r, b, h)
+	}
+}
+
+func TestTrafficAlertsFireAndClear(t *testing.T) {
+	f := newFx(t)
+	s := f.s
+	const mb = 1 << 20
+	now := time.Now()
+	s.mu.Lock()
+	m := s.st.Mappings[0]
+	o := s.findOrg(m.OrgID)
+	o.TrafficSpikeFactor, o.TrafficDailyMB = 4, 500
+	id := m.ID
+	s.mu.Unlock()
+	for min := 3 * 60; min > spikeWindow; min-- {
+		s.db.addTrafficAt(id, model.Stat{BytesIn: mb}, now.Add(-time.Duration(min)*time.Minute))
+	}
+
+	problems := func() map[string]problem {
+		s.mu.Lock()
+		defer s.mu.Unlock()
+		return s.problemsLocked(now)
+	}
+	if p := problems(); p["spike:"+id].msg != "" || p["quota:"+id].msg != "" {
+		t.Fatalf("steady traffic raised a problem: %+v", p)
+	}
+	s.db.addTrafficAt(id, model.Stat{BytesOut: 300 * mb}, now)
+	p := problems()
+	if !strings.Contains(p["spike:"+id].msg, "usual") || p["spike:"+id].kind != "traffic" {
+		t.Fatalf("no spike problem: %+v", p)
+	}
+	if p["quota:"+id].msg != "" {
+		t.Errorf("under the daily amount, yet: %+v", p["quota:"+id])
+	}
+	s.db.addTrafficAt(id, model.Stat{BytesOut: 300 * mb}, now)
+	if q := problems()["quota:"+id]; !strings.Contains(q.msg, "over the") {
+		t.Fatalf("no quota problem: %+v", q)
+	}
+
+	s.mu.Lock()
+	o.TrafficSpikeFactor, o.TrafficDailyMB = 0, 0
+	s.mu.Unlock()
+	if p := problems(); p["spike:"+id].msg != "" || p["quota:"+id].msg != "" {
+		t.Errorf("alerts off, yet: %+v", p)
+	}
+}

web/index.html

@@ -9,6 +9,7 @@
   /* tokens:begin  Design tokens, the Lightwell design language (docs/design.md). This is the only place raw colors and sizes are defined. */
   :root {
     --accent:#1f5c5c; --accent-contrast:#ffffff;
+    --terminal-bg:#0c0c0c; /* a terminal stays dark in both themes */
     --bg:#f6f5f3; --surface:#ffffff; --surface-2:#efede9; --border:#e0ddd7; --border-strong:#c4c0b8;
     --text:#1d1c1a; --text-muted:#5f5b54; --text-faint:#7a766e;
     --accent-text:var(--accent);
@@ -102,6 +103,8 @@
   .grid > label.chk { margin-top:calc(var(--text-sm) * var(--leading) + var(--s1) + var(--s2)); }
   .sect-h { padding:var(--s3) var(--s4); font-weight:600; border-bottom:var(--border-w) solid var(--border); }
   .scroll { overflow-x:auto; }
+  td.wrap { overflow-wrap:anywhere; min-width:150px; max-width:260px; }
+  .ports { display:flex; flex-wrap:wrap; gap:var(--s1); }
   .actions { display:flex; gap:var(--s2); flex-wrap:wrap; }
   .form-actions { justify-content:flex-end; margin-top:var(--s2); }
   .toolbar { display:flex; align-items:center; gap:var(--s3); margin-bottom:var(--s3); flex-wrap:wrap; }
@@ -159,6 +162,8 @@
   .badge-warning { background:var(--warning-bg); color:var(--warning); }
   .badge-danger { background:var(--danger-bg); color:var(--danger); }
   .badge-accent { background:var(--accent-soft); color:var(--accent-text); }
+  .term { background:var(--terminal-bg); border:var(--border-w) solid var(--border); border-radius:var(--radius); padding:var(--s2); height:min(60vh,460px); overflow:hidden; }
+  .term .xterm-viewport { background-color:var(--terminal-bg) !important; }
   .tags { display:flex; flex-wrap:wrap; gap:var(--s1); margin-top:var(--s1); }
   .tagbar { display:flex; flex-wrap:wrap; gap:var(--s2); margin-bottom:var(--s3); }
   .chip { min-height:44px; padding:0 var(--s3); border:var(--border-w) solid var(--border); border-radius:var(--radius); background:var(--surface); color:var(--text); font:inherit; cursor:pointer; }
@@ -179,7 +184,7 @@
   @keyframes shimmer { to { background-position:-200% 0; } }
 
   /* dialogs, tabs, toasts */
-  .shade { position:fixed; inset:0; background:var(--overlay); display:grid; place-items:start center; padding:var(--s7) var(--s4); overflow:auto; z-index:var(--z-overlay); }
+  .shade { position:fixed; inset:0; background:var(--overlay); display:grid; grid-template-columns:minmax(0,1fr); place-items:start center; padding:var(--s7) var(--s4); overflow:auto; z-index:var(--z-overlay); }
   .modal { background:var(--surface); border-radius:var(--radius-lg); box-shadow:var(--shadow); padding:var(--s5); width:min(480px,100%); }
   .modal.wide { width:min(780px,100%); }
   .modal h2 { margin:0 0 var(--s3); font-size:var(--text-lg); }
@@ -372,6 +377,12 @@ function modal(title, body, opts = {}) {
   function onKey(e) {
     const top = [...document.querySelectorAll('.shade')].pop();
     if (top !== shade) return;
+    // Inside a terminal every key belongs to the shell (Escape for vim, Tab for completion), except
+    // Ctrl+Alt+Escape, which moves focus out so the dialog can still be left from the keyboard.
+    if (opts.keys && e.target.closest && e.target.closest(opts.keys)) {
+      if (e.key === 'Escape' && e.ctrlKey && e.altKey) { e.preventDefault(); e.stopPropagation(); box.focus(); }
+      return;
+    }
     if (e.key === 'Escape') { e.stopPropagation(); close(); }
     else if (e.key === 'Tab') {
       const f = focusables(); if (!f.length) { e.preventDefault(); return; }
@@ -634,7 +645,7 @@ views.ports = async () => {
   S.headerAction = ['Add resource', () => portForm()];
   const q = h('input', {type:'search', placeholder:'Search resources…', oninput:() => { const t = q.value.toLowerCase(); table.querySelectorAll('tbody tr').forEach(tr => { tr.style.display = tr.textContent.toLowerCase().includes(t) ? '' : 'none'; }); }});
   return h('div', {}, unsetDash ? h('div', {class:'notice'}, 'Some ports have a login in front of them, but the dashboard URL isn\'t set. Visitors are sent back to this address\'s login page, so set it under Server → Settings if people reach Hammerhead by a different address.') : null,
-    tagBar('ports', tags), rows.length > 5 ? h('div', {class:'toolbar'}, q) : null, table);
+    tagBar('ports', tags), rows.length ? h('div', {class:'toolbar'}, h('button', {class:'ghost', onclick:() => trafficDialog('all', 'All resources')}, 'Traffic overview')) : null, rows.length > 5 ? h('div', {class:'toolbar'}, q) : null, table);
 };
 
 function spark(series) {
@@ -648,6 +659,57 @@ function spark(series) {
   return h('div', {title:'traffic per hour, last 24 hours'}, svg);
 }
 
+// Traffic chart for one resource (or "all"): stacked bars per time bucket.
+function trafficChart(d) {
+  const ns = 'http://www.w3.org/2000/svg', W = 640, H = 170, L = 52, B = 22, T = 8, n = d.points.length;
+  const mk = (tag, attrs, text) => { const e = document.createElementNS(ns, tag); for (const k in attrs) e.setAttribute(k, attrs[k]); if (text != null) e.textContent = text; return e; };
+  const per = d.bucketSeconds >= 3600 ? 'hour' : 'minute';
+  const svg = mk('svg', {viewBox:`0 0 ${W} ${H}`, width:'100%', role:'img', 'aria-label':`Traffic per ${per}: ${bytes(d.total.bytesOut)} out and ${bytes(d.total.bytesIn)} in`});
+  const max = Math.max(d.peakBucket, 1), plotH = H - B - T, bw = (W - L) / n;
+  for (const f of [0, .5, 1]) {
+    const y = T + plotH * (1 - f);
+    svg.append(mk('line', {x1:L, x2:W, y1:y, y2:y, stroke:'var(--border)', 'stroke-width':1}), mk('text', {x:L - 6, y:y + 4, 'text-anchor':'end', fill:'var(--text-faint)', 'font-size':11}, bytes(max * f)));
+  }
+  d.points.forEach((p, i) => {
+    const hOut = p.bytesOut / max * plotH, hIn = p.bytesIn / max * plotH, x = L + i * bw, g = mk('g', {});
+    g.append(mk('title', {}, `${new Date(p.t * 1000).toLocaleString(undefined, {month:'short', day:'numeric', hour:'numeric', minute:per === 'minute' ? '2-digit' : undefined})}: ${bytes(p.bytesOut)} out, ${bytes(p.bytesIn)} in, ${p.conns} conn`),
+      mk('rect', {x:x + .5, y:T + plotH - hOut, width:Math.max(bw - 1, .5), height:hOut, fill:'var(--accent)'}),
+      mk('rect', {x:x + .5, y:T + plotH - hOut - hIn, width:Math.max(bw - 1, .5), height:hIn, fill:'var(--text-faint)'}));
+    svg.append(g);
+  });
+  for (const i of [0, Math.floor(n / 2), n - 1]) {
+    const t = new Date(d.points[i].t * 1000);
+    svg.append(mk('text', {x:i === n - 1 ? W : L + i * bw, y:H - 6, 'text-anchor':i === 0 ? 'start' : i === n - 1 ? 'end' : 'middle', fill:'var(--text-faint)', 'font-size':11},
+      per === 'minute' || n <= 24 ? t.toLocaleTimeString(undefined, {hour:'numeric', minute:'2-digit'}) : t.toLocaleDateString(undefined, {month:'short', day:'numeric'})));
+  }
+  return svg;
+}
+
+// A dialog with the traffic chart, totals and busiest visitors of one resource ("all" = every resource).
+function trafficDialog(id, name) {
+  const body = h('div', {class:'stack'}, h('p', {class:'dim'}, 'Loading…'));
+  const close = modal('Traffic: ' + name, body, {wide:true});
+  const load = async r => {
+    const d = await api('GET', O(`/traffic/${id}?range=${r}`));
+    const per = d.bucketSeconds >= 3600 ? 'hour' : 'minute';
+    const ranges = [['1h', 'Hour'], ['6h', '6 hours'], ['24h', 'Day'], ['7d', 'Week'], ['30d', 'Month']];
+    const covered = d.sourcesHours >= 48 ? Math.round(d.sourcesHours / 24) + ' days' : d.sourcesHours === 1 ? 'hour' : d.sourcesHours + ' hours';
+    body.replaceChildren(
+      h('div', {class:'tabs2'}, ranges.map(([k, l]) => h('button', {class:k === r ? 'on' : '', onclick:() => load(k).catch(e => toast(e.message))}, l))),
+      d.total.bytesIn + d.total.bytesOut ? h('div', {}, trafficChart(d),
+        h('div', {class:'small dim'}, h('span', {style:'color:var(--accent)'}, '■ '), 'To visitors (out) ', h('span', {style:'margin-left:12px;color:var(--text-faint)'}, '■ '), 'From visitors (in) · per ' + per)) :
+        emptyState('ports', 'No traffic in this period', 'Traffic appears here a few seconds after it happens.'),
+      h('div', {class:'small'}, `${d.total.conns} connections or requests, ${bytes(d.total.bytesOut)} out, ${bytes(d.total.bytesIn)} in.`),
+      h('h3', {}, 'Busiest visitors'),
+      d.sources.length ? h('div', {class:'scroll'}, h('table', {},
+        h('thead', {}, h('tr', {}, ['Address', 'Connections', 'Out', 'In'].map(x => h('th', {}, x)))),
+        h('tbody', {}, d.sources.map(s => h('tr', {}, h('td', {class:'mono'}, s.ip), h('td', {}, s.conns), h('td', {}, bytes(s.bytesOut)), h('td', {}, bytes(s.bytesIn))))))) : h('p', {class:'dim small'}, 'None named yet.'),
+      h('p', {class:'dim small'}, `Counted over the last ${covered} (visitor addresses are kept for 7 days). Each report names the busiest ten per resource, so very small visitors may be missing.`),
+      h('div', {class:'actions form-actions'}, h('button', {class:'btn', onclick:() => close()}, 'Close')));
+  };
+  load('24h').catch(e => { close(); toast(e.message); });
+}
+
 const portText = m => m.portEnd > m.port ? m.port + '-' + m.portEnd : String(m.port);
 
 function mappingRow(m, st, series) {
@@ -666,24 +728,70 @@ function mappingRow(m, st, series) {
   const pubAddr = web ? m.proto + '://' + (m.hostname || '*') + (m.port === (m.proto === 'https' ? 443 : 80) ? '' : ':' + portText(m)) : (m.hostname ? m.hostname + ' @ ' : '') + m.bindIp + ':' + portText(m);
   return h('tr', {},
     h('td', {title:why}, badge(label, tone)),
-    h('td', {}, h('div', {}, m.name, ' ', h('span', {class:'tag'}, m.proto), m.proxyProtocol ? h('span', {class:'tag', style:'margin-left:4px', title:'the target is told the real client address'}, 'proxy ' + m.proxyProtocol) : null, prot ? h('span', {style:'margin-left:4px', title:'Visitors have to sign in'}, badge('Login', 'accent')) : null),
+    h('td', {}, h('div', {}, m.name, ' ', h('span', {class:'tag'}, m.proto), m.redirectTo ? h('span', {class:'tag', style:'margin-left:4px'}, 'redirect') : null,m.proxyProtocol ? h('span', {class:'tag', style:'margin-left:4px', title:'the target is told the real client address'}, 'proxy ' + m.proxyProtocol) : null, prot ? h('span', {style:'margin-left:4px', title:'Visitors have to sign in'}, badge('Login', 'accent')) : null),
       tagList(m),
       s.error ? h('div', {class:'err small mono clip', title:s.error}, s.error) : null,
       s.cert && m.proto === 'https' ? h('div', {class:'small clip ' + (s.cert.startsWith('error') ? 'err' : 'dim'), title:s.cert}, 'Certificate: ' + s.cert) : null),
     h('td', {}, h('span', {class:'mono'}, pubAddr), h('div', {class:'dim small'}, (edge ? edge.name : '?') + (web && m.bindIp !== '0.0.0.0' ? ' · ' + m.bindIp : ''))),
-    h('td', {class:'small'}, m.targets.map((t, i) => {
+    h('td', {class:'small'}, m.redirectTo ? h('div', {}, h('span', {class:'mono clip', title:m.redirectTo}, '→ ' + m.redirectTo),
+      h('div', {class:'dim'}, (m.redirectPermanent ? 'Permanent (301)' : 'Temporary (302)') + (m.redirectKeepPath ? ', keeps the path' : ''))) : m.targets.map((t, i) => {
       const th = (s.targets || [])[i], site = byId(st.sites, t.siteId);
       return h('div', {title:th && !th.healthy ? th.detail : '', style:'margin-bottom:4px'}, badge(th ? (th.healthy ? 'Up' : 'Down') : 'Unknown', th ? (th.healthy ? 'success' : 'danger') : ''), ' ', site ? site.name : '?', h('div', {class:'mono'}, t.host + ':' + t.port));
     })),
     h('td', {class:'small dim'}, m.total && m.total.conns ? [`${m.total.conns} conn`, h('br'), `${bytes(m.total.bytesIn)} in, ${bytes(m.total.bytesOut)} out`] : 'None yet', spark(series)),
     h('td', {}, h('div', {class:'actions'},
-      h('button', {class:'ghost', onclick:() => portForm(m)}, 'Edit'),
+      m.redirectTo ? null : h('button', {class:'ghost', onclick:() => trafficDialog(m.id, m.name)}, 'Traffic'),
+      h('button', {class:'ghost', onclick:() => m.redirectTo ? redirectForm(m) : portForm(m)}, 'Edit'),
       h('button', {class:'ghost', onclick: act(async () => { await api('PUT', O('/mappings/' + m.id), {...m, enabled:!m.enabled}); refresh(); })}, m.enabled ? 'Pause' : 'Enable'),
       web && prot ? h('button', {class:'ghost', onclick:() => shareDialog(m)}, 'Share') : null,
       web && prot ? h('button', {class:'ghost', title:'Everyone currently signed in to this resource has to sign in again', onclick: ask({title:`Sign everyone out of ${m.name}?`, body:'Everyone who is signed in to this resource has to sign in again. They can sign back in straight away.', confirm:'Sign out visitors', danger:false}, async () => { await api('POST', O(`/mappings/${m.id}/revoke`)); toast('Visitors signed out'); })}, 'Sign out visitors') : null,
       h('button', {class:'ghost danger', onclick: ask({title:`Close ${m.name}?`, body:'The port closes and visitors can no longer reach it. You can open it again later.', confirm:'Close resource'}, async () => { await api('DELETE', O('/mappings/' + m.id)); refresh(); })}, 'Close'))));
 }
 
+// A redirect sends visitors from one hostname to another address. New ones open the HTTPS side on port 443 and,
+// if asked, the plain HTTP side on port 80, so both ways of typing the old name land on the new one.
+function redirectForm(existing) {
+  const st = S.st, m = existing || {proto:'https', enabled:true, bindIp:'0.0.0.0', redirectPermanent:true, redirectKeepPath:true};
+  const edgeSel = h('select', {}, st.edges.map(e => h('option', {value:e.id, selected:e.id === m.edgeId}, e.name + (e.online ? '' : ' (offline)'))));
+  const ipIn = h('select', {});
+  const refreshIPs = () => {
+    const e = byId(st.edges, edgeSel.value), keep = ipIn.value || m.bindIp || '0.0.0.0';
+    const saved = existing && m.edgeId === edgeSel.value && m.bindIp !== '0.0.0.0' ? [m.bindIp] : [];
+    const ips = [...new Set([...((e && e.ips) || []), ...saved])];
+    const label = ip => { const d = st.domains.filter(x => x.ip === ip).map(x => x.name); return d.length ? `${ip} (${d.join(', ')})` : ip; };
+    ipIn.replaceChildren(h('option', {value:'0.0.0.0'}, 'Every address on the edge'), ...ips.map(ip => h('option', {value:ip}, label(ip))));
+    ipIn.value = ips.includes(keep) ? keep : '0.0.0.0';
+  };
+  edgeSel.onchange = refreshIPs; refreshIPs();
+  const from = h('input', {value:m.hostname || '', list:'rdoms', placeholder:'old.example.com or *.example.com'});
+  const doms = h('datalist', {id:'rdoms'}, st.domains.map(d => h('option', {value:d.name})));
+  const to = h('input', {value:m.redirectTo || '', placeholder:'https://new.example.com'});
+  const kind = h('select', {}, [[true, 'Permanent (301): browsers and search engines remember it'], [false, 'Temporary (302): checked every time']].map(([v, t]) => h('option', {value:String(v), selected:!!m.redirectPermanent === v}, t)));
+  const keep = chk('Keep the path and query, so old.example.com/a?b=1 goes to the same place on the new address', m.redirectKeepPath);
+  const both = chk('Also redirect plain HTTP (port 80)', true);
+  const name = h('input', {value:m.name || '', placeholder:'optional'}), tg = tagInput(m.tags, allTags(st.mappings));
+  const save = act(async () => {
+    const base = {name:name.value, edgeId:edgeSel.value, bindIp:ipIn.value || '0.0.0.0', hostname:from.value, redirectTo:to.value, redirectPermanent:kind.value === 'true',
+      redirectKeepPath:keep.box.checked, tags:tg.get(), enabled:m.enabled !== false, targets:[]};
+    if (existing) { await api('PUT', O('/mappings/' + m.id), {...base, proto:m.proto, port:m.port}); }
+    else {
+      await api('POST', O('/mappings'), {...base, proto:'https', port:443});
+      if (both.box.checked) {
+        try { await api('POST', O('/mappings'), {...base, proto:'http', port:80, name:name.value ? name.value + ' (HTTP)' : ''}); }
+        catch (e) { throw new Error('The HTTPS redirect was added, but the HTTP one could not be: ' + e.message); }
+      }
+    }
+    close(); toast(existing ? 'Redirect saved' : 'Redirect added'); refresh();
+  });
+  const close = modal(existing ? 'Edit redirect' : 'Add a domain redirect', h('div', {class:'stack'},
+    h('p', {class:'dim'}, existing ? `This redirect answers ${m.proto === 'https' ? 'HTTPS on port ' : 'HTTP on port '}${m.port}.` : 'Visitors who open the old address are sent to the new one. HTTPS gets its certificate automatically.'),
+    h('div', {class:'grid'}, field('Redirect this address', from, st.domains.length ? 'Must be under one of your domains' : 'Add a domain on the Domains tab first'), field('To', to, 'A full address, starting with http:// or https://')),
+    doms, field('Type', kind), keep, existing ? null : both,
+    h('div', {class:'grid'}, field('Edge', edgeSel), field('Public IP', ipIn, 'Domains pointed at an address are shown next to it')),
+    h('div', {class:'grid'}, field('Name', name), tg.el),
+    h('div', {class:'actions form-actions'}, h('button', {class:'ghost', onclick:() => close()}, 'Cancel'), h('button', {class:'btn', onclick:save}, existing ? 'Save changes' : 'Add redirect'))), {sticky:true});
+}
+
 function portForm(existing) {
   const st = S.st, m = existing || { proto:'tcp', enabled:true, bindIp:'0.0.0.0', targets:[{siteId:'', host:'127.0.0.1', port:''}], auth:{}, rules:[] };
   const a = m.auth || {};
@@ -743,21 +851,22 @@ function portForm(existing) {
       const th = h('input', {value:t.host, oninput:e => t.host = e.target.value, placeholder:'127.0.0.1 or 192.168.1.20'});
       const tp = h('input', {type:'number', value:t.port, placeholder:'port', oninput:e => t.port = +e.target.value});
       return h('div', {class:'tgt'}, field(i ? 'Target ' + (i + 1) : 'Target site', site), field('Host', th), field('Port', tp),
-        h('button', {class:'ghost', type:'button', title:'Pick a Docker container on that site', onclick:() => discover(t, th, tp)}, 'Containers'),
+        h('button', {class:'ghost', type:'button', title:'Pick a Docker container or discovered service on that site', onclick:() => discover(t, th, tp)}, 'Containers'),
         targets.length > 1 ? h('button', {class:'ghost danger', type:'button', 'aria-label':'Remove target', onclick:() => { targets.splice(i, 1); drawTargets(); }}, icon('close', 16)) : h('span'));
     }), many ? h('button', {class:'ghost', type:'button', style:'align-self:flex-start', title:'Spread visitors across targets, or fail over when one is down', onclick:() => { targets.push({siteId:targets[0] ? targets[0].siteId : '', host:'127.0.0.1', port:''}); drawTargets(); }}, icon('add', 16), 'Add another target') : null);
   };
   const discover = act(async (t, th, tp) => {
     if (!t.siteId) throw new Error('Choose a site first');
-    const list = await api('GET', O(`/sites/${t.siteId}/containers`));
-    if (!list.length) throw new Error('No containers reported. Run the agent with --docker (and make sure Docker is running).');
+    const [list, found] = await Promise.all([api('GET', O(`/sites/${t.siteId}/containers`)), api('GET', O(`/sites/${t.siteId}/discovered`))]);
+    if (!list.length && !found.length) throw new Error('Nothing reported. Run the agent with --docker for containers, or --discover (optionally --discover-subnets) for services on the network.');
     const opts = [];
+    for (const d of found) opts.push({label:`${d.service || 'tcp'}  ${d.host}:${d.port}${d.detail ? '  ' + d.detail : ''} (${d.source === 'local' ? 'listening here' : 'found by scan'})`, host:d.host, port:d.port});
     for (const c of list) for (const p of c.ports) {
       if (p.public) opts.push({label:`${c.name}  127.0.0.1:${p.public} (published)`, host:'127.0.0.1', port:p.public});
       for (const [n, ip] of Object.entries(c.networks)) opts.push({label:`${c.name}  ${ip}:${p.private} (${n} network)`, host:ip, port:p.private});
     }
     const sel = h('select', {size:Math.min(12, opts.length + 1)}, opts.map((o, i) => h('option', {value:i}, o.label)));
-    const close = modal('Pick a container', h('div', {class:'stack'}, sel, h('button', {class:'btn', onclick:() => { const o = opts[sel.value]; if (!o) return; t.host = th.value = o.host; t.port = o.port; tp.value = o.port; close(); }}, 'Use this')));
+    const close = modal('Pick a container or discovered service', h('div', {class:'stack'}, sel, h('button', {class:'btn', onclick:() => { const o = opts[sel.value]; if (!o) return; t.host = th.value = o.host; t.port = o.port; tp.value = o.port; close(); }}, 'Use this')));
   });
   const drawRules = () => ruleBox.replaceChildren(...rules.map((r, i) => h('div', {class:'grid', style:'grid-template-columns:110px 110px 1fr auto'},
     h('select', {onchange:e => r.action = e.target.value}, ['allow', 'deny'].map(v => h('option', {value:v, selected:r.action === v}, v === 'allow' ? 'Skip login' : 'Block'))),
@@ -817,6 +926,7 @@ function portForm(existing) {
       ['both', 'TCP + UDP', 'The same port for both, as many games want.'],
       ['tls', 'TLS passthrough', 'Route encrypted traffic by hostname without decrypting it.'],
       ['http', 'Website, plain HTTP', 'No certificate: for internal use or behind another proxy.'],
+      ['redirect', 'Domain redirect', 'Send visitors from one address to another, such as old.example.com to new.example.com.'],
     ];
     const [accessFs, rulesFs, advFs] = [...webBox.children];
     let step = 0, chosen = false;
@@ -856,6 +966,7 @@ function portForm(existing) {
       port.value = t === 'https' || t === 'tls' ? '443' : t === 'http' ? '80' : (targets[0] && targets[0].port ? String(targets[0].port) : '');
     };
     const pick = v => {
+      if (v === 'redirect') { close(); redirectForm(); return; }
       protoSel.value = v; chosen = true; protoSel.onchange();
       if (v === 'https') redir.box.checked = true;
       port.value = ''; step = 1; draw();
@@ -928,6 +1039,143 @@ async function shareDialog(m) {
 }
 
 // ----- sites -----
+// What an agent reports about its network: Docker containers (-docker) and listening services (-discover).
+async function siteFound(s) {
+  const [list, found] = await Promise.all([api('GET', O(`/sites/${s.id}/containers`)), api('GET', O(`/sites/${s.id}/discovered`))]);
+  // a published port is "8777→8777"; TCP and UDP on the same port share one entry
+  const portChips = ports => {
+    const by = new Map();
+    for (const p of ports) { const k = (p.public ? p.public + '→' : '') + p.private; by.set(k, [...(by.get(k) || []), p.type]); }
+    return h('div', {class:'ports'}, [...by].map(([k, types]) => h('span', {class:'tag mono'}, k + '/' + [...new Set(types)].join(','))));
+  };
+  const section = (title, count, body, empty) => h('div', {}, h('h3', {style:'margin:0 0 var(--s2)'}, title + (count ? ` (${count})` : '')), count ? h('div', {class:'scroll'}, body) : h('p', {class:'dim', style:'margin:0'}, empty));
+  const containers = h('table', {}, h('thead', {}, h('tr', {}, ['Name', 'Image', 'Ports'].map(t => h('th', {}, t)))),
+    h('tbody', {}, list.map(c => h('tr', {}, h('td', {class:'wrap'}, c.name), h('td', {class:'dim wrap'}, c.image), h('td', {}, portChips(c.ports))))));
+  const services = h('table', {}, h('thead', {}, h('tr', {}, ['Service', 'Address', 'Found', 'Detail'].map(t => h('th', {}, t)))),
+    h('tbody', {}, found.map(d => h('tr', {}, h('td', {}, d.service || 'tcp'), h('td', {class:'mono small'}, d.host + ':' + d.port),
+      h('td', {class:'dim'}, d.source === 'local' ? 'Listening here' : 'Network scan'), h('td', {class:'dim small wrap'}, d.detail || '')))));
+  const close = modal('Found by ' + s.name, h('div', {class:'stack'},
+    h('p', {class:'dim', style:'margin:0'}, 'These show up as suggestions when you pick a target for a resource. Nothing is shared until you choose it.'),
+    section('Docker containers', list.length, containers, 'None reported. Start the agent with -docker (BAMBOO_DOCKER=1, or BLACKTIP_DOCKER=1 for an edge) on a machine where Docker is running.'),
+    section('Services on the network', found.length, services, 'None found. Start the agent with -discover (BAMBOO_DISCOVER=1) for services listening on its machine, and add -discover-subnets (BAMBOO_DISCOVER_SUBNETS) to scan other addresses.'),
+    h('div', {class:'actions form-actions'}, h('button', {class:'btn', onclick:() => close()}, 'Close'))), {wide:true});
+}
+
+// ----- terminal -----
+// A shell on a site, in the dashboard. xterm.js is served by Hammerhead itself and loaded the first time it is needed.
+let termLoad = null;
+function loadTerm() {
+  if (!termLoad) {
+    const script = src => new Promise((ok, bad) => { const el = h('script', {src}); el.onload = ok; el.onerror = () => bad(new Error('Could not load the terminal.')); document.head.append(el); });
+    document.head.append(h('link', {rel:'stylesheet', href:'/static/term/xterm.css'}));
+    termLoad = script('/static/term/xterm.js').then(() => script('/static/term/addon-fit.js')).catch(e => { termLoad = null; throw e; });
+  }
+  return termLoad;
+}
+
+async function openTerminal(s) {
+  await loadTerm();
+  const box = h('div', {class:'term', role:'group', 'aria-label':'Terminal on ' + s.name});
+  const status = h('div', {class:'small dim', role:'status'}, 'Connecting…');
+  const forget = h('button', {class:'ghost', type:'button', style:'display:none', onclick: act(async () => { await api('DELETE', O(`/sites/${s.id}/terminal/hostkey`)); toast('Saved host key forgotten. Open the terminal again.'); })}, 'Forget saved host key');
+  const term = new Terminal({cursorBlink:true, fontFamily:'ui-monospace, Menlo, Consolas, monospace', fontSize:14, scrollback:5000,
+    theme:{background:getComputedStyle(document.documentElement).getPropertyValue('--terminal-bg').trim()}});
+  const fit = new FitAddon.FitAddon();
+  term.loadAddon(fit);
+  let ws = null, ended = false, ro = null;
+  const close = modal('Terminal: ' + s.name, h('div', {class:'stack'}, box, status,
+    h('div', {class:'actions form-actions'}, forget, h('button', {class:'btn', onclick:() => close()}, 'Close'))),
+    {sticky:true, wide:true, keys:'.term', onClose:() => { ended = true; if (ro) ro.disconnect(); if (ws) ws.close(); term.dispose(); }});
+  term.open(box); fit.fit(); term.focus();
+  const say = (text, bad) => { status.textContent = text; status.className = 'small ' + (bad ? 'err' : 'dim'); };
+  ws = new WebSocket(`${location.protocol === 'https:' ? 'wss' : 'ws'}://${location.host}${O(`/sites/${s.id}/terminal`)}?cols=${term.cols}&rows=${term.rows}`);
+  ws.binaryType = 'arraybuffer';
+  ws.onmessage = ev => {
+    if (typeof ev.data !== 'string') { term.write(new Uint8Array(ev.data)); return; } // the shell's output
+    let m; try { m = JSON.parse(ev.data); } catch { return; }
+    if (m.type === 'ready') say('Connected. Esc and Tab go to the shell. Press Ctrl+Alt+Esc to move focus out of the terminal.');
+    else if (m.type === 'error') { say(m.message, true); if (m.hostKeyChanged) forget.style.display = ''; }
+    else if (m.type === 'exit') say('The shell ended' + (m.code ? ` (exit code ${m.code})` : '') + '.');
+    else if (m.type === 'idle') say('Closed after 30 minutes without typing.', true);
+  };
+  ws.onclose = () => {
+    if (ended) return;
+    if (status.textContent === 'Connecting…') say('Could not open the terminal. Sign in again and retry.', true);
+    term.write('\r\n\x1b[2m[disconnected]\x1b[0m\r\n');
+  };
+  const enc = new TextEncoder();
+  term.onData(d => { if (ws.readyState === 1) ws.send(enc.encode(d)); });
+  term.onBinary(d => { if (ws.readyState === 1) ws.send(Uint8Array.from(d, c => c.charCodeAt(0))); });
+  ro = new ResizeObserver(() => { fit.fit(); if (ws.readyState === 1) ws.send(JSON.stringify({type:'resize', cols:term.cols, rows:term.rows})); });
+  ro.observe(box);
+}
+
+// Bamboo's built-in SSH server (started with -ssh) listens here on the site by default.
+const SSH_TARGET = '127.0.0.1:2222';
+
+// Reach a site's Bamboo SSH server with an ordinary ssh client: a TCP port on an edge whose target is that server.
+async function sshAccess(s) {
+  const st = S.st, key = await api('GET', O('/terminal'));
+  const [tHost, tPort] = [SSH_TARGET.split(':')[0], +SSH_TARGET.split(':')[1]];
+  const isSSH = m => m.proto === 'tcp' && !m.redirectTo && m.targets.some(t => t.siteId === s.id && (t.port === tPort || (m.tags || []).includes('ssh')));
+  let ports = st.mappings.filter(isSSH);
+  // what to type after the @: a domain pointed at the address when there is one, else the address itself
+  const edgeHost = (e, m) => {
+    if (m.bindIp && m.bindIp !== '0.0.0.0') { const d = st.domains.find(x => x.ip === m.bindIp); return d ? d.name : m.bindIp; }
+    const d = st.domains.find(x => x.ip && (!x.edgeId || x.edgeId === m.edgeId));
+    return d ? d.name : ((e && e.ips) || [])[0] || 'your-edge-address';
+  };
+  const cmds = h('div', {class:'stack'});
+  const connectBox = h('div', {}, h('h3', {style:'margin:0 0 var(--s2)'}, 'Connect with'), cmds, h('p', {class:'dim small'}, 'The user name is ignored: you get a shell as the user Bamboo runs as.'));
+  // Bamboo reports its SSH server; when it does, the terminal reaches it through the agent and needs no port
+  const viaAgent = !!(s.sshRelay && s.ssh);
+  const termBtn = h('button', {class:'btn', type:'button', onclick: act(async () => { close(); await openTerminal(s); })}, 'Open terminal');
+  const termNote = h('p', {class:'dim small', style:'margin:0'});
+  const draw = () => {
+    connectBox.style.display = ports.length ? '' : 'none';
+    termBtn.disabled = !viaAgent && !ports.length;
+    termNote.textContent = viaAgent ? 'No port needs to be open: Bamboo on the site connects out to Hammerhead for the session.'
+      : ports.length ? 'This uses the SSH port below. Updating Bamboo or turning on its SSH server makes a port unnecessary.'
+      : 'Needs Bamboo\'s SSH server to be running (see above), or an SSH port below.';
+    cmds.replaceChildren(...ports.map(m => {
+      const cmd = `ssh -p ${m.port} you@${edgeHost(byId(st.edges, m.edgeId), m)}`;
+      return h('div', {}, h('div', {class:'small dim'}, m.name + (m.enabled ? '' : ' (paused)')), cmdBox(cmd),
+        h('button', {class:'ghost', type:'button', onclick: act(async () => { await navigator.clipboard.writeText(cmd); toast('Copied'); })}, 'Copy'));
+    }));
+  };
+  draw();
+  const edgeSel = h('select', {}, st.edges.map(e => h('option', {value:e.id}, e.name + (e.online ? '' : ' (offline)'))));
+  const ipSel = h('select', {});
+  const refreshIPs = () => { const e = byId(st.edges, edgeSel.value); ipSel.replaceChildren(h('option', {value:'0.0.0.0'}, 'Every address on the edge'), ...((e && e.ips) || []).map(ip => h('option', {value:ip}, ip))); };
+  edgeSel.onchange = refreshIPs; refreshIPs();
+  const port = h('input', {type:'number', min:1, max:65535, value:2222, placeholder:'2222'});
+  const target = h('input', {value:SSH_TARGET});
+  const create = act(async () => {
+    const [h2, p2] = target.value.trim().split(':');
+    if (!st.edges.length) throw new Error('Add an edge first');
+    const m = await api('POST', O('/mappings'), {name:'SSH to ' + s.name, proto:'tcp', edgeId:edgeSel.value, bindIp:ipSel.value || '0.0.0.0', port:+port.value, tags:['ssh'], enabled:true,
+      targets:[{siteId:s.id, host:h2 || tHost, port:+p2 || tPort}]});
+    ports = [...ports, m]; draw(); toast('SSH port opened'); refresh();
+  });
+  const close = modal('SSH into ' + s.name, h('div', {class:'stack'},
+    h('p', {class:'dim', style:'margin:0'}, 'Bamboo has a small SSH server, so you can open a shell on this machine without installing sshd. Start Bamboo with -ssh (BAMBOO_SSH=1) and put the public keys that may sign in in authorized_keys in its data folder. Only keys are accepted, never passwords.'),
+    h('div', {style:'display:flex;gap:var(--s2);align-items:center;flex-wrap:wrap'},
+      s.sshRelay ? (s.ssh ? badge('SSH server running', 'success') : badge('SSH server off', 'warning')) : badge('Bamboo needs an update', 'warning'),
+      h('span', {class:'small dim'}, s.sshRelay ? (s.ssh ? 'Listening on ' + s.ssh + ' on that machine.' : 'Start Bamboo there with -ssh.') : 'This Bamboo is too old to report its SSH server.')),
+    h('fieldset', {}, h('legend', {}, 'Terminal in Hammerhead'), h('div', {class:'stack'},
+      h('p', {class:'dim', style:'margin:0'}, 'Open a shell here in the dashboard. Hammerhead signs in with a key of its own, so add this line to authorized_keys on the site first (Bamboo picks it up without a restart).'),
+      cmdBox(key.publicKey),
+      termNote,
+      h('div', {class:'actions'}, h('button', {class:'ghost', type:'button', onclick: act(async () => { await navigator.clipboard.writeText(key.publicKey); toast('Copied'); })}, 'Copy key'), termBtn))),
+    connectBox,
+    h('fieldset', {}, h('legend', {}, 'Use an ordinary ssh client (optional)'), h('div', {class:'stack'},
+      h('p', {class:'dim small', style:'margin:0'}, 'Only needed to connect from your own computer. The terminal above does not use this port.'),
+      h('div', {class:'grid'}, field('Edge', edgeSel), field('Public IP', ipSel), field('Public port', port, 'Anyone can reach it, but only listed keys sign in')),
+      field('Bamboo\'s SSH server on the site', target, 'Leave as is unless you changed -ssh-listen'),
+      h('div', {class:'actions form-actions'}, h('button', {class:'btn', type:'button', onclick:create}, 'Open SSH port')))),
+    h('div', {class:'actions form-actions'}, h('button', {class:'btn', onclick:() => close()}, 'Close'))), {wide:true});
+}
+
 views.sites = async () => {
   const st = S.st, name = h('input', {placeholder:'e.g. Home lab'});
   const add = act(async () => { const s = await api('POST', O('/sites'), {name:name.value}); name.value = ''; showCmd(s); refresh(); });
@@ -947,17 +1195,15 @@ views.sites = async () => {
   };
   return h('div', {},
     tagBar('sites', tags),
-    card('', h('table', {}, h('thead', {}, h('tr', {}, ['Status', 'Name', 'Kind', 'Tunnels to', 'Version', 'Containers', ''].map(t => h('th', {}, t)))),
+    card('', h('table', {}, h('thead', {}, h('tr', {}, ['Status', 'Name', 'Kind', 'Tunnels to', 'Version', 'Discovered', ''].map(t => h('th', {}, t)))),
       h('tbody', {}, st.sites.filter(s => hasTag(s, TAGSEL.sites)).map(s => h('tr', {},
         h('td', {}, st.online[s.id] ? badge('Online', 'success') : badge('Offline', 'danger')), h('td', {}, s.name, tagList(s)),
         h('td', {}, s.local ? 'This machine' : 'Bamboo agent'),
         h('td', {}, s.local ? 'Direct' : ((s.connected || []).map(id => (byId(st.edges, id) || {name:id}).name).join(', ') || 'nothing connected')),
         s.local ? h('td', {class:'dim'}, 'Same as its edge') : versionCell(s, latest.bamboo, !!s.updateId, O(`/sites/${s.id}/update`), O(`/sites/${s.id}/update/log`)),
-        h('td', {}, h('button', {class:'ghost', onclick: act(async () => {
-          const l = await api('GET', O(`/sites/${s.id}/containers`));
-          modal('Containers on ' + s.name, l.length ? h('table', {}, h('thead', {}, h('tr', {}, ['Name', 'Image', 'Ports'].map(t => h('th', {}, t)))), h('tbody', {}, l.map(c => h('tr', {}, h('td', {}, c.name), h('td', {class:'dim'}, c.image), h('td', {class:'mono small'}, c.ports.map(p => (p.public ? p.public + '→' : '') + p.private + '/' + p.type).join(', ')))))) :
-            h('p', {class:'dim'}, 'None reported. Start the agent with --docker (or BLACKTIP_DOCKER=1 / BAMBOO_DOCKER=1) on a machine where Docker is running.')); })}, 'View')),
+        h('td', {}, h('button', {class:'ghost', onclick: act(() => siteFound(s))}, 'View')),
         h('td', {}, s.local ? null : h('div', {class:'actions'},
+          h('button', {class:'ghost', onclick: act(() => sshAccess(s))}, 'SSH'),
           h('button', {class:'ghost', onclick:() => editSite(s)}, 'Edit'),
           h('button', {class:'ghost', onclick:() => showCmd(s)}, 'Show command'),
           h('button', {class:'ghost danger', onclick: ask({title:`Delete ${s.name}?`, body:'Every port that uses only this site is deleted with it, and its agent loses its connection. This cannot be undone.', confirm:'Delete site'}, async () => { await api('DELETE', O('/sites/' + s.id)); refresh(); })}, 'Delete')))))))));
@@ -982,23 +1228,58 @@ views.domains = async () => {
 // ----- mail egress -----
 const spfFor = ip => ip.includes(':') ? `v=spf1 ip6:${ip} ~all` : `v=spf1 ip4:${ip} ~all`;
 
+const findingBadge = f => f.status === 'pass' ? badge('Pass', 'success') : f.status === 'fail' ? badge('Fix', 'danger')
+  : f.status === 'none' ? (f.id.startsWith('mtasts:') ? badge('Optional') : badge('Set up', 'danger')) : badge('Check', 'warning');
+const findingsOk = e => !!e.check && (e.findings || []).every(f => f.status === 'pass' || f.status === 'warn' || (f.status === 'none' && f.id.startsWith('mtasts:')));
+
 function checklist(e) {
-  const c = e.check, ip = e.sourceIp, helo = e.heloName;
-  const item = (ok, title, fix) => h('div', {style:'display:flex;gap:10px;align-items:flex-start;margin-bottom:12px'},
-    h('span', {style:'margin-top:1px;flex:none'}, ok === true ? badge('Pass', 'success') : ok === false ? badge('Fix', 'danger') : badge('Check', 'warning')),
-    h('div', {}, h('div', {}, title), fix ? h('div', {class:'dim small'}, fix) : null));
-  const body = h('div', {},
+  const c = e.check, ip = e.sourceIp;
+  const row = f => h('div', {style:'display:flex;gap:10px;align-items:flex-start;margin-bottom:12px'},
+    h('span', {style:'margin-top:1px;flex:none'}, findingBadge(f)),
+    h('div', {}, h('div', {}, h('strong', {}, f.label + (f.domain ? ' · ' + f.domain : '')), ' ', f.detail), f.hint ? h('div', {class:'dim small'}, f.hint) : null));
+  const o = c && c.outcomes;
+  return h('div', {},
     c ? h('p', {class:'dim small'}, 'Checked ' + ago(c.checkedAt) + ' from the edge, using ' + ip + '.') : h('p', {class:'dim'}, 'No report yet. The edge sends one shortly after it picks up this egress.'),
-    c ? item(c.ptrMatchesHelo, 'Reverse DNS of ' + ip + (c.ptr.length ? ' is ' + c.ptr.join(', ') : ' is not set'),
-      c.ptrMatchesHelo ? null : `Receiving servers check this. Ask your VPS provider to set the reverse DNS (PTR) of ${ip} to ${helo}.`) : null,
-    c ? item(c.heloResolves, helo + (c.heloResolves ? ' resolves to ' + ip : ' does not resolve to ' + ip),
-      c.heloResolves ? null : `Create an A record: ${helo}  →  ${ip}  (it must match the reverse DNS).`) : null,
-    c ? item(c.port25 === 'open', 'Outbound port 25 from ' + ip + ': ' + (c.port25 === 'open' ? 'open' : 'blocked'),
-      c.port25 === 'open' ? null : c.port25) : null,
-    item(null, 'SPF', 'On each domain you send from, publish a TXT record, or add this address to your existing one:'),
-    cmdBox(spfFor(ip)),
-    item(null, 'DKIM and DMARC', 'Keep signing mail with DKIM in your mail server, as before. This relay passes the message through untouched, so signatures stay valid.'));
-  return body;
+    ...(e.findings || []).map(row),
+    !(e.findings || []).some(f => f.id.startsWith('spf:')) && c ? row({id:'spf', label:'SPF', status:'warn', detail:'Create a DKIM key for each domain you send from and the edge will check its SPF, DMARC and MTA-STS records too.', hint:'Meanwhile, publish this on each sending domain: ' + spfFor(ip)}) : null,
+    o ? h('div', {style:'margin-top:16px'}, h('h3', {}, 'Delivery, last 24 hours'),
+      h('p', {class:'small'}, `${o.delivered} delivered, ${o.deferred} deferred, ${o.failed} refused.`),
+      Object.keys(o.byDomain || {}).length ? h('table', {}, h('thead', {}, h('tr', {}, ['Recipient domain', 'Delivered', 'Deferred', 'Refused'].map(t => h('th', {}, t)))),
+        h('tbody', {}, Object.entries(o.byDomain).sort((a, b) => (b[1].failed + b[1].deferred) - (a[1].failed + a[1].deferred)).slice(0, 15).map(([d, n]) =>
+          h('tr', {}, h('td', {}, d), h('td', {}, n.delivered), h('td', {}, n.deferred), h('td', {}, n.failed))))) : null,
+      (o.recent || []).length ? h('div', {style:'margin-top:8px'}, h('div', {class:'dim small'}, 'Latest problems'),
+        ...o.recent.slice(0, 5).map(r => h('div', {class:'mono small'}, `${ago(r.time)} · ${r.domain} · ${r.code} ${r.text}`))) : null) : null);
+}
+
+async function dmarcCard() {
+  const doms = await api('GET', O('/dmarc'));
+  const upload = h('input', {type:'file', multiple:true, accept:'.xml,.gz,.zip,.eml,.txt', style:'display:none'});
+  upload.onchange = act(async () => {
+    let added = 0, dup = 0, msgs = 0;
+    for (const f of upload.files) {
+      const r = await fetch(O('/dmarc'), {method:'POST', body: await f.arrayBuffer()});
+      const j = await r.json().catch(() => ({}));
+      if (!r.ok) throw new Error(f.name + ': ' + (j.error || r.statusText));
+      j.duplicate ? dup++ : (added++, msgs += j.messages);
+    }
+    upload.value = '';
+    toast(`${added} report${added === 1 ? '' : 's'} added (${msgs} messages)` + (dup ? `, ${dup} already uploaded` : ''));
+    refresh();
+  });
+  const pct = (a, b) => b ? Math.round(100 * a / b) + '%' : '-';
+  return card('DMARC reports',
+    h('p', {class:'dim'}, 'Add rua=mailto:you@yourdomain to the DMARC record of a domain and receivers send a daily summary of every message that claimed to be from it. Upload those emails or their attachments (.xml, .gz, .zip or .eml) to see what passed and which senders are failing. Reports for the last 30 days are shown.'),
+    doms.length ? h('div', {}, ...doms.map(d => h('div', {style:'margin-bottom:18px'},
+      h('div', {class:'row-h'}, h('strong', {}, d.domain), ' ', badge('Policy: ' + (d.policy || '?')), ' ',
+        badge(pct(d.passed, d.messages) + ' passed', d.messages && d.passed / d.messages >= 0.9 ? 'success' : 'danger'),
+        h('span', {class:'dim small', style:'margin-left:8px'}, `${d.messages} messages, ${d.reports} reports from ${d.reporters.join(', ')}`),
+        h('button', {class:'ghost danger', style:'margin-left:auto', onclick: ask({title:`Delete the DMARC reports for ${d.domain}?`, body:'The uploaded reports are removed. You can upload them again.', confirm:'Delete reports'}, async () => { await api('DELETE', O('/dmarc/' + d.domain)); refresh(); })}, 'Delete')),
+      ...(d.advice || []).map(a => h('p', {class:'small'}, a)),
+      h('table', {}, h('thead', {}, h('tr', {}, ['Sending address', 'Messages', 'Passed', 'DKIM', 'SPF', 'Fix'].map(t => h('th', {}, t)))),
+        h('tbody', {}, d.sources.slice(0, 12).map(s => h('tr', {}, h('td', {class:'mono'}, s.ip, s.ours ? h('span', {style:'margin-left:6px'}, badge('Your relay', 'accent')) : null),
+          h('td', {}, s.messages), h('td', {}, pct(s.passed, s.messages)), h('td', {}, pct(s.dkimPass, s.messages)), h('td', {}, pct(s.spfPass, s.messages)),
+          h('td', {class:'dim small'}, s.hint || '')))))))) : h('p', {class:'dim'}, 'No reports yet.'),
+    h('div', {class:'row-h'}, h('button', {class:'ghost', onclick:() => upload.click()}, icon('add', 16), 'Upload reports'), upload));
 }
 
 views.egress = async () => {
@@ -1015,7 +1296,7 @@ views.egress = async () => {
         else if (e.status) { tone = 'danger'; label = 'Error'; why = e.status; }
         else { tone = 'success'; label = 'Listening'; why = 'listening'; }
       }
-      const good = c && c.ptrMatchesHelo && c.heloResolves && c.port25 === 'open';
+      const good = findingsOk(e);
       return h('tr', {},
         h('td', {title:why}, badge(label, tone)),
         h('td', {}, e.name, e.statusKnown && e.status ? h('div', {class:'err small mono'}, e.status) : null,
@@ -1034,6 +1315,7 @@ views.egress = async () => {
     !remote.length ? h('div', {class:'notice'}, 'This runs on a Bamboo site: add the network your mail server lives in on the Sites page and start Bamboo there first.') : null,
     h('div', {class:'card flush'}, table),
     await dkimCard(),
+    await dmarcCard(),
     card('How it works', h('p', {class:'dim'}, 'Bamboo opens a small SMTP relay on your mail server\'s network. Point your mail server at it, and each outgoing message is carried through the WireGuard tunnel to the edge, which delivers it straight to the recipient\'s mail server from the edge\'s IP, announcing itself with the HELO name you set. The recipient\'s replies come straight back, so your server\'s queue and retries work as usual.'),
       cmdBox('# Postfix (main.cf)\nrelayhost = [127.0.0.1]:2525\n\n# Exim: use a smarthost router pointing at 127.0.0.1 port 2525')));
 };
@@ -1093,40 +1375,52 @@ const FORMATS = [['json', 'JSON'], ['slack', 'Slack'], ['discord', 'Discord'], [
 function channelForm(base, existing, trusted) {
   const c = existing || {kind:'email', format:'json', enabled:true};
   const name = h('input', {value:c.name || '', placeholder:'e.g. On-call'});
-  const kind = h('select', {}, [['email', 'Email'], ['sandbar', 'Sandbar'], ['webhook', 'Webhook']].map(([v, t]) => h('option', {value:v, selected:v === c.kind}, t)));
+  const kind = h('select', {}, [['email', 'Email'], ['sandbar', 'Sandbar (pages on-call)'], ['topic', 'Sandbar topic'], ['webhook', 'Webhook']].map(([v, t]) => h('option', {value:v, selected:v === c.kind}, t)));
   const target = h('input', {value:c.kind === 'email' ? c.target || '' : '', placeholder:c.kind === 'webhook' && existing ? 'unchanged' : ''});
   const token = h('input', {type:'password', autocomplete:'off', placeholder:existing && c.kind === 'sandbar' ? 'unchanged' : 'sb_…'});
   const policy = h('input', {value:c.policy || '', placeholder:'Optional'});
+  const topic = h('input', {value:c.topic || '', placeholder:'e.g. alerts'});
+  const topicToken = h('input', {type:'password', autocomplete:'off', placeholder:existing && c.kind === 'topic' ? 'unchanged' : 'Optional'});
   const sandbarBox = h('div', {class:'grid'}, field('API key', token, 'Create one on Sandbar\'s Account page.'), field('Escalation policy ID', policy, 'Who gets paged. Leave blank to send alerts without paging anyone.'));
+  const topicBox = h('div', {class:'grid'}, field('Topic', topic, 'Messages are published here, so anyone subscribed (including the ntfy app) sees them.'), field('API key (optional)', topicToken, 'Leave blank for a server that lets anyone publish. Sandbar itself asks for a key.'));
   const fmt = h('select', {}, FORMATS.map(([v, t]) => h('option', {value:v, selected:v === (c.format || 'json')}, t)));
   const priv = chk('This address is on a private network (like a home server)', c.allowPrivate);
   const en = chk('Enabled', c.enabled !== false);
   const fmtBox = field('Webhook format', fmt, 'Slack and Discord incoming-webhook URLs work as they are. ntfy sends a push notification.');
   const targetBox = field('Where', target);
   const redraw = () => {
-    const w = kind.value === 'webhook', sb = kind.value === 'sandbar';
+    const w = kind.value === 'webhook', sb = kind.value === 'sandbar' || kind.value === 'topic';
     fmtBox.style.display = w ? '' : 'none';
-    sandbarBox.style.display = sb ? '' : 'none';
+    sandbarBox.style.display = kind.value === 'sandbar' ? '' : 'none';
+    topicBox.style.display = kind.value === 'topic' ? '' : 'none';
     priv.style.display = (w || sb) && trusted ? '' : 'none';
     targetBox.firstChild.textContent =sb ? 'Sandbar address' : 'Where';
-    target.placeholder = sb ? (existing && c.kind === 'sandbar' ? 'unchanged' : 'https://sandbar.example.com') :
+    target.placeholder = sb ? (existing && c.kind === kind.value ? 'unchanged' : 'https://sandbar.example.com') :
       w ? (existing && c.kind === 'webhook' ? 'unchanged (it may contain a secret token)' : 'https://hooks.example.com/…') : 'you@example.com, oncall@example.com';
   };
   kind.onchange = redraw; redraw();
   const save = act(async () => {
-    await api(existing ? 'PUT' : 'POST', base + '/channels' + (existing ? '/' + existing.id : ''), {name:name.value, kind:kind.value, target:target.value, format:kind.value === 'webhook' ? fmt.value : '', token:token.value, policy:policy.value, allowPrivate:trusted && priv.box.checked, enabled:en.box.checked});
+    await api(existing ? 'PUT' : 'POST', base + '/channels' + (existing ? '/' + existing.id : ''), {name:name.value, kind:kind.value, target:target.value, format:kind.value === 'webhook' ? fmt.value : '', token:kind.value === 'topic' ? topicToken.value : token.value, policy:policy.value, topic:topic.value, allowPrivate:trusted && priv.box.checked, enabled:en.box.checked});
     close(); toast(existing ? 'Alert channel saved' : 'Alert channel added'); refresh();
   });
   const close = modal(existing ? 'Edit channel' : 'Add an alert channel', h('div', {class:'stack'},
-    h('div', {class:'grid'}, field('Name', name), field('Type', kind)), targetBox, sandbarBox, fmtBox, priv, en,
+    h('div', {class:'grid'}, field('Name', name), field('Type', kind)), targetBox, sandbarBox, topicBox, fmtBox, priv, en,
     h('div', {class:'actions form-actions'}, h('button', {class:'ghost', onclick:() => close()}, 'Cancel'), h('button', {class:'btn', onclick:save}, 'Save'))), {sticky:true});
 }
 
+function trafficAlertCard(base, settings) {
+  const spike = h('input', {type:'number', min:0, max:1000, value:settings.spikeFactor || 0}), daily = h('input', {type:'number', min:0, value:settings.dailyMb || 0});
+  return card('Traffic', h('p', {class:'dim'}, 'Be told when a port suddenly carries far more than it usually does, or more than you expect in a day. These apply to every port in this organization. Leave them at 0 to switch them off.'),
+    h('div', {class:'grid'}, field('Spike: times the usual traffic', spike, 'Judged on the last 5 minutes against the previous hours, once a port has an hour of history and carried over 10 MB in those minutes. Try 5.'),
+      field('Most traffic a port may carry in a day (MB)', daily, 'Counted per port from midnight UTC, in and out together.'),
+      h('button', {class:'ghost', onclick: act(async () => { await api('PUT', base + '/traffic', {spikeFactor:+spike.value, dailyMb:+daily.value}); toast('Traffic alerts saved'); })}, 'Save traffic alerts')));
+}
+
 async function alertsView(base, scoped, trusted) {
   const d = await api('GET', base);
   const channels = h('table', {}, h('thead', {}, h('tr', {}, ['Name', 'Type', 'Sends to', ''].map(x => h('th', {}, x)))),
     h('tbody', {}, d.channels.map(c => h('tr', {}, h('td', {}, c.name, c.enabled ? null : h('span', {class:'tag', style:'margin-left:6px'}, 'off')),
-      h('td', {}, c.kind === 'email' ? 'Email' : c.kind === 'sandbar' ? 'Sandbar' : 'Webhook (' + c.format + ')'), h('td', {class:'mono small'}, c.preview),
+      h('td', {}, c.kind === 'email' ? 'Email' : c.kind === 'sandbar' ? 'Sandbar' : c.kind === 'topic' ? 'Sandbar topic' : 'Webhook (' + c.format + ')'), h('td', {class:'mono small'}, c.preview),
       h('td', {}, h('div', {class:'actions'},
         h('button', {class:'ghost', onclick: act(async () => { await api('POST', base + '/channels/' + c.id + '/test'); toast('Test sent'); })}, 'Send test'),
         h('button', {class:'ghost', onclick:() => channelForm(base, c, trusted)}, 'Edit'),
@@ -1142,6 +1436,7 @@ async function alertsView(base, scoped, trusted) {
     scoped ? card('Timing', h('p', {class:'dim'}, 'A problem has to last this long before you are told, so a restart or a brief blip stays quiet. You get one message when it starts and one when it clears.'),
       h('div', {class:'grid'}, field('Wait this many minutes first', debounce), field('Remind me every (hours)', repeat, 'Use 0 to send no reminders'),
         h('button', {class:'ghost', onclick: act(async () => { await api('PUT', base + '/settings', {debounceMin:+debounce.value, repeatHours:+repeat.value}); toast('Alert timing saved'); })}, 'Save timing'))) : null,
+    scoped ? trafficAlertCard(base, d.settings) : null,
     card('History', d.history.length ? h('table', {}, h('tbody', {}, d.history.map(e => h('tr', {}, h('td', {class:'dim small'}, fmtTime(e.time)),
       h('td', {}, h('span', {class:'tag'}, e.status)), h('td', {}, e.message))))) : emptyState('alerts', 'No alerts yet', 'Alerts that fired and recovered are listed here.')));
 }
@@ -1307,13 +1602,19 @@ serverViews.idps = async () => {
 };
 
 serverViews.settings = async () => {
-  const { settings: s, smtpPassSet } = await api('GET', '/api/admin/settings');
+  const { settings: s, smtpPassSet, sandbarTokenSet, sandbarSync } = await api('GET', '/api/admin/settings');
   const f = {url:h('input', {value:s.dashboardUrl, placeholder:'https://hammerhead.example.com'}), acme:h('input', {type:'email', value:s.acmeEmail, placeholder:'you@example.com'}),
     host:h('input', {value:s.smtp.host}), port:h('input', {type:'number', value:s.smtp.port || ''}), user:h('input', {value:s.smtp.user}), pass:h('input', {type:'password', placeholder:smtpPassSet ? 'unchanged' : ''}), from:h('input', {value:s.smtp.from, placeholder:'Hammerhead <no-reply@example.com>'}),
     mode:h('select', {}, [['starttls', 'STARTTLS'], ['tls', 'TLS'], ['none', 'None']].map(([m, l]) => h('option', {value:m, selected:(s.smtp.mode || 'starttls') === m}, l)))};
   const staging = chk('Use the Let\'s Encrypt staging server (untrusted test certificates)', s.acmeStaging);
+  const sb = s.sandbar || {};
+  const sbOn = chk('Create a Sandbar monitor for every resource that has health checks', sb.enabled);
+  const sbf = {url:h('input', {value:sb.url || '', placeholder:'https://sandbar.example.com'}), key:h('input', {type:'password', autocomplete:'off', placeholder:sandbarTokenSet ? 'unchanged' : 'sb_…'}), policy:h('input', {value:sb.policyId || '', placeholder:'Optional'})};
+  const sbPublic = chk('Show the monitors on the status page', sb.public), sbPriv = chk('Sandbar and the edge addresses are on a private network', sb.allowPrivate);
+  const sbOut = h('div', {class:'dim'}, sandbarSync && sandbarSync.at && sandbarSync.at.slice(0, 4) !== '0001' ? (sandbarSync.error ? 'The last sync failed: ' + sandbarSync.error : 'Last synced ' + ago(sandbarSync.at) + ', ' + sandbarSync.monitors + ' monitors.') : 'Not synced yet.');
   const save = act(async () => { await api('PUT', '/api/admin/settings', {dashboardUrl:f.url.value, acmeEmail:f.acme.value, acmeStaging:staging.box.checked,
-    smtp:{host:f.host.value, port:+f.port.value, user:f.user.value, pass:f.pass.value, from:f.from.value, mode:f.mode.value}}); S.me = await api('GET', '/api/me'); toast('Settings saved'); });
+    smtp:{host:f.host.value, port:+f.port.value, user:f.user.value, pass:f.pass.value, from:f.from.value, mode:f.mode.value},
+    sandbar:{enabled:sbOn.box.checked, url:sbf.url.value, token:sbf.key.value, policyId:sbf.policy.value, public:sbPublic.box.checked, allowPrivate:sbPriv.box.checked}}); S.me = await api('GET', '/api/me'); toast('Settings saved'); });
   const updOut = h('div', {class:'dim'}, 'Not checked yet.');
   const updNow = h('button', {class:'ghost', style:'display:none', onclick: act(async () => {
     await api('POST', '/api/admin/update/apply');
@@ -1341,6 +1642,9 @@ serverViews.settings = async () => {
     card('Email', h('p', {class:'dim'}, 'Needed for invitation emails and one-time sign-in codes.'),
       h('div', {class:'grid'}, field('SMTP host', f.host), field('Port', f.port), field('Security', f.mode), field('Username', f.user), field('Password', f.pass), field('From', f.from)),
       h('div', {class:'actions', style:'margin-top:10px'}, h('button', {class:'ghost', onclick: act(async () => { await save(); await api('POST', '/api/admin/settings/test-mail'); toast('Test email sent to ' + S.me.email); })}, 'Save and send a test email'))),
+    card('Sandbar monitors', h('p', {class:'dim'}, 'Hammerhead adds a monitor to Sandbar for each enabled resource with health checks (a website is fetched at its address, a port is connected to), so it gets uptime history, a status page and on-call paging. Monitors you add in Sandbar yourself are never touched. Hammerhead keeps them up to date every minute and removes the ones it created when a resource is deleted or its checks are turned off.'),
+      sbOn, h('div', {class:'grid'}, field('Sandbar address', sbf.url), field('Administrator API key', sbf.key, 'Create one on the Account page in Sandbar. Managing monitors needs an administrator.'), field('Escalation policy ID', sbf.policy, 'Who gets paged when a monitor goes down. Leave blank to page nobody.')), sbPublic, sbPriv,
+      h('div', {class:'actions', style:'margin-top:10px'}, h('button', {class:'ghost', onclick: act(async () => { await save(); const r = await api('POST', '/api/admin/settings/sandbar-sync'); sbOut.textContent = 'Synced, ' + r.monitors + ' monitors.'; toast('Sandbar monitors synced'); })}, 'Save and sync now')), sbOut),
     h('div', {class:'actions form-actions', style:'margin-bottom:24px'}, h('button', {class:'btn', onclick:save}, 'Save settings')),
     card('Updates', h('p', {class:'dim'}, 'Hammerhead, Blacktip and Bamboo can update themselves from new releases when auto-update is on (HAMMERHEAD_AUTO_UPDATE=1). This checks the newest release for Hammerhead; it installs nothing.'),
       updOut, h('div', {class:'actions', style:'margin-top:10px'}, h('button', {class:'ghost', onclick:checkUpdate}, 'Check for updates'), updNow), updBox));

web/term/LICENSE-xterm.txt

@@ -0,0 +1,21 @@
+Copyright (c) 2017-2019, The xterm.js authors (https://github.com/xtermjs/xterm.js)
+Copyright (c) 2014-2016, SourceLair Private Company (https://www.sourcelair.com)
+Copyright (c) 2012-2013, Christopher Jeffrey (https://github.com/chjj/)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

web/term/README.txt

@@ -0,0 +1,3 @@
+The web terminal uses xterm.js 6.0.0 and its fit addon 0.11.0 (MIT license, see LICENSE-xterm.txt),
+copied from the npm packages @xterm/xterm and @xterm/addon-fit. The only change is that the trailing
+source map comment is removed, because the map files are not shipped.

web/term/addon-fit.js

@@ -0,0 +1 @@
+!function(e,t){"object"==typeof exports&&"object"==typeof module?module.exports=t():"function"==typeof define&&define.amd?define([],t):"object"==typeof exports?exports.FitAddon=t():e.FitAddon=t()}(globalThis,(()=>(()=>{"use strict";var e={};return(()=>{var t=e;Object.defineProperty(t,"__esModule",{value:!0}),t.FitAddon=void 0,t.FitAddon=class{activate(e){this._terminal=e}dispose(){}fit(){const e=this.proposeDimensions();if(!e||!this._terminal||isNaN(e.cols)||isNaN(e.rows))return;const t=this._terminal._core;this._terminal.rows===e.rows&&this._terminal.cols===e.cols||(t._renderService.clear(),this._terminal.resize(e.cols,e.rows))}proposeDimensions(){if(!this._terminal)return;if(!this._terminal.element||!this._terminal.element.parentElement)return;const e=this._terminal._core._renderService.dimensions;if(0===e.css.cell.width||0===e.css.cell.height)return;const t=0===this._terminal.options.scrollback?0:this._terminal.options.overviewRuler?.width||14,r=window.getComputedStyle(this._terminal.element.parentElement),i=parseInt(r.getPropertyValue("height")),o=Math.max(0,parseInt(r.getPropertyValue("width"))),s=window.getComputedStyle(this._terminal.element),n=i-(parseInt(s.getPropertyValue("padding-top"))+parseInt(s.getPropertyValue("padding-bottom"))),l=o-(parseInt(s.getPropertyValue("padding-right"))+parseInt(s.getPropertyValue("padding-left")))-t;return{cols:Math.max(2,Math.floor(l/e.css.cell.width)),rows:Math.max(1,Math.floor(n/e.css.cell.height))}}}})(),e})()));

web/term/xterm.css

@@ -0,0 +1,285 @@
+/**
+ * Copyright (c) 2014 The xterm.js authors. All rights reserved.
+ * Copyright (c) 2012-2013, Christopher Jeffrey (MIT License)
+ * https://github.com/chjj/term.js
+ * @license MIT
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ *
+ * Originally forked from (with the author's permission):
+ *   Fabrice Bellard's javascript vt100 for jslinux:
+ *   http://bellard.org/jslinux/
+ *   Copyright (c) 2011 Fabrice Bellard
+ *   The original design remains. The terminal itself
+ *   has been extended to include xterm CSI codes, among
+ *   other features.
+ */
+
+/**
+ *  Default styles for xterm.js
+ */
+
+.xterm {
+    cursor: text;
+    position: relative;
+    user-select: none;
+    -ms-user-select: none;
+    -webkit-user-select: none;
+}
+
+.xterm.focus,
+.xterm:focus {
+    outline: none;
+}
+
+.xterm .xterm-helpers {
+    position: absolute;
+    top: 0;
+    /**
+     * The z-index of the helpers must be higher than the canvases in order for
+     * IMEs to appear on top.
+     */
+    z-index: 5;
+}
+
+.xterm .xterm-helper-textarea {
+    padding: 0;
+    border: 0;
+    margin: 0;
+    /* Move textarea out of the screen to the far left, so that the cursor is not visible */
+    position: absolute;
+    opacity: 0;
+    left: -9999em;
+    top: 0;
+    width: 0;
+    height: 0;
+    z-index: -5;
+    /** Prevent wrapping so the IME appears against the textarea at the correct position */
+    white-space: nowrap;
+    overflow: hidden;
+    resize: none;
+}
+
+.xterm .composition-view {
+    /* TODO: Composition position got messed up somewhere */
+    background: #000;
+    color: #FFF;
+    display: none;
+    position: absolute;
+    white-space: nowrap;
+    z-index: 1;
+}
+
+.xterm .composition-view.active {
+    display: block;
+}
+
+.xterm .xterm-viewport {
+    /* On OS X this is required in order for the scroll bar to appear fully opaque */
+    background-color: #000;
+    overflow-y: scroll;
+    cursor: default;
+    position: absolute;
+    right: 0;
+    left: 0;
+    top: 0;
+    bottom: 0;
+}
+
+.xterm .xterm-screen {
+    position: relative;
+}
+
+.xterm .xterm-screen canvas {
+    position: absolute;
+    left: 0;
+    top: 0;
+}
+
+.xterm-char-measure-element {
+    display: inline-block;
+    visibility: hidden;
+    position: absolute;
+    top: 0;
+    left: -9999em;
+    line-height: normal;
+}
+
+.xterm.enable-mouse-events {
+    /* When mouse events are enabled (eg. tmux), revert to the standard pointer cursor */
+    cursor: default;
+}
+
+.xterm.xterm-cursor-pointer,
+.xterm .xterm-cursor-pointer {
+    cursor: pointer;
+}
+
+.xterm.column-select.focus {
+    /* Column selection mode */
+    cursor: crosshair;
+}
+
+.xterm .xterm-accessibility:not(.debug),
+.xterm .xterm-message {
+    position: absolute;
+    left: 0;
+    top: 0;
+    bottom: 0;
+    right: 0;
+    z-index: 10;
+    color: transparent;
+    pointer-events: none;
+}
+
+.xterm .xterm-accessibility-tree:not(.debug) *::selection {
+  color: transparent;
+}
+
+.xterm .xterm-accessibility-tree {
+  font-family: monospace;
+  user-select: text;
+  white-space: pre;
+}
+
+.xterm .xterm-accessibility-tree > div {
+  transform-origin: left;
+  width: fit-content;
+}
+
+.xterm .live-region {
+    position: absolute;
+    left: -9999px;
+    width: 1px;
+    height: 1px;
+    overflow: hidden;
+}
+
+.xterm-dim {
+    /* Dim should not apply to background, so the opacity of the foreground color is applied
+     * explicitly in the generated class and reset to 1 here */
+    opacity: 1 !important;
+}
+
+.xterm-underline-1 { text-decoration: underline; }
+.xterm-underline-2 { text-decoration: double underline; }
+.xterm-underline-3 { text-decoration: wavy underline; }
+.xterm-underline-4 { text-decoration: dotted underline; }
+.xterm-underline-5 { text-decoration: dashed underline; }
+
+.xterm-overline {
+    text-decoration: overline;
+}
+
+.xterm-overline.xterm-underline-1 { text-decoration: overline underline; }
+.xterm-overline.xterm-underline-2 { text-decoration: overline double underline; }
+.xterm-overline.xterm-underline-3 { text-decoration: overline wavy underline; }
+.xterm-overline.xterm-underline-4 { text-decoration: overline dotted underline; }
+.xterm-overline.xterm-underline-5 { text-decoration: overline dashed underline; }
+
+.xterm-strikethrough {
+    text-decoration: line-through;
+}
+
+.xterm-screen .xterm-decoration-container .xterm-decoration {
+	z-index: 6;
+	position: absolute;
+}
+
+.xterm-screen .xterm-decoration-container .xterm-decoration.xterm-decoration-top-layer {
+	z-index: 7;
+}
+
+.xterm-decoration-overview-ruler {
+    z-index: 8;
+    position: absolute;
+    top: 0;
+    right: 0;
+    pointer-events: none;
+}
+
+.xterm-decoration-top {
+    z-index: 2;
+    position: relative;
+}
+
+
+
+/* Derived from vs/base/browser/ui/scrollbar/media/scrollbar.css */
+
+/* xterm.js customization: Override xterm's cursor style */
+.xterm .xterm-scrollable-element > .scrollbar {
+    cursor: default;
+}
+
+/* Arrows */
+.xterm .xterm-scrollable-element > .scrollbar > .scra {
+	cursor: pointer;
+	font-size: 11px !important;
+}
+
+.xterm .xterm-scrollable-element > .visible {
+	opacity: 1;
+
+	/* Background rule added for IE9 - to allow clicks on dom node */
+	background:rgba(0,0,0,0);
+
+	transition: opacity 100ms linear;
+	/* In front of peek view */
+	z-index: 11;
+}
+.xterm .xterm-scrollable-element > .invisible {
+	opacity: 0;
+	pointer-events: none;
+}
+.xterm .xterm-scrollable-element > .invisible.fade {
+	transition: opacity 800ms linear;
+}
+
+/* Scrollable Content Inset Shadow */
+.xterm .xterm-scrollable-element > .shadow {
+	position: absolute;
+	display: none;
+}
+.xterm .xterm-scrollable-element > .shadow.top {
+	display: block;
+	top: 0;
+	left: 3px;
+	height: 3px;
+	width: 100%;
+	box-shadow: var(--vscode-scrollbar-shadow, #000) 0 6px 6px -6px inset;
+}
+.xterm .xterm-scrollable-element > .shadow.left {
+	display: block;
+	top: 3px;
+	left: 0;
+	height: 100%;
+	width: 3px;
+	box-shadow: var(--vscode-scrollbar-shadow, #000) 6px 0 6px -6px inset;
+}
+.xterm .xterm-scrollable-element > .shadow.top-left-corner {
+	display: block;
+	top: 0;
+	left: 0;
+	height: 3px;
+	width: 3px;
+}
+.xterm .xterm-scrollable-element > .shadow.top.left {
+	box-shadow: var(--vscode-scrollbar-shadow, #000) 6px 0 6px -6px inset;
+}

web/term/xterm.js

@@ -0,0 +1 @@
+!function(e,t){if("object"==typeof exports&&"object"==typeof module)module.exports=t();else if("function"==typeof define&&define.amd)define([],t);else{var i=t();for(var s in i)("object"==typeof exports?exports:e)[s]=i[s]}}(globalThis,(()=>(()=>{"use strict";var e={2840:function(e,t,i){var s=this&&this.__decorate||function(e,t,i,s){var r,n=arguments.length,o=n<3?t:null===s?s=Object.getOwnPropertyDescriptor(t,i):s;if("object"==typeof Reflect&&"function"==typeof Reflect.decorate)o=Reflect.decorate(e,t,i,s);else for(var a=e.length-1;a>=0;a--)(r=e[a])&&(o=(n<3?r(o):n>3?r(t,i,o):r(t,i))||o);return n>3&&o&&Object.defineProperty(t,i,o),o},r=this&&this.__param||function(e,t){return function(i,s){t(i,s,e)}};Object.defineProperty(t,"__esModule",{value:!0}),t.AccessibilityManager=void 0;const n=i(7721),o=i(4292),a=i(7150),l=i(7098),h=i(6501),c=i(7093);let d=class extends a.Disposable{constructor(e,t,i,s){super(),this._terminal=e,this._coreBrowserService=i,this._renderService=s,this._rowColumns=new WeakMap,this._liveRegionLineCount=0,this._charsToConsume=[],this._charsToAnnounce="";const r=this._coreBrowserService.mainDocument;this._accessibilityContainer=r.createElement("div"),this._accessibilityContainer.classList.add("xterm-accessibility"),this._rowContainer=r.createElement("div"),this._rowContainer.setAttribute("role","list"),this._rowContainer.classList.add("xterm-accessibility-tree"),this._rowElements=[];for(let e=0;e<this._terminal.rows;e++)this._rowElements[e]=this._createAccessibilityTreeNode(),this._rowContainer.appendChild(this._rowElements[e]);if(this._topBoundaryFocusListener=e=>this._handleBoundaryFocus(e,0),this._bottomBoundaryFocusListener=e=>this._handleBoundaryFocus(e,1),this._rowElements[0].addEventListener("focus",this._topBoundaryFocusListener),this._rowElements[this._rowElements.length-1].addEventListener("focus",this._bottomBoundaryFocusListener),this._accessibilityContainer.appendChild(this._rowContainer),this._liveRegion=r.createElement("div"),this._liveRegion.classList.add("live-region"),this._liveRegion.setAttribute("aria-live","assertive"),this._accessibilityContainer.appendChild(this._liveRegion),this._liveRegionDebouncer=this._register(new o.TimeBasedDebouncer(this._renderRows.bind(this))),!this._terminal.element)throw new Error("Cannot enable accessibility before Terminal.open");this._terminal.element.insertAdjacentElement("afterbegin",this._accessibilityContainer),this._register(this._terminal.onResize((e=>this._handleResize(e.rows)))),this._register(this._terminal.onRender((e=>this._refreshRows(e.start,e.end)))),this._register(this._terminal.onScroll((()=>this._refreshRows()))),this._register(this._terminal.onA11yChar((e=>this._handleChar(e)))),this._register(this._terminal.onLineFeed((()=>this._handleChar("\n")))),this._register(this._terminal.onA11yTab((e=>this._handleTab(e)))),this._register(this._terminal.onKey((e=>this._handleKey(e.key)))),this._register(this._terminal.onBlur((()=>this._clearLiveRegion()))),this._register(this._renderService.onDimensionsChange((()=>this._refreshRowsDimensions()))),this._register((0,c.addDisposableListener)(r,"selectionchange",(()=>this._handleSelectionChange()))),this._register(this._coreBrowserService.onDprChange((()=>this._refreshRowsDimensions()))),this._refreshRowsDimensions(),this._refreshRows(),this._register((0,a.toDisposable)((()=>{this._accessibilityContainer.remove(),this._rowElements.length=0})))}_handleTab(e){for(let t=0;t<e;t++)this._handleChar(" ")}_handleChar(e){this._liveRegionLineCount<21&&(this._charsToConsume.length>0?this._charsToConsume.shift()!==e&&(this._charsToAnnounce+=e):this._charsToAnnounce+=e,"\n"===e&&(this._liveRegionLineCount++,21===this._liveRegionLineCount&&(this._liveRegion.textContent+=n.tooMuchOutput.get())))}_clearLiveRegion(){this._liveRegion.textContent="",this._liveRegionLineCount=0}_handleKey(e){this._clearLiveRegion(),/\p{Control}/u.test(e)||this._charsToConsume.push(e)}_refreshRows(e,t){this._liveRegionDebouncer.refresh(e,t,this._terminal.rows)}_renderRows(e,t){const i=this._terminal.buffer,s=i.lines.length.toString();for(let r=e;r<=t;r++){const e=i.lines.get(i.ydisp+r),t=[],n=e?.translateToString(!0,void 0,void 0,t)||"",o=(i.ydisp+r+1).toString(),a=this._rowElements[r];a&&(0===n.length?(a.textContent=" ",this._rowColumns.set(a,[0,1])):(a.textContent=n,this._rowColumns.set(a,t)),a.setAttribute("aria-posinset",o),a.setAttribute("aria-setsize",s),this._alignRowWidth(a))}this._announceCharacters()}_announceCharacters(){0!==this._charsToAnnounce.length&&(this._liveRegion.textContent+=this._charsToAnnounce,this._charsToAnnounce="")}_handleBoundaryFocus(e,t){const i=e.target,s=this._rowElements[0===t?1:this._rowElements.length-2];if(i.getAttribute("aria-posinset")===(0===t?"1":`${this._terminal.buffer.lines.length}`))return;if(e.relatedTarget!==s)return;let r,n;if(0===t?(r=i,n=this._rowElements.pop(),this._rowContainer.removeChild(n)):(r=this._rowElements.shift(),n=i,this._rowContainer.removeChild(r)),r.removeLine truncated