Show a progress bar and live log while Hammerhead, edges and sites update

The updater reports each step (checking, downloading with percent, verifying, installing, restarting).
Edges and sites send that progress and their log with their reports; Hammerhead serves it to a dialog
that opens when an update starts, and keeps its own last update log on disk across the restart.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

.gitea/workflows/ci.yml

@@ -44,7 +44,7 @@ jobs:
       # Catches platform-specific breakage before a release is tagged.
       - name: Every release platform compiles
         run: |
-          for t in linux/amd64 linux/arm64 linux/arm windows/amd64 darwin/amd64 darwin/arm64; do
+          for t in linux/amd64 windows/amd64; do
             echo "building for $t"
             GOOS=${t%/*} GOARCH=${t#*/} GOARM=7 CGO_ENABLED=0 go build -o /dev/null ./cmd/hammerhead
           done

.gitea/workflows/release.yml

@@ -59,4 +59,4 @@ jobs:
             *-*) ;;                       # pre-releases (v1.0.0-rc1) don't move :latest
             *) tags="$tags -t $image:latest" ;;
           esac
-          docker buildx build --platform linux/amd64,linux/arm64 $tags --push .
+          docker buildx build --platform linux/amd64 $tags --push .

README.md

@@ -41,7 +41,7 @@ ports. Then open ports on the **Resources** page.
 
 ## Deploying
 
-Releases are built by Gitea Actions. Pushing a tag such as `v0.2.0` runs `.gitea/workflows/release.yml`, which tests the code, builds Hammerhead for Linux (amd64, arm64, armv7), Windows (amd64) and macOS (amd64, arm64), publishes the archives with a `SHA256SUMS` file on the Releases page, and pushes a multi-architecture container image to this server's registry (`git.veillax.com/veillax/hammerhead`). Every push and pull request runs `.gitea/workflows/ci.yml`: formatting, vet, tests, and a check that every release platform still compiles.
+Releases are built by Gitea Actions. Pushing a tag such as `v0.2.0` runs `.gitea/workflows/release.yml`, which tests the code, builds Hammerhead for Linux (amd64) and Windows (amd64), publishes the archives with a `SHA256SUMS` file on the Releases page, and pushes a container image (linux/amd64) to this server's registry (`git.veillax.com/veillax/hammerhead`). Every push and pull request runs `.gitea/workflows/ci.yml`: formatting, vet, tests, and a check that every release platform still compiles.
 
 **Fastest: one script on a fresh VPS**
 
@@ -67,12 +67,14 @@ sudoedit /etc/hammerhead/hammerhead.env   # your settings and secrets
 sudo systemctl start hammerhead
 ```
 
-Use `linux_arm64` or `linux_arm` for Raspberry Pi and other ARM machines. To upgrade, repeat the same steps: your settings and data are kept. `sudo ./install.sh --uninstall` removes the service and binary but leaves `/etc/hammerhead` and `/var/lib/hammerhead` alone. `hammerhead -version` tells you which build is running.
+Releases are published for x86-64 only; for ARM machines build it yourself (`TARGETS=linux/arm64 sh deploy/build.sh`). To upgrade, repeat the same steps: your settings and data are kept. `sudo ./install.sh --uninstall` removes the service and binary but leaves `/etc/hammerhead` and `/var/lib/hammerhead` alone. `hammerhead -version` tells you which build is running.
 
 **Updates**
 
 Set `HAMMERHEAD_AUTO_UPDATE=1` (or pass `-auto-update`) and Hammerhead checks this server for a newer release every few hours, verifies it against `SHA256SUMS`, checks that the new binary runs, swaps it in and restarts itself. It only follows full releases (never pre-releases), never downgrades, and keeps the old binary if anything looks wrong. `setup.sh` turns it on. It is not available on Windows, and it is for the systemd and plain-binary installs: with Docker, pull a new image instead.
 
+Edges and sites report their version, so Server → Edges and Sites show which release each one runs and flag a newer one. An **Update** button appears for any machine that runs with auto-update on (`BLACKTIP_AUTO_UPDATE=1` / `BAMBOO_AUTO_UPDATE=1`): it asks that machine to install the newest release now, and the result is shown in the table. Under Server → Settings, **Check for updates** does the same lookup for Hammerhead itself, with an **Update now** button when auto-update is on. A machine without auto-update never installs anything on request. Starting an update opens a progress bar with the live log of what the machine is doing (checking, downloading, verifying, installing, restarting). Hammerhead keeps the log of its own last update in `update.log` in its data folder, so it is still there after the restart.
+
 **With Docker**
 
 ```

agent.go

@@ -76,6 +76,7 @@ func (s *Server) edgeConfig(w http.ResponseWriter, r *http.Request) {
 	if m := s.selfMapping(e); m != nil {
 		cfg.Mappings = append(cfg.Mappings, *m)
 	}
+	cfg.UpdateID = e.UpdateID
 	cfg.MailEgress = []model.MailPolicy{}
 	for _, x := range s.st.MailEgress {
 		if x.EdgeID == e.ID && x.Enabled {
@@ -110,6 +111,9 @@ func (s *Server) edgeReport(w http.ResponseWriter, r *http.Request) {
 	}
 	e.LastSeen = time.Now()
 	e.IPs = in.IPs
+	oldVersion := e.Version
+	answered := noteUpdateReport(&e.Version, &e.AutoUpdate, &e.UpdateID, &e.UpdateMsg, in.Version, in.AutoUpdate, in.UpdateID, in.UpdateMsg)
+	s.recordUpdateProgress(e.ID, in.UpdateID, in.UpdateStage, in.UpdatePct, in.UpdateLog, oldVersion, e.Version)
 	for _, x := range s.st.MailEgress {
 		if x.EdgeID == e.ID {
 			if chk, found := in.MailChecks[x.ID]; found {
@@ -146,6 +150,8 @@ func (s *Server) edgeReport(w http.ResponseWriter, r *http.Request) {
 	if e.PublicKey != in.PublicKey {
 		e.PublicKey = in.PublicKey
 		s.bump() // sites need the edge's new key
+	} else if answered {
+		s.bump() // the edge's config no longer carries the update request
 	} else {
 		s.save()
 	}
@@ -205,7 +211,7 @@ func (s *Server) siteConfig(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 	st.LastSeen = time.Now()
-	cfg := model.SiteConfig{Version: s.st.Version, SiteID: st.ID, TunnelIP: st.TunnelIP, Edges: []model.EdgeRef{}, MailEgress: []model.MailSpec{}}
+	cfg := model.SiteConfig{Version: s.st.Version, SiteID: st.ID, TunnelIP: st.TunnelIP, Edges: []model.EdgeRef{}, MailEgress: []model.MailSpec{}, UpdateID: st.UpdateID}
 	seen := map[string]bool{}
 	addEdge := func(id string) {
 		if seen[id] {
@@ -249,6 +255,9 @@ func (s *Server) siteReport(w http.ResponseWriter, r *http.Request) {
 		in.Connected = []string{}
 	}
 	st.Connected = in.Connected
+	oldVersion := st.Version
+	answered := noteUpdateReport(&st.Version, &st.AutoUpdate, &st.UpdateID, &st.UpdateMsg, in.Version, in.AutoUpdate, in.UpdateID, in.UpdateMsg)
+	s.recordUpdateProgress(st.ID, in.UpdateID, in.UpdateStage, in.UpdatePct, in.UpdateLog, oldVersion, st.Version)
 	s.containers[st.ID] = in.Containers
 	for _, x := range s.st.MailEgress {
 		if x.SiteID != st.ID {
@@ -263,6 +272,8 @@ func (s *Server) siteReport(w http.ResponseWriter, r *http.Request) {
 	if in.PublicKey != "" && st.PublicKey != in.PublicKey {
 		st.PublicKey = in.PublicKey
 		s.bump() // every edge this site talks to needs the new key
+	} else if answered {
+		s.bump() // the site's config no longer carries the update request
 	}
 	ok(w)
 }

cmd/hammerhead/main.go

@@ -78,6 +78,8 @@ func main() {
 		SelfDomain:       os.Getenv("HAMMERHEAD_SELF_DOMAIN"),
 		SelfTarget:       localAddr(*listen),
 		SelfTunnel:       os.Getenv("HAMMERHEAD_SELF_TUNNEL"),
+		Version:          version,
+		AutoUpdate:       *autoUpdate,
 	})
 	if err != nil {
 		log.Fatal(err)

deploy/build.sh

@@ -8,12 +8,13 @@
 #   dist/hammerhead_<version>_<os>_<arch>.tar.gz or .zip
 #   dist/SHA256SUMS
 #
-# Set TARGETS="linux/amd64 linux/arm64" to build fewer platforms.
+# Releases are published for linux/amd64 and windows/amd64 only. Set TARGETS="linux/arm64 darwin/arm64"
+# (any GOOS/GOARCH pair Go supports) to build something else by hand.
 set -eu
 NAME=hammerhead
 cd "$(dirname "$0")/.."
 VERSION=${1:-$(git describe --tags --always 2>/dev/null || echo dev)}
-TARGETS=${TARGETS:-"linux/amd64 linux/arm64 linux/arm windows/amd64 darwin/amd64 darwin/arm64"}
+TARGETS=${TARGETS:-"linux/amd64 windows/amd64"}
 
 rm -rf dist
 mkdir -p dist/bin dist/stage

deploy/setup.sh

@@ -51,9 +51,7 @@ command -v systemctl > /dev/null 2>&1 || die "this needs a systemd-based Linux (
 
 case "$(uname -m)" in
 	x86_64 | amd64) ARCH=amd64 ;;
-	aarch64 | arm64) ARCH=arm64 ;;
+	*) die "releases are published for x86-64 Linux only, not $(uname -m); build it yourself with: TARGETS=linux/arm64 sh deploy/build.sh" ;;
-	armv7* | armv6* | arm) ARCH=arm ;;
-	*) die "unsupported CPU: $(uname -m)" ;;
 esac
 
 pkg_install() { # installs packages with whichever package manager there is

e2e/e2e_test.go

@@ -241,6 +241,9 @@ func selfSigned(t *testing.T, host string) *tls.Certificate {
 
 // ---------- the environment ----------
 
+// siteCfgHook lets a test adjust the Bamboo that setup starts.
+var siteCfgHook func(*bamboo.Config)
+
 type world struct {
 	t       *testing.T
 	ts      *httptest.Server
@@ -297,7 +300,11 @@ func setup(t *testing.T, tr blacktip.TraefikConfig, opts ...func(*blacktip.Confi
 		t.Fatal(err)
 	}
 	go bt.Run(ctx)
-	bb, err := bamboo.New(bamboo.Config{Hammerhead: w.ts.URL, Token: site.Token, DataDir: t.TempDir()})
+	bbCfg := bamboo.Config{Hammerhead: w.ts.URL, Token: site.Token, DataDir: t.TempDir()}
+	if siteCfgHook != nil {
+		siteCfgHook(&bbCfg)
+	}
+	bb, err := bamboo.New(bbCfg)
 	if err != nil {
 		t.Fatal(err)
 	}

e2e/updates_test.go

@@ -0,0 +1,132 @@
+package e2e
+
+import (
+	"context"
+	"strings"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"git.veillax.com/Veillax/bamboo"
+	"git.veillax.com/Veillax/blacktip"
+	"git.veillax.com/Veillax/reef/model"
+)
+
+type edgeRow struct {
+	ID, Version, UpdateMsg string
+	AutoUpdate             bool
+	UpdatePending          bool
+	Online                 bool
+}
+
+func (w *world) edgeRow() edgeRow {
+	var edges []edgeRow
+	w.admin.must("GET", "/api/edges", nil, &edges)
+	for _, e := range edges {
+		if e.ID == w.edge {
+			return e
+		}
+	}
+	w.t.Fatal("edge not listed")
+	return edgeRow{}
+}
+
+func (w *world) siteRow() model.Site {
+	var st struct{ Sites []model.Site }
+	w.admin.must("GET", "/api/orgs/"+w.org+"/state", nil, &st)
+	for _, s := range st.Sites {
+		if s.ID == w.site {
+			return s
+		}
+	}
+	w.t.Fatal("site not listed")
+	return model.Site{}
+}
+
+func waitFor(t *testing.T, what string, ok func() bool) {
+	t.Helper()
+	for i := 0; i < 150; i++ {
+		if ok() {
+			return
+		}
+		time.Sleep(100 * time.Millisecond)
+	}
+	t.Fatalf("timed out waiting for %s", what)
+}
+
+func TestEdgesAndSitesReportVersionsAndUpdateOnRequest(t *testing.T) {
+	var edgeCalls, siteCalls atomic.Int32
+	siteCfgHook = func(c *bamboo.Config) {
+		c.Version, c.AutoUpdate = "v0.1.0", true
+		c.UpdateNow = func(ctx context.Context, progress func(string, int, string)) (string, error) {
+			siteCalls.Add(1)
+			progress("checking", 10, "Checking for the newest release")
+			progress("done", 100, "Already on the newest release")
+			return "", nil
+		}
+	}
+	defer func() { siteCfgHook = nil }()
+	w := setup(t, blacktip.TraefikConfig{}, func(c *blacktip.Config) {
+		c.Version, c.AutoUpdate = "v0.1.0", true
+		c.UpdateNow = func(ctx context.Context, progress func(string, int, string)) (string, error) {
+			edgeCalls.Add(1)
+			progress("downloading", 40, "Downloading blacktip_v0.2.0_linux_amd64.tar.gz")
+			progress("downloading", 70, "Downloaded 28.0 of 40.0 MB (70%)")
+			progress("restarting", 100, "Installed v0.2.0; restarting into it")
+			return "v0.2.0", nil
+		}
+	})
+
+	waitFor(t, "the edge to report its version", func() bool { e := w.edgeRow(); return e.Version == "v0.1.0" && e.AutoUpdate })
+	waitFor(t, "the site to report its version", func() bool { s := w.siteRow(); return s.Version == "v0.1.0" && s.AutoUpdate })
+
+	w.admin.must("POST", "/api/edges/"+w.edge+"/update", nil, nil)
+	waitFor(t, "the edge to answer the update request", func() bool {
+		e := w.edgeRow()
+		return !e.UpdatePending && e.UpdateMsg == "updated to v0.2.0, restarting"
+	})
+	if edgeCalls.Load() != 1 {
+		t.Fatalf("the edge ran its update %d times, want 1", edgeCalls.Load())
+	}
+	var log struct {
+		Running bool
+		Stage   string
+		Pct     int
+		Lines   []string
+	}
+	w.admin.must("GET", "/api/edges/"+w.edge+"/update/log", nil, &log)
+	text := strings.Join(log.Lines, "; ")
+	if log.Running || log.Stage != "restarting" || log.Pct != 100 || !strings.Contains(text, "Update requested") || !strings.Contains(text, "Downloaded 28.0 of 40.0 MB") || !strings.Contains(text, "restarting into it") {
+		t.Fatalf("edge update log: %+v (%s)", log, text)
+	}
+
+	w.admin.must("POST", "/api/orgs/"+w.org+"/sites/"+w.site+"/update", nil, nil)
+	waitFor(t, "the site to answer the update request", func() bool {
+		s := w.siteRow()
+		return s.UpdateID == "" && s.UpdateMsg == "already on the newest release"
+	})
+	if siteCalls.Load() != 1 {
+		t.Fatalf("the site ran its update %d times, want 1", siteCalls.Load())
+	}
+	w.admin.must("GET", "/api/orgs/"+w.org+"/sites/"+w.site+"/update/log", nil, &log)
+	if log.Running || log.Stage != "done" || !strings.Contains(strings.Join(log.Lines, "; "), "Already on the newest release") {
+		t.Fatalf("site update log: %+v", log)
+	}
+
+	// asking again works and is a fresh request, not a leftover answer
+	w.admin.must("POST", "/api/edges/"+w.edge+"/update", nil, nil)
+	waitFor(t, "a second update request to be answered", func() bool { return edgeCalls.Load() == 2 && !w.edgeRow().UpdatePending })
+}
+
+func TestUpdatesNeedAutoUpdateOnTheMachine(t *testing.T) {
+	w := setup(t, blacktip.TraefikConfig{}) // no version, auto-update off
+	waitFor(t, "the edge to be online", func() bool { return w.edgeRow().Online })
+	code, body := w.admin.do("POST", "/api/edges/"+w.edge+"/update", nil, nil)
+	if code != 400 || !strings.Contains(body, "BLACKTIP_AUTO_UPDATE") {
+		t.Fatalf("update on an edge without auto-update: %d %s", code, body)
+	}
+	code, body = w.admin.do("POST", "/api/orgs/"+w.org+"/sites/"+w.local+"/update", nil, nil)
+	if code != 400 {
+		t.Fatalf("update on a local site: %d %s", code, body)
+	}
+}

go.mod

@@ -3,9 +3,9 @@ module git.veillax.com/Veillax/hammerhead
 go 1.27.3
 
 require (
-	git.veillax.com/Veillax/bamboo v0.1.3
+	git.veillax.com/Veillax/bamboo v0.1.5
-	git.veillax.com/Veillax/blacktip v0.1.4
+	git.veillax.com/Veillax/blacktip v0.1.7
-	git.veillax.com/Veillax/reef v0.1.3
+	git.veillax.com/Veillax/reef v0.1.7
 	github.com/emersion/go-msgauth v0.7.0
 	golang.org/x/crypto v0.58.0
 	modernc.org/sqlite v1.61.0

go.sum

@@ -2,14 +2,28 @@ git.veillax.com/Veillax/bamboo v0.1.0 h1:16BG1P8982dIkHA2l0LPS0xYQc8d86PruVAGxc8
 git.veillax.com/Veillax/bamboo v0.1.0/go.mod h1:i4l4uJgG6Ymfwbjnlz0xeTt3qW/+1W1L2JBToMw50n0=
 git.veillax.com/Veillax/bamboo v0.1.3 h1:A66gz9GmmtUq1iIGUAL2ItIeC6iTy5pvqJOoIjRHjFg=
 git.veillax.com/Veillax/bamboo v0.1.3/go.mod h1:UuHV6aNHmV9jpow0xeGlXX0QhMB9glPJUGzgIM7S1MM=
+git.veillax.com/Veillax/bamboo v0.1.4 h1:pgiBJA46CBJx7T3bloCjPekb6wl6W5H8teokFAH5gj0=
+git.veillax.com/Veillax/bamboo v0.1.4/go.mod h1:wzSpfql7U4a3b9mCVH01OPG9H9szrzwSFDufWw02e0g=
+git.veillax.com/Veillax/bamboo v0.1.5 h1:i81c75/7wjayP9tOQ3tSnJWTBhvbjAwNBEPhT+3P2Xk=
+git.veillax.com/Veillax/bamboo v0.1.5/go.mod h1:se4kkcP6c3uCGD4HQd+nDLoblS0lN8qa+eFWLo5Horw=
 git.veillax.com/Veillax/blacktip v0.1.0 h1:L8WpW56gqtvZ1IAapE95Ynl6NKjqTA98WwmEH01kFFk=
 git.veillax.com/Veillax/blacktip v0.1.0/go.mod h1:sj+mr8393rajzWbl0+RRZrstGyDtZ/GHsut6ebScVxg=
 git.veillax.com/Veillax/blacktip v0.1.4 h1:o3uNoA0ucB/qcN9ccofysdFA6RTEU4jKdgeTWuE2Uw0=
 git.veillax.com/Veillax/blacktip v0.1.4/go.mod h1:f7zXCYlQf/9ffDLjPrMHylKAyXX4Nb+aQ7kErmOOhqI=
+git.veillax.com/Veillax/blacktip v0.1.6 h1:dUJzNGXgieWX05ftWjnMfh6norhaeKTqo1P13lJvjtk=
+git.veillax.com/Veillax/blacktip v0.1.6/go.mod h1:ZnPRc2gPh9ElILfPH39G/x3B6KAiHQyeoDLycSDgVPM=
+git.veillax.com/Veillax/blacktip v0.1.7 h1:PjxEkhE4fnGWht9Pj2/l6S8fOX2Iqjir3rc9vfpMFiQ=
+git.veillax.com/Veillax/blacktip v0.1.7/go.mod h1:dVm4wC6wufPjFzK8pvHnqNJJlxroRYmnq8mdkCqOKGc=
 git.veillax.com/Veillax/reef v0.1.0 h1:O0OhEKFx2CsvACJ20sfzJNjGsngomtfDdKdxu4ROpCs=
 git.veillax.com/Veillax/reef v0.1.0/go.mod h1:ZBUd/BARO5kkpUNTP81TTY1aQXLHFR6bKDhCxckjHso=
 git.veillax.com/Veillax/reef v0.1.3 h1:RxSq2xwaup1+MieAkhlvOTZnYwPf19thFHEAe/7VxsI=
 git.veillax.com/Veillax/reef v0.1.3/go.mod h1:ZBUd/BARO5kkpUNTP81TTY1aQXLHFR6bKDhCxckjHso=
+git.veillax.com/Veillax/reef v0.1.4 h1:T2+B8P1GAitEyTYRkwMT3UHbG9mN5zXb/inE2muXdQ4=
+git.veillax.com/Veillax/reef v0.1.4/go.mod h1:ZBUd/BARO5kkpUNTP81TTY1aQXLHFR6bKDhCxckjHso=
+git.veillax.com/Veillax/reef v0.1.6 h1:NgZ4u2Z3hJ+BYD/zX9lov8fox4s2i8txtYNOpIsikVo=
+git.veillax.com/Veillax/reef v0.1.6/go.mod h1:ZBUd/BARO5kkpUNTP81TTY1aQXLHFR6bKDhCxckjHso=
+git.veillax.com/Veillax/reef v0.1.7 h1:5C2udonlJac2xvDOE5e+zDxej+AbZ9ttrNSE4owmwuU=
+git.veillax.com/Veillax/reef v0.1.7/go.mod h1:ZBUd/BARO5kkpUNTP81TTY1aQXLHFR6bKDhCxckjHso=
 github.com/Microsoft/go-winio v0.6.4 h1:kJF2pamtYCWXf0xAT8sAkwKC0pMl2HYBrRPHAqyUc+g=
 github.com/Microsoft/go-winio v0.6.4/go.mod h1:1xl5JXG4ze/m74GH6Dc4Mwxfc3VL5siRVQjIAlHH07U=
 github.com/dustin/go-humanize v1.1.0 h1:dbKTrvD0klcbBV/h4AWJdMuZogJACoMlvWIWZ5b2xWg=

infra.go

@@ -24,6 +24,11 @@ type edgeView struct {
 	TunnelAddr string   `json:"tunnelAddr,omitempty"`
 	Token      string   `json:"token,omitempty"`
 	OrgIDs     []string `json:"orgIds,omitempty"`
+
+	Version       string `json:"version,omitempty"`
+	AutoUpdate    bool   `json:"autoUpdate,omitempty"`
+	UpdatePending bool   `json:"updatePending,omitempty"`
+	UpdateMsg     string `json:"updateMsg,omitempty"`
 }
 
 func (s *Server) viewEdge(e *model.Edge, admin bool) edgeView {
@@ -33,6 +38,7 @@ func (s *Server) viewEdge(e *model.Edge, admin bool) edgeView {
 	}
 	if admin {
 		v.TunnelAddr, v.Token, v.OrgIDs = e.TunnelAddr, e.Token, e.OrgIDs
+		v.Version, v.AutoUpdate, v.UpdatePending, v.UpdateMsg = e.Version, e.AutoUpdate, e.UpdateID != "", e.UpdateMsg
 		if v.OrgIDs == nil {
 			v.OrgIDs = []string{}
 		}

selfedge_test.go

@@ -96,3 +96,26 @@ func TestNoSelfEdgeUnlessAsked(t *testing.T) {
 		t.Fatal("token file written without HAMMERHEAD_SELF_DOMAIN")
 	}
 }
+
+func TestSavingSettingsKeepsTheSelfEdge(t *testing.T) {
+	dir := t.TempDir()
+	opts := Options{AdminEmail: "admin@test.local", AdminPassword: "password123", MasterKey: key, SelfDomain: "hh.example.com", SelfTarget: "127.0.0.1:8080"}
+	s, err := New(dir, opts)
+	if err != nil {
+		t.Fatal(err)
+	}
+	token := tokenFile(t, dir)
+	do := apiClient(t, s)
+	if code, b := do("PUT", "/api/admin/settings", map[string]any{"dashboardUrl": "https://hh.example.com", "acmeEmail": "me@example.com"}); code != 200 {
+		t.Fatalf("settings: %d %s", code, b)
+	}
+	s.Close()
+	s2, err := New(dir, opts)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s2.Close()
+	if len(s2.st.Edges) != 1 || tokenFile(t, dir) != token {
+		t.Fatalf("saving settings lost the self edge: %d edges", len(s2.st.Edges))
+	}
+}

server.go

@@ -97,6 +97,14 @@ func (s *Server) Handler() http.Handler {
 	mux.HandleFunc("GET /api/admin/settings", s.withServerAdmin(s.getSettings))
 	mux.HandleFunc("PUT /api/admin/settings", s.withServerAdmin(s.putSettings))
 	mux.HandleFunc("POST /api/admin/settings/test-mail", s.withServerAdmin(s.testMail))
+	mux.HandleFunc("GET /api/admin/update", s.withServerAdmin(s.checkUpdate))
+	mux.HandleFunc("POST /api/admin/update/apply", s.withServerAdmin(s.applyUpdate))
+	mux.HandleFunc("GET /api/admin/update/status", s.withServerAdmin(s.updateStatus))
+	mux.HandleFunc("GET /api/versions", s.withUser(s.latestVersions))
+	mux.HandleFunc("GET /api/edges/{id}/update/log", s.withServerAdmin(s.edgeUpdateLog))
+	mux.HandleFunc("GET /api/orgs/{org}/sites/{id}/update/log", s.withOrg(true, s.siteUpdateLog))
+	mux.HandleFunc("POST /api/edges/{id}/update", s.withServerAdmin(s.updateEdge))
+	mux.HandleFunc("POST /api/orgs/{org}/sites/{id}/update", s.withOrg(true, s.updateSite))
 	mux.HandleFunc("GET /api/admin/audit", s.withServerAdmin(func(w http.ResponseWriter, r *http.Request, p *principal) {
 		writeJSON(w, 200, s.logs.queryAudit("", 300, 0))
 	}))

settings.go

@@ -41,6 +41,7 @@ func (s *Server) putSettings(w http.ResponseWriter, r *http.Request, p *principa
 	if s.opts.DashboardURL != "" {
 		in.DashboardURL = s.st.Settings.DashboardURL // pinned by flag/env
 	}
+	in.SelfEdgeID = s.st.Settings.SelfEdgeID // not editable here; losing it would make the next start create a second self edge
 	s.st.Settings = in
 	s.bump() // edges receive the dashboard URL and ACME settings
 	s.mu.Unlock()

store.go

@@ -180,6 +180,14 @@ type Options struct {
 	SelfDomain string
 	SelfTarget string
 	SelfTunnel string
+
+	// Version is the running release (e.g. v0.1.5), shown under Settings with a check for updates.
+	Version string
+	// AutoUpdate says whether this process updates itself (HAMMERHEAD_AUTO_UPDATE), for the same screen.
+	AutoUpdate bool
+	// UpdateServer and UpdateRepo say where releases are published; the defaults are this project's own.
+	UpdateServer string
+	UpdateRepo   string
 }
 
 type Server struct {
@@ -200,8 +208,13 @@ type Server struct {
 	limits  map[string][]time.Time
 	oidc    map[string]*oidcDoc
 
-	alerts  map[string]*alertState // active problems by key (see alerts.go)
+	alerts   map[string]*alertState // active problems by key (see alerts.go)
-	mtaSent map[string]time.Time   // reminder throttle for repeat notifications
+	updRuns  map[string]*updRun     // progress of updates running on edges and sites, by ID
+	selfUpd  selfUpdate             // progress of this server's own update
+	verMu    sync.Mutex             // guards verCache and verAt, the latest releases seen by /api/versions
+	verCache map[string]string
+	verAt    time.Time
+	mtaSent  map[string]time.Time // reminder throttle for repeat notifications
 
 	logs *logStore
 	db   *DB
@@ -242,6 +255,7 @@ func New(dir string, opts Options) (*Server, error) {
 		oidc:       map[string]*oidcDoc{},
 		alerts:     map[string]*alertState{},
 		mtaSent:    map[string]time.Time{},
+		updRuns:    map[string]*updRun{},
 	}
 	s.st.Sessions = map[string]*Session{}
 	s.st.Secrets = map[string]*Secret{}
@@ -271,6 +285,7 @@ func New(dir string, opts Options) (*Server, error) {
 	if opts.SelfDomain != "" && opts.SelfTunnel == "" {
 		s.opts.SelfTunnel = opts.SelfDomain + ":51820"
 	}
+	s.finishSelfUpdateLog()
 	if err := s.ensureSelfEdge(); err != nil {
 		db.Close()
 		return nil, err

update.go

@@ -0,0 +1,183 @@
+package hammerhead
+
+import (
+	"context"
+	"errors"
+	"net/http"
+	"os"
+	"strings"
+	"sync"
+	"time"
+
+	"git.veillax.com/Veillax/reef/selfupdate"
+)
+
+func (s *Server) updateConfig(app, current string) selfupdate.Config {
+	o := s.opts
+	server := o.UpdateServer
+	if server == "" {
+		server = "https://git.veillax.com"
+	}
+	owner := "Veillax"
+	if o.UpdateRepo != "" {
+		owner = strings.SplitN(o.UpdateRepo, "/", 2)[0]
+	}
+	return selfupdate.Config{Server: server, Repo: owner + "/" + app, App: app, Current: current}
+}
+
+// checkUpdate asks the release server whether a newer Hammerhead exists. It only looks.
+func (s *Server) checkUpdate(w http.ResponseWriter, r *http.Request, p *principal) {
+	o := s.opts
+	ctx, cancel := context.WithTimeout(r.Context(), 20*time.Second)
+	defer cancel()
+	out := map[string]any{"current": o.Version, "autoUpdate": o.AutoUpdate, "checked": time.Now()}
+	latest, newer, err := selfupdate.Check(ctx, s.updateConfig("hammerhead", o.Version))
+	out["latest"], out["newer"] = latest, newer
+	if err != nil {
+		out["error"] = err.Error()
+	}
+	writeJSON(w, 200, out)
+}
+
+// applyUpdate installs the newest release of Hammerhead itself and restarts into it. Like the
+// scheduled check it needs auto-update to be on, which is also what makes the binary replaceable.
+func (s *Server) applyUpdate(w http.ResponseWriter, r *http.Request, p *principal) {
+	if !s.opts.AutoUpdate {
+		fail(w, 400, errors.New("auto-update is off: set HAMMERHEAD_AUTO_UPDATE=1 and restart Hammerhead, or update it by hand"))
+		return
+	}
+	u := &s.selfUpd
+	u.mu.Lock()
+	if u.running {
+		u.mu.Unlock()
+		fail(w, 409, errors.New("an update is already running"))
+		return
+	}
+	u.running, u.stage, u.pct, u.lines = true, "starting", 0, nil
+	u.mu.Unlock()
+	os.WriteFile(s.updateLogPath(), nil, 0o600) // a new run starts a new log
+	cfg := s.updateConfig("hammerhead", s.opts.Version)
+	cfg.Progress = func(e selfupdate.Event) { s.selfNote(e.Stage, e.Pct, e.Line) }
+	cfg.Restart = func(exe string) error {
+		s.selfNote("restarting", 100, "Closing the database and restarting")
+		s.Close()
+		return selfupdate.Restart(exe)
+	}
+	s.selfNote("starting", 0, "Update requested by "+p.User.Email)
+	s.audit("", p.User.Email, "server.update", "")
+	go func() {
+		ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
+		defer cancel()
+		selfupdate.CheckOnce(ctx, cfg) // on success the process is replaced and this never returns
+		u.mu.Lock()
+		u.running = false
+		u.mu.Unlock()
+	}()
+	writeJSON(w, 200, map[string]any{"started": true})
+}
+
+// latestVersions returns the newest release of each program, for "update available" badges. Results
+// are cached for ten minutes so a busy dashboard doesn't hammer the release server.
+func (s *Server) latestVersions(w http.ResponseWriter, r *http.Request, p *principal) {
+	s.verMu.Lock()
+	if time.Since(s.verAt) > 10*time.Minute || s.verCache == nil {
+		ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
+		out := map[string]string{}
+		var mu sync.Mutex
+		var wg sync.WaitGroup
+		for _, app := range []string{"hammerhead", "blacktip", "bamboo"} {
+			wg.Add(1)
+			go func(app string) {
+				defer wg.Done()
+				if tag, _, err := selfupdate.Check(ctx, s.updateConfig(app, "")); err == nil && tag != "" {
+					mu.Lock()
+					out[app] = tag
+					mu.Unlock()
+				}
+			}(app)
+		}
+		wg.Wait()
+		cancel()
+		s.verCache, s.verAt = out, time.Now()
+	}
+	out := s.verCache
+	s.verMu.Unlock()
+	writeJSON(w, 200, out)
+}
+
+// noteUpdateReport records what an edge or site says about its version and any update that was
+// requested of it. It reports whether a pending request was answered, which means the config the
+// agent polls has changed (it no longer carries the request).
+func noteUpdateReport(version *string, auto *bool, updateID, updateMsg *string, inVersion string, inAuto bool, inID, inMsg string) bool {
+	upgraded := *version != "" && inVersion != "" && inVersion != *version
+	if inVersion != "" {
+		*version = inVersion
+	}
+	*auto = inAuto
+	if *updateID == "" {
+		return false
+	}
+	switch {
+	case inID == *updateID && inMsg != "":
+		*updateMsg = inMsg
+	case upgraded:
+		*updateMsg = "updated to " + inVersion
+	default:
+		return false
+	}
+	*updateID = ""
+	return true
+}
+
+func (s *Server) updateEdge(w http.ResponseWriter, r *http.Request, p *principal) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	e := s.findEdge(r.PathValue("id"))
+	if e == nil {
+		fail(w, 404, errors.New("no such edge"))
+		return
+	}
+	if err := requestUpdate(&e.AutoUpdate, &e.UpdateID, &e.UpdateMsg, s.edgeOnline(e), "BLACKTIP"); err != nil {
+		fail(w, 400, err)
+		return
+	}
+	delete(s.updRuns, e.ID) // a new request starts a new log
+	s.bump()
+	s.audit("", p.User.Email, "edge.update", e.Name)
+	ok(w)
+}
+
+func (s *Server) updateSite(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	st := s.findSite(r.PathValue("id"))
+	if st == nil || !s.siteVisible(c, st) {
+		fail(w, 404, errors.New("no such site"))
+		return
+	}
+	if st.Local {
+		fail(w, 400, errors.New("this site is the edge's own machine; update the edge instead"))
+		return
+	}
+	if err := requestUpdate(&st.AutoUpdate, &st.UpdateID, &st.UpdateMsg, s.siteOnline(st), "BAMBOO"); err != nil {
+		fail(w, 400, err)
+		return
+	}
+	delete(s.updRuns, st.ID)
+	s.bump()
+	s.audit(c.Org.ID, c.Actor, "site.update", st.Name)
+	ok(w)
+}
+
+// requestUpdate marks an update as wanted. The agent installs it only if it has auto-update on, and
+// the request stays until the agent answers (or reports a new version). Asking again replaces it.
+func requestUpdate(auto *bool, updateID, updateMsg *string, online bool, prefix string) error {
+	if !online {
+		return errors.New("it is offline right now")
+	}
+	if !*auto {
+		return errors.New("it isn't running with auto-update on, so it won't install updates from here (set " + prefix + "_AUTO_UPDATE=1 and restart it)")
+	}
+	*updateID, *updateMsg = randHex(8), ""
+	return nil
+}

update_test.go

@@ -0,0 +1,90 @@
+package hammerhead
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func TestCheckForUpdates(t *testing.T) {
+	gitea := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path != "/api/v1/repos/Veillax/hammerhead/releases/latest" {
+			http.NotFound(w, r)
+			return
+		}
+		fmt.Fprint(w, `{"tag_name":"v0.3.0","prerelease":false,"assets":[]}`)
+	}))
+	defer gitea.Close()
+	for _, c := range []struct {
+		version string
+		newer   bool
+	}{{"v0.2.0", true}, {"v0.3.0", false}, {"dev", false}} {
+		s, err := New(t.TempDir(), Options{AdminEmail: "admin@test.local", AdminPassword: "password123", MasterKey: key,
+			Version: c.version, UpdateServer: gitea.URL, UpdateRepo: "Veillax/hammerhead", AutoUpdate: true})
+		if err != nil {
+			t.Fatal(err)
+		}
+		do := apiClient(t, s)
+		code, b := do("GET", "/api/admin/update", nil)
+		var out struct {
+			Current, Latest, Error string
+			Newer, AutoUpdate      bool
+		}
+		json.Unmarshal(b, &out)
+		if code != 200 || out.Current != c.version || out.Latest != "v0.3.0" || out.Newer != c.newer || !out.AutoUpdate || out.Error != "" {
+			t.Errorf("%s: %d %s", c.version, code, b)
+		}
+		s.Close()
+	}
+
+	// an unreachable release server is reported, not fatal
+	s, err := New(t.TempDir(), Options{AdminEmail: "admin@test.local", AdminPassword: "password123", MasterKey: key, Version: "v0.1.0", UpdateServer: "http://127.0.0.1:1"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	code, b := apiClient(t, s)("GET", "/api/admin/update", nil)
+	var out struct{ Error string }
+	json.Unmarshal(b, &out)
+	if code != 200 || out.Error == "" {
+		t.Errorf("unreachable server: %d %s", code, b)
+	}
+}
+
+func TestSelfUpdateLogSurvivesTheRestartItEndsWith(t *testing.T) {
+	dir := t.TempDir()
+	if err := os.WriteFile(filepath.Join(dir, "update.log"), []byte("12:00:00 Checking\n12:00:09 Installed v0.3.0; restarting into it\n"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	s, err := New(dir, Options{AdminEmail: "admin@test.local", AdminPassword: "password123", MasterKey: key, Version: "v0.3.0"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer s.Close()
+	do := apiClient(t, s)
+	code, b := do("GET", "/api/admin/update/status", nil)
+	var out struct {
+		Running bool
+		Stage   string
+		Pct     int
+		Lines   []string
+		Version string
+	}
+	json.Unmarshal(b, &out)
+	last := ""
+	if len(out.Lines) > 0 {
+		last = out.Lines[len(out.Lines)-1]
+	}
+	if code != 200 || out.Running || out.Stage != "done" || out.Pct != 100 || len(out.Lines) != 3 || !strings.Contains(last, "now running v0.3.0") {
+		t.Fatalf("status: %d %s", code, b)
+	}
+	// starting an update needs auto-update on
+	if code, b := do("POST", "/api/admin/update/apply", nil); code != 400 || !strings.Contains(string(b), "HAMMERHEAD_AUTO_UPDATE") {
+		t.Fatalf("apply without auto-update: %d %s", code, b)
+	}
+}

updatelog.go

@@ -0,0 +1,152 @@
+package hammerhead
+
+import (
+	"errors"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"time"
+)
+
+const maxUpdateLines = 200
+
+// updRun is the progress of one update an edge or site is carrying out, as that agent reported it.
+type updRun struct {
+	ID    string // the update request it belongs to
+	Stage string
+	Pct   int
+	Lines []string
+}
+
+// recordUpdateProgress keeps what an agent says about the update it is running. When the agent
+// reports a new version, the update is finished: its log gets a closing line and is then left alone,
+// so the freshly restarted program (which starts a new, empty log) can't overwrite it.
+// Caller holds s.mu.
+func (s *Server) recordUpdateProgress(agentID, reqID, stage string, pct int, lines []string, oldVersion, newVersion string) {
+	run := s.updRuns[agentID]
+	if run != nil && run.Stage != "done" && oldVersion != "" && newVersion != "" && newVersion != oldVersion {
+		run.Lines = append(run.Lines, time.Now().Format("15:04:05")+" Now running "+newVersion)
+		run.Stage, run.Pct = "done", 100
+		return
+	}
+	if reqID == "" || (stage == "" && len(lines) == 0) {
+		return
+	}
+	if run != nil && run.ID == reqID && run.Stage == "done" {
+		return
+	}
+	s.updRuns[agentID] = &updRun{ID: reqID, Stage: stage, Pct: pct, Lines: lines}
+}
+
+func (s *Server) updateLogView(agentID string, pending bool, msg, version string) map[string]any {
+	out := map[string]any{"running": pending, "msg": msg, "version": version, "stage": "", "pct": 0, "lines": []string{}}
+	if run := s.updRuns[agentID]; run != nil {
+		out["stage"], out["pct"], out["lines"] = run.Stage, run.Pct, run.Lines
+	}
+	return out
+}
+
+func (s *Server) edgeUpdateLog(w http.ResponseWriter, r *http.Request, p *principal) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	e := s.findEdge(r.PathValue("id"))
+	if e == nil {
+		fail(w, 404, errors.New("no such edge"))
+		return
+	}
+	writeJSON(w, 200, s.updateLogView(e.ID, e.UpdateID != "", e.UpdateMsg, e.Version))
+}
+
+func (s *Server) siteUpdateLog(w http.ResponseWriter, r *http.Request, c *orgCtx) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	st := s.findSite(r.PathValue("id"))
+	if st == nil || !s.siteVisible(c, st) {
+		fail(w, 404, errors.New("no such site"))
+		return
+	}
+	writeJSON(w, 200, s.updateLogView(st.ID, st.UpdateID != "", st.UpdateMsg, st.Version))
+}
+
+// selfUpdate is the progress of this server's own update. It is kept in memory and in <data>/update.log,
+// so the page can still show it after the restart that ends a successful update.
+type selfUpdate struct {
+	mu      sync.Mutex
+	running bool
+	stage   string
+	pct     int
+	lines   []string
+}
+
+func (s *Server) updateLogPath() string { return filepath.Join(s.dir, "update.log") }
+
+func (s *Server) selfNote(stage string, pct int, line string) {
+	u := &s.selfUpd
+	u.mu.Lock()
+	defer u.mu.Unlock()
+	if stage != "" {
+		u.stage = stage
+	}
+	if pct >= 0 {
+		u.pct = pct
+	}
+	if line == "" {
+		return
+	}
+	line = time.Now().Format("15:04:05") + " " + line
+	u.lines = append(u.lines, line)
+	if len(u.lines) > maxUpdateLines {
+		u.lines = u.lines[len(u.lines)-maxUpdateLines:]
+	}
+	if f, err := os.OpenFile(s.updateLogPath(), os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o600); err == nil {
+		f.WriteString(line + "\n")
+		f.Close()
+	}
+}
+
+func readLines(path string) []string {
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return nil
+	}
+	lines := strings.Split(strings.TrimRight(string(b), "\n"), "\n")
+	if len(lines) == 1 && lines[0] == "" {
+		return nil
+	}
+	if len(lines) > maxUpdateLines {
+		lines = lines[len(lines)-maxUpdateLines:]
+	}
+	return lines
+}
+
+// finishSelfUpdateLog runs at startup: if the last run ended by restarting into this process, say so.
+func (s *Server) finishSelfUpdateLog() {
+	lines := readLines(s.updateLogPath())
+	if n := len(lines); n > 0 && strings.Contains(lines[n-1], "restarting") && s.opts.Version != "" {
+		if f, err := os.OpenFile(s.updateLogPath(), os.O_APPEND|os.O_WRONLY, 0o600); err == nil {
+			f.WriteString(time.Now().Format("15:04:05") + " Hammerhead restarted: now running " + s.opts.Version + "\n")
+			f.Close()
+		}
+	}
+}
+
+// updateStatus returns the progress and log of this server's own update, for the Settings page.
+func (s *Server) updateStatus(w http.ResponseWriter, r *http.Request, p *principal) {
+	u := &s.selfUpd
+	u.mu.Lock()
+	out := map[string]any{"running": u.running, "stage": u.stage, "pct": u.pct, "lines": append([]string(nil), u.lines...), "version": s.opts.Version}
+	u.mu.Unlock()
+	if lines, _ := out["lines"].([]string); len(lines) == 0 {
+		// nothing in memory: this is a fresh process, so show the last update's log from disk
+		if saved := readLines(s.updateLogPath()); len(saved) > 0 {
+			stage := "done"
+			if strings.Contains(saved[len(saved)-1], "Failed:") {
+				stage = "failed"
+			}
+			out["lines"], out["stage"], out["pct"] = saved, stage, 100
+		}
+	}
+	writeJSON(w, 200, out)
+}

web/index.html

@@ -70,6 +70,16 @@
   .row-h h2 { display:none; }
   .grid { display:grid; grid-template-columns:repeat(auto-fit,minmax(170px,1fr)); gap:var(--s4); align-items:end; }
   .stack { display:flex; flex-direction:column; gap:var(--s3); }
+  .choices { display:grid; grid-template-columns:repeat(auto-fill,minmax(210px,1fr)); gap:var(--s3); }
+  .choice { text-align:left; background:var(--surface); border:1px solid var(--border); border-radius:var(--radius-lg); padding:var(--s3) var(--s4); color:var(--text); cursor:pointer; font:inherit; display:flex; flex-direction:column; gap:var(--s1); }
+  .choice:hover { border-color:var(--border-strong); background:var(--surface-2); }
+  .choice.on { border-color:var(--accent); background:var(--accent-soft); }
+  .choice-t { font-weight:600; }
+  .sect-h { padding:var(--s3) var(--s4); font-weight:600; border-bottom:1px solid var(--border); }
+  .progress { height:8px; background:var(--surface-2); border-radius:999px; overflow:hidden; }
+  .progress > div { height:100%; background:var(--accent); transition:width .4s ease; }
+  .progress > div.bad { background:var(--danger); }
+  .logbox { margin:0; background:var(--bg); border:1px solid var(--border); border-radius:var(--radius); padding:var(--s3); font:var(--text-xs)/1.6 var(--font-mono); max-height:260px; overflow:auto; white-space:pre-wrap; word-break:break-word; }
 
   /* forms */
   label { display:flex; flex-direction:column; gap:var(--s1); font-size:var(--text-sm); font-weight:600; color:var(--text); }
@@ -184,6 +194,60 @@ const ago = d => { if (!d || d.startsWith('0001')) return 'never'; const s = (Da
 const byId = (list, id) => (list || []).find(x => x.id === id);
 const field = (label, el, hint) => h('label', {}, label, el, hint ? h('span', {class:'sub'}, hint) : null);
 const chk = (label, checked, onchange) => { const i = h('input', {type:'checkbox', checked: !!checked}); if (onchange) i.onchange = () => onchange(i.checked); return Object.assign(h('label', {class:'chk'}, i, label), {box:i}); };
+// ----- versions and updates -----
+// A progress bar and live log for an update. load() returns {running, stage, pct, lines}; the panel polls it
+// until the update is over, and keeps going through the gap while a program restarts into its new version.
+function updatePanel(load, onDone) {
+  const bar = h('div', {style:'width:0%'}), stageEl = h('div', {class:'small'}, 'Starting…'), logEl = h('pre', {class:'logbox'}, 'Waiting for the first message…');
+  const el = h('div', {class:'stack'}, stageEl, h('div', {class:'progress'}, bar), logEl);
+  const label = {starting:'Starting', checking:'Checking for the release', downloading:'Downloading', verifying:'Verifying', installing:'Installing', restarting:'Restarting', done:'Done', failed:'Failed'};
+  let misses = 0;
+  const tick = async () => {
+    if (!el.isConnected) return; // the dialog or card was closed
+    let again = true;
+    try {
+      const s = await load(); misses = 0;
+      const pct = Math.max(0, Math.min(100, s.pct || 0)), lines = s.lines || [];
+      bar.style.width = pct + '%'; bar.className = s.stage === 'failed' ? 'bad' : '';
+      stageEl.textContent = (label[s.stage] || s.stage || 'Waiting') + (pct ? ' · ' + pct + '%' : '');
+      if (lines.length) { const atEnd = logEl.scrollTop + logEl.clientHeight >= logEl.scrollHeight - 4; logEl.textContent = lines.join('\n'); if (atEnd) logEl.scrollTop = logEl.scrollHeight; }
+      if (!s.running && (s.stage === 'done' || s.stage === 'failed')) { again = false; if (onDone) onDone(s); }
+    } catch (e) {
+      misses++; // the server is probably restarting into the new version
+      if (misses > 2) stageEl.textContent = 'Restarting… waiting for it to come back';
+      if (misses > 150) { again = false; stageEl.textContent = 'It has not come back after a few minutes. Check its log on the server.'; }
+    }
+    if (again) setTimeout(tick, 800);
+  };
+  setTimeout(tick, 100);
+  return el;
+}
+
+// compares vX.Y.Z strings: -1 if a is older than b, 1 if newer, 0 if equal or not comparable
+const cmpVer = (a, b) => {
+  const p = v => (/^v(\d+)\.(\d+)\.(\d+)$/.exec(v || '') || []).slice(1).map(Number), x = p(a), y = p(b);
+  if (x.length < 3 || y.length < 3) return 0;
+  for (let i = 0; i < 3; i++) if (x[i] !== y[i]) return x[i] < y[i] ? -1 : 1;
+  return 0;
+};
+const latestVersions = async () => { try { return await api('GET', '/api/versions'); } catch (e) { return {}; } };
+// A table cell with a machine's version, a note when a newer release exists, and an Update button
+// when that machine allows being updated from here (it must run with auto-update on).
+const updateDialog = (m, logUrl) => modal('Updating ' + m.name, h('div', {class:'stack'}, updatePanel(() => api('GET', logUrl), () => refresh()), h('p', {class:'small dim'}, 'You can close this; the update carries on.')));
+const versionCell = (m, latest, pending, updateUrl, logUrl) => {
+  const behind = latest && m.version && cmpVer(m.version, latest) < 0, kids = [h('span', {class:'mono small'}, m.version || 'unknown')];
+  if (behind) kids.push(h('div', {class:'small'}, latest + ' is available'));
+  if (pending) {
+    kids.push(h('div', {class:'small dim'}, 'Updating…'));
+    clearTimeout(S.updTimer); S.updTimer = setTimeout(() => refresh(), 4000); // look again for the result
+  } else if (m.updateMsg) kids.push(h('div', {class:'small dim'}, m.updateMsg));
+  if (behind && !pending) kids.push(h('button', {class:'ghost', disabled:!m.autoUpdate,
+    title:m.autoUpdate ? 'Install ' + latest + ' now' : 'Turn on auto-update on that machine (…_AUTO_UPDATE=1) to update it from here',
+    onclick: act(async () => { await api('POST', updateUrl); updateDialog(m, logUrl); refresh(); })}, 'Update'));
+  if (pending || m.updateMsg) kids.push(h('button', {class:'ghost', onclick:() => updateDialog(m, logUrl)}, 'Log'));
+  return h('td', {}, h('div', {class:'stack'}, ...kids));
+};
+
 const card = (title, ...kids) => h('div', {class:'card'}, title ? h('h2', {}, title) : null, ...kids);
 function dialogForm(title, intro, fields, label, submit) {
   const close = modal(title, h('div', {class:'stack'}, intro ? h('p', {class:'dim'}, intro) : null, ...fields,
@@ -391,13 +455,15 @@ views.ports = async () => {
   const unsetDash = !st.dashboardUrl && st.mappings.some(m => m.auth && (m.auth.sso || m.auth.password || m.auth.pin || m.auth.email));
   const traffic = await api('GET', O('/traffic?hours=24')).catch(() => ({}));
   const rows = st.mappings.slice().sort((a, b) => a.edgeId.localeCompare(b.edgeId) || a.bindIp.localeCompare(b.bindIp) || a.port - b.port);
-  const table = rows.length ? h('div', {class:'scroll'}, h('table', {},
+  const isWebRow = m => m.proto === 'http' || m.proto === 'https' || m.proto === 'tls';
-    h('thead', {}, h('tr', {}, ['', 'Name', 'Public address', 'Targets', 'Traffic', ''].map(t => h('th', {}, t)))),
+  const section = (title, list) => list.length ? h('div', {class:'card flush'}, h('div', {class:'sect-h'}, title + ' (' + list.length + ')'), h('div', {class:'scroll'}, h('table', {},
-    h('tbody', {}, rows.map(m => mappingRow(m, st, traffic[m.id]))))) : h('div', {class:'empty'}, 'No ports open yet.');
+    h('thead', {}, h('tr', {}, ['', 'Name', title === 'Websites' ? 'Address' : 'Public address', 'Targets', 'Traffic', ''].map(t => h('th', {}, t)))),
-  S.headerAction = ['Open a port', () => portForm()];
+    h('tbody', {}, list.map(m => mappingRow(m, st, traffic[m.id])))))) : null;
+  const table = rows.length ? h('div', {}, section('Websites', rows.filter(isWebRow)), section('Ports', rows.filter(m => !isWebRow(m)))) : h('div', {class:'card'}, h('div', {class:'empty'}, 'Nothing here yet. Use Add resource to share a website or open a port.'));
+  S.headerAction = ['Add resource', () => portForm()];
   const q = h('input', {type:'search', placeholder:'Search resources…', oninput:() => { const t = q.value.toLowerCase(); table.querySelectorAll('tbody tr').forEach(tr => { tr.style.display = tr.textContent.toLowerCase().includes(t) ? '' : 'none'; }); }});
   return h('div', {}, unsetDash ? h('div', {class:'banner'}, 'Some ports have a login in front of them, but the dashboard URL isn\'t set. Visitors are sent back to this address\'s login page, so set it under Server → Settings if people reach Hammerhead by a different address.') : null,
-    rows.length > 5 ? h('div', {class:'toolbar'}, q) : null, h('div', {class:'card flush'}, table));
+    rows.length > 5 ? h('div', {class:'toolbar'}, q) : null, table);
 };
 
 function spark(series) {
@@ -426,13 +492,13 @@ function mappingRow(m, st, series) {
   }
   const web = m.proto === 'http' || m.proto === 'https';
   const prot = m.auth && (m.auth.sso || m.auth.password || m.auth.pin || m.auth.email);
-  const pubAddr = (m.hostname ? m.hostname + ' @ ' : '') + m.bindIp + ':' + portText(m);
+  const pubAddr = web ? m.proto + '://' + (m.hostname || '*') + (m.port === (m.proto === 'https' ? 443 : 80) ? '' : ':' + portText(m)) : (m.hostname ? m.hostname + ' @ ' : '') + m.bindIp + ':' + portText(m);
   return h('tr', {},
     h('td', {title:why}, h('span', {class:dot})),
     h('td', {}, h('div', {}, m.name, ' ', h('span', {class:'tag'}, m.proto), m.proxyProtocol ? h('span', {class:'tag', style:'margin-left:4px', title:'the target is told the real client address'}, 'proxy ' + m.proxyProtocol) : null, prot ? h('span', {class:'tag', style:'margin-left:4px', title:'login required'}, '🔒') : null),
       s.error ? h('div', {class:'err small mono'}, s.error) : null,
       s.cert && m.proto === 'https' ? h('div', {class:'small ' + (s.cert.startsWith('error') ? 'err' : 'dim')}, '🔐 ' + s.cert) : null),
-    h('td', {class:'mono'}, pubAddr, h('div', {class:'dim small'}, edge ? edge.name : '?')),
+    h('td', {class:'mono'}, pubAddr, h('div', {class:'dim small'}, (edge ? edge.name : '?') + (web && m.bindIp !== '0.0.0.0' ? ' · ' + m.bindIp : ''))),
     h('td', {class:'mono small'}, m.targets.map((t, i) => {
       const th = (s.targets || [])[i], site = byId(st.sites, t.siteId);
       return h('div', {title:th && !th.healthy ? th.detail : ''}, h('span', {class:'dot ' + (th ? (th.healthy ? 'ok' : 'bad') : '')}), (site ? site.name : '?') + ' → ' + t.host + ':' + t.port);
@@ -547,10 +613,109 @@ function portForm(existing) {
     else for (const p of protos) await api('POST', O('/mappings'), {...base, proto:p, targets:p === 'udp' ? targets.slice(0, 1) : targets, proxyProtocol:['tcp', 'tls'].includes(p) ? proxySel.value : ''});
     close(); toast('Saved'); refresh();
   });
-  const close = modal(existing ? 'Edit port' : 'Open a port', h('div', {class:'stack'},
+  const close = existing ? modal('Edit port', h('div', {class:'stack'},
     h('div', {class:'grid'}, field('Name', name), field('Edge', edgeSel), field('Public IP', ipIn), field('Protocol', protoSel), field('Public port', port, 'One port, or a range like 2456-2458 (tcp/udp): each port goes to the target port plus the same offset')),
     ipList, domList, hostBox, proxyBox, h('fieldset', {}, h('legend', {}, 'Where does the traffic go?'), tgtBox), adv, webBox,
-    h('div', {class:'actions'}, h('button', {class:'btn', onclick:save}, existing ? 'Save changes' : 'Open port'), h('button', {class:'ghost', onclick:() => close()}, 'Cancel'))), {sticky:true});
+    h('div', {class:'actions'}, h('button', {class:'btn', onclick:save}, existing ? 'Save changes' : 'Open port'), h('button', {class:'ghost', onclick:() => close()}, 'Cancel'))), {sticky:true}) : wizardModal();
+
+  // The add flow for a new resource: a few short steps that reuse the controls above.
+  function wizardModal() {
+    const TYPES = [
+      ['https', '🌐', 'Website', 'A site or web app, with an automatic HTTPS certificate.'],
+      ['tcp', '🎮', 'Game or app port (TCP)', 'Minecraft, SSH, databases: anything over TCP.'],
+      ['udp', '📡', 'UDP port', 'Voice chat, some games, DNS.'],
+      ['both', '🔁', 'TCP + UDP', 'The same port for both, as many games want.'],
+      ['tls', '🔒', 'TLS passthrough', 'Route encrypted traffic by hostname without decrypting it.'],
+      ['http', '🌍', 'Website, plain HTTP', 'No certificate: for internal use or behind another proxy.'],
+    ];
+    const [accessFs, rulesFs, advFs] = [...webBox.children];
+    let step = 0, chosen = false;
+    const isWeb = () => protoSel.value === 'http' || protoSel.value === 'https';
+    const flow = () => isWeb() ? ['type', 'target', 'reach', 'access', 'finish'] : ['type', 'target', 'reach', 'finish'];
+    const titles = {type:'What are you sharing?', target:'Where does it point?', reach:'How do people reach it?', access:'Who can open it?', finish:'Name it and review'};
+    const body = h('div', {class:'stack'}), crumbs = h('div', {class:'small dim'}), err = h('div', {class:'err small'});
+    const back = h('button', {class:'ghost', type:'button', onclick:() => go(-1)}, 'Back');
+    const next = h('button', {class:'btn', type:'button', onclick:() => go(1)}, 'Next');
+    const portRe = /^(\d+)(?:\s*-\s*(\d+))?$/;
+
+    const summary = () => {
+      const t = targets.map(x => ((byId(st.sites, x.siteId) || {name:'?'}).name) + ' → ' + x.host + ':' + x.port).join(', '), pr = (port.value || '').trim();
+      const protoName = {both:'TCP + UDP', tcp:'TCP', udp:'UDP', tls:'TLS'}[protoSel.value];
+      const where = isWeb() ? protoSel.value + '://' + (host.value.trim() || '*') + (pr && pr !== '443' && pr !== '80' ? ':' + pr : '')
+        : protoName + ' port ' + pr + (protoSel.value === 'tls' && host.value.trim() ? ' for ' + host.value.trim() : '') + ' on ' + (ipIn.value.trim() || 'every address');
+      const login = ssoC.box.checked || pwC.box.checked || pinC.box.checked || mailC.box.checked;
+      return h('div', {class:'stack'}, h('div', {}, h('b', {class:'mono'}, where)), h('div', {class:'small'}, '→ ' + t),
+        isWeb() ? h('div', {class:'small dim'}, login ? 'Visitors have to sign in.' : 'Anyone can open it.') : null);
+    };
+    const validate = cur => {
+      if (cur === 'type') return chosen ? '' : 'Choose what you are sharing.';
+      if (cur === 'target') return targets.every(t => t.siteId && String(t.host).trim() && t.port > 0) ? '' : 'Choose a site and enter the host and port for every target.';
+      if (cur === 'reach') {
+        if ((isWeb() || protoSel.value === 'tls') && !host.value.trim()) return 'Enter the hostname people will visit.';
+        return portRe.test((port.value || '').trim()) ? '' : 'Enter a public port like 25565, or a range like 2456-2458.';
+      }
+      if (cur === 'access') {
+        if (pwC.box.checked && !pwIn.value) return 'Enter the shared password, or turn that option off.';
+        if (pinC.box.checked && !pinIn.value) return 'Enter a PIN, or turn that option off.';
+      }
+      return '';
+    };
+    const prefill = () => {
+      if (port.value) return;
+      const t = protoSel.value;
+      port.value = t === 'https' || t === 'tls' ? '443' : t === 'http' ? '80' : (targets[0] && targets[0].port ? String(targets[0].port) : '');
+    };
+    const pick = v => {
+      protoSel.value = v; chosen = true; protoSel.onchange();
+      if (v === 'https') redir.box.checked = true;
+      port.value = ''; step = 1; draw();
+    };
+    const go = dir => {
+      const cur = flow()[step];
+      if (dir > 0) {
+        const msg = validate(cur);
+        if (msg) { err.textContent = msg; return; }
+        if (cur === 'finish') {
+          if (!name.value.trim()) name.value = host.value.trim() || (protoSel.value.toUpperCase() + ' ' + port.value.trim());
+          save(); return;
+        }
+        step++;
+      } else step = Math.max(0, step - 1);
+      draw();
+    };
+    function draw() {
+      const f = flow(), cur = f[step];
+      err.textContent = '';
+      crumbs.textContent = 'Step ' + (step + 1) + ' of ' + f.length;
+      next.textContent = cur === 'finish' ? (isWeb() ? 'Add website' : 'Open port') : 'Next';
+      next.style.display = cur === 'type' ? 'none' : '';
+      back.style.display = step ? '' : 'none';
+      const kids = [h('h3', {style:'margin:0'}, titles[cur])];
+      if (cur === 'type') {
+        kids.push(h('div', {class:'choices'}, TYPES.map(([v, icon, t, d]) => h('button', {type:'button', class:'choice' + (chosen && protoSel.value === v ? ' on' : ''), onclick:() => pick(v)},
+          h('div', {class:'choice-t'}, icon + ' ' + t), h('div', {class:'small dim'}, d)))));
+      } else if (cur === 'target') {
+        if (st.edges.length > 1) kids.push(field('Public server (edge)', edgeSel, 'The server that listens for visitors.'));
+        kids.push(h('fieldset', {}, h('legend', {}, 'Where does the traffic go?'), tgtBox), proxyBox);
+      } else if (cur === 'reach') {
+        prefill();
+        const web = isWeb();
+        kids.push(h('div', {class:'grid'}, web || protoSel.value === 'tls' ? hostBox : null,
+          field('Public port', port, web ? 'Websites normally use ' + (protoSel.value === 'https' ? '443' : '80') + '.' : 'One port, or a range like 2456-2458: each port goes to the target port plus the same offset.'),
+          field('Public IP', ipIn, '0.0.0.0 means every address on the edge.')), ipList, domList);
+      } else if (cur === 'access') {
+        kids.push(accessFs);
+      } else {
+        kids.push(field('Name', name, 'Optional. Shown in the list.'), h('div', {class:'card', style:'background:var(--surface-2); margin:0'}, summary()),
+          h('details', {}, h('summary', {class:'small dim', style:'cursor:pointer'}, 'More options (health checks' + (isWeb() ? ', rules, redirects' : '') + ')'),
+            h('div', {class:'stack', style:'margin-top:10px'}, adv, ...(isWeb() ? [rulesFs, advFs] : []))));
+      }
+      body.replaceChildren(...kids);
+    }
+    draw();
+    return modal('Add a resource', h('div', {class:'stack'}, crumbs, body, err,
+      h('div', {class:'actions'}, back, next, h('button', {class:'ghost', type:'button', onclick:() => close()}, 'Cancel'))), {sticky:true});
+  }
 }
 
 async function shareDialog(m) {
@@ -581,12 +746,14 @@ views.sites = async () => {
   S.headerAction = ['Add site', () => { const close = modal('Add a site', h('div', {class:'stack'},
     h('p', {class:'dim'}, 'A site is a private network that holds your services. Every edge also has a built-in "this machine" site for services on the same box as Blacktip.'), field('Name', name),
     h('div', {class:'actions'}, h('button', {class:'btn', onclick:() => { close(); add(); }}, 'Add site'), h('button', {class:'ghost', onclick:() => close()}, 'Cancel')))); }];
+  const latest = await latestVersions();
   return h('div', {},
-    card('Sites', h('table', {}, h('thead', {}, h('tr', {}, ['', 'Name', 'Kind', 'Tunnels to', 'Containers', ''].map(t => h('th', {}, t)))),
+    card('Sites', h('table', {}, h('thead', {}, h('tr', {}, ['', 'Name', 'Kind', 'Tunnels to', 'Version', 'Containers', ''].map(t => h('th', {}, t)))),
       h('tbody', {}, st.sites.map(s => h('tr', {},
         h('td', {}, h('span', {class:'dot ' + (st.online[s.id] ? 'ok' : 'bad')})), h('td', {}, s.name),
         h('td', {}, s.local ? 'this machine (direct)' : 'bamboo agent'),
         h('td', {}, s.local ? '—' : ((s.connected || []).map(id => (byId(st.edges, id) || {name:id}).name).join(', ') || 'nothing connected')),
+        s.local ? h('td', {class:'dim'}, '—') : versionCell(s, latest.bamboo, !!s.updateId, O(`/sites/${s.id}/update`), O(`/sites/${s.id}/update/log`)),
         h('td', {}, h('button', {class:'ghost', onclick: act(async () => {
           const l = await api('GET', O(`/sites/${s.id}/containers`));
           modal('Containers on ' + s.name, l.length ? h('table', {}, h('thead', {}, h('tr', {}, ['Name', 'Image', 'Ports'].map(t => h('th', {}, t)))), h('tbody', {}, l.map(c => h('tr', {}, h('td', {}, c.name), h('td', {class:'dim'}, c.image), h('td', {class:'mono small'}, c.ports.map(p => (p.public ? p.public + '→' : '') + p.private + '/' + p.type).join(', ')))))) :
@@ -857,10 +1024,11 @@ serverViews.edges = async () => {
     h('div', {class:'actions'}, h('button', {class:'btn', onclick: act(async () => {
       const e = await api('POST', '/api/edges', {name:name.value, tunnelAddr:addr.value, orgIds:orgChecks.filter(x => x.c.box.checked).map(x => x.o.id)}); close(); showCmd(e); refresh(); })}, 'Add edge'),
       h('button', {class:'ghost', onclick:() => close()}, 'Cancel')))); }];
+  const latest = await latestVersions();
   return h('div', {},
-    card('Edges', edges.length ? h('table', {}, h('thead', {}, h('tr', {}, ['', 'Name', 'WireGuard endpoint', 'IP addresses', 'Orgs', ''].map(t => h('th', {}, t)))),
+    card('Edges', edges.length ? h('table', {}, h('thead', {}, h('tr', {}, ['', 'Name', 'WireGuard endpoint', 'IP addresses', 'Version', 'Orgs', ''].map(t => h('th', {}, t)))),
       h('tbody', {}, edges.map(e => h('tr', {}, h('td', {}, h('span', {class:'dot ' + (e.online ? 'ok' : 'bad'), title:e.online ? 'online' : 'offline'})), h('td', {}, e.name), h('td', {class:'mono'}, e.tunnelAddr),
-        h('td', {class:'mono small'}, (e.ips || []).join(', ') || '—'), h('td', {class:'small dim'}, (e.orgIds || []).length ? e.orgIds.map(id => (byId(orgs, id) || {name:id}).name).join(', ') : 'all'),
+        h('td', {class:'mono small'}, (e.ips || []).join(', ') || '—'), versionCell(e, latest.blacktip, e.updatePending, '/api/edges/' + e.id + '/update', '/api/edges/' + e.id + '/update/log'), h('td', {class:'small dim'}, (e.orgIds || []).length ? e.orgIds.map(id => (byId(orgs, id) || {name:id}).name).join(', ') : 'all'),
         h('td', {}, h('div', {class:'actions'}, h('button', {class:'ghost', onclick:() => showCmd(e)}, 'Show command'),
           h('button', {class:'ghost danger', onclick: act(async () => { if (confirm('Delete this edge and all its ports?')) { await api('DELETE', '/api/edges/' + e.id); refresh(); } })}, 'Delete'))))))) : h('div', {class:'empty'}, 'None yet.')));
 };
@@ -925,12 +1093,35 @@ serverViews.settings = async () => {
   const staging = chk('Use the Let\'s Encrypt staging server (untrusted test certificates)', s.acmeStaging);
   const save = act(async () => { await api('PUT', '/api/admin/settings', {dashboardUrl:f.url.value, acmeEmail:f.acme.value, acmeStaging:staging.box.checked,
     smtp:{host:f.host.value, port:+f.port.value, user:f.user.value, pass:f.pass.value, from:f.from.value, mode:f.mode.value}}); S.me = await api('GET', '/api/me'); toast('Saved'); });
+  const updOut = h('div', {class:'dim'}, 'Not checked yet.');
+  const updNow = h('button', {class:'btn', style:'display:none', onclick: act(async () => {
+    await api('POST', '/api/admin/update/apply');
+    updNow.style.display = 'none';
+    showPanel();
+  })}, 'Update now');
+  const updBox = h('div', {style:'margin-top:12px'});
+  const showPanel = () => { updOut.textContent = 'Updating Hammerhead. The page reconnects by itself after the restart.'; updBox.replaceChildren(updatePanel(() => api('GET', '/api/admin/update/status'), async s => { if (s.stage === 'done') updOut.textContent = 'Updated: Hammerhead is now running ' + s.version + '.'; })); };
+  api('GET', '/api/admin/update/status').then(s => {
+    if (s.running) showPanel();
+    else if ((s.lines || []).length) updBox.append(h('button', {class:'ghost', onclick:() => { updBox.replaceChildren(updatePanel(() => api('GET', '/api/admin/update/status'))); }}, 'Show the last update log'));
+  }).catch(() => {});
+  const checkUpdate = act(async () => {
+    updOut.textContent = 'Checking…'; updNow.style.display = 'none';
+    const u = await api('GET', '/api/admin/update');
+    const dev = !/^v\d+\.\d+\.\d+$/.test(u.current || '');
+    if (u.error) updOut.textContent = 'Running ' + (u.current || 'an unknown version') + '. Could not check for updates: ' + u.error;
+    else if (dev) updOut.textContent = 'This is a development build (' + (u.current || 'unversioned') + '). The newest release is ' + u.latest + '.';
+    else if (u.newer) { updOut.textContent = 'Running ' + u.current + '. ' + u.latest + ' is available. ' + (u.autoUpdate ? 'Auto-update is on, so it installs by itself within about six hours, or you can install it now.' : 'Auto-update is off: install it yourself, or set HAMMERHEAD_AUTO_UPDATE=1 and restart.'); if (u.autoUpdate) updNow.style.display = ''; }
+    else updOut.textContent = 'Running ' + u.current + ', which is the newest release.';
+  });
   return h('div', {},
     card('General', h('div', {class:'stack'}, field('Dashboard URL', f.url, 'The address people use to reach Hammerhead. Used for login redirects and SSO. Include https:// if you serve it over TLS.'),
       field('Let\'s Encrypt contact email', f.acme, 'Used for certificate expiry notices'), staging)),
     card('Email', h('p', {class:'dim'}, 'Needed for invitation emails and one-time sign-in codes.'),
       h('div', {class:'grid'}, field('SMTP host', f.host), field('Port', f.port), field('Security', f.mode), field('Username', f.user), field('Password', f.pass), field('From', f.from)),
       h('div', {class:'actions', style:'margin-top:10px'}, h('button', {class:'ghost', onclick: act(async () => { await save(); await api('POST', '/api/admin/settings/test-mail'); toast('Test email sent to ' + S.me.email); })}, 'Save and send a test email'))),
+    card('Updates', h('p', {class:'dim'}, 'Hammerhead, Blacktip and Bamboo can update themselves from new releases when auto-update is on (HAMMERHEAD_AUTO_UPDATE=1). This checks the newest release for Hammerhead; it installs nothing.'),
+      updOut, h('div', {class:'actions', style:'margin-top:10px'}, h('button', {class:'ghost', onclick:checkUpdate}, 'Check for updates'), updNow), updBox),
     h('button', {class:'btn', onclick:save}, 'Save settings'));
 };