297 lines
21 KiB
Markdown
297 lines
21 KiB
Markdown
# Hammerhead
|
|
|
|
Self-hosted control plane for tunnelled ports and websites: a web dashboard, accounts and SSO, alerts, and a database, with no config files.
|
|
|
|
Hammerhead is the hub of the sharks stack. It runs the dashboard and API, and keeps your ports, domains, organizations, users, alerts and logs in an encrypted SQLite database. Edges and site agents connect to it and do whatever you set up in the UI.
|
|
|
|
| Repo | Role | Runs on |
|
|
|---|---|---|
|
|
| **[Hammerhead](https://git.veillax.com/Veillax/hammerhead)** (this repo) | Control plane: dashboard, API, accounts, database | anywhere the other two can reach |
|
|
| [Blacktip](https://git.veillax.com/Veillax/blacktip) | Edge: owns public IPs, opens ports, WireGuard endpoint, enforces logins (optionally drives Traefik) | public server(s) |
|
|
| [Bamboo](https://git.veillax.com/Veillax/bamboo) | Site agent: WireGuard peer inside a private network, outbound only | each private network |
|
|
| [Reef](https://git.veillax.com/Veillax/reef) | Shared library imported by all three | build time only |
|
|
|
|
Every Blacktip also gets a built-in **"this machine"** site, so services on the same box
|
|
(e.g. several Minecraft proxies) work without running Bamboo at all.
|
|
|
|
## Quick start
|
|
|
|
```
|
|
go env -w GOPRIVATE=git.veillax.com # once, so Go fetches straight from this server
|
|
|
|
# 1. control plane (this repo); open http://localhost:8080 and create the first admin
|
|
git clone https://git.veillax.com/Veillax/hammerhead && cd hammerhead
|
|
go build -o bin/ ./cmd/hammerhead
|
|
cp .env.example .env
|
|
bin/hammerhead -genkey # paste the output into .env as HAMMERHEAD_MASTER_KEY=
|
|
bin/hammerhead -listen :8080 -url https://hammerhead.example.com
|
|
|
|
# 2. Server -> Edges -> add an edge (name + public host:UDP-port, e.g. 203.0.113.10:51820);
|
|
# the UI shows the command to run on the public server
|
|
go install git.veillax.com/Veillax/blacktip/cmd/blacktip@latest
|
|
blacktip -hammerhead https://hammerhead.example.com -token <edge token> -tunnel :51820 -docker
|
|
|
|
# 3. optional, for another network: Sites -> add a site, run on a machine there
|
|
go install git.veillax.com/Veillax/bamboo/cmd/bamboo@latest
|
|
bamboo -hammerhead https://hammerhead.example.com -token <site token> -docker
|
|
```
|
|
|
|
Open the edge's WireGuard UDP port (and any ports you open) in its firewall. Bamboo needs no inbound
|
|
ports. Then open ports on the **Resources** page.
|
|
|
|
## Deploying
|
|
|
|
Releases are built by Gitea Actions. Pushing a tag such as `v0.2.0` runs `.gitea/workflows/release.yml`, which tests the code, builds Hammerhead for Linux (amd64) and Windows (amd64), publishes the archives with a `SHA256SUMS` file on the Releases page, and pushes a container image (linux/amd64) to this server's registry (`git.veillax.com/veillax/hammerhead`). Every push and pull request runs `.gitea/workflows/ci.yml`: formatting, vet, tests, and a check that every release platform still compiles.
|
|
|
|
**Fastest: one script on a fresh VPS**
|
|
|
|
`deploy/setup.sh` (also inside the Linux archives) installs everything for you on a systemd Linux: the services, a generated master key, the firewall rules and HTTPS.
|
|
|
|
```
|
|
sudo sh setup.sh --domain hammerhead.example.com # Hammerhead and an edge on this machine
|
|
sudo sh setup.sh agent --hammerhead https://hammerhead.example.com --token <site token> # on each site
|
|
sudo sh setup.sh edge --hammerhead https://hammerhead.example.com --token <edge token> # more public servers
|
|
```
|
|
|
|
With `--domain`, Hammerhead listens only on `127.0.0.1:8080` and gets an edge of its own (`HAMMERHEAD_SELF_DOMAIN`): the Blacktip on the same machine serves the dashboard at `https://<domain>` with an automatic certificate, so no Caddy or other web server is needed. The route is built in, not a mapping you can edit or delete by mistake. DNS for the name must point at the server and ports 80 and 443 must be free. `--caddy` keeps the old style: Hammerhead alone, with Caddy providing HTTPS. Add `--ports 25565,30000-30100` to open game or mail ports in the firewall, and `--no-auto-update` to leave updates to you.
|
|
|
|
**On a Linux server (systemd)**
|
|
|
|
```
|
|
curl -LO https://git.veillax.com/Veillax/hammerhead/releases/download/v0.1.0/hammerhead_v0.1.0_linux_amd64.tar.gz
|
|
curl -LO https://git.veillax.com/Veillax/hammerhead/releases/download/v0.1.0/SHA256SUMS
|
|
sha256sum -c --ignore-missing SHA256SUMS
|
|
tar xzf hammerhead_v0.1.0_linux_amd64.tar.gz && cd hammerhead_v0.1.0_linux_amd64
|
|
sudo ./install.sh # installs the binary and a hardened systemd service
|
|
sudoedit /etc/hammerhead/hammerhead.env # your settings and secrets
|
|
sudo systemctl start hammerhead
|
|
```
|
|
|
|
Releases are published for x86-64 only; for ARM machines build it yourself (`TARGETS=linux/arm64 sh deploy/build.sh`). To upgrade, repeat the same steps: your settings and data are kept. `sudo ./install.sh --uninstall` removes the service and binary but leaves `/etc/hammerhead` and `/var/lib/hammerhead` alone. `hammerhead -version` tells you which build is running.
|
|
|
|
**Updates**
|
|
|
|
Set `HAMMERHEAD_AUTO_UPDATE=1` (or pass `-auto-update`) and Hammerhead checks this server for a newer release every few hours, verifies it against `SHA256SUMS`, checks that the new binary runs, swaps it in and restarts itself. It only follows full releases (never pre-releases), never downgrades, and keeps the old binary if anything looks wrong. `setup.sh` turns it on. It is not available on Windows, and it is for the systemd and plain-binary installs: with Docker, pull a new image instead.
|
|
|
|
Edges and sites report their version, so Server → Edges and Sites show which release each one runs and flag a newer one. An **Update** button appears for any machine that runs with auto-update on (`BLACKTIP_AUTO_UPDATE=1` / `BAMBOO_AUTO_UPDATE=1`): it asks that machine to install the newest release now, and the result is shown in the table. Under Server → Settings, **Check for updates** does the same lookup for Hammerhead itself, with an **Update now** button when auto-update is on. A machine without auto-update never installs anything on request. Starting an update opens a progress bar with the live log of what the machine is doing (checking, downloading, verifying, installing, restarting). Hammerhead keeps the log of its own last update in `update.log` in its data folder, so it is still there after the restart.
|
|
|
|
**With Docker**
|
|
|
|
```
|
|
cp .env.example .env # fill it in
|
|
docker compose -f deploy/docker-compose.yml up -d
|
|
```
|
|
|
|
The image is `git.veillax.com/veillax/hammerhead:<version>` (and `:latest` for full releases). The compose file is in the repository and inside each Linux archive.
|
|
|
|
**Setting up the build (once)**
|
|
|
|
- A Gitea Actions runner with the label `ubuntu-latest`, and Docker on it if you want the container image.
|
|
- Nothing else is required: the workflows use the automatic job token. If your server doesn't let that token publish releases or packages, add the secrets `RELEASE_TOKEN` and `REGISTRY_TOKEN` (access tokens with release and package write permission) and optionally the variable `REGISTRY_USER`.
|
|
- To skip the container image, set the repository variable `PUBLISH_DOCKER` to `false`.
|
|
- Build locally with `sh deploy/build.sh` (archives in `dist/`).
|
|
|
|
## Configuration and secrets
|
|
|
|
All three programs read a `.env` file from their working directory before they start (or the file
|
|
named by `-env path`). Real environment variables always win over the file. Copy `.env.example`,
|
|
which lists every setting, and keep `.env` out of version control (`.gitignore` already does).
|
|
|
|
What belongs in it: `HAMMERHEAD_MASTER_KEY`, the first admin's credentials, the edge and site
|
|
`..._TOKEN` values, `HAMMERHEAD_METRICS_TOKEN`, and DNS-provider credentials for wildcard certificates.
|
|
|
|
**Secrets at rest.** Hammerhead keeps everything in one SQLite file (`hammerhead.db` in its data
|
|
folder) and encrypts the secrets inside it with AES-256-GCM: edge and site tokens, 2FA secrets,
|
|
OIDC client secrets, SMTP and relay passwords, DKIM private keys, webhook URLs. Passwords are bcrypt
|
|
hashes; PINs and resource passwords likewise. With the wrong key Hammerhead refuses to start rather
|
|
than reading garbage. If you don't set `HAMMERHEAD_MASTER_KEY`, a key is generated into
|
|
`<data>/master.key` so it works out of the box, but then a backup of the folder carries the key too;
|
|
copy the file's contents into `.env` and delete it. **Back the key up separately: without it the
|
|
encrypted secrets cannot be recovered.**
|
|
|
|
## Database
|
|
|
|
SQLite (pure Go, no C compiler needed), in WAL mode. Configuration is written incrementally in one
|
|
transaction per change, so it survives crashes and power loss. Logs, traffic history and alerts are
|
|
proper indexed tables: access logs are searchable and paginated, old entries are pruned
|
|
(`HAMMERHEAD_LOG_DAYS`, default 30; audit log a year). An existing `state.json` and JSONL logs from an
|
|
earlier version are imported automatically on first start and kept as `*.migrated`.
|
|
|
|
Back up by copying the data folder while Hammerhead is stopped (or use `sqlite3 .backup`), plus your
|
|
master key.
|
|
|
|
## The tunnel: WireGuard
|
|
|
|
Blacktip and Bamboo are WireGuard peers, using [wireguard-go](https://git.zx2c4.com/wireguard-go)
|
|
with a userspace network stack (the same approach Newt uses), so there is no root requirement, no kernel
|
|
interface and no driver; it works on Windows, macOS and Linux alike.
|
|
|
|
- Hammerhead allocates each edge and site an address in `100.89.0.0/16` (it exists only inside the
|
|
tunnel, so it can't clash with any LAN or VPN) and distributes public keys. Private keys never leave
|
|
the machine that made them (`wg.key` in each agent's data directory).
|
|
- Bamboo dials out to every edge it is used from and keeps the NAT mapping open (keepalive 25s),
|
|
so sites behind NAT or CGNAT work. Each peer is limited to its own `/32`.
|
|
- TCP goes over a TCP connection inside the tunnel; UDP uses native datagrams (no UDP-over-TCP
|
|
head-of-line blocking, good for game servers).
|
|
- Adding or removing a site changes peers in place; existing tunnels are not reset.
|
|
|
|
## Two ways to serve web resources
|
|
|
|
| | built-in proxy (default) | Traefik (`-traefik /path/to/traefik`) |
|
|
|---|---|---|
|
|
| http/https routing, HTTP/2, websockets | yes | yes |
|
|
| Let's Encrypt | TLS-ALPN-01, per-subdomain certificates for wildcard hostnames | TLS-ALPN-01, or DNS-01 with `-acme-dns <provider>` for true wildcard certificates |
|
|
| Logins, rules, share links | in Blacktip | same logic, called by Traefik's forward-auth |
|
|
| Dependency | none | a Traefik v3 binary |
|
|
|
|
With Traefik, Blacktip supervises the process: it writes Traefik's static config, restarts it only when
|
|
the set of web `ip:port` listeners changes (Traefik declares its ports statically), and hot-reloads
|
|
routes, logins and targets through the file provider. Targets are reached through a loopback relay port
|
|
per resource into the WireGuard tunnel, so health checks and failover work the same. **Raw TCP, UDP and
|
|
TLS-passthrough ports never go through Traefik**: it can't open ports without a restart, and opening
|
|
ports from the UI instantly is the point of this project, so Blacktip's own listeners handle those.
|
|
|
|
`-no-acme` makes Traefik serve its built-in self-signed certificate (useful on private networks).
|
|
DNS-challenge credentials are passed as environment variables to Blacktip, exactly as Traefik documents
|
|
for each provider (e.g. `CF_DNS_API_TOKEN` for `cloudflare`); `.env` is the place for them.
|
|
|
|
## Features
|
|
|
|
**Ports**
|
|
- The same port on different IPs (`x.y.z.104:51000` and `x.y.z.209:51000` go to different targets).
|
|
Blacktip lists the IPs it finds; you can also type one.
|
|
- **Port ranges** for tcp and udp: `2456-2458` opens all three, each sent to the target port plus the
|
|
same offset (up to 1000 ports per entry; a busy port inside a range is named in the status).
|
|
- TCP, UDP, "TCP + UDP" in one click, HTTP and HTTPS reverse proxy, and TLS passthrough by SNI.
|
|
- **PROXY protocol (v1/v2)** on tcp and tls ports: the target is told the real client address, so IP
|
|
bans, rate limits, spam checks and logs see the visitor instead of the tunnel. Your server must be
|
|
set to expect it and to trust only the edge or site: Velocity/BungeeCord `haproxy-protocol`, Postfix
|
|
`postscreen_upstream_proxy_protocol = haproxy`, Dovecot `haproxy_trusted_networks`, nginx `proxy_protocol`.
|
|
- Many hostnames share one `ip:port`; `*.example.com` wildcards work.
|
|
- Several targets per port: round-robin load balancing with automatic failover.
|
|
- Health checks (TCP connect or an HTTP path); unhealthy targets are skipped and shown in the UI.
|
|
- Conflict detection (same ip/port/protocol, overlapping ranges, or `0.0.0.0` overlapping a specific IP).
|
|
- Traffic totals and a 24-hour sparkline per port, plus a **Traffic** chart per resource (and a
|
|
**Traffic overview** for the whole organization) with bytes in and out, connections and the
|
|
busiest visitor addresses. See *Traffic analytics* below.
|
|
|
|
**HTTPS**: automatic Let's Encrypt certificates, optional port-80 redirect, `X-Forwarded-*`
|
|
headers, Host rewrite, HTTPS to the backend, certificate status per port.
|
|
|
|
**Login in front of any web port** (like Pangolin's resource auth)
|
|
- Sign in with a Hammerhead account (limited to roles or everyone in the org), a shared
|
|
password, a PIN, or an emailed one-time code.
|
|
- Expiring, use-limited share links that skip the login.
|
|
- Path and IP/CIDR rules that skip the login or block requests.
|
|
- The backend gets `X-Hammerhead-User` (client-supplied copies and header aliases are removed) and
|
|
never sees the session cookie.
|
|
- **Sessions can be revoked.** Visitor cookies are stateless tokens carrying a per-resource epoch.
|
|
Hammerhead bumps it, and edges refuse older tokens within seconds, when a member is removed, a role
|
|
changes or is deleted, a user is deleted, the password/PIN or allowed methods change, a share link is
|
|
revoked, or you press **Sign out visitors**.
|
|
|
|
**Accounts**
|
|
- Organizations, roles (admin and custom), invitations (by link or email), member management.
|
|
- Two-factor authentication (TOTP) with backup codes. Password changes sign out other sessions.
|
|
- **Signed-in devices** on the Account page: see where you're signed in and end any session, or all the
|
|
others; admins can sign a user out everywhere.
|
|
- OpenID Connect single sign-on (Google, Authentik, Keycloak, Entra ID…) with auto-provisioning.
|
|
- Org-scoped API keys, a "My apps" page for members, brute-force throttling.
|
|
- Domains are owned per org and checked against DNS; hostnames must sit under your own domains.
|
|
|
|
**Alerts**
|
|
- Email or webhook (JSON, Slack, Discord, ntfy) when an edge or site goes quiet, a port can't bind, a
|
|
target fails its health check, a certificate errors, or a mail relay isn't deliverable (reverse DNS,
|
|
HELO, port 25, DKIM record missing).
|
|
- A problem must last the **debounce time** (default 2 minutes, per organization) before you hear about
|
|
it, so restarts and blips stay quiet. One message when it starts, one when it clears, optional
|
|
reminders. Active alerts are stored, so restarting Hammerhead doesn't repeat them, and an alert is
|
|
only marked resolved once the thing it describes has been re-checked.
|
|
- A dead edge silences its own ports' alerts (the edge alert already says it). Channels are per
|
|
organization; **server channels** get edge alerts.
|
|
- Webhooks can't be pointed at internal addresses (no metadata service, no LAN) unless a server admin
|
|
allows it for the organization, and redirects are not followed.
|
|
- **`/metrics`** (Prometheus) when `HAMMERHEAD_METRICS_TOKEN` is set: edge/site/port/target/mail-check
|
|
gauges, traffic counters, alerts firing. Off (404) otherwise.
|
|
|
|
**Traffic analytics**
|
|
- Every edge counts connections and bytes in and out per port, plus the ten busiest client addresses
|
|
per port, and sends them with its regular report (every 20 seconds). Hammerhead adds them up at
|
|
three resolutions: per minute (kept 48 hours), per hour (31 days) and, for visitor addresses, per
|
|
hour (7 days). Older rows are deleted by the same hourly clean-up as the logs.
|
|
- The charts are on the Resources page: **Traffic** on a row, or **Traffic overview** for everything.
|
|
Pick an hour or 6 hours (per minute), a day, a week or a month (per hour). Visitor addresses cover
|
|
at most the last 7 days, and very small visitors may be missing because each report names ten.
|
|
- `GET /api/orgs/{org}/traffic/{resource id or "all"}?range=1h|6h|24h|7d|30d` returns the series,
|
|
totals and busiest addresses as JSON, for scripts or your own dashboards.
|
|
- **Traffic alerts** (Alerts → Traffic, per organization, off by default), delivered like every other
|
|
alert, with the same debounce:
|
|
- *Spike*: a port carried at least N times its usual traffic in the last 5 minutes, compared with
|
|
the average over the previous 6 hours. A port needs an hour of history, and the burst must be over
|
|
10 MB, so quiet ports don't page you.
|
|
- *Daily amount*: a port carried more than the set number of MB (in plus out) since midnight UTC.
|
|
Neither blocks traffic; they only tell you.
|
|
|
|
**Mail egress (send your mail server's outgoing mail from the edge's IP)**
|
|
- Receiving servers judge mail by the IP it comes from: its reverse DNS, your HELO name and your SPF
|
|
record all have to agree. A mail server at home usually can't satisfy that (residential IP, port 25
|
|
blocked). Mail egress lets it send from a VPS IP you control instead.
|
|
- Bamboo opens a small SMTP relay on the mail server's network (default `127.0.0.1:2525`). Point your
|
|
MTA at it (Postfix: `relayhost = [127.0.0.1]:2525`). Each message crosses the WireGuard tunnel and
|
|
Blacktip delivers it straight to the recipient's server (MX lookup, MX fallback rules, opportunistic
|
|
STARTTLS) **from the source IP you choose**, announcing the HELO name you set. The recipient's replies
|
|
go straight back, so your server's queue, retries and bounces behave normally. Or set a smarthost to
|
|
relay through a provider that allows your edge IP.
|
|
- **DKIM signing at the edge.** Create a key per sending domain on the Mail egress page (RSA 2048,
|
|
generated by Hammerhead, private half encrypted at rest and held only by the edge); publish the TXT
|
|
record it shows. Mail is signed by From domain (a parent domain's key covers subdomains); a message
|
|
your mail server already signed for that domain is left alone, and a signing problem sends the mail
|
|
unsigned rather than dropping it. Hammerhead checks the DNS record and alerts if it's missing or wrong.
|
|
- **MTA-STS enforcement.** When a recipient domain publishes an `enforce` policy, mail only goes to the
|
|
servers it lists, over TLS with a verified certificate; otherwise it's a temporary failure so your
|
|
server retries. (`testing` mode doesn't block.) Policies are cached for their `max_age`. The policy
|
|
fetch can't reach internal addresses or follow redirects.
|
|
- The page shows an edge-side deliverability checklist: reverse DNS matches the HELO name, the HELO name
|
|
resolves back to the IP, outbound port 25 is open, plus the SPF record to publish.
|
|
- Guard rails: only the owning site can use an egress; recipients' servers on internal address ranges
|
|
are refused; a per-minute send limit (default 60) protects the IP's reputation if the mail server is
|
|
compromised; the relay requires a password if it listens beyond `127.0.0.1`; optional "require
|
|
verified TLS"; every delivery and refusal is in the access log (recipient domain, server used, SMTP
|
|
status; never message content).
|
|
- One transaction goes to one recipient domain; extra domains get `452` and your MTA resends them
|
|
separately, as MTAs expect. 50 MB limit.
|
|
- **Deliberately no queue at the edge.** Delivery is synchronous, so your mail server's own queue does
|
|
the retrying. Accepting mail at the edge and bouncing it later would create backscatter (bounces to
|
|
forged senders, which gets IPs blacklisted) and leave stored mail on a public server.
|
|
- Not included: smarthost authentication (smarthosts must allow the edge IP), DANE, and receiving
|
|
mail (use a normal TCP port with PROXY protocol for that).
|
|
|
|
**Operations**
|
|
- Searchable, paginated access log (per request for web ports, per connection for TCP, per message for
|
|
mail) and audit log.
|
|
- Docker container discovery on any site or edge (`-docker`): pick a container as a target.
|
|
- Server admins decide which orgs may use each edge, and whether an org may target the edge
|
|
machine itself (off by default for extra orgs, since that includes Hammerhead).
|
|
|
|
## How logins work
|
|
|
|
Blacktip redirects an unauthenticated visitor to Hammerhead's login page. Hammerhead signs a short-lived
|
|
token with a per-edge key and sends the visitor back; Blacktip verifies it and sets a cookie. The same
|
|
decision function runs in both proxy modes.
|
|
|
|
## Design
|
|
|
|
The dashboard follows the Lightwell design language: warm neutrals, one teal accent, system fonts, status badges with text, and dialogs that name what they affect. See [docs/design.md](docs/design.md). `node scripts/lint-design.mjs` checks the rules that a machine can check.
|
|
|
|
## Limits and notes
|
|
|
|
- No client-to-site VPN (Pangolin's Olm/private resources) and no blueprints/YAML import.
|
|
- Hammerhead serves plain HTTP; put it behind HTTPS (set `-url`) or keep it on a private network.
|
|
- SQLite means one Hammerhead instance (no clustering); fine for personal and small-team use.
|
|
- PROXY protocol covers tcp and tls ports, not udp.
|
|
- A Blacktip can only bind IPs that exist on its machine. Cloud elastic IPs that are NATed must be
|
|
bound by their private address.
|
|
- Let's Encrypt needs port 443 on the edge reachable from the internet and DNS pointing at it.
|
|
- Traefik's access log is imported into Hammerhead; the file is truncated when it passes 64 MB.
|
|
- Tests: `go test ./...`. The `e2e/` folder runs Hammerhead, Blacktip and Bamboo together (it pulls the other two in as test dependencies); the Traefik test runs when `TRAEFIK_BIN` points at a Traefik binary.
|