# Hammerhead

Self-hosted control plane for tunnelled ports and websites: a web dashboard, accounts and SSO, alerts, and a database, with no config files.

Hammerhead is the hub of the sharks stack. It runs the dashboard and API, and keeps your ports, domains, organizations, users, alerts and logs in an encrypted SQLite database. Edges and site agents connect to it and do whatever you set up in the UI.

| Repo | Role | Runs on |
|---|---|---|
| **[Hammerhead](https://git.veillax.com/Veillax/hammerhead)** (this repo) | Control plane: dashboard, API, accounts, database | anywhere the other two can reach |
| [Blacktip](https://git.veillax.com/Veillax/blacktip) | Edge: owns public IPs, opens ports, WireGuard endpoint, enforces logins (optionally drives Traefik) | public server(s) |
| [Bamboo](https://git.veillax.com/Veillax/bamboo) | Site agent: WireGuard peer inside a private network, outbound only | each private network |
| [Reef](https://git.veillax.com/Veillax/reef) | Shared library imported by all three | build time only |

Every Blacktip also gets a built-in **"this machine"** site, so services on the same box
(e.g. several Minecraft proxies) work without running Bamboo at all.

## Quick start

```
go env -w GOPRIVATE=git.veillax.com      # once, so Go fetches straight from this server

# 1. control plane (this repo); open http://localhost:8080 and create the first admin
git clone https://git.veillax.com/Veillax/hammerhead && cd hammerhead
go build -o bin/ ./cmd/hammerhead
cp .env.example .env
bin/hammerhead -genkey                   # paste the output into .env as HAMMERHEAD_MASTER_KEY=
bin/hammerhead -listen :8080 -url https://hammerhead.example.com

# 2. Server -> Edges -> add an edge (name + public host:UDP-port, e.g. 203.0.113.10:51820);
#    the UI shows the command to run on the public server
go install git.veillax.com/Veillax/blacktip/cmd/blacktip@latest
blacktip -hammerhead https://hammerhead.example.com -token <edge token> -tunnel :51820 -docker

# 3. optional, for another network: Sites -> add a site, run on a machine there
go install git.veillax.com/Veillax/bamboo/cmd/bamboo@latest
bamboo -hammerhead https://hammerhead.example.com -token <site token> -docker
```

Open the edge's WireGuard UDP port (and any ports you open) in its firewall. Bamboo needs no inbound
ports. Then open ports on the **Resources** page.

## Deploying

Releases are built by Gitea Actions. Pushing a tag such as `v0.2.0` runs `.gitea/workflows/release.yml`, which tests the code, builds Hammerhead for Linux (amd64) and Windows (amd64), publishes the archives with a `SHA256SUMS` file on the Releases page, and pushes a container image (linux/amd64) to this server's registry (`git.veillax.com/veillax/hammerhead`). Every push and pull request runs `.gitea/workflows/ci.yml`: formatting, vet, tests, and a check that every release platform still compiles.

**Fastest: one script on a fresh VPS**

`deploy/setup.sh` (also inside the Linux archives) installs everything for you on a systemd Linux: the services, a generated master key, the firewall rules and HTTPS.

```
sudo sh setup.sh --domain hammerhead.example.com            # Hammerhead and an edge on this machine
sudo sh setup.sh agent --hammerhead https://hammerhead.example.com --token <site token>   # on each site
sudo sh setup.sh edge  --hammerhead https://hammerhead.example.com --token <edge token>   # more public servers
```

With `--domain`, Hammerhead listens only on `127.0.0.1:8080` and gets an edge of its own (`HAMMERHEAD_SELF_DOMAIN`): the Blacktip on the same machine serves the dashboard at `https://<domain>` with an automatic certificate, so no Caddy or other web server is needed. The route is built in, not a mapping you can edit or delete by mistake. DNS for the name must point at the server and ports 80 and 443 must be free. `--caddy` keeps the old style: Hammerhead alone, with Caddy providing HTTPS. Add `--ports 25565,30000-30100` to open game or mail ports in the firewall, and `--no-auto-update` to leave updates to you.

**On a Linux server (systemd)**

```
curl -LO https://git.veillax.com/Veillax/hammerhead/releases/download/v0.1.0/hammerhead_v0.1.0_linux_amd64.tar.gz
curl -LO https://git.veillax.com/Veillax/hammerhead/releases/download/v0.1.0/SHA256SUMS
sha256sum -c --ignore-missing SHA256SUMS
tar xzf hammerhead_v0.1.0_linux_amd64.tar.gz && cd hammerhead_v0.1.0_linux_amd64
sudo ./install.sh                 # installs the binary and a hardened systemd service
sudoedit /etc/hammerhead/hammerhead.env   # your settings and secrets
sudo systemctl start hammerhead
```

Releases are published for x86-64 only; for ARM machines build it yourself (`TARGETS=linux/arm64 sh deploy/build.sh`). To upgrade, repeat the same steps: your settings and data are kept. `sudo ./install.sh --uninstall` removes the service and binary but leaves `/etc/hammerhead` and `/var/lib/hammerhead` alone. `hammerhead -version` tells you which build is running.

**Updates**

Set `HAMMERHEAD_AUTO_UPDATE=1` (or pass `-auto-update`) and Hammerhead checks this server for a newer release every few hours, verifies it against `SHA256SUMS`, checks that the new binary runs, swaps it in and restarts itself. It only follows full releases (never pre-releases), never downgrades, and keeps the old binary if anything looks wrong. `setup.sh` turns it on. It is not available on Windows, and it is for the systemd and plain-binary installs: with Docker, pull a new image instead.

Edges and sites report their version, so Server → Edges and Sites show which release each one runs and flag a newer one. An **Update** button appears for any machine that runs with auto-update on (`BLACKTIP_AUTO_UPDATE=1` / `BAMBOO_AUTO_UPDATE=1`): it asks that machine to install the newest release now, and the result is shown in the table. Under Server → Settings, **Check for updates** does the same lookup for Hammerhead itself, with an **Update now** button when auto-update is on. A machine without auto-update never installs anything on request. Starting an update opens a progress bar with the live log of what the machine is doing (checking, downloading, verifying, installing, restarting). Hammerhead keeps the log of its own last update in `update.log` in its data folder, so it is still there after the restart.

**With Docker**

```
cp .env.example .env              # fill it in
docker compose -f deploy/docker-compose.yml up -d
```

The image is `git.veillax.com/veillax/hammerhead:<version>` (and `:latest` for full releases). The compose file is in the repository and inside each Linux archive.

**Setting up the build (once)**

- A Gitea Actions runner with the label `ubuntu-latest`, and Docker on it if you want the container image.
- Nothing else is required: the workflows use the automatic job token. If your server doesn't let that token publish releases or packages, add the secrets `RELEASE_TOKEN` and `REGISTRY_TOKEN` (access tokens with release and package write permission) and optionally the variable `REGISTRY_USER`.
- To skip the container image, set the repository variable `PUBLISH_DOCKER` to `false`.
- Build locally with `sh deploy/build.sh` (archives in `dist/`).

## Configuration and secrets

All three programs read a `.env` file from their working directory before they start (or the file
named by `-env path`). Real environment variables always win over the file. Copy `.env.example`,
which lists every setting, and keep `.env` out of version control (`.gitignore` already does).

What belongs in it: `HAMMERHEAD_MASTER_KEY`, the first admin's credentials, the edge and site
`..._TOKEN` values, `HAMMERHEAD_METRICS_TOKEN`, and DNS-provider credentials for wildcard certificates.

**Secrets at rest.** Hammerhead keeps everything in one SQLite file (`hammerhead.db` in its data
folder) and encrypts the secrets inside it with AES-256-GCM: edge and site tokens, 2FA secrets,
OIDC client secrets, SMTP and relay passwords, DKIM private keys, webhook URLs. Passwords are bcrypt
hashes; PINs and resource passwords likewise. With the wrong key Hammerhead refuses to start rather
than reading garbage. If you don't set `HAMMERHEAD_MASTER_KEY`, a key is generated into
`<data>/master.key` so it works out of the box, but then a backup of the folder carries the key too;
copy the file's contents into `.env` and delete it. **Back the key up separately: without it the
encrypted secrets cannot be recovered.**

## Database

SQLite (pure Go, no C compiler needed), in WAL mode. Configuration is written incrementally in one
transaction per change, so it survives crashes and power loss. Logs, traffic history and alerts are
proper indexed tables: access logs are searchable and paginated, old entries are pruned
(`HAMMERHEAD_LOG_DAYS`, default 30; audit log a year). An existing `state.json` and JSONL logs from an
earlier version are imported automatically on first start and kept as `*.migrated`.

Back up by copying the data folder while Hammerhead is stopped (or use `sqlite3 .backup`), plus your
master key.

## The tunnel: WireGuard

Blacktip and Bamboo are WireGuard peers, using [wireguard-go](https://git.zx2c4.com/wireguard-go)
with a userspace network stack (the same approach Newt uses), so there is no root requirement, no kernel
interface and no driver; it works on Windows, macOS and Linux alike.

- Hammerhead allocates each edge and site an address in `100.89.0.0/16` (it exists only inside the
  tunnel, so it can't clash with any LAN or VPN) and distributes public keys. Private keys never leave
  the machine that made them (`wg.key` in each agent's data directory).
- Bamboo dials out to every edge it is used from and keeps the NAT mapping open (keepalive 25s),
  so sites behind NAT or CGNAT work. Each peer is limited to its own `/32`.
- TCP goes over a TCP connection inside the tunnel; UDP uses native datagrams (no UDP-over-TCP
  head-of-line blocking, good for game servers).
- Adding or removing a site changes peers in place; existing tunnels are not reset.

## Two ways to serve web resources

| | built-in proxy (default) | Traefik (`-traefik /path/to/traefik`) |
|---|---|---|
| http/https routing, HTTP/2, websockets | yes | yes |
| Let's Encrypt | TLS-ALPN-01, per-subdomain certificates for wildcard hostnames | TLS-ALPN-01, or DNS-01 with `-acme-dns <provider>` for true wildcard certificates |
| Logins, rules, share links | in Blacktip | same logic, called by Traefik's forward-auth |
| Dependency | none | a Traefik v3 binary |

With Traefik, Blacktip supervises the process: it writes Traefik's static config, restarts it only when
the set of web `ip:port` listeners changes (Traefik declares its ports statically), and hot-reloads
routes, logins and targets through the file provider. Targets are reached through a loopback relay port
per resource into the WireGuard tunnel, so health checks and failover work the same. **Raw TCP, UDP and
TLS-passthrough ports never go through Traefik**: it can't open ports without a restart, and opening
ports from the UI instantly is the point of this project, so Blacktip's own listeners handle those.

`-no-acme` makes Traefik serve its built-in self-signed certificate (useful on private networks).
DNS-challenge credentials are passed as environment variables to Blacktip, exactly as Traefik documents
for each provider (e.g. `CF_DNS_API_TOKEN` for `cloudflare`); `.env` is the place for them.

## Features

**Ports**
- The same port on different IPs (`x.y.z.104:51000` and `x.y.z.209:51000` go to different targets).
  Blacktip lists the IPs it finds; you can also type one.
- **Port ranges** for tcp and udp: `2456-2458` opens all three, each sent to the target port plus the
  same offset (up to 1000 ports per entry; a busy port inside a range is named in the status).
- TCP, UDP, "TCP + UDP" in one click, HTTP and HTTPS reverse proxy, and TLS passthrough by SNI.
- **PROXY protocol (v1/v2)** on tcp and tls ports: the target is told the real client address, so IP
  bans, rate limits, spam checks and logs see the visitor instead of the tunnel. Your server must be
  set to expect it and to trust only the edge or site: Velocity/BungeeCord `haproxy-protocol`, Postfix
  `postscreen_upstream_proxy_protocol = haproxy`, Dovecot `haproxy_trusted_networks`, nginx `proxy_protocol`.
- Many hostnames share one `ip:port`; `*.example.com` wildcards work.
- Several targets per port: round-robin load balancing with automatic failover.
- Health checks (TCP connect or an HTTP path); unhealthy targets are skipped and shown in the UI.
- Conflict detection (same ip/port/protocol, overlapping ranges, or `0.0.0.0` overlapping a specific IP).
- Traffic totals and a 24-hour sparkline per port.

**HTTPS**: automatic Let's Encrypt certificates, optional port-80 redirect, `X-Forwarded-*`
headers, Host rewrite, HTTPS to the backend, certificate status per port.

**Login in front of any web port** (like Pangolin's resource auth)
- Sign in with a Hammerhead account (limited to roles or everyone in the org), a shared
  password, a PIN, or an emailed one-time code.
- Expiring, use-limited share links that skip the login.
- Path and IP/CIDR rules that skip the login or block requests.
- The backend gets `X-Hammerhead-User` (client-supplied copies and header aliases are removed) and
  never sees the session cookie.
- **Sessions can be revoked.** Visitor cookies are stateless tokens carrying a per-resource epoch.
  Hammerhead bumps it, and edges refuse older tokens within seconds, when a member is removed, a role
  changes or is deleted, a user is deleted, the password/PIN or allowed methods change, a share link is
  revoked, or you press **Sign out visitors**.

**Accounts**
- Organizations, roles (admin and custom), invitations (by link or email), member management.
- Two-factor authentication (TOTP) with backup codes. Password changes sign out other sessions.
- **Signed-in devices** on the Account page: see where you're signed in and end any session, or all the
  others; admins can sign a user out everywhere.
- OpenID Connect single sign-on (Google, Authentik, Keycloak, Entra ID…) with auto-provisioning.
- Org-scoped API keys, a "My apps" page for members, brute-force throttling.
- Domains are owned per org and checked against DNS; hostnames must sit under your own domains.

**Alerts**
- Email or webhook (JSON, Slack, Discord, ntfy) when an edge or site goes quiet, a port can't bind, a
  target fails its health check, a certificate errors, or a mail relay isn't deliverable (reverse DNS,
  HELO, port 25, DKIM record missing).
- A problem must last the **debounce time** (default 2 minutes, per organization) before you hear about
  it, so restarts and blips stay quiet. One message when it starts, one when it clears, optional
  reminders. Active alerts are stored, so restarting Hammerhead doesn't repeat them, and an alert is
  only marked resolved once the thing it describes has been re-checked.
- A dead edge silences its own ports' alerts (the edge alert already says it). Channels are per
  organization; **server channels** get edge alerts.
- Webhooks can't be pointed at internal addresses (no metadata service, no LAN) unless a server admin
  allows it for the organization, and redirects are not followed.
- **`/metrics`** (Prometheus) when `HAMMERHEAD_METRICS_TOKEN` is set: edge/site/port/target/mail-check
  gauges, traffic counters, alerts firing. Off (404) otherwise.

**Mail egress (send your mail server's outgoing mail from the edge's IP)**
- Receiving servers judge mail by the IP it comes from: its reverse DNS, your HELO name and your SPF
  record all have to agree. A mail server at home usually can't satisfy that (residential IP, port 25
  blocked). Mail egress lets it send from a VPS IP you control instead.
- Bamboo opens a small SMTP relay on the mail server's network (default `127.0.0.1:2525`). Point your
  MTA at it (Postfix: `relayhost = [127.0.0.1]:2525`). Each message crosses the WireGuard tunnel and
  Blacktip delivers it straight to the recipient's server (MX lookup, MX fallback rules, opportunistic
  STARTTLS) **from the source IP you choose**, announcing the HELO name you set. The recipient's replies
  go straight back, so your server's queue, retries and bounces behave normally. Or set a smarthost to
  relay through a provider that allows your edge IP.
- **DKIM signing at the edge.** Create a key per sending domain on the Mail egress page (RSA 2048,
  generated by Hammerhead, private half encrypted at rest and held only by the edge); publish the TXT
  record it shows. Mail is signed by From domain (a parent domain's key covers subdomains); a message
  your mail server already signed for that domain is left alone, and a signing problem sends the mail
  unsigned rather than dropping it. Hammerhead checks the DNS record and alerts if it's missing or wrong.
- **MTA-STS enforcement.** When a recipient domain publishes an `enforce` policy, mail only goes to the
  servers it lists, over TLS with a verified certificate; otherwise it's a temporary failure so your
  server retries. (`testing` mode doesn't block.) Policies are cached for their `max_age`. The policy
  fetch can't reach internal addresses or follow redirects.
- The page shows an edge-side deliverability checklist: reverse DNS matches the HELO name, the HELO name
  resolves back to the IP, outbound port 25 is open, plus the SPF record to publish.
- Guard rails: only the owning site can use an egress; recipients' servers on internal address ranges
  are refused; a per-minute send limit (default 60) protects the IP's reputation if the mail server is
  compromised; the relay requires a password if it listens beyond `127.0.0.1`; optional "require
  verified TLS"; every delivery and refusal is in the access log (recipient domain, server used, SMTP
  status; never message content).
- One transaction goes to one recipient domain; extra domains get `452` and your MTA resends them
  separately, as MTAs expect. 50 MB limit.
- **Deliberately no queue at the edge.** Delivery is synchronous, so your mail server's own queue does
  the retrying. Accepting mail at the edge and bouncing it later would create backscatter (bounces to
  forged senders, which gets IPs blacklisted) and leave stored mail on a public server.
- Not included: smarthost authentication (smarthosts must allow the edge IP), DANE, and receiving
  mail (use a normal TCP port with PROXY protocol for that).

**Operations**
- Searchable, paginated access log (per request for web ports, per connection for TCP, per message for
  mail) and audit log.
- Docker container discovery on any site or edge (`-docker`): pick a container as a target.
- Server admins decide which orgs may use each edge, and whether an org may target the edge
  machine itself (off by default for extra orgs, since that includes Hammerhead).

## How logins work

Blacktip redirects an unauthenticated visitor to Hammerhead's login page. Hammerhead signs a short-lived
token with a per-edge key and sends the visitor back; Blacktip verifies it and sets a cookie. The same
decision function runs in both proxy modes.

## Limits and notes

- No client-to-site VPN (Pangolin's Olm/private resources) and no blueprints/YAML import.
- Hammerhead serves plain HTTP; put it behind HTTPS (set `-url`) or keep it on a private network.
- SQLite means one Hammerhead instance (no clustering); fine for personal and small-team use.
- PROXY protocol covers tcp and tls ports, not udp.
- A Blacktip can only bind IPs that exist on its machine. Cloud elastic IPs that are NATed must be
  bound by their private address.
- Let's Encrypt needs port 443 on the edge reachable from the internet and DNS pointing at it.
- Traefik's access log is imported into Hammerhead; the file is truncated when it passes 64 MB.
- Tests: `go test ./...`. The `e2e/` folder runs Hammerhead, Blacktip and Bamboo together (it pulls the other two in as test dependencies); the Traefik test runs when `TRAEFIK_BIN` points at a Traefik binary.