diff --git a/.env.example b/.env.example index 87b9985..ebe7b3b 100644 --- a/.env.example +++ b/.env.example @@ -30,9 +30,14 @@ SANDBAR_MASTER_KEY= #SANDBAR_UPDATE_SERVER=https://git.veillax.com #SANDBAR_UPDATE_REPO=Veillax/sandbar +# Behind a reverse proxy, take the client address from X-Forwarded-For. Needed for website analytics and +# the sign-in limit to tell visitors apart. Only set it when the proxy is the only way to reach Sandbar. +#SANDBAR_TRUST_PROXY=1 + # How long topic messages and monitor results are kept, in days. #SANDBAR_MESSAGE_DAYS=30 #SANDBAR_CHECK_DAYS=30 +#SANDBAR_ANALYTICS_DAYS=365 # Outgoing email, for email channels and for paging people by email. #SANDBAR_SMTP_HOST=smtp.example.com diff --git a/README.md b/README.md index 1e897c0..9353372 100644 --- a/README.md +++ b/README.md @@ -3,12 +3,13 @@ Self-hosted notifications, monitoring, on-call and incident response in one small program. One binary, one SQLite file, and everything is done in the web UI. -Sandbar covers the ground of four tools: +Sandbar covers the ground of five tools: | Area | Like | What it does | |---|---|---| | **Topics** | ntfy | Publish a message to a topic with one HTTP request. Subscribe in the browser, with `curl`, or with the ntfy phone app | | **Monitors** | Tianji, Uptime Kuma | Website, port, DNS and certificate checks, heartbeats for cron jobs, and a public status page | +| **Websites** | Tianji, Plausible, Umami | Page views, visitors, referrers, browsers, countries and custom events for your own sites, with no cookies and no personal data | | **On-call** | Opsgenie | Rotating schedules, escalation policies, and alerts that are acknowledged, snoozed or resolved | | **Incidents** | incident.io | Declare, assign, update and close an incident, with a timeline and a post-mortem | @@ -112,7 +113,8 @@ new releases itself, in Docker and as a plain binary, on Linux and macOS. ### HTTPS and a reverse proxy Sandbar serves plain HTTP, so put a reverse proxy in front of it and set `SANDBAR_URL` to the public address, since -links in notifications use it. Caddy needs one line: +links in notifications use it. Also set `SANDBAR_TRUST_PROXY=1`, so website analytics and the sign-in limit see the +real client address instead of the proxy's. Caddy needs one line: ``` sandbar.example.com { @@ -200,6 +202,35 @@ in a small per-day table, so the bars go back 90 days even though raw check resu Monitors cannot probe private or internal addresses unless you tick **Allow targets on private networks** on that monitor, so a user who can add a monitor cannot use Sandbar to scan your network. +## Websites (analytics) + +Add a website on the Websites page, then paste its one-line tracking code into the head of your pages: + +```html + +``` + +The page for each website shows visitors, page views, visits, bounce rate and who is online now, a chart over the last +24 hours, 7, 30 or 90 days, and top lists for pages, referrers, browsers, operating systems, devices, countries, +languages and events. + +- **Single-page apps** are counted as people move between pages. Add `data-hash` to count `#hash` changes as pages, + or `data-auto="false"` to send nothing until your code calls `sandbar.view()`. +- **Events:** call `sandbar.track("signup", { plan: "pro" })` from your code. +- **Private by design:** no cookies, nothing stored in the browser, no IP address or user agent kept (only the labels + "Chrome", "macOS", "Mobile" and so on), and nothing is sent when the visitor has Do Not Track on. A visitor is a hash + of the address, the browser and the website with a secret and the date, so the same person is one visitor for a day + and cannot be followed beyond it. Query strings are dropped from paths and only the referrer's host is kept. +- **Domain:** a website's optional domain makes Sandbar ignore visits from other hosts. Bots, and anything sending + more than 120 hits a minute from one address, are ignored. +- **Countries** come from the `CF-IPCountry` or `CloudFront-Viewer-Country` header, so they appear only behind + Cloudflare or CloudFront. Sandbar bundles no location database. Without the header, visits count as Unknown. +- **Behind a reverse proxy,** set `SANDBAR_TRUST_PROXY=1` so the client address comes from `X-Forwarded-For`. Without + it every visitor looks like the proxy. Only set it when the proxy is the only way to reach Sandbar. +- Hits are kept for `SANDBAR_ANALYTICS_DAYS` (default 365). + +Not included: funnels, goals, session replay, a public share link, and comparison with the previous period. + ## On-call - A **schedule** rotates through people, handing over every N hours from a start time. **Overrides** cover a @@ -256,7 +287,7 @@ than their retention are pruned hourly. Back up by copying the data folder while - One instance, one organization. SQLite means no clustering. - No ICMP ping monitor, no SMS or voice, and no two-factor or single sign-on yet. - Sandbar serves plain HTTP. Put it behind HTTPS and set `-url`, or keep it on a private network. -- Sign-in attempts are limited to 10 per 10 minutes per address. +- Sign-in attempts are limited to 10 per 10 minutes per address (the client address, see `SANDBAR_TRUST_PROXY`). - Tests: `go test ./...`. Style: `node scripts/lint-design.mjs` (see `docs/design.md`). ## License diff --git a/cmd/sandbar/main.go b/cmd/sandbar/main.go index 37d1855..5d55065 100644 --- a/cmd/sandbar/main.go +++ b/cmd/sandbar/main.go @@ -86,6 +86,8 @@ func main() { MetricsToken: os.Getenv("SANDBAR_METRICS_TOKEN"), MessageDays: atoi("SANDBAR_MESSAGE_DAYS"), CheckDays: atoi("SANDBAR_CHECK_DAYS"), + HitDays: atoi("SANDBAR_ANALYTICS_DAYS"), + TrustProxy: os.Getenv("SANDBAR_TRUST_PROXY") == "1", Version: version, AutoUpdate: *autoUpdate, UpdateServer: os.Getenv("SANDBAR_UPDATE_SERVER"), diff --git a/docs/design.md b/docs/design.md index 933a5a9..1042b76 100644 --- a/docs/design.md +++ b/docs/design.md @@ -28,15 +28,17 @@ The helpers at the top of the script in `index.html` are the same as Hammerhead' | `rowList(addLabel, make, initial)` | A list the person adds to and removes from, such as people in a rotation or steps in a policy | | `timeline(events)` | Dated entries for an alert or an incident | -## Status page bars +## Status page bars and charts * **Uptime bars** on the public page are one bar per UTC day, 90 of them. Green is no failures, amber is at least 99% passing, red is below that, grey is no data. Color is never the only signal: the row carries a text status badge and the uptime percentage, each bar's tooltip says the date and the numbers in words, and the whole strip has an `aria-label` summary. +* **Analytics charts** are plain bars made of tokens, with two series (visitors in the accent, page views in a lighter mix of it), a legend with text, a native tooltip per bar, and an `aria-label` that gives the totals. There are no chart libraries. +* **Top lists** are tables with a proportional bar behind the name, so the numbers stay readable without the bar. ## Icons Lucide style, outline only, stroke 1.75, round caps, `currentColor`. Sizes are 16, 18 and 22 pixels. They are inline in the `ICONS` table. An icon goes with a text label, and icon-only buttons carry an `aria-label`. -Approved names: `sandbar`, `alerts`, `incidents`, `oncall`, `monitors`, `topics`, `channels`, `status`, `users`, `account`, `out`, `menu`, `close`, `add`, `check`, `back`, `light`, `dark`, `alert`. Add a name here and in `ICONS` together. +Approved names: `sandbar`, `alerts`, `incidents`, `oncall`, `monitors`, `websites`, `topics`, `channels`, `status`, `users`, `account`, `out`, `menu`, `close`, `add`, `check`, `back`, `light`, `dark`, `alert`. Add a name here and in `ICONS` together. ## Patterns diff --git a/internal/sandbar/analytics.go b/internal/sandbar/analytics.go new file mode 100644 index 0000000..2825ec1 --- /dev/null +++ b/internal/sandbar/analytics.go @@ -0,0 +1,431 @@ +package sandbar + +import ( + "crypto/sha256" + _ "embed" + "encoding/hex" + "encoding/json" + "errors" + "io" + "net" + "net/http" + "net/url" + "regexp" + "sort" + "strings" + "time" +) + +// Website analytics, in the manner of Tianji, Plausible and Umami: a small script on a site sends +// page views and custom events here, and the Websites page shows visitors, pages, referrers, +// browsers, systems, devices, countries and events. +// +// Privacy by construction: no cookies, no stored IP address or user agent. A visitor is a hash of the +// IP address, the user agent and the website, salted with a secret and the UTC date, so the same +// person is the same visitor within a day and cannot be followed beyond it. + +//go:embed web/script.js +var trackerJS []byte + +// Website is a site whose visits are counted. +type Website struct { + ID string `json:"id"` + Name string `json:"name"` + Domain string `json:"domain"` // optional: when set, only hits from this host and its subdomains count +} + +func prepareWebsite(v any, id string) error { + w := v.(*Website) + w.ID = id + w.Name = strings.TrimSpace(w.Name) + if w.Name == "" { + return errors.New("give the website a name") + } + d := strings.ToLower(strings.TrimSpace(w.Domain)) + if d != "" { + if !strings.Contains(d, "//") { + d = "//" + d + } + u, err := url.Parse(d) + if err != nil || u.Hostname() == "" { + return errors.New("the domain should look like example.com") + } + d = u.Hostname() + } + w.Domain = d + return nil +} + +func (s *Server) deleteWebsiteData(id string) { + s.st.db.Exec(`DELETE FROM hits WHERE website=?`, id) +} + +// realIP is the address of the client. Behind a reverse proxy, set TrustProxy so the address the +// proxy added (the last entry of X-Forwarded-For) is used; otherwise every visitor looks like the proxy. +func (s *Server) realIP(r *http.Request) string { + if s.opts.TrustProxy { + if xff := r.Header.Get("X-Forwarded-For"); xff != "" { + parts := strings.Split(xff, ",") + if ip := strings.TrimSpace(parts[len(parts)-1]); net.ParseIP(ip) != nil { + return ip + } + } + if ip := strings.TrimSpace(r.Header.Get("X-Real-IP")); net.ParseIP(ip) != nil { + return ip + } + } + h, _, err := net.SplitHostPort(r.RemoteAddr) + if err != nil { + return r.RemoteAddr + } + return h +} + +func (s *Server) serveTracker(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/javascript; charset=utf-8") + w.Header().Set("Cache-Control", "public, max-age=3600") + w.Header().Set("Access-Control-Allow-Origin", "*") + w.Write(trackerJS) +} + +var botRe = regexp.MustCompile(`(?i)bot|crawl|spider|slurp|headless|lighthouse|preview|monitor|curl/|wget|python-requests|go-http-client|okhttp`) + +// parseUA reduces a user agent to a browser, a system and a kind of device. Only these labels are stored. +func parseUA(ua string) (browser, os, device string) { + switch { + case strings.Contains(ua, "Edg/") || strings.Contains(ua, "EdgA/") || strings.Contains(ua, "EdgiOS/"): + browser = "Edge" + case strings.Contains(ua, "OPR/") || strings.Contains(ua, "Opera"): + browser = "Opera" + case strings.Contains(ua, "SamsungBrowser"): + browser = "Samsung Internet" + case strings.Contains(ua, "Firefox/") || strings.Contains(ua, "FxiOS/"): + browser = "Firefox" + case strings.Contains(ua, "Chrome/") || strings.Contains(ua, "CriOS/"): + browser = "Chrome" + case strings.Contains(ua, "Safari/"): + browser = "Safari" + default: + browser = "Other" + } + switch { + case strings.Contains(ua, "Windows"): + os = "Windows" + case strings.Contains(ua, "Android"): + os = "Android" + case strings.Contains(ua, "iPhone") || strings.Contains(ua, "iPad") || strings.Contains(ua, "iPod"): + os = "iOS" + case strings.Contains(ua, "Mac OS X") || strings.Contains(ua, "Macintosh"): + os = "macOS" + case strings.Contains(ua, "CrOS"): + os = "ChromeOS" + case strings.Contains(ua, "Linux") || strings.Contains(ua, "X11"): + os = "Linux" + default: + os = "Other" + } + switch { + case strings.Contains(ua, "iPad") || strings.Contains(ua, "Tablet") || (strings.Contains(ua, "Android") && !strings.Contains(ua, "Mobile")): + device = "Tablet" + case strings.Contains(ua, "Mobi") || strings.Contains(ua, "iPhone"): + device = "Mobile" + default: + device = "Desktop" + } + return +} + +// secret is the random value that salts visitor hashes. It is made once and kept with the other settings. +func (s *Server) analyticsSecret() string { + s.anaMu.Lock() + defer s.anaMu.Unlock() + if s.anaSecret != "" { + return s.anaSecret + } + var v struct{ Secret string } + if s.st.get("setting", "analytics", &v) != nil || v.Secret == "" { + v.Secret = newToken() + s.st.put("setting", "analytics", &v) + } + s.anaSecret = v.Secret + return v.Secret +} + +func (s *Server) visitorID(website, ip, ua string, now time.Time) string { + h := sha256.Sum256([]byte(s.analyticsSecret() + "|" + now.UTC().Format("2006-01-02") + "|" + website + "|" + ip + "|" + ua)) + return hex.EncodeToString(h[:8]) +} + +// hitAllowed limits how fast one address can send hits: a script that loops must not fill the database. +func (s *Server) hitAllowed(ip string) bool { + now := s.now() + s.anaMu.Lock() + defer s.anaMu.Unlock() + if len(s.anaTry) > 20000 { + s.anaTry = map[string][]time.Time{} + } + recent := s.anaTry[ip][:0] + for _, t := range s.anaTry[ip] { + if now.Sub(t) < time.Minute { + recent = append(recent, t) + } + } + if len(recent) >= 120 { + s.anaTry[ip] = recent + return false + } + s.anaTry[ip] = append(recent, now) + return true +} + +func clip(s string, n int) string { + s = strings.TrimSpace(s) + if len(s) > n { + s = strings.ToValidUTF8(s[:n], "") + } + return s +} + +// collect receives a page view or event from the tracker script. It is public by design, so it +// answers 204 to everything it ignores and gives nothing away. +func (s *Server) collect(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Access-Control-Allow-Origin", "*") + w.Header().Set("Access-Control-Allow-Methods", "POST, OPTIONS") + w.Header().Set("Access-Control-Allow-Headers", "Content-Type") + if r.Method == http.MethodOptions { + w.WriteHeader(204) + return + } + defer w.WriteHeader(204) + body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 4<<10)) + if err != nil { + return + } + var in struct { + Website, Path, Referrer, Hostname, Language, Event string + Data map[string]any + } + if json.Unmarshal(body, &in) != nil { + return + } + var site Website + if s.st.get("website", in.Website, &site) != nil { + return + } + host := strings.ToLower(in.Hostname) + if site.Domain != "" && host != site.Domain && !strings.HasSuffix(host, "."+site.Domain) { + return + } + ua := r.UserAgent() + if ua == "" || botRe.MatchString(ua) { + return + } + ip := s.realIP(r) + if !s.hitAllowed(ip) { + return + } + path := in.Path + if i := strings.IndexAny(path, "?"); i >= 0 { + path = path[:i] + } + if !strings.HasPrefix(path, "/") { + path = "/" + path + } + ref := "" + if u, err := url.Parse(in.Referrer); err == nil && u.Host != "" { + if rh := strings.TrimPrefix(strings.ToLower(u.Hostname()), "www."); rh != strings.TrimPrefix(host, "www.") { + ref = rh + } + } + country := strings.ToUpper(clip(r.Header.Get("CF-IPCountry"), 2)) + if country == "" { + country = strings.ToUpper(clip(r.Header.Get("CloudFront-Viewer-Country"), 2)) + } + if country == "XX" || country == "T1" { + country = "" + } + lang := strings.ToLower(clip(in.Language, 12)) + if i := strings.IndexAny(lang, "-_"); i > 0 { + lang = lang[:i] + } + data := "" + if in.Event != "" && len(in.Data) > 0 { + if b, err := json.Marshal(in.Data); err == nil && len(b) <= 1024 { + data = string(b) + } + } + browser, osName, device := parseUA(ua) + now := s.now() + s.st.db.Exec(`INSERT INTO hits(website,time,visitor,event,path,referrer,browser,os,device,country,lang,data) VALUES(?,?,?,?,?,?,?,?,?,?,?,?)`, + site.ID, now.Unix(), s.visitorID(site.ID, ip, ua, now), clip(in.Event, 80), clip(path, 200), ref, browser, osName, device, country, lang, data) +} + +// ---------- reading ---------- + +type topRow struct { + Name string `json:"name"` + Visitors int `json:"visitors"` + Views int `json:"views"` // page views, or the number of times for an event +} + +type seriesPoint struct { + Time int64 `json:"t"` + Views int `json:"views"` + Visitors int `json:"visitors"` +} + +type siteStats struct { + Range string `json:"range"` + Bucket string `json:"bucket"` // "hour" or "day" + Visitors int `json:"visitors"` + Views int `json:"views"` + Visits int `json:"visits"` + BounceRate float64 `json:"bounceRate"` // percent of visits with one page view; -1 if no visits + Online int `json:"online"` // distinct visitors in the last 5 minutes + Series []seriesPoint `json:"series"` + Pages []topRow `json:"pages"` + Referrers []topRow `json:"referrers"` + Browsers []topRow `json:"browsers"` + Systems []topRow `json:"systems"` + Devices []topRow `json:"devices"` + Countries []topRow `json:"countries"` + Languages []topRow `json:"languages"` + Events []topRow `json:"events"` +} + +var ranges = map[string]time.Duration{"24h": 24 * time.Hour, "7d": 7 * 24 * time.Hour, "30d": 30 * 24 * time.Hour, "90d": 90 * 24 * time.Hour} + +// visitGap is how long a visitor can be quiet before their next page view starts a new visit. +const visitGap = 30 * 60 + +func (s *Server) websiteStats(id, rng string) (*siteStats, error) { + span, ok := ranges[rng] + if !ok { + rng, span = "7d", ranges["7d"] + } + now := s.now() + from := now.Add(-span).Unix() + out := &siteStats{Range: rng, Bucket: "day", BounceRate: -1, Series: []seriesPoint{}} + step := int64(86400) + if rng == "24h" { + out.Bucket, step = "hour", 3600 + } + db := s.st.db + + db.QueryRow(`SELECT count(DISTINCT visitor), count(*) FROM hits WHERE website=? AND time>=? AND event=''`, id, from).Scan(&out.Visitors, &out.Views) + db.QueryRow(`SELECT count(DISTINCT visitor) FROM hits WHERE website=? AND time>=?`, id, now.Add(-5*time.Minute).Unix()).Scan(&out.Online) + + // Visits and bounces: a visitor's page views, split wherever they were quiet for half an hour. + if rows, err := db.Query(`SELECT visitor,time FROM hits WHERE website=? AND time>=? AND event='' ORDER BY visitor,time`, id, from); err == nil { + var prev string + var prevT int64 + views, bounces := 0, 0 + closeVisit := func() { + if views > 0 { + out.Visits++ + } + if views == 1 { + bounces++ + } + } + for rows.Next() { + var v string + var t int64 + if rows.Scan(&v, &t) != nil { + continue + } + if v != prev || t-prevT > visitGap { + closeVisit() + views = 0 + } + views++ + prev, prevT = v, t + } + closeVisit() + rows.Close() + if out.Visits > 0 { + out.BounceRate = float64(bounces) * 100 / float64(out.Visits) + } + } + + // The chart: one point per hour or UTC day, empty ones included so the line has no gaps. + buckets := map[int64]seriesPoint{} + if rows, err := db.Query(`SELECT (time/?)*?, count(*), count(DISTINCT visitor) FROM hits WHERE website=? AND time>=? AND event='' GROUP BY 1`, step, step, id, from); err == nil { + for rows.Next() { + var p seriesPoint + if rows.Scan(&p.Time, &p.Views, &p.Visitors) == nil { + buckets[p.Time] = p + } + } + rows.Close() + } + for t := (from / step) * step; t <= now.Unix(); t += step { + p := buckets[t] + p.Time = t + out.Series = append(out.Series, p) + } + + top := func(col string, events bool, empty string) []topRow { + cond := "event=''" + sel := col + if events { + cond, sel = "event!=''", "event" + } + rows, err := db.Query(`SELECT `+sel+`, count(DISTINCT visitor), count(*) FROM hits WHERE website=? AND time>=? AND `+cond+ + ` GROUP BY 1 ORDER BY 3 DESC, 1 LIMIT 10`, id, from) + list := []topRow{} + if err != nil { + return list + } + defer rows.Close() + for rows.Next() { + var r topRow + if rows.Scan(&r.Name, &r.Visitors, &r.Views) == nil { + if r.Name == "" { + r.Name = empty + } + list = append(list, r) + } + } + return list + } + out.Pages = top("path", false, "/") + out.Referrers = top("referrer", false, "Direct") + out.Browsers = top("browser", false, "Other") + out.Systems = top("os", false, "Other") + out.Devices = top("device", false, "Other") + out.Countries = top("country", false, "Unknown") + out.Languages = top("lang", false, "Unknown") + out.Events = top("", true, "") + sort.SliceStable(out.Series, func(i, j int) bool { return out.Series[i].Time < out.Series[j].Time }) + return out, nil +} + +func (s *Server) websiteStatsHandler(w http.ResponseWriter, r *http.Request) { + id := r.PathValue("id") + var site Website + if s.st.get("website", id, &site) != nil { + writeJSON(w, 404, map[string]string{"error": "No such website."}) + return + } + st, _ := s.websiteStats(id, r.URL.Query().Get("range")) + writeJSON(w, 200, st) +} + +// websiteSummary gives the list page a number for each site: visitors and page views in the last day. +func (s *Server) websiteSummary(w http.ResponseWriter, r *http.Request) { + out := map[string]map[string]int{} + since := s.now().Add(-24 * time.Hour).Unix() + rows, err := s.st.db.Query(`SELECT website, count(DISTINCT visitor), count(*) FROM hits WHERE time>=? AND event='' GROUP BY website`, since) + if err == nil { + defer rows.Close() + for rows.Next() { + var id string + var v, p int + if rows.Scan(&id, &v, &p) == nil { + out[id] = map[string]int{"visitors": v, "views": p} + } + } + } + writeJSON(w, 200, out) +} diff --git a/internal/sandbar/analytics_test.go b/internal/sandbar/analytics_test.go new file mode 100644 index 0000000..d85b1bc --- /dev/null +++ b/internal/sandbar/analytics_test.go @@ -0,0 +1,175 @@ +package sandbar + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +const ( + uaChrome = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" + uaPhone = "Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1" +) + +func TestParseUA(t *testing.T) { + for _, c := range []struct{ ua, browser, os, device string }{ + {uaChrome, "Chrome", "Windows", "Desktop"}, + {uaPhone, "Safari", "iOS", "Mobile"}, + {"Mozilla/5.0 (X11; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0", "Firefox", "Linux", "Desktop"}, + {"Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 Chrome/126.0 Safari/537.36 Edg/126.0", "Edge", "Windows", "Desktop"}, + {"Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 Chrome/126.0 Mobile Safari/537.36", "Chrome", "Android", "Mobile"}, + {"Mozilla/5.0 (Linux; Android 13; SM-X700) AppleWebKit/537.36 Chrome/126.0 Safari/537.36", "Chrome", "Android", "Tablet"}, + } { + b, o, d := parseUA(c.ua) + if b != c.browser || o != c.os || d != c.device { + t.Errorf("%q: got %s/%s/%s, want %s/%s/%s", c.ua, b, o, d, c.browser, c.os, c.device) + } + } +} + +func TestWebsiteAnalytics(t *testing.T) { + s, _, _ := newTestServer(t) + s.st.put("website", "w1", &Website{ID: "w1", Name: "Blog", Domain: "example.com"}) + ts := httptest.NewServer(s.Handler()) + defer ts.Close() + + send := func(ua string, body string) int { + req, _ := http.NewRequest("POST", ts.URL+"/api/send", strings.NewReader(body)) + req.Header.Set("Content-Type", "text/plain") + req.Header.Set("User-Agent", ua) + resp, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatal(err) + } + resp.Body.Close() + return resp.StatusCode + } + view := func(ua, path, ref string) { + b, _ := json.Marshal(map[string]string{"website": "w1", "hostname": "www.example.com", "path": path, "referrer": ref, "language": "en-GB"}) + if code := send(ua, string(b)); code != 204 { + t.Fatalf("send: %d", code) + } + } + view(uaChrome, "/", "https://news.example.org/story?id=1") + view(uaChrome, "/about?utm=x", "") + view(uaPhone, "/", "") + send(uaChrome, `{"website":"w1","hostname":"example.com","path":"/pricing","event":"signup","data":{"plan":"pro"}}`) + + // Things that must not be counted. + send("Googlebot/2.1", `{"website":"w1","hostname":"example.com","path":"/bot"}`) + send(uaChrome, `{"website":"w1","hostname":"evil.test","path":"/wrong-host"}`) + send(uaChrome, `{"website":"nope","hostname":"example.com","path":"/unknown-site"}`) + send(uaChrome, `not json`) + + c := newAPIClient(t, ts) + if code := c.do("GET", "/api/v1/websites/w1/stats", nil, nil); code != 401 { + t.Fatalf("stats without sign-in: %d", code) + } + c.setup() + var st siteStats + if code := c.do("GET", "/api/v1/websites/w1/stats?range=7d", nil, &st); code != 200 { + t.Fatalf("stats: %d", code) + } + if st.Views != 3 || st.Visitors != 2 || st.Visits != 2 || st.BounceRate != 50 { + t.Fatalf("totals: views %d visitors %d visits %d bounce %.0f", st.Views, st.Visitors, st.Visits, st.BounceRate) + } + has := func(rows []topRow, name string, views int) bool { + for _, r := range rows { + if r.Name == name && r.Views == views { + return true + } + } + return false + } + for _, c := range []struct { + what string + rows []topRow + name string + n int + }{ + {"page /", st.Pages, "/", 2}, {"page /about without the query", st.Pages, "/about", 1}, + {"referrer host only", st.Referrers, "news.example.org", 1}, {"direct", st.Referrers, "Direct", 2}, + {"browser", st.Browsers, "Chrome", 2}, {"os", st.Systems, "iOS", 1}, + {"device", st.Devices, "Mobile", 1}, {"language", st.Languages, "en", 3}, {"event", st.Events, "signup", 1}, + } { + if !has(c.rows, c.name, c.n) { + t.Errorf("%s: %q x%d not in %+v", c.what, c.name, c.n, c.rows) + } + } + for _, p := range st.Pages { + if p.Name == "/bot" || p.Name == "/wrong-host" || p.Name == "/unknown-site" || p.Name == "/pricing" { + t.Errorf("counted something it should not: %+v", p) + } + } + if st.Online != 2 { + t.Fatalf("online %d", st.Online) + } + nonEmpty := 0 + for _, p := range st.Series { + if p.Views > 0 { + nonEmpty++ + } + } + if nonEmpty != 1 || len(st.Series) < 7 { + t.Fatalf("series: %d points, %d with views", len(st.Series), nonEmpty) + } + + // No IP address or user agent is stored. + var n int + s.st.db.QueryRow(`SELECT count(*) FROM hits WHERE visitor LIKE '%127.0.0.1%' OR browser LIKE '%Mozilla%'`).Scan(&n) + if n != 0 { + t.Fatal("raw identifiers stored") + } + + // The script and the browser preflight. + resp, _ := http.Get(ts.URL + "/script.js") + body := make([]byte, 64) + resp.Body.Read(body) + resp.Body.Close() + if resp.StatusCode != 200 || !strings.Contains(resp.Header.Get("Content-Type"), "javascript") { + t.Fatalf("script.js: %d %s", resp.StatusCode, resp.Header.Get("Content-Type")) + } + req, _ := http.NewRequest("OPTIONS", ts.URL+"/api/send", nil) + resp, _ = http.DefaultClient.Do(req) + resp.Body.Close() + if resp.StatusCode != 204 || resp.Header.Get("Access-Control-Allow-Origin") != "*" { + t.Fatalf("preflight: %d", resp.StatusCode) + } + + // Deleting the website deletes its hits. + if code := c.do("DELETE", "/api/v1/websites/w1", nil, nil); code != 200 { + t.Fatalf("delete: %d", code) + } + s.st.db.QueryRow(`SELECT count(*) FROM hits`).Scan(&n) + if n != 0 { + t.Fatalf("%d hits left after deleting the website", n) + } +} + +func TestHitRateLimitAndProxy(t *testing.T) { + s, _, _ := newTestServer(t) + for i := 0; i < 120; i++ { + if !s.hitAllowed("1.2.3.4") { + t.Fatalf("hit %d refused too early", i) + } + } + if s.hitAllowed("1.2.3.4") { + t.Fatal("the 121st hit in a minute should be refused") + } + if !s.hitAllowed("5.6.7.8") { + t.Fatal("another address has its own allowance") + } + + r := httptest.NewRequest("GET", "/", nil) + r.RemoteAddr = "10.0.0.1:5555" + r.Header.Set("X-Forwarded-For", "6.6.6.6, 9.9.9.9") + if ip := s.realIP(r); ip != "10.0.0.1" { + t.Fatalf("without TrustProxy the connection address is used, got %s", ip) + } + s.opts.TrustProxy = true + if ip := s.realIP(r); ip != "9.9.9.9" { + t.Fatalf("the proxy's entry is the last one, got %s", ip) + } +} diff --git a/internal/sandbar/api.go b/internal/sandbar/api.go index 7d5a612..3065b03 100644 --- a/internal/sandbar/api.go +++ b/internal/sandbar/api.go @@ -25,6 +25,9 @@ func (s *Server) Handler() http.Handler { mux.HandleFunc("GET /status.json", s.statusJSON) mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) { io.WriteString(w, "ok") }) mux.HandleFunc("GET /metrics", s.metrics) + mux.HandleFunc("GET /script.js", s.serveTracker) + mux.HandleFunc("POST /api/send", s.collect) + mux.HandleFunc("OPTIONS /api/send", s.collect) mux.HandleFunc("GET /api/v1/hb/{token}", s.heartbeat) mux.HandleFunc("POST /api/v1/hb/{token}", s.heartbeat) @@ -105,6 +108,9 @@ func (s *Server) Handler() http.Handler { } return nil }, nil) + s.crud(mux, "websites", "website", true, func() any { return &Website{} }, prepareWebsite, nil) + mux.HandleFunc("GET /api/v1/websites/{id}/stats", s.auth(false, s.websiteStatsHandler)) + mux.HandleFunc("GET /api/v1/website-summary", s.auth(false, s.websiteSummary)) mux.HandleFunc("GET /api/v1/system/update", s.auth(true, s.updateStatus)) mux.HandleFunc("POST /api/v1/system/update", s.auth(true, s.updateApply)) mux.HandleFunc("POST /api/v1/channels/{id}/test", s.auth(true, s.testChannel)) @@ -349,6 +355,9 @@ func (s *Server) crud(mux *http.ServeMux, path, kind string, adminWrite bool, ne if kind == "monitor" { s.deleteMonitorData(id) } + if kind == "website" { + s.deleteWebsiteData(id) + } fail2(w, s.st.del(kind, id)) })) } diff --git a/internal/sandbar/auth.go b/internal/sandbar/auth.go index e5c5ba3..e8ad949 100644 --- a/internal/sandbar/auth.go +++ b/internal/sandbar/auth.go @@ -4,7 +4,6 @@ import ( "context" "database/sql" "errors" - "net" "net/http" "strings" "time" @@ -144,19 +143,11 @@ func (s *Server) noteFailure(ip string) { s.loginMu.Unlock() } -func clientIP(r *http.Request) string { - h, _, err := net.SplitHostPort(r.RemoteAddr) - if err != nil { - return r.RemoteAddr - } - return h -} - var errThrottled = errors.New("too many attempts, try again in a few minutes") // login checks credentials and starts a session. func (s *Server) login(w http.ResponseWriter, r *http.Request, email, password string) (*User, error) { - ip := clientIP(r) + ip := s.realIP(r) if s.throttled(ip) { return nil, errThrottled } diff --git a/internal/sandbar/bars_test.go b/internal/sandbar/bars_test.go index 9876fd4..cc1a19d 100644 --- a/internal/sandbar/bars_test.go +++ b/internal/sandbar/bars_test.go @@ -38,7 +38,7 @@ func TestUptimeBars(t *testing.T) { } // The rollup survives pruning the raw checks. - s.st.prune(clk.Now().AddDate(0, 0, 60), 30, 1) + s.st.prune(clk.Now().AddDate(0, 0, 60), 30, 1, 365) if got := s.dailyUptime(m.ID, 5, clk.Now()); got[2] != [2]int{4, 1} { t.Fatalf("rollup lost after pruning: %v", got) } diff --git a/internal/sandbar/server.go b/internal/sandbar/server.go index d246405..977ec98 100644 --- a/internal/sandbar/server.go +++ b/internal/sandbar/server.go @@ -23,8 +23,10 @@ type Options struct { MasterKey string MetricsToken string SMTP SMTPConfig - MessageDays int // how long topic messages are kept (default 30) - CheckDays int // how long monitor results are kept (default 30) + MessageDays int // how long topic messages are kept (default 30) + CheckDays int // how long monitor results are kept (default 30) + HitDays int // how long website analytics are kept (default 365) + TrustProxy bool // take the client address from X-Forwarded-For, for use behind a reverse proxy // Self-update (see update.go). Version is the running release, such as v0.2.0; anything else // ("dev") never updates. Restart is called after a new binary is installed and should close @@ -55,6 +57,10 @@ type Server struct { upd updateState + anaMu sync.Mutex // website analytics: the visitor salt and a per-address rate limit + anaSecret string + anaTry map[string][]time.Time + stop chan struct{} wg sync.WaitGroup } @@ -66,7 +72,10 @@ func New(opts Options, st *Store) *Server { if opts.CheckDays <= 0 { opts.CheckDays = 30 } - return &Server{opts: opts, st: st, hub: newHub(), running: map[string]bool{}, loginTry: map[string][]time.Time{}, stop: make(chan struct{})} + if opts.HitDays <= 0 { + opts.HitDays = 365 + } + return &Server{opts: opts, st: st, hub: newHub(), running: map[string]bool{}, loginTry: map[string][]time.Time{}, anaTry: map[string][]time.Time{}, stop: make(chan struct{})} } func (s *Server) now() time.Time { @@ -80,7 +89,7 @@ func (s *Server) now() time.Time { func (s *Server) Start() { s.every(time.Second, func() { s.runDueMonitors(s.now()) }) s.every(5*time.Second, func() { s.escalate(s.now()) }) - s.every(time.Hour, func() { s.st.prune(s.now(), s.opts.MessageDays, s.opts.CheckDays) }) + s.every(time.Hour, func() { s.st.prune(s.now(), s.opts.MessageDays, s.opts.CheckDays, s.opts.HitDays) }) } func (s *Server) Close() { diff --git a/internal/sandbar/store.go b/internal/sandbar/store.go index b55ec62..23b3201 100644 --- a/internal/sandbar/store.go +++ b/internal/sandbar/store.go @@ -48,6 +48,12 @@ CREATE INDEX IF NOT EXISTS checks_monitor ON checks(monitor_id, time); -- One row per monitor per UTC day, kept far longer than raw checks: it feeds the status page's uptime bars. CREATE TABLE IF NOT EXISTS daily ( monitor_id TEXT NOT NULL, day TEXT NOT NULL, checks INTEGER NOT NULL, fails INTEGER NOT NULL, PRIMARY KEY (monitor_id, day)); +-- Website analytics: one row per page view or event. No IP address or user agent is kept, only labels. +CREATE TABLE IF NOT EXISTS hits ( + website TEXT NOT NULL, time INTEGER NOT NULL, visitor TEXT NOT NULL, event TEXT NOT NULL, path TEXT NOT NULL, + referrer TEXT NOT NULL, browser TEXT NOT NULL, os TEXT NOT NULL, device TEXT NOT NULL, country TEXT NOT NULL, + lang TEXT NOT NULL, data TEXT NOT NULL); +CREATE INDEX IF NOT EXISTS hits_site ON hits(website, time); CREATE TABLE IF NOT EXISTS mstate ( monitor_id TEXT PRIMARY KEY, status TEXT NOT NULL, since INTEGER NOT NULL, last_check INTEGER NOT NULL, last_ms INTEGER NOT NULL, fails INTEGER NOT NULL, last_ping INTEGER NOT NULL, detail TEXT NOT NULL); @@ -246,7 +252,8 @@ func (s *Store) events(scope, ref string) []Event { } // prune deletes data past its retention. -func (s *Store) prune(now time.Time, messageDays, checkDays int) { +func (s *Store) prune(now time.Time, messageDays, checkDays, hitDays int) { + s.db.Exec(`DELETE FROM hits WHERE time < ?`, now.AddDate(0, 0, -hitDays).Unix()) s.db.Exec(`DELETE FROM messages WHERE time < ?`, now.AddDate(0, 0, -messageDays).Unix()) s.db.Exec(`DELETE FROM checks WHERE time < ?`, now.AddDate(0, 0, -checkDays).Unix()) s.db.Exec(`DELETE FROM daily WHERE day < ?`, now.UTC().AddDate(0, 0, -400).Format("2006-01-02")) diff --git a/internal/sandbar/web/index.html b/internal/sandbar/web/index.html index 818e905..adcaf35 100644 --- a/internal/sandbar/web/index.html +++ b/internal/sandbar/web/index.html @@ -227,6 +227,28 @@ .feed .empty { margin:0; } .msg { border:var(--border-w) solid var(--border); border-radius:var(--radius); padding:var(--s2) var(--s3); display:flex; flex-direction:column; gap:var(--s1); } .msg .tag { margin-right:var(--s1); } + /* website analytics */ + .stats { display:grid; grid-template-columns:repeat(auto-fit,minmax(104px,1fr)); gap:var(--s3); margin-bottom:var(--s4); } + .stat { background:var(--surface); border:var(--border-w) solid var(--border); border-radius:var(--radius-lg); padding:var(--s3) var(--s4); } + .stat .n { font-size:var(--text-xl); font-weight:600; font-variant-numeric:tabular-nums; } + .stat .l { font-size:var(--text-sm); color:var(--text-muted); } + .chart { display:flex; align-items:flex-end; gap:2px; height:180px; margin-top:var(--s3); } + .chart .col { position:relative; flex:1; min-width:0; height:100%; display:flex; align-items:flex-end; } + .chart .col:hover { background:var(--surface-2); } + .chart .v, .chart .u { position:absolute; left:0; right:0; bottom:0; border-radius:2px 2px 0 0; } + .chart .v { background:color-mix(in srgb, var(--accent) 35%, var(--surface)); } + .chart .u { background:var(--accent); } + .axis2 { display:flex; justify-content:space-between; margin-top:var(--s1); font-size:var(--text-xs); color:var(--text-faint); } + .legend { display:flex; gap:var(--s4); font-size:var(--text-sm); color:var(--text-muted); flex-wrap:wrap; } + .legend i { display:inline-block; width:10px; height:10px; border-radius:2px; margin-right:var(--s1); vertical-align:baseline; } + .cols { display:grid; grid-template-columns:repeat(auto-fit,minmax(300px,1fr)); gap:var(--s4); margin-top:var(--s4); } + .cols .card { margin:0; } + .top { width:100%; border-collapse:collapse; font-size:var(--text-sm); } + .top td { padding:var(--s1) var(--s2); } + .top td:first-child { background:linear-gradient(to right, var(--accent-soft) var(--w), transparent var(--w)); overflow-wrap:anywhere; } + .top td.num { text-align:right; white-space:nowrap; width:1%; } + .top th { text-align:right; font-weight:500; color:var(--text-muted); padding:var(--s1) var(--s2); } + .top th:first-child { text-align:left; } @@ -360,6 +382,7 @@ const ICONS = { incidents:'', oncall:'', monitors:'', + websites:'', topics:'', channels:'', status:'', @@ -420,6 +443,7 @@ const PAGES = { incidents: ['Incidents', 'Coordinate the response when something is wrong, and keep a record of what happened.'], oncall: ['On-call', 'Who is responsible right now, and who is told next when nobody answers.'], monitors: ['Monitors', 'Checks that tell you when a website, port, certificate or scheduled job stops working.'], + websites: ['Websites', 'Count visitors and page views on your sites, without cookies or personal data.'], topics: ['Topics', 'Publish and subscribe to messages with plain HTTP requests. Phones can use the ntfy app.'], channels: ['Channels', 'Email addresses and webhooks that receive every alert and incident update.'], users: ['People', 'Everyone who can sign in, and how to reach them.'], @@ -427,7 +451,7 @@ const PAGES = { }; const NAV = [ ['Respond', [['alerts', 'Alerts', 'alerts'], ['incidents', 'Incidents', 'incidents'], ['oncall', 'On-call', 'oncall']]], - ['Observe', [['monitors', 'Monitors', 'monitors'], ['topics', 'Topics', 'topics']]], + ['Observe', [['monitors', 'Monitors', 'monitors'], ['websites', 'Websites', 'websites'], ['topics', 'Topics', 'topics']]], ['Set up', [['channels', 'Channels', 'channels'], ['users', 'People', 'users']]], ]; function route() { @@ -754,6 +778,75 @@ function passwordDialog(u) { dialogForm('Set a password for ' + u.name, 'They are signed out everywhere.', [field('New password', pw, 'At least 10 characters')], 'Set password', async () => { await api('POST', `${V1}/users/${u.id}/password`, {password:pw.value}); toast('Password changed.'); }); } +// ---------- websites (analytics) ---------- +const RANGES = [['24h', 'Last 24 hours'], ['7d', 'Last 7 days'], ['30d', 'Last 30 days'], ['90d', 'Last 90 days']]; +const num = n => Number(n || 0).toLocaleString(); +function regionName(code) { try { return new Intl.DisplayNames(undefined, {type:'region'}).of(code) || code; } catch (e) { return code; } } +function languageName(code) { try { return new Intl.DisplayNames(undefined, {type:'language'}).of(code) || code; } catch (e) { return code; } } + +views.websites = async () => { + if (S.arg) return websiteDetail(S.arg); + const admin = S.me.role === 'admin'; + S.headerAction = admin ? ['Add website', () => websiteDialog()] : null; + const [sites, sum] = await Promise.all([api('GET', V1 + '/websites'), api('GET', V1 + '/website-summary')]); + if (!sites.length) return emptyState('websites', 'No websites yet', 'Add a website, then paste one line of code into its pages. Visitors and page views show up here. No cookies are set and no personal data is kept.', + admin ? h('button', {class:'btn', onclick:() => websiteDialog()}, 'Add website') : null); + return table(['Website', 'Visitors, last 24 hours', 'Page views, last 24 hours', ''], sites.map(w => { const s = sum[w.id] || {}; + return h('tr', {}, h('td', {}, h('a', {href:'#websites/' + w.id}, w.name), w.domain ? h('div', {class:'small dim'}, w.domain) : null), + h('td', {class:'num'}, num(s.visitors)), h('td', {class:'num'}, num(s.views)), + h('td', {}, h('div', {class:'actions'}, h('button', {class:'ghost', onclick:() => trackingDialog(w)}, 'Tracking code'), + admin ? h('button', {class:'ghost', onclick:() => websiteDialog(w)}, 'Edit') : null, + admin ? h('button', {class:'ghost danger', onclick:ask({title:`Delete ${w.name}?`, body:'The website and all of its visits and events are removed. This cannot be undone.', confirm:'Delete website'}, async () => { await api('DELETE', `${V1}/websites/${w.id}`); toast('Website deleted.'); refresh(); })}, 'Delete') : null))); + })); +}; +function websiteDialog(w) { + const e = !!w, f = { name: h('input', {value:w ? w.name : ''}), domain: h('input', {value:w ? w.domain : '', placeholder:'example.com'}) }; + dialogForm(e ? 'Edit ' + w.name : 'Add a website', null, [field('Name', f.name), field('Domain', f.domain, 'Optional. When set, visits from other domains are ignored, which stops others counting against your site.')], e ? 'Save changes' : 'Add website', + async () => { const saved = await api(e ? 'PUT' : 'POST', e ? `${V1}/websites/${w.id}` : V1 + '/websites', {...(w || {}), name:f.name.value, domain:f.domain.value}); toast(e ? 'Website saved.' : 'Website added.'); await refresh(); if (!e) trackingDialog(saved); }); +} +function trackingDialog(w) { + const tag = ` + It sets no cookies and stores nothing in the browser. It sends the page path, the referrer, the + language and the browser's own user agent, and it honors Do Not Track. + Options on the tag: data-hash (count #hash changes as pages), data-auto="false" (send nothing + until you call sandbar.view() yourself). Custom events: sandbar.track("signup", {plan: "pro"}). */ +(function () { + var tag = document.currentScript; + if (!tag) return; + var site = tag.getAttribute('data-website'); + if (!site) return; + var base = tag.src.replace(/\/script\.js(\?.*)?$/, ''); + var dnt = navigator.doNotTrack === '1' || window.doNotTrack === '1' || navigator.msDoNotTrack === '1'; + var withHash = tag.hasAttribute('data-hash'); + var last = ''; + var first = true; + + function send(body) { + if (dnt) return; + body.website = site; + body.hostname = location.hostname; + body.language = navigator.language || ''; + var json = JSON.stringify(body); + // text/plain keeps this a "simple" cross-origin request, so no preflight is needed + try { + if (navigator.sendBeacon && navigator.sendBeacon(base + '/api/send', new Blob([json], { type: 'text/plain' }))) return; + } catch (e) { /* fall through to fetch */ } + try { + fetch(base + '/api/send', { method: 'POST', body: json, headers: { 'Content-Type': 'text/plain' }, keepalive: true, mode: 'cors', credentials: 'omit' }); + } catch (e) { /* analytics must never break the page */ } + } + + function view() { + var path = location.pathname + (withHash ? location.hash : ''); + if (path === last) return; + last = path; + send({ path: path, referrer: first ? document.referrer : '' }); + first = false; + } + + function hook(name) { + var orig = history[name]; + if (!orig) return; + history[name] = function () { + var r = orig.apply(this, arguments); + setTimeout(view, 0); + return r; + }; + } + hook('pushState'); + hook('replaceState'); + window.addEventListener('popstate', function () { setTimeout(view, 0); }); + if (withHash) window.addEventListener('hashchange', function () { setTimeout(view, 0); }); + + window.sandbar = { + view: view, + track: function (name, data) { + if (!name) return; + send({ path: location.pathname, event: String(name).slice(0, 80), data: data }); + } + }; + + if (tag.getAttribute('data-auto') === 'false') return; + if (document.readyState === 'complete') view(); + else window.addEventListener('load', view); +})();