# Alerting and public status page

The signaling server runs a small monitor (`internal/alerting`) that probes its own
dependencies every `ALERT_INTERVAL`, turns state changes into alerts (Slack, generic
webhook, e-mail), keeps a 24 h history for the public status page and lets you declare
incidents by hand.

- Status page: **`<PUBLIC_URL>/status/`** (e.g. <http://localhost:8080/status/>)
- JSON: `GET /v1/status`, `GET /v1/status/history?hours=24`
- Admin: `GET /v1/admin/alerts`, `POST /v1/admin/alerts/test`, `GET|POST /v1/admin/incidents`, `PATCH /v1/admin/incidents/{id}`

No extra process is needed: the monitor starts with the API server.

## Environment variables

| Variable | Default | Meaning |
|----------|---------|---------|
| `ALERT_INTERVAL` | `30s` | Probe period. Go duration (`30s`, `2m`) or seconds. Also the sample resolution of the 24 h history. |
| `ALERT_COOLDOWN` | `15m` | A component that *stays* down/degraded is re-alerted at most this often. |
| `ALERT_MAX_PEERS` | `5000` | `hub_load` is **degraded** at 80 % of this and **down** at or above it. |
| `ALERT_MIN_DISK_MB` | `500` | `disk` is **down** below this many MB free under `DATA_DIR`, **degraded** below 2×. |
| `ALERT_SLACK_WEBHOOK` | – | Slack incoming-webhook URL. |
| `ALERT_WEBHOOK_URL` | – | Any HTTPS endpoint; receives the JSON payload below. |
| `ALERT_EMAIL_TO` | – | Comma-separated recipients. Requires `SMTP_HOST`. |
| `SMTP_HOST`, `SMTP_PORT` | –, `587` | Port 465 = implicit TLS, 587 = STARTTLS required, others = STARTTLS when offered. |
| `SMTP_USER`, `SMTP_PASS` | – | PLAIN auth when `SMTP_USER` is set. |
| `SMTP_FROM` | `SMTP_USER` | Sender address. |

Probes reuse the existing settings: `ANALYTICS_URL`, `TURN_PUBLIC_HOST`, `TURN_PORT`, `TURN_SECRET`,
`DATA_DIR`, `PUBLIC_URL` (linked from Slack messages and included in webhook payloads).

State is persisted to `$DATA_DIR/alerting.json` (last 200 alerts, incidents, 24 h history) so a
restart does not blank the status page.

## Probes

Every round runs the probes concurrently (each bounded by a 3 s timeout) and produces one result per
component: `{component, status, latency_ms, message, checked_at}` with `status` one of
`ok | degraded | down | unconfigured`.

| Component | What it checks | degraded | down | unconfigured |
|-----------|----------------|----------|------|--------------|
| `api_self` | The API process itself; message carries uptime. | – | – | – |
| `analytics` | `GET $ANALYTICS_URL/healthz` | 3xx/4xx, or > 1.5 s | connection error, timeout, 5xx | `ANALYTICS_URL` empty |
| `turn` | UDP STUN binding request **and** a real TURN allocation on `TURN_PUBLIC_HOST:TURN_PORT`, authenticated with a credential signed by `TURN_SECRET` (user `statusprobe`). | STUN answers but the allocation fails (wrong secret, relay range exhausted) or > 1.5 s | no answer / timeout | `TURN_PUBLIC_HOST` empty |
| `hub_load` | Connected peers from the signaling hub vs `ALERT_MAX_PEERS`. | ≥ 80 % | ≥ 100 % | – |
| `disk` | Free space on the filesystem holding `DATA_DIR`. | < 2 × `ALERT_MIN_DISK_MB` | < `ALERT_MIN_DISK_MB` | statfs unavailable (Windows) |

`overall` on `/v1/status` is `outage` if any component is down, `degraded` if any is degraded **or an
incident is open**, otherwise `operational`. `unconfigured` components never affect it and are not
counted in uptime percentages.

Note on the event pipeline: the analytics emitter drops events when its queue is full and logs
`event queue full, dropping event`. That counter is not exposed by the emitter, so it is not a probe;
watch the log line (or `/metrics`) for it.

## Alert rules

Per component, between rounds:

| From | To | Alert |
|------|----|-------|
| ok / unconfigured / first run | down or degraded | `down` / `degraded` immediately |
| degraded | down (or down → degraded) | immediately (severity change) |
| down / degraded | same status | repeat only after `ALERT_COOLDOWN` |
| down / degraded | ok / unconfigured | `recovered` immediately |

Alerts are delivered synchronously to every configured notifier with a 10 s timeout each; the outcome
per notifier is stored on the alert (`delivered: {"slack":"ok","webhook":"502 Bad Gateway"}`) and
visible in `GET /v1/admin/alerts`.

## Payloads

**Generic webhook** (`ALERT_WEBHOOK_URL`), `POST`, `Content-Type: application/json`:

```json
{
  "type": "alert",
  "service": "callingserver",
  "public_url": "https://api.example.com",
  "sent_at": "2026-10-27T10:15:30Z",
  "alert": {
    "id": "alt_1761560130_7",
    "kind": "down",
    "component": "turn",
    "status": "down",
    "message": "turn allocate: timeout after 3s",
    "latency_ms": 3001,
    "at": "2026-10-27T10:15:30Z"
  }
}
```

`kind` is `down`, `degraded`, `recovered` or `test`. A 2xx response marks delivery as ok; anything
else (or a transport error) is recorded on the alert and logged.

**Slack** (`ALERT_SLACK_WEBHOOK`): `{"text": "[callingserver] DOWN: turn is down (…)", "attachments": [{color, text, fields[Component, Status, Latency], title_link → /status/}]}`.
Red for down, amber for degraded, green for recovered, blue for test.

**E-mail**: plain text, subject equal to the Slack `text` line, body with message, component, status,
kind, latency, timestamp and alert id.

**Test**: `POST /v1/admin/alerts/test {"message":"ping"}` (header `X-Admin-Key`) sends a `kind: test`
alert through every notifier and answers `{"succeeded":["slack"],"failed":{"email":"auth: 535 …"},"notifiers":[…],"alert":{…}}`.

## Incidents

Incidents are written by you, not by probes, and show on the status page while open and for 7 days
after resolution.

```bash
curl -X POST $API/v1/admin/incidents -H "X-Admin-Key: $ADMIN_API_KEY" -H 'Content-Type: application/json' \
  -d '{"title":"TURN latency in eu-west","status":"investigating","message":"Relay node is saturated; adding capacity.","components":["turn"]}'
# -> 201 {"incident":{"id":"inc_…","status":"investigating",…}}
curl -X PATCH $API/v1/admin/incidents/inc_… -H "X-Admin-Key: $ADMIN_API_KEY" -H 'Content-Type: application/json' \
  -d '{"status":"resolved","message":"Second relay online."}'
curl $API/v1/admin/incidents -H "X-Admin-Key: $ADMIN_API_KEY"
```

`status` is one of `investigating | identified | monitoring | resolved`. `PATCH` accepts any subset of
`title`, `status`, `message`, `components`. An open incident degrades `overall`.

## Public API

`GET /v1/status` (no auth, `Cache-Control: no-store`):

```json
{
  "overall": "operational",
  "components": [
    {"component":"api_self","status":"ok","latency_ms":0,"message":"up 3h12m0s","checked_at":"…"},
    {"component":"analytics","status":"ok","latency_ms":4,"message":"healthz 200 OK","checked_at":"…"},
    {"component":"turn","status":"ok","latency_ms":12,"message":"relay 127.0.0.1:52345","checked_at":"…"},
    {"component":"hub_load","status":"ok","message":"2 rooms, 4 peers (limit 5000)","checked_at":"…"},
    {"component":"disk","status":"ok","message":"120404 MB free of 460000 MB (min 500 MB)","checked_at":"…"}
  ],
  "incidents": [],
  "uptime_seconds": 11520,
  "generated_at": "…"
}
```

`GET /v1/status/history?hours=24` (1–24) returns `{"hours", "interval_seconds", "samples",
"components": {"turn": {"samples": [{"at","status"}…], "uptime_pct": 99.93}, …}}`. The ring holds
24 h at `ALERT_INTERVAL` resolution (2 880 samples at the default 30 s); the page buckets it into
15-minute bars, worst status wins per bucket.

## Using Prometheus / Grafana instead

The built-in monitor is deliberately small. If you already run Prometheus:

1. Scrape `GET /metrics` on the API (`calling_active_rooms`, `calling_connected_peers`,
   `calling_uptime_seconds`) and the analytics service, and use `blackbox_exporter` for the same
   checks the probes do: `http_2xx` on `$ANALYTICS_URL/healthz`, and a UDP/STUN check on the relay
   (coturn exposes its own Prometheus endpoint with `prometheus` in `coturn.conf`).
2. Alert in Alertmanager (`up == 0`, `calling_connected_peers > 4000`, `node_filesystem_avail_bytes < 500e6`)
   and route to Slack/PagerDuty there. Leave `ALERT_SLACK_WEBHOOK` / `ALERT_WEBHOOK_URL` unset so you
   are not paged twice.
3. Keep the status page: `/v1/status` and `/status/` still work with no notifier configured. If you
   prefer Grafana's public dashboards, point a panel at `/v1/status/history`
   (JSON API data source) or simply link the dashboard from the incident message.
4. Alternatively point `ALERT_WEBHOOK_URL` at an Alertmanager-compatible receiver or at a tiny
   relay that converts the payload above into whatever your on-call tool expects.

## Files

- `internal/alerting/` – monitor, probes, notifiers, env parsing, persistence, tests
- `internal/api/ext_status.go` – routes (registered through the extension hook)
- `web/status/index.html` – the public page (served at `/status/`)