# SEO (settings, content pages, redirects, sitemap, health check)

The public site (landing page, documentation, status page, blog and landing pages written
by staff) carries search-engine metadata that marketing manages from the admin panel
without a deploy: site-wide settings, per-route `<head>` overrides, Markdown content pages,
redirects, the sitemap and `robots.txt`, plus an in-process health check that crawls the
public pages and scores them.

- Admin API: `/v1/admin/seo/*` (below). Code: `internal/api/ext_seo.go`; state and
  rendering helpers: `internal/seo`.
- Permissions (`docs/ROLES.md`): reads need `seo.read` (superadmin, marketing, viewer);
  writes need `seo.write` (superadmin, marketing). The raw `head_html` setting can only be
  changed by a superadmin. Sales, ops, finance and support get `403` on every SEO route.
- State: `DATA_DIR/seo.json` (settings, routes, pages, redirects) and
  `DATA_DIR/seo-check.json` (last health report). Writes are atomic (temp file + rename)
  and recorded in the audit log like every other admin call.
- Public routes rendered from this state (served by the core server): `/blog/` (index of
  published posts), `/blog/<slug>/` (a post), `/<slug>/` (a landing page), `/sitemap.xml`,
  `/robots.txt`, staff redirects on any path, and `<head>` injection on `/` and `/docs/`.
- Configuration: `PUBLIC_URL` is the origin the health check crawls and the fallback for
  absolute URLs until `canonical_host` is set; `WEB_DIR` locates the docs
  (`<WEB_DIR>/../docs`) that the sitemap and the check include.

All responses are JSON; errors are `{"error":{"code","message"}}` (`invalid`, `not_found`,
`conflict`, `forbidden`). Timestamps are RFC 3339 UTC.

## Settings

`GET /v1/admin/seo/settings` → `{"settings": {...}}`;
`PATCH /v1/admin/seo/settings` with any subset of the fields → the full settings.

| Field | Meaning | Validation |
|---|---|---|
| `site_name` | `og:site_name`, Organization schema | 1–80 chars |
| `title_suffix` | appended to page titles, e.g. `" · BitCall"` (leading space kept) | ≤ 80 |
| `default_title`, `default_description` | used by `/` and any route without an override | ≤ 120 / ≤ 300 |
| `canonical_host` | origin of every canonical URL and the sitemap, e.g. `https://bitcall.stream` | `https://host`, no path, no trailing slash |
| `og_image` | default social image | absolute URL or `/path` |
| `twitter_handle` | `twitter:site` | starts with `@`, or empty |
| `locale` | `og:locale` | like `en_IN` |
| `google_site_verification`, `bing_verification` | verification meta tags | plain tokens |
| `ga4_measurement_id` | enables the gtag snippet | `^G-[A-Z0-9]+$`, or empty |
| `plausible_domain` | enables the Plausible snippet instead of / besides GA4 | bare host |
| `noindex_paths` | prefixes served with `noindex, nofollow` and listed as `Disallow` in robots.txt | absolute paths; default `/admin/`, `/portal/`, `/demo/` |
| `organization_jsonld` | overrides the generated Organization schema | valid JSON |
| `head_html` | raw tags appended to every `<head>` **unescaped** | ≤ 20 KB; **superadmin only** (`403 forbidden` for anyone else; echoing the unchanged value back is allowed) |

`updated_at` / `updated_by` are set by the server (the staff id, or `admin` for the admin
key) and ignored in the request body.

## Routes (built-in pages)

Overrides for the `<head>` of pages the server itself renders: `/`, `/docs/`, `/status/`,
`/blog/` by default. Each override may set `title` (the suffix is appended),
`description`, `og_image`, `canonical`, `noindex`, and the sitemap `priority` (0–1) and
`changefreq` (`always|hourly|daily|weekly|monthly|yearly|never`).

- `GET /v1/admin/seo/routes` → `{"routes": [...]}`
- `PUT /v1/admin/seo/routes` body `{"path": "/pricing/", "title": "Pricing", ...}` upserts
  by path (`201` when created, `200` when updated; fields left out keep their value).
  `path` must start with `/`.
- `DELETE /v1/admin/seo/routes?path=/pricing/` → `204` (`404` when there is no override).
  Deleting the last override restores the four defaults.

A route with `noindex: true` (or under a `noindex_paths` prefix) is served with a noindex
tag and left out of the sitemap.

## Pages: blog posts and landing pages

Pages are written in Markdown (GitHub-flavoured, heading ids, tables, raw HTML allowed —
only staff write them) and rendered by the server with the same `<head>` machinery as the
built-in routes, plus `Article` (blog) and `FAQPage` (when `faq` is set) structured data.

| Section | URL | Use |
|---|---|---|
| `blog` | `/blog/<slug>/` | posts: product updates, integration guides; listed on `/blog/`, `og:type` article with published/modified times and author |
| `page` | `/<slug>/` | landing pages: "calling API for astrology apps", comparisons, campaign pages |

Fields: `title` (≤ 120, warns above 60), `section`, `slug` (`^[a-z0-9-]{1,80}$`, default
`Slugify(title)`), `description` (≤ 200, warns above 160 or when empty), `body` (Markdown,
≤ 200 KB), and optional `h1`, `excerpt`, `keywords`, `tags`, `author`, `og_image`,
`canonical`, `noindex`, `faq` (`[{question, answer}]`), `published`.

Landing-page slugs that collide with a built-in path are refused (`400`): `admin`,
`portal`, `demo`, `status`, `docs`, `sdk`, `blog`, `v1`, `healthz`, `metrics`, `og.png`,
`styles.css`, `robots.txt`, `sitemap.xml`. A slug must be unique within its section
(`409 conflict`).

- `GET /v1/admin/seo/pages?section=blog|page&q=&status=published|draft` →
  `{"pages": [...], "counts": {"published","draft","blog","page"}, "total"}`; rows omit
  `body` and add `word_count` and `path`; `q` matches title, slug, description, tags and
  keywords; newest update first.
- `POST /v1/admin/seo/pages` → `201 {"page", "path", "warnings": [...]}`. A page created
  with `published: true` gets `published_at` immediately.
- `GET /v1/admin/seo/pages/{id}` → `{"page", "path", "html", "head", "word_count",
  "warnings"}` where `html` is the rendered body and `head` is what the public page emits
  (`title`, `description`, `canonical`, `og_image`, `og_type`, `noindex`, `jsonld`,
  `published`, `modified`, `author`, and `html` — the rendered tags).
- `PATCH /v1/admin/seo/pages/{id}` with the same fields (slug changes allowed; `409` if
  taken). The first `published: true` sets `published_at`; later unpublish/republish keeps
  it.
- `DELETE /v1/admin/seo/pages/{id}` → `204`.
- `POST /v1/admin/seo/pages/{id}/publish` and `/unpublish` → `{"page", "path"}`
  (idempotent).
- `POST /v1/admin/seo/preview` body `{"body": "...markdown...", "title"?}` →
  `{"html", "title", "word_count"}` for the editor's live preview.

Drafts and `noindex` pages are rendered with `noindex, nofollow` when previewed on the
public URL and never appear in `/blog/` or the sitemap. `warnings` are advisory (title and
description length, empty description, fewer than 300 words); they never block a save.

## Redirects

`GET /v1/admin/seo/redirects` → `{"redirects": [{from, to, code, hits, created_at,
created_by}]}`; `POST /v1/admin/seo/redirects` body `{"from": "/old", "to": "/new",
"code": 301}` → `201 {"redirect"}`; `DELETE /v1/admin/seo/redirects?from=/old` → `204`.

- `from` is an exact path starting with `/`, or a prefix when it ends with `*`
  (`/old-docs/*`). `to` is a path or an absolute URL. `code` is `301` (default) or `302`.
- Rejected with `400`: `from == to`, a prefix that matches its own target, and loops — the
  target is followed through the existing redirects (up to 16 hops) and must not come
  back to `from`. A second redirect from the same path is `409`.
- The public router applies redirects before any other route and counts `hits`.

## Sitemap and robots.txt

`GET /v1/admin/seo/sitemap` → `{"entries": [{loc, lastmod, changefreq, priority}],
"count", "xml"}` — exactly what the public `/sitemap.xml` serves (the public handler
calls `Server.SEOSitemapEntries()`), built from:

1. route overrides, in order, skipping `noindex` ones and those under a `noindex_paths`
   prefix (a route whose `canonical` points elsewhere is skipped too);
2. published pages that are not `noindex` (`lastmod` = `updated_at`; priority 0.7 for
   landing pages, 0.6 for posts, `changefreq` monthly);
3. documentation pages: every `*.md` under `<WEB_DIR>/../docs` except `index.md` and
   `NEXT-CONTRACT.md`, as `<host>/docs/<relative path>` with the file's mtime (skipped
   entirely when `/docs/` is noindex).

`<host>` is `canonical_host`, or `PUBLIC_URL` until one is configured. Entries are
de-duplicated by `loc`.

`GET /v1/admin/seo/robots` → `{"text"}`: `Allow: /`, a `Disallow` per `noindex_paths`,
`Disallow: /v1/` and the `Sitemap:` line when `canonical_host` is set. The public
`/robots.txt` serves the same text.

## Health check

`POST /v1/admin/seo/check` crawls the public site over real HTTP against `PUBLIC_URL` and
returns `{"report"}`; `GET /v1/admin/seo/check` returns `{"report", "running"}` with the
last report (persisted in `DATA_DIR/seo-check.json`, `report: null` before the first
run). A second `POST` while one runs is `409`.

What it fetches (max 60 URLs, 5 s timeout each, 4 in parallel, redirects not followed):
`/`, `/docs/`, `/blog/`, every published page, `/status/`, then every docs `*.md` with
`Accept: text/html` (the documentation viewer). HTML is parsed with
`golang.org/x/net/html`. Afterwards up to 100 distinct internal links found on those
pages (same host as `PUBLIC_URL` or `canonical_host`) are fetched once each.

Per page: `status`, `response_ms`, `title` / `title_length`, `description` /
`description_length`, `h1_count`, `canonical` / `canonical_present` /
`canonical_matches` (equals `<canonical_host or PUBLIC_URL><path>`), `noindex` (meta tag
or `X-Robots-Tag`), `og_image`, `word_count` (visible text, scripts and styles excluded),
`internal_links`, `links_not_ok` (`[{url, status, error}]` for anything not `200`),
`images_missing_alt` (images without an `alt` attribute; empty `alt` is fine) and
`issues`.

Issue types and their score weights: `fetch_error` 25, `bad_status` 20, `not_html` 10,
`missing_title` 10, `missing_description` 8, `missing_h1` 6, `canonical_mismatch` 6,
`broken_link` 6 (a link with an error or status ≥ 400), `redirect` 5,
`missing_canonical` 4, `multiple_h1` 3, `noindex` 3, `missing_og_image` 3,
`thin_content` 3 (under 150 words), `title_too_long` 2 (> 60), `description_too_long` 2
(> 160), `image_alt_missing` 2, `slow_response` 2 (> 1.5 s), `title_too_short` 1
(< 10), `description_too_short` 1 (< 50).

`summary` holds `pages`, `pages_ok`, `links_checked`, `links_broken`, `issues` (count per
type), `issue_count` and `score`: `100 − (sum of weights ÷ pages)`, clamped to 0–100. A
clean site scores 100; treat anything under 80 as something to fix before a campaign.

## How the public pages render

- `/` and `/docs/`: the static landing page and the documentation viewer are served as
  before; the server injects the `<head>` built from settings + the route override
  (`title`, description, robots, canonical, Open Graph, Twitter card, verification tags,
  Organization JSON-LD, analytics snippets, `head_html`).
- `/blog/`: index of published posts, newest first, with excerpts and tags.
- `/blog/<slug>/` and `/<slug>/`: the page's Markdown rendered inside the site template,
  `h1` (or the title), author and dates for posts, FAQ section when `faq` is set; drafts
  render with `noindex` so staff can preview them by URL.
- `/sitemap.xml`, `/robots.txt`: as described above.
- Redirects are checked first on every request; paths under `noindex_paths` get the
  noindex meta tag and `X-Robots-Tag: noindex`.