Docs Automation Plan — Automated Refresh After Releases

Status: Phase 1 complete · Phase 2 screenshot coverage complete: 83 automated and 86 skipped Date: 2026-09-04 Owner: Docs / Frontend

Resuming work? Jump to Current state & next steps at the bottom — it lists exactly what is done, what is next, and the workflow for adding new screenshot entries.

Goal

After each saas-frontend release, automatically:

1. Re-capture screenshots of the platform used in the docs.
2. Detect stale content — factual claims in the docs (button names, column headers, menu labels) that no longer match the live UI.
3. Keep image assets tidy — no broken references, no orphaned files.
4. Deliver everything as a pull request for human review. Nothing is published automatically.

Non-goals (for now)

• No LLM. The system detects and reports drift; it does not rewrite prose.
• No auto-merge. PR review is the editorial gate.
• Writing docs for brand-new features stays a human task — the system only reports what is missing or outdated.

Why

Two manual audits (DOCS_AUDIT_2026-07-31.md, FAQ_AUDIT_2026-08-18.md) each took days of human effort and found ~30 stale screenshots, dozens of outdated statements, and orphaned images. This system runs that audit automatically after every release.

Architecture

flowchart LR
    A[saas-frontend release] -->|repository_dispatch| B[docs-refresh workflow]
    B --> C[Playwright capture run<br/>HAR replay — no live backend]
    C --> D[screenshot-manifest.json<br/>route + steps + selector per image]
    D --> E[pixelmatch diff vs<br/>committed PNGs]
    E -->|changed| F[Auto-commit new PNGs]
    B --> G[claims-check run<br/>assert UI text matches docs claims]
    G -->|drift| H[Drift report]
    B --> I2[reference hygiene<br/>broken refs + orphans]
    F & H & I2 --> I[PR to firetail-documentation<br/>human reviews & merges]

The core pattern: screenshots and factual claims stop being static assets and become build artifacts generated from manifests — the same approach this repo already uses for API specs, tags, findings and dynamic variables (scripts/*/download-*.js + build-*.js).

Phases

Phase 1 — Reference hygiene ✅ (implemented)

Static analysis, zero risk, pure win.

• scripts/images/check-image-refs.js:
  • Parses all docs/**/*.md (and releases/, api/) for image references (<img src="..."> and ![](...)).
  • Fails if any referenced image file does not exist (broken reference).
  • Reports image files under docs/**/images/ never referenced by any page (orphans). Use --fail-on-orphans to make orphans fail CI too.
• Run locally: yarn check:images
• Wired into CI (pull.yaml) so every PR is checked.

Phase 2 — Screenshot manifest + capture runner ✅ (complete)

Make every doc screenshot reproducible from a recipe.

Implemented (pilot, 2026-08-21):

• screenshots/manifest.json + screenshots/manifest.schema.json in this repo — capture recipes with shared defaults (viewport 1440×900 @2x, maxDiffPixelRatio, fixedTime, orgId) and per-entry route, steps, capture, mask.
• Capture runner in saas-frontend/e2e/docs-screenshots/ (own Playwright config + capture.spec.ts), reusing the e2e auth setup and harReplay.ts.
• npm scripts in saas-frontend:
  • npm run docs:screenshots — replay mode (deterministic, hermetic; CI-safe)
  • npm run docs:screenshots:record — record HARs against a real env, then sanitize credentials out of them
• HAR fixtures committed in saas-frontend under e2e/docs-screenshots/fixtures/<entry-id>.har — not in this repo. Two reasons: the recordings contain internal API data (org/user identifiers, usage numbers) that must not live in the public docs repo even after credential sanitization, and a HAR records API response shapes that track app versions, so it belongs with the app code (next to the existing e2e HAR fixtures) and is re-recorded by frontend developers. The manifest (docs-owned: which screenshots exist, where they land) stays here; the fixtures (app-owned: what the API returned) live with the runner.
• Pixel diff via pixelmatch; PNG overwritten only above maxDiffPixelRatio.
• Run summary written to saas-frontend/e2e/docs-screenshots/capture-report.md (feeds the future docs-refresh PR body).
• Determinism verified: repeated replay runs produce byte-identical PNGs (unchanged in the report) for the two pilot entries (workforce-platforms-list, workforce-platform).

Scaled out — workforce section complete (2026-08-21):

• 13 of 14 docs/workforce/images/*.png are now manifest-managed: platforms list + drawer, employee drawer + edit form, group drawer + edit form, device drawer + edit form, application drawer, Create Policy form, scoped Create Policy form (opened from an employee's Policies tab, showing the read-only "Scope set to"), provided-guardrail detail modal, Platform Rules tab (clipped to tabs + table). All recorded, captured, and verified deterministic (4+ replay runs → byte-identical PNGs, all unchanged).
• Deliberately not manifest-managed:
  • ai-policy-precedence.png — an illustration/diagram of policy evaluation logic, not an app screenshot; cannot be captured from the UI.
• Step/capture features added during review: within (scopes a step's locator to a role/name container, e.g. a drawer, to disambiguate from the page behind), capture.target: "clip" (viewport-relative rectangle for partial-page shots), and capture.target: "role" for dialog-only screenshots.

Workload screenshots (2026-09-02):

• All 9 docs/workload/images/*.png are now manifest-managed: model details, agent details, prompt messages, the services listing, Available Integrations, and the four AWS Bedrock service-detail views. Each has a sanitized HAR fixture and passed hermetic replay. The service-detail routes use a 30-day dateTime query range to stay within the sandbox plan's 90-day retention window.

Dashboard screenshots (2026-09-02):

• 21 dashboard screenshots are now manifest-managed: workforce risk, workforce usage, workload risk, workload usage, Logs by Service, Logs by Risk, Finding Volume by Status, Finding Volume by Severity, Top 10 AI Models by risk score, Resources Grouped by Provider, Logs Grouped by Provider, Log Volume by Log Type, Log Volume by Region, Input Tokens by Provider, Output Tokens by Provider, Total Tokens by Model, Total Tokens, Total Tokens by Project, Token Consumption, Top Consumers, and Usage Grouped by Platforms. Each has a sanitized HAR fixture and passed hermetic replay. The dashboard routes use a 30-day dateTime query range so the fixed capture clock stays within the sandbox plan's 90-day retention window.
• The remaining dashboard and logging screenshots are now also manifest-managed: Top Platforms by Employee Activity, the chart download example, Audit Logs, Audit Log details, and Employee Log. The Audit Logs capture currently reflects the sandbox's empty-state listing.

Alerts, FAQ download and integrations catalogue screenshots (2026-09-04):

• 5 further images are manifest-managed: the Static Alert form and the delete-alert confirmation (docs/alerts/), the Findings and Events Download Data dialogs (docs/faqs/), and the Settings > Integrations > Available catalogue (docs/integrations/). Each has a sanitized HAR fixture and passed hermetic replay twice with byte-identical PNGs.
• The two alert entries depend on the sandbox alert AI Static Alerting; the delete entry opens the confirmation popconfirm but never confirms it.

Current screenshot inventory (2026-09-04):

• 83 screenshot entries are currently manifest-managed: 13 workforce, 9 workload, 23 dashboard, 5 logging, 1 model-scanning, 2 alerts, 2 FAQ download dialogs, 12 integrations, 3 projects, 3 organizations, 2 getting-started and 8 posture-management entries.
• All 83 manifest outputs exist on disk and pass hermetic replay. The docs contain 169 images under docs/**/images/; 86 intentionally skipped images are third-party flows, generated pages, logos, or illustrations. No images remain in planned or needs-triage.
• Skipped images include third-party console/browser setup screenshots, Auth0 and Stripe flows, generated report pages, logos, and illustrations. Each skip has a reason in screenshots/image-inventory.json; ai-policy-precedence.png is intentionally excluded because it is a diagram, not a UI screenshot.
• Image-reference hygiene currently reports no broken references in published pages and no orphaned images. Four draft browser-extension pages reference missing Edge, Firefox, Opera, and Safari screenshots.

Lessons learned in the pilot:

• page.clock.setFixedTime freezes requestAnimationFrame, hanging Ant Design drawer animations and page.screenshot — use page.clock.install({ time }) instead (clock starts fixed but keeps ticking).
• animations: 'disabled' on screenshots must be avoided for the same reason.
• Cursor-paginated endpoints send the page cursor in the request body, which the HAR match key ignores — harReplay.ts now replays duplicate-key responses in recorded order (repeating the last), otherwise the app refetches page 1 forever.
• A networkidle wait after the steps is needed in record mode so lazy panel requests are fully recorded rather than captured as aborted; the same wait is needed after the capture (before the context closes) or trailing requests record as aborted too.
• Requests the app itself cancels (RTK Query aborts) are recorded with status -1 and can never complete; harReplay.ts leaves them pending on replay (an instant abort beats the app's own cancellation and pops the global "Network Error" modal into the screenshot — which then silently overwrites the good PNG). The runner now also refuses to capture any frame showing the app's error dialog, failing the entry instead.
• waitForLoadState('networkidle') is not enough to guarantee complete recordings — it only needs a 500 ms lull and times out on slow endpoints, leaving unfinished (status -1) HAR entries that replay as permanent spinners. The runner now tracks in-flight API requests explicitly and waits until all have settled (plus a quiet period) before capturing and before closing a recording context, and refuses to capture frames with visible spinners/skeletons.
• Prefer deep routes (e.g. .../policies/platform-rules) over clicking through tabs: intermediate tab mounts fire extra requests non-deterministically, producing racy HAR misses.
• The HAR shape-key fallback (query minus filter params) keeps listings populated when a filter param drifts, but it can serve an unfiltered recording to a filtered request — putting data in a screenshot that does not belong to the entity on screen. It logs [har] no exact query match … falling back but does not fail. Treat any occurrence as a re-record signal: grep the run output for it. The suite currently triggers it zero times.
• Beautifier rules must be idempotent. fallbackMac originally recognised only 02:00:00:00:00:xx as already-masked while generating 02:00:00:xx:xx:xx, so every re-run re-hashed masked MACs and rewrote 15 untouched fixtures. docIp had the same latent bug for the RFC 5737 ranges. Both now short-circuit on their own output.
• A record run must never write the PNG. It runs against the live app, so its screenshot contains unsanitized data, and the subsequent replay only overwrites the file when the diff exceeds maxDiffPixelRatio — a name or an email is far below 0.1% of the pixels, so real data silently survived into the committed image. The runner now stops after the guards in record mode.
• docs:screenshots:sanitize must run both scrubbers: sanitize-har.mjs (auth headers and bearer tokens) and then beautify-docs-fixtures.mjs (PII and scruffy names). The npm script previously ran only the latter, so hand-recorded fixtures kept live JWTs. The beautifier's "review leftovers: jwt-like token" line is the signal that the first pass was skipped.
• Ant Design dropdowns opened while their modal is still animating land a pixel or two off, and the offset is not stable between runs. Insert a wait step after the dialog appears and before opening the select; stretching the post-step settle does not fix it.
• The browser clock must sit near that entry's own HAR recording date, not a global constant. defaults.fixedTime is 2026-08-21; fixtures recorded on 2026-09-04 rendered every timestamp in the future ("Created in 14 days"). Entries now carry an optional fixedTime override — set it whenever you record on a date far from the default.
• maxDiffPixelRatio (0.001) is too coarse to catch text. A changed name, email or relative date is a few hundred pixels out of ~5M, so replay reports unchanged and keeps the stale PNG. Delete the output file to force a rewrite after any change that only affects text. This also weakens Phase 2's drift detection and should be revisited with cross-environment evidence before tightening the default.
• Dashboard routes need an explicit 30-day dateTime query (?dateTime=%7B%22value%22%3A2592000%7D). The default range is 90 days and the sandbox plan's retention is exactly 90 days, so recording fails with "Bad Request — Start date is older than your plans retention period".

Projects and organizations screenshots (2026-09-04):

• 6 further images are manifest-managed: the Create Project and Edit Project forms and the project detail header (docs/projects/), and the Members table, Edit Team Member role selector and the all-organizations grid (docs/organizations/).
• Filling forms needed new step actions in the runner: fill (role/name + value, with optional nth for repeated key/value rows), press (a single keyboard key, e.g. Enter to commit an Ant Design tag chip) and wait (ms). nth also works on clickRole/waitForRole, which is how the Members table's per-row edit buttons are addressed — they all share the accessible name edit.
• Two pieces of drift surfaced immediately: project details is now a drawer rather than a page, the Create Project form no longer has the Propagate Custom Tags / Propagate Owners toggles, and the member role list offers Project Admin where the docs still said Application Admin.

Getting-started and posture-management screenshots (2026-09-04):

• 6 further images are manifest-managed: the Findings page and the Workforce Risk dashboard (docs/getting-started/), and the Events listing, Provided Policies grid, the empty Create Resource Policy drawer and its Add check menu (docs/posture-management/).
• docs/posture-management/images/event-details.png was pulled from this batch: the Events page has neither a working free-text search for event types nor a filter, so a RESOURCE_POLICY:CREATED event cannot be reached deterministically. Events are deep-linkable (/platform/events/<uuid>), so the entry becomes trivial once a stable sandbox event of that type is identified.

Integration forms and suggested checks (2026-09-04):

• 11 further images are manifest-managed: the SMTP, ServiceNow, Email, Jira, Lambda, HTTP Webhook, AWS Bedrock Invoke, AWS Bedrock Logging, OpenAI Discovery and PagerDuty forms (docs/integrations/), plus the Resource Policy Suggested Checks picker (docs/posture-management/). All replay from sanitized HAR fixtures.
• The current catalog labels differ from the historic docs: ServiceNow Issue, Jira Issue, Lambda Function, PagerDuty Issue, and HTTP Webhook (HMAC Signed). The latter replaces the documented generic HTTP webhook name.
• The populated-check screenshot remains planned. Suggested-policy rows do not enable the picker confirmation in the sandbox under replay, so it cannot be captured without a stable selectable suggested-policy fixture.

Remaining: add the docs-refresh workflow — see Current state & next steps.

Phase 3 — Claims manifests (stale-content detection)

Turn fragile factual claims in the docs into machine-checkable assertions.

• claims/<doc-slug>.json per high-churn page:

  {
    "doc": "docs/workforce/workforce-platforms.md",
    "assertions": [
      { "route": ".../platforms", "role": "columnheader",
        "names": ["Name", "Capabilities", "Employees", "Logs"] },
      { "route": ".../platforms", "role": "button", "name": "Download" }
    ]
  }

• A Playwright run asserts these against the released frontend (using the same HAR replay fixtures as Phase 2, so no live backend is needed) and emits a markdown drift report: "workforce-platforms.md claims column 'Risk Level'; UI now shows 'Risk Score'".

• Assertions use getByRole-style semantic selectors, matching the saas-frontend e2e conventions.

Phase 4 — Expand generated content

Anything that is a list of facts should be rendered by Eleventy from downloaded data, like findings/specs/tags already are. Candidates from the audits:

• Export-modal attribute lists (findings, events, models, prompts)
• Supported platform/browser lists
• Plan feature tables

Once generated, this content can never go stale.

Trigger & delivery

• The saas-frontend release workflow fires repository_dispatch on this repo (fallback: nightly cron).
• The docs-refresh workflow runs: capture → diff → claims check → reference hygiene → opens a PR containing changed PNGs + a markdown drift report.
• Never auto-merge. A human reviews and approves.

What stays manual

• Deciding a new feature needs a new doc/screenshot.
• Deleting docs for removed features.
• Rewriting prose flagged by the drift report.

The system detects all three (manifest coverage vs. app routes, failing claims, orphaned images) and surfaces them in the PR report.

Future: LLM slot

When/if an LLM is added, it slots between drift report → PR: given the precise, grounded drift findings, it drafts the prose updates for human review. All the manifest/diff infrastructure built here is exactly the grounding it needs — nothing is throwaway.

Rollout estimate

Non-technical summary

flowchart TB
    A["🚀 New product release goes live"] --> B["🤖 Docs Checker starts automatically"]
    B --> C["📸 Re-takes every screenshot<br/>by clicking through the product,<br/>exactly like a user would"]
    B --> D["🔍 Reads the docs and checks<br/>every fact against the product<br/>(button names, menus, columns...)"]
    B --> E["🧹 Tidy-up check:<br/>finds broken or unused images"]
    C --> F{"Did anything change?"}
    D --> F
    E --> F
    F -- "No" --> G["✅ Docs are up to date.<br/>Nothing to do."]
    F -- "Yes" --> H["📦 Prepares an update package:<br/>• fresh screenshots<br/>• list of outdated sentences<br/>• list of images to remove"]
    H --> I["🧑‍💻 A person reviews the package<br/>and approves with one click"]
    I --> J["📚 Documentation website<br/>is updated"]

Current state & next steps

Everything needed to resume this work in a fresh session.

Repos & key files

Environment prerequisites (record mode only)

• saas-frontend dev server running (npm start, serves https://localhost:3000)
• e2e/.env with BASE_URL=https://localhost:3000 and TEST_USER_EMAIL
• AWS SSO authenticated (ftauth) — the auth setup fetches the test user password from SSM; a still-valid token in e2e/.playwright/.auth/user.json skips login entirely
• Sandbox org 94520fbd-7863-465b-bc77-70038e014aea (manifest defaults.orgId)
• Replay mode (npm run docs:screenshots) needs none of the above except the dev server (or any deployed frontend via BASE_URL)

Workflow: adding a new screenshot entry

1. Find the doc image and its context: which page/state does it show? (grep -rn "<image>.png" firetail-documentation/docs/)
2. Explore the live UI with playwright-cli -s=firetail-sandbox-authenticated-session (or npm run e2e:auth) to find the route, step selectors (role/name), and the dialog's accessible name.
3. Add an entry to screenshots/manifest.json. Prefer deep routes over tab clicking; use within to scope steps inside drawers; pick capture.target: viewport (full page), role (a dialog only), or clip (rectangle).
4. Record: npm run docs:screenshots:record (records HAR + captures + sanitizes). To re-record a single entry: UPDATE_HAR=1 npx playwright test --config=e2e/docs-screenshots/playwright.config.ts -g "<entry-id>" && npm run docs:screenshots:sanitize A record run never writes the PNG — it only proves the recipe resolves and produces the fixture.
5. Verify: run npm run docs:screenshots twice — everything must pass and all PNG hashes must be identical (shasum docs/**/images/*.png). If a change only affects text, delete the output PNG first — the diff threshold treats it as unchanged.
6. Visually review the changed PNGs on the rendered docs page (yarn dev in the docs repo) — data, framing, crop.
7. Check the HAR has no unfinished entries: node -e "console.log(JSON.parse(require('fs').readFileSync('e2e/docs-screenshots/fixtures/<id>.har','utf8')).log.entries.filter(e=>e.response.status<0).length)" (should print 0).

Image inventory workflow

• screenshots/image-inventory.json lists every image under docs/**/images/, its documentation references, and one of four statuses: automated, planned, skipped, or needs-triage.
• automationId and references are derived from the screenshot manifest and Markdown files. Reviewers maintain planned decisions with required capture notes and skipped decisions with required reasons; manifest outputs are always marked automated.
• Run yarn build:image-inventory after adding, removing, or reclassifying an image. CI runs yarn check:image-inventory and fails when the inventory is stale, incomplete, or points to a missing manifest output.

Done so far (2026-09-04)

• Phase 1 — image hygiene check wired into docs build + pull.yaml CI. Current warning: 4 broken refs in draft browser-extension docs. Published pages have no broken references, and there are no orphaned images.
• Phase 2 infra — manifest/schema, runner, record/replay modes, pixel diff, capture report, sanitizer; all hardening lessons are in the "Lessons learned" list above (clock, animations, pagination, pending-request semantics, in-flight API tracking, spinner/error-dialog guards).
• Phase 2 workforce — 13/14 images manifest-managed (all except ai-policy-precedence.png, a hand-drawn diagram). All 13 entries are verified deterministic. Sandbox data the steps depend on: employee charles.miller@tropical.com, group sales team, device DESKTOP-7804TS5, application Claude Desktop, platform ChatGPT, guardrail Personal PII Blocking — if the sandbox org loses these records, re-pick targets and re-record.
• Phase 2 workload — all 9 images manifest-managed and deterministic. Sandbox data: model Claude Sonnet 4, agent Langchain Agent Python, prompt langchain_agent_python/main.py:84-88, and service AWS Bedrock. The four service-detail routes use a 30-day query range to remain within the sandbox plan's 90-day retention window.
• Phase 2 dashboard and logging — 23 dashboard and 3 logging images are manifest-managed and deterministic.
• Phase 2 alerts, FAQ downloads and integrations catalogue — 5 further images are manifest-managed and deterministic, bringing the total to 53 entries.
• Phase 2 projects and organizations — 6 further images are manifest-managed and deterministic, bringing the total to 59 entries. Sandbox data the steps depend on: project test-new-project (94057967-a46a-4ef6-9c2d-e9b37de0ba5d, beautified to customer-portal) and the second row of Settings > Members. The whole suite (60 tests) was run twice end to end with byte-identical PNGs and no updated rows.
• Phase 2 getting-started and posture-management — 6 further images are manifest-managed and deterministic, bringing the total to 65 entries. The whole suite (66 tests) was run twice end to end with byte-identical PNGs and no updated rows. Sandbox data the steps depend on: the sandbox must have at least one custom resource policy so the Create Resource Policy button is present, and the Events listing must be non-empty.
• Phase 2 integration forms and suggested checks — 11 further images are manifest-managed and pass hermetic replay, bringing the total to 76 entries: 10 notification/AI setup forms and the Resource Policy Suggested Checks picker. The product’s current labels are recorded in the manifest selectors.
• Phase 2 final coverage — 7 further images are manifest-managed and pass hermetic replay, completing all planned FireTail UI captures. This includes notification history with a successful response, Cloud Log input/output metadata, Model Scanning history, populated Resource Policy checks, Event details, and Resource Policy history. The latter two use stable deep links; Event details uses AI_WORKFORCE_USER_CONSUMPTION:DISCOVERED and Resource Policy history uses a preselected matching-resource history entry.
• Fixture drift resolved (2026-09-04) — the employee drawer now scopes its consumption search by userUUID, which the workforce-employee, workforce-employee-form and workforce-policy-scoped fixtures predated. All three were re-recorded and the whole suite (54 tests) passes. workforce-employee.png changed as a result: the Consumptions tab is now empty, because charles.miller@tropical.com genuinely has no consumptions. The previous screenshot's populated table was org-wide data substituted by the shape-key fallback — it never belonged to that employee. Open decision: only three sandbox employees have consumption data and all three are personas for real staff accounts, so targeting one would put a real colleague's address in this public manifest. Either keep the accurate empty state or seed consumption data for a fictional employee (charles.miller@tropical.com) in the sandbox.

Next steps (in order)

1. Docs-refresh GitHub Actions workflow (lives in saas-frontend — it has the runner, fixtures and app build; triggered by release + nightly cron): checkout both repos → build/serve frontend → npm run docs:screenshots (replay) → if PNGs changed, open a PR against firetail-documentation with the changed images + capture-report.md as the PR body. Never auto-merge.
2. Phase 3 — claims manifests (see section above); reuse the same runner infra and HAR fixtures.
3. Phase 4 — move fact-lists into the Eleventy generated-content pipeline.

Branch/PR state (as of 2026-09-04)

• saas-frontend: current branch is FIRE-4033-navigation-system; the docs screenshot runner, fixtures, HAR replay changes, npm scripts, and dependencies are present in the working repository.
• firetail-documentation: current branch is FIRE-4037-docs-automation; Phase 1, the manifest, and refreshed screenshots are present here.
• The re-captured PNGs replace hand-taken originals (different data/framing) — they need a visual editorial review before merging.