carlosarias/seedproject-web
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
seedproject-web/.memory/qa-agent.md

7.5 KiB
Raw Blame History

The QA agent — automated site testing with a self-healing loop

Status: Implemented · Date: 2026-07-24 Related: webdesigner.md · ../.claude/skills/qa/SKILL.md · ../agents/console/SETUP.md · ../agents/console/PLAN.md

What it is

The QA agent tests the live site (carlosarias.co) — links, images, the contact form, the API, SEO/meta, and accessibility — records findings in the admin, and auto-fixes the safe ones by feeding the Web Designer queue. It runs on a heartbeat (hourly) and on demand from /admin/qa.

The honest architecture. The confined agent (claude -p, no Bash/WebFetch) cannot make HTTP requests or drive a browser, and no headless browser is usable by the runner. Since the site is static (Astro → baked HTML), that's fine: a deterministic Node crawler inside the console runner does the actual testing (Node 22 global fetch, zero deps), findings are stored via PHP CLI + DB (the runner has no DB driver), and a thin LLM step writes a plain-English summary using the qa skill. The crawl is the source of truth; the LLM only triages.

The closed loop

heartbeat (hourly)  ──►  runQa (crawl)  ──►  cja_qa_findings
                                              │
                                              ├─ safe findings (alt, internal links)
                                              │     └─ qa-autofix.php → cja_tasks (queued)
                                              │            └─ Web Designer fixes → publishes
                                              │                   └─ self-logs to /changelog
                                              │                          └─ next QA run: green
                                              └─ judgment calls (external links, etc.)
                                                     └─ shown in /admin/qa with an
                                                        instructions box → you dispatch a fix

The site went 14 warnings → 0 through this loop (sitemap, résumé portrait alt, blog cover alt auto-fixed; GitHub link removed by hand).

What the crawler checks

check_type Verifies Severity
page every route returns 200 + HTML error
link every internal <a href> resolves error
image every <img>/og:image/icon loads error
external off-site links reachable warning (bot-blocking is common)
form /api/contact/submit alive (honeypot probe → 200) + rejects bad input (→ 422) error/warning
health /api/health → 200 & db:"connected" (read at data.db — API wraps in {ok,data,error}) error
seo each page has <title> + description; canonical host = carlosarias.co; no duplicate titles warning (error on wrong canonical)
sitemap every route is in sitemap.xml warning
a11y images have alt — empty/bare alt is flagged too (valid only for decorative images) warning

Route set = static list + DB slugs (qa-routes.php) + sitemap + discovered internal links. HTML parsed with regex (clean static HTML). Each finding carries a templated fix_hint.

Auto-fix (safe findings)

  • Runs after every QA pass when QA_AUTOFIX ≠ 0 (default on). qa-autofix.php queues a Web Designer cja_tasks row for each open finding in a safe category — ['a11y', 'link'] (see SAFE in that file) — then the runner kicks drainTasks().
  • Dedup: skips a finding if a same-title task is queued/running or was created in the last 6 hours — so hourly heartbeats never spam or loop on the same issue.
  • Judgment calls (external, image, seo, sitemap, form, health, page) are not auto-fixed — they wait in /admin/qa for a human decision.

Report + instructions (the human path)

/admin/qa lists runs (counts, summary) and the latest run's findings grouped by severity. Each open finding has:

  • Create fix task → queues a Web Designer fix (uses the fix_hint).
  • an instructions input → type how to handle it ("remove it", "use https://…", "reword") and that becomes the Web Designer's brief instead of the default hint. This is where decisions like a wrong external URL get made — in the admin, not in chat.
  • Ignore → mark the finding handled.

Heartbeat

In-process timer in server.mjs (the runner is always up, so no cron needed):

  • QA_HEARTBEAT_MIN (default 60, 0 disables) — set in the systemd unit.
  • One pulse ~90s after each restart; then every N minutes.
  • Triage is gated to save tokens: a scheduled run with 0 errors gets a cheap templated summary (no LLM call); only manual runs or runs with errors get an LLM summary.

Storage & pruning

  • cja_qa_runs (status, trigger manual/scheduled, summary, counts JSON, timestamps)
  • cja_qa_findings (run_id, check_type, severity, url, detail, fix_hint, status open/fix_queued/ignored) — migration api/db/migrations/013_create_cja_qa.sql.
  • qa-finish.php prunes to the 50 most recent runs (hourly runs accumulate).

Files

File Role
agents/console/server.mjs runQa, qaTriage, runQaFlow, POST /devconsole/qa/run, heartbeat, QA_AUTOFIX/QA_HEARTBEAT_MIN
api/db/migrations/013_create_cja_qa.sql runs + findings tables
api/cli/qa-start.php open a run → run_id
api/cli/qa-finish.php store findings (temp JSON file) + prune
api/cli/qa-routes.php DB-driven slugs for the crawl
api/cli/qa-autofix.php queue Web Designer fixes for safe findings (6h dedup)
api/public/controllers/adminqa.php runs / findings / fix (optional instructions) / ignore
app/src/pages/admin/qa.astro report page + instructions box + "Run QA now"
.claude/skills/qa/SKILL.md what QA checks, severity model, gotchas

Operating it

  • Run now: the "Run QA now" button, or curl -s -X POST http://127.0.0.1:3011/devconsole/qa/run.
  • Change cadence / disable heartbeat: QA_HEARTBEAT_MIN in the unit (0 = off), then systemctl daemon-reload && systemctl restart astroagent-console.service.
  • Disable auto-fix: QA_AUTOFIX=0 in the unit env.
  • View: /admin/qa.

Safety & gotchas

  • Read-only against the live site — no git, no build, no writes. The two contact-form probes write nothing (the honeypot company field is filled on purpose; the validation probe is rejected with 422). Verified: a full run leaves cja_contact untouched.
  • Fixes only ever go through the Web Designer queue — build-gated, scoped commit, git-revertable, self-logged. Auto-fix ≠ unsafe; a bad fix is one git revert away.
  • External links = warnings, never errors (403/429/timeouts are usually bot-blocking; only a clear 404/410/DNS failure is flagged).
  • Empty alt IS a finding. alt=""/bare alt is valid only for decorative images; a content image (photo, cover, screenshot) needs descriptive alt. (An earlier version wrongly treated empty alt as a pass — corrected.)
  • Health check reads data.db — the API wraps every response in {ok, data, error}.
  • Site-coupling: QA_BASE, the static route list, and carlosarias.co are hard-coded in runQa — same site-agnosticism gap noted for the Web Designer; parameterise from config for a clean clone.
  • Timezone: the server (and thus the runner's timestamps) is America/New_York as of 2026-07-24 (was UTC). New Web Designer changelog stamps are Eastern.