seedproject-web/.memory/qa-agent.md

The QA agent — automated site testing with a self-healing loop

Status: Implemented · Date: 2026-07-24 Related: webdesigner.md · ../agents/console/FRAMEWORK.md · ../.claude/skills/qa/SKILL.md · ../agents/console/SETUP.md

Framework note (2026-07-24): the console runner was refactored into a generic host + pluggable agent modules. QA now lives in agents/console/agents/qa.mjs (exports manifest

• register(host, settings)), is enabled via the astroagent.config.json roster, and reads heartbeatMin/autofix from agents.settings.qa. Its logic is unchanged from below; see FRAMEWORK.md.

---

What it is

The QA agent tests the live site (carlosarias.co) — links, images, the contact form, the API, SEO/meta, and accessibility — records findings in the admin, and auto-fixes the safe ones by feeding the Web Designer queue. It runs on a heartbeat (hourly) and on demand from /admin/qa.

The honest architecture. The confined agent (claude -p, no Bash/WebFetch) cannot make HTTP requests or drive a browser, and no headless browser is usable by the runner. Since the site is static (Astro → baked HTML), that's fine: a deterministic Node crawler inside the console runner does the actual testing (Node 22 global fetch, zero deps), findings are stored via PHP CLI + DB (the runner has no DB driver), and a thin LLM step writes a plain-English summary using the qa skill. The crawl is the source of truth; the LLM only triages.

The closed loop

heartbeat (hourly)  ──►  runQa (crawl)  ──►  cja_qa_findings
                                              │
                                              ├─ safe findings (alt, internal links)
                                              │     └─ qa-autofix.php → cja_tasks (queued)
                                              │            └─ Web Designer fixes → publishes
                                              │                   └─ self-logs to /changelog
                                              │                          └─ next QA run: green
                                              └─ judgment calls (external links, etc.)
                                                     └─ shown in /admin/qa with an
                                                        instructions box → you dispatch a fix

The site went 14 warnings → 0 through this loop (sitemap, résumé portrait alt, blog cover alt auto-fixed; GitHub link removed by hand).

What the crawler checks

check_type	Verifies	Severity
page	every route returns 200 + HTML	error
link	every internal <a href> resolves	error
image	every <img>/og:image/icon loads	error
external	off-site links reachable	warning (bot-blocking is common)
form	/api/contact/submit alive (honeypot probe → 200) + rejects bad input (→ 422)	error/warning
health	/api/health → 200 & db:"connected" (read at data.db — API wraps in {ok,data,error})	error
seo	each page has <title> + description; canonical host = carlosarias.co; no duplicate titles	warning (error on wrong canonical)
sitemap	every route is in sitemap.xml	warning
a11y	images have alt — empty/bare alt is flagged too (valid only for decorative images)	warning

Route set = static list + DB slugs (qa-routes.php) + sitemap + discovered internal links. HTML parsed with regex (clean static HTML). Each finding carries a templated fix_hint.

Auto-fix (safe findings)

• Runs after every QA pass when QA_AUTOFIX ≠ 0 (default on). qa-autofix.php queues a Web Designer cja_tasks row for each open finding in a safe category — ['a11y', 'link'] (see SAFE in that file) — then the runner kicks drainTasks().
• Dedup: skips a finding if a same-title task is queued/running or was created in the last 6 hours — so hourly heartbeats never spam or loop on the same issue.
• Judgment calls (external, image, seo, sitemap, form, health, page) are not auto-fixed — they wait in /admin/qa for a human decision.

Report + instructions (the human path)

/admin/qa lists runs (counts, summary) and the latest run's findings grouped by severity. Each open finding has:

• Create fix task → queues a Web Designer fix (uses the fix_hint).
• an instructions input → type how to handle it ("remove it", "use https://…", "reword") and that becomes the Web Designer's brief instead of the default hint. This is where decisions like a wrong external URL get made — in the admin, not in chat.
• Ignore → mark the finding handled.

Heartbeat

In-process timer in server.mjs (the runner is always up, so no cron needed):

• QA_HEARTBEAT_MIN (default 60, 0 disables) — set in the systemd unit.
• One pulse ~90s after each restart; then every N minutes.
• Triage is gated to save tokens: a scheduled run with 0 errors gets a cheap templated summary (no LLM call); only manual runs or runs with errors get an LLM summary.

Storage & pruning

• cja_qa_runs (status, trigger manual/scheduled, summary, counts JSON, timestamps)
• cja_qa_findings (run_id, check_type, severity, url, detail, fix_hint, status open/fix_queued/ignored) — migration api/db/migrations/013_create_cja_qa.sql.
• qa-finish.php prunes to the 50 most recent runs (hourly runs accumulate).

Files

File	Role
agents/console/server.mjs	runQa, qaTriage, runQaFlow, POST /devconsole/qa/run, heartbeat, QA_AUTOFIX/QA_HEARTBEAT_MIN
api/db/migrations/013_create_cja_qa.sql	runs + findings tables
api/cli/qa-start.php	open a run → run_id
api/cli/qa-finish.php	store findings (temp JSON file) + prune
api/cli/qa-routes.php	DB-driven slugs for the crawl
api/cli/qa-autofix.php	queue Web Designer fixes for safe findings (6h dedup)
api/public/controllers/adminqa.php	runs / findings / fix (optional instructions) / ignore
app/src/pages/admin/qa.astro	report page + instructions box + "Run QA now"
.claude/skills/qa/SKILL.md	what QA checks, severity model, gotchas

Operating it

• Run now: the "Run QA now" button, or curl -s -X POST http://127.0.0.1:3011/devconsole/qa/run.
• Change cadence / disable heartbeat: QA_HEARTBEAT_MIN in the unit (0 = off), then systemctl daemon-reload && systemctl restart astroagent-console.service.
• Disable auto-fix: QA_AUTOFIX=0 in the unit env.
• View: /admin/qa.

Safety & gotchas

• Read-only against the live site — no git, no build, no writes. The two contact-form probes write nothing (the honeypot company field is filled on purpose; the validation probe is rejected with 422). Verified: a full run leaves cja_contact untouched.
• Fixes only ever go through the Web Designer queue — build-gated, scoped commit, git-revertable, self-logged. Auto-fix ≠ unsafe; a bad fix is one git revert away.
• External links = warnings, never errors (403/429/timeouts are usually bot-blocking; only a clear 404/410/DNS failure is flagged).
• Empty alt IS a finding. alt=""/bare alt is valid only for decorative images; a content image (photo, cover, screenshot) needs descriptive alt. (An earlier version wrongly treated empty alt as a pass — corrected.)
• Health check reads data.db — the API wraps every response in {ok, data, error}.
• Site-coupling: QA_BASE, the static route list, and carlosarias.co are hard-coded in runQa — same site-agnosticism gap noted for the Web Designer; parameterise from config for a clean clone.
• Timezone: the server (and thus the runner's timestamps) is America/New_York as of 2026-07-24 (was UTC). New Web Designer changelog stamps are Eastern.