seedproject-web/.memory/qa-agent.md

# The QA agent — automated site testing with a self-healing loop

**Status:** Implemented · **Date:** 2026-07-24
**Related:** [webdesigner.md](webdesigner.md) · [../agents/console/FRAMEWORK.md](../agents/console/FRAMEWORK.md) · [../.claude/skills/qa/SKILL.md](../.claude/skills/qa/SKILL.md) · [../agents/console/SETUP.md](../agents/console/SETUP.md)

> **Framework note (2026-07-24):** the console runner was refactored into a generic host +
> pluggable agent modules. QA now lives in `agents/console/agents/qa.mjs` (exports `manifest`
> + `register(host, settings)`), is enabled via the `astroagent.config.json` roster, and reads
> `heartbeatMin`/`autofix` from `agents.settings.qa`. Its logic is unchanged from below; see
> [FRAMEWORK.md](../agents/console/FRAMEWORK.md).

---

## What it is

The QA agent tests the live site (carlosarias.co) — links, images, the contact form, the
API, SEO/meta, and accessibility — records findings in the admin, and **auto-fixes the safe
ones** by feeding the [Web Designer](webdesigner.md) queue. It runs on a **heartbeat** (hourly)
and on demand from `/admin/qa`.

**The honest architecture.** The confined agent (`claude -p`, no Bash/WebFetch) *cannot make
HTTP requests or drive a browser*, and no headless browser is usable by the runner. Since the
site is **static** (Astro → baked HTML), that's fine: a **deterministic Node crawler inside
the console runner** does the actual testing (Node 22 global `fetch`, zero deps), findings are
stored via **PHP CLI + DB** (the runner has no DB driver), and a **thin LLM step** writes a
plain-English summary using the `qa` skill. The crawl is the source of truth; the LLM only
triages.

## The closed loop

```
heartbeat (hourly)  ──►  runQa (crawl)  ──►  cja_qa_findings
                                              │
                                              ├─ safe findings (alt, internal links)
                                              │     └─ qa-autofix.php → cja_tasks (queued)
                                              │            └─ Web Designer fixes → publishes
                                              │                   └─ self-logs to /changelog
                                              │                          └─ next QA run: green
                                              └─ judgment calls (external links, etc.)
                                                     └─ shown in /admin/qa with an
                                                        instructions box → you dispatch a fix
```

The site went 14 warnings → 0 through this loop (sitemap, résumé portrait alt, blog cover alt
auto-fixed; GitHub link removed by hand).

## What the crawler checks

| check_type | Verifies | Severity |
|---|---|---|
| `page` | every route returns 200 + HTML | error |
| `link` | every internal `<a href>` resolves | error |
| `image` | every `<img>`/og:image/icon loads | error |
| `external` | off-site links reachable | warning (bot-blocking is common) |
| `form` | `/api/contact/submit` alive (honeypot probe → 200) + rejects bad input (→ 422) | error/warning |
| `health` | `/api/health` → 200 & `db:"connected"` (read at `data.db` — API wraps in `{ok,data,error}`) | error |
| `seo` | each page has `<title>` + description; canonical host = carlosarias.co; no duplicate titles | warning (error on wrong canonical) |
| `sitemap` | every route is in `sitemap.xml` | warning |
| `a11y` | images have alt — **empty/bare `alt` is flagged too** (valid only for decorative images) | warning |

Route set = static list + DB slugs (`qa-routes.php`) + sitemap + discovered internal links.
HTML parsed with regex (clean static HTML). Each finding carries a templated `fix_hint`.

## Auto-fix (safe findings)

- Runs after every QA pass when `QA_AUTOFIX` ≠ `0` (default **on**). `qa-autofix.php` queues a
  Web Designer `cja_tasks` row for each **open** finding in a **safe category** —
  `['a11y', 'link']` (see `SAFE` in that file) — then the runner kicks `drainTasks()`.
- **Dedup:** skips a finding if a same-title task is queued/running *or was created in the last
  6 hours* — so hourly heartbeats never spam or loop on the same issue.
- Judgment calls (`external`, `image`, `seo`, `sitemap`, `form`, `health`, `page`) are **not**
  auto-fixed — they wait in `/admin/qa` for a human decision.

## Report + instructions (the human path)

`/admin/qa` lists runs (counts, summary) and the latest run's findings grouped by severity.
Each open finding has:
- **Create fix task** → queues a Web Designer fix (uses the `fix_hint`).
- an **instructions input** → type how to handle it ("remove it", "use https://…", "reword")
  and that becomes the Web Designer's brief instead of the default hint. This is where
  decisions like a wrong external URL get made — in the admin, not in chat.
- **Ignore** → mark the finding handled.

## Heartbeat

In-process timer in `server.mjs` (the runner is always up, so no cron needed):
- `QA_HEARTBEAT_MIN` (default **60**, `0` disables) — set in the systemd unit.
- One pulse ~90s after each restart; then every N minutes.
- **Triage is gated** to save tokens: a scheduled run with 0 errors gets a cheap templated
  summary (no LLM call); only manual runs or runs with errors get an LLM summary.

## Storage & pruning

- `cja_qa_runs` (status, trigger manual/scheduled, summary, counts JSON, timestamps)
- `cja_qa_findings` (run_id, check_type, severity, url, detail, fix_hint, status
  open/fix_queued/ignored) — migration `api/db/migrations/013_create_cja_qa.sql`.
- `qa-finish.php` prunes to the **50 most recent runs** (hourly runs accumulate).

## Files

| File | Role |
|---|---|
| `agents/console/server.mjs` | `runQa`, `qaTriage`, `runQaFlow`, `POST /devconsole/qa/run`, heartbeat, `QA_AUTOFIX`/`QA_HEARTBEAT_MIN` |
| `api/db/migrations/013_create_cja_qa.sql` | runs + findings tables |
| `api/cli/qa-start.php` | open a run → run_id |
| `api/cli/qa-finish.php` | store findings (temp JSON file) + prune |
| `api/cli/qa-routes.php` | DB-driven slugs for the crawl |
| `api/cli/qa-autofix.php` | queue Web Designer fixes for safe findings (6h dedup) |
| `api/public/controllers/adminqa.php` | runs / findings / fix (optional instructions) / ignore |
| `app/src/pages/admin/qa.astro` | report page + instructions box + "Run QA now" |
| `.claude/skills/qa/SKILL.md` | what QA checks, severity model, gotchas |

## Operating it

- **Run now:** the "Run QA now" button, or `curl -s -X POST http://127.0.0.1:3011/devconsole/qa/run`.
- **Change cadence / disable heartbeat:** `QA_HEARTBEAT_MIN` in the unit (`0` = off), then
  `systemctl daemon-reload && systemctl restart astroagent-console.service`.
- **Disable auto-fix:** `QA_AUTOFIX=0` in the unit env.
- **View:** `/admin/qa`.

## Safety & gotchas

- **Read-only against the live site** — no git, no build, no writes. The two contact-form
  probes write nothing (the honeypot `company` field is filled on purpose; the validation
  probe is rejected with 422). Verified: a full run leaves `cja_contact` untouched.
- **Fixes only ever go through the Web Designer queue** — build-gated, scoped commit,
  git-revertable, self-logged. Auto-fix ≠ unsafe; a bad fix is one `git revert` away.
- **External links = warnings, never errors** (403/429/timeouts are usually bot-blocking; only
  a clear 404/410/DNS failure is flagged).
- **Empty alt IS a finding.** `alt=""`/bare `alt` is valid only for *decorative* images; a
  content image (photo, cover, screenshot) needs descriptive alt. (An earlier version wrongly
  treated empty alt as a pass — corrected.)
- **Health check reads `data.db`** — the API wraps every response in `{ok, data, error}`.
- **Site-coupling:** `QA_BASE`, the static route list, and `carlosarias.co` are hard-coded in
  `runQa` — same site-agnosticism gap noted for the Web Designer; parameterise from config for
  a clean clone.
- **Timezone:** the server (and thus the runner's timestamps) is `America/New_York` as of
  2026-07-24 (was UTC). New Web Designer changelog stamps are Eastern.