Build Assessment — PartnerHub v3
The headline finding is the gap between those two verdicts: the quality stack (page judge + functional verify) passed a build the owner rejects. The judge grades source code against PRD text; it cannot see that the app loses all data on restart, ignored the mandated tech stack, and reads as shallow in a real browser. Closing that measurement gap is worth more than any single fix below.
Critical defects
- No persistence.
core.py: STORE = InMemoryStore()— every restart/deploy/auto-upgrade wipes all business data; SAMPLE_DATA reseeds demo rows so nobody notices. Disqualifying for a live business app. - Mandated stack ignored. The idea required Flask + jinja2 + htmx; delivered is stdlib ThreadingHTTPServer with hand-rolled routing/sessions. Root cause is in the factory, not the model: the staged-build scaffold prompt hardcodes the stdlib contract and the page prompt says "Only stdlib", silently overriding customer directives on every big build.
- Judge leniency. Missing CSRF was noted on 3+ money-mutating pages yet they scored 93–96; a queue with no pagination/search scored 95. Findings must gate, not decorate.
- Interaction depth. htmx ~absent (0–5 attrs/page), drag-and-drop is mouse-only with a single drop handler, document "previews" are anchors, uploads store filenames only.
- Possibly broken feature. Payout Schedule's
action_generatehandler was flagged truncated/incomplete by the judge itself — and still passed.
Page-by-page: judge score vs live reality
| Page | Route | Judge | Live crawl | Top gaps |
|---|---|---|---|---|
| Dashboard / Nav Shell | /dashboard | 76 | 17.5KB · 7 charts · 0 htmx | Flat sidebar, no breadcrumbs; weakest judge score in the build |
| Partner List | /partners | 94 | 18KB · 13 rows · 5 htmx | Search chip remove doesn't re-query live; sort state fragile |
| Partner Profile | /partners/:id | 94 | drill-down OK | Notes are delete-only (no edit); Resources card mis-mapped |
| Add / Edit Partner | /partners/new | 89 | 9 inputs, validation OK | No client-side duplicate pre-check; toast quirk |
| Application Intake | /apply | 88 | 23KB · 14 inputs | 5MB limit UI-only (not enforced server-side); documents stored as filenames only — no real upload |
| Applications Queue | /applications | 95 | 1 table · 12 rows · 0 htmx | NO pagination or search — judge still scored 95 (leniency example) |
| Application Review | /applications/:id/review | 93 | actions POST OK | Doc 'preview' links are #anchors, not previews; no CSRF on approve/reject |
| Bulk Import | /partners/import | 91 | wizard present | Duplicate check only within file, not vs existing data; limits asserted in copy only |
| Deal Board | /deals | 93 | kanban cols · 14 draggable · 1 drop handler | Drag is mouse-only (no keyboard/a11y); card position never persisted properly |
| Register Deal | /deals/new | 96 | 7 inputs OK | NO CSRF on POST; duplicate detection name-only exact match |
| Deal Detail | /deals/:id | 94 | timeline OK | Stage transitions unconstrained (any→any); owner reassignment unvalidated |
| Commission Rules | /commission-rules | 90 | 2 forms · 3 htmx | Preview matches approved rules only; timeline bodies truncated |
| Payout Schedule | /payouts/schedule | 92 | 4 rows | action_generate handler truncated/INCOMPLETE (possibly broken feature); no re-generate guard |
| Payout Statements | /payouts/statements | 93 | 11 rows · 11 forms | Manual recalc only; no filter/search by partner or period |
| Payout Approvals | /payouts/approvals | 95 | 13 forms · 14 buttons | NO CSRF; webhook has no retry/idempotency — one failure loses the event |
What worked — keep these
Mandate coverage gate delivered 15/15 pages (v2 shipped 8). Parallel staged build + janitor survived 5 interruptions and still went live end-to-end. Functional verify passed 6/8 eval cases. Infra automation (VLAN VM, NAT, public DNS, Let's Encrypt) worked unattended. Styling is at least consistent (Inter, indigo palette) and every page carries sample data, empty states, and an audit-trail helper. 6,616 LOC across 17 files.
Delta plan — make the next E2E better
The app stores EVERYTHING in memory (core.py STORE = InMemoryStore()); any service restart, deploy, or unattended-upgrade wipes all business data, and SAMPLE_DATA quietly reseeds demo rows so the loss is invisible. Fix: (a) live-mode builds require a durable store — sqlite file minimum, postgres when flagged; (b) add a RESTART-SURVIVAL smoke to the assembly gate: create a row → restart the process → assert the row is still there. That one check makes in-memory stores impossible to ship.
The idea explicitly mandated Flask + jinja2 + htmx; the staged-build scaffold prompt hardcodes 'ThreadingHTTPServer + ROUTES dict' and the page prompt says 'Only stdlib' — the system prompt silently overrides the customer directive for every ≥5-page build. Fix: parameterize the scaffold contract by the lang directive (a Flask variant of _SCAFFOLD_SYSTEM/_PAGE_SYSTEM), and add an assembly-gate check: when a framework is mandated, `import flask` must exist and htmx must actually be served/used.
The page judge reads code against PRD text — it passed this build 92 mean while the owner called it bad. Blind spots to add: framework/persistence mandate compliance, security (it NOTED missing CSRF on 3+ pages yet scored them 93-96 — make CSRF a gating deduction), pagination presence on list pages, and a rendered-page pass (headless screenshot of each page scored for visual quality/consistency instead of trusting design_match from source).
Every page carries its own inline <style>; consistency currently depends on the model re-typing the same CSS. Move the chrome/typography/palette into one static stylesheet emitted by the scaffold, and have the judge diff pages against it.
12-13 seeded rows per entity makes every list page trivially shallow and hides pagination gaps. Seed configurable volume (200+ rows on medium builds) so lists must implement paging/search to stay usable — and the judge sees the real behavior.
Put token issue/verify helpers in core.py's contract so page modules get CSRF for free, and gate on it in scoring.
Manual cleanup checklist (for the hand-fix pass on the live app)
- Replace InMemoryStore with a sqlite-backed store keeping the same STORE API (list/get/create/update/delete/count/audit) — schema is derivable from current usage
- Add CSRF tokens to every POST form (register deal, approvals, review actions first — they mutate money/state)
- Add pagination + search to /applications (queue is unbounded)
- Fix Payout Schedule action_generate — judge flagged the handler as truncated/incomplete; verify and finish it
- Real file handling on intake: enforce the 5MB limit server-side; store content, not just filenames; make review 'preview' links actually open the document
- Deal board: persist card position; add keyboard alternative for drag
- Constrain deal stage transitions (define the legal lifecycle); validate owner reassignment
- Bulk import: check duplicates against existing partners, not just within the uploaded file
- Webhook on payout approvals: add retry + idempotency key
- Notes: allow edit, not just delete; fix Resources card mapping on Partner Profile
When the cleanup lands, diff it against this page — that delta is the ground truth for the next round of factory prompt/gate changes.