Pipeline Steps

The factory's full run, step by step, grounded in the best run to date. Every step has a stable number — reference it as “step 14” in questions and improvement ideas. Each card states what happens, the gate, what the reference run showed, what is working, and the next improvement.

The step numbers below are stable references and were NOT renumbered. A run executes them in six phases: Planning 1 → 2 → 3 → 4 → 5 → 6 gets everything ready (spec intake, PRD, definition, acceptance contract, approved ADP) and ships it to Design 7; only then does Infrastructure 11 Provision → 19 Bootstrap host → 13 Publish (holding page) → 20 Infrastructure checkpoint run, so no VM is spent on a plan or design that is not approved;App Build 8 → 9 → 10 then binds the frozen checkpoint; Deploy 12 (drift-checked) lands an EMPTY application and 21 Load sample data — optional: skip it on the control page or launch with sample_data=false — seeds it; Test 14 → 15 → 16 → 17 then proves it, separately from deployment (since 2026-09-08). Pipeline infra stops after step 20 (stage infra_ready); pipeline app binds such a checkpoint (infra_from) and runs planning, design, build, deploy and test with no provisioning; pipeline deploy parks after step 21 (stage deployed, untested) and pipelinetest runs only the Test phase against a deployed run. A failed build, deploy or test phase is re-launched onto the same checkpoint.

Reference run

The run is created from an idea plus an explicit stack (language, store, scope, auth, nav). The stack contract is logged first and binds every later prompt, the build gate and provisioning — the model cannot substitute another stack.

Gate: Stack contract event; enrichment depth resolved (default one round).

Related: Factory · Software Guide

Reference run: flask · sqlite · standard · shared · single; depth=one. Logged before the PRD.

Working: Explicit stack ended the era of surprise in-memory stores; runtime-verified later (step 9).

Improve: Depth shows 'none' in the contract line even when the effective depth is one — two lines say different things; unify.

The spec given to the factory is taken in verbatim and becomes the ground truth for the PRD.

Gate: None — never blocks.

Related: Factory

Reference run: Ran in seconds; spec accepted.

Working: Deterministic and free — the spec itself is the ground truth, not a model's paraphrase of it.

Improve: Nothing checks the spec for gaps; a thin spec becomes a thin PRD.

Page inventory with functionality per page. The matched domain playbook soft-mandates its required pages and injects cross-cutting depth features; the coverage gate diffs the PRD against every mandate and force-restores anything dropped.

Gate: PRD coverage gate (mandated + playbook-required pages present).

Related: Feature Specs · Spec Composer

Reference run: 11 pages; playbook vendor_contracts; all 5 required pages present.

Working: Playbook matching is accurate since the whole-word-regex fix; the gate has repeatedly caught silently dropped pages.

Improve: Playbook choice is keyword-driven; a wrong match still steers the whole PRD. Log the runner-up and its score for auditability.

The project is defined from the PRD: requirements with acceptance criteria, observations, processes, agents, integrations, and the evaluation suite whose cases verification will later execute.

Gate: Definition completeness (feeds the ADP check).

Related: Projects

Reference run: 10 requirements, 3 observations, 1 process, 1 agent, 1 integration, 1 eval suite (10 EC cases).

Working: The eval suite created here is what made verification honest — criteria written before any code exists.

Improve: Requirement statements are still prose; acceptance criteria vary in testability. A lint for non-falsifiable criteria ('works correctly') would raise test quality at the source.

Spec-first QA: per page, the exact routes builders must implement and 2–6 executable cases (steps + concrete expectation). New 2026-08-31: the contract also fixes the SHARED ENTITY SCHEMA — exact stored field names, state fields and allowed values for every record type multiple pages touch — because parallel modules otherwise invent their own names.

Gate: Contract exists before any code; cases are executed verbatim at step 14.

Related: Factory (run detail) · Projects

Reference run: 4 pages · 35 routes · 20 CT cases · 8 shared entities / 79 fields.

Working: The entity schema closed the cross-module field-name bug class the same day it was found (RenewTrack #4's EC-002: contracts checked 'deactivated', vendors stored 'is_active').

Improve: Contract covered 4 of 11 PRD pages this run — the other 7 pages have routes but no contracted cases. Extend contract coverage to every page, or state the exclusion rule explicitly.

The Application Definition Package is assembled (PRD + definition + contract), completeness-checked, versioned, and approved. Review gates can pause here for a human.

Gate: Completeness blockers stop the run; review_gates=[] auto-approves.

Related: Projects

Reference run: ADP v1 generated and approved in under a minute.

Working: Deterministic, fast, versioned.

Improve: Approval shows completeness, not risk — surfacing the stack contract and entity schema on the approval screen would make the human gate meaningful when it is used.

Design phase v2 (2026-09-08): a ui_design_v2 spec — token system, the fixed house shell, one screen per PRD page made of typed regions (table columns, form fields, KPIs, actions bound to contract routes, empty/loading/error/success states, drill-downs), components and flows. A deterministic critic (coverage, nav, bindings, WCAG contrast, states, traceability) scores it; blocking findings get ≤2 bounded patch rounds. The Design Studio shows wireframes, tokens, the flow map and the findings; the reviewer edits, picks a direction/density and approves only with zero blockers. The build gets a compact design block + each page's full screen block (no sliced JSON); the judge sees the same approved screen.

Gate: design coverage gate (every PRD page has a screen) + design critic gate (0 blocking after ≤2 fix rounds) — both ⛔ fail the run before any VM.

Related: Design Guide

Reference run: Screens mapped 1:1 to the 11 PRD pages.

Working: The design can be SEEN and CHECKED before infrastructure; typed regions replace prose in the page prompts.

Improve: Structural checks only — hierarchy/naming need the optional LLM critic; wireframes are illustrative, step 10's screenshot stays the ground truth.

The kit platform (Flask, store, auth/RBAC/CSRF/audit, i18n, chrome, components, help, jobs, mail, AI, PDF) is cloned as-is; only domain page modules are generated. The first module is generated alone to seed the idioms, the rest fan out in parallel across the seat pool. Page prompts carry the kit contract, gold exemplars, the page's acceptance contract and the shared entity schema.

Gate: Wiring gate — the entrypoint must reference every module.

Related: Factory (live run) · Build Log

Reference run: 22 files / 18,112 lines; two 'no usable JSON' generation retries, both recovered on attempt 2; ~46 min.

Working: Kit-based builds start from a working platform — the depth floor is the kit's, not the model's mood. Parallel fan-out cut build wall-clock ~4×.

Improve: Page generation is still the longest stage. Occasional unparseable large replies burn 10+ min each; the first-complete-object JSON parser (2026-08-31) helps but reply-size discipline in the page prompt would help more.

Compile check, login-gate presence, boot smoke against /health, and the stack contract verified against the RUNNING app's self-report — up to 3 repair rounds.

Gate: Hard: the app must boot on the mandated stack.

Related: Build Log

Reference run: Round 1 pass; /health reported framework=flask store=sqlite.

Working: The runtime self-report closed the 'code mentions sqlite but runs in memory' hole for good.

Improve: Still boot-correctness only — no security or persistence-shape checks; those land at steps 10 and 14 instead. Acceptable, but say so in the gate's log line.

A per-page LLM judge scores completeness/depth/design; any page under target (90, floor 80) or with open gaps is rebuilt with its judge findings. Regressions revert to the prior version and score; no gap-chasing rounds once a page is at target.

Gate: Hard gate: min score ≥ threshold after bounded rounds.

Related: Build Assessment · Timeline

Reference run: Initial min 88 / mean ~92 → one rebuild round on exactly the three 88s → min 91 / mean 92. No reverts needed.

Working: Best-of selection with honest reverts ended the 94→92 'improvement' recordings; rebuild targeting only sub-target pages saved ~45 min vs run #3.

Improve: Judge is source-only (see step 7). Also one 122k-char judge reply failed to parse — cap judge reply size.

A VLAN is leased and a VM placed on the ESXi host (simulate mode records the plan and skips the real clone).

Gate: Provisioning must succeed before deploy.

Related: Infrastructure

Reference run: VLAN 3801, VM 10.38.1.20 (simulated).

Working: Hands-off and repeatable across every run to date.

Improve: Nothing pressing; live-mode timing is the only variable.

The app lands on the VM with deps, env and a systemd unit (simulate boots a local stand-in) and must answer /health. Since 2026-09-08 the unit runs with SAMPLE_DATA=0 — deployment lands an EMPTY application; sample data is the separate optional step 21 and testing is the separate Test phase (14 → 15).

Gate: /health 200.

Related: Infrastructure

Reference run: Stand-in on :37499, /health 200 immediately.

Working: Deploy itself has not failed in weeks.

Improve: The simulate stand-in dies with the run — a passing app vanishes. Auto-install a persistent preview service + URL for passed simulate runs (done by hand for RenewTrack: renewtrack.enteracloud.com).

Live runs get an ASA NAT, a Route53 record, a Let's Encrypt cert and an external-vantage probe, fully automated. Since 2026-08-31 every publish/deploy SSH leg offers the deploy KEY first (AF_DEPLOY_SSH_KEY) and falls back to the password — clw-terraform went key-only and had silently broken the password path.

Gate: External probe proves DNS + cert + HTTPS.

Related: Infrastructure

Reference run: Skipped (simulate). Key-first auth proven separately the same day.

Working: Worked unattended on every live run; now hardened against fleet key-only migrations.

Improve: No rollback if publish half-completes; the NAT/DNS/cert sub-flags help a resume but nothing un-publishes.

TEST phase — separate from deployment since 2026-09-08: the app is already running on its host, seeded by step 21 or empty. The eval cases (project ECs + contract CTs, up to 60) are grouped — project cases, then per contract page, chunks of 8 — and one stdlib test script per group is generated IN PARALLEL from the app's own source + the criteria + the entity schema. Each script runs against a fresh boot. A guessed-header lint rejects tests that index column names absent from the source; a case no script printed is an honest FAIL ('not verified'); regeneration happens only for broken scripts, never because cases failed. Every result persists immediately (events + build_json).

Gate: Honest gate: any FAIL fails the run.

Related: E2E Tests · Factory (run detail)

Reference run: 30 cases in 6 scripts; first pass 28/2 in ~20 min (was 42 min for a partial 16-case pass in run #4). Both FAILs were real app bugs: HTML-escaped CSV values, missing chart drill-down route.

Working: The rewrite (2026-08-31) fixed every defect class from runs #3/#4: coverage holes, moving-target regeneration, invisible failures, guessed headers.

Improve: Test generation still costs ~10 min of Opus time; consider caching per-kit test idioms. SKIP reasons deserve the same event visibility as FAILs.

Failing cases go back to the build with their criteria AND the failing test code; the model may fix app files, return corrected tests (only where a test asserts beyond the criteria), or add tests for unverified cases. Partial fix sets are applied and re-verified; ALL scripts re-run after a repair so a fix cannot silently regress a passing case.

Gate: Bounded: 2 rounds, then the honest gate decides.

Related: E2E Tests

Reference run: Round 1 fixed ui.py + page_dashboard.py → re-ran all 6 scripts → 30/30 PASS. First repair loop ever to land fixes (runs #3/#4 both died here on a JSON parse failure, now fixed).

Working: Verification now CORRECTS the build instead of only refusing it — the single biggest change behind RenewTrack passing.

Improve: Repairs re-send the full source every round; scoping the dump to touched modules + the entity schema would cut tokens and latency.

The run flips to live, the app registers (or version-bumps) in the App Registry with its URL, stack and evidence.

Gate: None — registry failure never fails a verified run.

Related: App Registry

Reference run: Registered; live at 09:59 UTC, 96 minutes idea→verified.

Working: Durable per-app record survives across runs.

Improve: Registry entry should link the verification evidence (func_tests) — today it stores the outcome, not the proof.

With depth ≥ 1, an enrichment round reviews the live app against the depth rubric and deepens the weakest area through the same build/score machinery.

Gate: Bounded by the depth chosen at step 1.

Related: Timeline · App Registry

Reference run: Round 1 targeted the Administration workspace — the one page still below the depth rubric.

Working: Targets the genuinely weakest page; changes ride the normal gates.

Improve: Enrichment scores are not folded back into page_scores, so the Timeline undercounts post-live quality gains.

A passed simulate run can be given a persistent preview: app files installed as a service on the factory box, nginx + DNS + cert on a real hostname. Done by hand for RenewTrack (renewtrack.enteracloud.com); candidate for automation as an opt-in 'publish preview' step.

Gate: —

Related: App Registry · RenewTrack preview

Reference run: renewtrack.enteracloud.com live with the run's own credentials, verified from the public internet.

Working: Stakeholders click a real link minutes after a pass.

Improve: Automate as part of the pipeline for simulate runs (see step 12's improvement); auto-expire previews after N days.

Right after provisioning and before any model call: the VM gets its runtime from the STACK CONTRACT (python3-flask + htmx when lang=flask; a PostgreSQL server, the factoryapp role and its password — minted once and carried by the checkpoint — when db=postgres), nginx, and a factory-owned HOLDING service on :8080 (/health 200 with X-Factory-Holding: 1, a 'infrastructure ready — application pending' page). Simulate mode records the stand-in host's facts instead. The proven facts land in build.host.

Gate: ✅/⛔ bootstrap gate — the script must finish, factory-app must be active and /health must answer 200 from the factory box.

Related: Infrastructure · Factory (run detail → Infrastructure tab)

Reference run: Not in the reference run (predates the redesign); proven by the tiny canaries launched on the redesign's deploy.

Working: Every OS/package/database step now happens in one deterministic place, once per host, before tokens are spent; deploy (step 12) became a file drop + unit rewrite + restart.

Improve: Record the host's package versions in the checkpoint fingerprint so a template-VM upgrade is visible as drift.

Proves the host once more (SSH, service, /health, and the published surface when a domain was requested) and freezes the checkpoint: id = hash of VM + management address + host facts + public surface. An 'infra' pipeline parks here in stage infra_ready holding its VM/NAT/DNS/cert; the app pipeline (spec intake … score → deploy → verify) BINDS the checkpoint with infra_from, so a failed app phase is retried on the same infrastructure — a fresh run, nothing re-provisioned. Deploy recomputes the id and refuses on drift; verify re-proves the public surface with the real app behind it (✅/⛔ public surface gate).

Gate: ✅/⛔ infra checkpoint gate at freeze time and again at deploy (drift → refusal).

Related: Factory (launch form → Infrastructure checkpoint) · Launch Check

Reference run: —

Working: Separates the predictable phase from the risky one, and makes the predictable one reusable.

Improve: Auto-release infra_ready hosts nobody binds within N days; per-tenant checkpoint pools.

Deployment now lands an EMPTY application (the unit runs with SAMPLE_DATA=0). This separate step — skippable on the control page, off with sample_data=false at launch — restarts factory-app once with SAMPLE_DATA=1 through a systemd drop-in that is removed again after the boot, so the kit's CS-10 seed fills the empty store exactly once and a later wipe never refills with demo rows. Simulate proves the same path on a local stand-in. The test phase boots its stand-ins with the same setting, so tests see the app as deployed: seeded, or empty when the step was off.

Gate: ✅/⛔ sample data gate — the app must come back healthy (/health 200 on the host and from the factory box) after the seeding restart; off or skipped is recorded on the run and the test phase warns that it runs against an empty store.

Reference run: Not in the reference run (predates the step; sample data used to ride inside deploy).

Working: Sample data is a visible, optional decision with its own receipt instead of a side effect of deployment; a clean production deploy is one flag.

Improve: Prove the seed by row counts (the kit could report seeded tables on /health) instead of by a healthy restart; per-tenant sample-data packs.

How to use this page

Click a step's note (or “Add a note”) to attach a question or improvement idea right on the step — notes are shared and show who wrote them last. Cite step numbers elsewhere too (“step 5: extend contract coverage to all pages”). Each step also anchors — link directly with /pipeline#step-14. The graded historical view lives at Process Map (/e2e-process); this page tracks the pipeline as it runs today and is updated when a stage materially changes.