DeployProof — Reliability Report v0.1

LIVE DATA

T1 · Discovery

Raw + converged pass rates on a declared scenario envelope, with per-failure attribution. Java × GCP complete; Java × AWS and Python × AWS running.

DESIGNED · PRE-REGISTERED

T2 · Certification

≥99% converged success at 95% confidence (Clopper-Pearson), N=480 template-routed runs on frozen versions. Runs after the AWS discovery campaigns close.

DESIGNED · PRE-REGISTERED

T3 · Generalization

Converged pass rate on 24–30 public repos the agent has never seen — pinned SHAs, fixed prompt, publish-regardless. Intake tooling is live; first candidate pinned.

Headline · Java agent × GCP (closed 15 Jul 2026)

64.1%raw pass, 64-run discovery (41/64) — published first, always

95.7%converged after class fixes: 22/23 failed slots recovered on the strict gate

100%deploy-health after the Cloud Run platform fix (23/23 recovered)¹

1real platform defect in 23 failures — caught by re-run, fixed, class green

$0.75avg LLM cost per deploy session (median $0.51, 94% cache hit)

23/23failures root-caused with public dossiers

¹ One scenario (Cloud Run, scale-to-zero) passes health+connection+metrics; its idle container emits no recent log entries by design — disclosed, counted as deploy-health only.

The success gate

What "deployed" means here

A run counts only if the deployed application proves itself on four independent checks, probed from outside the platform:

Health — the app's endpoint returns 200, twice.
Connection — the app exercises its backing stores (writes a Redis queue entry and a Mongo heartbeat), twice. Gate v1.1 (14 Jul 2026→): measured and published, non-gating — the failing class proved environmentally dominated.
Logs — the platform's observability discovers ≥1 service and streams ≥1 real log entry.
Metrics — ≥1 live datapoint in metrics history.

Load-balancer-generated 502/503/504 pages never count as "up." Every result row is labeled with the gate version it ran under.

T1 results · discovery

Campaign A — 64 runs, 8 scenarios, deterministic + generative

Scenario	Topology	Route	Raw	Re-run of failed slots²
01_java_cloudrun	Cloud Run, scale-to-zero	det	7/8	0/1 → closed by platform fix¹
04_quarkus_jvm_gke	Quarkus on GKE	det	6/8	2/2
10_det_3vm_dev	3-VM (api + 2 workers)	det	7/8	1/1
11_interactive_prod_generative	Prod 3-VM, interactive	gen	5/8	3/3
12_learning_curve_gcp	Learning-curve 3-VM	gen	3/8	5/5
12b_clean_3vm_gcp	Clean-slate 3-VM	gen	3/8	5/5
14_oos_prod_mig_global_lb	MIG + global HTTPS LB (out-of-spec ask)	gen	5/8	3/3
98_learning_curve	Learning-curve variant	gen	5/8	3/3
TOTAL			41/64 · 64.1%	22/23 · 95.7%

² Fix-then-rerun protocol: every failure root-caused → failure class fixed platform-wide → the exact failed slots re-run in a stable window. Raw and converged are always published together.

Where the 23 raw failures actually came from

Attribution	Count	Evidence pattern
Store-egress transients	12	App up + logging; backing-store connection 500s during operator-network flap windows; same scenarios pass in stable windows
Network blips	5	LLM-API DNS 502s, a full connectivity outage, churn timeouts, post-apply flaps
Platform defect (probe contract)	1	Read as a network blip at close; re-runs exposed the Cloud Run /healthz probe-contract bug (below) — the campaign's one AGENT-attributed failure, fixed and re-run green
Account credit exhaustion	2	LLM account hit zero mid-campaign (402) — operator error, not platform
Environment residuals	2	A leaked-secret 409 collision + a quota flap
Observability fail-open	1	Metrics discovery skipped during a blip; same render passed with metrics elsewhere
Total attributed	23	22 environmental/harness + 1 platform defect — which the instrument caught, fixed, and re-ran green

Evidence: per-failure dossiers · raw rows (JSONL) · campaign metadata · re-run rows

Campaign B — 4 unseen production topologies, first-contact ×3

Scenario	Topology (first time the agent ever saw it)	Final	Note
20_springboot_gke_hardened	Hardened GKE (private nodes, shielded, netpol)	3/3	Logs+metrics path proven on hardened clusters
21_springboot_cloudrun_multiservice	Cloud Run multi-service (api + workers)	3/3	Recovered from 0/3 — headless-worker architecture mismatch found, deterministic servlet fixer shipped
22_springboot_mig_https_lb	MIG behind HTTPS LB	2/3	Lone fail = an observability display bug on a healthy deploy (deploy-health 3/3)
23_springboot_gke_multizone_ha	Multi-zone HA GKE	3/3	Interpolation-hygiene fixer shipped en route
TOTAL		11/12 gate · 12/12 deploy-health

Evidence: raw rows · the one failure's dossier

Root cause worth reading · the Cloud Run class

Every Cloud Run deploy failed its external health probe on /healthz — while the container's own startup and liveness probes on the same path succeeded. Live-evidence diagnosis: Google's front end reserves /healthz externally, returning 404 to outside callers even when the app serves it. No app, module, or probe change could ever have passed. The platform's public probe now falls back across candidate health paths; the fix unblocked the entire Cloud Run class (scenario 01 and the 0/3 → 3/3 recovery above). This is the kind of failure only volume testing on real clouds finds.

Self-healing & the flywheel

Discovery isn't just a score — it feeds the product

Classes close and stay closed. The JVM-OOM class started 0-for-6; after a learned lesson re-seed + runtime hardening it went 5/5 — and the same class then passed first-try on AWS. Fixes transfer across clouds.
Classes transfer across agents, too. Before the Python agent's AWS campaign opened, every infrastructure- and deployment-level failure class the Java agent had hit on AWS was ported into the Python agent's templates, guard rules and validators. The Java × GCP discovery opened at 64.1% raw; the Python × AWS ledger below opened at 100% raw on 62 runs. That is the flywheel working as designed — and it is why the number needs its caveats read.
The platform writes its own fixers. From its repair history it distilled a Docker-socket readiness fixer that a human engineer had independently hand-written days earlier — and now applies it deterministically in ~300 ms with no LLM call.
Generative successes become templates. Every verified generative deploy is captured, quarantined, and ratified into the deterministic library. Template-routed deploys cost $0.05–0.15 vs $0.75 average generative — reliability and unit cost compound together.

In flight · Python agent × AWS · snapshot 19 Sep 2026 (IST)

Second agent, second cloud — 62 runs, 0 deploy failures. Read the three caveats before the number.

Same instrument, new agent, new cloud: 64 reliability runs (8 scenarios × 8) with 12 coverage runs queued behind them, us-east-1, one operator environment, every run torn down to a verified-zero account afterwards. This is a raw, in-flight number — no converged claim is made until closure.

62/62raw pass so far (100%) — 62 of 64 reliability runs banked; 0 deploy failures, 0 teardown leaks

40/40on the strict 4-part gate (health + logs + metrics gating, connection measured 40/40)

22/22on the learning-curve evaluation gate — endpoint reachability only, as those scenario specs declare³

15runs that drifted from a declared-generative ask onto a template — pass, but do not test the generative path⁴

2off-ledger incidents disclosed below — neither a deploy failure, both slots re-run green

$26metered LLM cost across 73 sessions (≈$0.42/run, max session $2.12); provider-invoice reconciliation pending

Scenario	Topology	Declared → observed route	Gate	Raw	Median deploy
01_py_fargate	Fargate service behind an ALB	det → det (template)	4-part	8/8	326 s
04_py_eks	EKS cluster, api + 2 worker deployments	det → det (template)	4-part	8/8	1630 s
10_det_3vm_dev	3-VM (api + 2 workers) behind an ALB	det → det (template)	4-part	8/8	274 s
11_interactive_prod_generative	Prod request, interactive, full spec	gen → det (as designed: complete spec matches a template)	4-part	8/8	700 s
14_oos_prod_asg_alb	Auto-scaling group + ALB (out-of-spec ask)	det → det (template)	4-part	8/8	354 s
12_learning_curve_aws	Learning-curve 3-VM + extra data volume per VM	gen → det (drift⁴)	reachability³	8/8	585 s
12b_clean_3vm_aws	Clean-slate 3-VM + extra data volume per VM	gen → det (drift⁴)	reachability³	7/7	517 s
98_learning_curve	Learning-curve variant, generative 3-VM	gen → gen (1 run needed 2 pre-apply verify/repair rounds)	reachability³	7/7	575 s
TOTAL (in flight)				62/62 · 100%	535 s

³ Scenarios 12, 12b and 98 are learning-curve evaluations by design: their gate is endpoint reachability (200 on the app port), with logs and metrics measured where available but not gating. Every row in the JSONL carries its gate notes, so the two gate modes can be separated by a reader.

⁴ Route drift, disclosed as an open defect. Scenarios 12 and 12b ask for a 3-VM topology plus a second data volume per VM — a requirement no template has a field for, put there precisely to force the generative path. The classifier judged the request complete, routed it onto the 3-VM template and silently dropped the out-of-scope requirement. The deploys are healthy, the number is honest, and the runs still don't test what they were written to test. The same class was recorded on GCP (a requested data disk never delivered). Fix direction, not yet landed: a stated requirement with no schema field must mark the request incomplete, not vanish. Until then, real generative evidence on this ledger is scenario 98 (7 rows).

Two incidents that never reached the ledger

Incident	What happened	Outcome
15 Sep · 01_py_fargate#3	Model-provider credit balance hit zero mid-session (HTTP 402). The harness recognised the class and halted cleanly instead of banking a failure.	Operator error, not platform. Topped up; slot re-run → pass. Disclosed here because it is invisible in the JSONL.
15 Sep · 01_py_fargate#5	A local DNS drop mid-apply left the deploy worker blocked inside a subprocess read for 2.5 hours — deployment read "deploying" on every poll, process alive, log growing, nothing a liveness check could see. A queued repair task then resurrected the torn-down deployment.	Harness + platform defect. Root fix: bounded subprocess reads that kill and reap on timeout; a triple-signal watchdog (queue backlog ∧ stale log ∧ dead ping); teardown order now purges the queue before marking terminal. Slot re-run → pass.

Six defects only running it found

Three were caught by pre-campaign canary runs and fixed before the ledger opened — which is why the raw number is 100% and should be read as "after canaries", not "first contact". Three struck mid-campaign.

Availability-zone capability blind spot (canary). The default VPC has a subnet in every AZ, and one us-east-1 AZ offers neither the chosen instance type nor Fargate. The 3-VM template picked a subnet without checking; Fargate handed all six subnets to ECS. A 1-in-6 failure on 19 of 76 planned runs. Fix: subnets are now filtered by live instance-type offerings, and Fargate excludes the incapable zone id.
HTTPS requested without a domain (canary). The extractor turned public HTTPS on with an empty subdomain, sending the deploy into an unrepairable hosted-zone lookup; a second template exposed certificate-via-DNS with no domain field at all. Fix: schema validators degrade to plain HTTP / self-signed and say so, instead of failing 40 minutes in.
A hollow logs gate on EKS (canary). The only log group the template created was the image build's, so the gate could pass on build output while no container log existed. Fix: control-plane logging plus container-level log collection, and the observability path reads the application group. Confirmed live before launch.
Provider credit exhaustion (mid-campaign) — the 402 incident above.
Worker wedge (mid-campaign) — the DNS-drop incident above, and the structural fix it forced.
The sweeper said "verified clean" with 42 resources live (mid-campaign). Ownership matching used a name prefix, so provider-managed names carrying the run id mid-string were invisible; generatively named resources had no owner rule; the verify pass read an unpaginated first page. Fix: run-id token anywhere in the name, a whitelisted generative-name shape (never applied to IAM), full pagination, security groups in the verify inventory — then the account was swept to a verified zero and every run since has ended there.

Evidence: raw rows (JSONL, with declared and observed route per row) · campaign metadata and caveats · ledger summary. Final rows, the coverage-12 leg and the closure note publish when the campaign closes.

In flight · Java agent × AWS · snapshot 18 Jul 2026

Java on AWS — discovery running

The same instrument, second cloud: 64 reliability runs (16 deterministic / 48 generative, 9 scenarios) at concurrency 2, with 12 coverage runs queued. Early signal: the deterministic 3-VM scenario passed 4/4 first-try (304-second deploy), and GCP's worst class passed first-try. The failure classes this leg surfaced were ported to the Python agent before its campaign (above). Results publish at closure as report v0.2. Snapshot rows (not citable as final — see metadata).

Roadmap · pre-registered designs

What gets measured next

Pass	Design (committed before any runs)	Cost / duration
T2 Certification	N=480 template-routed runs, production-weighted 7-topology mix, failure budget k≤1, one-sided 95% Clopper-Pearson lower bound ≥99%, AGENT/ENVIRONMENTAL/HARNESS attribution, certificate bound to frozen platform+template versions with published invalidation rules	≈$100–260 · ~2 days
T3 Generalization	24–30 public OSS repos, pinned SHAs, pre-registered list + fixed prompt + adapted gate, pre-run build control, promotion frozen mid-wave, results publish regardless of outcome. Intake tooling live; first candidate pinned (spring-petclinic @ f182358d).	≈$50–150 · ~2–4 days

Limitations

What this report does not claim

Rates cover a declared scenario envelope (Java agent × GCP topologies above; Python agent × AWS as an in-flight snapshot) — not arbitrary repos until T3 lands, not other agents/clouds until their campaigns close.
The Python × AWS 100% is raw and in flight, on a ledger that opened after canary fixes. 22 of its 62 rows are on a reachability-only gate by design, and 15 rows drifted onto a template instead of exercising the generative path they were written for — both disclosed per row. It becomes a citable number only at closure, with the coverage leg and the route-drift defect either fixed or counted.
Runs executed from a single operator environment; its network contributed most raw-rate pollution. (Mitigation shipped for future campaigns: cloud-hosted backing stores + environmental re-draw rules.)
Failure attribution is self-scored pre-certification — mitigated by pre-registration, per-failure dossiers, and the re-run convergence itself (18 of 23 environmentally-attributed slots passed on re-run).
This is a deploy-time claim (through the gate window + clean teardown), not a runtime SLO.

Reproducibility

Versioned, inspectable, re-runnable

Each report release binds to a platform commit, template-library version, scenario-catalog version, and gate version. Published alongside: the ledger summary, ledger documentation and regeneration commands, per-run rows, and per-failure dossiers. Certificates (T2) carry explicit invalidation rules — any deploy-path change voids them until re-certification.