deploy-health after the Cloud Run platform fix (23/23 recovered)¹
DeployProofReport v0.1updated 31 Oct 2026Envelope:Java agent × GCP (complete)Python agent × AWS (complete)MongoDB agent × AWS (complete — reliability, coverage, re-run pass)Java agent × AWS (in flight)
The reliability ledger for AI-driven cloud deployments
HiveDeploy's agents deploy applications to customers' own clouds. This report publishes how often that works — raw and attributed, with every failure root-caused, every fix named, and every run's evidence one click away. Nothing here is a demo.
Raw + converged pass rates on a declared scenario envelope, with per-failure attribution. Java × GCP, Python × AWS and MongoDB × AWS complete; Java × AWS running.
≥99% converged success at 95% confidence (Clopper-Pearson), N=480 template-routed runs on frozen versions. Runs after the AWS discovery campaigns close.
Converged pass rate on 24–30 public repos the agent has never seen — pinned SHAs, fixed prompt, publish-regardless. Intake tooling is live; first candidate pinned.
real platform defect in 23 failures — caught by re-run, fixed, class green
avg LLM cost per deploy session (median $0.51, 94% cache hit)
failures root-caused with public dossiers
One scenario (Cloud Run, scale-to-zero) passes health+connection+metrics; its idle container emits no recent log entries by design — disclosed, counted as deploy-health only.
What "deployed" means here
A run counts only if the deployed application proves itself on four independent checks, probed from outside the platform:
the app's endpoint returns 200, twice.
the app exercises its backing stores (writes a Redis queue entry and a Mongo heartbeat), twice.
Gate v1.1 (14 Jul 2026→): measured and published, non-gating — the failing class proved environmentally dominated.the platform's observability discovers ≥1 service and streams ≥1 real log entry.
≥1 live datapoint in metrics history.
Load-balancer-generated 502/503/504 pages never count as "up." Every result row is labeled with the gate version it ran under.
Campaign A — 64 runs, 8 scenarios, deterministic + generative
| Scenario | Topology | Route | Raw | Re-run of failed slots² |
|---|---|---|---|---|
| 01_ | Cloud Run, scale-to-zero | det | 7/8 | 0/1 → closed by platform fix¹ |
| 04_ | Quarkus on GKE | det | 6/8 | 2/2 |
| 10_ | 3-VM (api + 2 workers) | det | 7/8 | 1/1 |
| 11_ | Prod 3-VM, interactive | gen | 5/8 | 3/3 |
| 12_ | Learning-curve 3-VM | gen | 3/8 | 5/5 |
| 12b_ | Clean-slate 3-VM | gen | 3/8 | 5/5 |
| 14_ | MIG + global HTTPS LB (out-of-spec ask) | gen | 5/8 | 3/3 |
| 98_ | Learning-curve variant | gen | 5/8 | 3/3 |
| TOTAL | 41/64 · 64.1% | 22/23 · 95.7% |
Fix-then-rerun protocol: every failure root-caused → failure class fixed platform-wide → the exact failed slots re-run in a stable window. Raw and converged are always published together.
Where the 23 raw failures actually came from
| Attribution | Count | Evidence pattern |
|---|---|---|
| Store-egress transients | 12 | App up + logging; backing-store connection 500s during operator-network flap windows; same scenarios pass in stable windows |
| Network blips | 5 | LLM-API DNS 502s, a full connectivity outage, churn timeouts, post-apply flaps |
| Platform defect (probe contract) | 1 | Read as a network blip at close; re-runs exposed the Cloud Run /healthz probe-contract bug (below) — the campaign's one AGENT-attributed failure, fixed and re-run green |
| Account credit exhaustion | 2 | LLM account hit zero mid-campaign (402) — operator error, not platform |
| Environment residuals | 2 | A leaked-secret 409 collision + a quota flap |
| Observability fail-open | 1 | Metrics discovery skipped during a blip; same render passed with metrics elsewhere |
| Total attributed | 23 | 22 environmental/harness + 1 platform defect — which the instrument caught, fixed, and re-ran green |
Evidence:per-failure dossiersraw rows (JSONL)JSONLcampaign metadataJSONre-run rowsJSONL
Campaign B — 4 unseen production topologies, first-contact ×3
| Scenario | Topology (first time the agent ever saw it) | Final | Note |
|---|---|---|---|
| 20_ | Hardened GKE (private nodes, shielded, netpol) | 3/3 | Logs+metrics path proven on hardened clusters |
| 21_ | Cloud Run multi-service (api + workers) | 3/3 | Recovered from 0/3 — headless-worker architecture mismatch found, deterministic servlet fixer shipped |
| 22_ | MIG behind HTTPS LB | 2/3 | Lone fail = an observability display bug on a healthy deploy (deploy-health 3/3) |
| 23_ | Multi-zone HA GKE | 3/3 | Interpolation-hygiene fixer shipped en route |
| TOTAL | 11/12 gate · 12/12 deploy-health |
Evidence:raw rowsJSONLthe one failure's dossierMD
Every Cloud Run deploy failed its external health probe on /healthz — while the container's own startup and liveness probes on the same path succeeded. Live-evidence diagnosis: Google's front end reserves /healthz externally, returning 404 to outside callers even when the app serves it. No app, module, or probe change could ever have passed. The platform's public probe now falls back across candidate health paths; the fix unblocked the entire Cloud Run class (scenario 01 and the 0/3 → 3/3 recovery above). This is the kind of failure only volume testing on real clouds finds.
Discovery isn't just a score — it feeds the product
Classes close and stay closed.
The JVM-OOM class started 0-for-6; after a learned lesson re-seed + runtime hardening it went 5/5 — and the same class then passed first-try on AWS. Fixes transfer across clouds.
Classes transfer across agents, too.
Before the Python agent's AWS campaign opened, every infrastructure- and deployment-level failure class the Java agent had hit on AWS was ported into the Python agent's templates, guard rules and validators. The Java × GCP discovery opened at 64.1% raw and needed a fix-then-rerun pass to converge; the Python × AWS campaign below went 76/76 raw with no re-runs at all. That is the flywheel working as designed — and it is why the number ships with its caveats attached rather than instead of them.
The platform writes its own fixers.
From its repair history it distilled a Docker-socket readiness fixer that a human engineer had independently hand-written days earlier — and now applies it deterministically in ~300 ms with no LLM call.
Generative successes become templates.
Every verified generative deploy is captured, quarantined, and ratified into the deterministic library. Template-routed deploys cost $0.05–0.15 vs $0.75 average generative — reliability and unit cost compound together.
Second agent, second cloud — 76 runs, 76 passes, no re-runs. Read the caveats before the number.
The same instrument pointed at a different agent and a different cloud: 64 reliability runs (8 scenarios × 8) plus 12 coverage runs on four production topologies the Python agent had never deployed on AWS. One operator environment, us-east-1, every run torn down to a verified-zero account afterwards. Every row passed on first attempt — this campaign needed no fix-then-rerun pass, so raw and converged are the same number. That is a stronger result than the Java × GCP campaign and a weaker claim than it looks: the three caveats below are load-bearing.
raw pass, first attempt (100%) — 64 reliability + 12 coverage, 0 deploy failures
on the strict 4-part gate (health + logs + metrics gating; connection measured true on all 52)
on the learning-curve evaluation gate — endpoint reachability only, as those scenario specs declare³
runs that drifted from a declared-generative ask onto a template — healthy, but not a test of the generative path⁴
clean teardowns; the account was swept to a verified zero after the campaign closed
metered LLM cost across 87 sessions (≈$0.36/run, max session $2.12); provider-invoice reconciliation pending
Campaign A — 64 reliability runs, 8 scenarios × 8
| Scenario | Topology | Declared → observed route | Gate | Raw | Median deploy |
|---|---|---|---|---|---|
| 01_ | Fargate service behind an ALB | det → det (template) | 4-part | 8/8 | 326 s |
| 04_ | EKS cluster, api + 2 worker deployments | det → det (template) | 4-part | 8/8 | 1630 s |
| 10_ | 3-VM (api + 2 workers) behind an ALB | det → det (template) | 4-part | 8/8 | 274 s |
| 11_ | Prod request, interactive, full spec | gen → det (as designed: a complete spec matches a template) | 4-part | 8/8 | 700 s |
| 14_ | Auto-scaling group + ALB (out-of-spec ask) | det → det (template) | 4-part | 8/8 | 354 s |
| 12_ | Learning-curve 3-VM + an extra data volume per VM | gen → det (drift⁴) | reachability³ | 8/8 | 585 s |
| 12b_ | Clean-slate 3-VM + an extra data volume per VM | gen → det (drift⁴) | reachability³ | 8/8 | 517 s |
| 98_ | Learning-curve variant, generative 3-VM | gen → gen (1 run needed 2 pre-apply verify/repair rounds; all 8 captured for promotion) | reachability³ | 8/8 | 575 s |
| TOTAL | 64/64 · 100% | 535 s |
Campaign B — 4 unseen production topologies, first-contact ×3
| Scenario | Topology (first time this agent deployed it on AWS) | Gate | Final | Median deploy |
|---|---|---|---|---|
| 20_ | Hardened EKS (restricted networking, container-level log collection) | 4-part | 3/3 | 1450 s |
| 21_ | Multi-service Fargate (api + workers, one task definition each) | 4-part | 3/3 | 605 s |
| 22_ | Auto-scaling group behind an HTTPS ALB | 4-part | 3/3 | 414 s |
| 23_ | Multi-AZ HA EKS | 4-part | 3/3 | 1857 s |
| TOTAL | 12/12 · 100% | 1027 s |
Scenarios 12, 12b and 98 are learning-curve evaluations by design: their gate is endpoint reachability (200 on the app port), with logs and metrics measured where available but not gating. Every row in the JSONL carries its gate notes, so a reader can separate the two gate modes.
Route drift, published as an open defect. Scenarios 12 and 12b ask for a 3-VM topology plus a second data volume per VM — a requirement no template has a field for, put there precisely to force the generative path. The classifier judged the request complete, routed it onto the 3-VM template, and silently dropped the out-of-scope requirement. The deploys are healthy and the rows are honest, and those 16 runs still don't test what they were written to test. The same class was recorded on GCP (a requested data disk never delivered), so it was never an AWS or a Python problem. The generative evidence on this ledger is therefore scenario 98, 8 rows, not 76.
Fixed on 19 Sep, after this campaign closed. Asking a model for one boolean — "does this request fit the template?" — makes the failure invisible: nothing in the answer says a requirement was dropped. So the verdict moved out of the model and into code. The router now asks only for an enumeration of what the chosen template cannot deliver, and refuses the template route whenever that list is non-empty. Verified with live calls on the real campaign prompts: both learning-curve scenarios now route generative and name the dropped volume verbatim, on AWS and on GCP, while all eleven deterministic and coverage scenarios still route to their template. The 16 rows above stay as they ran — a ledger is not something you re-write once you understand it.
Two incidents that never reached the ledger
| Incident | What happened | Outcome |
|---|---|---|
| 15 Sep · 01_ | The model provider's credit balance hit zero mid-session (HTTP 402). The harness recognised the class and halted cleanly instead of banking a failure. | Operator error, not platform. Topped up; slot re-run → pass. Published here because it is invisible in the JSONL. |
| 15 Sep · 01_ | A local DNS drop mid-apply left the deploy worker blocked inside a subprocess read for 2.5 hours — the deployment read "deploying" on every poll, the process was alive and the log kept growing, so nothing a liveness check watches looked wrong. A queued repair task then resurrected the torn-down deployment. | Harness + platform defect. Root fix: bounded subprocess reads that kill and reap on timeout; a triple-signal watchdog (queue backlog ∧ stale log ∧ dead worker ping); teardown now purges the queue before marking a deployment terminal. Slot re-run → pass. |
Six defects only running it found
Three were caught by pre-campaign canary runs and fixed before the ledger opened — which is why the raw number is 100%, and why it should be read as "after canaries", not "first contact". Three struck mid-campaign.
Availability-zone capability blind spot
(canary). The default VPC has a subnet in every zone, and one us-east-1 zone offers neither the chosen instance type nor Fargate. The 3-VM template picked a subnet without checking; the Fargate template handed all six subnets to the scheduler. A 1-in-6 failure across 19 of the 76 planned runs. Fix: subnets are filtered by live instance-type offerings, and Fargate excludes the incapable zone.
HTTPS requested without a domain
(canary). The extractor turned public HTTPS on with an empty subdomain, sending the deploy into an unrepairable hosted-zone lookup; a second template exposed certificate-via-DNS with no domain field at all. Fix: schema validators degrade to plain HTTP or a self-signed certificate and say so, instead of failing forty minutes in.
A hollow logs gate on EKS
(canary). The only log group the template created was the image build's, so the gate could pass on build output while no container log existed. Fix: control-plane logging plus container-level log collection, with the observability path reading the application group. Confirmed live before launch, and visible in every scenario-20 and -23 row.
Provider credit exhaustion
(mid-campaign) — the 402 incident above.
Worker wedge
(mid-campaign) — the DNS-drop incident above, and the structural fix it forced.
The sweeper reported "verified clean" with 42 resources live
(mid-campaign). Ownership matching used a name prefix, so provider-managed names carrying the run id mid-string were invisible; generatively named resources had no owner rule at all; and the verify pass read an unpaginated first page. Fix: run-id token anywhere in the name, a whitelisted generative-name shape, full pagination, and security groups added to the verify inventory. The account was then swept to a verified zero, and every run since has ended there. One deliberate gap remains: the generative-name rule is never extended to identity objects, because a shape-matched delete of the wrong role is unrecoverable. The cost of that choice showed up at closure — three generatively named identity objects from the leak window survived every automated sweep and were removed by hand during the final audit. Identity residue is the one class this instrument still hands to a human on purpose.
Evidence:reliability rows (JSONL, declared and observed route per row)JSONLreliability metadata and caveatsJSONcoverage rowsJSONLcoverage metadataJSONledger summaryMD
Third agent, a database this time — 64 reliability runs, 56 passes first attempt; 15 first-contact coverage runs, 6 passes; then every failed topology passed a fresh attempt after its class was fixed.
The same instrument pointed at a stateful workload: 64 reliability runs (8 scenarios × 8) of the MongoDB agent deploying self-managed MongoDB on AWS — a single VM, a Percona replica set on EKS, three-member replica sets on EC2 — then 15 coverage runs on five topologies the agent had never deployed (a multi-AZ replica set with backups, a hardened Percona EKS, a sharded cluster on EKS, a multi-region active-passive pair, a single VM in a non-default region), then a re-run pass: one fresh attempt per failed scenario after its failure class was fixed. One operator environment, us-east-1 unless the scenario says otherwise, every deployment torn down and the account swept to a verified zero at every boundary and after closure. The whole program ran fix-to-green: each failure was root-caused while the campaign ran, the fix committed and loaded between slots (never mid-slot), and every number below is still the raw first attempt of its slot. The ledger therefore has eras, and the row dates tell them apart.
reliability, raw pass first attempt (87.5%) — 8 scenarios × 8, no re-runs counted; 39/43 after the database-layer evidence sense landed on 10 Oct
coverage, first contact on five topologies this agent had never deployed (40%); 9 failures → 5 AGENT, 4 PLATFORM, each root-caused, fixed, dossier published⁸
re-run pass: one fresh attempt per failed scenario after its class was fixed — 11, 12b, 20, 21, 22 and the single-VM region case passed; 07 failed its first fresh attempt on a new class (logs never shipped from a healthy database), a rule was added, and its second attempt passed (7/8)
gate for coverage and re-run rows: health, logs, metrics and an authenticated write-and-read from outside the VPC, all gating (rule of 26 Oct); the reliability ledger stays on its 3-leg gate as published⁵
rules the coverage failures became: plan-stage refusals with the exact fix named, runtime verdicts, two probe corrections — and one platform-side defect of our own gates, found by a slot it cost
metered LLM cost: reliability (64 sessions), coverage (15 rows; $473 counting the superseded attempts below), re-run pass (8 rows); ~85% of chat tokens served from the prompt cache
Campaign A — 64 reliability runs, 8 scenarios × 8 · closed 13 Oct
| Scenario | Topology | Declared → observed route | Gate | Raw | Median deploy |
|---|---|---|---|---|---|
| 10_ | Single mongod on one EC2 VM, SCRAM auth | det → det (template) | 3-part⁵ | 8/8 | 275 s |
| 11_ | Percona operator replica set on EKS | det → det (template) | 3-part⁵ | 7/8 | 1429 s |
| 13_ | Single VM, tuned (WiredTiger cache, backups) | gen → gen | 3-part⁵ | 8/8 | 786 s |
| 08_ | Cheapest credible single node | gen → gen | 3-part⁵ | 7/8 | 778 s |
| 14_ | Production ask, interactive, full spec (EKS or EC2 replica set as the model chooses) | gen → gen | 3-part⁵ | 7/8 | 835 s |
| 07_ | Under-specified ask; the agent must elicit the rest | gen → gen (1 run drifted onto the Percona EKS template⁷) | 3-part⁵ | 6/8 | 777 s |
| 12_ | Learning-curve 3-member replica set, no keyFile given | gen → gen | reachability⁵ | 6/8 | 877 s |
| 12b_ | Clean-slate 3-member replica set on EC2 | gen → gen | reachability⁵ | 7/8 | 846 s |
| TOTAL | 56/64 · 87.5% | 812 s |
Campaign B — 5 unseen topologies, first-contact ×3 · 14–16 Oct, four rows re-judged 30 Oct⁹
| Scenario | Topology (first time this agent deployed it on AWS) | Route | Gate | Raw | Failures (bucket · class) | Median deploy (pass) |
|---|---|---|---|---|---|---|
| 24_ | Single VM in a non-default region (us-west-2) — tests region plumbing, not topology | det → det (template) | 4-part | 2/3 | #1 PLATFORM · single-shot route dropped the named region | 268 s |
| 20_ | Multi-AZ 3-member replica set on EC2, keyFile + TLS, PBM backups to S3 | gen → gen | 4-part | 1/3 | #2 AGENT · users created through an unauthenticated check-then-create; #3 AGENT · bootstrap shell aimed at a public name, localhost exception lost | 1185 s |
| 21_ | Hardened Percona EKS (private nodes, default-deny network policy, encryption, PITR) | gen → gen | 4-part | 0/3 | #1 PLATFORM · request assembly refused by the provider (chat never deployed); #2 AGENT · three faults in one render (pmm image, operator egress, certificate-less clients); #3 AGENT · kubernetes_manifest beside its own cluster, Helm namespace ownership | — |
| 22_ | Sharded cluster on EKS (config replica set + 2 shards + mongos) | gen → gen | 4-part | 0/3 | #1 PLATFORM · empty spec patches accepted (output cut at the token ceiling); #2 PLATFORM · our own file gate false-positived a templated chart (zero applies); #3 AGENT · operator 1.15 paired with a 7.0 image | — |
| 23_ | Multi-region active-passive replica set, us-east-1 + us-west-2 with a hidden member | gen → gen | 4-part | 3/3 | — | 1081 s |
| TOTAL | 6/15 · 40% | 5 AGENT · 4 PLATFORM · 0 HARNESS on the ledger | 1014 s |
Campaign C — the re-run pass: one fresh attempt per failed scenario, after its class was fixed · 26–31 Oct
| Row | Re-runs | Why it had failed | Fresh attempt | Deploy |
|---|---|---|---|---|
| 11_ | reliability 11#5 | HARNESS — our gate probe raised on a healthy deploy | ✓ PASS · 4 legs | 1346 s |
| 24_ | coverage 24#1 | PLATFORM — named region dropped | ✓ PASS · 4 legs | 240 s |
| 12b_ | reliability 12b#7 | PLATFORM — open-ingress fixer scoped an egress rule | ✓ PASS · reachability + write⁵ | 1063 s |
| 20_ | coverage 20#2, 20#3 | AGENT — localhost-exception classes, now plan-stage rules | ✓ PASS · 4 legs | 1190 s |
| 21_ | coverage 21#1–3 | PLATFORM + AGENT — request assembly; pmm/egress/TLS; manifest-beside-cluster | ✓ PASS · 4 legs | 2575 s |
| 22_ | coverage 22#1–3 | PLATFORM + AGENT — empty patches; our file gate; operator/image major | ✓ PASS · 4 legs (first ever pass of this topology) | 2649 s |
| 07_ | reliability 07#6 | PLATFORM — probe ignored the CA the render exported | ✕ FAIL · logs — a new class: the CloudWatch agent started from a path that exists only on preinstalled images (AGENT; rule added) | 930 s |
| 07_ | second fresh attempt, after the rule | AGENT — CloudWatch agent control path; plan-stage rule added 30 Oct | ✓ PASS · 4 legs (31 Oct; 25 log entries shipped from the first render) | 684 s |
| TOTAL | 7/8 | 1268 s median (pass) |
Gate modes and the 26 Oct rule. The reliability ledger (Campaign A) was judged as published on 13 Oct: 45 rows on the strict gate with health, logs and metrics gating and the authenticated write measured but not gating; 13 learning-curve rows (12, 12b) on endpoint reachability, as their specs declare. On 26 Oct the operator made the write leg gating for everything that followed (health AND logs AND metrics AND write). Campaign A is not re-judged under that rule, and its write-leg values are not evidence either way: 18 of its 56 passes carry a true write and 38 a false one, and the false ones came from the harness probe writing into a database the rendered application user had never been granted — a harness defect, fixed on 26 Oct (the probe now writes into the database the credentials are scoped to). A ledger is not re-written once you understand it; it is annotated.
Attribution for Campaign A, published per failure. AGENT: 08#1, 12#1, 07#2 and 12#2 fell in the pre-nets era — mongod never came up or the replica set was never initiated, and the repair ladder, lacking any view of the database itself, repeated the same signature until it gave up. 14#4 chained five distinct errors on a production EKS render. HARNESS: 11#5, the campaign's own gate probe raised on a healthy deploy and banked an empty verdict. PLATFORM: 07#6, our probe refused a correctly deployed verifiable-TLS replica set because it never read the CA the render exported; 12b#7, our open-ingress fixer read a standalone egress rule as ingress and locked the instances' egress to the operator's address on every render. All three non-agent classes were fixed and their scenarios re-run in Campaign C.
One route drift: 07#1, declared generative, was judged a complete request by the classifier and routed onto the Percona EKS template. It passed; it is not generative evidence.
Attribution for Campaign B, published per failure. PLATFORM (4): 24#1, the single-shot route built the spec from the account's default region and ignored the one the user named; 21#1, two request-assembly defects that surfaced with a newer model generation (a waiting turn dropped the message it was handed; unrequested thinking blocks were replayed) so the chat never deployed; 22#1, five of ten spec patches arrived empty because the output was cut at the token ceiling and the handler accepted an empty patch as a commit; 22#2, a static file gate we had added the previous evening judged a Go-templated chart file on its own text and refused a correct render five times without one apply. AGENT (5): 20#2, every createUser was guarded by a lookup the unauthenticated localhost shell cannot run, so no user was ever created; 20#3, a split-horizon fix pointed the bootstrap's own shell at the public name and lost the localhost exception; 21#2, three faults in one hardened render (a pmm block without an image, a default-deny policy that cut the operator off from the cluster API, a TLS cluster that refused certificate-less clients); 21#3, a kubernetes_manifest placed beside the cluster the same configuration creates, then a Helm release fighting the Terraform-created namespace; 22#3, operator 1.15.0 paired with a 7.0 image whose legacy shell it still execs. Every one is now a rule, and the two harness-adjacent weaknesses they exposed (the give-up marker the harness missed on 20#2; the operator log collected only after a restart on 22#3) are fixed. Two rows carry teardown_ok=false for honest reasons: 21#1 never created a deployment, and 22#3 was hand-banked after the operator stopped a blind ladder, its cluster removed by the campaign sweep and the account verified clean twice.
Four coverage rows were run twice, and the ledger says which run it shows. 20#1, 23#1, 23#2 and 23#3 first deployed on 14–16 Oct and passed health, logs and metrics with the write leg false — the harness probe defect of footnote 5. When the write leg became gating on 26 Oct those four verdicts were unverified, so on 30 Oct the operator had the slots run again with the corrected probe; all four passed on all four legs, and those 30 Oct rows are the ones published. The 14–16 Oct deployments were healthy; they are disclosed here, not counted. Campaign B's first-attempt rows ran on five more off-ledger interruptions, all re-run: an expired harness token (13 Oct, four slots cut before deploying), an operator IP rotation under a /32 allow-list (16 Oct, 22#3's gate, ladder and teardown all timed out on the stale address; the operator chose a /16 allow-list for campaign runs), a single 502 on session create (26 Oct; the harness now retries), and two provider credit exhaustions mid-ladder (16 and 26 Oct) — each of which made the ladder's repair turn fail and the platform fall to a blind backstop redeploy that the operator stopped to cap EKS spend. A third credit exhaustion struck the re-run pass on 30 Oct (22's attempt, re-run on 31 Oct). None of these is a deploy verdict and none is on the ledger.
What the plan stage learned to refuse
Campaign A taught the agent to see the database (the read-only evidence sense: per-endpoint TCP, an unauthenticated hello, a login per advertised credential, replica-set status, the mongod log tail, security-group facts — now 25 named verdicts, including the operator's own log and the EKS control plane). Campaign B taught it to refuse before apply. Each of the nine failures became at least one plan-stage rule with the exact fix in the finding: a check-then-create through the localhost exception; a bootstrap shell aimed at anything but localhost; a pmm block without an image; a default-deny policy that leaves the operator no route to the API; a TLS cluster that refuses certificate-less clients; a kubernetes_manifest beside its own cluster; a Helm release creating a namespace Terraform already owns; an operator older than its image's major; a CloudWatch agent started from a path the package does not install. Two more layers landed with them on 15 Oct: every mongod configuration block of a render is rehearsed in a real mongod of its own major before plan, and the ladder's attempt budget is progress-aware — it stops grinding when the same class recurs three times and escalates instead. The honest cost of adding rules fast: one of them was wrong (22#2), and a slot paid for it. That gate now judges the render, not a file, and the lesson is written down.
Incidents that never reached the ledger
| Incident | What happened | Outcome |
|---|---|---|
| 11 Oct · 14#6, 10#7 | The model provider's credit balance hit zero mid-campaign. One chat was cut mid-turn and one session creation refused; the harness recognised the class and halted cleanly. | Operator error, not platform. Credits restored; both slots re-run → pass. Invisible in the JSONL, disclosed here. |
| 13 Oct · coverage 20#1–23#1 | The harness's own API token expired exactly 30 days after it was minted, mid-chat on 20#1 and before 21–23#1 started. | Harness, not platform. Token re-minted; the four slots re-run. |
| 11–16 Oct · operator IP rotation | The operator's public IP changed seven times across the two campaigns. Each deployment's ingress is scoped to the operator's address at slot start, so a rotation mid-slot made the probe, the ladder's control-plane dial and even the teardown unreachable (13#7, 22#3). | Environmental; the evidence sense named it exactly. Campaign runs now allow-list the operator's /16 rather than a /32 — a decision recorded on 16 Oct. |
| 16, 26, 30 Oct · credit exhaustion ×3 | Three more provider credit exhaustions, each landing mid-ladder. With no repair turn available the platform fell to a blind backstop redeploy; the operator stopped it each time to cap EKS spend. | Operator error. Each affected slot re-run after top-up; the fall-to-blind-backstop behaviour on a credit error is a known platform weakness, recorded. |
| 26 Oct · the write leg itself | The harness probe had been writing into a database the rendered application user was never granted, so the write leg read false on every generative render that scoped its user correctly. | Harness. Probe corrected to write into the database the credentials are scoped to; four coverage rows and three re-run rows re-run under it (footnote 9). |
| 30 Oct · 22 re-run, twice | The platform's health probe first verified the operator's self-signed CA strictly on a serving sharded cluster, then assumed TLS on a render that had declared it disabled. | Platform, fixed twice (the probe follows the posture the render declares; an operator cluster with no exported CA is accepted as self-signed). The slot's third fresh attempt is the 22 row in Campaign C. |
Defects only running it found — and the nets that followed
Bootstraps too large for the provider.
A base64 user-data script crossed the 16 KB limit three times. Fix: the exemplar, a pre-plan rewrite to gzip encoding, and a deterministic fixer for the apply error.
Operator CIDR never reached a render that declared it.
Fix: the platform injects declared scope variables into every render.
mongod startup rules that are deterministic yet reached apply.
Authorization with a replica set and no keyFile; a TLS mode with no chain of trust. Fix: both blocked before plan with the exact fix in the finding.
An EKS add-on failing silently for 40 minutes.
The EBS CSI controller without IRSA. Fix: exemplar via IRSA, and an add-on sensor that reads those logs into the repair turn.
The platform's own probe disagreeing with a correct deployment — four times across the program.
A self-signed posture the render advertised; a CA the render exported; an operator CA with nothing exported; a declared tls.mode = disabled the probe overrode. Fix, cumulatively: the probe follows the posture the render declares.
A security fixer that broke security.
The open-ingress scoper rewrote an egress rule it mistook for ingress. Fix: egress resources are egress everywhere, egress is a sensed fact with its own verdict, and a pre-plan gate blocks an egress scoped to a CIDR list.
A region the user named, silently replaced.
Fix: a named region wins. Proven live by 24#2, 24#3 and the re-run.
Request assembly that a newer model generation refuses.
A waiting turn dropped the message it was handed; thinking blocks nobody asked for were replayed. Older models had continued silently without ever seeing the dropped message. Fix: the handed tail survives a refresh, thinking is stripped on replay, a request never ends with the assistant.
An empty patch accepted as a commit.
Output cut at the token ceiling arrived as an empty tool input and was merged as success. Fix: an empty patch is an error the model must answer, the cut-off is logged, the ceiling is 64k.
The localhost exception, lost two ways.
A check-then-create through the unauthenticated shell; a shell aimed at the public name. Fix: both refused before plan; a data-layer verdict is the ladder's signature so identical findings count as one.
Hardened Percona renders: three faults, then a manifest beside its own cluster.
Fix: pmm image, operator egress, certificate-less clients, kubectl_manifest and namespace ownership are rules; the repeat guards (reference-suspect block, gate self-demotion) came out of 21#3.
Our own gate, wrong.
A per-file check on a Go-templated chart. Fix: gates judge the render; plan-stage resolved checks are preferred over file text.
An operator older than its image's major
— and an evidence gap that hid it (the operator log read only after a restart). Fix: refused at plan with the minimum version named; the operator log is read whenever the operator runs.
Logs that never shipped on a healthy database.
The CloudWatch agent started from a path only preinstalled images have. Fix: refused at plan when the script installs the package. The product's Logs view would otherwise have stayed blank — a class a deploy gate catches only because it reads the log group.
Evidence:reliability rows (JSONL, declared and observed route per row)JSONLreliability metadata and caveatsJSONcoverage rowsJSONLcoverage metadataJSONre-run rowsJSONLre-run metadataJSONper-failure dossiersledger summaryMD
Java on AWS — discovery running
The same instrument, second cloud: 64 reliability runs (16 deterministic / 48 generative, 9 scenarios) at concurrency 2, with 12 coverage runs queued. Early signal: the deterministic 3-VM scenario passed 4/4 first-try (304-second deploy), and GCP's worst class passed first-try. The failure classes this leg surfaced were ported to the Python agent before its campaign (above). Results publish at closure as report v0.2. Snapshot rowsJSONL (not citable as final — see metadataJSON).
What gets measured next
| Pass | Design (committed before any runs) | Cost / duration |
|---|---|---|
| T2 Certification | N=480 template-routed runs, production-weighted 7-topology mix, failure budget k≤1, one-sided 95% Clopper-Pearson lower bound ≥99%, AGENT/ENVIRONMENTAL/HARNESS attribution, certificate bound to frozen platform+template versions with published invalidation rules | ≈$100–260 · ~2 days |
| T3 Generalization | 24–30 public OSS repos, pinned SHAs, pre-registered list + fixed prompt + adapted gate, pre-run build control, promotion frozen mid-wave, results publish regardless of outcome. Intake tooling live; first candidate pinned (spring-petclinic @ f182358d). | ≈$50–150 · ~2–4 days |
What this report does not claim
- Rates cover a declared scenario envelope (Java agent × GCP, Python agent × AWS and MongoDB agent × AWS topologies above) — not arbitrary repos until T3 lands, not other agents/clouds until their campaigns close.
- The MongoDB × AWS numbers are three ledgers, not one: 56/64 reliability (raw, first attempt, fix-to-green, on the 3-leg gate as published on 13 Oct), 6/15 coverage (first contact on five unseen topologies, on the 4-leg gate with the write gating; four rows re-judged under the corrected write probe and disclosed per row), and a re-run pass of 7/8 fresh attempts after the class fixes. The re-run pass is converged evidence, not first-contact evidence; it is banked separately and never folded into the other two. The reliability ledger's write-leg values are not evidence either way (footnote 5).
- The Python × AWS 76/76 is raw, first-attempt, and measured after canary fixes — not a first-contact number. 24 of its 76 rows ran a reachability-only gate by design, and 16 rows drifted onto a template instead of exercising the generative path they were written for; both are labeled per row. The generative path on this envelope rests on 8 rows, not 76.
- Runs executed from a single operator environment; its network contributed most raw-rate pollution. (Mitigation shipped for future campaigns: cloud-hosted backing stores + environmental re-draw rules.)
- Failure attribution is self-scored pre-certification — mitigated by pre-registration, per-failure dossiers, and the re-run convergence itself (18 of 23 environmentally-attributed slots passed on re-run).
- This is a deploy-time claim (through the gate window + clean teardown), not a runtime SLO.
Versioned, inspectable, re-runnable
Each report release binds to a platform commit, template-library version, scenario-catalog version, and gate version. Published alongside: the ledger summary, ledger documentation and regeneration commands, per-run rows, and per-failure dossiers. Certificates (T2) carry explicit invalidation rules — any deploy-path change voids them until re-certification.
DEPLOYPROOF (WORKING TITLE) · A HIVEDEPLOY PUBLICATION · REPORT V0.1 PREVIEW · JAVA × GCP DATA THROUGH 2026-07-18 · PYTHON × AWS CLOSED 2026-09-19 · MONGODB × AWS RELIABILITY CLOSED 2026-10-13, COVERAGE AND RE-RUN PASS CLOSED 2026-10-31 · GATE V1/V1.1 AND GATE MODE AS LABELED