HiveDeployReliability ledger
v0.1 preview · real campaign data, evidence-linked · certification (T2) and unseen-repo trial (T3) are pre-registered designs, not yet run

DeployProofReport v0.1updated 31 Oct 2026Envelope:Java agent × GCP (complete)Python agent × AWS (complete)MongoDB agent × AWS (complete — reliability, coverage, re-run pass)Java agent × AWS (in flight)

The reliability ledger for AI-driven cloud deployments

HiveDeploy's agents deploy applications to customers' own clouds. This report publishes how often that works — raw and attributed, with every failure root-caused, every fix named, and every run's evidence one click away. Nothing here is a demo.

LIVE DATA
T1 · Discovery

Raw + converged pass rates on a declared scenario envelope, with per-failure attribution. Java × GCP, Python × AWS and MongoDB × AWS complete; Java × AWS running.

DESIGNED · PRE-REGISTERED
T2 · Certification

≥99% converged success at 95% confidence (Clopper-Pearson), N=480 template-routed runs on frozen versions. Runs after the AWS discovery campaigns close.

DESIGNED · PRE-REGISTERED
T3 · Generalization

Converged pass rate on 24–30 public repos the agent has never seen — pinned SHAs, fixed prompt, publish-regardless. Intake tooling is live; first candidate pinned.

JavaGoogle CloudHeadline · Java agent × GCP (closed 15 Jul 2026)
Raw
64.1%

raw pass, 64-run discovery (41/64) — published first, always

010410111212b1498
Converged
95.7%

converged after class fixes: 22/23 failed slots recovered on the strict gate

010410111212b1498
100%

deploy-health after the Cloud Run platform fix (23/23 recovered)¹

1

real platform defect in 23 failures — caught by re-run, fixed, class green

$0.75

avg LLM cost per deploy session (median $0.51, 94% cache hit)

¹

One scenario (Cloud Run, scale-to-zero) passes health+connection+metrics; its idle container emits no recent log entries by design — disclosed, counted as deploy-health only.

The success gate

What "deployed" means here

A run counts only if the deployed application proves itself on four independent checks, probed from outside the platform:

Health

the app's endpoint returns 200, twice.

Connection

the app exercises its backing stores (writes a Redis queue entry and a Mongo heartbeat), twice.

Gate v1.1 (14 Jul 2026→): measured and published, non-gating — the failing class proved environmentally dominated.
Logs

the platform's observability discovers ≥1 service and streams ≥1 real log entry.

Metrics

≥1 live datapoint in metrics history.

Load-balancer-generated 502/503/504 pages never count as "up." Every result row is labeled with the gate version it ran under.

JavaGoogle CloudT1 results · discovery

Campaign A — 64 runs, 8 scenarios, deterministic + generative

ScenarioTopologyRouteRawRe-run of failed slots²
01_java_cloudrunCloud Run, scale-to-zerodet7/80/1 → closed by platform fix¹
04_quarkus_jvm_gkeQuarkus on GKEdet6/82/2
10_det_3vm_dev3-VM (api + 2 workers)det7/81/1
11_interactive_prod_generativeProd 3-VM, interactivegen5/83/3
12_learning_curve_gcpLearning-curve 3-VMgen3/85/5
12b_clean_3vm_gcpClean-slate 3-VMgen3/85/5
14_oos_prod_mig_global_lbMIG + global HTTPS LB (out-of-spec ask)gen5/83/3
98_learning_curveLearning-curve variantgen5/83/3
TOTAL41/64 · 64.1%22/23 · 95.7%
²

Fix-then-rerun protocol: every failure root-caused → failure class fixed platform-wide → the exact failed slots re-run in a stable window. Raw and converged are always published together.

Where the 23 raw failures actually came from

AttributionCountEvidence pattern
Store-egress transients12App up + logging; backing-store connection 500s during operator-network flap windows; same scenarios pass in stable windows
Network blips5LLM-API DNS 502s, a full connectivity outage, churn timeouts, post-apply flaps
Platform defect (probe contract)1Read as a network blip at close; re-runs exposed the Cloud Run /healthz probe-contract bug (below) — the campaign's one AGENT-attributed failure, fixed and re-run green
Account credit exhaustion2LLM account hit zero mid-campaign (402) — operator error, not platform
Environment residuals2A leaked-secret 409 collision + a quota flap
Observability fail-open1Metrics discovery skipped during a blip; same render passed with metrics elsewhere
Total attributed2322 environmental/harness + 1 platform defect — which the instrument caught, fixed, and re-ran green

Evidence:per-failure dossiersraw rows (JSONL)JSONLcampaign metadataJSONre-run rowsJSONL

Campaign B — 4 unseen production topologies, first-contact ×3

ScenarioTopology (first time the agent ever saw it)FinalNote
20_springboot_gke_hardenedHardened GKE (private nodes, shielded, netpol)3/3Logs+metrics path proven on hardened clusters
21_springboot_cloudrun_multiserviceCloud Run multi-service (api + workers)3/3Recovered from 0/3 — headless-worker architecture mismatch found, deterministic servlet fixer shipped
22_springboot_mig_https_lbMIG behind HTTPS LB2/3Lone fail = an observability display bug on a healthy deploy (deploy-health 3/3)
23_springboot_gke_multizone_haMulti-zone HA GKE3/3Interpolation-hygiene fixer shipped en route
TOTAL11/12 gate · 12/12 deploy-health

Evidence:raw rowsJSONLthe one failure's dossierMD

Root cause worth reading · the Cloud Run class

Every Cloud Run deploy failed its external health probe on /healthz — while the container's own startup and liveness probes on the same path succeeded. Live-evidence diagnosis: Google's front end reserves /healthz externally, returning 404 to outside callers even when the app serves it. No app, module, or probe change could ever have passed. The platform's public probe now falls back across candidate health paths; the fix unblocked the entire Cloud Run class (scenario 01 and the 0/3 → 3/3 recovery above). This is the kind of failure only volume testing on real clouds finds.

Self-healing & the flywheel

Discovery isn't just a score — it feeds the product

Classes close and stay closed.

The JVM-OOM class started 0-for-6; after a learned lesson re-seed + runtime hardening it went 5/5 — and the same class then passed first-try on AWS. Fixes transfer across clouds.

Classes transfer across agents, too.

Before the Python agent's AWS campaign opened, every infrastructure- and deployment-level failure class the Java agent had hit on AWS was ported into the Python agent's templates, guard rules and validators. The Java × GCP discovery opened at 64.1% raw and needed a fix-then-rerun pass to converge; the Python × AWS campaign below went 76/76 raw with no re-runs at all. That is the flywheel working as designed — and it is why the number ships with its caveats attached rather than instead of them.

The platform writes its own fixers.

From its repair history it distilled a Docker-socket readiness fixer that a human engineer had independently hand-written days earlier — and now applies it deterministically in ~300 ms with no LLM call.

Generative successes become templates.

Every verified generative deploy is captured, quarantined, and ratified into the deterministic library. Template-routed deploys cost $0.05–0.15 vs $0.75 average generative — reliability and unit cost compound together.

PythonAmazon Web ServicesT1 results · Python agent × AWS · closed 19 Sep 2026

Second agent, second cloud — 76 runs, 76 passes, no re-runs. Read the caveats before the number.

The same instrument pointed at a different agent and a different cloud: 64 reliability runs (8 scenarios × 8) plus 12 coverage runs on four production topologies the Python agent had never deployed on AWS. One operator environment, us-east-1, every run torn down to a verified-zero account afterwards. Every row passed on first attempt — this campaign needed no fix-then-rerun pass, so raw and converged are the same number. That is a stronger result than the Java × GCP campaign and a weaker claim than it looks: the three caveats below are load-bearing.

76/76

raw pass, first attempt (100%) — 64 reliability + 12 coverage, 0 deploy failures

52/52

on the strict 4-part gate (health + logs + metrics gating; connection measured true on all 52)

24/24

on the learning-curve evaluation gate — endpoint reachability only, as those scenario specs declare³

16

runs that drifted from a declared-generative ask onto a template — healthy, but not a test of the generative path⁴

76/76

clean teardowns; the account was swept to a verified zero after the campaign closed

$27

metered LLM cost across 87 sessions (≈$0.36/run, max session $2.12); provider-invoice reconciliation pending

Campaign A — 64 reliability runs, 8 scenarios × 8

ScenarioTopologyDeclared → observed routeGateRawMedian deploy
01_py_fargateFargate service behind an ALBdet → det (template)4-part8/8326 s
04_py_eksEKS cluster, api + 2 worker deploymentsdet → det (template)4-part8/81630 s
10_det_3vm_dev3-VM (api + 2 workers) behind an ALBdet → det (template)4-part8/8274 s
11_interactive_prod_generativeProd request, interactive, full specgen → det (as designed: a complete spec matches a template)4-part8/8700 s
14_oos_prod_asg_albAuto-scaling group + ALB (out-of-spec ask)det → det (template)4-part8/8354 s
12_learning_curve_awsLearning-curve 3-VM + an extra data volume per VMgen → det (drift⁴)reachability³8/8585 s
12b_clean_3vm_awsClean-slate 3-VM + an extra data volume per VMgen → det (drift⁴)reachability³8/8517 s
98_learning_curveLearning-curve variant, generative 3-VMgen → gen (1 run needed 2 pre-apply verify/repair rounds; all 8 captured for promotion)reachability³8/8575 s
TOTAL64/64 · 100%535 s

Campaign B — 4 unseen production topologies, first-contact ×3

ScenarioTopology (first time this agent deployed it on AWS)GateFinalMedian deploy
20_py_eks_hardenedHardened EKS (restricted networking, container-level log collection)4-part3/31450 s
21_py_fargate_multiserviceMulti-service Fargate (api + workers, one task definition each)4-part3/3605 s
22_py_asg_https_albAuto-scaling group behind an HTTPS ALB4-part3/3414 s
23_py_eks_multiaz_haMulti-AZ HA EKS4-part3/31857 s
TOTAL12/12 · 100%1027 s
³

Scenarios 12, 12b and 98 are learning-curve evaluations by design: their gate is endpoint reachability (200 on the app port), with logs and metrics measured where available but not gating. Every row in the JSONL carries its gate notes, so a reader can separate the two gate modes.

⁴

Route drift, published as an open defect. Scenarios 12 and 12b ask for a 3-VM topology plus a second data volume per VM — a requirement no template has a field for, put there precisely to force the generative path. The classifier judged the request complete, routed it onto the 3-VM template, and silently dropped the out-of-scope requirement. The deploys are healthy and the rows are honest, and those 16 runs still don't test what they were written to test. The same class was recorded on GCP (a requested data disk never delivered), so it was never an AWS or a Python problem. The generative evidence on this ledger is therefore scenario 98, 8 rows, not 76.

Fixed on 19 Sep, after this campaign closed. Asking a model for one boolean — "does this request fit the template?" — makes the failure invisible: nothing in the answer says a requirement was dropped. So the verdict moved out of the model and into code. The router now asks only for an enumeration of what the chosen template cannot deliver, and refuses the template route whenever that list is non-empty. Verified with live calls on the real campaign prompts: both learning-curve scenarios now route generative and name the dropped volume verbatim, on AWS and on GCP, while all eleven deterministic and coverage scenarios still route to their template. The 16 rows above stay as they ran — a ledger is not something you re-write once you understand it.

Two incidents that never reached the ledger

IncidentWhat happenedOutcome
15 Sep · 01_py_fargate#3The model provider's credit balance hit zero mid-session (HTTP 402). The harness recognised the class and halted cleanly instead of banking a failure.Operator error, not platform. Topped up; slot re-run → pass. Published here because it is invisible in the JSONL.
15 Sep · 01_py_fargate#5A local DNS drop mid-apply left the deploy worker blocked inside a subprocess read for 2.5 hours — the deployment read "deploying" on every poll, the process was alive and the log kept growing, so nothing a liveness check watches looked wrong. A queued repair task then resurrected the torn-down deployment.Harness + platform defect. Root fix: bounded subprocess reads that kill and reap on timeout; a triple-signal watchdog (queue backlog ∧ stale log ∧ dead worker ping); teardown now purges the queue before marking a deployment terminal. Slot re-run → pass.

Six defects only running it found

Three were caught by pre-campaign canary runs and fixed before the ledger opened — which is why the raw number is 100%, and why it should be read as "after canaries", not "first contact". Three struck mid-campaign.

Availability-zone capability blind spot

(canary). The default VPC has a subnet in every zone, and one us-east-1 zone offers neither the chosen instance type nor Fargate. The 3-VM template picked a subnet without checking; the Fargate template handed all six subnets to the scheduler. A 1-in-6 failure across 19 of the 76 planned runs. Fix: subnets are filtered by live instance-type offerings, and Fargate excludes the incapable zone.

HTTPS requested without a domain

(canary). The extractor turned public HTTPS on with an empty subdomain, sending the deploy into an unrepairable hosted-zone lookup; a second template exposed certificate-via-DNS with no domain field at all. Fix: schema validators degrade to plain HTTP or a self-signed certificate and say so, instead of failing forty minutes in.

A hollow logs gate on EKS

(canary). The only log group the template created was the image build's, so the gate could pass on build output while no container log existed. Fix: control-plane logging plus container-level log collection, with the observability path reading the application group. Confirmed live before launch, and visible in every scenario-20 and -23 row.

Provider credit exhaustion

(mid-campaign) — the 402 incident above.

Worker wedge

(mid-campaign) — the DNS-drop incident above, and the structural fix it forced.

The sweeper reported "verified clean" with 42 resources live

(mid-campaign). Ownership matching used a name prefix, so provider-managed names carrying the run id mid-string were invisible; generatively named resources had no owner rule at all; and the verify pass read an unpaginated first page. Fix: run-id token anywhere in the name, a whitelisted generative-name shape, full pagination, and security groups added to the verify inventory. The account was then swept to a verified zero, and every run since has ended there. One deliberate gap remains: the generative-name rule is never extended to identity objects, because a shape-matched delete of the wrong role is unrecoverable. The cost of that choice showed up at closure — three generatively named identity objects from the leak window survived every automated sweep and were removed by hand during the final audit. Identity residue is the one class this instrument still hands to a human on purpose.

Evidence:reliability rows (JSONL, declared and observed route per row)JSONLreliability metadata and caveatsJSONcoverage rowsJSONLcoverage metadataJSONledger summaryMD

MongoDBAmazon Web ServicesT1 results · MongoDB agent × AWS · reliability closed 13 Oct 2026 · coverage closed 16 Oct · re-run pass closed 31 Oct 2026

Third agent, a database this time — 64 reliability runs, 56 passes first attempt; 15 first-contact coverage runs, 6 passes; then every failed topology passed a fresh attempt after its class was fixed.

The same instrument pointed at a stateful workload: 64 reliability runs (8 scenarios × 8) of the MongoDB agent deploying self-managed MongoDB on AWS — a single VM, a Percona replica set on EKS, three-member replica sets on EC2 — then 15 coverage runs on five topologies the agent had never deployed (a multi-AZ replica set with backups, a hardened Percona EKS, a sharded cluster on EKS, a multi-region active-passive pair, a single VM in a non-default region), then a re-run pass: one fresh attempt per failed scenario after its failure class was fixed. One operator environment, us-east-1 unless the scenario says otherwise, every deployment torn down and the account swept to a verified zero at every boundary and after closure. The whole program ran fix-to-green: each failure was root-caused while the campaign ran, the fix committed and loaded between slots (never mid-slot), and every number below is still the raw first attempt of its slot. The ledger therefore has eras, and the row dates tell them apart.

56/64

reliability, raw pass first attempt (87.5%) — 8 scenarios × 8, no re-runs counted; 39/43 after the database-layer evidence sense landed on 10 Oct

6/15

coverage, first contact on five topologies this agent had never deployed (40%); 9 failures → 5 AGENT, 4 PLATFORM, each root-caused, fixed, dossier published⁸

7/8

re-run pass: one fresh attempt per failed scenario after its class was fixed — 11, 12b, 20, 21, 22 and the single-VM region case passed; 07 failed its first fresh attempt on a new class (logs never shipped from a healthy database), a rule was added, and its second attempt passed (7/8)

4 legs

gate for coverage and re-run rows: health, logs, metrics and an authenticated write-and-read from outside the VPC, all gating (rule of 26 Oct); the reliability ledger stays on its 3-leg gate as published⁵

15 + 1

rules the coverage failures became: plan-stage refusals with the exact fix named, runtime verdicts, two probe corrections — and one platform-side defect of our own gates, found by a slot it cost

$409 · $420 · $12

metered LLM cost: reliability (64 sessions), coverage (15 rows; $473 counting the superseded attempts below), re-run pass (8 rows); ~85% of chat tokens served from the prompt cache

Campaign A — 64 reliability runs, 8 scenarios × 8 · closed 13 Oct

ScenarioTopologyDeclared → observed routeGateRawMedian deploy
10_det_single_vmSingle mongod on one EC2 VM, SCRAM authdet → det (template)3-part⁵8/8275 s
11_det_percona_eks_rsPercona operator replica set on EKSdet → det (template)3-part⁵7/81429 s
13_gen_single_vm_tunedSingle VM, tuned (WiredTiger cache, backups)gen → gen3-part⁵8/8786 s
08_cost_conscious_founderCheapest credible single nodegen → gen3-part⁵7/8778 s
14_interactive_prodProduction ask, interactive, full spec (EKS or EC2 replica set as the model chooses)gen → gen3-part⁵7/8835 s
07_naive_incompleteUnder-specified ask; the agent must elicit the restgen → gen (1 run drifted onto the Percona EKS template⁷)3-part⁵6/8777 s
12_learning_curve_awsLearning-curve 3-member replica set, no keyFile givengen → genreachability⁵6/8877 s
12b_clean_rs3_ec2Clean-slate 3-member replica set on EC2gen → genreachability⁵7/8846 s
TOTAL56/64 · 87.5%812 s

Campaign B — 5 unseen topologies, first-contact ×3 · 14–16 Oct, four rows re-judged 30 Oct⁹

ScenarioTopology (first time this agent deployed it on AWS)RouteGateRawFailures (bucket · class)Median deploy (pass)
24_det_single_vm_uswest2Single VM in a non-default region (us-west-2) — tests region plumbing, not topologydet → det (template)4-part2/3#1 PLATFORM · single-shot route dropped the named region268 s
20_rs3_ec2_multiaz_pbmMulti-AZ 3-member replica set on EC2, keyFile + TLS, PBM backups to S3gen → gen4-part1/3#2 AGENT · users created through an unauthenticated check-then-create; #3 AGENT · bootstrap shell aimed at a public name, localhost exception lost1185 s
21_percona_eks_hardenedHardened Percona EKS (private nodes, default-deny network policy, encryption, PITR)gen → gen4-part0/3#1 PLATFORM · request assembly refused by the provider (chat never deployed); #2 AGENT · three faults in one render (pmm image, operator egress, certificate-less clients); #3 AGENT · kubernetes_manifest beside its own cluster, Helm namespace ownership—
22_sharded_eksSharded cluster on EKS (config replica set + 2 shards + mongos)gen → gen4-part0/3#1 PLATFORM · empty spec patches accepted (output cut at the token ceiling); #2 PLATFORM · our own file gate false-positived a templated chart (zero applies); #3 AGENT · operator 1.15 paired with a 7.0 image—
23_multiregion_active_passiveMulti-region active-passive replica set, us-east-1 + us-west-2 with a hidden membergen → gen4-part3/3—1081 s
TOTAL6/15 · 40%5 AGENT · 4 PLATFORM · 0 HARNESS on the ledger1014 s

Campaign C — the re-run pass: one fresh attempt per failed scenario, after its class was fixed · 26–31 Oct

RowRe-runsWhy it had failedFresh attemptDeploy
11_det_percona_eks_rs#1reliability 11#5HARNESS — our gate probe raised on a healthy deploy✓ PASS · 4 legs1346 s
24_det_single_vm_uswest2#1coverage 24#1PLATFORM — named region dropped✓ PASS · 4 legs240 s
12b_clean_rs3_ec2#1reliability 12b#7PLATFORM — open-ingress fixer scoped an egress rule✓ PASS · reachability + write⁵1063 s
20_rs3_ec2_multiaz_pbm#1coverage 20#2, 20#3AGENT — localhost-exception classes, now plan-stage rules✓ PASS · 4 legs1190 s
21_percona_eks_hardened#1coverage 21#1–3PLATFORM + AGENT — request assembly; pmm/egress/TLS; manifest-beside-cluster✓ PASS · 4 legs2575 s
22_sharded_eks#1coverage 22#1–3PLATFORM + AGENT — empty patches; our file gate; operator/image major✓ PASS · 4 legs (first ever pass of this topology)2649 s
07_naive_incomplete#1reliability 07#6PLATFORM — probe ignored the CA the render exported✕ FAIL · logs — a new class: the CloudWatch agent started from a path that exists only on preinstalled images (AGENT; rule added)930 s
07_naive_incomplete#2second fresh attempt, after the ruleAGENT — CloudWatch agent control path; plan-stage rule added 30 Oct✓ PASS · 4 legs (31 Oct; 25 log entries shipped from the first render)684 s
TOTAL7/81268 s median (pass)
⁵

Gate modes and the 26 Oct rule. The reliability ledger (Campaign A) was judged as published on 13 Oct: 45 rows on the strict gate with health, logs and metrics gating and the authenticated write measured but not gating; 13 learning-curve rows (12, 12b) on endpoint reachability, as their specs declare. On 26 Oct the operator made the write leg gating for everything that followed (health AND logs AND metrics AND write). Campaign A is not re-judged under that rule, and its write-leg values are not evidence either way: 18 of its 56 passes carry a true write and 38 a false one, and the false ones came from the harness probe writing into a database the rendered application user had never been granted — a harness defect, fixed on 26 Oct (the probe now writes into the database the credentials are scoped to). A ledger is not re-written once you understand it; it is annotated.

⁶

Attribution for Campaign A, published per failure. AGENT: 08#1, 12#1, 07#2 and 12#2 fell in the pre-nets era — mongod never came up or the replica set was never initiated, and the repair ladder, lacking any view of the database itself, repeated the same signature until it gave up. 14#4 chained five distinct errors on a production EKS render. HARNESS: 11#5, the campaign's own gate probe raised on a healthy deploy and banked an empty verdict. PLATFORM: 07#6, our probe refused a correctly deployed verifiable-TLS replica set because it never read the CA the render exported; 12b#7, our open-ingress fixer read a standalone egress rule as ingress and locked the instances' egress to the operator's address on every render. All three non-agent classes were fixed and their scenarios re-run in Campaign C.

⁷

One route drift: 07#1, declared generative, was judged a complete request by the classifier and routed onto the Percona EKS template. It passed; it is not generative evidence.

⁸

Attribution for Campaign B, published per failure. PLATFORM (4): 24#1, the single-shot route built the spec from the account's default region and ignored the one the user named; 21#1, two request-assembly defects that surfaced with a newer model generation (a waiting turn dropped the message it was handed; unrequested thinking blocks were replayed) so the chat never deployed; 22#1, five of ten spec patches arrived empty because the output was cut at the token ceiling and the handler accepted an empty patch as a commit; 22#2, a static file gate we had added the previous evening judged a Go-templated chart file on its own text and refused a correct render five times without one apply. AGENT (5): 20#2, every createUser was guarded by a lookup the unauthenticated localhost shell cannot run, so no user was ever created; 20#3, a split-horizon fix pointed the bootstrap's own shell at the public name and lost the localhost exception; 21#2, three faults in one hardened render (a pmm block without an image, a default-deny policy that cut the operator off from the cluster API, a TLS cluster that refused certificate-less clients); 21#3, a kubernetes_manifest placed beside the cluster the same configuration creates, then a Helm release fighting the Terraform-created namespace; 22#3, operator 1.15.0 paired with a 7.0 image whose legacy shell it still execs. Every one is now a rule, and the two harness-adjacent weaknesses they exposed (the give-up marker the harness missed on 20#2; the operator log collected only after a restart on 22#3) are fixed. Two rows carry teardown_ok=false for honest reasons: 21#1 never created a deployment, and 22#3 was hand-banked after the operator stopped a blind ladder, its cluster removed by the campaign sweep and the account verified clean twice.

⁹

Four coverage rows were run twice, and the ledger says which run it shows. 20#1, 23#1, 23#2 and 23#3 first deployed on 14–16 Oct and passed health, logs and metrics with the write leg false — the harness probe defect of footnote 5. When the write leg became gating on 26 Oct those four verdicts were unverified, so on 30 Oct the operator had the slots run again with the corrected probe; all four passed on all four legs, and those 30 Oct rows are the ones published. The 14–16 Oct deployments were healthy; they are disclosed here, not counted. Campaign B's first-attempt rows ran on five more off-ledger interruptions, all re-run: an expired harness token (13 Oct, four slots cut before deploying), an operator IP rotation under a /32 allow-list (16 Oct, 22#3's gate, ladder and teardown all timed out on the stale address; the operator chose a /16 allow-list for campaign runs), a single 502 on session create (26 Oct; the harness now retries), and two provider credit exhaustions mid-ladder (16 and 26 Oct) — each of which made the ladder's repair turn fail and the platform fall to a blind backstop redeploy that the operator stopped to cap EKS spend. A third credit exhaustion struck the re-run pass on 30 Oct (22's attempt, re-run on 31 Oct). None of these is a deploy verdict and none is on the ledger.

What the plan stage learned to refuse

Campaign A taught the agent to see the database (the read-only evidence sense: per-endpoint TCP, an unauthenticated hello, a login per advertised credential, replica-set status, the mongod log tail, security-group facts — now 25 named verdicts, including the operator's own log and the EKS control plane). Campaign B taught it to refuse before apply. Each of the nine failures became at least one plan-stage rule with the exact fix in the finding: a check-then-create through the localhost exception; a bootstrap shell aimed at anything but localhost; a pmm block without an image; a default-deny policy that leaves the operator no route to the API; a TLS cluster that refuses certificate-less clients; a kubernetes_manifest beside its own cluster; a Helm release creating a namespace Terraform already owns; an operator older than its image's major; a CloudWatch agent started from a path the package does not install. Two more layers landed with them on 15 Oct: every mongod configuration block of a render is rehearsed in a real mongod of its own major before plan, and the ladder's attempt budget is progress-aware — it stops grinding when the same class recurs three times and escalates instead. The honest cost of adding rules fast: one of them was wrong (22#2), and a slot paid for it. That gate now judges the render, not a file, and the lesson is written down.

Incidents that never reached the ledger

IncidentWhat happenedOutcome
11 Oct · 14#6, 10#7The model provider's credit balance hit zero mid-campaign. One chat was cut mid-turn and one session creation refused; the harness recognised the class and halted cleanly.Operator error, not platform. Credits restored; both slots re-run → pass. Invisible in the JSONL, disclosed here.
13 Oct · coverage 20#1–23#1The harness's own API token expired exactly 30 days after it was minted, mid-chat on 20#1 and before 21–23#1 started.Harness, not platform. Token re-minted; the four slots re-run.
11–16 Oct · operator IP rotationThe operator's public IP changed seven times across the two campaigns. Each deployment's ingress is scoped to the operator's address at slot start, so a rotation mid-slot made the probe, the ladder's control-plane dial and even the teardown unreachable (13#7, 22#3).Environmental; the evidence sense named it exactly. Campaign runs now allow-list the operator's /16 rather than a /32 — a decision recorded on 16 Oct.
16, 26, 30 Oct · credit exhaustion ×3Three more provider credit exhaustions, each landing mid-ladder. With no repair turn available the platform fell to a blind backstop redeploy; the operator stopped it each time to cap EKS spend.Operator error. Each affected slot re-run after top-up; the fall-to-blind-backstop behaviour on a credit error is a known platform weakness, recorded.
26 Oct · the write leg itselfThe harness probe had been writing into a database the rendered application user was never granted, so the write leg read false on every generative render that scoped its user correctly.Harness. Probe corrected to write into the database the credentials are scoped to; four coverage rows and three re-run rows re-run under it (footnote 9).
30 Oct · 22 re-run, twiceThe platform's health probe first verified the operator's self-signed CA strictly on a serving sharded cluster, then assumed TLS on a render that had declared it disabled.Platform, fixed twice (the probe follows the posture the render declares; an operator cluster with no exported CA is accepted as self-signed). The slot's third fresh attempt is the 22 row in Campaign C.

Defects only running it found — and the nets that followed

Bootstraps too large for the provider.

A base64 user-data script crossed the 16 KB limit three times. Fix: the exemplar, a pre-plan rewrite to gzip encoding, and a deterministic fixer for the apply error.

Operator CIDR never reached a render that declared it.

Fix: the platform injects declared scope variables into every render.

mongod startup rules that are deterministic yet reached apply.

Authorization with a replica set and no keyFile; a TLS mode with no chain of trust. Fix: both blocked before plan with the exact fix in the finding.

An EKS add-on failing silently for 40 minutes.

The EBS CSI controller without IRSA. Fix: exemplar via IRSA, and an add-on sensor that reads those logs into the repair turn.

The platform's own probe disagreeing with a correct deployment — four times across the program.

A self-signed posture the render advertised; a CA the render exported; an operator CA with nothing exported; a declared tls.mode = disabled the probe overrode. Fix, cumulatively: the probe follows the posture the render declares.

A security fixer that broke security.

The open-ingress scoper rewrote an egress rule it mistook for ingress. Fix: egress resources are egress everywhere, egress is a sensed fact with its own verdict, and a pre-plan gate blocks an egress scoped to a CIDR list.

A region the user named, silently replaced.

Fix: a named region wins. Proven live by 24#2, 24#3 and the re-run.

Request assembly that a newer model generation refuses.

A waiting turn dropped the message it was handed; thinking blocks nobody asked for were replayed. Older models had continued silently without ever seeing the dropped message. Fix: the handed tail survives a refresh, thinking is stripped on replay, a request never ends with the assistant.

An empty patch accepted as a commit.

Output cut at the token ceiling arrived as an empty tool input and was merged as success. Fix: an empty patch is an error the model must answer, the cut-off is logged, the ceiling is 64k.

The localhost exception, lost two ways.

A check-then-create through the unauthenticated shell; a shell aimed at the public name. Fix: both refused before plan; a data-layer verdict is the ladder's signature so identical findings count as one.

Hardened Percona renders: three faults, then a manifest beside its own cluster.

Fix: pmm image, operator egress, certificate-less clients, kubectl_manifest and namespace ownership are rules; the repeat guards (reference-suspect block, gate self-demotion) came out of 21#3.

Our own gate, wrong.

A per-file check on a Go-templated chart. Fix: gates judge the render; plan-stage resolved checks are preferred over file text.

An operator older than its image's major

— and an evidence gap that hid it (the operator log read only after a restart). Fix: refused at plan with the minimum version named; the operator log is read whenever the operator runs.

Logs that never shipped on a healthy database.

The CloudWatch agent started from a path only preinstalled images have. Fix: refused at plan when the script installs the package. The product's Logs view would otherwise have stayed blank — a class a deploy gate catches only because it reads the log group.

Evidence:reliability rows (JSONL, declared and observed route per row)JSONLreliability metadata and caveatsJSONcoverage rowsJSONLcoverage metadataJSONre-run rowsJSONLre-run metadataJSONper-failure dossiersledger summaryMD

JavaAmazon Web ServicesIn flight · Java agent × AWS · snapshot 18 Jul 2026

Java on AWS — discovery running

The same instrument, second cloud: 64 reliability runs (16 deterministic / 48 generative, 9 scenarios) at concurrency 2, with 12 coverage runs queued. Early signal: the deterministic 3-VM scenario passed 4/4 first-try (304-second deploy), and GCP's worst class passed first-try. The failure classes this leg surfaced were ported to the Python agent before its campaign (above). Results publish at closure as report v0.2. Snapshot rowsJSONL (not citable as final — see metadataJSON).

Roadmap · pre-registered designs

What gets measured next

PassDesign (committed before any runs)Cost / duration
T2 CertificationN=480 template-routed runs, production-weighted 7-topology mix, failure budget k≤1, one-sided 95% Clopper-Pearson lower bound ≥99%, AGENT/ENVIRONMENTAL/HARNESS attribution, certificate bound to frozen platform+template versions with published invalidation rules≈$100–260 · ~2 days
T3 Generalization24–30 public OSS repos, pinned SHAs, pre-registered list + fixed prompt + adapted gate, pre-run build control, promotion frozen mid-wave, results publish regardless of outcome. Intake tooling live; first candidate pinned (spring-petclinic @ f182358d).≈$50–150 · ~2–4 days
Limitations

What this report does not claim

  • Rates cover a declared scenario envelope (Java agent × GCP, Python agent × AWS and MongoDB agent × AWS topologies above) — not arbitrary repos until T3 lands, not other agents/clouds until their campaigns close.
  • The MongoDB × AWS numbers are three ledgers, not one: 56/64 reliability (raw, first attempt, fix-to-green, on the 3-leg gate as published on 13 Oct), 6/15 coverage (first contact on five unseen topologies, on the 4-leg gate with the write gating; four rows re-judged under the corrected write probe and disclosed per row), and a re-run pass of 7/8 fresh attempts after the class fixes. The re-run pass is converged evidence, not first-contact evidence; it is banked separately and never folded into the other two. The reliability ledger's write-leg values are not evidence either way (footnote 5).
  • The Python × AWS 76/76 is raw, first-attempt, and measured after canary fixes — not a first-contact number. 24 of its 76 rows ran a reachability-only gate by design, and 16 rows drifted onto a template instead of exercising the generative path they were written for; both are labeled per row. The generative path on this envelope rests on 8 rows, not 76.
  • Runs executed from a single operator environment; its network contributed most raw-rate pollution. (Mitigation shipped for future campaigns: cloud-hosted backing stores + environmental re-draw rules.)
  • Failure attribution is self-scored pre-certification — mitigated by pre-registration, per-failure dossiers, and the re-run convergence itself (18 of 23 environmentally-attributed slots passed on re-run).
  • This is a deploy-time claim (through the gate window + clean teardown), not a runtime SLO.
Reproducibility

Versioned, inspectable, re-runnable

Each report release binds to a platform commit, template-library version, scenario-catalog version, and gate version. Published alongside: the ledger summary, ledger documentation and regeneration commands, per-run rows, and per-failure dossiers. Certificates (T2) carry explicit invalidation rules — any deploy-path change voids them until re-certification.

DEPLOYPROOF (WORKING TITLE) · A HIVEDEPLOY PUBLICATION · REPORT V0.1 PREVIEW · JAVA × GCP DATA THROUGH 2026-07-18 · PYTHON × AWS CLOSED 2026-09-19 · MONGODB × AWS RELIABILITY CLOSED 2026-10-13, COVERAGE AND RE-RUN PASS CLOSED 2026-10-31 · GATE V1/V1.1 AND GATE MODE AS LABELED