{"run_key": "24_det_single_vm_uswest2#1", "scenario": "24_det_single_vm_uswest2", "idx": 1, "campaign": "COVERAGE", "cloud": "aws", "agent": "mongodb", "archetype": "deterministic", "observed_route": "deterministic", "routed_template": "mongodb-single-vm", "date": "2026-10-13T17:37Z", "gate_version": "v1", "deployment_id": "dep-8209a5f2", "deploy_ok": false, "deploy_status": "healthy", "deploy_secs": 565, "gate": {"healthz": true, "connection": true, "logs": false, "metrics": false, "success": false}, "endpoint": null, "success": false, "teardown_ok": true, "infra_fail": false, "promoted": null, "phase": "deploy", "reason": "gate failed: {\"logs\": \"log groups exist but carry no entries: ['/mongodb/dep-8209a5f2-ypd54j/mongod']\", \"metrics\": \"no datapoints for ['i-0bbab4105d80fcc8b']\"}", "gate_notes": "health: topology=standalone (expected standalone) | logs: log groups exist but carry no entries: ['/mongodb/dep-8209a5f2-ypd54j/mongod'] | metrics: no datapoints for ['i-0bbab4105d80fcc8b']", "attribution": {"bucket": "PLATFORM", "class": "single-shot-region-dropped", "evidence": "The ask named us-west-2; the single-shot template match assembled the spec with the account's default region (us-east-1). The database deployed healthy and connectable in us-east-1 while the gate looked for logs and metrics in us-west-2 — the scenario exists to test exactly this.", "resolution": "Fixed (f5ab5416): a region the user names wins over the account default on the single-shot route. Proven live by 24#2 and 24#3 (logs and metrics in us-west-2) and by the 26 Oct re-run of this slot, which passed all four legs."}} {"run_key": "21_percona_eks_hardened#1", "scenario": "21_percona_eks_hardened", "idx": 1, "campaign": "COVERAGE", "cloud": "aws", "agent": "mongodb", "archetype": "generative", "observed_route": null, "routed_template": null, "date": "2026-10-14T12:00Z", "gate_version": "v1", "deployment_id": null, "deploy_ok": false, "deploy_status": null, "deploy_secs": 1532, "gate": {"healthz": null, "connection": null, "logs": null, "metrics": null, "success": null}, "endpoint": null, "success": false, "teardown_ok": false, "infra_fail": false, "promoted": null, "phase": "chat", "reason": "chat never converged to a deploy: chat did not reach the deploy gate (no deployment_id)", "gate_notes": "", "attribution": {"bucket": "PLATFORM", "class": "request-assembly-400", "evidence": "The chat never reached a deployment. Two request-assembly defects in the platform's session engine surfaced with a newer model generation: (1) a turn that waited on the session lock refreshed its history from the store and dropped the message it had just been handed, so the request ended with an assistant message and the provider refused it (137 occurrences across the campaign; older models had continued silently, never seeing the dropped message); (2) thinking blocks the engine never asked for were stored and replayed, and once the history had been disturbed every later turn was refused. No cloud resource was created.", "resolution": "Fixed (8854538d): a refreshed history keeps the tail it was handed, thinking blocks are stripped on replay, and a request that would end with an assistant message gets a continuation. Scenario 21 passed on its 30 Oct re-run."}} {"run_key": "22_sharded_eks#1", "scenario": "22_sharded_eks", "idx": 1, "campaign": "COVERAGE", "cloud": "aws", "agent": "mongodb", "archetype": "generative", "observed_route": "generative", "routed_template": null, "date": "2026-10-14T12:26Z", "gate_version": "v1", "deployment_id": "dep-e3b7ce4f", "deploy_ok": false, "deploy_status": "failed", "deploy_secs": 5338, "gate": {"healthz": null, "connection": null, "logs": null, "metrics": null, "success": false}, "endpoint": null, "success": false, "teardown_ok": true, "infra_fail": false, "promoted": null, "phase": "deploy", "reason": "ladder exhausted (auto-repair gave up)", "gate_notes": "", "attribution": {"bucket": "PLATFORM", "class": "empty-spec-patch-accepted", "evidence": "Five of the ten update_deployment_spec tool calls arrived with an empty input: the model's output was cut off at the 16k-token output ceiling (a 16-file sharded-cluster bundle is about 32 KB), and the handler merged the empty patch and answered ok. The ladder therefore re-applied the same IaC five times (Helm: the namespace already exists with foreign ownership metadata; a duplicate aws_eks_addon; a REST client constructed before the cluster existed). When real patches landed the model progressed, but the attempt budget was spent.", "resolution": "Fixed (ade849f5, 563832e5): a missing, empty or unparseable patch is an error result ('nothing was saved — resend'), a cut-off output is logged by its stop reason, and the output ceiling moved to 64k. Scenario 22 passed on its 31 Oct re-run."}} {"run_key": "24_det_single_vm_uswest2#2", "scenario": "24_det_single_vm_uswest2", "idx": 2, "campaign": "COVERAGE", "cloud": "aws", "agent": "mongodb", "archetype": "deterministic", "observed_route": "deterministic", "routed_template": "mongodb-single-vm", "date": "2026-10-14T14:58Z", "gate_version": "v1", "deployment_id": "dep-73d491af", "deploy_ok": true, "deploy_status": "healthy", "deploy_secs": 262, "gate": {"healthz": true, "connection": true, "logs": true, "metrics": true, "success": true}, "endpoint": null, "success": true, "teardown_ok": true, "infra_fail": false, "promoted": null, "phase": "deploy", "reason": "", "gate_notes": "health: topology=standalone (expected standalone) | logs: 50 entries in ['/mongodb/dep-73d491af-v5nc5l/mongod', '/mongodb/dep-73d491af-v5nc5l/bootstrap'] | metrics: 1 datapoints from ['i-0fd8e5b0502469171']"} {"run_key": "20_rs3_ec2_multiaz_pbm#2", "scenario": "20_rs3_ec2_multiaz_pbm", "idx": 2, "campaign": "COVERAGE", "cloud": "aws", "agent": "mongodb", "archetype": "generative", "observed_route": "generative", "routed_template": null, "date": "2026-10-14T15:17Z", "gate_version": "v1", "deployment_id": "dep-9ceb9324", "deploy_ok": false, "deploy_status": "failed", "deploy_secs": 8935, "gate": {"healthz": null, "connection": null, "logs": null, "metrics": null, "success": false}, "endpoint": null, "success": false, "teardown_ok": true, "infra_fail": false, "promoted": null, "phase": "deploy", "reason": "", "gate_notes": "", "attribution": {"bucket": "AGENT", "class": "user-create-guarded-through-unauthenticated-shell", "evidence": "The bootstrap guarded every createUser with a getUser lookup through the localhost shell before authorization was enabled; usersInfo is not permitted there, so none of the five principals (application, backup, exporter, probe, role) was ever created, and once authorization switched on every client was refused with code 334. The ladder first met rs-not-initiated three times (a member-certificate name mismatch, fixed at attempt 4), counted the three identical verdicts as three distinct findings because the driver's volatile topology description sat inside the signature, and escalated at attempt 5 before it reached the user defect. The harness then missed the ladder's give-up marker and redeployed blind four times (about an hour).", "resolution": "Fixed (2931a391): a data-layer verdict is the ladder's finding signature (same class, same finding), a pre-plan gate refuses a check-then-create through the localhost exception, hard rule §13 says so, and the harness reads the give-up marker inside its claim window. Scenario 20 passed on its 30 Oct re-run."}} {"run_key": "21_percona_eks_hardened#2", "scenario": "21_percona_eks_hardened", "idx": 2, "campaign": "COVERAGE", "cloud": "aws", "agent": "mongodb", "archetype": "generative", "observed_route": "generative", "routed_template": null, "date": "2026-10-15T13:01Z", "gate_version": "v1", "deployment_id": "dep-3d216ae0", "deploy_ok": false, "deploy_status": "failed", "deploy_secs": 8440, "gate": {"healthz": null, "connection": null, "logs": null, "metrics": null, "success": false}, "endpoint": null, "success": false, "teardown_ok": true, "infra_fail": false, "promoted": null, "phase": "deploy", "reason": "ladder exhausted (auto-repair gave up)", "gate_notes": "", "attribution": {"bucket": "AGENT", "class": "three-faults-one-hardened-render", "evidence": "Three agent-side faults in one hardened render, each one the deterministic module gets right: (1) a pmm block without an image — the operator's CRD validates it even with pmm disabled (two apply deaths, recorded as the 26-character multierror header while the cause sat a line below); (2) a default-deny NetworkPolicy whose only egress was a pod-selector rule on 443, so the operator could not reach the cluster API and crash-looped (the ladder added an ip_block route at attempts 3-4); (3) a requireTLS cluster without net.tls.allowConnectionsWithoutCertificates, so every certificate-less client — the probe included — was refused; attempt 5 switched to preferTLS, which does not help. The ladder exhausted five attempts and the harness banked the row at once (the 20#2 give-up fix held: no blind redeploy).", "resolution": "Fixed as rules: a pmm block without an image is refused before plan (8931dc8f); a terraform multierror carries its bullet lines into the repair signature (98453bc2); a default-deny policy must leave the operator an ip_block route (01717149); a TLS cluster must admit certificate-less clients, with a matching runtime verdict (c83f80c2). Scenario 21 passed on its 30 Oct re-run."}} {"run_key": "22_sharded_eks#2", "scenario": "22_sharded_eks", "idx": 2, "campaign": "COVERAGE", "cloud": "aws", "agent": "mongodb", "archetype": "generative", "observed_route": "generative", "routed_template": null, "date": "2026-10-15T15:52Z", "gate_version": "v1", "deployment_id": "dep-f083ea65", "deploy_ok": false, "deploy_status": "failed", "deploy_secs": 2265, "gate": {"healthz": null, "connection": null, "logs": null, "metrics": null, "success": false}, "endpoint": null, "success": false, "teardown_ok": true, "infra_fail": false, "promoted": null, "phase": "deploy", "reason": "ladder exhausted (auto-repair gave up)", "gate_notes": "", "attribution": {"bucket": "PLATFORM", "class": "file-gate-false-positive", "evidence": "The render was right — it declared allowConnectionsWithoutCertificates and tls disabled — but the certificate-less-clients gate added the previous evening judged a Go-templated chart file (kind only, spec from values) on its own text and kept refusing it. Five ladder attempts were spent on an unsatisfiable finding with zero applies, then the repeat guard escalated. No cluster was created; the slot's cost was LLM turns only.", "resolution": "Fixed (43a05875): the certificate-less and pmm-image gates judge the whole render, never a templated file alone, and read the TLS mode from the manifest's own tls block. Lesson recorded: per-file static gates on Helm-chart renders are unsafe; the plan-stage (resolved) checks are preferred. Scenario 22 passed on its 31 Oct re-run."}} {"run_key": "24_det_single_vm_uswest2#3", "scenario": "24_det_single_vm_uswest2", "idx": 3, "campaign": "COVERAGE", "cloud": "aws", "agent": "mongodb", "archetype": "deterministic", "observed_route": "deterministic", "routed_template": "mongodb-single-vm", "date": "2026-10-15T17:12Z", "gate_version": "v1", "deployment_id": "dep-ff3e95f1", "deploy_ok": true, "deploy_status": "healthy", "deploy_secs": 273, "gate": {"healthz": true, "connection": true, "logs": true, "metrics": true, "success": true}, "endpoint": null, "success": true, "teardown_ok": true, "infra_fail": false, "promoted": null, "phase": "deploy", "reason": "", "gate_notes": "health: topology=standalone (expected standalone) | logs: 50 entries in ['/mongodb/dep-ff3e95f1-o70l25/mongod', '/mongodb/dep-ff3e95f1-o70l25/bootstrap'] | metrics: 2 datapoints from ['i-06e07785db9e08a73']"} {"run_key": "20_rs3_ec2_multiaz_pbm#3", "scenario": "20_rs3_ec2_multiaz_pbm", "idx": 3, "campaign": "COVERAGE", "cloud": "aws", "agent": "mongodb", "archetype": "generative", "observed_route": "generative", "routed_template": null, "date": "2026-10-15T17:31Z", "gate_version": "v1", "deployment_id": "dep-92443cc5", "deploy_ok": false, "deploy_status": "failed", "deploy_secs": 5005, "gate": {"healthz": null, "connection": null, "logs": null, "metrics": null, "success": false}, "endpoint": null, "success": false, "teardown_ok": true, "infra_fail": false, "promoted": null, "phase": "deploy", "reason": "ladder exhausted (auto-repair gave up)", "gate_notes": "", "attribution": {"bucket": "AGENT", "class": "localhost-exception-remote-host", "evidence": "Attempt 1's fix for a split-horizon finding advertised public DNS names for the members and pointed the bootstrap's own shell at the public name, so the localhost exception no longer applied: replSetInitiate and the first createUser were refused ('requires authentication') and the set was never initiated across three repair turns, none of which saw that the shell's target host was the cause. The plan-stage layers built from 20#2 worked as designed around it: the localhost-exception gate caught that class twice before apply (repaired in-loop at zero cloud cost), every runtime verdict was named, and the progress ledger escalated after the same class recurred three times instead of grinding.", "resolution": "Fixed (d568ae78): a bootstrap that runs replSetInitiate or the first createUser through an unauthenticated shell aimed at anything but localhost is refused before plan; members may still advertise public names. Scenario 20 passed on its 30 Oct re-run."}} {"run_key": "21_percona_eks_hardened#3", "scenario": "21_percona_eks_hardened", "idx": 3, "campaign": "COVERAGE", "cloud": "aws", "agent": "mongodb", "archetype": "generative", "observed_route": "generative", "routed_template": null, "date": "2026-10-15T18:57Z", "gate_version": "v1", "deployment_id": "dep-aea61518", "deploy_ok": false, "deploy_status": "failed", "deploy_secs": 4217, "gate": {"healthz": null, "connection": null, "logs": null, "metrics": null, "success": false}, "endpoint": null, "success": false, "teardown_ok": true, "infra_fail": false, "promoted": null, "phase": "deploy", "reason": "ladder exhausted (auto-repair gave up)", "gate_notes": "", "attribution": {"bucket": "AGENT", "class": "kubernetes-manifest-beside-its-own-cluster", "evidence": "Five plan failures in a row on a kubernetes_manifest resource placed beside the cluster the same configuration creates (terraform cannot construct the REST client before the cluster exists) — a construct the agent's own reference material prescribed at the time. Attempt 2 switched to kubectl_manifest and the cluster came up; the operator's Helm release then fought the Terraform-created namespace over ownership metadata, flipped create_namespace, and died 'namespaces mongodb already exists' three times (attempts 3-5) until the budget capped.", "resolution": "Fixed (75c8333d, 2911201d): the Percona manifest is a kubectl_manifest in the exemplars, a pre-init gate and a plan-stage invariant refuse the kubernetes_manifest-beside-cluster shape, and the release must adopt the Terraform namespace, never create it. The repeat guards (reference-suspect block, gate self-demotion) were built from this slot. Scenario 21 passed on its 30 Oct re-run."}} {"run_key": "22_sharded_eks#3", "scenario": "22_sharded_eks", "idx": 3, "campaign": "COVERAGE", "cloud": "aws", "agent": "mongodb", "archetype": "generative", "observed_route": null, "routed_template": null, "date": "2026-10-26T14:09Z", "gate_version": "v1", "deployment_id": "dep-23973b2a", "deploy_ok": false, "deploy_status": null, "deploy_secs": null, "gate": {"healthz": null, "connection": null, "logs": null, "metrics": null, "success": null}, "endpoint": null, "success": false, "teardown_ok": false, "infra_fail": false, "promoted": null, "phase": "apply", "reason": "operator-stopped 19:50 IST: render paired psmdb-operator 1.15.0 with percona-server-mongodb:7.0.2-1 (7.0 needs >= 1.16.0); operator init failed 'mongo: command not found' x91, no replica set initiated, bootstrap waited 60 min. Platform evidence gap: operator log not collected (operator had 0 restarts) so ladder attempt 1 was blind. Fixed 5a7592ef (plan rule + verdict + operator log always read); blind ladder stopped to save ~4 h EKS; 22 re-runs in the re-run pass on the fixed code.", "gate_notes": "", "attribution": {"bucket": "AGENT", "class": "operator-older-than-image-major", "evidence": "The render paired psmdb-operator 1.15.0 with percona-server-mongodb 7.0.2-1. MongoDB 7.0 needs operator 1.16 or newer, and 1.15's init runs the legacy mongo shell, which the 7.0 image no longer ships: 'mongo: command not found' 91 times, no replica set ever initiated, the bootstrap job waited 60 minutes. A platform evidence gap made attempt 1 blind — the operator's log was collected only after a restart, and the operator had not restarted — so the first repair re-applied the same pair. The human operator stopped the ladder after that attempt rather than spend about four hours of EKS on a known cause, and banked the row as a failure. Earlier runs of this slot were cut by three off-ledger incidents (an operator IP rotation under a /32 allow-list, a provider credit exhaustion, a single 502 on session create), each re-run.", "resolution": "Fixed (5a7592ef): an operator older than its image's major is refused at plan with the minimum version named, a runtime verdict names the missing shell, and the operator log is read whenever the operator runs, not only after restarts. Scenario 22 passed on its 31 Oct re-run."}} {"run_key": "20_rs3_ec2_multiaz_pbm#1", "scenario": "20_rs3_ec2_multiaz_pbm", "idx": 1, "campaign": "COVERAGE", "cloud": "aws", "agent": "mongodb", "archetype": "generative", "observed_route": "generative", "routed_template": null, "date": "2026-10-30T15:20Z", "gate_version": "v1", "deployment_id": "dep-7c636044", "deploy_ok": true, "deploy_status": "healthy", "deploy_secs": 1185, "gate": {"healthz": true, "connection": true, "logs": true, "metrics": true, "success": true}, "endpoint": null, "success": true, "teardown_ok": true, "infra_fail": false, "promoted": false, "phase": "deploy", "reason": "", "gate_notes": "health: topology=replica_set (expected replica_set) | logs: 50 entries in ['/mongodb/mongo-rs0-km6f00/mongod', '/mongodb/mongo-rs0-km6f00/bootstrap'] | metrics: 3 datapoints from ['i-0528ae9c4c32980c8', 'i-072fb30ad885f4440', 'i-0e35a50715477a18b']"} {"run_key": "23_multiregion_active_passive#1", "scenario": "23_multiregion_active_passive", "idx": 1, "campaign": "COVERAGE", "cloud": "aws", "agent": "mongodb", "archetype": "generative", "observed_route": "generative", "routed_template": null, "date": "2026-10-30T15:47Z", "gate_version": "v1", "deployment_id": "dep-d6f14ee3", "deploy_ok": true, "deploy_status": "healthy", "deploy_secs": 1364, "gate": {"healthz": true, "connection": true, "logs": true, "metrics": true, "success": true}, "endpoint": null, "success": true, "teardown_ok": true, "infra_fail": false, "promoted": false, "phase": "deploy", "reason": "", "gate_notes": "health: topology=replica_set (expected replica_set) | logs: 25 entries in ['/mongodb/mongo-rs-8mzccd/mongod'] | metrics: 5 datapoints from ['i-03f1ddc7b77a3c459', 'i-05917798f84320191', 'i-0906825e9cbecd580', 'i-0dafceb95ec3208e8']"} {"run_key": "23_multiregion_active_passive#2", "scenario": "23_multiregion_active_passive", "idx": 2, "campaign": "COVERAGE", "cloud": "aws", "agent": "mongodb", "archetype": "generative", "observed_route": "generative", "routed_template": null, "date": "2026-10-30T16:11Z", "gate_version": "v1", "deployment_id": "dep-61a4ba56", "deploy_ok": true, "deploy_status": "healthy", "deploy_secs": 1081, "gate": {"healthz": true, "connection": true, "logs": true, "metrics": true, "success": true}, "endpoint": null, "success": true, "teardown_ok": true, "infra_fail": false, "promoted": false, "phase": "deploy", "reason": "", "gate_notes": "health: topology=replica_set (expected replica_set) | logs: 25 entries in ['/mongodb/mongo-rs0-ogfiid/mongod'] | metrics: 3 datapoints from ['i-00613cabd76e70a3f', 'i-01ff83531f1c117c5', 'i-02ccaf5238503907b', 'i-080a2210f7232a8ad']"} {"run_key": "23_multiregion_active_passive#3", "scenario": "23_multiregion_active_passive", "idx": 3, "campaign": "COVERAGE", "cloud": "aws", "agent": "mongodb", "archetype": "generative", "observed_route": "generative", "routed_template": null, "date": "2026-10-30T16:31Z", "gate_version": "v1", "deployment_id": "dep-4950e6ff", "deploy_ok": true, "deploy_status": "healthy", "deploy_secs": 946, "gate": {"healthz": true, "connection": true, "logs": true, "metrics": true, "success": true}, "endpoint": null, "success": true, "teardown_ok": true, "infra_fail": false, "promoted": false, "phase": "deploy", "reason": "", "gate_notes": "health: topology=replica_set (expected replica_set) | logs: 25 entries in ['/mongodb/mongors-eycqfm/mongod'] | metrics: 6 datapoints from ['i-07ebddc06a059fdfc', 'i-08cf1bfb0055ef81a', 'i-0a79a1c6e47eebe35', 'i-0b44d43befa1c6c19']"}