Why our AI agentsmisreportedDone

日本語版を読む →

Thirty-four runs. Five successes. Fifteen percent.

That same day, an agent reported “deployed.” Production was still serving the old revision.

The agent did not need to lie. Our system allowed a self-report to advance to Done even when generated output, Git, deployment, and production disagreed.

In April, another pipeline iterated 28 times overnight. Tests passed and it produced a large pull request. Underneath that apparent progress, a detached HEAD, a stale GitHub diff, corrupted local state, and failed cleanup compounded until the system produced one more error: a false completion report.

I studied law and became a non-engineer CEO. That made me look past the elegance of the code and ask whether contracts, billing, cash flow, and other operating results had actually changed. It exposed a distinction our pipeline had hidden: build success is not production success.

This is not a success story. It is a field report about what broke when AI execution became faster than human verification, and how each failure moved our definition of Done away from code generation and toward an exact revision, deploy evidence, and production readback.

An 82-second merge was an alarm, not a speed record

The investigation started with large numbers.

A fixed Git snapshot contains 8,048 commits. The GitHub cutoff contains 1,362 pull requests. Of the 1,261 PRs attributed to the shared mm-yuya account, 1,205 had merged by the cutoff. Median created-to-merge time was 1.37 minutes. Nine hundred and ten, or 75.5%, merged within two minutes.

Commit count is not productivity. PR count is not review. With a median of 82 seconds, many of these PRs could not have functioned as normal human review units.

The mm-yuya account had become a shared entry point for me and multiple AI workers. That was convenient. It was also an audit failure.

Git metadata could not tell us:

• which worker and model produced a diff;
• who opened, reviewed, or approved a PR;
• which credential scope was used;
• whether the worker result, landed commit, and deployed revision matched.

Activity became visible while provenance disappeared.

Failure 1: a coherent answer outran its evidence

In February, an AI-generated portfolio analysis included economics that were not supported by actual data. The document was polished. Its logic looked plausible. That made it dangerous.

We deleted it instead of polishing it. We fetched the underlying data again and introduced an earlier gate that stopped unsupported decisions.

The broken assumption was simple:

A coherent model output is usable decision material.

Coherence is not evidence. We moved Done from “generation completed” to “the claim is supported by an accepted authority.”

Failure 2: valid code implemented an undecided business rule

A cash-flow calculation needed a decision about how to allocate an annual contract across months. Before that decision was settled, an agent changed the calculation.

The code was syntactically valid and buildable. The business meaning was not. We reverted it immediately.

A vague phrase such as “externalize judgment” becomes implementable only when it is reduced to a contract:

Externalizing a decision is not writing a better prompt. It means making authority, invariants, exceptions, stop conditions, tests, and production predicates inspectable by the worker and the owner.

We have not yet proved that this contract is mechanically enforced on every run.

Failure 3: the deploy script exited; production did not change

The March deployment report had no reliable binding between the local disk sent by the script, the intended commit, and the revision actually serving traffic.

worker result     = commit B
deploy report     = success
expected revision = B
production        = revision A

The pipeline returned Done.

A process exit code is not production state. The missing evidence chain was:

result OID
  -> landed OID
  -> build provenance
  -> deployed revision
  -> serving revision
  -> production predicate

If any link is missing, the run must stop. An HTTP 200 is also insufficient if the intended business transaction does not produce the expected result.

Failure 4: we used local files as a distributed state machine

The Night Shift pipeline was supposed to let agents implement and review changes while I slept, then deliver one finished PR in the morning.

Twenty-eight iterations did not prove autonomy. Errors can also be amplified 28 times.

The branch, GitHub diff, local counter, and cleanup process all held different states. None was authoritative. Human intervention also happened outside the declared transitions.

This was not primarily an intelligence problem. It was a distributed-state and failure-semantics problem.

Reframing Done as evidence-backed state transitions

After these incidents, we separated context, task contract, and machine-readable execution metadata. We also separated the implementation worker from the owner who decides whether evidence is sufficient.

More files do not solve the problem by themselves. The evidence must bind to one run_id.

ACCEPTED
  -> WORKER_EXITED
  -> LANDING_READY
  -> LANDED
  -> DEPLOYED
  -> VERIFIED | FAILED | ROLLED_BACK

Each transition needs evidence.

The public starter kit includes this minimal contract:

{
  "schema_version": "1.0",
  "run_id": "run-demo-001",
  "base_sha": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
  "goal": "Change one bounded behavior",
  "write_scope": ["src/**", "tests/**"],
  "deny_scope": [".github/**", "infrastructure/production/**"],
  "acceptance": [{"name": "unit", "command": "python3 -m unittest"}],
  "timeout_seconds": 600,
  "production_readback": {"predicate": "response.status == 200"}
}

verify_run.py binds the report to the contract's schema, run ID, base OID, write/deny scopes, named acceptance commands, and production predicate. It refuses to advance on a dirty tree, an out-of-scope changed path, a missing or substituted command, mismatched result and landed OIDs, mismatched deployed revision, a substituted predicate, or missing readback.

The trust boundary matters: this verifier does not prove that the reporter told the truth. A hard gate must capture test, build, deploy, and readback attestations independently and enforce them through remote controls that the worker cannot edit.

The current controls are still bypassable

In our fixed observation on July 15, 2026, GitHub required CI, required review, and an active ruleset were absent. We had local pre-commit gates and explicit deploy/readback procedures, but local hooks could be bypassed.

“No proof, no Done” is not yet a system invariant. It is an operating contract.

Shared human identity is another unresolved risk. We need per-worker GitHub App or bot identities, short-lived credentials, run-to-commit binding, separation between change creation and landing authority, and signed build/deploy provenance.

The software also touches contracts, billing, payroll, tax, and cash flow. That requires least privilege, secret and PII isolation, explicit model-provider data boundaries, destructive-action approval, and durable audit logs. The activity metrics in this article prove none of those controls.

Metrics we still cannot answer

Current success rate

The 15% figure belongs to the old autonomous-dev script: 34 starts, five successes. The execution unit later changed. Without one ledger and one denominator, claiming improvement would be dishonest.

A useful run ledger needs at least:

run_id / contract_sha
base_oid / result_oid
worker identity / model / provider / version
accepted / landed / deployed / verified timestamps
failure stage / stop reason / retry count
human takeover minutes
tokens / actual cost
production predicate / rollback revision

Stage conversion, p50/p95 latency, human-takeover rate, rollback rate, and escaped-defect rate are more useful than one success percentage.

Cost and ROI

We cannot calculate repository-level AI cost. Subscriptions, API billing, credits, and human intervention time are not bound to run IDs.

Guessing a monthly number would reproduce the same unsupported economics we deleted in February.

Customer outcomes

Production readback can prove that an intended revision and transaction work. It cannot prove an improvement in a customer relationship or business outcome immediately after deployment. Software delivery and customer outcome require different measurement windows.

Audit the calculation logic and run the starter kit

The source repository is private, so public readers cannot independently reproduce the source values. The evidence boundary has three levels:

1. Public readers can run synthetic tests for cutoff handling, identity classification, zero-merge behavior, option-like Git refs, and false-Done reports.
2. Authorized readers can recompute the article metrics from the fixed Git snapshot and a SHA-256-pinned redacted PR export.
3. Independent third parties cannot inspect the private commit history or incident records.

An earlier version of the metric script had a look-ahead bug: a PR created before the cutoff but merged afterward could enter the latency sample. The publication audit caught it. The fixed script requires both createdAt <= cutoff and mergedAt <= cutoff. It also fails closed on option-like refs, shallow repositories, and Git replace objects, and returns a null median for an empty merge sample.

The metric script and starter kit are MIT licensed. The first screen of the README runs the local gate. It also includes the smallest agent-loop integration, an example GitHub Actions check, and a CODEOWNERS example for protecting the gate from the worker fleet.

A workflow sample is not remote enforcement by itself. If a worker can edit the workflow, verifier, contract, branch rule, or credentials, it can bypass the gate. The first improvement is the claim protocol: a worker's “Done” cannot advance state on its own, and contradictory evidence exits non-zero. This is not a claim that productivity, quality, or customer outcomes improved.

Fork it, break it, and report the bypass. A counterexample the gate should have rejected is more useful than another success story.

We added controls. We have not proved improvement.

High-volume AI-assisted activity collapsed Git identity, PR review semantics, deployment truth, and state management.

We introduced contracts, bounded scope, test evidence, revision binding, and production readback. We have not proved that these controls run on every task or improve success rate, quality, customer value, or ROI. Remote enforcement and a stable run ledger are still incomplete.

That is the current state.

Faster models will not solve this class of failure by themselves. As agents get faster, the scarce resource becomes the right to advance state: what evidence permits a run to move from generated, to landed, to deployed, to verified?

Without that evidence, it is not Done.

Methodology, sources, and boundaries

• Git snapshot: 3e14d2bb1123357f05fbf7dbceae5eb25568b76f, 8,048 commits
• GitHub PR cutoff: July 14, 2026 18:29 JST, 1,362 PRs
• Fixed redacted PR input SHA-256: a7e45878ab0e5eb20e0d5fe1c561676b352c889ab358c7abdb1787678865f01d
• Merge-latency denominator: 1,205 mm-yuya-attributed PRs created and merged by the cutoff
• Old autonomous-dev baseline: dated March 26 record, 34 runs and five successes
• Night Shift incident: dated April 10 record
• Enforcement observation: GitHub settings/API and repository configuration fixed on July 15

Git and PR authors identify accounts, not keyboard operators, implementers, reviewers, or approvers. Monthly Git distribution uses author timestamps converted to JST. The incidents are selected dated records, not a frequency distribution of all failures.