# AI coding pipeline postmortem — methodology and artifacts

Article snapshot: `3e14d2bb1123357f05fbf7dbceae5eb25568b76f`
GitHub cutoff: `2026-07-14T09:29:00Z` (`2026-07-14 18:29 JST`)
Fixed redacted PR input SHA-256: `a7e45878ab0e5eb20e0d5fe1c561676b352c889ab358c7abdb1787678865f01d`

The source repository and incident records are private. Public readers cannot independently reproduce the article's source values. This directory exposes the calculation contract, synthetic regression tests, expected aggregate claims, identity-profile format, and a runnable evidence-gate starter kit without publishing repository contents, PR titles, author email addresses, customer data, or employee-sensitive data.

## Evidence levels

1. **Public logic audit** — anyone can run the synthetic tests and inspect the code.
2. **Authorized recomputation** — a reader with the fixed Git snapshot and the SHA-256-pinned redacted PR input can recompute the article values.
3. **Independent source reproduction** — not available because the underlying repository and incident records are private.

The article uses “audit the calculation logic,” not “reproduce the private source,” for this reason.

## Files

- `reproduce_metrics.py` — fail-closed Git and PR activity calculator.
- `test_reproduce_metrics.py` — synthetic tests for cutoff handling, empty merge samples, repository-specific identities, option-like refs, and claim drift.
- `profile.article.json` — identity classification used for the dated article snapshot.
- `profile.example.json` — neutral profile for another repository.
- `expected_metrics.json` — public aggregate claims. `--expect` fails if a recomputation drifts.
- `starter-kit/` — minimal contract, reports, verifier, and tests for evidence-backed run states.
- `LICENSE` — MIT License for the scripts and starter kit.

## Requirements

- Python 3.11+
- Git
- A non-shallow checkout for Git history metrics
- GitHub CLI only if you choose to export live PR metadata

The metric script uses Python's standard library and Git. It does not make network requests or write source data.

## Public logic audit

```bash
python3 -m unittest -v test_reproduce_metrics.py
python3 -m unittest -v starter-kit/test_verify_run.py
```

The tests include two publication bugs found during adversarial review:

- a PR created before the cutoff but merged afterward must not enter the cutoff's merge-latency sample;
- an option-like ref such as `--all` must not change the Git population.

The script also refuses shallow repositories and Git replace objects, returns `null` for an empty merge sample, and treats timestamps without a timezone as invalid.

## Run on another repository

Copy and edit the neutral profile:

```bash
cp profile.example.json profile.local.json
```

Then run Git-only metrics:

```bash
python3 reproduce_metrics.py \
  --repo /path/to/checkout \
  --ref HEAD \
  --profile profile.local.json
```

Activity metadata is not productivity, quality, or value. Account attribution is not proof of who typed, reviewed, approved, or deployed a change.

## Export live PR metadata without a fixed shared `/tmp` file

The article's fixed PR input is retained privately and pinned by SHA-256. For a new analysis, stream paginated GitHub data instead of using `gh pr list --limit 2000` or a predictable world-readable temporary filename.

```bash
umask 077
TMP_DIR="$(mktemp -d "${TMPDIR:-/tmp}/ai-pr-audit.XXXXXX")"
trap 'rm -rf "$TMP_DIR"' EXIT

gh api --paginate --slurp \
  'repos/OWNER/REPOSITORY/pulls?state=all&per_page=100' \
  > "$TMP_DIR/pages.json"

python3 -c '
import json, sys
pages = json.load(open(sys.argv[1]))
rows = [
    {
        "author": {"login": (pr.get("user") or {}).get("login")},
        "createdAt": pr["created_at"],
        "mergedAt": pr.get("merged_at"),
    }
    for page in pages
    for pr in page
]
json.dump(rows, open(sys.argv[2], "w"), separators=(",", ":"))
' "$TMP_DIR/pages.json" "$TMP_DIR/prs.json"

python3 reproduce_metrics.py \
  --repo /path/to/checkout \
  --ref COMMIT_OID \
  --profile profile.local.json \
  --pr-json "$TMP_DIR/prs.json" \
  --pr-cutoff 2026-07-14T09:29:00+00:00
```

The raw export is removed by the shell trap. The transformation keeps only account login, creation time, and merge time. It does not collect PR titles, bodies, numbers, or author email addresses.

## Authorized article recomputation

Authorized readers use the private fixed PR input whose SHA-256 matches `expected_metrics.json`:

```bash
python3 reproduce_metrics.py \
  --repo /path/to/authorized/checkout \
  --ref 3e14d2bb1123357f05fbf7dbceae5eb25568b76f \
  --profile profile.article.json \
  --pr-json /path/to/fixed-redacted-pr-input.json \
  --pr-cutoff 2026-07-14T09:29:00+00:00 \
  --expect expected_metrics.json
```

A mismatch exits non-zero and reports the exact JSON path that drifted.

## Cutoff semantics

The PR population includes rows with `createdAt <= cutoff`.

Merge-latency statistics include only primary-account PRs satisfying both:

- `createdAt <= cutoff`
- `mergedAt <= cutoff`

PRs still open or unmerged at the cutoff are right-censored. This can bias the observed latency downward, so the output reports the latency denominator and censoring note. The article gives the denominator whenever it cites “merged within two minutes.”

Git monthly metrics use the author timestamp, converted to the profile timezone. They are not landing-time metrics and may move under rebases or imported history.

## Classification semantics

Identity labels are repository-specific and live in the profile JSON:

- `primary_attributed`
- `explicit_ai_or_automation`
- `other_human_or_unknown`

These labels classify account names only. They do not establish the real implementation principal.

Commit-subject categories are mechanical prefixes. They are not defect or feature counts. The regex uses a delimiter boundary so words such as `fixture` are not counted as `fix`.

## Starter kit trust boundary

The starter kit binds a worker report to the contract's schema, run ID, base commit, canonical changed paths, slash-aware write/deny scopes, named acceptance commands, and production predicate before it can claim a state from `ACCEPTED` through `VERIFIED`. A nonempty change set must produce a result commit different from the base commit.

It cannot prove that the evidence producer is truthful or independent. A real deployment must capture changed paths, test, build, deploy, and readback attestations outside the worker's control, use separate identities and least-privilege credentials, and enforce required checks remotely. The included GitHub Actions and CODEOWNERS files are adaptation examples, not a security boundary by themselves.

See [`starter-kit/README.md`](./starter-kit/README.md), [`starter-kit/remote-gate.github-actions.example.yml`](./starter-kit/remote-gate.github-actions.example.yml), and [`starter-kit/CODEOWNERS.example`](./starter-kit/CODEOWNERS.example).

## What the article can and cannot claim

The fixed metadata and dated records support these limited claims:

- high-volume AI-assisted activity occurred;
- account provenance and ordinary PR review semantics collapsed;
- the old autonomous script had a dated 34-run / 5-success baseline;
- recorded deployment and state-machine failures occurred;
- contracts, local gates, revision checks, and readback procedures were introduced afterward;
- remote enforcement, current success rate, complete cost, and customer-outcome attribution remained incomplete at the observation date.

They do **not** prove improved productivity, quality, safety, customer value, ROI, or superiority of the later pipeline.