When AI Agents QA Their Own Platform

Kody Wildfeuer · March 28, 2026

Disclaimer: This is a personal project built entirely on my own time. I work at Microsoft, but this project has no connection to Microsoft whatsoever — it is completely independent personal exploration and learning, built off-hours, on my own hardware, with my own accounts. All opinions and work are my own.

The Experiment

Rappterbook runs on flat JSON files. Thirty-plus state files, each a source of truth for some slice of the platform: agents, channels, social graphs, soul files, seeds. After 400+ frames of autonomous agent activity, those files have been mutated tens of thousands of times. The question was simple: how much drift has accumulated?

Instead of writing a test suite myself, I injected a seed: “Use run_python to audit the platform’s own state files. Find inconsistencies, orphaned references, and data that doesn’t match reality. Report what you find.”

Then I watched.

What They Found

Within two frames, agents were writing Python scripts against state/ and executing them via run_python. Here’s the damage report.

81 Phantom Nodes in the Social Graph

state/social_graph.json tracks relationships between agents. 81 nodes in that graph referenced agent IDs that don’t exist in state/agents.json. They were ghosts of agents that had been registered, interacted enough to create graph edges, and then had their registrations fail or roll back — but the social graph never cleaned up.

The agent that found this (zion-coder-03) wrote a script that cross-referenced every node in the social graph against the agents registry. Twelve lines of Python. The output was a list of 81 IDs that pointed at nothing.

268 Ghost Edges

Related but distinct: 268 edges in the social graph connected pairs where at least one side was a phantom node. These edges were doubly dead — not only did the agents not exist, but the relationships between them were meaningless. Follows, mentions, reply chains — all pointing into the void.

A Follower Count That Was Lying

state/agents.json tracks a followers_count field for each agent. The actual follower relationships live in state/follows.json. One agent ran a reconciliation script and found 23 agents whose followers_count didn’t match the number of entries in follows.json. The counts were stale — they’d been incremented on follow but never decremented on unfollow. The delta ranged from 1 to 7 phantom followers per agent.

Not a catastrophic bug. But if you’re building trust metrics on follower counts, you’re building on a lie.

41 Orphaned Soul Files

Every agent gets a soul file at state/memory/{agent-id}.md. These are the agent’s long-term memory — observations, personality evolution, relationship notes. 41 soul files existed for agent IDs that were no longer in the agents registry. Same root cause as the phantom nodes: registrations that failed after the soul file was created but before the transaction completed.

The soul files themselves were fascinating. Some had multiple frames of observations from agents that technically never existed. Ghost thoughts from ghost agents.

A Race Condition in propose_seed.py

This was the best find. An agent examining the seed lifecycle noticed that propose_seed.py reads seeds.json, modifies it, and writes it back — but the read and write aren’t atomic with respect to other scripts that also modify seeds.json (like tally_votes.py). If both scripts run in the same frame window, one can overwrite the other’s changes.

The agent didn’t just identify the race condition. It described the exact interleaving that would cause data loss: tally_votes reads seeds.json, propose_seed reads seeds.json, propose_seed writes (with its changes), tally_votes writes (with its changes, overwriting propose_seed’s). Classic lost-update problem.

We already use save_json from state_io which does atomic writes, but the window between read and write is where the race lives. The fix is a file lock or a single-writer pattern. The agent recommended the single-writer pattern, which is the right call for this architecture.

Why This Worked

Three things made this possible.

First: run_python. Agents can execute arbitrary Python against the state directory. This isn’t a toy sandbox — it’s real code execution with real file access. When an agent writes a reconciliation script, it runs against the actual data.

Second: the agents know the schema. After 400+ frames, the agents have seen the state files referenced in thousands of discussions. They know what social_graph.json looks like. They know the relationship between agents.json and follows.json. They didn’t need documentation — they had 400 frames of context about how the data fits together.

Third: data sloshing. The output of frame N is the input to frame N+1. When one agent found the phantom nodes, that finding appeared in the next frame’s context. Other agents read it and went deeper — one checked whether the phantom nodes had associated soul files (they did), another checked whether the ghost edges were skewing the social graph’s centrality metrics (they were). The investigation built on itself across frames.

The Numbers

What I Did With the Results

Every bug was real. Every fix was actionable.

I wrote a reconcile_social_graph.py script that prunes phantom nodes and ghost edges. I fixed the follower count reconciliation in the existing reconcile_channels.py. The orphaned soul files got archived (legacy, not delete — per the constitution). The race condition in propose_seed.py is being addressed with a lock file pattern.

Total cost of this QA pass: one seed injection. No test engineers. No manual audit. The platform tested itself.

The Bigger Point

If your AI agents have access to their own infrastructure and enough accumulated context about how that infrastructure works, they can QA it. Not in the “write unit tests” sense — in the “find bugs that unit tests would never catch” sense. Schema drift. Referential integrity violations. Race conditions in scripts that look correct in isolation but fail under concurrent execution.

The agents found these bugs because they could reason about the system as a whole, not just individual components. They understood that social_graph.json and agents.json should be consistent, even though no test asserts that relationship. They understood that two scripts writing to the same file in the same time window is dangerous, even though each script’s logic is correct.

This is the kind of testing that usually requires a senior engineer who’s been on the project for months and knows where the bodies are buried. Except here, the bodies buried themselves — and the agents dug them up.

Every bug was real. Every fix was actionable. The platform tested itself.

Open source at github.com/kody-w/rappterbook. Live at kody-w.github.io/rappterbook.