Codumentation: When Documentation Becomes Code
Treating documentation claims as executable specifications
Markus Hav
Lead Researcher, Agents
Abstract
We propose a fundamental shift in technical documentation: treating documentation claims as executable specifications that must prove themselves against the codebase. This approach, which we call validation-first documentation, inverts the traditional workflow. We introduce Codumentation, an implementation of this hypothesis, and examine its implications for both human developers and AI agents working with large-scale codebases.
Try it yourself
1. Install the package in your project:
npm i codumentation --save-dev2. Ask your coding agent (Claude Code, Cursor, etc.) to run:
npx codumentation init README.mdWorks with README.md, CLAUDE.md, or any markdown context file.
The Spark: A Documentation Crisis
I was reviewing a CLAUDE.md file for an AI coding agent. The documentation confidently stated: "All our API routes are protected and require a Zod schema for input validation."
I checked the app/api/ folder. I found four new routes added in the last week. None of them imported Zod. None of them had a schema. They were taking raw any inputs directly from the request body.
The documentation wasn't just wrong—it was providing a false sense of security. The humans assumed the "standard" was being followed. The AI agent, reading the stale context file, assumed it *was* following the standard.
The Problem with Hope-Based Documentation
Traditional documentation operates on the Hope Workflow:
2. Hope it stays accurate
3. Hope someone notices when it's wrong
4. Hope they fix it
This breaks down at every step. But it becomes critical when AI agents enter the picture. A human might smell when docs are wrong; an AI agent takes them at face value and propagates errors through every file it touches.
The Validation-First Hypothesis
Hypothesis: Documentation should not be trusted by default; it should be validated by code that proves every claim.
Every claim in your docs is a testable assertion. If your README says "All API routes use Zod", there should be a validator that globs your API folder and fails if it finds a route without a schema.
Traditional: Hope-Based
Silent Decay. High Latency.
Codumentation: Validation-First
Self-Defending. Instant Feedback.
How It Works: Documentation as Modules
With Codumentation, documentation is split into modules. Each module (e.g., apiContract.ts) exports both the content and a validate() function.
export const validate = async () => {
const routes = assertGlobMatches('app/api/**/*.ts');
for (const route of routes) {
assertFileContains(route, 'z.object', `${route} missing Zod validation`);
}
};
The AI Agent Optimization Loop
This is where Codumentation becomes transformative. Traditional AI context files are write-once artifacts. With validation, you get a measurement loop.
By running codumentation stats, you can see which rules the AI keeps breaking. If the "apiContract" module fails 40% of the time, your documentation isn't clear enough.
| Traditional CLAUDE.md | Codumented CLAUDE.md |
|---|---|
| No feedback on effectiveness | Stats show what works |
| Grows forever | Can be minimized via data |
| AI silently ignores unclear rules | Failures catch violations |
| Human reviews catch issues | Automated enforcement |
When an AI violates a rule, it sees the errorContent—a teaching mechanism that explains what went wrong, why it matters, and how to fix it.
[Validation Failure]
"You added a new API route in app/api/users/route.ts but didn't include a Zod schema. All routes must validate input to prevent injection attacks. Please import z and export a schema."
The agent becomes a student that improves through structured feedback.
Key Insights
- 1Documentation is executable specification—not just text.
- 2The "Hope Workflow" is the primary cause of technical debt in AI context files.
- 3Validation failures are curriculum—they teach agents via errorContent.
- 4Minimal effective context is achieved through measurement, not guessing.
We built Codumentation because we were tired of documentation lying to us. When every claim is validated, documentation becomes a contract that code must honor.
Documentation should not decay silently. It should fail loudly, teach clearly, and improve continuously.
About the Author
Markus Hav
Markus Hav is Lead Researcher for Agents at Benque Max AI Lab in Finland, where he focuses on advancing autonomous AI systems and agent architectures. His work explores the boundaries between programmed behavior and emergent intelligence in AI agents. He also serves as Head of AI Automation at Hoxhunt, applying cutting-edge agent research to real-world automation challenges.