How to Serve Markdown to AI Agents Using Accept Headers
A plain-English setup guide for making your internal wikis and tools readable by the AI assistants your team already built.
- 01Serving Markdown instead of HTML can reduce an AI agent's token usage by over 99%.
- 02Content negotiation uses the Accept header to serve Markdown from standard URLs.
- 03Always set the Vary: Accept header to prevent CDNs from caching the wrong format.
- 04Only three of seven major coding agents currently request Markdown automatically.
The short answer
To format a website or wiki for AI agents, serve a clean Markdown version of every page at the same URL your browser uses, and hand it over when the request's Accept header asks for text/markdown. This is called content negotiation. It has been part of HTTP since 1997, and it is the single highest-leverage change you can make to get your internal tools ready for AI agents.1
Most internal wikis were built for people clicking around in a browser. Every page ships with navigation bars, sidebars, embedded scripts, and styling markup that a human never notices and an AI agent cannot use. When your custom assistant fetches that page to answer an employee's question, it has to wade through all of it first.
Why this matters more than it sounds like it should
When an AI agent requests a web page, it does not see what your browser renders. It sees raw HTML: div wrappers, class names, ARIA labels, tracking scripts, and nav items repeated on every single page.2 None of that has any value to a language model, but every character of it counts against the model's context window and your token bill.
The numbers are not subtle. Cloudflare measured an 80% reduction in tokens when serving Markdown instead of HTML.2 Vercel found that a typical page dropped from roughly 500KB of HTML to about 2 to 3KB of Markdown, a reduction north of 99%.3 Checkly ran its own test on its documentation and got nearly identical results: 615.4KB and 180,573 tokens of HTML versus 2.3KB and 478 tokens of Markdown for the same page.2
That gap is not cosmetic. It is the difference between an internal assistant that gives a fast, accurate answer from your wiki and one that burns its context budget parsing div soup, then hallucinates the parts it never actually read.
How content negotiation actually works
Content negotiation is a single mechanism: the client tells the server what format it wants using the Accept request header, and the server decides what to send back from the same URL.
An agent's request looks like this:
Accept: text/markdown, text/html, */*
That header says: give me Markdown if you have it, otherwise HTML is fine. If your server understands text/markdown, it returns a clean, structured text version of the page. If not, the agent falls back to parsing HTML like it always did. Nothing breaks for anyone, human or machine.31
This is different from publishing a separate .md URL for every page, which requires the agent to already know that convention exists. Content negotiation needs no site-specific knowledge. Any agent that sends the right header gets Markdown automatically, from any site that supports it.3
The four things to actually build
1. Detect the header and route to a Markdown response. On most modern frameworks this is a rewrite rule plus a route handler. Vercel's implementation, for example, checks whether the incoming Accept header contains text/markdown and routes the request to a dedicated Markdown endpoint instead of the default HTML page.3 If your wiki content is already authored in Markdown, as most internal wikis are, you often do not need a conversion step at all. You just need to stop wrapping it in a template before you send it.
2. Set Vary: Accept on every response. Without this header, any CDN or browser cache sitting in front of your wiki will happily cache the Markdown response and serve it to a human browser next, or vice versa. This is the single most common mistake teams make when they add content negotiation, and it is invisible until someone reports a broken page.1
3. Handle the edge cases honestly. If a request's Accept header genuinely cannot be satisfied, return a proper 406 Not Acceptable rather than guessing. And respect quality values (q=) when an agent ranks its preferences, for example text/markdown;q=1.0, text/html;q=0.7, rather than only checking whether the string appears anywhere in the header.1
4. Add a discovery path for agents that do not negotiate yet. Not every agent sends the right Accept header today. Publish an llms.txt file at your site root that lists your key pages with links to their Markdown versions, and add a <link rel="alternate" type="text/markdown"> tag in your HTML head. Both give agents a fallback route to the clean version even without perfect content negotiation.3
Not every agent asks for it yet
A February 2026 test by Checkly sent the same request to seven common coding agents and checked which ones actually asked for Markdown. Claude Code, Cursor, and OpenCode all sent Accept: text/markdown and got the lean response. OpenAI Codex, Gemini CLI, GitHub Copilot, and Windsurf did not, either sending a plain Accept: */* or a browser-style HTML preference list.2
That is roughly three out of seven. It is a useful reminder that this is an emerging convention, not a finished standard, and that supporting it is still worth doing even though adoption is uneven. The agents that do send the header are already getting a faster, cheaper, more accurate experience out of your wiki. The ones that do not will eventually catch up, and your server will already be ready when they do.
Where this fits if you are building your own internal assistant
If your team has already built a custom AI assistant on top of your wiki, your ticketing system, or your internal docs, and it does not yet respect content negotiation, this is a low-cost, high-value fix. It sits squarely inside the same discipline as writing your own internal tools instead of renting someone else's black box: you control the format your data is served in, so you can make sure the model you built actually reads what you meant it to read. Tools built for exactly this kind of internal agent orchestration, like Remy, depend on your source systems being legible in the first place, which is why the plumbing described here matters even before you think about prompts or models.
Start with your highest-traffic internal pages: the onboarding wiki, the runbook library, the API reference your support team's assistant leans on hardest. Add the header check, set Vary: Accept, and ship it. It is a few hours of engineering work that pays back every time an agent touches your docs from now on.
The Accept header is a standard part of every HTTP request that tells a server what format the client prefers. AI agents like Claude Code and Cursor increasingly send Accept: text/markdown, and a server that checks for this can return a clean, structured version of a page instead of full HTML, saving the agent tokens and parsing time.
No. Content negotiation serves both the HTML and Markdown versions from the exact same URL. The server decides which one to return based on the Accept header, so humans and agents hit the same address and get the version that suits them.
llms.txt is a file at your site root that lists your most important pages, similar to a curated sitemap for AI systems. It complements content negotiation rather than replacing it: the Accept header optimizes how a given page is served, while llms.txt helps an agent discover which pages exist and matter in the first place.
No. Browsers and search crawlers do not send Accept: text/markdown, so they continue to receive the same HTML they always have. The Markdown response only goes to clients that explicitly ask for it.
As of a February 2026 test, Claude Code, Cursor, and OpenCode requested Markdown by default. OpenAI Codex, Gemini CLI, GitHub Copilot, and Windsurf did not yet negotiate for it, though this is expected to change as the convention spreads.
