--- title: How to Serve Markdown to AI Agents Using Accept Headers url: "https://goremy.ai/blog/articles/how-to-serve-markdown-to-ai-agents-using-accept-headers" author: Luis Chavez-Mattos editor: Luis Chavez-Mattos editor_title: Director, Product published: "2026-09-01T19:00:11.623Z" updated: "2026-09-01T19:00:11.623Z" topic: AI Agents & Automation description: A plain-English guide to configuring wikis and internal tools with HTTP content negotiation so AI agents can read your data cleanly. --- # How to Serve Markdown to AI Agents Using Accept Headers > A plain-English guide to configuring wikis and internal tools with HTTP content negotiation so AI agents can read your data cleanly. Edited by Luis Chavez-Mattos, Director, Product · Published 2026-09-01 ## How do you serve Markdown to AI agents using Accept headers? To format a website or wiki for AI agents, serve a clean Markdown version of every page at the same URL your browser uses, and hand it over when the request's `Accept` header asks for `text/markdown`. This is called content negotiation. It has been part of HTTP since 1997, and it is the single highest-leverage change you can make to get your internal tools ready for AI agents. [1] Most internal wikis were built for people clicking around in a browser. Every page ships with navigation bars, sidebars, embedded scripts, and styling markup that a human never notices and an AI agent cannot use. When your custom assistant fetches that page to answer an employee's question, it has to wade through all of it first. ## TL;DR - Serving **Markdown instead of HTML** can cut an AI agent's token usage by about 80% (Cloudflare), and page size by over 99% on a typical page (Vercel). - Content negotiation uses the **`Accept` header** to serve Markdown from the same standard URLs your browser already uses. - Always set the **`Vary: Accept` header** so CDNs and browser caches do not serve the wrong format to the wrong client. - Only **three of seven** major coding agents currently request Markdown automatically, so adoption is still uneven. - Publishing an **`llms.txt` file** gives agents that do not negotiate yet a fallback route to the clean version. - If your wiki content is **already authored in Markdown**, you often need no conversion step, just a way to stop wrapping it in a template. ## Why does this matter more than it sounds like it should? When an AI agent requests a web page, it does not see what your browser renders. It sees raw HTML: div wrappers, class names, ARIA labels, tracking scripts, and nav items repeated on every single page. [2] None of that has any value to a language model, but every character of it counts against the model's context window and your token bill. The numbers are not subtle. Cloudflare measured an 80% reduction in tokens when serving Markdown instead of HTML. [2] Vercel found that a typical page dropped from roughly 500KB of HTML to about 2 to 3KB of Markdown, a reduction north of 99%. [3] Checkly ran its own test on its documentation and got nearly identical results: 615.4KB and 180,573 tokens of HTML versus 2.3KB and 478 tokens of Markdown for the same page. [2] **What Markdown saves an AI agent** - 80%: Token reduction serving Markdown vs HTML - 99.6%: Payload size reduction, Vercel blog page _Figures independently measured by Cloudflare and Vercel on their own production pages._ That gap is not cosmetic. It is the difference between an internal assistant that gives a fast, accurate answer from your wiki and one that burns its context budget parsing div soup, then hallucinates the parts it never actually read. ## How does content negotiation actually work? Content negotiation is a single mechanism: the client tells the server what format it wants using the `Accept` request header, and the server decides what to send back from the same URL. An agent's request looks like this: ``` Accept: text/markdown, text/html, */* ``` That header says: give me Markdown if you have it, otherwise HTML is fine. If your server understands `text/markdown`, it returns a clean, structured text version of the page. If not, the agent falls back to parsing HTML like it always did. Nothing breaks for anyone, human or machine. [3] [1] This is different from publishing a separate `.md` URL for every page, which requires the agent to already know that convention exists. Content negotiation needs no site-specific knowledge. Any agent that sends the right header gets Markdown automatically, from any site that supports it. [3] ## What are the four things to actually build? **1. Detect the header and route to a Markdown response.** On most modern frameworks this is a rewrite rule plus a route handler. Vercel's implementation, for example, checks whether the incoming `Accept` header contains `text/markdown` and routes the request to a dedicated Markdown endpoint instead of the default HTML page. [3] If your wiki content is already authored in Markdown, as most internal wikis are, you often do not need a conversion step at all. You just need to stop wrapping it in a template before you send it. **2. Set `Vary: Accept` on every response.** Without this header, any CDN or browser cache sitting in front of your wiki will happily cache the Markdown response and serve it to a human browser next, or vice versa. This is the single most common mistake teams make when they add content negotiation, and it is invisible until someone reports a broken page. [1] **3. Handle the edge cases honestly.** If a request's `Accept` header genuinely cannot be satisfied, return a proper `406 Not Acceptable` rather than guessing. And respect quality values (`q=`) when an agent ranks its preferences, for example `text/markdown;q=1.0, text/html;q=0.7`, rather than only checking whether the string appears anywhere in the header. [1] **4. Add a discovery path for agents that do not negotiate yet.** Not every agent sends the right `Accept` header today. Publish an `llms.txt` file at your site root that lists your key pages with links to their Markdown versions, and add a `` tag in your HTML head. Both give agents a fallback route to the clean version even without perfect content negotiation. [3] ## Does every agent ask for Markdown yet? A February 2026 test by Checkly sent the same request to seven common coding agents and checked which ones actually asked for Markdown. Claude Code, Cursor, and OpenCode all sent `Accept: text/markdown` and got the lean response. OpenAI Codex, Gemini CLI, GitHub Copilot, and Windsurf did not, either sending a plain `Accept: */*` or a browser-style HTML preference list. [2] **Which AI agents request Markdown via Accept header (Feb 2026 test)** | | Requests Markdown | | --- | --- | | Claude Code | 1 agents | | Cursor | 1 agents | | OpenCode | 1 agents | | OpenAI Codex | 0 agents | | Gemini CLI | 0 agents | | GitHub Copilot | 0 agents | | Windsurf | 0 agents | _1 = agent sent Accept: text/markdown when fetching a URL, 0 = it did not. Based on Checkly's direct header test of seven agent tools._ That is roughly three out of seven. It is a useful reminder that this is an emerging convention, not a finished standard, and that supporting it is still worth doing even though adoption is uneven. The agents that do send the header are already getting a faster, cheaper, more accurate experience out of your wiki. The ones that do not will eventually catch up, and your server will already be ready when they do. ## Where does this fit if you are building your own internal assistant? If your team has already built a custom AI assistant on top of your wiki, your ticketing system, or your internal docs, and it does not yet respect content negotiation, this is a low-cost, high-value fix. It sits squarely inside the same discipline as writing your own internal tools instead of renting someone else's black box: you control the format your data is served in, so you can make sure the model you built actually reads what you meant it to read. Tools built for exactly this kind of internal agent orchestration, like [Remy](https://goremy.ai), depend on your source systems being legible in the first place, which is why the plumbing described here matters even before you think about prompts or models. Start with your highest-traffic internal pages: the onboarding wiki, the runbook library, the API reference your support team's assistant leans on hardest. Add the header check, set `Vary: Accept`, and ship it. It is a few hours of engineering work that pays back every time an agent touches your docs from now on. ## Frequently Asked Questions ### What is the Accept header and why does it matter for AI agents? The Accept header is a standard part of every HTTP request that tells a server what format the client prefers. AI agents like Claude Code and Cursor increasingly send Accept: text/markdown, and a server that checks for this can return a clean, structured version of a page instead of full HTML, saving the agent tokens and parsing time. ### Do I need a separate URL for the Markdown version of my page? No. Content negotiation serves both the HTML and Markdown versions from the exact same URL. The server decides which one to return based on the Accept header, so humans and agents hit the same address and get the version that suits them. ### What is llms.txt and do I still need it if I support content negotiation? llms.txt is a file at your site root that lists your most important pages, similar to a curated sitemap for AI systems. It complements content negotiation rather than replacing it: the Accept header optimizes how a given page is served, while llms.txt helps an agent discover which pages exist and matter in the first place. ### Will serving Markdown break my site for regular visitors or search engines? No. Browsers and search crawlers do not send Accept: text/markdown, so they continue to receive the same HTML they always have. The Markdown response only goes to clients that explicitly ask for it. ### Which AI agents actually send the Accept: text/markdown header today? As of a February 2026 test, Claude Code, Cursor, and OpenCode requested Markdown by default. OpenAI Codex, Gemini CLI, GitHub Copilot, and Windsurf did not yet negotiate for it, though this is expected to change as the convention spreads. ## Sources 1. [Guides — Markdown content negotiation for AI agents](https://acceptmarkdown.com/guides) (acceptmarkdown.com) 2. [The Current State of Content Negotiation for AI Agents (Feb 2026)](https://www.checklyhq.com/blog/state-of-ai-agent-content-negotation/) (Checkly) 3. [Making agent-friendly pages with content negotiation](https://vercel.com/blog/making-agent-friendly-pages-with-content-negotiation) (Vercel)