llms.txt Hype vs. AI Agent Reality
This episode cuts through the hype around Generative Engine Optimization and llms.txt, showing why the file matters far more for business-to-agent routing than for search rankings. The hosts also explore how to design WordPress sites for AI agents, measure agent traffic through server logs, and keep clean Markdown, JSON-LD, and APIs accessible behind the scenes.
Show Notes
- https://thewpweekly.com/feed/: https://thewpweekly.com/feed/
Chapter 1
The GEO Mirage and the Reality of LLM Files
Nick d
So everybody has been talking about Generative Engine Optimization, or GEO, like it is the shiny new gold rush. Agencies are out here charging thousands of dollars to build this single file, llms dot txt, telling site owners it is the secret key to ranking number one on ChatGPT or Perplexity.
Ido
Wait, wait, but, but is it not? I mean, everyone is saying that file is basically the new sitemap for AI crawlers.
Nick d
Well, that is the hype. But if you look at actual server data, out of over five hundred million LLM crawler visits analyzed in 2026, search bots like GPTBot or PerplexityBot fetched that file exactly four hundred and eight times. Four hundred and eight times out of five hundred million!
Ben cohen
Four hundred and eight times! That is, uh, that is practically zero. It is a rounding error!
Nick d
Exactly. And SE Ranking ran a study on three hundred thousand domains and found absolutely zero statistical correlation between having an llms dot txt file and getting cited more often in AI search responses. In fact, John Mueller over at Google outright called it the modern equivalent of meta keywords.
Ido
Ouch. So search engine engineers are saying, look, our crawlers just read the raw HTML directly, stop overcomplicating it.
Nick d
Right. But, but here is where people get confused, because the file is not totally useless. It is just being completely sold for the wrong job. It is not for search engines. It is for Business to Agent routing.
Ben cohen
Right, right, exactly! When you use developer tools like Cursor or Claude Code, or LangChain with their mcpdoc server, those coding agents fetch the Markdown file directly. Why? Because they do not want to parse two megabytes of HTML headers, scripts, and navigation menus. They just want clean, raw documentation. It cuts token overhead by ten times!
Ido
Ah, so it is, it is an agent navigation file for actual task execution, not a marketing gimmick for search ranking.
Nick d
Precisely. But even if you do want to set it up for agents, WordPress site owners are shooting themselves in the foot. Plugins like Yoast SEO and Rank Math added these one click toggles to generate the file, which is great. But then site owners leave their robots dot txt file configured to block user agents like ClaudeBot or GPTBot from accessing the root directory in the first place.
Ben cohen
They block the very bots at the door while hanging a welcome sign on the window! I, I see this all the time.
Chapter 2
Designing WordPress for AI Agents
Ben cohen
And that really brings us to the core issue of how we build WordPress sites today. Most themes are designed for human eyeballs, packed with heavy page builders, DOM clutter, popups, and scripts that completely exhaust an AI agent's context window. That is why we built the free AI Content Reader at GetAISO, to strip away all CSS and JavaScript and show webmasters what an AI assistant actually sees when it visits their page.
Ido
And when you strip away the pretty visual layer, a lot of sites look like a jumbled mess of navigation links and consent banners before you even get to the actual content, right?
Ben cohen
Mm, completely. It is pure noise to an LLM. But when you clean up that site architecture, incredible things happen. Benjamin Intal, founder of Stackable and WP Interactions, shared Ahentic AI workspace experiment recently. And it shows how an AI agent can understand an entire WordPress site, help launch it, and orchestrate plugin setups locally.
Nick d
Wait, with, with no external servers? No third party MCP bridge?
Ben cohen
None at all! Just pure WordPress, clean site architecture, and the core WordPress APIs. The agent just talks directly to the site.
Ido
That is awesome. But, okay, how do site builders or agencies actually measure if agents are visiting? Because Google Analytics does not show Claude Code or custom terminal agents browsing your pages.
Nick d
You have to look at server logs. Traditional analytics scripts rely on JavaScript executing in a browser, which headless AI agents rarely run. So you have to monitor your CDN access logs for direct hits to llms dot txt, or even embed single use honeypot URLs inside your Markdown indexes to track when an agent follows a link.
Ben cohen
That honeypot trick is brilliant. Moving forward into 2026, site builders have to treat raw Markdown delivery and structured JSON LD schema with the exact same priority as visual theme design. If an agent cannot read your site cleanly, your site does not exist to the automated web.
Ido
So build for human eyes on the front, but keep clean Markdown and clear APIs open for the bots in the back. Sounds like a solid plan. Good chatting, guys!
Nick d
Talk soon!