Is llms.txt worth adding to your website?

Whether llms.txt is worth adding depends on your actual goal and how much you're willing to gamble on a file format that AI companies might ignore tomorrow. The short answer: it probably won't hurt, but it won't replace real content strategy either.

llms.txt is a text file you place in your site root that tells AI crawlers what they're allowed to do on your domain. It's similar in spirit to robots.txt but written specifically for AI systems. The idea is that by being explicit and welcoming, you might get better crawling or indexing. In practice, no one can guarantee that. AI companies don't publish how they decide what to crawl, what to index, or what to cite. Adding llms.txt might signal cooperation, but it's not a ranking factor and it's not a contract.

What llms.txt actually does

llms.txt is a plain text file that sits at yourdomain.com/llms.txt. It uses a simple format to tell AI crawlers which pages they can fetch, which they should skip, and optionally where to find your privacy policy or terms. You can allow all crawlers, block specific ones, or set rules by path. For example, you might allow GPTBot and PerplexityBot but block ClaudeBot. You might exclude your admin pages or customer account areas.

The file is purely advisory. AI companies did not invent it together or agree on enforcement. Some crawlers check it. Some don't. Some check it sometimes. There's no standard, no audit, and no way to know if a crawler is actually reading your file or just ignoring it. If you want to be sure a crawler doesn't access certain pages, robots.txt is older and more widely respected, though still not guaranteed.

One practical thing: if you use Cloudflare, it began blocking AI crawlers by default on new domains in 2025. Many site owners have that setting turned on without realizing it. If you want AI crawlers to reach your site at all, you may need to adjust your Cloudflare settings first. llms.txt won't override that.

Does it help with AI search visibility?

This is where the honest answer gets uncomfortable. Adding llms.txt does not guarantee that your pages will appear in ChatGPT, Perplexity, Claude, or Google AI Overviews. A crawler requesting your page proves it fetched the page, not that the page was used, ranked, or cited anywhere.

AI search engines use their own indexing logic. Some crawl the web proactively ahead of time, building an index before anyone asks a question. Others fetch pages on demand when a person asks a question. Whether they cite your content depends on relevance, quality, and how the AI system was built. No file format changes that.

llms.txt might make crawlers more efficient by signaling which pages matter. It might also do nothing. The AI companies haven't published research on whether llms.txt adoption correlates with better indexing or visibility. Until they do, any claim that llms.txt improves your AI search ranking is speculation.

What you can actually measure is whether crawlers are visiting your site. Dominaition includes a WordPress plugin that logs server-side requests from AI crawlers. Because the plugin runs in PHP on the server, it sees every request that reaches WordPress, including requests from crawlers that don't run JavaScript and never appear in Google Analytics. That data is real. Whether llms.txt caused the visits is harder to prove.

Should you add it anyway?

If you're already doing content strategy well, llms.txt is low friction. It takes a short time to create and deploy. It signals that you're aware of AI crawlers and you're willing to cooperate. That's not nothing.

But it's not a substitute for the work that actually moves the needle. You need pages that answer questions your competitors' pages don't. You need those pages to be well-written, accurate, and genuinely useful. You need them indexed by search engines and crawled by AI systems. llms.txt doesn't do any of that. It just says "you're allowed to be here."

If you're not doing that foundational work, llms.txt is a distraction. If you are doing it, llms.txt is a small courtesy that might help a little. The risk is minimal. The upside is uncertain but not zero.

The robots.txt question

You probably already have a robots.txt file. It tells Google, Bing, and other crawlers where they can and can't go. llms.txt is supposed to do something similar for AI crawlers, but it's not a replacement for robots.txt. Google still uses robots.txt. So does Bing. So do some AI crawlers.

If you want to block a crawler, you can do it in robots.txt. If you want to allow it, robots.txt usually allows everything by default unless you explicitly block it. llms.txt gives you more granular control if you want it, and it's specifically designed for AI companies. But it's a newer convention with less adoption.

The practical move: keep your robots.txt clean and simple. Add llms.txt if you want to be explicit about AI crawler access. Don't expect either file to be a magic switch.

What about your Cloudflare settings?

If you use Cloudflare, check your security settings. Cloudflare began blocking AI crawlers by default on new domains in 2025. Many site owners have that setting on without knowing it. If you want AI crawlers to reach your site, you need to allow them in Cloudflare first. Cloudflare also offers a managed robots.txt feature that can help you control crawler access without editing files manually.

If Cloudflare is blocking crawlers at the edge, llms.txt won't help. The request never reaches your server. So if you're adding llms.txt, make sure your Cloudflare settings aren't blocking the crawlers you want to allow. This is a common reason a site appears invisible to AI crawlers despite having all the right files in place.

The real work: content that answers questions

llms.txt is a tactical detail. The strategic work is knowing what questions your audience is asking and whether your competitors' pages already answer them. If they do and yours doesn't, you have a gap. That gap is where AI search visibility comes from.

The process is straightforward. Research the questions your competitors answer that you don't. Write articles that answer those questions completely and accurately. Publish them to your site. Track which AI crawlers request them. Pay attention to whether they show up in AI search results over time. That's the real game.

llms.txt is a small part of that picture. It might make crawlers slightly more efficient at finding what you've published. It signals cooperation. But it's not the reason your pages get indexed or cited. Content quality and relevance are.

The objection: "Isn't this just another SEO tactic?"

Yes and no. SEO is about making your site visible in search results. AI search is a new channel with different rules that are still being written. llms.txt is a tactic within that space, but it's not a replacement for real content strategy.

The difference is that llms.txt is about permission, not ranking. You're not optimizing for an algorithm. You're just saying "crawlers are welcome here and here's what you should know." That's different from keyword density or backlink building.

But the core principle is the same: you need good content that answers real questions. llms.txt doesn't create that. It just makes it easier for crawlers to find what you've already built. So yes, add it if you want. But don't confuse it with the actual work of building a content strategy that holds up in AI search.

About Dominaition

Dominaition is AI search visibility software for agencies and small businesses in the United States and Canada. It researches the questions competitors answer that your site doesn't, writes and edits articles for those gaps, and publishes them to WordPress on a schedule or on autopilot. The platform includes a server-side AI crawler log that shows which AI systems are requesting your pages, so you can see what's actually happening.