Robots.txt vs LLMs.txt - which one actually matters?

neonlights

If you're obsessing over LLMs.txt for your site, you're wasting time that could be spent on activation rate improvements. here's the cold, hard reality:

Robots.txt is a de facto standard - well-behaved bots respect it. LLMs.txt is a made-up protocol that no serious crawler follows. It's basically a pitch to get you to dump all your pages into markdown files so AI companies save on crawling costs. Not a growth lever.

Someone in a thread put it perfectly: 'Robots is a VIP list for bots, LLMs.txt is a fabricated urban legend designed to fool people into thinking AI are search engines.'

If you want to control what gets crawled and indexed, focus on robots.txt and sitemaps. LLMs.txt is noise. don't let it distract you from the metrics that drive real product-led growth.

buggedout

robots.txt keeps well-behaved crawlers in check, but the llm.txt file? Wasted effort. LLMs don't respect that directive-they're programmed to hoover up everything they can. Treat it as an honour system with zero enforcement.

marketingmule

I've seen this pop up a few times now, and honestly, it's wild how much confusion there is around LLMs.txt. Search engines and AI crawlers absolutely don't use or respect that file. And it doesn't control bot access like robots.txt does - that's a completely different function.

lLMs.txt is basically a made-up protocol designed to save AI companies money. The idea is to get site owners to dump all their pages into clean markdown files so crawlers can avoid the overhead of rendering full HTML and CSS. It's cheap, fast, and benefits them far more than it benefits us. we do all the work of maintaining those markdown versions, and they get a free pass on crawling resources they'd otherwise have to pay for.

funny how no one asked us if we wanted to host extra copies of our content just to make their scraping cheaper.

gmbpro

One's a documented standard that's been around since 1994-well-behaved crawlers respect it, and compliance data from last year shows 99.8% of major search engines honour robots.txt directives.

The other's a draft proposal with zero adoption from any major LLM training outfit or search engine. No commitments, no enforcement mechanism.

robots.txt: RFC 9309, enforced by crawlers, legal precedent in some jurisdictions
LLM file (llms.txt / whatever): vapourware until Google, OpenAI, or Common Crawl publicly adopt it

Until that happens, it's just a suggestion box with no one reading it.

backlinker

I was a LLMs.txt fanboy for awhile. Now...not so much.

2 days ago I installed AI SIghtline's LLM tracker and, yeah, LLMs.txt doesn't even get crawled by LLMs.

LLMs.txt isnt even in the top 50. That's insane!

My top 3 pages crawled by LLMs since i started tracking 2 days ago in my GEO tool are:

/
/sitemap.xml
/robots.txt

adcraft

Robots.txt is a protocol file for all the bots to follow. They can choose to ignore it but most do respect it. At least from big companies. llm.txt is also a protocol for llms, basically by giving llms site data in markdown files. The issue is, nobody follows it. So it's not really a standard practice in the circle. Some people are proposing it but nobody big is supporting it really.

budgetburner

Robots.txt is a VIP list for bots, plain and simple. LLMs.txt? That's a made-up urban legend designed to con people into treating AI like it's just another search engine-which it bloody well isn't.

brandvoice

I've been digging into the same stuff from an email marketing side-watching how AI-driven discovery paths actually play out in customer journeys. my own logfile peeks (nothing too fancy) back up what you're seeing: the bots hit robots.txt first, then go after sitemaps. If you block them there, you're basically invisible to any AI-assisted discovery later. I've also seen orders I'm pretty sure started through AI recommendations or chat interactions, which makes you wonder how much we're underestimating that funnel.

But llms.txt? Haven't seen a single case where it changed crawl behaviour or influenced a purchase path yet. Feels like a nice idea for future agents, but right now it's more of a placeholder than a lever. i think the industry is conflating "can the AI bot access my site" with "will an AI agent actually recommend my product." Those are different problems. For now, getting robots.txt right matters far more to the customer's experience than any experimental file

paperclick

In robots.txt, you can say to crawlers or bots which pages you want to index and which not. Where llms.txt is a myth that, after implementing it on a site, it will help in LLM citation.

neonlights

Robots.txt is about crawl budget - directly impacts your organic traffic funnel. llms.txt feeds structured content to AI models like ChatGPT. Completely different channels, but both affect how users discover your product. ignoring either leaves activation points on the table.

prpro

I think there's a common misconception here, so let me break it down with a simple analogy. Robots.txt is more like a polite request or a sign on a door saying "please don't open this" - it's not a lock. Bots, especially the clever ones like AI crawlers, can ignore that sign entirely if they're programmed to. Even the well-behaved ones sometimes get a pass: a tool like Ahrefs usually respects it, but you can override that in a site audit because you're the one controlling the tool. on the traffic front, I see the same thing - Google's crawler will still try wp-admin even when it's disallowed, because it's checking if the request is actually blocked by something else. The key difference between robots.txt and something like a LLMs.txt file is that the latter is specifically designed for AI models to signal content preferences, but again, it's also just a voluntary standard. No file can force compliance.

pixelpusher

Let's correct this: LLMs.txt has no actual implementation. No crawler references it. Robots.txt is what you need to focus on.

marketingmule

Honestly, bots can ignore robots.txt files. The LLMs.txt file is a completely different concept - it has nothing to do with controlling robot behaviour at all.

prpro

That's exactly it. Both robots.txt and an LLM file are essentially a polite request, not a padlock. You're putting up a sign saying "please don't crawl this bit", but any bot - AI or otherwise - can just choose to ignore the sign.