What is a website in a post-AI marketplace?

There is a file you are now told to put on your website. It’s called llms.txt. Jeremy Howard of Answer.AI proposed it in 2024 for a sensible, narrow purpose: giving AI systems a clean version of a site’s content to read, mostly so coding agents could make sense of software documentation.

Since then it has been sold far more broadly, as something every site needs if it wants to be seen by answer engines.

A file sold as essential for AI visibility goes almost entirely unread

Earlier this year, Ahrefs checked the server logs of 137,000 domains to see what happens to those files. Twenty-eight percent of the sites had one — a sample that skews technical and SEO-aware, so treat that as a ceiling. Ninety-seven percent of those files received no requests at all in the month measured. Among the few that did, the AI systems that retrieve pages to answer questions accounted for about one percent of the traffic. The heaviest AI category was agentic tooling, and apart from OpenAI’s GPTBot and one indexing bot, Anthropic’s coding agent out-fetched every other AI system that came looking, which is roughly what the file’s author had in mind. The largest single category of requests came from generic SEO crawlers that hit the file incidentally on their way through everything else, nearly half of them Ahrefs’ own. Another twelve percent came from bots built to audit, validate and catalogue llms.txt files.

So the file does roughly what its author intended, for coding agents, and almost nothing of what it’s been sold to do.

As a visibility tactic, its most reliable audience is the industry that promotes it, checking on itself. Google’s John Mueller put the search side of it plainly a year earlier: “no AI system currently uses llms.txt.”

I keep thinking about that file, or rather about what it’s been turned into. It’s the purest artifact of a bad idea, which is that the job of a website in a post-AI world is to be more easily consumed by AI.

A website’s job now is to confirm what people heard elsewhere

The website’s job has changed, I think, but in nearly the opposite direction. People increasingly arrive at a website having already heard of you somewhere else — a recommendation, a citation in an answer, a colleague’s post. They come to check. So the question worth asking is less how a machine reads your site than what a person needs to find there to believe what they heard.

Adding schema markup didn’t improve AI citations

The machine-first advice has been consistent for two years: restructure your content for answer engines, add markup, add the file, or disappear. It has also been tested. Ahrefs tracked 1,885 pages that added structured data markup between August 2025 and March 2026, matched them against four thousand control pages, and measured their citations in AI answers. The effect on Google’s AI Mode was 2.4 percent, and on ChatGPT 2.2 percent — both “statistically indistinguishable from zero,” in the authors’ own words. On AI Overviews the effect was negative and statistically significant, though the authors decline to pin it on the schema.

The two remedies the industry sells hardest have been measured, and they do essentially nothing.

There’s a structural reason. Search was never fully knowable. Google has ranked with machine learning for more than a decade, and SEO was always part science and part superstition. But a ranking was at least observable: search the same phrase and you got roughly the same list, so you could measure where you stood, change something, and measure again. An entire profession grew up doing exactly that. An answer engine doesn’t hold still long enough to measure. Asking one a question is closer to asking a knowledgeable colleague — the phrasing changes the answer, the context changes the answer, and asking twice gets you two different responses.

Being talked about beats being optimized

You cannot game a colleague with markup. You can influence what they say about you, though, and everyone already knows how. Be genuinely known. Be cited by people they trust. Be worth recommending.

I don’t mean that metaphorically. When Ahrefs looked at what correlates with showing up in AI answers across 75,000 brands, YouTube mentions were the strongest signal of anything they measured, around 0.74 on a Spearman scale, with mentions across the rest of the web close behind at 0.66 to 0.71. Domain Rating landed between 0.27 and 0.33, and of raw backlink counts Ahrefs reports only “very weak correlations,” publishing no figure. Correlation isn’t causation, and they say so. Still, being talked about beat being optimized.

The irony underneath is worth thinking about. These systems learned to read websites by reading websites, overwhelmingly pages that people built for other people. Now we’re told to rebuild those pages so the machines can manage. What genuinely helps a machine parse a site — semantic structure, real text in the document, clear hierarchy, pages that load — is what you should have done for humans a decade ago. If you built it well, you’re already finished. Don’t buy into the hype. It’s wrong.

AI readers won’t replace human ones

The failed tactics aren’t what worry me, though. What worries me is the premise hiding underneath them.

If an agent can visit your site, parse it, and take what it needs, the reasoning goes, then a person never will. And if a person never will, the website doesn’t need to be a designed thing at all. It can be a pile of structured files. The visual experience was always overhead.

I’ve heard this “shape of argument “logic” a lot over the last couple of years. When AI started generating images and text, the immediate conclusion was that people were done making things. But a world in which a machine can write is a world in which both machines and people write, and a world in which a machine can read a website is one in which machines and people both read websites.

I don’t know about you, but when an agent cites a website, I like to check it. I realize I’m likely in the minority on that point, and perhaps increasingly so. But even when I trust the citation, I still want to see it for myself. I think that will hold for many people, in many contexts.

The real question is which sites people bother to visit

That doesn’t mean nothing changes. When something survives a new technology, its economics usually don’t: fewer people make a living from it, and the ones who do have to be good at something different. Websites will exist. The question is which ones anyone bothers to visit.

Search still happens, but fewer searches end in a visit

I have to correct something I’ve said myself here — that answer engines trump search engines. The numbers don’t bear that out…yet.

The intuitive story is that the audience moved to AI: people ask a chatbot instead of searching, and your traffic went with them. The numbers don’t support it. Ahrefs, looking at roughly 112,000 sites in August 2026, found Google Search accounting for 24.24 percent of measured traffic and ChatGPT 0.32 percent. Seventy-six to one — and that flatters traditional search slightly, since clicks from Google’s own AI Overviews are counted as search.

Search demand didn’t collapse either. Gartner predicted in February 2024 that traditional search volume would fall a quarter by 2026; instead Google’s search revenue grew seventeen percent year over year last quarter, down from nineteen the quarter before. Revenue is not volume, and Google’s claim that its AI features are driving more queries is a self-report with no numbers behind it. Still, nobody has produced evidence of the collapse.

What changed is narrower and stranger: the searches still happen, but they stop producing visits. Rand Fishkin, working from Similarweb’s clickstream panel, put zero-click at 68 percent of US Google searches in the first four months of 2026; a 2024 measurement on a different panel put it around 60 percent, and he’s careful to say the two aren’t strictly comparable. The first randomized evidence on the question is more persuasive — 1,065 US desktop Chrome users, 68,000 searches, run by researchers at the Indian School of Business and Carnegie Mellon. When an AI Overview appeared, outbound organic clicks fell 39.8 percent. It’s a working paper, not yet peer-reviewed, and Google has published nothing to rebut it. Basically, Google injects AI into search, muddying the data water.

Most B2B buyers pick a favorite before they contact you

The audience is still out there, looking and forming opinions. They’re just forming them somewhere other than your site, and by the time they arrive, they’ve usually decided. 6sense — which sells software premised on exactly this problem — surveyed more than four thousand buyers last year and found that 94 percent of buying groups had ranked a preferred vendor before making first contact, and bought from that preliminary favorite 77 percent of the time.

Visitors now ask a fourth question: did a person make this?

Which brings me back to what a website is for.

For the last twenty years, the strategy was one continuous motion, all of it on the site: attract an audience, inform them, engage them. The site had to do everything because the site was where everything happened. It still has to inform and engage. It no longer does the attracting.

Visitors decide what a page is, and whether it’s for them, within seconds

I’ve told clients for years that within the first few seconds of a page loading, a visitor asks and answers three questions without knowing they’re doing it. What is this? Is it relevant to me? What should I do next? Everything about the page — its structure, its density, its visual language — either makes those answers clear or doesn’t.

Those questions haven’t changed. There’s a fourth one now, and it comes before the others: did a person make this?

Proving a person made your site takes evidence that’s expensive to fake

That question is harder to answer than it looks, because a website is precisely the artifact where humanity is cheapest to counterfeit. A model can produce a warm, idiosyncratic, human-seeming page faster than a person can, including the small imperfections that read as authorship. Looking human is the cheapest thing on the menu.

So proof of humanity has to rest on what’s expensive to fake rather than on tone: specificity that only comes from having done the work, a named person with a traceable history, claims that can be checked somewhere other than this page, a position that costs something to hold, evidence of accumulated time.

The example I know best is my own. I’ve published essays here on Newfangled.com for twenty years, and on my own website (chrbutler.com) for almost as long. What matters is what an archive does that a page can’t. It’s dated. It’s checkable. It holds positions I took years ago, still sitting there to be defended or revised. A model generating a thought-leadership page this afternoon has no old positions to answer for. Nobody fully reads an archive that deep before deciding whether to call someone, but its existence answers the fourth question before the visitor has finished asking it.

You can start building that evidence on day one

The obvious objection is that most people don’t have an archive like that, and someone starting out now can’t conjure one. That’s fair, and time is the one proof nobody can buy.

But an archive matters less for its length than for its properties, and most of those are available on the first day. Be specific in the way only someone who did the work could be: the decision you made and why, what went wrong first, the detail a summary would smooth away. Put your name on it, and connect that name to things you don’t control — a work history, people who will take a call, places you’ve been cited. Hold a position some visitors will disagree with, since generated content drifts toward positions nobody could object to. And date what you publish, then leave it standing; an archive starts with its first entry, and what makes it evidence later is that you didn’t quietly revise it along the way. The fastest proof of all, for someone new, is other people vouching for you — which is to say the same off-site signal the website exists to confirm.

That turns out to be the same material as the other proofs a visitor is looking for. Proof of fit: that you work with people like them, on problems like theirs. Proof of quality: visible in the thing itself, in how your work is made. Proof of results: checkable, attributable, someone’s name on it. Validation is an act of checking, and all four proofs are answered by the same evidence. If the presence of an archive proves the presence of a human mind, the presence of testimonials and case studies prove that other humans have trusted it.

The preference is real and it is measured. The Reuters Institute surveyed six countries in mid-2025 and found 12 percent of people comfortable with news made entirely by AI, against 62 percent comfortable with news made entirely by a human. Gallup found the same gradient in a commercial setting this May: three-quarters of Americans accept AI for brainstorming and early drafts when that use is disclosed, 53 percent accept it producing the final copy or images, and 62 percent reject AI-generated people in advertising even when they’re told.

Meanwhile Pew, which classified Common Crawl pages back to 2021 and drew a random sample of the July 2026 web, found that about ten percent of English-language pages show significant signs of AI authorship. Among pages published since ChatGPT’s release, more than a third do. Detection is unreliable on any single page, Pew is careful to say, though it holds up in aggregate. People are drafting with it and publishing it, and they’d rather you didn’t.

What makes a site credible to people also makes it legible to machines

There is a tension in all this, isn’t there? I’ve said the AI part takes care of itself. But an answer engine is increasingly likely to be what decides whether a person ever arrives to validate anything. For now the likelier referrer is still a colleague’s post on LinkedIn, or a line in a newsletter. Either way something upstream is doing the deciding, and that sounds like it ought to be the whole game.

It isn’t, and the reason is the thing I keep coming back to. What makes a site legible to a machine and what makes it credible to a person turn out to be the same properties. Clear structure. Real substance. Claims that survive checking. Evidence that someone was here. There is no version of this where you optimize for the machine and the human loses, or the reverse. There’s only the version where you do the work, and the version where you add a file nobody reads.

The site that wins the next decade won’t be the one that was easiest for an agent to parse. It’ll be the one that was worth the visit after the agent was done.