Menu

Programmatic: WTF Is robots.txt? From Search Crawlers to AI Bots

Robots.txt rules for search engine crawlers and AI bots on publisher websites

The Lion King was in cinemas, IAM's Le Mia was on French playlists, and the Web was young. The year was 1994. What could that era possibly have to do with today's AI-powered search? One tiny text file: robots.txt. 🤖

In a nutshell: A robots.txt file tells compliant web crawlers which URLs they may request. It is a public, voluntary protocol—not a security barrier, not a copyright licence, and not a guarantee that a page will never appear in search results. With AI crawlers entering the picture, publishers must decide which kinds of access they want to permit.

1. Back to 1994: when web crawlers became a problem

Internet connections were slow, search engines were just taking shape, and early crawler bots were exploring websites to build searchable indexes. That was useful—but it could also be disruptive. Automated requests consumed scarce bandwidth, and poorly configured crawlers could fetch far more content than site owners expected.

The answer was surprisingly simple: a publicly accessible text file, placed at the root of a website, giving crawlers instructions about which paths to avoid. The Robots Exclusion Protocol emerged in 1994, associated with the work of Dutch-born engineer Martijn Koster.

Fast-forward more than thirty years: the same basic mechanism now sits at the center of debates about AI training, content licensing and the future of independent publishers.

2. What exactly is robots.txt?

Robots.txt is a machine-readable set of crawling rules. A website typically publishes it at https://example.com/robots.txt. Crawlers that choose to follow the protocol can consult the file before fetching pages.

It helps manage crawler access; it does not authenticate visitors, encrypt content, or grant and revoke legal rights by itself. And contrary to a common misconception, a Disallow rule does not necessarily prevent a URL from being indexed if search engines discover that URL elsewhere.

You can inspect a real publisher's file in France by visiting lefigaro.fr/robots.txt (its contents may change).

3. Three directives every publisher should understand

DirectivePurposeExample
User-agentNames the crawler to which a rule group appliesUser-agent: *
DisallowAsks that crawler not to fetch matching pathsDisallow: /private/
AllowPermits an exception within otherwise restricted paths, for crawlers supporting the directiveAllow: /private/public-guide/

For example, a publisher might use the following:

User-agent: *
Disallow: /internal/

User-agent: ExampleBot
Disallow: /

The first group asks all crawlers to avoid /internal/. The second asks a hypothetical crawler named ExampleBot not to crawl the site at all (actual matching and precedence depend on the crawler's implementation and the protocol's rules).

Note that robots.txt is not the same thing as ads.txt: the former concerns crawling; the latter declares authorized digital advertising sellers.

4. A polite request, not a locked door

Robots.txt works because responsible crawlers respect it. A malicious scraper can ignore it completely. If a page contains genuinely sensitive information, the right protection is authentication or access control—not a line in robots.txt advertising where the sensitive material lives.

There are also important legal nuances. A crawler's refusal to comply may raise questions under copyright, contract, database rights or other laws depending on jurisdiction and circumstances; robots.txt alone does not determine whether scraping is lawful or unlawful.

And blocking a crawler can have side effects. For search engines, restricting access may affect how a site is understood and surfaced. Publishers should test changes carefully rather than applying blanket rules without understanding the consequences.

5. Why AI bots have turned a technical file into a strategic issue

Historically, publishers mainly thought about crawlers as the bots that helped search engines discover their pages. Now, some bots collect material for AI model development, while others fetch pages to support AI-assisted answers and search experiences. These are different uses, even when they involve similar HTTP requests.

Here in France, where publishers are already confronting pressure on direct traffic and advertising revenue, the stakes are significant. An AI answer may satisfy a user's question without requiring a visit to the original article. But AI discovery can also create new ways for a publisher to be cited or found (the balance varies by query, platform and publisher).

That connects directly to our broader discussion of the Open Web's attention problem and the risk of building distribution on someone else's platform.

One key distinction: blocking a training crawler is not necessarily the same as blocking a search or retrieval crawler. Providers may operate different bots with different declared purposes. Publishers should review the current documentation for each relevant user-agent before making decisions.

6. Allow, restrict or negotiate? The publisher's dilemma

Publishers face a choice that is more commercial than technical: how can they preserve discoverability while retaining leverage over the use of their content?

ApproachPotential benefitTrade-off
Allow relevant crawlersMaintain opportunities for discovery and referralsLess control over some automated uses of content
Restrict selected user-agentsDifferentiate between crawling purposesRequires monitoring and depends on crawler compliance
Block broadlyExpress a strong access preferenceMay reduce visibility or useful automated access
Negotiate licences or partnershipsSeek contractual terms and compensationDepends on bargaining power and enforcement

In my view, publishers should avoid treating this as a binary choice between “let everyone scrape everything” and “disappear from the AI ecosystem.” They can audit bot traffic, define access policies by use case, protect premium content appropriately, and explore commercial arrangements where possible.

And they should keep investing in their own audience relationships—an idea explored in our proposal for a Spotify-style super app for the press.

7. Why this matters to GreyFox Ads

At GreyFox Ads, we believe a healthy, independent Open Web is essential for advertisers and publishers alike. If original journalism becomes harder to fund, the supply of trustworthy editorial environments becomes harder to sustain too.

Our role is to help international Media Traders reach quality audiences here in France through a single Master Curated Deal, with attention to inventory quality, efficient activation and transparent economics. The debate about robots.txt is another reminder that the future of programmatic advertising depends on the future of publishers.

Final thought: In 1994, robots.txt was about asking crawlers not to overwhelm websites. In the AI era, it is also about who gets to use publishers' content—and on what terms. Quite a career for a tiny text file. 🤖

About the author

Max Legrand is the founder of GreyFox Ads, a French programmatic ad network specializing in curation and the quality Open Web.

Originally published as a LinkedIn post. This article expands on the original analysis with technical and legal distinctions.

Activate quality French Open Web inventory

Looking for a straightforward way to reach French audiences through transparent programmatic curation?

Explore our Master DealContact GreyFox Ads