Programmatic: WTF Is robots.txt? From Search Crawlers to AI Bots
The Lion King was in cinemas, IAM's Le Mia was on French playlists, and the Web was young. The year was 1994. What could that era possibly have to do with today's AI-powered search? One tiny text file: robots.txt. 🤖
1. Back to 1994: when web crawlers became a problem
Internet connections were slow, search engines were just taking shape, and early crawler bots were exploring websites to build searchable indexes. That was useful—but it could also be disruptive. Automated requests consumed scarce bandwidth, and poorly configured crawlers could fetch far more content than site owners expected.
The answer was surprisingly simple: a publicly accessible text file, placed at the root of a website, giving crawlers instructions about which paths to avoid. The Robots Exclusion Protocol emerged in 1994, associated with the work of Dutch-born engineer Martijn Koster.
Fast-forward more than thirty years: the same basic mechanism now sits at the center of debates about AI training, content licensing and the future of independent publishers.
2. What exactly is robots.txt?
Robots.txt is a machine-readable set of crawling rules. A website typically publishes it at https://example.com/robots.txt. Crawlers that choose to follow the protocol can consult the file before fetching pages.
It helps manage crawler access; it does not authenticate visitors, encrypt content, or grant and revoke legal rights by itself. And contrary to a common misconception, a Disallow rule does not necessarily prevent a URL from being indexed if search engines discover that URL elsewhere.
You can inspect a real publisher's file in France by visiting lefigaro.fr/robots.txt (its contents may change).
3. Three directives every publisher should understand
| Directive | Purpose | Example |
|---|---|---|
User-agent | Names the crawler to which a rule group applies | User-agent: * |
Disallow | Asks that crawler not to fetch matching paths | Disallow: /private/ |
Allow | Permits an exception within otherwise restricted paths, for crawlers supporting the directive | Allow: /private/public-guide/ |
For example, a publisher might use the following:
User-agent: *
Disallow: /internal/
User-agent: ExampleBot
Disallow: /
The first group asks all crawlers to avoid /internal/. The second asks a hypothetical crawler named ExampleBot not to crawl the site at all (actual matching and precedence depend on the crawler's implementation and the protocol's rules).
Note that robots.txt is not the same thing as ads.txt: the former concerns crawling; the latter declares authorized digital advertising sellers.
4. A polite request, not a locked door
Robots.txt works because responsible crawlers respect it. A malicious scraper can ignore it completely. If a page contains genuinely sensitive information, the right protection is authentication or access control—not a line in robots.txt advertising where the sensitive material lives.
There are also important legal nuances. A crawler's refusal to comply may raise questions under copyright, contract, database rights or other laws depending on jurisdiction and circumstances; robots.txt alone does not determine whether scraping is lawful or unlawful.
And blocking a crawler can have side effects. For search engines, restricting access may affect how a site is understood and surfaced. Publishers should test changes carefully rather than applying blanket rules without understanding the consequences.
5. Why AI bots have turned a technical file into a strategic issue
Historically, publishers mainly thought about crawlers as the bots that helped search engines discover their pages. Now, some bots collect material for AI model development, while others fetch pages to support AI-assisted answers and search experiences. These are different uses, even when they involve similar HTTP requests.
Here in France, where publishers are already confronting pressure on direct traffic and advertising revenue, the stakes are significant. An AI answer may satisfy a user's question without requiring a visit to the original article. But AI discovery can also create new ways for a publisher to be cited or found (the balance varies by query, platform and publisher).
That connects directly to our broader discussion of the Open Web's attention problem and the risk of building distribution on someone else's platform.
One key distinction: blocking a training crawler is not necessarily the same as blocking a search or retrieval crawler. Providers may operate different bots with different declared purposes. Publishers should review the current documentation for each relevant user-agent before making decisions.
6. Allow, restrict or negotiate? The publisher's dilemma
Publishers face a choice that is more commercial than technical: how can they preserve discoverability while retaining leverage over the use of their content?
| Approach | Potential benefit | Trade-off |
|---|---|---|
| Allow relevant crawlers | Maintain opportunities for discovery and referrals | Less control over some automated uses of content |
| Restrict selected user-agents | Differentiate between crawling purposes | Requires monitoring and depends on crawler compliance |
| Block broadly | Express a strong access preference | May reduce visibility or useful automated access |
| Negotiate licences or partnerships | Seek contractual terms and compensation | Depends on bargaining power and enforcement |
In my view, publishers should avoid treating this as a binary choice between “let everyone scrape everything” and “disappear from the AI ecosystem.” They can audit bot traffic, define access policies by use case, protect premium content appropriately, and explore commercial arrangements where possible.
And they should keep investing in their own audience relationships—an idea explored in our proposal for a Spotify-style super app for the press.
7. Why this matters to GreyFox Ads
At GreyFox Ads, we believe a healthy, independent Open Web is essential for advertisers and publishers alike. If original journalism becomes harder to fund, the supply of trustworthy editorial environments becomes harder to sustain too.
Our role is to help international Media Traders reach quality audiences here in France through a single Master Curated Deal, with attention to inventory quality, efficient activation and transparent economics. The debate about robots.txt is another reminder that the future of programmatic advertising depends on the future of publishers.
Final thought: In 1994, robots.txt was about asking crawlers not to overwhelm websites. In the AI era, it is also about who gets to use publishers' content—and on what terms. Quite a career for a tiny text file. 🤖
Activate quality French Open Web inventory
Looking for a straightforward way to reach French audiences through transparent programmatic curation?
Explore our Master DealContact GreyFox Ads