Test Lab / Data study 5 Aug 2026
105 European financial firms. Two decided anything about AI crawlers.
I fetched the robots.txt of 105 European banks, asset managers, insurers, fintechs and brokers. The result is not that European finance is locking AI out. It is that almost nobody has decided anything. The AI Act rules this collides with are live obligations for marketing teams.
why this matters: AI answers first, and the Observatory shows how.
Your AI visibility is not a strategy. It is a side effect: whatever a rule written for Googlebot, by someone who has probably left, happens to let through.
The matrix: who decided, who inherited
One row per firm, one column per citation-critical crawler. Green means allowed, amber means partially restricted, red means blocked site-wide, grey means the robots.txt never answered. Hover any cell for the source of the rule.
partial
blocked
no robots
| Firm | GPTBot | OAI-Search | ClaudeBot | Perplexity | Google-Ext. | CCBot |
|---|
Read it as a wall: almost every cell is amber inherited from a wildcard. The two firms that named a crawler, and the one that blocked on purpose, stand out immediately.
The firm explorer
Search and sort the full 105-firm table. The dataset is the audit itself: every robots.txt was fetched, parsed as a crawler would parse it, and saved raw.
| Firm | Segment | Country | robots.txt | names AI | blocks critical | llms.txt |
|---|
Three readings from the same file
Every crawler verdict, the llms.txt status, and the segment pattern, drawn from the same 105 rows.
Verdicts by crawler
How all 105 firms treat each citation-critical crawler. Explicit allows are rare; almost everything is inherited or partial.
The robots.txt verdict
What the 85 readable files actually do.
llms.txt status
Who serves a real file, who dresses up an HTML page as one, and who has no file.
The pattern by segment
Readable, fully open, and naming an AI crawler, per segment. The ratio is the same everywhere.
The operator read
Run the numbers together and the shape is clear. European finance has not made an anti-AI choice. It has made no choice. Eighty-three of 85 readable firms are one wildcard rule away from either opening the door properly or closing it on purpose, and almost none of them know which way that rule currently points.
I have argued before that volume is not the game in this niche, and that the winners are the firms doing the unglamorous structural work before the message. This is another instance of exactly that. The AI-visibility advantage in European financial services over the next year will not go to whoever produces the most content. It will go to whoever first treats their crawler policy as something a human decided, rather than something a 2019 SEO rule is quietly deciding for them. That is a half-day of work and a conversation with whoever owns the robots file. Almost nobody has had it.
Which is the recurring lesson from the regulated seat: the edge is rarely capability. It is looking at the thing everyone assumes is handled and finding out that it is not.
Method and limits
Sample: 105 European financial firms across 15 countries: 27 asset managers, 25 banks, 13 insurers and reinsurers, 10 fintechs, 8 brokers, plus pensions and bank-AM hybrids. A curated list of the largest and most visible names, not a random sample.
Collection: each firm’s /robots.txt and /llms.txt fetched over HTTPS with redirects followed and a browser user-agent, on 5 August 2026. Raw files saved per domain.
Parsing: robots.txt read as grouped User-agent blocks. A crawler is blocked only on a site-wide Disallow: / with no more specific Allow, restricted on any narrower disallow, and otherwise falls back to the wildcard group. String presence alone was never treated as a block.
Limits: 20 firms returned a bot challenge or no file to a scripted request and are reported as unreadable, not as blocking. robots.txt changes silently, so these figures are a single-day snapshot. llms.txt presence was content-checked to exclude HTML pages served at a 200.
Frequently asked
How many European financial firms block AI crawlers?
Almost none by explicit rule. In an audit of 105 European banks, asset managers, insurers, fintechs and brokers, only 85 served a machine-readable robots.txt, and of those just two named any AI crawler at all. A single firm, the Italian asset manager Anima Holding, blocked citation-critical crawlers site-wide. The rest inherit whatever their generic wildcard rule allows, which was almost always written for Googlebot long before AI crawlers existed.
Does having no AI rule in robots.txt mean AI can read your site?
Usually yes. If robots.txt does not name a crawler such as GPTBot or ClaudeBot, that crawler falls back to the wildcard user-agent group. In this audit 28 of 85 readable sites left the wildcard fully open, and 56 carried path-level disallows written for generic bots that also happen to restrict AI crawlers from those same paths. Only one site blocked AI crawlers site-wide. So for most firms, AI access is an accident of an old rule, not a decision.
What is llms.txt and do financial firms use it?
llms.txt is a proposed file that offers a clean, curated summary of a site for large language models. Adoption in European finance is early: of 105 firms checked, 16 served a genuine llms.txt, most of them fintechs and a handful of asset managers. Another cluster returned an HTTP 200 at the llms.txt path but served a normal HTML page, a soft 404 that looks present to a crawler but carries no useful content. llms.txt is not confirmed as a ranking or citation lever, but its presence is a clear signal of an AI-aware web team.
Which AI crawlers matter for citation in search?
The user-agents that decide whether an AI assistant can quote your page today are GPTBot and OAI-SearchBot for ChatGPT, ClaudeBot for Claude, PerplexityBot for Perplexity, and Google-Extended for Gemini and AI Overviews grounding. Blocking any of these removes your pages from that engine’s answer set. CCBot, which feeds Common Crawl and many model training sets, matters more for training than for live citation.
How do you check your own site’s AI crawler rules?
Fetch your own robots.txt and read it as an AI crawler would. Confirm whether GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended are named, and if not, read your wildcard user-agent group, because that is the rule they inherit. Then request /llms.txt and confirm it returns either a real file or an honest 404, not an HTML page dressed as one.
Cite this study
Roodhorst, V. (2026). AI Crawler Rules at 105 European Financial Firms. roodhorst.eu. https://roodhorst.eu/ai-crawler-rules-european-finance/
Sources & data
- Each firm’s /robots.txt and /llms.txt, fetched 5 August 2026, raw files saved per domain.
- Parsing: grouped User-agent blocks; blocked only on site-wide Disallow: / without a specific Allow.
- The full per-firm dataset with raw files is available on request: victor@roodhorst.eu.
2 thoughts on “AI Crawler Rules at 105 European Financial Firms”