Back to Workshop resources
SEO · AEO · GEO1 min read
Allowing AI crawlers in robots.txt — the explicit list.
Don't rely on the implicit allow. List GPTBot, ClaudeBot, PerplexityBot, etc. explicitly.
Overview
Some hosts and CDNs block AI crawlers by default. If you want to be cited by AI engines, you need to explicitly allow them in robots.txt.
The crawlers to allow
• OpenAI: GPTBot, ChatGPT-User, OAI-SearchBot
• Anthropic: ClaudeBot, Claude-Web, anthropic-ai
• Perplexity: PerplexityBot, Perplexity-User
• Google AI: Google-Extended
• Apple Intelligence: Applebot-Extended
• Meta AI: Meta-ExternalAgent, Meta-ExternalFetcher
How to allow them in Next.js
Use Next.js's app/robots.ts to generate the file programmatically. For each crawler, add a rule with
userAgent and allow: "/".Reversible
If you decide to block AI crawlers later (e.g. for paid content), you can switch the rule from allow to disallow. Effect within hours.
Previousllms.txt explained — the new convention for AI.Next Google Search Console for solo founders.
Want to Apply This to Your Business?
We're a Singapore AI development and automation agency. Let's discuss how we can help solve your specific challenges.