robots.txt Generator

Free online robots.txt generator with per-bot control. Allow or block GPTBot, ClaudeBot, PerplexityBot and other AI crawlers individually, then download instantly.

Default Policy

All crawlers

Applies to crawlers without their own rule: Allow = crawl everything, Disallow = block everything

AI Crawler Control

Individually control LLM crawlers such as GPTBot, ClaudeBot, and PerplexityBot

OpenAI GPTBot (model training)GPTBot

OpenAI SearchBot (ChatGPT Search)OAI-SearchBot

ChatGPT-User (user-initiated fetching)ChatGPT-User

Anthropic ClaudeBotClaudeBot

Anthropic Claude-WebClaude-Web

Anthropic anthropic-aianthropic-ai

Google-Extended (Gemini training)Google-Extended

PerplexityBotPerplexityBot

Bytespider (ByteDance / Doubao)Bytespider

Common Crawl CCBotCCBot

Meta External Agentmeta-externalagent

AmazonbotAmazonbot

Applebot-Extended (Apple Intelligence)Applebot-Extended

Rules & Misc

Sitemap

Crawl-delayseconds; 0 = not set

Top Comment

robots.txt

User-agent: *
Disallow: /wp-admin/

Sitemap: https://www.029xazx.com/sitemap.xml

Lines: 4

💡 Upload the file to your site root, accessible at https://yourdomain/robots.txt

💡 robots.txt is voluntary — malicious crawlers may ignore it

FAQ

Why control AI crawlers separately?

GPTBot, ClaudeBot, Google-Extended and friends scrape content for LLM training. Set allow/block per bot to protect original content — or deliberately feed AI search engines.

How do I deploy the file?

Download robots.txt and upload it to your site root so https://yourdomain/robots.txt resolves. It usually takes effect within a day or two.

What is Crawl-delay?

Seconds between requests to ease server load. Google ignores it; Bing and others honor it.