Does blocking Google-Extended hurt my Google Search rankings?

No. Google has explicitly confirmed that 'Google-Extended' controls access to Gemini and Vertex AI training data only. Blocking Google-Extended has zero impact on your organic search indexing, which is handled exclusively by 'Googlebot'.

Why should I block Bytespider and CCBot?

Bytespider (ByteDance/TikTok) and CCBot (Common Crawl) are notorious for aggressive, high-frequency crawling that can overload server CPU, exhaust bandwidth quotas, and scrape proprietary articles without attribution or licensing compensation.

How do I install the generated robots.txt on my website?

Download the generated robots.txt file and place it in the public root directory of your web server (e.g. https://yourdomain.com/robots.txt). For Next.js applications, place it inside the /public folder or define it dynamically via app/robots.ts.

Back to Tools

SEO & Website Tools

AI Crawler & Bot Blocker robots.txt Generator

Generate bulletproof robots.txt rules to block AI model training scrapers (GPTBot, ClaudeBot, Bytespider, CCBot) while allowing search engine citations and Perplexity traffic.

Instant ResultsNo Sign-up Required100% Free Utility

Quick Security Presets

Allow search engines

Googlebot, Bingbot, DuckDuckBot, Applebot. Turning this off removes your site from Google and Bing results.

Configurable AI Crawlers & Scrapers

9 of 15 Bots Blocked

Sitemap Directive URL:

Optional. Tells crawlers where your sitemap is.

Sensitive Paths to Disallow (One per line):

Live Generated robots.txt Preview

# robots.txt generated with Smart Toolverse AI robots.txt Generator
# Note: robots.txt is advisory and public. Protect private areas with authentication.

# Blocked completely
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Bytespider
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: cohere-ai
User-agent: Amazonbot
Disallow: /

# Allowed, except the private paths (repeated here because named bots ignore the * group)
User-agent: Googlebot
User-agent: Bingbot
User-agent: DuckDuckBot
User-agent: Applebot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Allow: /

# Every other crawler
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Allow: /

AI Robots.txt Policy Advisor

Describe your website and traffic goals to get an optimal balance between AI search citations and content theft protection.

Website Industry / Domain:

Primary Objective:

About AI Crawler & Bot Blocker robots.txt Generator

Generate standards-compliant (RFC 9309) robots.txt rules to block generative AI foundation scrapers (GPTBot, ClaudeBot, Google-Extended, Bytespider, CCBot) while preserving Google Search indexation and allowing conversational AI search engines (PerplexityBot, ChatGPT-User) to cite your domain.

Why Choose Our AI robots.txt Generator

Feature	Smart Toolverse	Others
Modern AI Crawler Coverage	15+ Dedicated AI agents (GPTBot, ClaudeBot, Perplexity, Apple)	Only supports legacy 2015 search engines (Google/Bing)
Citation vs Training Separation	Separate training scrapers from search citation agents	All-or-nothing wildcard blocking that breaks SEO
AI Strategy Advisor	Tailored policy recommendations based on your business type	Generic static code blocks without explanation
1-Click Direct Export	Clean valid UTF-8 .txt download with sitemap integration	Cluttered with ad popups and captcha hurdles

Common Uses & Applications

Protecting digital intellectual property from unauthorized LLM model training, stopping server bandwidth depletion caused by aggressive web scrapers (like Bytespider), optimizing AI citation discoverability, and auditing search crawler access paths.

Frequently Asked Questions

GPTBot is OpenAI's large-scale automated web crawler used to harvest web data for training future foundation models. In contrast, ChatGPT-User is triggered in real-time when an active ChatGPT user asks the assistant to browse a specific URL. Disallowing GPTBot protects your content from model training without preventing ChatGPT users from reading and citing your links.