Free Robots.txt Generator

Create a valid robots.txt file in seconds. Control search engine crawlers, block AI bots, set crawl rules, and download your file ready to upload.

Quick Presets

Search Engine Crawlers

AI Crawlers

2026

Control which AI bots can crawl your site for training data. Blocking these does NOT affect your search rankings.

Blocked Directories

Custom Paths

Advanced Settings

Seconds between requests. Only Bing and Yandex honor this. Google ignores it — use Search Console instead.

Generated robots.txt

29 active rules

# robots.txt generated by Kleap - 2026-05-31
# https://kleap.co/tools/robots-txt-generator

User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /private/

# AI Crawlers
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: cohere-ai
Disallow: /

How to use this file

Upload this file to the root of your website so it is accessible at yoursite.com/robots.txt. Most hosting platforms (Vercel, Netlify, WordPress) have a public/ or root directory where you drop it. Changes take effect immediately — no restart required.

What Is a Robots.txt File?

A robots.txt file is a plain text file placed at the root of your website (yoursite.com/robots.txt) that tells web crawlers which pages they can and cannot access. It follows the Robots Exclusion Protocol, a standard used by all major search engines including Google, Bing, Yahoo, and Yandex. When a crawler visits your site, it checks robots.txt first before crawling any page.

In 2026, robots.txt has become more important than ever. Beyond traditional search engine crawlers, a new wave of AI bots — GPTBot (OpenAI), Claude-Web (Anthropic), PerplexityBot, and others — now crawl websites to train large language models. A properly configured robots.txt gives you control over which bots can access your content, protecting your intellectual property while maintaining search engine visibility.

Why Use Our Robots.txt Generator?

One-Click Presets

Choose from 6 presets: Allow All, Block All, Standard, E-commerce, WordPress, and Block AI Bots. Start with a template and customize from there.

AI Bot Controls

Block or allow 11 AI crawlers including GPTBot, Claude-Web, PerplexityBot, CCBot, and Bytespider. Most generators in 2026 still do not cover these.

Live Validation

See warnings in real-time when you make risky choices, like blocking Googlebot or disallowing your entire site. Prevent costly SEO mistakes before they happen.

Search Engine Controls

Granular control over Googlebot, Bingbot, Slurp (Yahoo), DuckDuckBot, Baiduspider, and YandexBot. Allow or block each crawler individually.

Sitemap Declaration

Add your sitemap URL directly in robots.txt so crawlers discover all your pages. This is a best practice recommended by Google and Bing.

Copy & Download

Copy to clipboard or download as a .txt file. Upload to your site root and you are done. No signup, no account, no limitations.

Why You Need to Control AI Bots in 2026

Since 2024, AI companies have deployed web crawlers that scrape websites to train large language models. GPTBot (OpenAI), Claude-Web (Anthropic), PerplexityBot, CCBot (Common Crawl), Bytespider (ByteDance/TikTok), and others now crawl billions of pages. Unlike search engine crawlers that drive traffic to your site, AI crawlers take your content to train models — without sending visitors back.

Blocking AI bots in your robots.txt is the most effective way to prevent your content from being used for AI training. Major publishers like The New York Times, CNN, and Reuters have already added these blocks. The important distinction: blocking AI bots does NOT affect your Google or Bing rankings. Search engine crawlers and AI crawlers are separate user agents.

Our generator includes 11 known AI crawler user agents, updated for 2026. As new AI crawlers emerge, we add them. This is one of the few robots.txt generators that covers AI bots comprehensively — most competitors still only handle traditional search engine crawlers.

How to Set Up Your Robots.txt

Setting up robots.txt is straightforward. Here is a step-by-step guide to get it right.

1. Generate Your File

Use the generator above to configure your rules. Start with a preset (Standard is recommended for most sites) and adjust the AI bot settings based on your preference. Add your sitemap URL if you have one.

2. Upload to Your Site Root

The file must be at yoursite.com/robots.txt — not in a subdirectory. On most platforms: Vercel and Next.js use the public/ folder, WordPress uses the root directory or a plugin, Shopify auto-generates it (you can customize via the theme editor). For static sites, just drop the file alongside your index.html.

3. Test Your Robots.txt

After uploading, visit yoursite.com/robots.txt in a browser to confirm it is accessible. Then use Google Search Console's robots.txt Tester (under Crawl > robots.txt Tester) to verify Google can parse it correctly. Fix any errors before they affect your indexing.

4. Monitor and Update

Review your robots.txt quarterly. New AI bots emerge regularly — add their user agents as they appear. If you add new sections to your site (like /api/ or /admin/), make sure they are blocked. Check Search Console for crawl errors that might indicate robots.txt misconfiguration.

Kleap vs Other Robots.txt Generators

FeatureKleapSEOptimerSmallSEOTools
PriceFree, no limitsFree (basic)Free with ads
AI Bot Controls11 AI crawlers (GPTBot, Claude, etc.)NoneNone
Preset Templates6 presets (Standard, E-commerce, WordPress...)Basic onlyNone
Live WarningsReal-time validation warningsBasic syntax checkNo validation
Download .txtCopy + DownloadCopy onlyCopy only
No Signup RequiredYesYesYes

People Also Ask

What happens if I don't have a robots.txt file?+
Without a robots.txt file, all crawlers (search engines and AI bots) assume they can access every page on your site. This means AI companies can freely scrape your content for training data. While search engines will still index your site normally, you lose control over which bots can access your content.
Does robots.txt affect my SEO rankings?+
Robots.txt itself does not directly affect rankings. However, blocking Googlebot will remove your pages from search results entirely. Used correctly, robots.txt improves SEO by preventing crawl budget waste on low-value pages (admin panels, duplicate content, staging pages) while ensuring important pages are crawled efficiently.
Can I block AI bots without affecting Google?+
Yes. AI crawlers (GPTBot, Claude-Web, PerplexityBot, CCBot, etc.) use different user agents than Googlebot. Blocking them has zero impact on your Google rankings. Search engines and AI crawlers are completely independent.
Is robots.txt legally enforceable?+
Robots.txt is a voluntary standard — there is no technical enforcement mechanism. However, major AI companies (OpenAI, Anthropic, Google) have publicly committed to respecting robots.txt. In the EU and some US states, ignoring robots.txt may have legal implications under copyright and data protection laws.
How often should I update my robots.txt?+
Review it quarterly. New AI crawlers emerge regularly, and your site structure may change. After major site redesigns, new section launches, or when new AI bots are announced, update your robots.txt to reflect the changes.

Frequently Asked Questions

What is a robots.txt file?+
A robots.txt file is a plain text file at the root of your website that instructs web crawlers (search engine bots, AI bots, etc.) which pages they can and cannot access. It follows the Robots Exclusion Protocol, supported by all major search engines.
Where do I put the robots.txt file?+
It must be at the root of your domain: yoursite.com/robots.txt. On Next.js and Vercel, place it in the public/ folder. On WordPress, use the root directory or an SEO plugin like Yoast. On Shopify, edit it via the theme customizer under robots.txt.liquid.
Can robots.txt block all AI crawlers?+
Yes. Our generator includes 11 known AI crawler user agents as of 2026: GPTBot, ChatGPT-User, Google-Extended, Claude-Web, PerplexityBot, CCBot, Bytespider, Amazonbot, FacebookBot, anthropic-ai, and cohere-ai. New crawlers appear regularly, so check for updates.
Does blocking Googlebot remove my site from search?+
Yes. If you block Googlebot, Google will eventually remove all your pages from its search index. This is almost never what you want. Our generator warns you when you make this selection so you can avoid the mistake.
What is the difference between Disallow and noindex?+
Disallow in robots.txt prevents crawlers from accessing a page. The noindex meta tag (or X-Robots-Tag header) tells crawlers not to index a page they have already crawled. For best results, use noindex for pages you want crawled but not indexed, and Disallow for pages you do not want crawled at all.
Should I add a sitemap to robots.txt?+
Yes, it is a best practice. Adding a Sitemap: directive helps crawlers discover all your pages, especially new ones. Google and Bing both recommend including your sitemap URL in robots.txt. You should also submit it in Google Search Console.
What is crawl delay?+
Crawl-delay tells crawlers to wait a specified number of seconds between requests. This prevents server overload from aggressive crawling. Note: Google ignores crawl-delay — use Google Search Console to manage Google's crawl rate instead. Bing and Yandex do honor it.
Can I use wildcards in robots.txt?+
Yes. Googlebot and Bingbot support * (match any sequence) and $ (end of URL). For example, Disallow: /*.pdf$ blocks all PDF files. However, not all crawlers support wildcards, so use them carefully and test with Google's robots.txt Tester.
Is this robots.txt generator free?+
Completely free, with no account required and no usage limits. Generate as many robots.txt files as you need. We do not store your data or require any personal information.
How do I test if my robots.txt is working?+
After uploading, visit yoursite.com/robots.txt in your browser to confirm it is live. Then use Google Search Console's robots.txt Tester to verify Google can parse it correctly. You can also use the Bing Webmaster Tools robots.txt analyzer for Bing-specific validation.

Build a Website with SEO Built In

Kleap generates robots.txt, sitemaps, meta tags, and structured data automatically. Focus on your content, not technical SEO configuration.

Start Building Free

Related Free Tools

Schema Generator

Generate JSON-LD structured data for Google rich results

Meta Tags Generator

Generate SEO meta tags, Open Graph and Twitter cards

Privacy Policy Generator

Create a GDPR-compliant privacy policy for your site

Website Copy Generator

Write compelling website copy with AI assistance

Free Robots.txt Generator | Create & Validate Your Robots.txt 2026