Free Robots.txt Generator for News & Media Sites
Generate a robots.txt optimized for news websites. Maximize crawl efficiency for breaking news and archives. Free SEO tool.
News sites need aggressive crawling to get breaking stories indexed quickly. Your robots.txt must balance maximum crawlability for news content while blocking duplicate pages from syndication, print versions, and infinite pagination that waste your crawl budget.
Tips for News Sites
Never block Googlebot-News — it's a separate crawler that indexes content for Google News specifically
Block /print/ and /amp/ versions if they duplicate your main article content and aren't canonicalized
Allow fast crawling with a generous or no Crawl-delay directive — news content has a short freshness window
Block internal search result pages and paginated archive pages beyond the first 5-10 pages
Try This Example
“Generate a robots.txt for a news site that maximizes article crawling, allows Google News bot, and blocks print/duplicate pages”