Robots.txt Generator
Create a valid, WordPress-safe robots.txt file — control search engines and AI crawlers with full Allow/Block precision. No signup required.
Site Details
Default Access for Search Engines
This setting may prevent search engines from crawling your entire website and can seriously affect search visibility.
Recommended Presets
AI Crawlers
AI Training Crawlers
Block crawlers that may collect public website content for AI model development or training.
AI Search Crawlers
These crawlers may help your website appear in AI search results, answers and citations. Blocking them may reduce AI search visibility.
User-Initiated AI Browsing
These crawlers access a page when a user asks an AI assistant to visit, read or summarize the website.
Custom Rules (optional)
How it works
Set your rules
Choose a preset or fine-tune each AI crawler and search engine setting
Watch the live preview
Your robots.txt updates instantly as you change settings — no button needed
Download & upload
Save the file and upload it to your site's root directory
SEO best practices
- Your robots.txt file must live at the root: yoursite.com/robots.txt — not in a subfolder
- Blocking a page in robots.txt doesn't remove it from Google — use a noindex tag for that
- Always include your sitemap URL so crawlers can find your pages faster
- Never block CSS or JS files — Google needs them to render your pages correctly
- Test your file in Google Search Console's robots.txt report after uploading