Robots.txt Generator
A robots.txt file tells search engine crawlers which parts of your site they're allowed to visit. It lives at the root of your domain — https://yoursite.com/robots.txt — and is one of the first things crawlers check before indexing anything.
Use this tool to generate a correct robots.txt in seconds — allow everything, block everything, or write custom rules per user-agent, plus point crawlers at your sitemap.
robots.txt
User-agent: *
Disallow:Frequently asked questions
Where do I put the robots.txt file?
Upload it to the root of your domain so it's reachable at https://yoursite.com/robots.txt — it won't work in a subdirectory.
Does blocking a page in robots.txt remove it from Google?
No. Robots.txt only stops crawling, not indexing — a blocked URL can still appear in search results if other pages link to it. Use a noindex meta tag to actually keep a page out of results.
Should I list my sitemap in robots.txt?
Yes — adding a Sitemap: line makes it easy for any crawler to discover your sitemap.xml without needing to submit it manually first.