Robots.txt Generator
Create a correct robots.txt file for your website in seconds. Every option is explained in plain English, so you don't accidentally block search engines from your site.
Settings
What robots.txt actually does
A robots.txt file sits at the root of your website (yoursite.com/robots.txt) and gives search engine crawlers instructions about which parts of your site they're allowed to visit.
It is not a security tool — it doesn't password-protect anything, and well-behaved crawlers respect it while malicious bots can ignore it entirely. Its real job is guiding legitimate crawlers like Googlebot away from pages that shouldn't be indexed, such as admin areas or duplicate content.
Mistakes that can hurt your site
- ✕Blocking your entire site by accident. A single line — Disallow: / under User-agent: * — tells every search engine to ignore your whole website. This is the most common and most damaging robots.txt mistake.
- ✕Blocking CSS or JavaScript files. Google needs these to render your pages properly and judge mobile-friendliness. Blocking them can hurt how your site is understood, not just what's crawled.
- ✕Forgetting the sitemap line. Not required, but it's a simple way to point crawlers straight to your full list of pages.
- ✕Using robots.txt to hide sensitive pages. Disallowed URLs can still appear in search results without their content, and the file itself is publicly viewable — so it's not a way to keep something private.
Frequently asked questions
Where do I upload robots.txt?
It must sit at the root of your domain — for example waqarweb.com/robots.txt — not inside a subfolder. In cPanel, that's usually the public_html folder.
Does every website need a robots.txt file?
No — if one doesn't exist, search engines assume they can crawl everything. It's useful once you have specific areas you want to guide crawlers away from.
Can I block just one search engine?
Yes, by targeting a specific user-agent (like Googlebot) instead of the wildcard *. This generator currently applies rules to all crawlers — per-engine rules are a planned Pro feature.