Robots.txt is a small file with an outsized ability to help or seriously harm your website's visibility on Google. This guide explains what it does, how to write one correctly, and the single most common mistake to avoid.

What Is Robots.txt?

Robots.txt is a plain text file that sits at the root of your website - for example, yourwebsite.com/robots.txt - and gives instructions to search engine crawlers about which parts of your site they're allowed to visit.

It is not a security tool. It doesn't password-protect anything, and well-behaved crawlers respect it while malicious bots can simply ignore it. Its real job is guiding legitimate crawlers like Googlebot away from pages that shouldn't be indexed.

What Does a Robots.txt File Look Like?

A basic robots.txt file looks like this:

User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://yourwebsite.com/sitemap.xml

"User-agent: *" means the rules apply to all crawlers. Each "Disallow" line blocks a specific path from being crawled. The "Sitemap" line points crawlers to your sitemap.xml file.

The Mistake That Can Hide Your Entire Website

This is the most important thing to understand about robots.txt. A single line can accidentally block your whole website from every search engine:

User-agent: *
Disallow: /
That forward slash after "Disallow:" means "everything." This exact configuration is sometimes left in place accidentally after a website moves from a staging environment to live - always double check your robots.txt after launch.

What Should You Actually Block?

Common candidates for blocking include:

  • Admin or login areas
  • Internal search result pages
  • Duplicate content, like print-friendly versions of pages
  • Shopping cart and checkout pages, for online stores

What you generally should not block:

  • CSS and JavaScript files - Google needs these to properly render and understand your pages
  • Pages you actually want to appear in search results

How to Create a Robots.txt File

  1. Use a robots.txt generator to build the file with the rules you need
  2. Download or copy the generated text
  3. Upload it as a file named exactly "robots.txt" to the root folder of your website
  4. Confirm it loads by visiting yourwebsite.com/robots.txt directly in your browser

Frequently Asked Questions

Does every website need a robots.txt file?

No - if one doesn't exist, search engines assume they can crawl everything. It becomes useful once you have specific areas you want to guide crawlers away from.

Can I use robots.txt to hide a page from Google entirely?

Not reliably. A disallowed URL can sometimes still appear in search results without its content if other pages link to it. To reliably keep a page out of search results, use a "noindex" meta tag on the page itself instead.

Can I block just one search engine and not others?

Yes, by targeting a specific crawler name instead of the wildcard "*" - for example, "User-agent: Googlebot". This is a more advanced use case most websites don't need.

Ready to put this into practice?

Use our free Website SEO Checker — no signup required.

Open Website SEO Checker →

Ready to put this into practice?

Use our free Robots.txt Generator — no signup required.

Open Robots.txt Generator →

Ready to put this into practice?

Use our free Sitemap Generator — no signup required.

Open Sitemap Generator →