Every website has a quiet little gatekeeper sitting at its front door. Most site owners never look at it, and most of the time that is fine — until the day Google starts crawling pages you never wanted indexed, or worse, stops crawling the pages that matter. That gatekeeper is robots.txt, a plain text file with an outsized influence on how search engines treat your site.
The good news: robots.txt is one of the simplest files on the web. You can understand the whole thing in about ten minutes, and you can create a correct one in under a minute with a robots.txt generator. This guide explains what the file does, what it cannot do, the mistakes that cost sites real traffic, and how to check that Google is actually respecting your rules.
When a search engine bot arrives at your site, the polite ones first ask for yoursite.com/robots.txt. The file answers a simple question: which parts of this site are you allowed to crawl? A typical file looks like this:
User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
That is nearly the entire language. User-agent picks which bot the rule applies to (* means everyone). Disallow lists paths bots should not crawl. Allow carves out exceptions. And the Sitemap line points crawlers at your XML sitemap so they can discover your pages faster — which is why it pairs so well with a proper XML sitemap generator when you are setting up a new site.
Just as important is what robots.txt does not do. It does not hide a page from humans — anyone can still visit a disallowed URL directly. It does not guarantee a page stays out of Google's index either; if other sites link to a blocked page, Google may index the URL without ever crawling its content. And it is not a security tool. Think of it as a "please knock" sign, not a lock. If a page must truly stay private, it needs a password or a noindex tag, not just a Disallow line.
Robots.txt must live at the root of your domain: https://yoursite.com/robots.txt. A file at yoursite.com/blog/robots.txt will simply be ignored. The file is case-sensitive in two ways that trip people up: the filename itself must be lowercase, and paths are matched exactly as written — Disallow: /Admin/ will not block /admin/.
Here are the directives you will actually use:
* covers all of them; Googlebot targets Google specifically.Disallow: / blocks the entire site — the single most dangerous line in SEO.Rules are grouped into blocks, one block per user-agent, and a bot follows only the most specific block that names it. Keep the file tidy, comment generously with #, and avoid clever tricks — simplicity is the whole point.
Small files cause big outages. These are the errors I see most often:
1. Blocking CSS and JavaScript. Google needs your stylesheets and scripts to render pages the way visitors see them. Blocking /wp-content/ or /assets/ wholesale can make Google misjudge your pages and hurt rankings.
2. Leaving Disallow: / live after development. Staging sites routinely block all crawlers, and the file gets copied to production by accident. If your traffic suddenly falls off a cliff, robots.txt is the first place to look.
3. Typos and wrong paths. A missing trailing slash or a pluralized folder name (/product/ vs /products/) silently breaks the rule. There is no error message — the bot just does something different from what you intended.
4. Using robots.txt to "hide" sensitive pages. Login pages, staging areas, internal tools — blocking them in robots.txt actually advertises their existence, since the file itself is public. Use authentication instead.
5. Forgetting that subdomains need their own file. blog.yoursite.com needs its own robots.txt; the main domain's file does not cover it.
Writing the file is only half the job — you should verify how crawlers experience your site. A search engine spider simulator shows you your pages roughly the way a bot sees them, which is handy for spotting blocked resources or unexpected redirects. After that, a Google index checker tells you whether your important pages are actually in Google's index — because a page can be crawlable yet still unindexed for other reasons, like thin content or missing internal links.
Make this a routine, not a one-time task. Check robots.txt after every redesign, migration, or CMS update. It takes thirty seconds and it is one of the highest-leverage checks in technical SEO.
You do not need to hand-write the file unless you enjoy that sort of thing. A free robots.txt generator walks you through the options — which bots to address, which folders to block, where your sitemap lives — and produces a clean, valid file. The workflow is simple:
* block is enough.yoursite.com/robots.txt in a browser to confirm it is live.Then test it. Fetch the URL, run the crawler checks above, and confirm your key pages are reachable. Done well, this is a five-minute task that protects you for years.
Not directly. It does not boost rankings, but it protects your crawl budget — the limited attention Google gives your site — by steering bots away from junk pages (filters, internal search, duplicates) and toward the pages that deserve to rank. On large sites that protection is significant.
Yes: Disallow: /that-page/ stops crawling of that path. But remember, blocking crawling is not the same as removing it from the index. If the page must disappear from search results, use a noindex meta tag instead — and note that a Disallow rule can actually prevent Google from seeing the noindex tag, so do not combine them on the same URL.
Usually one of three things: the file is not at the domain root, it has a syntax error or wrong casing, or you are looking at cached behavior — Google can take days to re-fetch the file. Also check that you are not testing with a different user-agent than the one your rules target.
Strictly speaking, no — a small site works fine without one. But a minimal file with your sitemap line costs nothing and helps crawlers discover your content faster. There is no good reason to skip it.
Robots.txt is a tiny file with an important job: telling search engines where to spend their time on your site. Keep it simple, keep it at the root, never use it as a security measure, and re-check it after every major site change. Get those basics right and you will never have to think about it again — which is exactly how infrastructure should feel.
Build a valid robots.txt for your site in seconds — choose what to block, add your sitemap, and download the file. Free, no signup needed.
