A robots.txt file tells search engines which parts of your website they may crawl, and which they should skip. It lives at the root of your domain, at /robots.txt. If that file is wrong, Google can miss the pages that bring you work, or spend crawl time on pages that should stay private.

This guide is for business owners, not developers. You do not need to write the file by hand. You do need to know what it does, because a one-line mistake can hide your whole site from search. For the rest of the crawl picture, see our guides on XML sitemaps and technical SEO.

What is robots.txt?

Robots.txt is a plain text file that uses the Robots Exclusion Protocol. Search engine crawlers (Googlebot, Bingbot, and others) request that file first, then follow the rules you list. It is a request, not a lock. Well-behaved crawlers honour it. Scrapers and malware bots often ignore it.

The file is not a way to hide a page from Google's results. If a URL is already known (from a sitemap, a link, or an earlier crawl), Google can still index it even when robots.txt blocks crawling. To keep a page out of search results you need a noindex tag or a login wall, not robots.txt alone.

Typical uses:

  • Keep admin, cart, thank-you, and search-result URLs out of the crawl.
  • Point crawlers at your XML sitemap.
  • Protect duplicate or parameter URLs that waste crawl budget. See crawl budget.
  • Avoid crawling files that do not help search, such as staging copies or download folders.

Purpose and role in SEO

Think of robots.txt as a gatekeeper for crawlers, not for customers. People can still visit a blocked URL if they have the link. The file only tells bots how to spend their time on your site. Used well, it helps Google spend that time on the pages that should rank. Used badly, it is one of the fastest ways to make a healthy site disappear from search.

How search engines use robots.txt

When a crawler arrives, it looks for https://yourdomain.com/robots.txt. If the file is missing, most crawlers assume they may fetch any public URL they find. If the file exists, the crawler matches its own name against User-agent groups, then applies the longest matching Allow or Disallow rule.

Google still has to choose which URLs to fetch. Robots.txt does not replace a sitemap, internal links, or a fast, crawlable site. It is one instruction in a larger SEO program.

Creating and structuring a robots.txt file

Place the file in the website root so it is reachable at /robots.txt. One file per host. www and the bare domain are different hosts, so each needs its own file if both serve content. Keep the file UTF-8, with one directive per line.

Basic syntax and commands

A group starts with a user-agent, then lists rules for that bot:

User-agent: *
Disallow: /private/
Allow: /public/
Sitemap: https://www.example.com/sitemap.xml
  • User-agent names the crawler. * means every bot that does not have a more specific group.
  • Disallow is a path prefix the bot should not fetch. Disallow: / blocks the whole site.
  • Allow is a path the bot may fetch even if a broader Disallow would have blocked it. Google uses the longest matching path.
  • Sitemap is an absolute URL to your XML sitemap. You can list more than one.

Examples of whole-file patterns:

# Block every crawler from everything (almost never what a public site wants)
User-agent: *
Disallow: /

# Allow every crawler everywhere
User-agent: *
Disallow:

# Block one crawler from one folder
User-agent: Googlebot
Disallow: /private/

Crawl-delay appears in some older files. Bing may honour it. Google ignores it. Do not rely on it to throttle Googlebot.

Best practices

  • Keep the file short. Every extra rule is another chance to hide a page you need.
  • Block folders you truly do not want crawled: admin, cart, account, internal search, printer versions, and filter URLs that create infinite combinations.
  • Do not block CSS or JavaScript that Google needs to render the page. If Google cannot load the layout, it cannot judge the page fairly.
  • List your sitemap. Pair this file with a clean XML sitemap.
  • Test after every change. Google Search Console includes a robots.txt report. Fetch /robots.txt in a browser and confirm the live file is the one you uploaded.
  • Comments start with #. Crawlers ignore the rest of that line. Use them so the next person knows why a rule exists.
  • Malicious bots can ignore the file. Sensitive pages need authentication, not robots.txt.

Common mistakes to avoid

  • Disallow: / on a live site. That tells every crawler to skip every URL. Sites go dark in Google this way after a migrate or a plugin toggle.
  • Blocking resources Google needs to render. If CSS, fonts, or JS sit behind Disallow, the page can look broken to Google.
  • Using robots.txt as noindex. A blocked URL can still appear in results, often without a snippet. Use noindex on the page, then allow crawling so Google can see the tag.
  • Wrong host. The file must be on the same host as the pages: HTTPS, the www choice you use in search, and no stray staging copy.
  • Conflicting rules. Two groups for the same bot, or Allow and Disallow that fight each other, make the file hard to reason about. Prefer one group per bot and the longest-match rule.
  • Listing several paths on one Disallow line. Each path needs its own Disallow.
  • A Disallow with no User-agent. That group is invalid. The crawler may ignore it.

Testers and validators

Before you publish a change, test it. Google Search Console shows how Googlebot reads the live file. Google's documentation on robots.txt is at developers.google.com. After a change, request indexing on a key URL and watch the coverage report for a few days.

Impact of robots.txt on SEO

Benefits of a correct file

A correct file saves crawl budget for the pages that should rank: services, locations, and guides. It keeps thin or duplicate URLs out of the crawl so Google spends time on the ones that win calls. For a local business, that usually means service pages, city pages, and the pages behind your Google Business Profile.

Risks and pitfalls

The same file can hide a site overnight. Typical failures: a staging rule that shipped to production, a plugin that writes Disallow: /, or a migrate that copied the old host's file onto a new CMS. If organic traffic falls and Search Console shows "Blocked by robots.txt," start here, not with new content.

Most modern CMSs write a default robots.txt. That default is not automatically right for your business. Review it the same way you would review title tags.

Frequently asked questions

What happens if there is no robots.txt file?

Crawlers treat the site as fully crawlable. That is often fine for a small brochure site. It is a problem if you have cart URLs, faceted search, or an admin area that is reachable without a login. A short, tested file is safer than none.

Why is Google asking about robots?

Googlebot looks for robots.txt on every host it crawls. If Search Console mentions robots, it is checking that file for directives, or reporting that a URL was blocked by it.

Does robots.txt still work?

Yes. Google, Bing, and other major crawlers still read it. The protocol is old. The stakes are not. A bad file still takes pages out of the crawl.

Can robots.txt keep a page out of Google?

Not reliably. Blocking crawl can stop Google from seeing a noindex tag. If you need a page unpublished, noindex it (or password it), allow the crawl, then remove it from sitemaps.

What to do next

Open your live file at /robots.txt. Confirm it does not disallow the whole site, that CSS and JS are allowed, and that your sitemap URL is listed and loads. If you are unsure, send it with your site on a free growth review. We will tell you whether the file is helping, hurting, or doing nothing.

Read next: XML sitemaps, technical SEO, or the full SEO Education library. When you want the work done on your site, start with SEO services.