Is robots.txt required?

No. A site without one is crawled normally — crawlers treat a 404 as 'everything allowed'. It becomes worth adding once you have paths you want left alone, or once you want a Sitemap line for discovery.

Why does my robots.txt return 404 when the file is on my server?

Almost always because it is not being served from the domain root. Static-site frameworks often need the file in a specific folder (`public/`, `static/`, `www/`) to end up at `/robots.txt`, and some hosts intercept the path entirely. Fetch the URL directly rather than trusting that the file exists on disk.

Does blocking a page in robots.txt remove it from Google?

No, and this is the single most common misunderstanding. Blocking prevents crawling, so Google cannot read the page — but it can still list the URL if other pages link to it. Use a noindex meta tag on a crawlable page instead.

Can I block AI crawlers here?

Yes. GPTBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended all honour robots.txt user-agent rules. Note that this is a policy decision, not a security control — it stops the crawlers that choose to comply.

robots.txt checker

robots.txt is a plain text file at the root of your domain (yoursite.com/robots.txt) that tells crawlers which paths they may fetch. Enter a domain below and we'll fetch it, confirm whether the file is actually reachable, and check whether it points crawlers at your sitemap.

13%have no reachable robots.txt

Across 959 websites analyzed

Measured across every site in our public database.

Why this matters

A missing robots.txt is not fatal — a wrong one is

If the file is absent, crawlers assume everything is allowed and carry on. The real damage comes from a file that exists and blocks the wrong thing: a stray `Disallow: /` left over from a staging environment will deindex an entire site, and it is one of the most common causes of a sudden traffic collapse. That is why checking what the file actually says matters more than checking that it exists.

It is where crawlers look for your sitemap

A `Sitemap:` line in robots.txt is the one place every major crawler — Google, Bing, Naver, Yandex — checks without you submitting anything. On a new domain with no backlinks, it is often the only way your inner pages get discovered at all.

How to fix it

1. Put the file at the root, not in a subfolder
It must be served at `https://yoursite.com/robots.txt` — exactly there. `/public/robots.txt` or `/assets/robots.txt` is ignored. Each subdomain needs its own: `blog.yoursite.com/robots.txt` is a separate file from the apex domain's.
2. Start from a safe default
For most sites this is the whole file: `User-agent: *` on one line, `Allow: /` on the next, then a blank line and `Sitemap: https://yoursite.com/sitemap.xml`. Add `Disallow:` lines only for paths you genuinely do not want fetched — admin panels, cart and checkout URLs, internal search results.
3. Never use it to hide a page from search results
Disallow stops a page from being *crawled*, not from being *indexed* — a blocked URL with inbound links can still appear in results, just without a description. To keep a page out of search, let crawlers fetch it and serve `<meta name="robots" content="noindex">` instead. Blocking it in robots.txt actually prevents the noindex from ever being seen.
4. Verify after deploying
Open the URL in a private browser window — a 404, a redirect to your homepage, or an HTML error page all count as broken. Then re-run the check above and confirm the sitemap line is picked up.

Want this fixed rather than explained?

We do this as fixed-price work — no hourly billing, no retainer. You get a quote before anything starts.

See fixed prices →

Frequently asked questions

Is robots.txt required?: No. A site without one is crawled normally — crawlers treat a 404 as 'everything allowed'. It becomes worth adding once you have paths you want left alone, or once you want a Sitemap line for discovery.
Why does my robots.txt return 404 when the file is on my server?: Almost always because it is not being served from the domain root. Static-site frameworks often need the file in a specific folder (`public/`, `static/`, `www/`) to end up at `/robots.txt`, and some hosts intercept the path entirely. Fetch the URL directly rather than trusting that the file exists on disk.
Does blocking a page in robots.txt remove it from Google?: No, and this is the single most common misunderstanding. Blocking prevents crawling, so Google cannot read the page — but it can still list the URL if other pages link to it. Use a noindex meta tag on a crawlable page instead.
Can I block AI crawlers here?: Yes. GPTBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended all honour robots.txt user-agent rules. Note that this is a policy decision, not a security control — it stops the crawlers that choose to comply.