robots.txt is missing or not served

Audit check robots · worth 5 of 100 points · all checks

What the audit saw

GET /robots.txt did not return 200.

Why it costs you

robots.txt is where you point crawlers at your sitemap and keep them out of the paths that waste crawl budget. Missing is not fatal — everything is allowed — but you lose the sitemap pointer and the ability to exclude.

How to find it on your site

  1. Check the status 
    curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' https://your-site.com/robots.txt
  2. A minimal correct file 
    User-agent: *
Allow: /

Sitemap: https://your-site.com/sitemap.xml
  3. GitHub Pages serves it from the repo root 
    # put robots.txt in the branch Pages publishes, not in a subfolder

How to know it is fixed

HTTP 200 with content-type text/plain and a Sitemap line.

The part people get wrong

A 200 that returns your HTML shell (SPA catch-all route) counts as broken: the crawler reads markup as directives. Serve it as a real static file.

Check it against your own site

Open an issue with your URL and a bot runs this check plus 14 others, posts the failures with the measured values, and publishes a report page: free audit. If you want the fixes applied and re-verified instead: fixed-price tiers.

Other checks

The site does not answer on the URL you gave · The page is served over plain HTTP · The TLS certificate is expired or about to exp · No mobile viewport tag · The page title is missing or the wrong length · The meta description is missing or the wrong l · The page has no H1, or more than one · Images have no alt text

All published audits · Check reference