Quick answer
robots.txt controls whether a crawler may request URLs, while noindex tells a crawler that it may fetch the page but should not include it in search results. They solve different problems and are not interchangeable. Use robots rules to manage crawl access and a crawlable noindex directive, authentication, or removal when the goal is to keep content out of an index.
What the issue means
robots.txt is a public file at the origin root. A rule such as Disallow: /private/ asks matching crawlers not to fetch those paths. A page-level directive appears in HTML or a response header:
<meta name="robots" content="noindex">
X-Robots-Tag: noindex
The crawler must be able to request the page to see a meta or header directive. A robots block can prevent that request, and a blocked URL may still be known through links or other sources. Neither file is a substitute for authentication.
Why it matters for SEO
Use the control that matches the intent. Blocking large sets of unimportant resources can reduce crawl traffic, while noindex is appropriate for a crawlable page that should not appear in results. Blocking a page in robots.txt does not reliably remove an already known URL from search because the crawler may not be able to see a removal directive. An accidental noindex or broad disallow can hide valuable content from search systems.
How NOTABIS detects it
NOTABIS fetches and parses the origin’s robots.txt before crawling. It records robots.txt unavailable when the request fails before a response can be processed and applies the parsed policy to eligible URLs. For each measured HTML page, the noindex-directive rule checks the robots meta value and X-Robots-Tag header. The audit can classify common login, account, and filtered-query routes as intentional noindex contexts.
These checks describe the response and crawl policy observed from the audit network. They do not prove how every crawler interprets a rule, and robots syntax, user-agent groups, redirects, and caches can change the result.
How to check it yourself
Request both controls separately:
curl -sS https://example.com/robots.txt
curl -sS -D - https://example.com/page -o page.html
Check the user-agent group, matching Allow/Disallow paths, final URL, HTML robots meta, and X-Robots-Tag. Decide whether the URL should be public, crawlable but excluded, or protected by authentication.
How to fix it
Remove an accidental robots block from important public paths and keep robots.txt syntax and user-agent scope clear. If a page should stay crawlable but out of search, remove the disallow and emit noindex on the final response. For confidential content, require authentication or remove it; do not publish secrets in robots.txt.
After a change, fetch the exact URL, inspect redirects and headers, and rerun the audit. Search engines can cache robots responses and need time to recrawl a noindex change.
When this is intentional or does not need fixing
Blocking admin paths, crawl traps, or private resources can be sensible, and noindex is common for login, account, search, and filter pages. A site may use both controls for different URL sets. Document the policy and avoid changing it only to improve an audit count.
Related issues
Read noindex directives, robots.txt availability, sitemap access, and sitemap-only pages. Google’s robots.txt introduction explains crawl control and its limits.
Run a NOTABIS audit
NOTABIS shows the robots policy and page-level noindex evidence together, helping you tell a crawl restriction from an indexing directive.