Quick answer
robots.txt is a public text file that gives crawlers site-level crawl instructions. NOTABIS reports robots-unavailable when its request for the file fails before a response can be processed, such as through a connection, DNS, TLS, or timeout failure. A missing or inaccessible robots file can limit the crawler’s ability to apply your policy, but it does not itself remove pages from a search index.
What the issue means
The standard location is https://example.com/robots.txt (or the HTTP origin being audited). It contains groups of directives such as User-agent, Disallow, and Allow, plus optional Sitemap lines. The file controls crawling access; it is not a privacy boundary and is not the right way to de-index a page. Use authentication for private content and a crawlable noindex directive when a page should be removed from search.
Why it matters for SEO
Search engines use robots rules to manage crawler traffic and decide which resources they may request. If the file cannot be reached, a crawler may not receive the rules you intended. A bad robots policy can also block important pages or resources, while an overly broad policy can waste crawl activity. Google notes that robots.txt is for controlling crawling, not for preventing a URL from appearing in search results.
The finding is about the audit’s ability to retrieve the policy. It does not prove that every search engine treats an outage identically or that a page is indexed or blocked.
How NOTABIS detects it
At the start of an audit, NOTABIS requests the origin’s /robots.txt and parses the returned text when the request completes. The robots-unavailable issue is created only when fetchText throws before a usable response is recorded; a response such as a reachable 404 is not the same trigger in the current implementation. The crawler then uses a restrictive fallback policy for that run and records the issue at origin scope.
This result reflects the audit location, user agent, network path, timeout, redirects, and response observed at that time. It does not validate every directive’s business intent or guarantee that another crawler will see the same result.
How to check it yourself
Request the exact origin path and retain headers:
curl -sS -D - https://example.com/robots.txt -o robots.txt
Get-Content .\robots.txt
Check the final URL, status, content type, body encoding, TLS certificate, DNS, CDN or firewall logs, and whether the file is publicly reachable without login. Use Google’s robots.txt testing and inspection tools where available, but compare them with the deployed response.
How to fix it
Publish a plain-text robots.txt at the origin root, keep it small and syntactically clear, and make sure the web server, CDN, WAF, and TLS configuration allow the request. Remove accidental authentication or bot challenges from this path. Correct redirect loops and stale Sitemap URLs, then test from outside your office network. If you do not need a custom policy, a reachable file with an appropriate empty or permissive policy is still clearer than an unreachable path.
After deployment, request the file again and rerun the audit. Search engines may cache robots responses for a period, so policy changes are not necessarily reflected immediately.
When this is intentional or does not need fixing
An internal staging site may intentionally be unreachable to public crawlers, provided it is protected by authentication or network controls. A site that has no special crawl restrictions may use a minimal file. Do not publish sensitive paths or credentials in robots.txt; its contents are public.
Related issues
Read noindex directives and sitemap availability. Google’s robots.txt introduction explains the difference between crawl control and indexing control.
Run a NOTABIS audit
A NOTABIS audit records whether it could retrieve and process the origin’s robots.txt, so you can investigate policy and network failures with the rest of the crawl evidence.