← Back to SEO LearnNOTABIS SEO / LEARN

Crawlability · 16 September 2026

Edge and Bot Blocking: How It Affects Crawlers

An edge-blocked audit received a firewall, CDN, WAF, or anti-bot response instead of the site page. NOTABIS recognises selected 403/429 and challenge responses, then stops normal scoring.

Quick answer

An edge-blocked audit received a response that looks like a firewall, CDN, WAF, or anti-bot challenge rather than the site’s page. NOTABIS reports edge-blocked for HTTP 403 or 429 responses and for recognised challenge-page markers, then stops the crawl because page-level results would be misleading. Allow an authorised audit path or rerun after the protection challenge is cleared.

What the issue means

Edge services sit in front of an application and can deny, rate-limit, or challenge requests. A 403 may be an intentional access policy; a 429 may be rate limiting. Challenge pages can return 200 while asking a browser to run JavaScript or solve a verification step, so the status code alone does not show that the real page was delivered.

Why it matters for SEO

Search engines need to retrieve the actual HTML and resources they are allowed to crawl. An edge policy that blocks a legitimate crawler can leave important pages unmeasured or inaccessible, while an intentional private policy may be correct. This finding is about audit coverage and access, not proof of a ranking penalty or a request to disable security controls.

How NOTABIS detects it

When a page response arrives, NOTABIS checks for status 403 or 429 and scans the body for markers such as cf-chl-, challenge-platform, “performing security verification”, “incompatible browser”, “just a moment”, or “checking your browser”. If a marker or status matches, it records edge-blocked at audit scope, stores the reason, marks the crawl blocked, and does not calculate the normal score.

The marker list is a heuristic. A custom WAF page may be missed, and a genuine application 403 or 429 can look similar. Check the response body, headers, user agent, and edge logs before changing policy.

How to check it yourself

Request the exact URL from the audit’s network and compare it with a normal browser request. Record status, final URL, response headers, body title, cookies, and any challenge script. Review CDN/WAF rules, rate limits, IP reputation, and access logs for the request time.

How to fix it

Create a narrowly scoped allow rule for the authorised audit user agent or source, or provide a protected origin route that returns the real HTML. Tune rate limits and bot challenges without opening private routes to everyone. Keep authentication and WAF protections for sensitive areas, then test the public path from an external network and rerun the audit.

When this is intentional or does not need fixing

A private application, staging site, payment step, or deliberately protected API may correctly return 403, 429, or a challenge. Document the policy and interpret the audit as limited coverage. Do not remove edge protection or spoof a crawler identity merely to obtain a score.

Related issues

Compare fetch failures, no pages crawled, HTTP error responses, and robots.txt availability.

Run a NOTABIS audit

NOTABIS records when edge protection prevents a trustworthy crawl and preserves the response reason so you can work with your CDN or WAF team.

Measure before you change

See which issues affect your site.
Run a NOTABIS audit ↗