NOTABIS SEO Auditor crawler

A transparent, respectful crawler for authorised SEO site auditing. It does not bypass authentication, robots restrictions, rate limits, WAF controls, or anti-bot challenges.

Purpose and classification

SEO auditing and accessibility-oriented technical diagnostics for site owners and authorised users. It is not a search indexer, training crawler, scraper, vulnerability scanner, or transaction bot.

User agent

NOTABIS SEO Auditor/1.0 (+https://seo.notabis.com/crawler)

Identifier pattern: NOTABIS SEO Auditor

Robots behaviour

Fetches robots.txt before crawling, selects the most specific matching User-agent group, enforces Allow/Disallow using longest-match precedence, honors Crawl-delay, and applies the result to the seed URL, sitemap URLs and discovered links. If robots.txt fails due to a transport or server error, the audit fails closed; a normal HTTP 404 means no policy was published.

Request limits

up to 60 page requests/minute; robots.txt Crawl-delay takes precedence; maximum 100 pages per audit, bounded response size, timeout, redirect, retry, and sitemap limits.

Consent and safety

Audits are initiated by a user for an entered site. The crawler uses HTTP/HTTPS only, validates DNS destinations, stays same-origin, and stops on edge security challenges rather than evading them.

Verification

Machine-readable identity: /crawler.json. IP validation data: /crawler-ips.json. The IP list will contain only dedicated crawler egress addresses once configured.

Privacy and contact

Audit metadata and findings are retained to provide reports. Operational diagnostics avoid unnecessary page-content retention. To block or report the crawler, use your WAF controls or contact the site operator.