Methodology: This check follows wSEO methodology 1.1. How the assessment works
AI Crawler Access Configuration
Allow selected AI search crawlers to reach the right public pages without indiscriminately allowing every bot and use case.
- robots.txt and meta robots
- CDN, WAF, Cloudflare, or server rules
- Separation of search, user-triggered, and training crawlers
What to prepare
- the canonical URL and exact test journey;
- dated baseline evidence;
- a list of implemented changes;
- the same device, browser, region, and account where these affect the result.
Step-by-step verification
- The intended user agent receives the allowed URL
- No 403, CAPTCHA, redirect loop, or geographic block
- Private and utility sections remain protected
Repeat the check at least twice in a fresh window. For non-deterministic AI answers or experiments, use multiple repetitions and do not draw a conclusion from one response.
Acceptance criterion
An agreed access policy and verified successful responses for the selected crawlers.
Save the URL, date, screenshot, HTTP response, or analytics event—the evidence type that directly proves the promised outcome.
Excludes general site availability repair, hosting migration, and legal decisions about licensing policy.
If verification fails
Do not hide the symptom with additional changes. Return to the baseline, check caching and configuration propagation, then compare the actual result with the agreed scope.
