Confirm that robots.txt is available and does not block valuable URLs
The file's existence proves little. A good result returns 200, contains understandable rules and permits crawling of representative pages that should attract search visitors.
- open the file as an ordinary visitor;
- test a valuable and a service URL;
- save the file and result evidence.
How to assess the result
| Situation | Level | Action |
|---|---|---|
| The whole site or a valuable section is unintentionally blocked; the file returns a server error | Critical | Restore a safe version and retest immediately. |
| Sitemap is absent or rules are outdated or ambiguous, while valuable control URLs remain crawlable | Noncritical | Confirm intent and schedule a correction. |
| The file is available, syntax is clear, valuable URLs are allowed and service restrictions are intentional | No problem | Save evidence and retest after website changes. |
Quick browser check
- Open
https://example.com/robots.txt. - Confirm that it is a text file, not an HTML error or login page.
- Find
User-agent,Disallow,AllowandSitemap. - Look for an unexpected
Disallow: /. - Compare broad restrictions with valuable site URLs.
Google Search Console check
Use URL Inspection in Google Search Console for a verified property. Enter a valuable URL and check whether crawling is allowed. Google may temporarily use a cached robots.txt after a change.
- Test one valuable indexable page.
- Test one service page intentionally blocked.
- Repeat for WWW or a separate subdomain when used.
- Save screenshots of both outcomes.
HTTP response check
curl -I https://example.com/robots.txt
curl -sS https://example.com/robots.txtExpect 200 and the rule text. A redirect may work for compatibility, but direct availability on every used host is preferable.
Representative URL test
python -m pip install robotexclusionrulesparser
python - <<'PY'
from robotexclusionrulesparser import RobotExclusionRulesParser
import urllib.request
r = RobotExclusionRulesParser()
r.parse(urllib.request.urlopen('https://example.com/robots.txt').read().decode())
print(r.is_allowed('Googlebot', 'https://example.com/important-page'))
PYTrue means the selected crawler may access the URL under these rules.
Differences and test limitations
robots.txt is evaluated separately for every protocol, hostname and subdomain. A CDN or cache may serve an old version, and crawlers do not all support identical extensions. Repeat without cache and test the crawler relevant to the task.
Allowed crawling does not guarantee indexing, while blocked crawling does not guarantee removal of a known URL from search. noindex, canonical, HTTP responses and index presence require separate checks.
The automated wSEO audit for this service assesses robots.txt for one entered website and representative rules; it cannot replace a business review of every filter and service area. If a problem is confirmed, follow the repair guide or order configuration.
