Configure robots.txt without guessing broad restrictions
There is no universal robots.txt for every website. A safe file begins with understanding the site structure and testing representative valuable and service URLs.
- save the present file;
- test URL patterns, not isolated examples;
- publish only after validation.
Message for a developer
“Review and save the current /robots.txt. List valuable pages that must remain crawlable and service URL patterns that may be blocked. Add only justified rules, declare the working Sitemap, and show tests of representative URLs for Googlebot before publication. Do not apply a blanket parameter restriction until pagination, filters, CSS, JavaScript and images have been checked.”
What to prepare
- the current robots.txt and a backup of its generator or configuration;
- valuable page types: categories, products, services and articles;
- examples of search, filters, cart, account, parameters and duplicates;
- the current sitemap.xml location;
- CMS, repository or web-root access for publication.
Step-by-step configuration
- Open
https://example.com/robots.txtand save the file. - Confirm a 200 response for the correct protocol, hostname and subdomain.
- Collect real valuable and service URL patterns from a crawl, analytics and CMS.
- Start with a minimal file and add restrictions only for a clear reason.
- Test every rule against valuable, edge-case and service URLs.
- Publish in the host root, clear the relevant cache and retest.
- Monitor Search Console reports and crawl logs after the change.
Safe minimal example
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlAn empty Disallow does not block crawling. Add only rules justified by this website's structure.
Targeted example
User-agent: *
Disallow: /cart/
Disallow: /account/
Disallow: /internal-search/
Sitemap: https://example.com/sitemap.xmlTechnical check and rollback
curl -i https://example.com/robots.txt
curl -sS https://example.com/robots.txtThe file should be plain text, accessible without authentication or redirect loops. Compare syntax and behaviour with the official Google documentation.
If valuable URLs become blocked, restore the saved version or safe minimal file immediately, clear cache and retest. robots.txt controls crawling; it is not a secure way to hide confidential data or remove a URL from the index.
Use the manual verification guide afterwards. Fixing noindex, canonical, sitemap and URL architecture is separate work. If the rules remain uncertain, order configuration.
Content history
- — Added a safe change process, primary sources and rollback steps.
