Robots.txt gives cooperating crawlers instructions about which paths they may request. It is a public text file, normally requested at the root of the relevant host, such as https://example.com/robots.txt.
It is not a password system. It is also not a reliable way to remove a known URL from search results. Those distinctions matter before a broad Disallow rule is deployed.
Read the group before the path
User-agent: *
Disallow: /internal-search/
Disallow: /test-area/
Allow: /test-area/public-guide/
Sitemap: https://example.com/sitemap-index.xml
This illustrative group addresses crawlers covered by the wildcard identity. It restricts the named paths and permits the more specific public-guide path. It does not protect private files from clients that ignore the instructions.
A crawler-specific group can use a recognised crawler token. Google’s matching and group-selection rules are documented in its robots.txt specification. Do not assume every bot interprets unsupported directives the same way.
Test the exact URL pattern
Rules match paths, and their specificity matters. For Google’s supported interpretation, the more specific matching path takes precedence; where equally specific Allow and Disallow rules conflict, Allow takes precedence.
Wildcard patterns can match more than expected. $ can anchor a pattern to the URL ending. Before using them, create a test set containing URLs that should be allowed and URLs that should be blocked.
A simple prefix restriction such as /internal-search/ does not automatically cover every alternative URL format used by the search feature. Check the site’s real paths and query parameters instead of designing rules from a guessed pattern.
Keep crawling separate from indexing
If Google cannot fetch a page because robots.txt blocks it, it cannot read a noindex meta tag inside that response. A blocked URL can still be known through links.
Use meta robots for an appropriate page-level indexing instruction when Google can crawl the page. Use authentication or another real access control for private material. Publishing a confidential directory name in robots.txt does not make it private.
Validate a proposed change before publishing
Screaming Frog supports custom robots configuration for controlled tests. Use a saved candidate file to compare expected access before changing the live root file. Confirm which user agent the test represents and record the crawl settings.
A custom local test changes your crawler’s conditions; it does not change the site’s public instructions. After deployment, request the live file, inspect the response, and confirm the intended contents are actually being served.
Retain a readable maintenance record
Use comments to explain non-obvious decisions and keep a previous version for rollback. When an allowed page becomes blocked unexpectedly, compare the live rules with the last known version before changing unrelated metadata.
For larger sites, test by page type and URL pattern. My crawl and indexing analysis service covers access rules alongside actual responses. A technical SEO audit can identify where an overly broad rule affects content or rendering resources.