Robots.txt vs noindex: which control do you need?
Separate permission to crawl from permission to appear in search, then check the page that is actually served.
1. Start with the outcome
Robots.txt controls requests by crawlers that follow its rules. Noindex asks supporting search engines not to include a resource in search results. Neither is an access-control system for confidential information. Use authentication when content must be private.
| Intended outcome | Control to investigate |
|---|---|
| Manage crawler traffic to repetitive URLs | A carefully scoped robots.txt rule, after checking what the crawler needs to read. |
| Keep a public page out of Google results | A noindex directive on a page Google can fetch. |
| Keep customer information private | Authentication and authorization, not crawler instructions. |
| Choose one URL for equivalent content | Canonicalization or an appropriate redirect, rather than a blanket crawl block. |
2. Avoid the blocked-page trap
If robots.txt blocks a page, Google cannot read a noindex directive on that page. A blocked URL may still appear in results when linked elsewhere. Diagnose the crawl rule before assuming that adding another meta tag will solve the problem.
| Situation | Review decision |
|---|---|
| The public page has noindex but /thank-you is disallowed | If the page contains no private information, allow fetching so the directive can be read. Verify the delivered page and monitor later. |
| The page exposes a customer's booking details | Protect the content through authentication immediately. Do not make it crawlable merely to expose a noindex. |
3. Inspect the complete response
Record the requested URL, redirects and final destination. Read robots.txt on the exact host, then inspect the final page's response headers and HTML. A CMS plugin, application middleware or CDN may each contribute directives, so keep a copy of what the visitor and crawler actually receive.
Noindex can be delivered in a robots meta tag or an X-Robots-Tag HTTP header. Google's documentation does not support putting a noindex instruction inside robots.txt.
<meta name="robots" content="noindex, follow">X-Robots-Tag: noindex- Which host and user agent does the crawl rule apply to?
- Is the page accessible without authentication, and should it be?
- Is noindex present in HTML, headers, or both?
- Does a redirect lead to a page with different directives?
- Is the evidence from a live response or a previously stored search observation?
4. Preview one deliberate change
Write the intended outcome in the change ticket. Preserve the old rule, show the exact diff, get the owner's approval and apply the smallest correction. Recheck neighboring pages as well as the target, since a shared template or broad path rule can affect more than one URL.
- If a launch page should be discoverable, check for both unintended crawl blocks and unintended noindex.
- If a public utility page should stay out of search, verify its directive can be read.
- If duplicate URLs are the issue, investigate canonicalization separately.
- If information is private, test access permissions independently of SEO settings.
5. Keep a decision worksheet
Sources
Checked on 11 November 2026. Platform guidance changes, so follow the links for the current wording.
- Introduction to robots.txtGoogle Search Central
- Block Search indexing with noindexGoogle Search Central
- How to specify a canonical URLGoogle Search Central
Original CrawlrSEO working guide. Examples use a fictional business. Search outcomes vary; no ranking, indexing, traffic or AI inclusion is guaranteed.
Browser tools run locally. Hosted checks show whether the service is available before you submit.