Control crawl access and indexation

How to Set Noindex on Pages and Directories—and Verify Google Sees It

Set noindex on an HTML page or a directory of responses, then check what is served and whether Google has found the directive.

Editorial illustration for How to Set Noindex on Pages and Directories—and Verify Google Sees It

To exclude a URL from Google Search with noindex, publish the directive on a response Googlebot can fetch. For one HTML page, a robots meta tag in the document head may be simplest. For a directory containing HTML and PDFs, an X-Robots-Tag response header can apply the same rule across content types—but check the response of each kind of URL. A rule written in robots.txt is not a substitute: Google does not support noindex there.

Choose the URLs, not just the directory name

Suppose /preview.html is a single page you want to exclude, while /drafts/ contains /drafts/outline.html, /drafts/spec.pdf, and nested pages. The policy for /drafts/ needs to reach the responses for those files. Seeing a header on https://example.com/drafts/ alone does not establish that the PDF or a nested page returns it.

Make a short test list before changing configuration: the standalone HTML page, an HTML file in the directory, the PDF, and a nested URL if one exists. Include any URL whose handling might differ. This is a response-level check, not an assumption based on a folder name.

Publish noindex in the page or response

For https://example.com/preview.html, put the complete robots meta tag in the served HTML head:

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Preview</title>
  <meta name="robots" content="noindex">
</head>
<body>
  <h1>Preview</h1>
</body>
</html>

Google documents the head placement and treats a robots meta tag and an HTTP response header as two ways to implement noindex. A response header is particularly useful for a PDF, which has no HTML head. For a hypothetical GET of https://example.com/drafts/spec.pdf, the relevant part of the response would look like this:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex

If this site uses Apache, one possible directory-scoped implementation is to place the following in /drafts/.htaccess:

Header set X-Robots-Tag "noindex"

This path assumes Apache processes that .htaccess file, mod_headers is enabled, and the server configuration permits the Header directive there. The Apache .htaccess header example depends on mod_headers; it is not evidence that a particular hosting setup accepts the file. Check an HTML file, the PDF, and a nested URL after deployment rather than inferring their headers from the directory URL.

Keep the three controls separate. noindex addresses whether the URL appears in results. nofollow is a different robots directive concerning links; adding it is not required to say noindex. A robots.txt disallow controls crawling. In particular, Google discovers a meta tag or X-Robots-Tag when it crawls the URL. If robots.txt disallows the URL, Googlebot cannot fetch that response to find its noindex rule. A blocked URL may still appear in an index without its content having been crawled; blocking the crawl is not a reliable replacement for noindex.

Check publication, then check Google’s detection

Inspect the GET response for each header-based URL. For the PDF, for example:

curl -sS -D - -o /dev/null https://example.com/drafts/spec.pdf

The printed response headers should contain X-Robots-Tag: noindex. Repeat for representative HTML and nested URLs under /drafts/; inspect the actual response for the URL under test, not merely a different directory response. For /preview.html, fetch the served HTML and confirm that its <head> contains the robots meta tag. Also check that the applicable robots.txt rules do not disallow Googlebot from fetching any of these URLs. A successful live check establishes that the directive is being published and is available to a crawler permitted to fetch it; it does not establish that Google has read it.

Next, inspect the affected URLs in Google Search Console’s URL Inspection tool and monitor the Page Indexing report for URLs from which Googlebot has extracted a noindex rule. Compare that information with the live response and crawl access. If a URL still appears in results, inspect its crawl state rather than concluding that today’s response was ignored: Google says it may not have crawled the page since noindex was added, and a revisit can take months. URL Inspection can also be used to request a recrawl. The meaningful end state is not merely a correct header in curl; it is Google having crawled the affected URL and detected the directive.

Find a note

Search by topic, title, or keyword.