HTML NoIndex: Keep Pages Off Search

The HTML Meta Tag

To instruct search engine crawlers not to index a specific web page, the following tag should be placed within the section of your HTML document:

<meta name="robots" content="noindex">

Explanation

• meta: This HTML tag is used to specify metadata about an HTML document.
• name="robots": This attribute indicates that the meta tag's content is intended for web robots (search engine crawlers, spiders, bots).
• content="noindex": This is the directive given to the robots. It explicitly tells any compliant search engine crawler not to include this particular page in its search engine index.

Purpose and Use Cases

The primary purpose of the noindex meta tag is to prevent specific web pages from appearing in search engine results pages (SERPs). This is useful for various scenarios:

• Development or Staging Environments: Pages under construction, test pages, or internal development versions that should not be public.
• Private or Sensitive Content: Pages containing user-specific data, admin dashboards, login pages, thank you pages after a form submission, or intranet pages.
• Duplicate Content: To avoid potential SEO penalties or dilute page authority by telling search engines not to index pages that are very similar or identical to other indexed content (e.g., printer-friendly versions, filtered category pages with little unique content).
• Low-Quality or Unimportant Pages: Pages that provide little value to users searching via a search engine, such as internal search results pages or simple utility pages.
• Temporary Content: Pages that are only relevant for a short period and shouldn't remain in the index.

Placement and Considerations

• Placement: The tag must be placed within the section of the HTML document, typically before the closing tag.
• Crawler Access: For the noindex tag to be effective, the search engine crawler must be able to access and read the page where the tag is present.
  • If a page is blocked from crawling by a robots.txt file, the crawler will not be able to read the noindex tag, and thus, the page might still appear in search results (though without content snippets) if it was linked from other indexed pages.
  • The noindex directive explicitly tells crawlers not to index, even if they are allowed to crawl.
• Combining Directives:
  • noindex, nofollow: You can combine noindex with nofollow to also instruct crawlers not to follow any links present on that page. This is written as .
  • index, follow: This is the default behavior for most pages, so these directives are rarely explicitly needed: .

Related Concepts

• robots.txt file: This file is placed at the root of your website and tells crawlers which parts of your site they are allowed or disallowed to crawl. While robots.txt prevents crawling, the noindex meta tag prevents indexing. It's crucial to understand the difference: robots.txt prevents a bot from visiting a page, whereas noindex tells a bot that has visited a page not to show it in search results.
• X-Robots-Tag HTTP Header: For more robust control, especially for non-HTML files (like PDFs, images) or for server-side management, the noindex directive can be sent as an HTTP header. For example: X-Robots-Tag: noindex. This header is processed before the page content is even delivered.