NoIndex Tag: Understanding and Directing the Crawlers Part 2

January 26, 2010  |   Internal Architecture   |   0 Comment

Welcome to part 2 of the Understanding & Directing the Crawlers series. We’re going to discuss the No Index tag. It is an HTML meta tag that tells the crawlers not to index a page and display it in the search results.  Basically, what you’re telling the search engine crawlers is that “you can see my content, but you can’t steal it!”

The only time that a No Index tag should be used is when you want a page to be crawled for content, but not indexed. (i.e. an internal page with high pagerank that you are using to link to other pages within your site, but don’t want the search engines to display it in the search results)

Here is how the NoIndex tag is used in the HTML code:

NoIndex tag

This tag instructs the crawlers not to index the page.  Let me explain the many ways that this tag can be utilized:

<META name=”ROBOTS” content=”NOINDEX, NOFOLLOW”>

  • Don’t index the page & don’t follow any links on the page

<META name=”ROBOTS” content=”NOINDEX, FOLLOW”>

  • Don’t index the page & follow all links on the page

BUT WAIT!!!

Just like Robots.txt, the NoIndex tag will not be obeyed by all crawlers.  So if you want to prevent the search engines from indexing a specific page, I would suggest using a URL removal tool from the search engines.

For Google, you can go to https://www.google.com/webmasters/tools/removals

For Bing & Yahoo!, you need to send them an email with a request.

Hope this helps!  Stay tuned for 301/302 redirects!





About the author


Maximus Kang is the Director of SEO Strategy & Founder of Ranking Channel, a Seattle-based SEO consulting agency. With enterprise level experience at Expedia and agency experience at Optify, his SEO knowledge covers a wide spectrum. He also started his very . Follow him on Twitter or connect with him on Facebook.

New to SEO? You can learn How to Win Users & Influence Google.