Understanding and Directing the Crawlers Intro

As promised, I will be doing a 4 part series on directing the crawlers…or spiders…or bots…or whatever you want to call them.

The crawlers are computer programs that browse the World Wide Web (internet) in a methodical manner to gather information from various websites. When a spider crawls a page, it gets indexed (this means your website will show up in the search engines when users run a search) into the search engine’s database. In fact there are only two ways to get a site indexed into the search engine’s database.

1. Submit a page to the search engine via XML sitemaps to get crawled
2. Being linked to by an already indexed page

There are a few methods to direct the bots to perform specific tasks.

So without further ado, let me introduce you to the different techniques:

• Robots.txt
• NoIndex Tag
• 301/302 Redirects
• Content Barriers
  

Stay tuned for Part 1 – Robots.txt

Maximus Kang is the Director of SEO Strategy & Founder of Ranking Channel, a Seattle-based SEO consulting agency. With enterprise level experience at Expedia and agency experience at Optify, his SEO knowledge covers a wide spectrum. He also started his very first profitable business as a 1st grader. Follow him on Twitter or connect with him on Facebook.

New to SEO? You can learn How to Win Users & Influence Google.