Understanding and Directing the Crawlers Intro

January 20, 2010  |   Internal Architecture   |   0 Comment

As promised, I will be doing a 4 part series on directing the crawlers…or spiders…or bots…or whatever you want to call them.

The crawlers are computer programs that browse the World Wide Web (internet) in a methodical manner to gather information from various websites. When a spider crawls a page, it gets indexed (this means your website will show up in the search engines when users run a search) into the search engine’s database. In fact there are only two ways to get a site indexed into the search engine’s database.

  1. Submit a page to the search engine via XML sitemaps to get crawled
  2. Being linked to by an already indexed page

There are a few methods to direct the bots to perform specific tasks.

So without further ado, let me introduce you to the different techniques:

Stay tuned for Part 1 – Robots.txt





About the author


Maximus Kang is the Director of SEO Strategy & Founder of Ranking Channel, a Seattle-based SEO consulting agency. With enterprise level experience at Expedia and agency experience at Optify, his SEO knowledge covers a wide spectrum. He also started his very . Follow him on Twitter or connect with him on Facebook.

New to SEO? You can learn How to Win Users & Influence Google.