Search Bots, Crawlers, and Spiders

If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren't humans, you can't tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time?

Not quite sure what I am talking about? Here is a few examples of various bots searching my website:

207.68.146.40 (msnbot.msn.com)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
This is the MSN Search bot.

207.68.146.40 (lj2070.inktomisearch.com)
Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)
This is Yahoos Search Bot.

66.249.65.147 (crawl-66-249-65-147.googlebot.com)
Mediapartners-Google/2.1
This is Googles bot, that searches your webpages for AdSense.

What is a Bot, Crawler, Spider?
These terms are all the same, they all refer to an automated program that goes from website to website caching and processing the pages for search engines. As you know, "WWW" means World Wide Web, thus "Spider" seemed like an appropriate term. Crawler is another term that just describes what it does, crawling from site to site and page to page endlessly. Bot, is actually short for "robot" and again is just an automated program to index websites.

What is the purpose of a Spider?
A spider looks at all the pages of your website, and uses that information to rank you in search engines (how high you will list in a search result), and cache a copy of your page on their server for quick reference, and if your site ever goes down. Spiders jump from link to link on the Internet and run endlessly, even if you never submit your website to a search engine, odds are your site will still be spidered.

Can I stop bots and spiders from searching my website?
Yes and no. Legitimate spiders are run by reputable organizations that follow certain rules. For instance, most companies have a policy that their robot will search for a file called "robots.txt" in the root of your website. This text file is filled with information telling the bots what and what not is allowed to be viewed. Unfortunately, there are also bad bots out there, they search the internet harvesting e-mail addresses for spam and other bad things, these bots often don't comply with the "robots.txt" standard.

How many bots are there?
It's impossible to guess how many bots are out there searching websites. On any given day I will get roughly 10 different ones check my website. Some of them only search one or two pages, others go over my entire website. Not all of them give you a good description of what they do, or who owns them. If you cut and paste their name and IP address in to Google, quite often you can find more information about what they do.

How can I get my site spidered?
As I mentioned before, if your website is up long enough, it "will" get spidered eventually. However, if you want to ensure that it gets done within a few months, go to the various search engine websites and look for the "Add URL" or "Suggest a Link" pages. DMOZ is one of the big directories which you should submit your site. When you sign up for these search engines, your website is automatically queued up to be spidered. It may take several weeks or months to actually start showing up on the search engine, even after you see the robot spidering your website.

What about pay search engines?
There are a bunch of different search engines that make you pay to have your website listed. I personally don't support these search engines, I find that most people use the big free search engines anyway. However, if you do wish to get included in some search engines faster, many have payment options which will get your site listed within a couple of days.

Ken Dennis
http://KenDennis-RSS.homeip.net/

Brand Media Goes Green; SEO Brand Media Works on New Sta. Elena ... PR-CANADA.net (press release), Montenegro - 5 hours ago SEO Brand Media has officially signed the contract with the developer of Sta. Elena Golf and Country Club, Mr. Gippy Tantoco for the website development of ...

Learn How to Create Quality SEO Content with Google Snatch Corsavoo.com, France - 8 hours ago Throughout the pages of furniture cleaning service book he demystifies the process behind selecting keywords to build a site and writing SEO articles that ... Linking up: Search engine rankings still come down to links. Corsavoo.com Google Doesn't Care About Website Design Corsavoo.com all 3 news articles

SEO Elite really worth your money? Corsavoo.com, France - Aug 29, 2008 One of the greatest misconceptions about SEO Elite and other SEO software is that they’re a ‘get-traffic-quick’ source. Whereby, simply inserting your ... Search Engine Optimization(On Page) Corsavoo.com all 2 news articles

There's No Secret Recipe to SEO Search Engine Watch - Aug 29, 2008 But SEO isn't an omelet. While a general framework is necessary to be effective, the optimization process must be adaptable to each unique client. ...

Professional SEO services healthcare Business Feet, UK - Aug 29, 2008 Professional services for SEO should be treated like doctors because "websites are like patients" and won't get better if they don't reveal symptoms. ... Disclose information for SEO BCS all 2 news articles

Marketing Assistant/SEO/Web Analyst Bizcommunity.com, South Africa - Aug 29, 2008 Do you have a degree in marketing and would like to make the transition to e-marketing? We are looking for a talented individual with a marketing brain to ...

Internet Marketing News

Unique web content 'best for SEO' Epiphany Search News, UK - Aug 28, 2008 Such content can both give search engine optimisation (SEO) efforts a boost and attract more relevant visitors and potential sales leads to a company's ... Expert: Enhance SEO with unique web pages Digital Response Media Exclusivity vital for better search engine placement Business Feet Exclusive website content "crucial" DirectNews Internet Marketing News - Bluhalo all 8 news articles

Div ID=”Header” Different SEO Tactics Search Engine Journal - Aug 29, 2008 it is the most prominent part of the site - it is usually located at the top of the page source code (and hence is a good place for your keywords). an ...

Are You Giving Your SEO Enough Information to Succeed? Search Engine Land, CT - Aug 28, 2008 I think of that post nearly every time someone asks for SEO advice because the field has grown so complex that both yes and no are often mutually wrong ...

Electricity bourse gets SEO nod Tehran Times, Iran - Aug 26, 2008 TEHRAN – The director of Iran’s Securities and Exchange Organization (SEO) here on Tuesday announced that the SEO’s Bourse Council gave the green light to ...