Search Bots, Crawlers, and Spiders

If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren't humans, you can't tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time?

Not quite sure what I am talking about? Here is a few examples of various bots searching my website:

207.68.146.40 (msnbot.msn.com)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
This is the MSN Search bot.

207.68.146.40 (lj2070.inktomisearch.com)
Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)
This is Yahoos Search Bot.

66.249.65.147 (crawl-66-249-65-147.googlebot.com)
Mediapartners-Google/2.1
This is Googles bot, that searches your webpages for AdSense.

What is a Bot, Crawler, Spider?
These terms are all the same, they all refer to an automated program that goes from website to website caching and processing the pages for search engines. As you know, "WWW" means World Wide Web, thus "Spider" seemed like an appropriate term. Crawler is another term that just describes what it does, crawling from site to site and page to page endlessly. Bot, is actually short for "robot" and again is just an automated program to index websites.

What is the purpose of a Spider?
A spider looks at all the pages of your website, and uses that information to rank you in search engines (how high you will list in a search result), and cache a copy of your page on their server for quick reference, and if your site ever goes down. Spiders jump from link to link on the Internet and run endlessly, even if you never submit your website to a search engine, odds are your site will still be spidered.

Can I stop bots and spiders from searching my website?
Yes and no. Legitimate spiders are run by reputable organizations that follow certain rules. For instance, most companies have a policy that their robot will search for a file called "robots.txt" in the root of your website. This text file is filled with information telling the bots what and what not is allowed to be viewed. Unfortunately, there are also bad bots out there, they search the internet harvesting e-mail addresses for spam and other bad things, these bots often don't comply with the "robots.txt" standard.

How many bots are there?
It's impossible to guess how many bots are out there searching websites. On any given day I will get roughly 10 different ones check my website. Some of them only search one or two pages, others go over my entire website. Not all of them give you a good description of what they do, or who owns them. If you cut and paste their name and IP address in to Google, quite often you can find more information about what they do.

How can I get my site spidered?
As I mentioned before, if your website is up long enough, it "will" get spidered eventually. However, if you want to ensure that it gets done within a few months, go to the various search engine websites and look for the "Add URL" or "Suggest a Link" pages. DMOZ is one of the big directories which you should submit your site. When you sign up for these search engines, your website is automatically queued up to be spidered. It may take several weeks or months to actually start showing up on the search engine, even after you see the robot spidering your website.

What about pay search engines?
There are a bunch of different search engines that make you pay to have your website listed. I personally don't support these search engines, I find that most people use the big free search engines anyway. However, if you do wish to get included in some search engines faster, many have payment options which will get your site listed within a couple of days.

Ken Dennis
http://KenDennis-RSS.homeip.net/

Five Essential Elements of SEO SEO Consult (blog) All sorts of fancy techniques have been devised by the clever minds of the SEO community. If you're just starting out in SEO, it can all be very confusing. ... How to improve your organic trafficStuckOn Define your value for internet successSEO Consult (blog) all 3 news articles »

PR Web (press release)

SEO Company India Providing SEO Service at Very Affordable SEO Packages PR-inside.com (press release) 2010-03-13 07:48:44 - SEO India Designers, a leading organic Search Engine Optimization and Social Media Marketing company India offering SEO Services at ... Frequent Design Mistakes In SEOSearch Newz Wrong SEO advice may weaken business prospectsviploan.co.uk Few Words on Affordable SEO Services : Wildnet TechnologiesBigNews.biz (press release) PR Web (press release) -MarketingProfs.com (subscription) -Impact Media (blog) all 76 news articles »

CatchFriday: Delivering Quality Web Design And SEO OfficialWire (press release) CatchFriday can combine good web design and programming with effective SEO. By incorporating all the factors that made a website successful, CF can create a ... and more »

Establish SEO Authority For Your Site SEO Consult (blog) It emerged as an SEO tactic when businesses discovered how hard it was getting to achieve inbound links. The options for attracting inbound links are ... Hotel Internet Marketing and Hotel SEO Services – Why?The Brainchild Group - Marketing News (blog) Community building for successful internet marketingHelium all 3 news articles »

Book Review: The Art of SEO: Mastering Search Engine Optimization by Eric Enge ... Blogcritics.org (blog) The Art of SEO is a comprehensive book that covers every facet of search engine optimization. It thoroughly touches on the many factors that affect a ...

SearchEngineOptimization.co.uk Launch Free Online Alexa Site Rank Checker Tool PR Web (press release) SearchEngineOptimization.co.uk, a leading SEO firm based in the North West of England, have just launched a range of free SEO tools on their website which ... and more »

Team1 Real Estate and WebiMax Partner for SEO Campaign PR Web (press release) Team1 Real Estate sought the search engine services of WebiMax regarding the Web site, NaplesRealEstateGuys.com. The online marketing campaign involves ...

AffiliateTip.com

SEO Copywriting for WordPress AffiliateTip.com I have never been much of an SEO guy, so I was excited when I heard about this tool to improve my blog posts. I've been using the All in One SEO Pack plugin ... Scribe can optimize your blog posts in WordPressLast Click News all 2 news articles »

On-page Elements That Make or Break SEO SEO Consult (blog) The main elements of SEO are the practical things you do to your pages. When it comes to the work of search engine optimisation, people tend to get caught ... Elite SEO Marketing Launches New Blog ServicePR-inside.com (press release) Price Structures for SEO Services – Pricing an Effective SEO CampaignThe Brainchild Group - Marketing News (blog) 7 Ways a Copywriter Can Help Improve Your Site's SEOImpact Media (blog) PR-inside.com (press release) -SEO Consult (blog) -SEO Consult (blog) all 9 news articles »

Big Media or Big SEO Spammers? GigaOm (blog) I think that the work done to the duped content is done for the purpose of SEO (Search Engine Optimization). The idea here seems to be stripping content, ... Blogger Violet Blue Discovers SEO Spamming Of Her Columnsmediabistro.com (blog) Some newspapers may hate search engines, but they love SEO spamEconsultancy (blog) all 4 news articles »