Top Web Hosting Reviews
Top Web Hosting Provider of The Month:
Top Web Hosting
Visit Bluehost.com | Read Bluehost Review

>> Web Hosting Geeks // Web Hosting Articles // SEO - Search Engine Optimization  


Search Bots, Crawlers, and Spiders








If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren't humans, you can't tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time?

Not quite sure what I am talking about? Here is a few examples of various bots searching my website:

207.68.146.40 (msnbot.msn.com)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
This is the MSN Search bot.

207.68.146.40 (lj2070.inktomisearch.com)
Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)
This is Yahoos Search Bot.

66.249.65.147 (crawl-66-249-65-147.googlebot.com)
Mediapartners-Google/2.1
This is Googles bot, that searches your webpages for AdSense.

What is a Bot, Crawler, Spider?
These terms are all the same, they all refer to an automated program that goes from website to website caching and processing the pages for search engines. As you know, "WWW" means World Wide Web, thus "Spider" seemed like an appropriate term. Crawler is another term that just describes what it does, crawling from site to site and page to page endlessly. Bot, is actually short for "robot" and again is just an automated program to index websites.

What is the purpose of a Spider?
A spider looks at all the pages of your website, and uses that information to rank you in search engines (how high you will list in a search result), and cache a copy of your page on their server for quick reference, and if your site ever goes down. Spiders jump from link to link on the Internet and run endlessly, even if you never submit your website to a search engine, odds are your site will still be spidered.

Can I stop bots and spiders from searching my website?
Yes and no. Legitimate spiders are run by reputable organizations that follow certain rules. For instance, most companies have a policy that their robot will search for a file called "robots.txt" in the root of your website. This text file is filled with information telling the bots what and what not is allowed to be viewed. Unfortunately, there are also bad bots out there, they search the internet harvesting e-mail addresses for spam and other bad things, these bots often don't comply with the "robots.txt" standard.

How many bots are there?
It's impossible to guess how many bots are out there searching websites. On any given day I will get roughly 10 different ones check my website. Some of them only search one or two pages, others go over my entire website. Not all of them give you a good description of what they do, or who owns them. If you cut and paste their name and IP address in to Google, quite often you can find more information about what they do.

How can I get my site spidered?
As I mentioned before, if your website is up long enough, it "will" get spidered eventually. However, if you want to ensure that it gets done within a few months, go to the various search engine websites and look for the "Add URL" or "Suggest a Link" pages. DMOZ is one of the big directories which you should submit your site. When you sign up for these search engines, your website is automatically queued up to be spidered. It may take several weeks or months to actually start showing up on the search engine, even after you see the robot spidering your website.

What about pay search engines?
There are a bunch of different search engines that make you pay to have your website listed. I personally don't support these search engines, I find that most people use the big free search engines anyway. However, if you do wish to get included in some search engines faster, many have payment options which will get your site listed within a couple of days.

Ken Dennis
http://KenDennis-RSS.homeip.net/


MORE RESOURCES:

PR-Inside.com (Pressemitteilung)

Seopath.com offers SEO Packages for Quality Search Engine Optimization
BigNews.biz (press release)
Keeping this thing in mind Seopath.com takes part in fulfilling the SEO needs of a website that is newly launched to face the ongoing competition with other ...
SEO Satisfaction SurveyOnline PR News (press release)
It can be daunting but it's necessarySEO Consult - SEO News (blog)
Announcing The Most Affordable Most Powerful SEO Services From This Search ...PR-Inside.com (Pressemitteilung)
Earthtimes (press release) -Promotion World (press release) -DirectNews
all 41 news articles »


PR-Inside.com (Pressemitteilung)

Areas to watch out for in SEO
SEO Consult - SEO News (blog)
Search engines, when it comes down to it, sit in opposition to SEO. While a search engine's purpose is to guide the viewer to the most appropriate site for ...
Cheers to SEO: How It Pays To Be OptimizedSearch Engine Land (blog)
SEO Ranking Expert Introduces SEO Services for Small Static & Huge Dynamic ...PR.com (press release)
Domain names without Latin characters will broaden SEO impactBrafton
BigNews.biz (press release) -OfficialWire (press release) -PR-Inside.com (Pressemitteilung)
all 80 news articles »


Be In Control of Your SEO Venture With Web Analytics
SEO Consult - SEO News (blog)
Web analytics is a crucial part of an SEO campaign. In simple terms, web analytics is a tool used to measure, collect and analyse data. ...
The Benefits of Web AnalyticsSEO Consult - SEO News (blog)

all 2 news articles »


Caspio Introduces Breakthrough SEO Technology for Cloud Database Apps
Reuters
Cloud Computing Database Applications Can Now Be Fully Indexed by Search Engines Using Caspio`s New SEO Deployment Model MOUNTAIN VIEW, Calif. ...

and more »


Little things are important to SEO
SEO Consult - SEO News (blog)
There are so many things to remember when it comes to SEO. You need to keep abreast of rule changes and techniques, track keywords and keep an eye on what ...



Business Daily Africa

Social Search Google's newest Lab experiment – Webmaster & SEO News
Kingpin Webmaster News
We mentioned Google's new social search in a previous news post, however now we feel we should go into more detail for our readers, as this seems like a hot ...
Future Twitter trendsBlogstorm (blog)
My budget is tight, what's the best bang for my buck getting more traffic to ...SmartCompany.com.au
Teaching the Facebook GenerationBusinessWeek
ClickZ News
all 93 news articles »


Brafton

Groundwork for SEO could get tougher with encrypted browsing tool
Brafton
Catalan researchers at Rovira i Virgili University say that they have developed a way for internet users to prevent search engines and websites from ...

and more »


website Auditor Enterprise 1.8.5 (Windows)
ZDNet
website Auditor Enterprise is a quick SEO tool that lets you make effective web pages and provide your SEO clients with stunning onpage optimization reports ...



Caspio Deploys Cloud-based SEO
Editor & Publisher
But limitations surrounding SEO for dynamically generated pages has hindered the ability to fully exploit data. Caspio provides a programming-free way for ...
Caspio SEO technology for cloud database appsCIOL
Caspio Releases Version 6.5 with Advanced Data Import and Publishing CapabilitiesPR Web (press release)

all 6 news articles »


New IT Jobsites to Focus on PHP Jobs and SEO Jobs Launched by CV Screen
PR Web (press release)
CV Screen launch niche IT Job Boards to attract candidates to new PHP and SEO Jobs in the UK. (PRWeb UK/PRWEB ) November 7, 2009 -- Specialist IT ...

and more »

Google News





 
 
 

© 2004 - 2008 "Web Hosting Geeks" | Web Hosting Reviews | Customer Reviews | RealMetrics Reviews | Hosting Articles | Directory | Partners | Contacts
Over 7000 articles: web hosting, web development, domain names, ecommerce, web design, site promotion, ppc advertising, seo, site promotion and many others.
Web hosting reviews, ratings and awards are not based on any incentives or commissions. Names and trademarks are the properties of their respective owners.
A direct link to Web Hosting Geeks (http://webhostinggeeks.com) must be provided in order to use any of the above information. Contact us for more info.

Partners: Hosts by speed, Cheap Website Hosting, Free Website Hosting, Cheap Web Hosting, Top 10 Web Hosts, Top 10 Web Hosting Deals, Best Website Hosting, Free Web Hosting, Free Web Hosting, Dedicated Server Hosting, Adult Web Hosting, Web Hosting Discussions, Dedicated Server Reviews, Best Web Hosting, Web Hosting Discounts, HostProfessor.com, rsuog, halyava, PHP Website Hosting Services, Web Hosting Reviews, Hosting Uptime, Best Web Hosting Reviews, Cheap Webhosting, Web Hosting, Flash Templates, CMS Templates, Web Hosting Reviews, Website Hosting Reviews, Web Hosting Providers, Best Web Hosting, Top Web Hosting, RSUOG Web Hosting