| Top Web Hosting Provider of The Month: |
 |
| Visit Bluehost.com | Read Bluehost Review |
|
|
| >> Web Hosting Geeks // Web Hosting Articles // SEO - Search Engine Optimization |
|
|
 |
60 Day Sandbox for Google & AskJeeves; MSN Indexes Quickest, Yahoo Next
Search engine listing delays have come to be called the Google Sandbox effect are actually true in practice at each of four top tier search engines in one form or another. MSN, it seems has the shortest indexing delay at 30 days. This article is the second in a series following the spiders through a brand new web site beginning on May 11, 2005 when the site was first made live on that day under a newly purchased domain name. Previously we looked at the first 35 days and detailed the crawling behavior of Googlebot, Teoma, MSNbot and Slurp as they traversed the pages of this new site. We discovered the each robot spider displays distinctly different behavior in crawling frequency and similarly differing indexing patterns. For reference, there are about 15 to 20 new pages added to the site daily, which are each linked from the home page for a day. Site structure is non-traditional with no categories and a linking structure tied to author pages listing their articles as well as a "related articles" index varied by linking to relevant pages containing similar content. So let's review where we are with each spider crawling and look at pages crawled and compare pages indexed by engine. The AskJeeves spider, Teoma has crawled most of the pages on the site, yet indexes no pages 60 days later at this writing. This is clearly a site aging delay that's modeled on Google's Sandbox behavior. Although the Teoma spider from Ask.com has crawled more pages on this site than any other engine over a 60 day period and appears to be tired of crawling as they've not returned since July 13 - their first break in 60 days. In the first two days, Googlebot gobbled up 250 pages and didn't return until 60 days later, but has not indexed even a single page in 60 days since they made that initial crawl. But Googlebot is showing a renewed interest in crawling the site since this crawling case study article was published on several high traffic sites. Now Googlebot is looking at a few pages each day. So far no more than about 20 pages at a decidedly lackluster pace, a true "Crawl" that will keep it occupied for years if continued that slowly. MSNbot crawled timidly for the first 45 days, looking over 30 to 50 pages daily, but not until they found a robots.txt file, which we'd neglected to post to the site for a week and then bobbled the ball as we changed site structure, then failed to implement robots.txt in new subdomains until day 25 - and THEN MSNbot didn't return until day 30. If little else were discovered about initial crawls and indexing, we have seen that MSNbot relies heavily on that robots.txt file and proper implementation of that file will speed crawling. MSNbot is now crawling with enthusiasm at anywhere between 200 to 800 pages daily. As a matter of fact, we had to use a "crawl-delay" command in the robots.txt file after MSNbot began hitting 6 pages per second last week. The MSN index now shows 4905 pages 60 days into this experiment. Cached pages change weekly. MSNbot has apparently found that it likes how we changed the page structure to include a new feature which links to questions from several other article pages. Slurp gets strangely inactive then alternately hyperactive for periods of time. The Yahoo crawler will look at 40 pages one day and then 4000 the next, then simply look at the home page for a few days and then jump back in for 3000 pages the next day and back to only reviewing robots.txt for two days. Consistency is not a curse suffered by Slurp. Yahoo now shows 6 pages in their index, one an errors page and another is a "index/of" page as we have not posted a home page to several subdomains. But Slurp has crawled easily 15,000 pages to date. Lessons learned in the first 60 days on a new site follow: 1) Google crawls 250 pages on first discovery of links to site. Then they don't return until they find more links and crawl slowly. Google has failed to index new domain for 60 days. 2) Yahoo looks for errors pages and once they find bad links will crawl them ceaselessly until you tell them to stop it. Then won't crawl at all for weeks until crawling heavily one day and lightly the next in random fashion. 3) MSNbot requires robots.txt files and once they decide they like your site, may crawl too fast, requiring "crawl-delay" instructions in that robots.txt file. Implement immediately. 4) Bad bots can strain resources and hit too many pages too quickly until you tell them to stay out. We banned 3 bots outright after they slammed our servers for a day or two. Noted "aipbot" crawled first then "BecomeBot" came along and then "Pbot" from Picsearch.com crawled heavily looking for image files we don't have. Bad bots, stay out. Best to implement robots.txt exclusions for all but top engines if their crawlers strain your server resources. We considered excluding the Chinese search engine named Baidu.com when they began crawling heavily early on. We don't expect much traffic from China, but why exclude one billion people? Especially since Google is rumored to be considering a possible purchase of Baidu.com as entry to Chinese market. The bottom line is that we've discovered all engines seem to delay indexing of new domain names for at least thirty days. Google so far has delayed indexing THIS new domain for 60 days since first crawling it. AskJeeves has crawled thousands of pages, while indexing none of them. MSN indexes faster than all engines but requires robots.txt file. Yahoo's Slurp crawls on again off again for 60 days, but indexes only six of total 15,000 or more pages crawled to date. We seem to have settled that there is a clear indexing delay, but whether this site specifically is "Sandboxed" and whether delays apply universally is less clear. Many webmasters claim that they have been indexed fully within 30 days of first posting a new domain. We'd love to see others track spiders through new sites following launch to document their results publicly so that indexing and crawling behavior are proven. © Copyright July 18, 2005 Mike Banks Valentine Mike Banks Valentine is a search engine optimization specialist who operates WebSite101 eCommerce Tutorial and will continue reports of case study chronicling search indexing of Publish101 Article Resource Click to Contact Mike Valentine
MORE RESOURCES:
Google News |
|
|
|
RELATED ARTICLES
Using Blogs for SEO Why Start A Blog?I knew about blogging and blogs for years before I actually started my first blog.. Utilizing Popular Directories as Free Link Sources If you're a webmaster, you've probably spent almost as much time going after link exchanges as actually building your website. Gaining in-bound links, especially from websites with a higher PageRank than your own, can increase your own PageRank on Google, and help your site attain a better overall ranking in Google search results. Link Popularity: Improve Your Search Engine Rankings What is link popularity?Link Popularity is simply the total number of pages that link to your website. Most search engines, including Google, consider that when one page links to another page, it is effectively casting a vote of confidence for the other page. The Role of the Robots.txt File to Improve Site Ranking! Not many web master take the time to use a robots.txt file for their website. How MSN and Yahoo Sells Your Traffic Yes, it really happens. Now you might find it hard to believe but you will understand after I explain. Does The Number Of Links On A Page Affect Ranking? Lots of research has focused on inbound links to a site, but little has focused on the number of links actually on a page (outbound or to other parts of a site). Many SEO gurus have recently been talking about something they call "PR Leak" which seems to be a theory that the more outbound links you have, the more your page rank on Google "leaks" away. Google Patent Application - Linking The recent patent application filed by Google details numerous items the search engine uses to rank web pages. The specific application is summarized as:"A method for scoring a document, comprising: identifying a document; obtaining one or more types of history data associated with the document; and generating a score for the document based on the one or more types of history data. Why SEO (as we know it) is Doomed to Failure and How You Can Avoid the Trap Search Engine Optimization (SEO) has become one of the biggest internet buzz-words recently. Everyone is talking about it. Effective Keyword Optimization and Analysis Techniques Keyword optimization involves vital keyword selection and placement strategy depends on successfully identifying your industry related important keywords and then where you can place those keyword for maximum effectiveness.Effective use of keywords optimization and Phrases on your websiteKey to successful web optimization begins with effective use of keywords selection and placement. Link Popularity --- Its Role and Importance In Getting Top Search Engine Rankings Introduction"Link Popularity" - these words may have caught your attention several times while you have been searching the Internet for tips on optimizing your website for top search engine rankings. Link popularity means popularizing a particular link of a website by increasing the number of websites that link to that site. Link Building Services In today scenario when we talk about Search Engine Optimization, we also talk about one of the most important aspect of SEO, which is Link Building. But there are different types, aspects and limitations of Link Building, which would be discussed now under1. Its Not Just All About Google Anymore Those webmasters that stick to the old ways and focus entirely on Google are missing out on a lot of search traffic these days if they are not also well ranked by Yahoo and MSN.For the first few months after Yahoo decided to go their own way with natural search (and MSN decided to get serious about the search business), the search results provided by those two could only be described as bizarre. Search Wars! - MSNs Opening Salvo With all the recent publicity given to Google as the Internet's number one search service, it's hardly surprising that Microsoft has already started work on re-vamping their MSN search service. Okay, it's still powered by the Yahoo engine, but according to Microsoft it has been "cleaned up" and the new service mirrors much more the kind of "non-commercial" results which are currently displayed by Google. Duplicate the Exact Steps Used to Get a Number 1 Yahoo Ranking in Less than 30 Days If you have ever been into a McDonalds you will understood the value of the Cookie Cutter Model. Every McDonalds you go in are the Same. How to Get One Way Backlinks Don't be fooled into believing that all backlinks are created equal because their not! Why, you may ask? It's no secret that many webmasters trade links left, and right, for the benefit of a higher ranking in the search results, but search engines have caught on to this technique, and are very aware that this is major threat to the relevance of the search results.Search engines such as Google have took steps in an effort to prevent this from becoming major problem by placing more importance on one way backlinks. Meta Tag Tactics - Give Your Website Traffic a Boost with the Meta Tag Basics Getting your site noticed by the search engines and rewarded with top rankings is most webmasters main goal, however there are a lot of different factors that play into what the search engines are looking for, including Meta tags. So, if you don't know anything about meta tags but are interested in learning about them so you can use them to possibly increase your rankings, then read the following basic tips regarding meta tags. Playing in Googlebots Sandbox with Slurp, Teoma, & MSNbot - Spiders Display Differing Personalities There has been endless webmaster speculation and worry about the so-called "Google Sandbox" - the indexing time delay for new domain names - rumored to last for at least 45 days from the date of first "discovery" by Googlebot. This recognized listing delay came to be called the "Google Sandbox effect. Why You Need a SEO Maintenance Plan A search engine optimization maintenance service plan will ensure that your site will continue to increase in its rankings, attract more visitors and make more sales. It's not enough to simply design your web site, have it optimized for the search engines and expect it to continually rank well. Being dumped by Google? Learn how to avoid becoming a victim next time around! After Google latest update nicknamed "Florida", many webmasters discovered that their traffic plummeted.What happened?More importantly what can you do about it?And what will Google do next?What happened was that Google made an algorithm change on how they rate web pages. 3 Principles Of Google When online "Use it. Use it.
|
|
|
|
|
| © 2004 - 2008 "Web Hosting Geeks" | Web Hosting Reviews | Customer Reviews | RealMetrics Reviews | Hosting Articles | Directory | Partners | Contacts Over 7000 articles: web hosting, web development, domain names, ecommerce, web design, site promotion, ppc advertising, seo, site promotion and many others. Web hosting reviews, ratings and awards are not based on any incentives or commissions. Names and trademarks are the properties of their respective owners. A direct link to Web Hosting Geeks (http://webhostinggeeks.com) must be provided in order to use any of the above information. Contact us for more info. Partners: Hosts by speed, Cheap Website Hosting, Free Website Hosting, Cheap Web Hosting, Top 10 Web Hosts, Top 10 Web Hosting Deals, Best Website Hosting, Free Web Hosting, Free Web Hosting, Dedicated Server Hosting, Adult Web Hosting, Web Hosting Discussions, Dedicated Server Reviews, Best Web Hosting, Web Hosting Discounts, HostProfessor.com, rsuog, halyava, PHP Website Hosting Services, Web Hosting Reviews, Hosting Uptime, Best Web Hosting Reviews, Cheap Webhosting, Web Hosting, Flash Templates, CMS Templates, Web Hosting Reviews, Website Hosting Reviews, Web Hosting Providers, Best Web Hosting, Top Web Hosting, RSUOG Web Hosting |
