| Top Web Hosting Provider of The Month: |
 |
| Visit Bluehost.com | Read Bluehost Review |
|
|
| >> Web Hosting Geeks // Web Hosting Articles // SEO - Search Engine Optimization |
|
|
 |
Search Engine Spiders Lost Without Guidance - Post This Sign!
The robots.txt file is an exclusion standard required by all web crawlers/robots to tell them what files and directories that you want them to stay OUT of on your site. Not all crawlers/bots follow the exclusion standard and will continue crawling your site anyway. I like to call them "Bad Bots" or trespassers. We block them by IP exclusion which is another story entirely. This is a very simple overview of robots.txt basics for webmasters. For a complete and thorough lesson, visit http://www.robotstxt.org/ To see the proper format for a somewhat standard robots.txt file look directly below. That file should be at the root of the domain because that is where the crawlers expect it to be, not in some secondary directory. Below is the proper format for a robots.txt file -----> User-agent: * User-agent: msnbot User-agent: Teoma User-agent: Slurp User-agent: aipbot User-agent: BecomeBot User-agent: psbot --------> End of robots.txt file This tiny text file is saved as a plain text document and ALWAYS with the name "robots.txt" in the root of your domain. A quick review of the listed information from the robots.txt file above follows. The "User Agent: MSNbot" is from MSN, Slurp is from Yahoo and Teoma is from AskJeeves. The others listed are "Bad" bots that crawl very fast and to nobody's benefit but their own, so we ask them to stay out entirely. The * asterisk is a wild card that means "All" crawlers/spiders/bots should stay out of that group of files or directories listed. The bots given the instruction "Disallow: /" means they should stay out entirely and those with "Crawl-delay: 10" are those that crawled our site too quickly and caused it to bog down and overuse the server resources. Google crawls more slowly than the others and doesn't require that instruction, so is not specifically listed in the above robots.txt file. Crawl-delay instruction is only needed on very large sites with hundreds or thousands of pages. The wildcard asterisk * applies to all crawlers, bots and spiders, including Googlebot. Those we provided that "Crawl-delay: 10" instruction to were requesting as many as 7 pages every second and so we asked them to slow down. The number you see is seconds and you can change it to suit your server capacity, based on their crawling rate. Ten seconds between page requests is far more leisurely and stops them from asking for more pages than your server can dish up. (You can discover how fast robots and spiders are crawling by looking at your raw server logs - which show pages requested by precise times to within a hundredth of a second - available from your web host or ask your web or IT person. Your server logs can be found in the root directory if you have server access, you can usually download compressed server log files by calendar day right off your server. You'll need a utility that can expand compressed files to open and read those plain text raw server log files.) To see the contents of any robots.txt file just type robots.txt after any domain name. If they have that file up, you will see it displayed as a text file in your web browser. Click on the link below to see that file for Amazon.com http://www.Amazon.com/robots.txt You can see the contents of any website robots.txt file that way. The robots.txt shown above is what we currently use at Publish101 Web Content Distributor, just launched in May of 2005. We did an extensive case study and published a series of articles on crawler behavior and indexing delays known as the Google Sandbox. That Google Sandbox Case Study is highly instructive on many levels for webmasters everywhere about the importance of this often ignored little text file. One thing we didn't expect to glean from the research involved in indexing delays (known as the Google Sandbox) was the importance of robots.txt files to quick and efficient crawling by the spiders from the major search engines and the number of heavy crawls from bots that will do no earthly good to the site owner, yet crawl most sites extensively and heavily, straining servers to the breaking point with requests for pages coming as fast as 7 pages per second. We discovered in our launch of the new site that Google and Yahoo will crawl the site whether or not you use a robots.txt file, but MSN seems to REQUIRE it before they will begin crawling at all. All of the search engine robots seem to request the file on a regular basis to verify that it hasn't changed. Then when you DO change it, they will stop crawling for brief periods and repeatedly ask for that robots.txt file during that time without crawling any additional pages. (Perhaps they had a list of pages to visit that included the directory or files you have instructed them to stay out of and must now adjust their crawling schedule to eliminate those files from their list.) Most webmasters instruct the bots to stay out of "image" directories and the "cgi-bin" directory as well as any directories containing private or proprietary files intended only for users of an intranet or password protected sections of your site. Clearly, you should direct the bots to stay out of any private areas that you don't want indexed by the search engines. The importance of robots.txt is rarely discussed by average webmasters and I've even had some of my client business' webmasters ask me what it is and how to implement it when I tell them how important it is to both site security and efficient crawling by the search engines. This should be standard knowledge by webmasters at substantial companies, but this illustrates how little attention is paid to use of robots.txt. The search engine spiders really do want your guidance and this tiny text file is the best way to provide crawlers and bots a clear signpost to warn off trespassers and protect private property - and to warmly welcome invited guests, such as the big three search engines while asking them nicely to stay out of private areas. Copyright © August 17, 2005 by Mike Banks Valentine Google Sandbox Case Study http://publish101.com/Sandbox2 Mike Banks Valentine operates http://Publish101.com Free Web Content Distribution for Article Marketers and Provides content aggregation, press release optimization and custom web content for Search Engine Positioning http://www.seoptimism.com/SEO_Contact.htm
MORE RESOURCES:
Google News |
|
|
|
RELATED ARTICLES
Monitor and Increase Your Search Engine Visibility with the DIY SEO Tools In this three part article, you'll find many tools that any webmaster can use to monitor your site's search engine position, and use to increase the visibility of your site in major search engines like Google, Yahoo and MSN.URL Trends http://www. Yahoos Back! I was all set to write an article predicting the future of search engines, when Yahoo dropped Google and replaced it with its own engine. Now that's big news. Banned By Google And Back Again The date: 29th July 2005. The time: early morning. How To Select The Right Keywords Keyword SelectionThe most important component of search engine optimization is keyword selection. Search engines use key words and phrases to find and rank websites. 10 Quick Ways To Kick-Start Your Profit Pulling Keywords First, you must realize that targeting the right keywords or phrases is the 'key' to making any kind of profit from your site. Choosing the 'right' keywords (the exact keyword or phrase surfers type into the search engines to find your site or product) can make or break your online venture. So, Where Has Your Search Engine Been Today? Visit Google, Yahoo, MSN or one of the lesser search engines, and you get a few million results for just about any search term. Despite this impressive depth of results, most users consider only a few of the WebPages being pointed to. How to Improve Your Search Engine Positioning and Increase Traffic Today Every website has times when traffic is higher than others. However, in the downtimes you need to figure out why your traffic is lower and what you can you about it. Easy And Simple Steps To Get Listed In Search Engines Search engines are one of THE best resource for free advertising. Better your web site's search engine position better be the traffic generation. Speed Indexing - 3 Steps to Getting Your Website Listed in Google Quickly Getting your website listed in Google quickly simply requires that you know what Google is looking for and how to apply that to your site. Fortunately, what Google is looking for is pretty easy to understand and use in your marketing plan. Promoting Home Business: Tips to Increase Web Site Sales You've selected an appropriate Online Business Opportunity. That is not ALL!To run a successful online ecommerce Home business, getting the targetted users to visit your site and converting them into customers is the first and foremost thing. How to Avoid Being Dropped by the Search Engines For websites, one of the most important things in their existence is their ranking with the search engines. The reason why this is so important is because when websites are ranked high by the search engines, they get flooded with free, targeted web traffic from visitors who are looking for information or products. Search Engine Position Report Since search engines are the first stop for people on the Internet looking for goods or services, the position your website appears in search results is an important factor. If your URL shows up far down the results list, the chances of the consumer never finding you increase incrementally. What Did We Learn from the Great Search Engine Experiment! Last Week I did a Search engine Experiment. I wanted to see if I could brand myself as the coolest guy in the universe. Meta Tags - What Are They and Which Search Engines Use Them? Defining Meta Tags is much easier than explaining how they are used, and by which engines. The reason is very few engines clearly lay out what they do and do not look at, and how much emphasis they put on any one factor. The Simple Formula To Search Engines Search engines are one of the best tools to bring targeted traffic to your business. Millions of people are always using them every day to search for information that's suitable to them. Importance of Keywords in Anchor Text or Title Text Keywords are indisputably, the single most important element of an anchor text.First of all, for those who are still learning the ropes let us define an anchor text. Secrets on Website Promotion: How You Can Get a #1 Ranking for Your Website Name Within 30 Days Launching a new website with enough acceleration to rise above this ever increasing daily din needs some force. It is common to see a website with a different name and various product or service offerings with equally unrelated names. Google News - Just another article announcer? In Google's recent battle towards becoming an international news center, I've come to notice that the results delivered from Google News seems like nothing more than the articles we publish everyday. So I ask, doesn't it seem like Google News resembles an article directory of some sorts?Google News World: http://news. How to Boost Your Traffic and Profits with Content! Are you aware of how vitally important and valuable CONTENT is to your online business? In fact, content can do more to build your business and profits than just about any other resource or service available.Following is a list of 5 key ways that content can help build your traffic, subscribers, and customers starting today!. Is Google Fair? If you are the owner of a new website, trying to get a decent ranking from the mighty google, you will no doubt answer with a resounding, NO! Recent findings indicate that Google's algorithm has an ageing filter, which put in simple terms, makes it harder for a new webmaster to get high ranking in the SERP's, in the short term at least. So does this mean google favours established sites over new ones?Broad keywords appear nigh on impossible to get top 50 rankings for, and Google's first page seems like an unattainable dream.
|
|
|
|
|
| © 2004 - 2008 "Web Hosting Geeks" | Web Hosting Reviews | Customer Reviews | RealMetrics Reviews | Hosting Articles | Directory | Partners | Contacts Over 7000 articles: web hosting, web development, domain names, ecommerce, web design, site promotion, ppc advertising, seo, site promotion and many others. Web hosting reviews, ratings and awards are not based on any incentives or commissions. Names and trademarks are the properties of their respective owners. A direct link to Web Hosting Geeks (http://webhostinggeeks.com) must be provided in order to use any of the above information. Contact us for more info. Partners: Hosts by speed, Cheap Website Hosting, Free Website Hosting, Cheap Web Hosting, Top 10 Web Hosts, Top 10 Web Hosting Deals, Best Website Hosting, Free Web Hosting, Free Web Hosting, Dedicated Server Hosting, Adult Web Hosting, Web Hosting Discussions, Dedicated Server Reviews, Best Web Hosting, Web Hosting Discounts, HostProfessor.com, rsuog, halyava, PHP Website Hosting Services, Web Hosting Reviews, Hosting Uptime, Best Web Hosting Reviews, Cheap Webhosting, Web Hosting, Flash Templates, CMS Templates, Web Hosting Reviews, Website Hosting Reviews, Web Hosting Providers, Best Web Hosting, Top Web Hosting, RSUOG Web Hosting |
