Working with robots.txt for better SEO
SEO July 26th, 2009
Well, if your blog is self-hosted on a shared server or dedicated, you are always need to set up the robots.txt and make sure the search engine is crawling your site without any issues. I am not talking about blog hosted at blogspot.com, wordpress.com or other blogging platform.
You will need a robots.txt when you have buy a hosting package from hosting company. Although this is not necessary to learn by blogger but it is an alternative knowledge which you can improve your blog SEO with better optimization.
AskApache.com comes out with a good and simple listing on how to work with robots.txt such as what can be added into robots.txt. Setting up your robots.txt is recommended only when you are facing bandwidth issue or search engine crawling issue. Otherwise, it is advisable not to modified the existing robots.txt when you have a blog that runs smoothly already without any problem.
Go to AshApache.com for a better description on how to improve SEO with robots.txt for new blog or existing blog.
Good info on robots.txt. I’ve never really understood that part of blogging guess I consider that something I want my webmaster to know more about. But after reading your post and then going over to apache.com I’m seeing where I need someone to get a page or two of mine not to be crawled… very interesting but kind of makes me feel like I’m going to open a Math Book ugh!! But necessary I know. Thanks for that feeling so early in the morning lol. jj
hi jj. Glad to know that this post does help you
Yeah. It is a bit difficult to learn whats inside robots.txt This post is dedicated to blogger who looks for robots.txt references and solve the crawling issue.
Anyway, thanks for your comment, jj. Good luck!