Comments on: Tell the Robots Where Not to Go http://www.lgr.ca/blog/2007/03/tell-the-robots-where-not-to-go.html Bringing the Internet into Focus! Thu, 15 Dec 2011 22:47:36 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: LGR http://www.lgr.ca/blog/2007/03/tell-the-robots-where-not-to-go.html#comment-7847 LGR Mon, 26 Sep 2011 16:51:37 +0000 http://www.blog2.lgr.ca/2007/03/tell-the-robots-where-not-to-go.html#comment-7847 Very true! My robots.txt file is actually out of date, since there are folders listed that no longer exist. In theory one could list a folder and use it to trap hackers looking for exploits. I am sure a quick Google search would find something. Very true! My robots.txt file is actually out of date, since there are folders listed that no longer exist. In theory one could list a folder and use it to trap hackers looking for exploits. I am sure a quick Google search would find something.

]]> By: Alex http://www.lgr.ca/blog/2007/03/tell-the-robots-where-not-to-go.html#comment-7842 Alex Mon, 26 Sep 2011 16:26:00 +0000 http://www.blog2.lgr.ca/2007/03/tell-the-robots-where-not-to-go.html#comment-7842 It's worth pointing out (and/or reminding) that robots.txt is always open and visible to anyone who puts that filename in the url. (so I could put in www.lgr.ca/robots.txt and see the robots.txt file) So don't hide important / super-secret files and folders by listing them in robots.txt . It's the first place hackers look to see if they can find what's now indexed. It’s worth pointing out (and/or reminding) that robots.txt is always open and visible to anyone who puts that filename in the url. (so I could put in http://www.lgr.ca/robots.txt and see the robots.txt file)

So don’t hide important / super-secret files and folders by listing them in robots.txt . It’s the first place hackers look to see if they can find what’s now indexed.

]]> By: WordPress Ninja Affiliate for Free - LGR Internet Solutions http://www.lgr.ca/blog/2007/03/tell-the-robots-where-not-to-go.html#comment-1058 WordPress Ninja Affiliate for Free - LGR Internet Solutions Wed, 15 Jul 2009 18:49:22 +0000 http://www.blog2.lgr.ca/2007/03/tell-the-robots-where-not-to-go.html#comment-1058 [...] BlogMechanics KeywordLink allow you to do this. You can also add your custom URL’s to your robots.txt file to prevent the search engine spiders from following your affiliate links. For example I could [...] [...] BlogMechanics KeywordLink allow you to do this. You can also add your custom URL’s to your robots.txt file to prevent the search engine spiders from following your affiliate links. For example I could [...]

]]> By: LGR http://www.lgr.ca/blog/2007/03/tell-the-robots-where-not-to-go.html#comment-84 LGR Mon, 02 Apr 2007 19:18:00 +0000 http://www.blog2.lgr.ca/2007/03/tell-the-robots-where-not-to-go.html#comment-84 It is just a little to metaphysical for me these days Rhett. It is just a little to metaphysical for me these days Rhett.

]]> By: Rhett http://www.lgr.ca/blog/2007/03/tell-the-robots-where-not-to-go.html#comment-83 Rhett Mon, 02 Apr 2007 18:22:00 +0000 http://www.blog2.lgr.ca/2007/03/tell-the-robots-where-not-to-go.html#comment-83 What about the stars Lee? :'( What about the stars Lee? :’(

]]> By: LGR http://www.lgr.ca/blog/2007/03/tell-the-robots-where-not-to-go.html#comment-82 LGR Sat, 31 Mar 2007 20:39:00 +0000 http://www.blog2.lgr.ca/2007/03/tell-the-robots-where-not-to-go.html#comment-82 Rhett, Yes usually the root web folder is called www or public_html. I know for a fact on your server that is what it is called. For Wordpress, I ran across a good post the other day at <a HREF="http://www.dailyblogtips.com/create-a-robotstxt-file/" REL="nofollow">Daily Blog Tips</a> that had a good sample robots.txt file. <br/><br/>As for where you don't want the bots to go? Look at all the files/folders in your www folder and ask yourself "do you want search engines to index this folder?", if the answer is no then disallow it. This is why I often disallow image folders. I don't need the images I use to build a website to be index and stored in Google Images for example. If it is a photo gallery that is different. <br/><br/>Remember though, by disallowing a folder in the robots.txt file, the robots will ignore the folder, but that does not mean people will, since people can read the robots.txt file. Often those you don't want reading the file will look at it to see if there are folders listed that might allow them some kind of access to your site. If you really don't want people or robots to have access to a folder I would password protect it using .htaccess. If you use cPanel this can be done easily under the Web Protect section.<br/><br/>As for the mp3 question, as long as you own the copyright to those works, you could put them in a folder and disallow robots from reading it. That will stop robots from looking in the folder. This does not mean that people won't find them and download them.<br/><br/>If the mp3's are others copyrighted work, don't put them on the web. Many hosts will remove your account if you have copyrighted material on your account.<br/><br/>What's in the cgi-bin folder? These days not much since PHP has become so popular. It was used a lot more in the past for PERL scripts and other programs. There also might be reasons to not block bots from that folder. If you actually used a script that output data from that folder you might want to allow bots access. I just disallow it since it is not used much anymore. Rhett, Yes usually the root web folder is called www or public_html. I know for a fact on your server that is what it is called. For WordPress, I ran across a good post the other day at Daily Blog Tips that had a good sample robots.txt file.

As for where you don’t want the bots to go? Look at all the files/folders in your www folder and ask yourself “do you want search engines to index this folder?”, if the answer is no then disallow it. This is why I often disallow image folders. I don’t need the images I use to build a website to be index and stored in Google Images for example. If it is a photo gallery that is different.

Remember though, by disallowing a folder in the robots.txt file, the robots will ignore the folder, but that does not mean people will, since people can read the robots.txt file. Often those you don’t want reading the file will look at it to see if there are folders listed that might allow them some kind of access to your site. If you really don’t want people or robots to have access to a folder I would password protect it using .htaccess. If you use cPanel this can be done easily under the Web Protect section.

As for the mp3 question, as long as you own the copyright to those works, you could put them in a folder and disallow robots from reading it. That will stop robots from looking in the folder. This does not mean that people won’t find them and download them.

If the mp3′s are others copyrighted work, don’t put them on the web. Many hosts will remove your account if you have copyrighted material on your account.

What’s in the cgi-bin folder? These days not much since PHP has become so popular. It was used a lot more in the past for PERL scripts and other programs. There also might be reasons to not block bots from that folder. If you actually used a script that output data from that folder you might want to allow bots access. I just disallow it since it is not used much anymore.

]]> By: Rhett http://www.lgr.ca/blog/2007/03/tell-the-robots-where-not-to-go.html#comment-81 Rhett Sat, 31 Mar 2007 17:27:00 +0000 http://www.blog2.lgr.ca/2007/03/tell-the-robots-where-not-to-go.html#comment-81 Lee do you ever wonder if when I leave my apartment and look up at the stars and you leave your house and you look up at the stars if we are gazing upon the same one, somehow connected through time and space... :D<br/><br/>I swear I was just reading about this the other day and now it's here. Well, I have a quick question. I think what's going on here is you are talking to a certain audience that knows one more thing than I do and so I can't quite make sense of what you are saying. <br/><br/>If I create said text file do I just put it in the "www" folder? Would this be a way for say bots not to find like an audio folder with mp3s so they don't get ripped off? (This question might be too large) How do I know where I want bots to go and not want them to go? What's in the cgi bin? Lee do you ever wonder if when I leave my apartment and look up at the stars and you leave your house and you look up at the stars if we are gazing upon the same one, somehow connected through time and space… :D

I swear I was just reading about this the other day and now it’s here. Well, I have a quick question. I think what’s going on here is you are talking to a certain audience that knows one more thing than I do and so I can’t quite make sense of what you are saying.

If I create said text file do I just put it in the “www” folder? Would this be a way for say bots not to find like an audio folder with mp3s so they don’t get ripped off? (This question might be too large) How do I know where I want bots to go and not want them to go? What’s in the cgi bin?

]]>