What Is Robots.Txt? Robots.txt is a file that contain path which cannot crawled by bot most of time search-engine bots like google bot or etc. It tells search-engine that this directory is private & can not be crawled by them. If yo are site owner & want to make robots.txt file , then go following link , it will create robots.txt file for you. http://www.mcanerin.com/EN/search-engine/robots-txt.asp so just for now , robots.txt is pretty much what websites use to block certain pages from search engines. Here is a sample : http://www.whitehouse.gov/robots.txt First Method Now this method is very rare & the web-master would have to be stupid to do this, but you'll be surprised how many stupid people there are in the world. This one is simple, go to one of the disallowed directories & look in the source. Sometimes web-master leave comments there to give hints like passwords/ or user-names. You never know you might find something juicy. :] With this info you could...
Comments
Post a Comment