Robots.txt Files - How to Create a Robots Text File - robots.txt

Question: How do I create a robot.stxt file?

A robots.txt is a plain text file that instructs search engine robots and web crawlers on how to crawl your website. It contains simple commands (directives) that allow or disallow (forbid) robots access to your website.

Answer: If your entire website is open for inspection, that is, you want to allow all robots and web crawlers to be able to access all parts of your website, creating a robots.txt file is incredibly simple:

Open Notepad (or another plain text editor – do not use Word!)

Type in the following exactly as it appears below:

User-agent: *
Disallow:

Save the file name as: robots.txt ("robots" should be all lower case and plural).

Upload it to your root directory – and you are done!

Tip: The correct and ONLY location to put a robots.txt file is in your root directory.

Location example: http://yourdomainname.com/robots.txt

via womeninbusiness.about.com