Meta Tag - Meta Tag Install - Meta Tag Service - Meta Tag Resources.
<-- NAVIGATION
Meta Tag
Using a Robots.txt
Description
Search engines will look in your root domain
for a special file named "robots.txt" (http://www.mydomain.com/robots.txt). The file tells the robot (spider) which files it may spider (download). This system is called, The Robots Exclusion Standard. The format for the robots.txt file is special. It consists of records. Each record consists of two fields : a User-Agent line and one or more Disallow: lines. The format is:
<Field> ":" <value>
The robots.txt file should be created in Unix
line ender mode! Most good text editors will have a Unix mode or your FTP client *should* do the conversion for you. Do not attempt to use an HTML editor that does not specifically have a text mode to create a robots.txt file. For more Information to create a Robots.txt please go here
