RoboGen Screenshot
| RoboGen is an editor for robot exclusion files, ROBOTS.TXT. Search engines use the information in the ROBOTS.TXT file when they index a web site. A properly maintained robot exclusion file can help web site owners gain better search rankings. The main RoboGen editor allows you to log into your web site through FTP and select the files and directories you would like to disallow. Once the ROBOTS.TXT file is created, you can save it to your local disk or upload it directly to the web server through FTP with a single click. RoboGen contains a pick-list of over 180 known search engine user agents to make more advanced customization easy. The RoboGen package also comes with a companion program called RoboTag. It can be used to quickly create robot exclusion meta-tags that can be inserted into HTML pages. The robot meta-tags have essentially the same purpose as ROBOTS.TXT files.
|
