help you create a robots.txt file
By Sumantra Roy
Robogen is a software for Windows
98 and Windows NT based computers which can help you create robots.txt
files without having to worry about the correct syntax of a robots.txt
file. It comes in two editions: the free Limited edition and the
which costs $12.95. You can get Robogen from http://www.rietta.com/robogen/
this article, I'll tell how you can use Robogen to simplify the process
of creating a robots.txt file. This article assumes that you have read
article on what a robots.txt file is and how you can manually create
If you missed the article, it is available at http://www.1stSearchRanking.com/t.cgi?3319&robots.htm
would recommend that you download the trial version of the Standard
Edition. The Limited edition does not support as many search engines as
the Standard Edition.
you have downloaded Robogen and installed it in your hard drive, start
the program. Here are the steps that you should follow to create a
1: Click on the Logon FTP
button. In the dialog box that appears, type in the address of the FTP
for your web site, make sure that the "This server requires me to log
option is checked, and then type in the username and the password which
use to access the server in the appropriate boxes.
depending on how your web host has set up the server, the files for
web site may be present in the root directory of the server itself. Or,
may be present in a sub-directory of the root directory. In the latter
case, click on the "Use specified root" option and type the relevant
sub-directory in the text box besides the option. If you don't know in
which sub-directory the files for your web site are present, ask your
web host to give it to you.
Now click on OK. After a while, Robogen will display the directories
files which are present in your site in the right pane of the main
program window. If you find that Robogen has not displayed the files in
site, it means that you did not enter the correct sub-directory in
the files of your web site are present. In this case, ask you web host
give the correct sub-directory to you.
2: In the left pane of the main
Robogen window, click on the small button present to the right of the
combo box at the top. In the menu that appears, select "Popular Search
3: Now, click on the drop-down
combo box and choose the search engine that you are targeting (say, AltaVista
). In the right pane, select a page
which is NOT meant for AltaVista. Now click on the "
button. You will find that the file which you had selected in the right
pane is now present in the left pane. In this way, click on each of the
pages which are not
meant for AltaVista and press the "
Repeat step 3 for
each of the
search engines that you are targeting. Just select the search engine
that you are targeting from the drop-down combo box, select the pages
which are not meant for the search engine one by one from the right
pane, and click on the "
<<<" button after clicking on each age. Repeat
all the search engines that you are targeting.
Step 4: Upload the
file to your
server using the "Upload Robots.Txt" button. If you want, you can also
Save menu item from the File menu to save a copy of the robots.txt file
in your hard drive.
Article by Sumantra
Sumantra is one of the most respected search engine positioning
specialists on the Internet. To have Sumantra's company place your site
at the top of the search engines, go to http://www.1stSearchRanking.com/t.cgi?3319 For more advice on how you can
web site to the top of the search engines, subscribe to his FREE
by going to http://www.1stSearchRanking.com/t.cgi?3319&newsletter.htm