robots.txt explained!

[Nomar]

The robots.txt file is a simple text file that must be placed in your root directory (http://www.example.com/robots.txt). It tells the search engine spider which web pages on your website should be indexed and which web pages should be ignored.


You can use a simple text editor to create a robots.txt file. The content of a robots.txt file consists of so-called "records".
A record contains the information for a special search engine. Each record consists of two fields: the user agent line and one or more Disallow lines. Here's an example:

User-agent: googlebot
Disallow: /cgi-bin/

This robots.txt file would allow the "googlebot", which is the search engine spider of Google, to retrieve every page from your site except for files from the "cgi-bin" directory. All files in the "cgi-bin" directory will be ignored by googlebot.

[Doskono]

Ah this is pretty interesting. Does it actually work? I try it out!

[Adulu]

It's working fine and so fast in google

[Nomar]

Yup, it does work !

Webmasters have found out that Google seems to be experimenting with a Noindex commands for the robots.txt file. It basically seems to do the same as the Disallow command so it's not clear why Google is using this command.

Other commands that might be tested by Google are Noarchive and Nofollow. However, none of these commands is official yet.

[Massive]

Didn't know that yet ! I never use these robots.txt, is that bad?

[surfville]

that one helps? i never had known the use of this robot .txt? thanks for sharing anyways.

[attagirl]

In all honesty can you please explain the benefit as far as PR goes. I think it would have to do something with improvement right?

[phntje]

cool, i am going to try this...

[surfville]

Do you have the code for it by the way? and some more useful tips in using it. This thing would be cool unless all people in your competition will know it also and again can't get the edge.

[Nomar]

Everyone knows about it ! It is quite old But you can use it if you want Search Engines to stay out of certain of your pages!