But...if you have a forum and not have a way of inserting robots doesn't this robots.txt get robots to your site or am I missing the point of this thing totally? yes there are disallow lines there but the first line is to allow all robots with a *
if you dont have a robots.txt it doesn't mean you wont get spidered. If your site is linked to and/or you submit it to search engine it will get spidered.
Robots.txt just tells the spiders which parts to not index, such as private areas etc
But...if you have a forum and not have a way of inserting robots doesn't this robots.txt get robots to your site or am I missing the point of this thing totally? yes there are disallow lines there but the first line is to allow all robots with a *
Posts: 7106 Reputation: 107
– / /
Joined: Mar 2002
RE: robots.txt?
quote:Originally posted by Savior
yes there are disallow lines there but the first line is to allow all robots with a *
The first line basically means "all bots should obey the following rules." You could use a bot name instead of the * to tell specific bots what not to spider.
quote:User-agent
The value of this field is the name of the robot the record is describing access policy for.
If more than one User-agent field is present the record describes an identical access policy for more than one robot. At least one field needs to be present per record.
The robot should be liberal in interpreting this field. A case insensitive substring match of the name without version information is recommended.
If the value is '*', the record describes the default access policy for any robot that has not matched any of the other records. It is not allowed to have multiple such records in the "/robots.txt" file.