What happened to the Messenger Plus! forums on msghelp.net?
Shoutbox » MsgHelp Archive » Skype & Technology » Tech Talk » robots.txt?

Pages: (2): « First [ 1 ] 2 » Last »
robots.txt?
Author: Message:
Savior
Full Member
***

Avatar

Posts: 292
Reputation: -1
Joined: Mar 2005
O.P. robots.txt?
Here is a question I know this site has a robots.txt file how much more traffic does something like that bring?

thanks
05-09-2005 01:34 PM
Profile E-Mail PM Web Find Quote Report
Jhrono
Veteran Member
*****


Posts: 1791
Reputation: 25
33 / Male / Flag
Joined: Jun 2004
RE: robots.txt?
quote:
Originally posted by Savior


Here is a question I know this site has a robots.txt file how much more traffic does something like that bring?

thanks

?!
what do you mean by a robot.txt file?:S...
05-09-2005 01:37 PM
Profile E-Mail PM Find Quote Report
Savior
Full Member
***

Avatar

Posts: 292
Reputation: -1
Joined: Mar 2005
O.P. RE: robots.txt?
http://shoutbox.menthix.net/robots.txt
05-09-2005 01:39 PM
Profile E-Mail PM Web Find Quote Report
-dt-
Scripting Contest Winner
*****

Avatar
;o

Posts: 1819
Reputation: 74
36 / Male / Flag
Joined: Mar 2004
RE: robots.txt?
robot.txt file just lets you block "spiders" from fetching certain parts of your website it dosnt bring more traffic...

[Image: dt2.0v2.png]      Happy Birthday, WDZ
05-09-2005 01:39 PM
Profile PM Web Find Quote Report
Savior
Full Member
***

Avatar

Posts: 292
Reputation: -1
Joined: Mar 2005
O.P. RE: robots.txt?
But don't robots supposed to index your entire site?
05-09-2005 01:42 PM
Profile E-Mail PM Web Find Quote Report
Dempsey
Scripting Contest Winner
*****

Avatar
http://AdamDempsey.net

Posts: 2395
Reputation: 53
38 / Male / Flag
Joined: Jul 2003
RE: robots.txt?
yea robots index your site, but a robots.txt file tells them which bits not to index
SoundPacks   -   Scripts   -   Skins

that's not a bug, thats an unexpected feature
05-09-2005 01:50 PM
Profile E-Mail PM Web Find Quote Report
Savior
Full Member
***

Avatar

Posts: 292
Reputation: -1
Joined: Mar 2005
O.P. RE: robots.txt?
But...if you have a forum and not have a way of inserting robots doesn't this robots.txt get robots to your site or am I missing the point of this thing totally?  yes there are disallow lines there but the first line is to allow all robots with a *
05-09-2005 01:56 PM
Profile E-Mail PM Web Find Quote Report
Dempsey
Scripting Contest Winner
*****

Avatar
http://AdamDempsey.net

Posts: 2395
Reputation: 53
38 / Male / Flag
Joined: Jul 2003
RE: robots.txt?
if you dont have a robots.txt it doesn't mean you wont get spidered.  If your site is linked to and/or you submit it to search engine it will get spidered.

Robots.txt just tells the spiders which parts to not index, such as private areas etc
SoundPacks   -   Scripts   -   Skins

that's not a bug, thats an unexpected feature
05-09-2005 01:59 PM
Profile E-Mail PM Web Find Quote Report
Savior
Full Member
***

Avatar

Posts: 292
Reputation: -1
Joined: Mar 2005
O.P. RE: robots.txt?
Ok thanks

But...if you have a forum and not have a way of inserting robots doesn't this robots.txt get robots to your site or am I missing the point of this thing totally?  yes there are disallow lines there but the first line is to allow all robots with a *
05-09-2005 02:03 PM
Profile E-Mail PM Web Find Quote Report
WDZ
Former Admin
*****

Avatar

Posts: 7106
Reputation: 107
– / Male / Flag
Joined: Mar 2002
RE: robots.txt?
quote:
Originally posted by Savior
yes there are disallow lines there but the first line is to allow all robots with a *
The first line basically means "all bots should obey the following rules." You could use a bot name instead of the * to tell specific bots what not to spider.

quote:
User-agent
The value of this field is the name of the robot the record is describing access policy for.

If more than one User-agent field is present the record describes an identical access policy for more than one robot. At least one field needs to be present per record.

The robot should be liberal in interpreting this field. A case insensitive substring match of the name without version information is recommended.

If the value is '*', the record describes the default access policy for any robot that has not matched any of the other records. It is not allowed to have multiple such records in the "/robots.txt" file.

http://www.robotstxt.org/wc/norobots.html
05-09-2005 03:06 PM
Profile PM Web Find Quote Report
Pages: (2): « First [ 1 ] 2 » Last »
« Next Oldest Return to Top Next Newest »


Threaded Mode | Linear Mode
View a Printable Version
Send this Thread to a Friend
Subscribe | Add to Favorites
Rate This Thread:

Forum Jump:

Forum Rules:
You cannot post new threads
You cannot post replies
You cannot post attachments
You can edit your posts
HTML is Off
myCode is On
Smilies are On
[img] Code is On