spidered/crawled but not to be seen.. - Printable Version
-Shoutbox (https://shoutbox.menthix.net)
+-- Forum: MsgHelp Archive (/forumdisplay.php?fid=58)
+--- Forum: Skype & Technology (/forumdisplay.php?fid=9)
+---- Forum: Tech Talk (/forumdisplay.php?fid=17)
+----- Thread: spidered/crawled but not to be seen.. (/showthread.php?tid=30268)
spidered/crawled but not to be seen.. by jren207 on 08-21-2004 at 06:20 PM
How is it that my site has been spidered many times but hasn't appeared on one search engine. (Except for Google, but it doesn't even have my site's description on it.)
Look at this screenshot...
Click for bigger picture
( this can be found at http://www.jren207.com/cgi-bin/spydertrax/spyderview.cgi )
As you can see, it has been spidered by MSN Search, sooo many times and I searched for "jren207" and all i got was pages from sites like these forums which I have subscribed to that have been spidered. Not even my site!!
RE: spidered/crawled but not to be seen.. by Titty on 08-21-2004 at 06:27 PM
Wow, that's interesting.
Did you install a script or something for that page?
RE: spidered/crawled but not to be seen.. by jren207 on 08-21-2004 at 06:35 PM
If you mean the page in the screenshot, yep. It's called Spydertrax and you can get it at http://www.darrinward.com/
It tracks bots that crawl your website.
You put: <?php virtual("/cgi-bin/spydertrax.cgi"); ?> in the pages you want to track and it tells you if they have been spidered.
RE: spidered/crawled but not to be seen.. by BEWARE^^ on 08-21-2004 at 11:50 PM
well i think that its usefull but at they other side i think i'll never download it srry 
RE: spidered/crawled but not to be seen.. by jren207 on 08-22-2004 at 12:06 AM
thtz ok, i was tellin ya the script I used for tracking....
... but still... can someone explain why my site isn't appearing all that often in MSN search even though it's been spidered at least 15 times!!!
RE: spidered/crawled but not to be seen.. by Mippo on 08-22-2004 at 06:34 AM
Maybe those sites (except for Google) require those meta tags before you can be added into their databases? Or do you already have those? 
RE: spidered/crawled but not to be seen.. by BEWARE^^ on 08-22-2004 at 10:19 AM
Maybe he has a meta tag coz else he would'nt have spidered it i guess
RE: spidered/crawled but not to be seen.. by jren207 on 08-22-2004 at 12:31 PM
Yep, I do have meta tags, or else there would be NO chance of it even being spidered lol.
look
code:
<META HTTP-EQUIV="pics-label" content='(pics-1.1 "http://www.icra.org/ratingsv02.html" comment "ICRAonline EN v2.0" l gen true for "http://jren207.no-ip.com" r (nz 1 vz 1 lz 1 oz 1 cb 1) "http://www.rsac.org/ratingsv01.html" l gen true for "http://jren207.no-ip.com" r (n 0 s 0 v 0 l 0))' >
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=ISO-8859-1" >
<META HTTP-EQUIV="EXPIRES" CONTENT="0" >
<META NAME="RESOURCE-TYPE" CONTENT="DOCUMENT" >
<META NAME="DISTRIBUTION" CONTENT="GLOBAL" >
<META NAME="AUTHOR" CONTENT="JR!" >
<META NAME="COPYRIGHT" CONTENT="Copyright JR! - JR! Web Design2004" >
<META NAME="KEYWORDS" CONTENT="JR,jren207,jdr207,computer,php,html,google,poll,coding,apache,jason,rennie,notepad,ie,internet,download,jasondanielrennie207,search,original,home" >
<META NAME="DESCRIPTION" CONTENT="A site with tips and tricks and many flavours of content to please and help with web development." >
<META NAME="ROBOTS" CONTENT="ALL" >
<META NAME="REVISIT-AFTER" CONTENT="1 DAYS" >
<META NAME="RATING" CONTENT="GENERAL" >
<META NAME="GENERATOR" CONTENT="JR! Hand Coding JR! Web Design" >
<META NAME="OWNER" CONTENT="JR!" >
<META NAME="MSSmartTagsPreventParsing" CONTENT="TRUE" >
RE: spidered/crawled but not to be seen.. by BEWARE^^ on 08-22-2004 at 04:37 PM
thats what i was saying lol 
RE: spidered/crawled but not to be seen.. by WDZ on 08-23-2004 at 12:55 AM
Wow... that's a lot of meta tags... maybe too many? 
Maybe some of them are conflicting with some search engines...
I've read that sometimes you will be penalized if some of your meta info is too long, or if you use keywords that aren't actually related to the content of your site.
RE: spidered/crawled but not to be seen.. by jren207 on 08-23-2004 at 02:06 PM
Right, I think it might be the keywords. The first tag makes sure my site isn't filtered by adult filters which is a must-have tag. And the rest make sure people know it's MY site 
and the rest instruct bots etc.
I'll try and think about the keywords.
RE: spidered/crawled but not to be seen.. by BEWARE^^ on 08-23-2004 at 03:58 PM
quote:
Originally posted by WDZ
Wow... that's a lot of meta tags... maybe too many?
Maybe some of them are conflicting with some search engines...
I've read that sometimes you will be penalized if some of your meta info is too long, or if you use keywords that aren't actually related to the content of your site.
maybe you have to check it out with some one else who has it and the compare it with yours and what wdz is telling could olso be a problem that its nor reacting to the search engines
RE: spidered/crawled but not to be seen.. by jren207 on 08-23-2004 at 04:24 PM
It's being spidered but not show in search results with the keywords I put in. It's in Google and now has a title and desc. . Stupid Microsoft lol.... 
RE: spidered/crawled but not to be seen.. by BEWARE^^ on 08-23-2004 at 05:17 PM
So he has found/detected him yet or not