Keith Devens .com

What browser addon automatically spiders all pages linked from a page?

In my referrers I commonly see things like this (first URI was the referrer for all the other requests):

/weblog/archive/2002/Dec/26/LordOfTheRings - 86

/weblog/archive/2002/Dec/22/PoorIraqiSoldiers... - 1
/weblog/archive/2002/Dec/30/JamesBrownRULES/rss - 1
/weblog/archive/2002/Dec/28/FunnyCoding/rss - 1
/weblog/archive/2002/Dec/22/WriteCode/rss - 1
/weblog/archive/2002/Dec/23/MetroidMusic/rss - 1
/weblog/archive/2002/Dec/22/WriteCode - 1
/weblog/archive/2002/Dec/31/ProgrammingLanguagesUserInterface - 1
/weblog/archive/2002/Dec/31/ProgrammingLanguagesUserInterface/rss - 1
/weblog/archive/2002/Dec/28/NoCompile-TimeIncludeFunctionPHP - 1
/weblog/archive/2002/Dec/23/FireflyTimePersonalStories/rss - 1
/weblog/archive/2002/Dec/23/GoogleKnowsSpellcheckName - 1
/weblog/archive/2002/Dec/28/NoCompile-TimeIncludeFunctionPHP/rss - 1
/weblog/archive/2002/Dec/28/FunnyCoding - 1
/weblog/archive/2002/Dec/23/GoogleKnowsSpellcheckName/rss - 1
/weblog/archive/2002/Dec/21/Firefly,OneMoreTime - 1
/weblog/archive/2003/Jan/01/WillFiltersKillSpam - 1
/weblog/archive/2002/Dec/23/TotalInformationAwareness - 1
/weblog/archive/2002/Dec/31/BenSteinAmericanEnterprise/rss - 1
/weblog/archive/2002/Dec/22/SeasonOfFriends - 1
/weblog/archive/2002/Dec/31/BenSteinAmericanEnterprise - 1
/weblog/archive/2002/Dec/23/MetroidMusic - 1
/weblog/archive/2002/Dec/28/TheLadderTheory - 1
/weblog/archive/2002/Dec/23/Schaeffer - 1
/weblog/archive/2002/Dec/23/ProgrammingFunForDay/rss - 1
/weblog/archive/2002/Dec/30/DontGoShoppingHungry - 1
/weblog/archive/2002/Dec/28/MeaningfulProgramming/rss - 1
/weblog/archive/2002/Dec/23/ProgrammingFunForDay - 1
/weblog/archive/2002/Dec/28/MoreQuickLinks/rss - 1
/weblog/archive/2002/Dec/28/MeaningfulProgramming - 1
/weblog/archive/2002/Dec/23/BabyFactories - 1
/weblog/archive/2002/Dec/23/ScienceMagazinesHighlightOf2002/rss - 1
/weblog/archive/2002/Dec/28/MoreQuickLinks - 1
/weblog/archive/2002/Dec/28/ReasonsDidntTwoTowersNearlyLotR/rss - 1
/weblog/archive/2002/Dec/23/ScienceMagazinesHighlightOf2002 - 1
/weblog/archive/2002/Dec/23/ChristianityAndFreeWill/rss - 1
/weblog/archive/2002/Dec/30/DontGoShoppingHungry/rss - 1
/weblog/archive/2002/Dec/28/ReasonsDidntTwoTowersNearlyLotR - 1
/weblog/archive/2002/Dec/30/JamesBrownRULES - 1
/weblog/archive/2002/Dec/30/ThePerfectParser/rss - 1
/weblog/archive/2002/Dec/23/TotalInformationAwareness/rss - 1
/weblog/archive/2002/Dec/23/ChristianityAndFreeWill - 1
/weblog/archive/2002/Dec/23/IraqInvitesTheCIA - 1
/weblog/archive/2002/Dec/25/CharlieBrownChristmas/rss - 1
/weblog/archive/2002/Dec/30/ThePerfectParser - 1
/weblog/archive/2002/Dec/30/Relbookmark/rss - 1
/weblog/archive/2002/Dec/28/TheLadderTheory/rss - 1
/weblog/archive/2002/Dec/23/BillFrist - 1
/weblog/archive/2002/Dec/23/Schaeffer/rss - 1
/weblog/archive/2002/Dec/30/Relbookmark - 1
/weblog/archive/2002/Dec/23/IraqInvitesTheCIA/rss - 1
/weblog/archive/2002/Dec/23/BillFrist/rss - 1
/weblog/archive/2002/Dec/25/SnowballFight/rss - 1
/weblog/archive/2002/Dec/28/PHP4.3.0Released - 1
/weblog/archive/2002/Dec/28/PHP4.3.0Released/rss - 1
/weblog/archive/2002/Dec/28/IanMcKellenPlayDumbledore - 1
/weblog - 1
/weblog/archive/2002/Dec/25/ItllBeGreat - 1
/weblog/archive/2002/Dec/24/YouGottaSeeThis - 1
/weblog/archive/2002/Dec/28/IanMcKellenPlayDumbledore/rss - 1
/weblog/archive/2002/Dec/30/PoliticsForToday... - 1
/weblog/archive/2002/Dec/28/D.B.WoodsideInterview - 1
/weblog/archive/2002/Dec/26/ConversationBeach/rss - 1
/weblog/archive/2002/Dec/26/LetsBlowStuffUp - 1
/weblog/archive/2002/Dec/25/ItllBeGreat/rss - 1
/weblog/archive/2002/Dec/25/ChineseFoodForChristmas - 1
/weblog/archive/2002/Dec/27/OfficialLogoForDarwin - 1
/weblog/archive/2002/Dec/23/BabyFactories/rss - 1
/weblog/archive/2002/Dec/27/Reformed.OrgDown - 1
/weblog/archive/2002/Dec/25/ChineseFoodForChristmas/rss - 1
/weblog/archive/2002/Dec/26/LetsBlowStuffUp/rss - 1
/weblog/archive/2002/Dec/25/SnowballFight - 1
/weblog/archive/2002/Dec/27/Reformed.OrgDown/rss - 1
/weblog/archive/2002/Dec/25/Best.Cheesecake.. - 1
/weblog/rss - 1
/weblog/archive/2002/Dec/26/ConversationBeach - 1
/weblog/archive/2002/Dec/27/OfficialLogoForDarwin/rss - 1
/weblog/archive/2002/Dec/28/QuickLinks - 1
/weblog/archive/2002/Dec/25/CharlieBrownChristmas - 1
/weblog/archive/2002/Dec/24/McGreeveysDamnMoron - 1
/weblog/archive/2002/Dec/24/McGreeveysDamnMoron/rss - 1
/weblog/archive/2002/Dec/23/FireflyTimePersonalStories - 1
/weblog/archive/2002/Dec/26/LordOfTheRings/rss - 1
/weblog/archive/2002/Dec/24/YouGottaSeeThis/rss - 1
/weblog/atom - 1
/weblog/archive/2002/Dec/28/QuickLinks/rss - 1
/weblog/archive/2002/Dec/28/D.B.WoodsideInterview/rss - 1

Sorry to post the whole thing here, but that was the only way to give a sense of how annoying this is to me, and how much bandwidth and server resources these people waste. And, annoyingly, it doesn't just spider, but drops referrers everywhere. This must be a browser plugin that some people are using (or could it be one of those "accelerators" that some dial-up Internet providers provide?). I've checked my Apache log and there are no identifying marks such as a user agent appended to the browser's user agent string.

Anyone have any clue what this is?

Permalink | 3 Comments | Tags: This website | id 7929

20:51 (utc-6)
edited: 21:55 (utc-6)