While checking the logs of The Virtual B ... / 2006-12-08

2006-12-08 While checking the logs of The Virtual B ... 11 years ago
While checking the logs of The Virtual Bookcase to see if I made any errors in the pages, I noticed another webbot requesting loads of pages, the bot related to Webclipping.com. I looked at their site and noticed that they are no search engine, they just scour the web looking for people having opinions about their paying customers. Which means: those requests by their spider will never result in visitors to The Virtual Bookcase. I only want to allow webbots that result in visitors to the site. So, I wanted to know what to put in robots.txt to stop them.. couldn't find that on their website so I asked via e-mail. The answer is (paraphrased): "We decide on our own which sites to visit.". My answer is to block them.

Tags: , , ,

, reachable as koos+website@idefix.net. PGP encrypted e-mail preferred.

PGP key 5BA9 368B E6F3 34E4 local copy PGP key 5BA9 368B E6F3 34E4 via keyservers pgp key statistics for 0x5BA9368BE6F334E4 Koos van den Hout
RSS
Other webprojects: Camp Wireless, wireless Internet access at campsites, The Virtual Bookcase, book reviews, Weather maps
This page generated in 0.008426 seconds.