While checking the logs of The Virtual B ... / 2006-12-08

2006-12-08 While checking the logs of The Virtual B ...
While checking the logs of The Virtual Bookcase to see if I made any errors in the pages, I noticed another webbot requesting loads of pages, the bot related to Webclipping.com. I looked at their site and noticed that they are no search engine, they just scour the web looking for people having opinions about their paying customers. Which means: those requests by their spider will never result in visitors to The Virtual Bookcase. I only want to allow webbots that result in visitors to the site. So, I wanted to know what to put in robots.txt to stop them.. couldn't find that on their website so I asked via e-mail. The answer is (paraphrased): "We decide on our own which sites to visit.". My answer is to block them.

Tags: , , ,

IPv6 check

Running test...
, reachable as koos+website@idefix.net. PGP encrypted e-mail preferred. PGP key 5BA9 368B E6F3 34E4 local copy PGP key 5BA9 368B E6F3 34E4 via keyservers

RSS
Meningen zijn die van mezelf, wat ik schrijf is beschermd door auteursrecht. Sommige publicaties bevatten een expliciete vermelding dat ze ongevraagd gedeeld mogen worden.
My opinions are my own, what I write is protected by copyrights. Some publications contain an explicit license statement which allows sharing without asking permission.
Other webprojects: Camp Wireless, wireless Internet access at campsites, The Virtual Bookcase, book reviews
This page generated by $Id: newsitem.cgi,v 1.57 2022/02/15 21:48:18 koos Exp $ in 0.006825 seconds.