While checking the logs of The Virtual B ... / 2006-12-08

While checking the logs of The Virtual Bookcase to see if I made any errors in the pages, I noticed another webbot requesting loads of pages, the bot related to Webclipping.com. I looked at their site and noticed that they are no search engine, they just scour the web looking for people having opinions about their paying customers. Which means: those requests by their spider will never result in visitors to The Virtual Bookcase. I only want to allow webbots that result in visitors to the site. So, I wanted to know what to put in robots.txt to stop them.. couldn't find that on their website so I asked via e-mail. The answer is (paraphrased): "We decide on our own which sites to visit.". My answer is to block them.