News items for tag virtualbookcase - Koos van den Hout

Recently I was looking at some reports of the affiliate income generated by The Virtual Bookcase and it hasn't generated a cent in a few years.

This is probably fully related to the fact I haven't paid any attention to the site both in code and content for years. The only commits in 2022 were due to a vulnerability found in the site. Most commits to the code for the site were before 2010. Time to admit to myself I need to stop doing this. There are other things that take my time and give me joy.

If someone else wants to take over: get in touch. I'm not sure which parts of the database are of any use to people and which parts I shouldn't transfer due to Dutch privacy laws but we'll figure it out. If nobody wants it, I will start giving 410 gone status from 1 september 2023 and end the domain registration in November 2023.

The original announcement of starting the site, dated 28 march 1999: I've created a virtual bookcase with an overview of books I like/read.. visit the site too! which is also the oldest newsitem in my archive.

I wanted to use wapiti as scanner to check for other vulnerabilities in The Virtual Bookcase after receiving a report about a cross-site scripting vulnerability. Wapiti is open source and free, which is a fitting price for scanning a hobby project site.

I quickly ran into wapiti taking hours to scan because of the URL structure of the site: all /book/detail/x/y URLs map to one handler that deals with the X and Y parameters in SQL queries. Yes those queries are surrounded by very defensive checking and I use positional parameters. Everything to avoid SQL injection and becoming the next Little Bobby Tables.

Wapiti has no simple method that I can find to crawl for a list of URLs and stop at that to allow for selecting the list of URLs to scan. But it has an option to minimize crawling and import a list of additional URLs to scan so I used that option to get at the same result.

Gathering URLs was done with wget:

$ wget --spider -r http://developer.virtualbookcase.com 2>&1 | grep '^--' | egrep -v '\.(css|jpg|gif|png)' | awk '{ print $3}' > developer.virtualbookcase.com-urls.txt

After that I sorted the file with URLs and threw out a lot of them, making sure all the scripts with several variants of input were still tested.

With that list I start wapiti with some special options. It still needs a starting url at -u so I give it the root but I limit the crawling with the depth parameter -d 1 and the max files parameter --max-files-per-dir 50. Then I add the additional urls from the earlier scan with the -s parameter. It's a lot of tweaking but it does the trick.

$ wapiti -u http://developer.virtualbookcase.com/ -d 1 --max-files-per-dir 50 -s developer.virtualbookcase.com-urls.txt -o ~/wapiti/ -v 2

No vulnerabilities were found. I found one PHP warning which only triggered in the kind of corner case a web vulnerability scanner causes, or an attacker. So I fixed that corner case too.

I received a responsible disclosure report of a vulnerability in The Virtual Bookcase.

I will directly admit I haven't done a lot of maintenance on this site in the past few years but I want to keep my sites secure.

The report came via openbugbounty.org and has no details about the vulnerability, so I am not 100% sure where the reported vulnerability is. But based on the report text XSS (Cross Site Scripting) and a peek in the access-log looking for specific requests I found I made a beginner mistake in dealing with a search query: displaying it as-is within an HTML context. I immediately fixed that error in the site.

Now I wonder why it took so long for me to realize the error of my ways or for someone to notice it!

Checking the logs some more finds huge amounts of attempts at SQL injection, which is a vulnerability I am very aware of and where I put up standard defenses. But this is the first time a security researcher made me aware of the cross-site scripting vulnerability.

Update: I contacted the reporter about the vulnerability who responded quickly inquiring about the possible bounty for finding the bug. As this is a site that hasn't delivered any income in years the best I can do is a mention in the credits of the site or on a separate hall of fame.

Update: I also started a vulnerability scanner on the site myself, to find any other vulnerabilities I might have missed. This scanner is going through the development site at the moment. Like many other scanners it doesn't see by default how certain urls all map to the same PHP script.

I already committed a few minor updates to improve handling of corner cases in not set variables and other things popping up in the scan.

Update 2022-09-23: I realized the reporter has never responded with the actual bug information.

I received a message from amazon that The Virtual Bookcase no longer qualifies as an amazon.com associate. That was no big surprise as I haven't done a lot of maintenance on the site and haven't added a lot of content in the last years.

The only serious maintenance was for the migration to the new web server where php 7.0 is the standard version. I wish to some day migrate to perl but haven't found time yet.

So I removed all amazon affiliate links I could find. This also means I can't use the amazon.com API anymore.

After spending an evening fixing scripts on The Virtual Bookcase to make them run in PHP 7 and make them safer at the same time I came to the conclusion that I still don't like php.

My conclusion is that if I want to maintain sites I'd rather redo them in perl. I noticed any serious maintenance on the scripts of The Virtual Bookcase was 9 years ago (!). That was also when I had the habit of writing maintenance scripts in perl and web code in php. The upside is that a part of the page-generating code is already available in perl.

But a rewrite is a task for another day. For now the site works cleanly in PHP 7 (and 5) and I can go on to the next task for moving the homeserver.

I received a notification from the google webmaster program that chrome browser would start showing security warnings on http://www.virtualbookcase.com/ due to the search box there.

The simple solution: make the site correctly available via https and redirect to the https version. I found out I already started doing the first bit and therefore the conversion was easy. Now with encrypted connections: The Virtual Bookcase.

On Camp Wireless and The Virtual Bookcase I see the following pattern in the access logs:

2620:108:700f::36bc:aade - - [27/Mar/2015:13:27:11 +0100] "GET / HTTP/1.1" 302 298 "-" "curl/7.36.0"
2406:da00:ff00::36e2:d963 - - [27/Mar/2015:13:27:38 +0100] "GET / HTTP/1.1" 302 298 "-" "curl/7.36.0"

Constant requests, 2 or 3 per minute from Amazon EC2 IPv6 addresses just requesting the / using curl. Over the day I now see 1334 unique addresses with at most 5 requests from one url.

The same pattern as described in Stange stream of HTTP GET requests in apache logs, from amazon ec2 instances - Server Fault with no real answer to the why.

It's not a problematic amount of traffic, I'd just like to understand what is happenning!

Time to count IPv6 visitor percentage to different websites again:

Site	July 2009	July 2010	July 2011	July 2012	July 2014
http://idefix.net/ my homepage	1%	2%	2%	3%	4%
http://netwerk.pcgg.nl/ hcc!pcgg netwerkgroep	2%	2%	2%	3%	1%
http://weather.idefix.net/ weather maps	< 1%	5%	6%	7%	6%
http://bbs.idefix.net/ BBS files		1%	1%	1%	3%
http://webcam.idefix.net/ the webcam	< 1%	1%	< 1%	2%	2%
http://www.virtualbookcase.com/ The Virtual Bookcase	< 1%	1%	1%	4%	87%
http://www.camp-wireless.org/ Camp Wireless	< 1%	1%	1%	3%	70%
http://weatherstation.idefix.net/ Weather station Utrecht Overvecht					1%

Interesting numbers. Results for The Virtual Bookcase and Camp Wireless are totally skewed thanks to some IPv6 bot constantly checking the site from constantly changing IPv6 addresses .. but without privacy extensions enabled.

Method: unique IPv6 addresses seen in the whole month / total unique addresses seen in the whole month.

Time to count IPv6 visitor percentage to different websites again:

Site	July 2009	July 2010	July 2011	July 2012
http://idefix.net/ my homepage	1%	2%	2%	3%
http://weather.idefix.net/ weather maps	< 1%	5%	6%	7%
http://netwerk.pcgg.nl/ hcc!pcgg netwerkgroep	2%	2%	2%	3%
http://bbs.idefix.net/ BBS files		1%	1%	1%
http://webcam.idefix.net/ the webcam	< 1%	1%	< 1%	2%
http://www.virtualbookcase.com/ The Virtual Bookcase	< 1%	1%	1%	4%
http://www.camp-wireless.org/ Camp Wireless	< 1%	1%	1%	3%

Interesting numbers. Websites with a more 'general' audience are now also growing this year.

Method: unique IPv6 addresses seen in the whole month / total unique addresses seen in the whole month.

After the updates to virtualbookcase.com last weekend I noticed one other area of serious bit rot: character set damage. The pages were still charset=US-ASCII and this went noticeably wrong in HTML For Dummies® by Ed Tittel and Steve James. But there are only a few places where I had to change that, and the databases are all UTF8 anyway.

Funny is how the amount of visitors is slowly growing since this weekend. A few improvements and a few new links from my homepage, and suddenly more visitors.