On Camp Wireless and The Virtual Bookcase I see the following pattern in the access logs:2620:108:700f::36bc:aade - - [27/Mar/2015:13:27:11 +0100] "GET / HTTP/1.1" 302 298 "-" "curl/7.36.0" 2406:da00:ff00::36e2:d963 - - [27/Mar/2015:13:27:38 +0100] "GET / HTTP/1.1" 302 298 "-" "curl/7.36.0"Constant requests, 2 or 3 per minute from Amazon EC2 IPv6 addresses just requesting the / using curl. Over the day I now see 1334 unique addresses with at most 5 requests from one url. The same pattern as described in Stange stream of HTTP GET requests in apache logs, from amazon ec2 instances - Server Fault with no real answer to the why. It's not a problematic amount of traffic, I'd just like to understand what is happenning!
Time to count IPv6 visitor percentage to different websites again:
Interesting numbers. Results for The Virtual Bookcase and Camp Wireless are totally skewed thanks to some IPv6 bot constantly checking the site from constantly changing IPv6 addresses .. but without privacy extensions enabled. Method: unique IPv6 addresses seen in the whole month / total unique addresses seen in the whole month.
Site July 2009 July 2010 July 2011 July 2012 July 2014 http://idefix.net/ my homepage 1% 2% 2% 3% 4% http://netwerk.pcgg.nl/ hcc!pcgg netwerkgroep 2% 2% 2% 3% 1% http://weather.idefix.net/ weather maps < 1% 5% 6% 7% 6% http://bbs.idefix.net/ BBS files 1% 1% 1% 3% http://webcam.idefix.net/ the webcam < 1% 1% < 1% 2% 2% http://www.virtualbookcase.com/ The Virtual Bookcase < 1% 1% 1% 4% 87% http://www.camp-wireless.org/ Camp Wireless < 1% 1% 1% 3% 70% http://weatherstation.idefix.net/ Weather station Utrecht Overvecht 1%
Time to count IPv6 visitor percentage to different websites again:
Interesting numbers. Websites with a more 'general' audience are now also growing this year. Method: unique IPv6 addresses seen in the whole month / total unique addresses seen in the whole month.
Site July 2009 July 2010 July 2011 July 2012 http://idefix.net/ my homepage 1% 2% 2% 3% http://weather.idefix.net/ weather maps < 1% 5% 6% 7% http://netwerk.pcgg.nl/ hcc!pcgg netwerkgroep 2% 2% 2% 3% http://bbs.idefix.net/ BBS files 1% 1% 1% http://webcam.idefix.net/ the webcam < 1% 1% < 1% 2% http://www.virtualbookcase.com/ The Virtual Bookcase < 1% 1% 1% 4% http://www.camp-wireless.org/ Camp Wireless < 1% 1% 1% 3%
After the updates to virtualbookcase.com last weekend I noticed one other area of serious bit rot: character set damage. The pages were still charset=US-ASCII and this went noticeably wrong in HTML For Dummies® by Ed Tittel and Steve James. But there are only a few places where I had to change that, and the databases are all UTF8 anyway. Funny is how the amount of visitors is slowly growing since this weekend. A few improvements and a few new links from my homepage, and suddenly more visitors.
I got around to posting reviews for books I read recently:
But I noticed quite some bit rot on The Virtual Bookcase. Some scripts needed updating before I could add a book or post a review. And the reviews queue had been filled with lots of spam.
- The Best of 2600: A Hacker Odyssey by Emmanuel Goldstein
- Wiring Up The Big Brother Machine...And Fighting It by Mark Klein
- Crypto : How the Code Rebels Beat the Government -- Saving Privacy in the Digital Age by Steven Levy
Big surprise yesterday: UPS logistics is able to do a 'deliver to neighbour' on a residential address. Since we have nice neighbours in the street a package was delivered there and we found the note to pick it up. That is quite an improvement over having to go to some warehouse in a far away industrial area only reachable by car.
In the package: two Amazon Kindles. Yes, we're going e-book! The main reason is quite simple: on cycling holidays books are a serious part of the weight we drag along. Having the entire library as a lightweight (240 gram) e-reader is nicer.
One effect is that I think I want to add something with editions / versions of books to detail pages about books on virtualbookcase so it is visible when a book is available in a kindle version. This has already been on my mind for UK/US versions (which happened for books like Harry Potter and the Half-Blood Prince (UK version) / Harry Potter and the Half-Blood Prince (US version). Ideally, I'd like to link all versions of the same book so I can show all reviews together. And as an added bonus I could show availability of hardback / paperback / kindle versions at places like amazon. But that will take some serious programming time...
I recently re-read The Fugitive Game: Online with Kevin Mitnick and noticed I never wrote a review for it. So I fixed that omission. It's a great book to read as it reads as a big adventure. But it took me a while to write a good review.
Redoing the IPv6 percentages visitors websites for November 2010:
- http://pictures.idefix.net/ pictures : 7%
- http://weather.idefix.net/ weather maps : 4%
- http://idefix.net/ my homepage : 4%
- http://netwerk.pcgg.nl/ hcc!pcgg netwerkgroep : 2%
- http://bbs.idefix.net/ BBS files : <1%
- http://webcam.idefix.net/ the webcam : <1%
- http://www.virtualbookcase.com/ The Virtual Bookcase : <1%
- http://www.camp-wireless.org/ Camp Wireless : <1%
There is an interesting relation between 'website with technical subject' and 'visitors with IPv6'. I counted the number of unique addresses of visitors via IPv6 to several websites I run for the month of July 2010 and found the following percentages:
Comparing it to July 2009 (percentages between parentheses) does show growth.
- http://weather.idefix.net/ weather maps : 5% (1%)
- http://pictures.idefix.net/ pictures : 4% (2%)
- http://idefix.net/ my homepage : 2% (1%)
- http://netwerk.pcgg.nl/ hcc!pcgg netwerkgroep : 2% (2%)
- http://bbs.idefix.net/ BBS files : 1%
- http://webcam.idefix.net/ the webcam : <1%
- http://www.virtualbookcase.com/ The Virtual Bookcase : < 1%
- http://www.camp-wireless.org/ Camp Wireless : < 1%
Time for some website programming: Amazon was notifying me that Amazon 'Product Advertising API' requests (formerly known as Amazon Web Service) now need to be digitally signed. The 'why' of this is probably something with security. For the 'how' I had to revive those braincells which once programmed Amazon Web Service into The Virtual Bookcase. Those braincells took time as I was first browsing in the wrong sourcefiles. But, after having a peek at Jaap's free Amazon PHP Scripts and incorporating the change to sign requests into my sources, I found the right way again.
One of those times I am glad I have a development version of the site (smaller database, code in development), a qa version (production database, candidate code) and a running version (production database, production code) to test changes like this. The visitors of The Virtual Bookcase never saw all the tries in getting the code to work right.It was a while since I did any serious work on that site. Otherwise it runs on autopilot and I only add new books and reviews from time to time. Ok, the income from the site has dropped in the same way.
Computers are better at... doing what you program to, not what you want. There is still an older computer running which once hosted my mailman mailinglists for several domains. I shut down mailman on that machine and migrated the lists. But mailman was never removed from the startup files and the machine rebooted today, resulting in three years of mailman monthly reminders mailed at once. So if you were looking why you got old mailman reminders from virtualbookcase, this is why.
The Virtual Bookcase is back online too, and mail is flowing again for all the domains. Lots of typing, checking and everything to move the stuff to the home server. But, finished (I think).
Update: and all the web statistics are working again and updated. Finished?
And it is back! Idefix 4 broke in a major way: the power supply let out the magic smoke in a big way: the hosting company called me to let me know the server was smelling funny and did not want to start up at all. Since the end of idefix 4 in a rack was near anyway the decision was made to move the server home. There I used another power supply to get access to my data again. The old powersupply was a 300 Watt powersupply which seems to be way underrated for a dual Xeon system. My best guess is that the instability the system had came from the powersupply anyway. So, time to move more domains home. Content from idefix.net is now here at home and virtualbookcase will be next when I find time. I had started migrating Camp Wireless so I finished that migration fast. Mail is diverted to a different place so I have a bit of time to configure all the mailing lists and other things.
Did some work on The Virtual Bookcase. First of all Amazon notified me that amazon web services was going to disable version 3 of the API which I was still using (yes, a year after the first notification that it was going to be ended .. not that much time for virtualbookcase at the moment). So time to do some PHP programming and redo the stuff for version 4. I also noticed that the bot for the russian search site Yandex was causing high amounts of hits but would not even return a direct link to www.virtualbookcase.com when searching on virtual bookcase at yandex. So I wanted to disable Yandex in robots.txt but I could not easily find the right name to put in robots.txt because all the help at Yandex is also in Russian. Finally I found some hints at this page describing robots.txt in russian that the right User-Agent probably is Yandex. And the robots.txt for cisco.com agrees.
While checking the logs of The Virtual Bookcase to see if I made any errors in the pages, I noticed another webbot requesting loads of pages, the bot related to Webclipping.com. I looked at their site and noticed that they are no search engine, they just scour the web looking for people having opinions about their paying customers. Which means: those requests by their spider will never result in visitors to The Virtual Bookcase. I only want to allow webbots that result in visitors to the site. So, I wanted to know what to put in robots.txt to stop them.. couldn't find that on their website so I asked via e-mail. The answer is (paraphrased): "We decide on our own which sites to visit.". My answer is to block them.
I finally redid a lot of the page design on The Virtual Bookcase. All is done with stylesheets now (no more crummy html 3.2 with tables in tables for making borders) and I created a bunch of new graphics for the site. I learned new stuff about graphic design, photographing and working with The Gimp in the process. For example, the bookcase-image in the background of the header took me about an hour to set up, photograph and adjust to get the result I wanted. I do like the current result although I am pondering some more changes. Time also goes into testing whether a neat stylesheet or image trick also works in other popular browsers (not everybody uses Firefox).
I picked up working on The Virtual Bookcase again (after letting it run for a while with just reviewing new books that came in). Page by page gets converted to html4 (I even try to pass the w3 validator) and I'm also looking at a restyle of the pages (html4 with css makes this a lot easier). The changes to html4 get incorporated in the site right away, the big changes will happen in one cutover. It is nice to note how html4+css makes the html source that much cleaner.
Doing lots of little stuff at The Virtual Bookcase. Updating pages to be a little more google-friendly, debugging some code to make stuff work again (seems php isn't too stable as a language, which made stuff fail in interesting ways). And keeping up with the book news. And finding out the stylesheet wasn't giving the wanted overviews in MSIE. Oops there.
idefix.net now lives on a new server, somewhat faster and more reliable. Time to update the picture on that page! I also moved stuff like The Virtual Bookcase and Camp Wireless. Only thing that gave me troubles right before moving was GNU mailman.
I guess it's the time for international book fairs.. after adding news about the International Book Fair of Havana to The Virtual Bookcase last week, this week the news is about the 17th World Book Fair in New Delhi.