In all the web content that has to be migrated to a new environment I noticed the weather map site weather.idefix.net depends on a load of complex scripts to generate it and never got any amount of visitors. So I decided to stop that site. I'll archive all the scripts around it so I can pick it up again some day.
Time to count IPv6 visitor percentage to different websites again:
Interesting numbers. Websites with a more 'general' audience are now also growing this year. Method: unique IPv6 addresses seen in the whole month / total unique addresses seen in the whole month.
Site July 2009 July 2010 July 2011 July 2012 http://idefix.net/ my homepage 1% 2% 2% 3% http://weather.idefix.net/ weather maps < 1% 5% 6% 7% http://netwerk.pcgg.nl/ hcc!pcgg netwerkgroep 2% 2% 2% 3% http://bbs.idefix.net/ BBS files 1% 1% 1% http://webcam.idefix.net/ the webcam < 1% 1% < 1% 2% http://www.virtualbookcase.com/ The Virtual Bookcase < 1% 1% 1% 4% http://www.camp-wireless.org/ Camp Wireless < 1% 1% 1% 3%
Another improvement to the homepage: the weather projects get a bit more attention, with the latest weathermap for the Netherlands and the current weather readings in the right-hand sidebar. This costs about 0.02 second so it may need some optimization. In the optimization process I could even make the weather data publicly available in some portable format.
Since almost all my websites are now hosted at home behind a not-too-bright ADSL link with 1 megabit upstream I sometimes wonder about what I'll do when something I host gets slashdotted. I don't write that much which might suddenly be interesting but sometimes the intarwebs decide something else.
I do have a server at work with enough upstream bandwidth, but I will not use that server for anything which would remotely be interpretable as 'commercial'. So I could only use that server when it would involve for example the weather maps.Another solution for bandwith-hogging binaries would be to host them on my xs4all homepage which has enough space. But copying stuff over there would be a lot of work or take some serious scripting.
What could be interesting (in the case of images or other binary stuff causing the problem) is trying Coral Cache which can be done using some simple mod_rewrite tricks as noted in Saving Bandwidth and Preventing Hotlinking With Coral Cache. The note about not using Coral Cache when not being slashdotted is true: the demo image loads slower at the moment. The article is from August 2007.Anyway, this is all very very theory: even when a comment on an article on a very popular site linked to a file on the bbs archive it still didn't cause any real trouble.
Update : A small test does not give me a lot of confidence in Coral Cache:koos@kolham:~$ host idefix.net.nyud.net nyud.net has DNAME record http.l2.l1.l0.nyucd.net. idefix.net.nyud.net is an alias for idefix.net.http.l2.l1.l0.nyucd.net. idefix.net.http.l2.l1.l0.nyucd.net is an alias for http.l2.l1.l0.nyucd.net. http.l2.l1.l0.nyucd.net has address 22.214.171.124 http.l2.l1.l0.nyucd.net has address 126.96.36.199 http.l2.l1.l0.nyucd.net has address 188.8.131.52 Host http.l2.l1.l0.nyucd.net not found: 3(NXDOMAIN) koos@kolham:~$ host idefix.net.nyud.net Host idefix.net.nyud.net not found: 3(NXDOMAIN) koos@kolham:~$ host idefix.net.nyud.net Host idefix.net.nyud.net not found: 3(NXDOMAIN)Seems like their DNS and the local resolver dislike eachother.
Redoing the IPv6 percentages visitors websites for November 2010:
- http://pictures.idefix.net/ pictures : 7%
- http://weather.idefix.net/ weather maps : 4%
- http://idefix.net/ my homepage : 4%
- http://netwerk.pcgg.nl/ hcc!pcgg netwerkgroep : 2%
- http://bbs.idefix.net/ BBS files : <1%
- http://webcam.idefix.net/ the webcam : <1%
- http://www.virtualbookcase.com/ The Virtual Bookcase : <1%
- http://www.camp-wireless.org/ Camp Wireless : <1%
With all the interesting weather today on the Northsea I added the Northsea area at weather.idefix.net with fresh weather maps every hour.
Het waait stevig.. let ook op de windvlaggetjes op weather.idefix.net Nederland 2010-08-23 14:00 .. vooral de waarnemingen over zee hebben tot 9 Beaufort! Waarnemingsposten EHKV en EHQE.
Update: Op 15:00 nog meer harde wind, windkracht 10 op EHKV weather.idefix.net Nederland 2010-08-23 15:00.
EHKV = oil platform K14-fa-1c in the Northsea.
There is an interesting relation between 'website with technical subject' and 'visitors with IPv6'. I counted the number of unique addresses of visitors via IPv6 to several websites I run for the month of July 2010 and found the following percentages:
Comparing it to July 2009 (percentages between parentheses) does show growth.
- http://weather.idefix.net/ weather maps : 5% (1%)
- http://pictures.idefix.net/ pictures : 4% (2%)
- http://idefix.net/ my homepage : 2% (1%)
- http://netwerk.pcgg.nl/ hcc!pcgg netwerkgroep : 2% (2%)
- http://bbs.idefix.net/ BBS files : 1%
- http://webcam.idefix.net/ the webcam : <1%
- http://www.virtualbookcase.com/ The Virtual Bookcase : < 1%
- http://www.camp-wireless.org/ Camp Wireless : < 1%
I've done some work on the weather map site. I'm improving the plotting script to have a better abstraction of data and plotting so I can do plots for multiple countries.
As a first I'm plotting Denmark. The reasons are simple: it's a not too big country and has a reasonable number of weather stations.
I noticed some funny stuff on the maps plotted at the weather map site. Non-fitting temperatures like 28°C in the south of the Netherlands where it is -4°C to -8°C according to the people who know. Or 5°C at site 'EHDV' which was plotted somewhere in the eastern part of the country, near Marknesse, but the reading for Marknesse was not the same.
The first error was caused by the recent upgrade at home: Ubuntu comes with Geo::METAR 1.14. I did not notice this right away because they did fix the error about only accepting metar data from the USA. So I put in my own Geo::METAR which fixes the parsing. The other error was that EHDV was in the wrong location in my data. Searching using google found that EHDV is the code for oil platform D15-FA-1 which is also hard to find, but somewhere deep in the data history of air traffic control the Netherlands is the right location out on the North Sea. Where it is indeed a lot warmer than in the east of the country, thanks to less cold wind from the east and more warm wather from the Gulf stream.
I noticed that the entry for De Bilt, Netherlands at the weather.gladstonefamily.net site had recent weather data. I asked Philip Gladstone where he gets that data and searched around a bit myself. The answer is that this is synop (surface synoptic observations) data. The synop data format is ofcourse very different from the METAR format. Time to find good data sources, write another parser and see what this can do for http://weather.idefix.net/. Probably time to split the whole script that generates the weathermaps in several parts, fetching, parsing and caching data, drawing the resulting data on the map.
After a 'complaint' that temperatures were mapped to almost the same shades of green I tried to find a better way to map temperatures to colours for the weather maps on http://weather.idefix.net. What I learned sofar: visualizing temperatures in an easy to understand way is quite hard. The human perception is that blue is cold and red is hot. The current mapping uses that to map temperature to Hue values and use the resulting HSV colour (after conversion to RGB). Adding a legend showing what temperature maps to what colour helps a bit, but with the Netherlands being not too big an area for weather and with usually not much differences in temperature I keep ending up with close shades of the same colour. Interesting problem, and I can't find any published work about this sofar. Lots of research in 3d weather visualization, but no temperature to colour mapping information.
A few updates to the weather maps recently: The maps are now generated each hour and there is a list of the recent weather maps.
Updates to the logic behind the weather maps at http://weather.idefix.net/ and updates to the stations list. Data is now fetched from multiple sources and with a lot of searching I found the locations of a number of stations such as EHFS (Vlissingen), EHSC (Lichteiland Goeree) and EHKV (oilplatform K14-FA-1C). Starting tomorrow morning the updates will be visible in the generated maps.
The weather keeps me interested and I have been working on processing weather data (ofcourse using Geo::METAR). Roger Burton West did a lot of work on visualizing weather data and I updated the result for Geo::METAR. Resulting maps now available on http://weather.idefix.net/. Stuff to add: caching, getting data from several sources (at least one metar for the Netherlands isn't available via the US national weather service) or using different sources.