News items for tag statistics - Koos van den Hout

2018-04-26 More radio maps and their limits 1 month ago
The PE4KH website has maps of the locations where I contacted radio amateurs all over the world. The maps with generated images are created by exporting my locators worked/confirmed from cqrlog and using gcmwin for linux with a whole set of different configurations to plot the results.

But now the 2 meter band has been added and on that band a 'record distance' is not as far as on HF. On HF my current distance record was a contact with Australia at 16581 kilometers. My current distance record on the 2 meter band is 363 kilometer in Germany. Quite a different scale!

So the maps part has been enhanced with a 2 meter contacts map, but gcmwin can't use 6-position maidenhead locators so the map is quite coarse compared to what I want. I don't know the solution at the moment to improve this. The recent qso map PE4KH does show the more precise gridsquares when available in the log, so maybe that page needs distances added.
Read the rest of More radio maps and their limits

Tags: , ,
2018-03-01 An interesting bug in age-old Perl Net::SNMP code 3 months ago
I recently noticed the network traffic statistics weren't updated correctly for the LAN interface of my Draytek Vigor 130 modem. These statistics were extracted using code that I originally started using at the computer science systems group somewhere in the previous decade. It's all Perl Net::SNMP and not very efficient. I don't know if I wrote it myself or copied from somewhere else, I do know a new bug was introduced.

To understand the code it is important to realize that interface index numbers in SNMP are dynamic. Across a reboot a certain number can change. Interface names are static, but those are never used directly in SNMP.

So to get from a static interface name to a dynamic interface index the interfaces.2.1.2 subtree (ifDescr) has to be fetched from the device and checked for the right names. To get the interface index from an snmp object identifier I used to use this bit of code:
# find the current interface indices for the wanted ^ interfaces
foreach my $oid (oid_lex_sort(keys(%table))) {
    if (oid_base_match($ifTable_ifDesc,$oid)){
#        printf("%s => %s\n", $oid, $table{$oid});
        if (defined $wantstuff{$table{$oid}}){
            # I am lazy. I fill a hash with the interface indices so I can
            # use it for lookups
        #    printf "Found ifindex %d for %s\n",$wantstuff{$table{$oid}}{ifindex},$table{$oid};
But note how the current ifDesc subtree is from the modem:
IF-MIB::ifDescr.1 = STRING: LAN
IF-MIB::ifDescr.4 = STRING: VDSL
IF-MIB::ifDescr.5 = STRING: Resrved
IF-MIB::ifDescr.6 = STRING: 
IF-MIB::ifDescr.7 = STRING: 
IF-MIB::ifDescr.8 = STRING: 
IF-MIB::ifDescr.20.101.1 = STRING: WAN1
IF-MIB::ifDescr.21.101.1 = STRING: WAN2
IF-MIB::ifDescr.22.101.1 = STRING: LAN_PORT1
Using that rindex function there are 4 instances of index 1. Which caused the very similar code looking for the ifInOctets, ifOutOctets and other counters to overwrite the result for index 1 with those from WAN1, WAN2 and LAN_PORT1.

So that code is now improved, no more rindex but a well-defined use of length:
# find the current interface indices for the wanted ^ interfaces
foreach my $oid (oid_lex_sort(keys(%table))) {
    if (oid_base_match($ifTable_ifDesc,$oid)){
        #printf("%s => %s\n", $oid, $table{$oid});
        if (defined $wantstuff{$table{$oid}}){
                        my $intindex=substr($oid,length($ifTable_ifDesc)+1);
                        #printf "Submatch found ifindex %d for %s\n",$intindex,$table{$oid};
            # I am lazy. I fill a hash with the interface indices so I can
            # use it for lookups
            #printf "Found ifindex %d for %s\n",$wantstuff{$table{$oid}}{ifindex},$table{$oid};

Tags: , ,
2018-02-11 Plotting the number of amateur radio contacts 4 months ago
After the end of January I decided to plot the number of contacts again. QSL count plot up to January 2018 January is a busy month with two contests for me but I did not make a lot of contacts outside of those contests this year. I added contacts from holidays and the PE4KH/P activities to the total count.

Some more work on the plot script, I think bars look better than a line graph. But you could spend hours in gnuplot making the plot just right...

The new script:
set output "qslcount.png"
set terminal png size 640,300 fontscale 0.7
set timefmt "%Y-%m"
set xlabel "Month"
set ylabel "Number of contacts"
set xdata time
set style fill solid
set xtics format "%b %Y"
set xtics rotate
set grid
set boxwidth 0.75 relative
set autoscale xfixmin
set autoscale xfixmax
plot "dataset-qsocount" using 1:2 title "Contacts/Month" with boxes
Update: And indeed the change in x autoscale was one bit more 'just right'. The first graph was in February 2017: Rising number of amateur radio contacts.

Tags: , , ,
2017-10-10 Plotting the number of contacts (again) 8 months ago
After working through the results of my participation in the Russian worldwide digimode contest 2017 I decided to run a graph again of contacts per month as I did in Februari 2017. And remember how I made those graphs this time and save it in a plot script. qsl count plot

And the plotscript:
set output "qslcount.png"
set terminal png size 440,300 fontscale 0.7
set timefmt "%Y-%m"
set xlabel "Month"
set ylabel "Number of contacts"
set xdata time
set style data lines
set xtics format "%b %Y" 
set xtics rotate
plot "dataset-qsocount" using 1:2 title "Contacts/Month"
The interesting peak in January 2017 is still visible, it was caused by two contests I participated in: the ARRL RTTY roundup 2017 and the UBA PSK63 prefix contest 2017.

Tags: , ,
2017-09-08 VDSL tijdelijk met ADSL2 snelheid 9 months ago
Ik had een hik vandaag met de DSL verbinding en toen was ineens de snelheid 15 mbit downstream en 1.2 mbit upstream in plaats van 103 mbit downstream en 27 mbit upstream wat ik meestal gewend ben. Het leken wel ADSL2 snelheden. Ook met maar 512 carriers van de 4096 in gebruik. Toch was het DSL profiel nog wel 17a, wat een VDSL profiel is.

Na een tweede hik was de snelheid weer gewoon wat ik gewend was met vectored VDSL.

Update 2017-09-11: De snelheid blijft toch inzakken. De verbinding is nog stabiel maar ik verwacht eigenlijk problemen in de komende tijd. Het regent veel de laatste dagen dus het kan zijn dat er ergens water in de telefoonkabels terecht komt.

Update 2017-09-12: VDSL downstream week 20170912 En na een nieuwe hik is de snelheid weer terug. Het blijft magie, maar omdat het radiofrequenties zijn verwacht ik dit soort magie.

Tags: , ,
2017-07-24 Getting a lot more entropy from the Raspberry Pi 11 months ago
Entropy graph of a Raspberry Pi On doing some research on randomness in Linux I found out about the rng-tools package which includes rngd which can get randomness from hardware random generators to linux /dev/random.

On the main homeserver greenblatt there was no hardware randomness source available, I already use randomsound to generate randomness from audio noise. I found out the Raspberry Pi has a hardware randomness source so I installed rng-tools and rngd was able to use it. The impact on the measured available entropy is quite visible.

Tags: , ,
2017-03-27 Meten en grafieken maken van Powerline Ethernet doorvoer 1 year ago
Uit de archieven omdat ik de server waar dit ooit op draaide wil herinstalleren voor andere projecten.

Bij de HCC PCgg netwerkgroep hadden we ooit een netwerk gebaseerd op PLC (powerline communicatie) waarbij ethernet over stroomdraden gaat. Daar heb ik ook scripts voor gebouwd om statistieken te verzamelen over de doorvoer. Op de locatie in Maarn was dat voor de verbinding naar buiten niet zo briljant, zie de rapporten in Bijeenkomst 20 maart 2010 - hcc!pc netwerkgroep.

Maar voordat ik de scripts weggooi die deze gegevens verzamelden en er grafiekjes van maakten nog even voor de eeuwigheid. Het testdlan script gaat uit van output van dlanlist als:
koos@metcalfe:~$ dlanlist eth0
Type    MAC address        Mbps TX/RX       Version/Product
local   00:0B:3B:5F:95:AB  ---.-- / ---.--  INT6000-MAC-3-3-3348-00-2764-20080808-FINAL-B devolo dLAN 200 AVplus [MT2165]
remote  00:0B:3B:6F:AE:90   73.50 / 112.88  INT6000-MAC-3-3-3348-00-2764-20080808-FINAL-B

  • testdlan vraag dlanlist van devolo voor de huidige snelheid van devolo interfaces op netwerkkaart(en)
  • graphdlan maak grafieken per dag
De opzet van de scripts was om altijd te draaien als de machine actief was en achteraf overzichten te hebben per dag activiteit (meestal zaterdagen van 10:00 tot 17:00).

Tags: , , ,
2017-02-12 Rising number of amateur radio contacts 1 year ago
I noticed recently the number of radio contacts made by my new callsign PE4KH which I started using in March 2016 was getting close to the number of radio contacts made by my previous callsign PD4KH between March 2013 and March 2016. A typical rise in contacts, mostly due to my skills improving and participating in contests. So I wanted to view the rise per month and did some searching how to ask the cqrlog databases and plot the results.

Oh, and now PE4KH has more contacts after a few new contacts logged in PSK31 mode on the 20 meter band today.
Read the rest of Rising number of amateur radio contacts

Tags: , , ,
2016-11-03 De gasmeter van de slimme meter ging over op wintertijd 1 year ago
Ik ontdekte dat de gasmeter niet meer uitgelezen werd in de scripts die de slimme meter bulletins verwerken. Bij het napluizen van de Dutch smart meter standard v4.0 P1 protocol bleek dat het bericht met de gasmeterstand (0-1:24.2.1) bij de 'Capture Time' een timestamp heeft met S voor summertime en W voor wintertime.

Het script is nu aangepast om dit goed te verwerken en nu zie ik weer het gasgebruik.

Tags: , ,
2016-09-04 De slimme meter uitlezen 1 year ago
Slimme meter uitlezen De slimme meter die meer dan een maand geleden ons huis in kwam wordt nu constant uitgelezen en de resulterende gegevens worden opgeslagen in rrdtool databases van het electriciteits- en gas gebruik.

Uitlezen van een seriele poort vanuit Perl bleek niet makkelijk stabiel te krijgen te zijn. Uiteindelijk heb ik maar een oplossing gekozen/geleend van iemand anders: een stuk python wat cu aanroept: P1/ at master · sanderjo/P1. Dit script zou nog iets robuuster zijn als het de CRC controleert, maar dat is dan een wens voor een toekomstige versie.

Vervolgens sla ik de output van dit script op in /var/run/telemetry/smartmeteroutput. De keuze voor /var/run is omdat ik niet elke vijf minuten op de SD kaart van de raspberry wil schrijven. Ik heb dus ook /etc/rc.local aangepast om een /var/run/telemetry met als eigenaar user telemetry te maken, er worden daar meer meetgegevens neergezet voor verdere verwerking in statistieken. Het telemetry concept wat ik gebruik is het verzamelen in ASCII leesbare vorm van meetgegevens op systemen waarna deze opgehaald kunnen worden door een verzamelaar die ze gaat importeren in rrdtool databases (of andere verzamelingen). Omdat de verbinding tussen het te monitoren systeem en de verzamelaar als onbetrouwbaar gezien wordt (sommige systemen waarvan ik meetgegevens verzamel zitten achter een hikkende wifi verbinding) worden de meetgegevens lokaal opgeslagen met een tijdsaanduiding in de bestandsnaam en worden ze in rrdtool geimporteerd met deze tijdsaanduiding.

Het script wat de meetgegevens van de slimme meter verwerkt moet er tegen kunnen dat er soms velden ontbreken. Ik heb er voor gekozen uiteindelijk de dag- en nacht tellerstanden afgenomen en teruggeleverd (meetwaardes 1.8.1, 1.8.2, 2.8.1 en 2.8.2 in de Dutch smart meter standard) als 'verplicht' te tellen en de overige meetwaardes als 'optioneel'.

Uiteindelijk komen er dan mooie grafieken uit. De conclusie die ik eerder trok dat uit energie meetwaarden per kwartier prima af te leiden is wat bewoners doen is nog steeds valide. Uit grafieken over langere termijn is ook keurig af te zien wanneer we op vakantie waren.

Tags: , , , ,
2016-06-23 A serious thunderstorm somewhat counted 1 year ago
Lightning strikes 20160623 Last night a serious lightning storm passed and it got counted, but clearly with the same problems in counting as seen before in counting thunderstorms from the shed while radio activity causes a lot higher counts. Looking at the graphs for thunderstorms counted from the attic before I was active on HF radio there is quite a difference in numbers.

I think I want the lightning strikes counter back up in the attic but with a low-pass filter somehow to filter out false counts from amateur radio traffic.

Tags: , , ,
2015-09-12 Electrisch energieverbruik en gewoontes meten 2 years ago
In mijn 1-wire projecten was ook nog het plan om de watt-uur pulsen van de electricteitsmeter te gaan tellen. Met een schakeling met lichtgevoelige leds en een teller is het me nooit gelukt de pulsen betrouwbaar te tellen.

Ik ontdekte de YouLess die het tellen van zo'n ledje of van een draaischijfmeter terdege opgelost heeft en die niet begint met de data naar een externe dienst te sturen voor ik er naar kan kijken. Er is gewoon een simpele interface waarmee de tellingen van de YouLess via het netwerk uit te lezen zijn en die gebruik ik natuurlijk en maak er interne statistieken van.

YouLess stroomgebruik per kwartier
Gemeten electriciteitsgebruik per kwartier
Van die statistieken maak ik ook een grafiek met een nauwkeurigheid per kwartier. Die keuze is omdat de 'slimme meter' per kwartier uitgelezen kan worden en ik wil wel eens weten wat je er dan uit kan halen. Heel veel dus! Uit de kwartierwaarden is een hoop te halen over onze dagelijkse gewoonten, zeker als je ze over een lange termijn beschikbaar hebt. Voor de meeste pieken kan ik een verklaring bedenken en als eenmaal bekend is welke actie welke hoeveelheid energie gebruikt kan je daarna andere pieken ook weer uitleggen. Opvallend is dat zelfs bij de resolutie van 15 minuten een kort maar duidelijk gebruik zoals de waterkoker of een lange termijn wijziging zoals het licht aan of uit schakelen prima te zien is.
Read the rest of Electrisch energieverbruik en gewoontes meten

Tags: , , ,
2015-08-31 A serious thunderstorm counted 2 years ago
Lightning strikes 2015-08-30 We finally had a serious thunderstorm last night, making us almost jump from the bed at one moment. And the lighting strike counter saw that thunderstorm nicely.

It's quite clear the counts are lower now the sensor is closer to the ground. I'm not sure what to do to get a "good" count again. Last week I placed the 10/20/40m endfed antenna outside for part of the day and noticed that setup also influences the lightning detector in the shed. Raising the end of the antenna helped a bit, I noticed I can use the back gate of our house as the low end of the antenna.

Tags: , ,
2015-07-05 Moving the lightning detector to the shed and testing 2 years ago
A year in lightning strikes and activity on 20M PSK31
The last year in counted lightning strikes, showing clearly that I got active on 20meter psk31 in October 2014. The 'blips' before that are real thunderstorms.
I thought of this ages ago: Moving the lightning strike detector to the shed but only today got around to it because we had some serious chances of a thunderstorm earlier today which showed up on the lightning strike detector but the graph was completely screwed up again after I tried some psk31 digital mode transmissions on the 20 meter amateur band (14.000 to 14.250 MHz for me).

So now it is in the shed. Moving it to a lower position does mean I will not get readings for thunderstorms as far away as I used to but I'd rather have usable readings at this moment. First tests with transmitting psk31/psk63 on 20 the meter amateur band after I changed it look like it doesn't count the transmissions anymore. Now to wait for the next good thunderstorm to see how that gets counted. Some blips are showing up.

Update 2015-07-14: The first result seems to be that using the lights in the shed (tubelights with starters) shows up clearly. Using the radio still has no effect. I now await the first thunderstorm for more results.

Update 2015-07-28: No thunderstorm has been reported by the KNMI weather institute thunderstorm archive within a short distance of my sensor. I guess the maximum range is quite limited now.

Tags: , , ,
2015-06-16 SSL implementation on the fritzbox isn't secure enough 3 years ago
The latest OpenSSL updates cause me a new problem:
Connecting to (||:49443... connected.
OpenSSL: error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small
Unable to establish SSL connection.
Which means the script to fetch the dsl status from the fritzbox can't connect until I find out how to convince wget how to negotiate a non-standard cipher set. Or switch to curl.

Getting the right answers with curl isn't working out either. I can get the SSL working and do a POST to the right URL but the 'best' thing I get back is:
<errorDescription>XML error</errorDescription>

Update: The solution was to keep using wget but disable(!) SSL, using the non-SSL port for upnp. The command now is:
wget --user=$FRITZUSER --password=$FRITZPASS --post-file=linkstatusrequest.xml \
--header="Content-Type: text/xml" \
--header="SOAPAction: \"urn:dslforum-org:service:WANCommonInterfaceConfig:1#GetCommonLinkProperties\"" \ -O linkstatusanswer.xml
VDSL downstream speed 20150618 And now the data is available again and the graph is updated.

So the recent upgrade in OpenSSL which disabled less secure Diffie-Hellman key negotiation results in having to disable all encryption on the connection with the fritzbox. A security update on the fritzbox may solve this.

Tags: , , , ,
2015-03-27 Overly interested Amazon EC2 nodes 3 years ago
On Camp Wireless and The Virtual Bookcase I see the following pattern in the access logs:
2620:108:700f::36bc:aade - - [27/Mar/2015:13:27:11 +0100] "GET / HTTP/1.1" 302 298 "-" "curl/7.36.0"
2406:da00:ff00::36e2:d963 - - [27/Mar/2015:13:27:38 +0100] "GET / HTTP/1.1" 302 298 "-" "curl/7.36.0"
Constant requests, 2 or 3 per minute from Amazon EC2 IPv6 addresses just requesting the / using curl. Over the day I now see 1334 unique addresses with at most 5 requests from one url.

The same pattern as described in Stange stream of HTTP GET requests in apache logs, from amazon ec2 instances - Server Fault with no real answer to the why.

It's not a problematic amount of traffic, I'd just like to understand what is happenning!

Tags: , , , , ,
2014-11-28 Moving the lightning strike detector to the shed 3 years ago
I have noticed the lightning strike detection in Weather station Utrecht Overvecht goes completely mad when I transmit on the 20 meter amateur band. With the detector being quite close to the antenna I can understand this.

The solution will be to find a place to mount the detector in the shed. It will be lower (less reception of the radio energy of the strikes) but it will also be further away from my interference.

That also means the reading of the detector will have to be done using w1retap since that is what I use on the shed weatherstation computer. I was a bit confused whether w1retap supports this counter but I found out it's based on the DS2423 counter chip which is supported in w1retap, as part of a wind speed meter in a TAI8515 weather station, but w1retap will give the count on readout and the conversion is up to the user.

Tags: , , , ,
2014-11-26 Getting the DSL linespeed from the Fritz!Box 7360 3 years ago
The fact I couldn't get the DSL linespeed from the Fritz!Box 7360 annoyed me a lot, especially since there is a new telephone wiring cabinet which should raise VDSL2 speeds. I went through a number of websites about getting data out of the Fritz!Box with upnp, and finally I made it work and I get the results I want:
The hint that worked for me was at MRTG en Fritz!Box 7360 (firmware 124.06.05) - tweakers (in Dutch) where it mentiones the changeover to TR-064 protocol which should be reachable over the url which will ask for authentication with the root username and the Fritz!Box password. More about the Fritz!Box TR-064 implementation at Schnittstellen für Entwickler | AVM Deutschland (in German) which has more documentation at AVM TR-064 – First Steps (pdf, English). This made me end up at doing a SOAP request (post) to which failed. All SOAP requests fail with an HTTP error code 500, but there is a separate SOAP error set in the HTTP status 500 body. I used tcpdump to look at the SOAP error body and found:
<errorCode>504</errorCode> <errorDescription>SSL needed</errorDescription></UPnPError>
The SSL port is (according to the TR-064 first steps document above) 49443 and the URL is over SSL: and this works, giving the answers I want.
Read the rest of Getting the DSL linespeed from the Fritz!Box 7360

Tags: , ,
2014-05-25 (#) 4 years ago
After testing the gps sky view it's now time to test with ntpd. First step was to recompile ntpd because the debian default package had no pps support. Recompiling on a 500 MHz AMD Geode takes a bit of time.

Results look ok for a first test:
root@ritchie:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
+greenblatt.idef    2 u   17   64  377    1.177  101.732  63.403
*metronoom.dmz.c .PPS.            1 u   17   64  377   19.499  100.512   6.722    2 u   11   64  377   18.403  104.008   3.669
oGPS_NMEA(0)     .GPS.            0 l    6    8  377    0.000  114.073   7.364
root@ritchie:~# ntpdc -c loopi
offset:               0.001986 s
frequency:            94.434 ppm
poll adjust:          -30
watchdog timer:       342 s
root@ritchie:~# ntpdc -c kerni
pll offset:           3.3e-08 s
pll frequency:        94.434 ppm
maximum error:        0.175258 s
estimated error:      2e-06 s
status:               2007  pll ppsfreq ppstime nano
pll time constant:    3
precision:            1e-09 s
frequency tolerance:  500 ppm
root@ritchie:~# ntpdc -c sysi
system peer:          GPS_NMEA(0)
system peer mode:     client
leap indicator:       00
stratum:              1
precision:            -19
root distance:        0.00000 s
root dispersion:      0.00749 s
reference ID:         [GPS]
reference time:       d72cb010.dc91595e  Sun, May 25 2014 20:08:16.861
system flags:         auth monitor ntp kernel stats pps 
jitter:               0.006714 s
stability:            0.000 ppm
broadcastdelay:       0.000000 s
authdelay:            0.000000 s
It will need some more calibration probably.

Update: It keeps looking nice after some calibration. Stats gathered at NTP server stats.

This does mean one of the old project sundial goals has been met: the weather station computer in the shed is now also a time server.

Tags: , , ,
2014-05-24 (#) 4 years ago
I was able to buy a real Garmin GPS 18 LVC secondhand. It's now on the roof of our shed. The first thing I want to do is repeat my plotting of GPS satellite positions from $GPGSV messages and plotting of GPS satellite positions and signal strengths from $GPGSV messages measurements with data from this unit. After that has run for a while I'll configure ntpd to get the correct time from the GPS unit and the PPS signal.

And again, the resulting plot of gps satellite positions versus signal strength is not very helpful in finding out which part of the sky is obscured.

Tags: , ,
  Older news items for tag statistics ⇒
, reachable as PGP encrypted e-mail preferred.

PGP key 5BA9 368B E6F3 34E4 local copy PGP key 5BA9 368B E6F3 34E4 via keyservers pgp key statistics for 0x5BA9368BE6F334E4 Koos van den Hout
Other webprojects: Camp Wireless, wireless Internet access at campsites, The Virtual Bookcase, book reviews