News items for tag statistics - Koos van den Hout

2015-09-12 Electrisch energieverbruik en gewoontes meten 3 years ago
In mijn 1-wire projecten was ook nog het plan om de watt-uur pulsen van de electricteitsmeter te gaan tellen. Met een schakeling met lichtgevoelige leds en een teller is het me nooit gelukt de pulsen betrouwbaar te tellen.

Ik ontdekte de YouLess die het tellen van zo'n ledje of van een draaischijfmeter terdege opgelost heeft en die niet begint met de data naar een externe dienst te sturen voor ik er naar kan kijken. Er is gewoon een simpele interface waarmee de tellingen van de YouLess via het netwerk uit te lezen zijn en die gebruik ik natuurlijk en maak er interne statistieken van.

YouLess stroomgebruik per kwartier
Gemeten electriciteitsgebruik per kwartier
Van die statistieken maak ik ook een grafiek met een nauwkeurigheid per kwartier. Die keuze is omdat de 'slimme meter' per kwartier uitgelezen kan worden en ik wil wel eens weten wat je er dan uit kan halen. Heel veel dus! Uit de kwartierwaarden is een hoop te halen over onze dagelijkse gewoonten, zeker als je ze over een lange termijn beschikbaar hebt. Voor de meeste pieken kan ik een verklaring bedenken en als eenmaal bekend is welke actie welke hoeveelheid energie gebruikt kan je daarna andere pieken ook weer uitleggen. Opvallend is dat zelfs bij de resolutie van 15 minuten een kort maar duidelijk gebruik zoals de waterkoker of een lange termijn wijziging zoals het licht aan of uit schakelen prima te zien is.
Read the rest of Electrisch energieverbruik en gewoontes meten

Tags: , , ,
2015-08-31 A serious thunderstorm counted 3 years ago
Lightning strikes 2015-08-30 We finally had a serious thunderstorm last night, making us almost jump from the bed at one moment. And the lighting strike counter saw that thunderstorm nicely.

It's quite clear the counts are lower now the sensor is closer to the ground. I'm not sure what to do to get a "good" count again. Last week I placed the 10/20/40m endfed antenna outside for part of the day and noticed that setup also influences the lightning detector in the shed. Raising the end of the antenna helped a bit, I noticed I can use the back gate of our house as the low end of the antenna.

Tags: , ,
2015-07-05 Moving the lightning detector to the shed and testing 3 years ago
A year in lightning strikes and activity on 20M PSK31
The last year in counted lightning strikes, showing clearly that I got active on 20meter psk31 in October 2014. The 'blips' before that are real thunderstorms.
I thought of this ages ago: Moving the lightning strike detector to the shed but only today got around to it because we had some serious chances of a thunderstorm earlier today which showed up on the lightning strike detector but the graph was completely screwed up again after I tried some psk31 digital mode transmissions on the 20 meter amateur band (14.000 to 14.250 MHz for me).

So now it is in the shed. Moving it to a lower position does mean I will not get readings for thunderstorms as far away as I used to but I'd rather have usable readings at this moment. First tests with transmitting psk31/psk63 on 20 the meter amateur band after I changed it look like it doesn't count the transmissions anymore. Now to wait for the next good thunderstorm to see how that gets counted. Some blips are showing up.

Update 2015-07-14: The first result seems to be that using the lights in the shed (tubelights with starters) shows up clearly. Using the radio still has no effect. I now await the first thunderstorm for more results.

Update 2015-07-28: No thunderstorm has been reported by the KNMI weather institute thunderstorm archive within a short distance of my sensor. I guess the maximum range is quite limited now.

Tags: , , ,
2015-06-16 SSL implementation on the fritzbox isn't secure enough 4 years ago
The latest OpenSSL updates cause me a new problem:
Connecting to (||:49443... connected.
OpenSSL: error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small
Unable to establish SSL connection.
Which means the script to fetch the dsl status from the fritzbox can't connect until I find out how to convince wget how to negotiate a non-standard cipher set. Or switch to curl.

Getting the right answers with curl isn't working out either. I can get the SSL working and do a POST to the right URL but the 'best' thing I get back is:
<errorDescription>XML error</errorDescription>

Update: The solution was to keep using wget but disable(!) SSL, using the non-SSL port for upnp. The command now is:
wget --user=$FRITZUSER --password=$FRITZPASS --post-file=linkstatusrequest.xml \
--header="Content-Type: text/xml" \
--header="SOAPAction: \"urn:dslforum-org:service:WANCommonInterfaceConfig:1#GetCommonLinkProperties\"" \ -O linkstatusanswer.xml
VDSL downstream speed 20150618 And now the data is available again and the graph is updated.

So the recent upgrade in OpenSSL which disabled less secure Diffie-Hellman key negotiation results in having to disable all encryption on the connection with the fritzbox. A security update on the fritzbox may solve this.

Tags: , , , ,
2015-03-27 Overly interested Amazon EC2 nodes 4 years ago
On Camp Wireless and The Virtual Bookcase I see the following pattern in the access logs:
2620:108:700f::36bc:aade - - [27/Mar/2015:13:27:11 +0100] "GET / HTTP/1.1" 302 298 "-" "curl/7.36.0"
2406:da00:ff00::36e2:d963 - - [27/Mar/2015:13:27:38 +0100] "GET / HTTP/1.1" 302 298 "-" "curl/7.36.0"
Constant requests, 2 or 3 per minute from Amazon EC2 IPv6 addresses just requesting the / using curl. Over the day I now see 1334 unique addresses with at most 5 requests from one url.

The same pattern as described in Stange stream of HTTP GET requests in apache logs, from amazon ec2 instances - Server Fault with no real answer to the why.

It's not a problematic amount of traffic, I'd just like to understand what is happenning!

Tags: , , , , ,
2014-11-28 Moving the lightning strike detector to the shed 4 years ago
I have noticed the lightning strike detection in Weather station Utrecht Overvecht goes completely mad when I transmit on the 20 meter amateur band. With the detector being quite close to the antenna I can understand this.

The solution will be to find a place to mount the detector in the shed. It will be lower (less reception of the radio energy of the strikes) but it will also be further away from my interference.

That also means the reading of the detector will have to be done using w1retap since that is what I use on the shed weatherstation computer. I was a bit confused whether w1retap supports this counter but I found out it's based on the DS2423 counter chip which is supported in w1retap, as part of a wind speed meter in a TAI8515 weather station, but w1retap will give the count on readout and the conversion is up to the user.

Tags: , , , ,
2014-11-26 Getting the DSL linespeed from the Fritz!Box 7360 4 years ago
The fact I couldn't get the DSL linespeed from the Fritz!Box 7360 annoyed me a lot, especially since there is a new telephone wiring cabinet which should raise VDSL2 speeds. I went through a number of websites about getting data out of the Fritz!Box with upnp, and finally I made it work and I get the results I want:
The hint that worked for me was at MRTG en Fritz!Box 7360 (firmware 124.06.05) - tweakers (in Dutch) where it mentiones the changeover to TR-064 protocol which should be reachable over the url which will ask for authentication with the root username and the Fritz!Box password. More about the Fritz!Box TR-064 implementation at Schnittstellen für Entwickler | AVM Deutschland (in German) which has more documentation at AVM TR-064 – First Steps (pdf, English). This made me end up at doing a SOAP request (post) to which failed. All SOAP requests fail with an HTTP error code 500, but there is a separate SOAP error set in the HTTP status 500 body. I used tcpdump to look at the SOAP error body and found:
<errorCode>504</errorCode> <errorDescription>SSL needed</errorDescription></UPnPError>
The SSL port is (according to the TR-064 first steps document above) 49443 and the URL is over SSL: and this works, giving the answers I want.
Read the rest of Getting the DSL linespeed from the Fritz!Box 7360

Tags: , ,
2014-05-25 (#) 5 years ago
After testing the gps sky view it's now time to test with ntpd. First step was to recompile ntpd because the debian default package had no pps support. Recompiling on a 500 MHz AMD Geode takes a bit of time.

Results look ok for a first test:
root@ritchie:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
+greenblatt.idef    2 u   17   64  377    1.177  101.732  63.403
*metronoom.dmz.c .PPS.            1 u   17   64  377   19.499  100.512   6.722    2 u   11   64  377   18.403  104.008   3.669
oGPS_NMEA(0)     .GPS.            0 l    6    8  377    0.000  114.073   7.364
root@ritchie:~# ntpdc -c loopi
offset:               0.001986 s
frequency:            94.434 ppm
poll adjust:          -30
watchdog timer:       342 s
root@ritchie:~# ntpdc -c kerni
pll offset:           3.3e-08 s
pll frequency:        94.434 ppm
maximum error:        0.175258 s
estimated error:      2e-06 s
status:               2007  pll ppsfreq ppstime nano
pll time constant:    3
precision:            1e-09 s
frequency tolerance:  500 ppm
root@ritchie:~# ntpdc -c sysi
system peer:          GPS_NMEA(0)
system peer mode:     client
leap indicator:       00
stratum:              1
precision:            -19
root distance:        0.00000 s
root dispersion:      0.00749 s
reference ID:         [GPS]
reference time:       d72cb010.dc91595e  Sun, May 25 2014 20:08:16.861
system flags:         auth monitor ntp kernel stats pps 
jitter:               0.006714 s
stability:            0.000 ppm
broadcastdelay:       0.000000 s
authdelay:            0.000000 s
It will need some more calibration probably.

Update: It keeps looking nice after some calibration. Stats gathered at NTP server stats.

This does mean one of the old project sundial goals has been met: the weather station computer in the shed is now also a time server.

Tags: , , ,
2014-05-24 (#) 5 years ago
I was able to buy a real Garmin GPS 18 LVC secondhand. It's now on the roof of our shed. The first thing I want to do is repeat my plotting of GPS satellite positions from $GPGSV messages and plotting of GPS satellite positions and signal strengths from $GPGSV messages measurements with data from this unit. After that has run for a while I'll configure ntpd to get the correct time from the GPS unit and the PPS signal.

And again, the resulting plot of gps satellite positions versus signal strength is not very helpful in finding out which part of the sky is obscured.

Tags: , ,
2014-04-18 (#) 5 years ago
A bit of searching later found the right incantation to make gnuplot adjust color based on a third value (signal level in my case). It isn't very complicated:
set size square
set angles degrees
set polar
set grid polar 30
set xtics axis 0,30
set ytics axis 0,30
unset border
unset param
set xrange[-90:90]
set yrange[-90:90]
set rrange[0:360]
set trange[0:90]

set title "GPS satellite tracks"
set xlabel "Azimuth"
set ylabel "Elevation"
set terminal png size 600,600
set output "gpsazelsig.png"
plot "gpsazelsig.dat" using 1:2:3 palette notitle
But the resulting plot isn't very helpful for my original question: in which direction radio signals are obstructed. There are some obstructions in the Southwest, but they are comparable to what is in the Northeast.

Tags: , ,
2014-04-16 (#) 5 years ago
I want to get an idea of the 'radio shadow' around our backyard to get a better idea of the minimum elevation to receive from and transmit to amateur radio satellites. Since there still is a gps receiver on the roof of the shed and the earlier ntp experiments aren't running at the moment I decided to stop ntp and log the $GPGSV GPS satellites in view messages from the gps unit. My idea is that the radio signals from GPS satellites get obstructed by houses at least the same as UHF signals, so a GPS satellite reception plot will be interesting. Something like the VisualGPS plot I made at a previous house with a different GPS unit. Note that the plotted satellite tracks are way outside the plotted contour which I recall was a nice approximation of the view during the test.

Now to get this data plotted with gnuplot in a polar plot. I found out the orientation of $GPGSV messages (true north is 0 degrees, east is 90 degrees, south is 180 degrees, west is 270 degrees) does not match the azimuth range available by the polar plot in gnuplot (0 degrees is to the right, 90 degrees is up, 180 degrees is to the left). And the horizon is 0 in $GPGSV messages and maximum range in gnuplot. Time for some perl massaging of the $GPGSV lines to gnuplot orientation:
#!/usr/bin/perl -w

use strict;

while (<>){
    if (/^\$GPGSV,\d+,\d+,\d+,([\d,]+)\*[0-9A-Z]{2}$/){
        my @fields=split(/,/,$1);
        while ($#fields>0){
            my $sv=shift @fields;
            my $elevation=shift @fields;
            my $azimuth=shift @fields;
            my $signal=shift @fields;
            if ($signal){
                warn sprintf "SV %d elevation %d azimuth %d signal %d\n",$sv,$elevation,$azimuth,$signal;
                if ($azimuth<0) {
                printf "%3d %3d\n",$azimuth,90-$elevation;
And indeed we have data:
SV 33 elevation 27 azimuth 205 signal 38
SV 29 elevation 83 azimuth 100 signal 44
SV 31 elevation 48 azimuth 227 signal 45
SV 21 elevation 47 azimuth 169 signal 44
SV 25 elevation 29 azimuth 122 signal 41
And azimuth/elevation in a file that gnuplot can handle:
245  63
  8   9
283  40
226  44
326  64
The azimuth/elevation data, modified for gnuplot. And the next step is a gnuplot plotscript:
set size square
set angles degrees
set polar
set grid polar 30
set xtics 30
unset border
unset param
set xrange[-90:90]
set yrange[-90:90]
set rrange[0:360]
set trange[0:90]

set title "GPS satellite tracks"
set xlabel "Azimuth"
set ylabel "Elevation"
set terminal png size 600,600
set output "gpsazel.png"
plot "gpsazel.dat" using 1:2 notitle
Which indeed gives a nice plot of some recent data.

Main conclusion: this sirf star II gps is 'too good' for this application. For example, one measurement:
SV 5 elevation 4 azimuth 86 signal 37
Satellite 5 seen at an elevation of 4 degrees above the horizon in easterly direction with a signal/noise ratio of 37 dB. There are high buildings (4 floors) in the easterly direction so I think I'm seeing the gps receiver being way too good at this.

The good part is that I'm not the first one to think of this: GPS Skyline: A Panorama in 1.6GHz Microwave-"Light" which suggests I need to find the right cutoff value for my type of GPS unit.

Tags: , ,
2014-02-18 (#) 5 years ago
After getting the gps running in the shed I noticed a bit of variation in the output location as logged from the NMEA $GPGGA strings in the clockstats file. And reading Tom van Baak testing the MG1613S GPS Receiver noting the variation in location made me decide to do a bit of plotting of location on my own. As Tom notes, plotting distance in meters gives a better idea of scale. So I wrote a bit of perl to massage the lat/long pairs into X/Y meters from a starting point. I was lazy: I used the first measurement as starting point. The resulting X/Y pairs are graphed using gnuplot.

Update: I'm a security specialist, not a programmer: I found some errors in the routines that convert output from the GPS to degrees to meters. Fixed them, so the first graph has been redrawn using data from 17 and 18 Februari.

Tags: , ,
2013-12-24 (#) 5 years ago
Modelled after the ntp server statistics at I created years ago I recently started gathering stats on my own. Today I had some time to spare to actually create some graphs from those ntp stats: NTP server stats.

Tags: , ,
2013-12-17 (#) 5 years ago
I looked up the details of the right configuration for ntpd to allow a reset of the packet counters without restarting ntpd for the ntp server project. Relevant part of /etc/ntp.conf:
keys /etc/ntp/ntp.keys
trustedkey 10
requestkey 10
controlkey 10
And in /etc/ntp/ntp.keys is one key 10. And it works:
ntpdc> syss
time since restart:     12
time since reset:       12
packets received:       10
packets processed:      8
current version:        8
previous version:       0
ntpdc> reset sys
Keyid: 10
MD5 Password: 
ntpdc> syss
time since restart:     19
time since reset:       3
packets received:       2
packets processed:      1
current version:        1
previous version:       0
Learned from How do I configure remote administration - ntp faq and miscellaneous commands and options - ntpd.

Tags: , ,
2013-07-19 (#) 5 years ago
My speedtest on T-Mobile umts The predicted change from KPN to T-Mobile took a bit longer than predicted but it has finally happened. Network speed is now 1 mbit down and 32 kbit up according to speedtest.

Somebody I spoke about it wondered whether there was a data subscription included at all or this was the rate at which things could get expensive fast but the T-Mobile business website confirms that this is the slowest data subscription available from T-Mobile NL.

Tags: , , ,
2013-02-19 (#) 6 years ago
Lots of problems have already been solved by people willing to share the solution. So I wasn't surprised somebody already learned Zabbix to work with HP UPS units.

I found Template gallery - Zabbix forums japan with a template for an HP UPS. It took some translating since "Panne d'alimentation" (literally translated: malnourishment, not enough food. Meaning: low input voltage) is too much thinking for me. But it works now and I have all the data, triggers and graphs I want.

Tags: , ,
2012-12-18 (#) 6 years ago
I updated the zabbix ssl certificate test script to be able to use starttls services and did some other changes (tests work better in days left). Current version which can also check for smtp tls and returns the certificate time left in days which makes for easier checks:
#!/usr/bin/perl -w

# monitor the number of days left on the SSL certificate on a publicly
# reachable service
# usage in zabbix, create an item in a template
# - Type: External check
# - Key:  ssl-expiry-left.monitor[443]
#   change this for other services and use ssl-expiry-left.monitor[587,"-smtp"]
#   for smtp+tls. Yes, you will need to set up a separate item (/template)
#   for each ssl port combination
# - Type of information: Numeric (unsigned)
# - Data type: Decimal
# - Units: Days
# - Update interval (in sec): 43200
# - Application: SSL+service
# possible trigger values:
# 0: certificate already expired or invalid or not retrievable
# you can add tests for less than 30 or 60 days left

use strict;
use Date::Parse;

my $protoadd="";

if (defined $ARGV[2]){
        if ($ARGV[2] eq "-smtp"){
                $protoadd="-starttls smtp ";

my ($host,$port) = ($ARGV[0],$ARGV[1]);

open(SSLINFO,"echo \"\" | openssl s_client -connect $host:$port $protoadd 2>/dev/null | openssl x509 -enddate -noout 2>/dev/null |");

my $expiry=0;

while (<SSLINFO>){
        if (/^notAfter=(.+)\n$/){

if ($expiry>0){
        my $daysleft=($expiry-time())/86400;
        printf "%d\n",$daysleft>=0?$daysleft:0;
} else {
        print "0\n";
Assumes a reasonably recent openssl.

And yes, this script has helped me avoid embarrasment over expired certificates.

Tags: , , ,
2012-11-06 (#) 6 years ago
En sommige 'slimme meters' zenden hun actuele gegevens ook continue via radiosignalen uit: Smart meters not so clever about privacy, researchers find - Networkworld (engelstalig artikel).
Researchers at the University of South Carolina have discovered that some types of electricity meter are broadcasting unencrypted information that, with the right software, would enable eavesdroppers to determine whether you're at home.
Met een SDR (software defined radio) is het dus mogelijk de signalen 'gewoon' te ontvangen en met wat zoeken uit te vinden bij welk huis deze horen.

Het bijbehorende research paper Neighborhood Watch: Security and Privacy Analysis of Automatic Meter Reading Systems (pdf) met veel meer technische details zoals frequentie en protocol.

Via Some Smart Meters Broadcast Readings in the Clear - Slashdot met de observatie (voor de US situatie) : Perhaps more distressing, given trends in 4th amendment interpretation, I bet the transmissions are open game for law enforcement.

Tags: , , ,
2012-09-23 (#) 6 years ago
Meer onveiligheid met 'slimme meters', maar dan echt uit de categorie 'privacy gevoelige data delen met iedereen die ervoor wil betalen': Smart meter data shared far and wide - The Age Australia (engelstalig).
DETAILED information about electricity customers' power usage, which gives insights into when a house is occupied, is being shared with third parties including mail houses, debt collectors, data processing analysts and government agencies.
Ik moest even opzoeken wat de term 'mail house' kan betekenen maar dat is denk ik wat we hier een 'marketing bedrijf' noemen, die dus graag zoveel mogelijk gegevens van mensen verzamelen om ze in passende marketing campagnes lastig te vallen.

Dit is wel heel absurd delen van data. En voorzover ik begrijp is de enige manier om achter je eigen gegevens te komen hiermee instemmen. Want je eigen meetgegevens moeten natuurlijk eerst naar een bedrijf wat ze nog net niet publiek post (maar als iemand daarvoor wil betalen..) voordat je ze zelf weer kan opvragen.

Ik blijf erbij: de uitgang voor meetgegevens van 'slimme meters' zit aan de verkeerde kant. Als ik iets wil met deze gegevens wil ik ze aan de binnenkant zelf uitlezen, verwerken, visualiseren en er conclusies uit trekken. Andere statistieken die ik thuis verzamel waar uit af te leiden valt of er mensen thuis zijn kunnen vanwege mijn keuze voor privacy ook niet door derden opgevraagd worden.

Via Australian Smart Meter Data Shared Far and Wide - Slashdot your rights online

Tags: , ,
2012-09-07 (#) 6 years ago
More work in zabbix: we got alerts a few times for load averages > 5. But on a 48-core system in use by people doing calculations that isn't a very useful trigger. My solution is to start monitoring the number of CPUs (a very boring number normally), and create a new trigger
This makes a lot more sense: a load of more than 3 times the number of cores is an issue, both on a 1-core (virtual) machine and on a 48-core calculating monster. On some of those calculation servers a load of less than 10 means some model crashed and a scientist will be trying to restart it.

And we can now set a trigger on any change in the number of cores. That would be interesting.

Tags: , , ,
⇐ Newer news items for tag statistics  Older news items for tag statistics ⇒
, reachable as PGP encrypted e-mail preferred.

PGP key 5BA9 368B E6F3 34E4 local copy PGP key 5BA9 368B E6F3 34E4 via keyservers pgp key statistics for 0x5BA9368BE6F334E4 Koos van den Hout
Other webprojects: Camp Wireless, wireless Internet access at campsites, The Virtual Bookcase, book reviews