News items for tag statistics - Koos van den Hout

2017-03-27 Meten en grafieken maken van Powerline Ethernet doorvoer 2 months ago
Uit de archieven omdat ik de server waar dit ooit op draaide wil herinstalleren voor andere projecten.

Bij de HCC PCgg netwerkgroep hadden we ooit een netwerk gebaseerd op PLC (powerline communicatie) waarbij ethernet over stroomdraden gaat. Daar heb ik ook scripts voor gebouwd om statistieken te verzamelen over de doorvoer. Op de locatie in Maarn was dat voor de verbinding naar buiten niet zo briljant, zie de rapporten in Bijeenkomst 20 maart 2010 - hcc!pc netwerkgroep.

Maar voordat ik de scripts weggooi die deze gegevens verzamelden en er grafiekjes van maakten nog even voor de eeuwigheid. Het testdlan script gaat uit van output van dlanlist als:
koos@metcalfe:~$ dlanlist eth0
Type    MAC address        Mbps TX/RX       Version/Product
local   00:0B:3B:5F:95:AB  ---.-- / ---.--  INT6000-MAC-3-3-3348-00-2764-20080808-FINAL-B devolo dLAN 200 AVplus [MT2165]
remote  00:0B:3B:6F:AE:90   73.50 / 112.88  INT6000-MAC-3-3-3348-00-2764-20080808-FINAL-B

  • testdlan vraag dlanlist van devolo voor de huidige snelheid van devolo interfaces op netwerkkaart(en)
  • graphdlan maak grafieken per dag
De opzet van de scripts was om altijd te draaien als de machine actief was en achteraf overzichten te hebben per dag activiteit (meestal zaterdagen van 10:00 tot 17:00).

Tags: , , ,
2016-11-03 De gasmeter van de slimme meter ging over op wintertijd 7 months ago
Ik ontdekte dat de gasmeter niet meer uitgelezen werd in de scripts die de slimme meter bulletins verwerken. Bij het napluizen van de Dutch smart meter standard v4.0 P1 protocol bleek dat het bericht met de gasmeterstand (0-1:24.2.1) bij de 'Capture Time' een timestamp heeft met S voor summertime en W voor wintertime.

Het script is nu aangepast om dit goed te verwerken en nu zie ik weer het gasgebruik.

Tags: , ,
2016-09-04 De slimme meter uitlezen 9 months ago
Slimme meter uitlezen De slimme meter die meer dan een maand geleden ons huis in kwam wordt nu constant uitgelezen en de resulterende gegevens worden opgeslagen in rrdtool databases van het electriciteits- en gas gebruik.

Uitlezen van een seriele poort vanuit Perl bleek niet makkelijk stabiel te krijgen te zijn. Uiteindelijk heb ik maar een oplossing gekozen/geleend van iemand anders: een stuk python wat cu aanroept: P1/P1-python-cu.py at master · sanderjo/P1. Dit script zou nog iets robuuster zijn als het de CRC controleert, maar dat is dan een wens voor een toekomstige versie.

Vervolgens sla ik de output van dit script op in /var/run/telemetry/smartmeteroutput. De keuze voor /var/run is omdat ik niet elke vijf minuten op de SD kaart van de raspberry wil schrijven. Ik heb dus ook /etc/rc.local aangepast om een /var/run/telemetry met als eigenaar user telemetry te maken, er worden daar meer meetgegevens neergezet voor verdere verwerking in statistieken. Het telemetry concept wat ik gebruik is het verzamelen in ASCII leesbare vorm van meetgegevens op systemen waarna deze opgehaald kunnen worden door een verzamelaar die ze gaat importeren in rrdtool databases (of andere verzamelingen). Omdat de verbinding tussen het te monitoren systeem en de verzamelaar als onbetrouwbaar gezien wordt (sommige systemen waarvan ik meetgegevens verzamel zitten achter een hikkende wifi verbinding) worden de meetgegevens lokaal opgeslagen met een tijdsaanduiding in de bestandsnaam en worden ze in rrdtool geimporteerd met deze tijdsaanduiding.

Het script wat de meetgegevens van de slimme meter verwerkt moet er tegen kunnen dat er soms velden ontbreken. Ik heb er voor gekozen uiteindelijk de dag- en nacht tellerstanden afgenomen en teruggeleverd (meetwaardes 1.8.1, 1.8.2, 2.8.1 en 2.8.2 in de Dutch smart meter standard) als 'verplicht' te tellen en de overige meetwaardes als 'optioneel'.

Uiteindelijk komen er dan mooie grafieken uit. De conclusie die ik eerder trok dat uit energie meetwaarden per kwartier prima af te leiden is wat bewoners doen is nog steeds valide. Uit grafieken over langere termijn is ook keurig af te zien wanneer we op vakantie waren.

Tags: , , , ,
2016-06-23 A serious thunderstorm somewhat counted 1 year ago
Lightning strikes 20160623 Last night a serious lightning storm passed and it got counted, but clearly with the same problems in counting as seen before in counting thunderstorms from the shed while radio activity causes a lot higher counts. Looking at the graphs for thunderstorms counted from the attic before I was active on HF radio there is quite a difference in numbers.

I think I want the lightning strikes counter back up in the attic but with a low-pass filter somehow to filter out false counts from amateur radio traffic.

Tags: , , ,
2015-09-12 Electrisch energieverbruik en gewoontes meten 1 year ago
In mijn 1-wire projecten was ook nog het plan om de watt-uur pulsen van de electricteitsmeter te gaan tellen. Met een schakeling met lichtgevoelige leds en een teller is het me nooit gelukt de pulsen betrouwbaar te tellen.

Ik ontdekte de YouLess die het tellen van zo'n ledje of van een draaischijfmeter terdege opgelost heeft en die niet begint met de data naar een externe dienst te sturen voor ik er naar kan kijken. Er is gewoon een simpele interface waarmee de tellingen van de YouLess via het netwerk uit te lezen zijn en die gebruik ik natuurlijk en maak er interne statistieken van.

YouLess stroomgebruik per kwartier
Gemeten electriciteitsgebruik per kwartier
Van die statistieken maak ik ook een grafiek met een nauwkeurigheid per kwartier. Die keuze is omdat de 'slimme meter' per kwartier uitgelezen kan worden en ik wil wel eens weten wat je er dan uit kan halen. Heel veel dus! Uit de kwartierwaarden is een hoop te halen over onze dagelijkse gewoonten, zeker als je ze over een lange termijn beschikbaar hebt. Voor de meeste pieken kan ik een verklaring bedenken en als eenmaal bekend is welke actie welke hoeveelheid energie gebruikt kan je daarna andere pieken ook weer uitleggen. Opvallend is dat zelfs bij de resolutie van 15 minuten een kort maar duidelijk gebruik zoals de waterkoker of een lange termijn wijziging zoals het licht aan of uit schakelen prima te zien is.
Read the rest of Electrisch energieverbruik en gewoontes meten

Tags: , , ,
2015-08-31 A serious thunderstorm counted 1 year ago
Lightning strikes 2015-08-30 We finally had a serious thunderstorm last night, making us almost jump from the bed at one moment. And the lighting strike counter saw that thunderstorm nicely.

It's quite clear the counts are lower now the sensor is closer to the ground. I'm not sure what to do to get a "good" count again. Last week I placed the 10/20/40m endfed antenna outside for part of the day and noticed that setup also influences the lightning detector in the shed. Raising the end of the antenna helped a bit, I noticed I can use the back gate of our house as the low end of the antenna.

Tags: , ,
2015-07-05 Moving the lightning detector to the shed and testing 1 year ago
A year in lightning strikes and activity on 20M PSK31
The last year in counted lightning strikes, showing clearly that I got active on 20meter psk31 in October 2014. The 'blips' before that are real thunderstorms.
I thought of this ages ago: Moving the lightning strike detector to the shed but only today got around to it because we had some serious chances of a thunderstorm earlier today which showed up on the lightning strike detector but the graph was completely screwed up again after I tried some psk31 digital mode transmissions on the 20 meter amateur band (14.000 to 14.250 MHz for me).

So now it is in the shed. Moving it to a lower position does mean I will not get readings for thunderstorms as far away as I used to but I'd rather have usable readings at this moment. First tests with transmitting psk31/psk63 on 20 the meter amateur band after I changed it look like it doesn't count the transmissions anymore. Now to wait for the next good thunderstorm to see how that gets counted. Some blips are showing up.

Update 2015-07-14: The first result seems to be that using the lights in the shed (tubelights with starters) shows up clearly. Using the radio still has no effect. I now await the first thunderstorm for more results.

Update 2015-07-28: No thunderstorm has been reported by the KNMI weather institute thunderstorm archive within a short distance of my sensor. I guess the maximum range is quite limited now.

Tags: , , ,
2015-06-16 SSL implementation on the fritzbox isn't secure enough 2 years ago
The latest OpenSSL updates cause me a new problem:
Connecting to fritz.koos.koffie.dot (fritz.koos.koffie.dot)|192.168.178.1|:49443... connected.
OpenSSL: error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small
Unable to establish SSL connection.
Which means the script to fetch the dsl status from the fritzbox can't connect until I find out how to convince wget how to negotiate a non-standard cipher set. Or switch to curl.

Getting the right answers with curl isn't working out either. I can get the SSL working and do a POST to the right URL but the 'best' thing I get back is:
<errorCode>502</errorCode>
<errorDescription>XML error</errorDescription>

Update: The solution was to keep using wget but disable(!) SSL, using the non-SSL port for upnp. The command now is:
wget --user=$FRITZUSER --password=$FRITZPASS --post-file=linkstatusrequest.xml \
--header="Content-Type: text/xml" \
--header="SOAPAction: \"urn:dslforum-org:service:WANCommonInterfaceConfig:1#GetCommonLinkProperties\"" \
http://192.168.178.1:49000/upnp/control/wancommonifconfig1 -O linkstatusanswer.xml
VDSL downstream speed 20150618 And now the data is available again and the graph is updated.

So the recent upgrade in OpenSSL which disabled less secure Diffie-Hellman key negotiation results in having to disable all encryption on the connection with the fritzbox. A security update on the fritzbox may solve this.

Tags: , , , ,
2015-03-27 Overly interested Amazon EC2 nodes 2 years ago
On Camp Wireless and The Virtual Bookcase I see the following pattern in the access logs:
2620:108:700f::36bc:aade - - [27/Mar/2015:13:27:11 +0100] "GET / HTTP/1.1" 302 298 "-" "curl/7.36.0"
2406:da00:ff00::36e2:d963 - - [27/Mar/2015:13:27:38 +0100] "GET / HTTP/1.1" 302 298 "-" "curl/7.36.0"
Constant requests, 2 or 3 per minute from Amazon EC2 IPv6 addresses just requesting the / using curl. Over the day I now see 1334 unique addresses with at most 5 requests from one url.

The same pattern as described in Stange stream of HTTP GET requests in apache logs, from amazon ec2 instances - Server Fault with no real answer to the why.

It's not a problematic amount of traffic, I'd just like to understand what is happenning!

Tags: , , , , ,
2014-11-28 Moving the lightning strike detector to the shed 2 years ago
I have noticed the lightning strike detection in Weather station Utrecht Overvecht goes completely mad when I transmit on the 20 meter amateur band. With the detector being quite close to the antenna I can understand this.

The solution will be to find a place to mount the detector in the shed. It will be lower (less reception of the radio energy of the strikes) but it will also be further away from my interference.

That also means the reading of the detector will have to be done using w1retap since that is what I use on the shed weatherstation computer. I was a bit confused whether w1retap supports this counter but I found out it's based on the DS2423 counter chip which is supported in w1retap, as part of a wind speed meter in a TAI8515 weather station, but w1retap will give the count on readout and the conversion is up to the user.

Tags: , , , ,
2014-11-26 Getting the DSL linespeed from the Fritz!Box 7360 2 years ago
The fact I couldn't get the DSL linespeed from the Fritz!Box 7360 annoyed me a lot, especially since there is a new telephone wiring cabinet which should raise VDSL2 speeds. I went through a number of websites about getting data out of the Fritz!Box with upnp, and finally I made it work and I get the results I want:
<NewLayer1UpstreamMaxBitRate>1480000</NewLayer1UpstreamMaxBitRate>
<NewLayer1DownstreamMaxBitRate>23144000</NewLayer1DownstreamMaxBitRate>
The hint that worked for me was at MRTG en Fritz!Box 7360 (firmware 124.06.05) - tweakers (in Dutch) where it mentiones the changeover to TR-064 protocol which should be reachable over the http://192.168.178.1/tr064/tr64desc.xml url which will ask for authentication with the root username and the Fritz!Box password. More about the Fritz!Box TR-064 implementation at Schnittstellen für Entwickler | AVM Deutschland (in German) which has more documentation at AVM TR-064 – First Steps (pdf, English). This made me end up at doing a SOAP request (post) to http://192.168.178.1/tr064/upnp/control/wancommonifconfig1 which failed. All SOAP requests fail with an HTTP error code 500, but there is a separate SOAP error set in the HTTP status 500 body. I used tcpdump to look at the SOAP error body and found:
<errorCode>504</errorCode> <errorDescription>SSL needed</errorDescription></UPnPError>
The SSL port is (according to the TR-064 first steps document above) 49443 and the URL is over SSL: https://192.168.179.1:49443/upnp/control/wancommonifconfig1 and this works, giving the answers I want.
Read the rest of Getting the DSL linespeed from the Fritz!Box 7360

Tags: , ,
2014-05-25 (#) 3 years ago
After testing the gps sky view it's now time to test with ntpd. First step was to recompile ntpd because the debian default package had no pps support. Recompiling on a 500 MHz AMD Geode takes a bit of time.

Results look ok for a first test:
root@ritchie:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+greenblatt.idef 131.211.8.244    2 u   17   64  377    1.177  101.732  63.403
*metronoom.dmz.c .PPS.            1 u   17   64  377   19.499  100.512   6.722
+auth1.xs4all.nl 193.67.79.202    2 u   11   64  377   18.403  104.008   3.669
oGPS_NMEA(0)     .GPS.            0 l    6    8  377    0.000  114.073   7.364
root@ritchie:~# ntpdc -c loopi
offset:               0.001986 s
frequency:            94.434 ppm
poll adjust:          -30
watchdog timer:       342 s
root@ritchie:~# ntpdc -c kerni
pll offset:           3.3e-08 s
pll frequency:        94.434 ppm
maximum error:        0.175258 s
estimated error:      2e-06 s
status:               2007  pll ppsfreq ppstime nano
pll time constant:    3
precision:            1e-09 s
frequency tolerance:  500 ppm
root@ritchie:~# ntpdc -c sysi
system peer:          GPS_NMEA(0)
system peer mode:     client
leap indicator:       00
stratum:              1
precision:            -19
root distance:        0.00000 s
root dispersion:      0.00749 s
reference ID:         [GPS]
reference time:       d72cb010.dc91595e  Sun, May 25 2014 20:08:16.861
system flags:         auth monitor ntp kernel stats pps 
jitter:               0.006714 s
stability:            0.000 ppm
broadcastdelay:       0.000000 s
authdelay:            0.000000 s
It will need some more calibration probably.

Update: It keeps looking nice after some calibration. Stats gathered at NTP server ritchie.idefix.net stats.

This does mean one of the old project sundial goals has been met: the weather station computer in the shed is now also a time server.

Tags: , , ,
2014-05-24 (#) 3 years ago
I was able to buy a real Garmin GPS 18 LVC secondhand. It's now on the roof of our shed. The first thing I want to do is repeat my plotting of GPS satellite positions from $GPGSV messages and plotting of GPS satellite positions and signal strengths from $GPGSV messages measurements with data from this unit. After that has run for a while I'll configure ntpd to get the correct time from the GPS unit and the PPS signal.

And again, the resulting plot of gps satellite positions versus signal strength is not very helpful in finding out which part of the sky is obscured.

Tags: , ,
2014-04-18 (#) 3 years ago
A bit of searching later found the right incantation to make gnuplot adjust color based on a third value (signal level in my case). It isn't very complicated:
set size square
set angles degrees
set polar
set grid polar 30
set xtics axis 0,30
set ytics axis 0,30
unset border
unset param
set xrange[-90:90]
set yrange[-90:90]
set rrange[0:360]
set trange[0:90]

set title "GPS satellite tracks"
set xlabel "Azimuth"
set ylabel "Elevation"
set terminal png size 600,600
set output "gpsazelsig.png"
plot "gpsazelsig.dat" using 1:2:3 palette notitle
But the resulting plot isn't very helpful for my original question: in which direction radio signals are obstructed. There are some obstructions in the Southwest, but they are comparable to what is in the Northeast.

Tags: , ,
2014-04-16 (#) 3 years ago
I want to get an idea of the 'radio shadow' around our backyard to get a better idea of the minimum elevation to receive from and transmit to amateur radio satellites. Since there still is a gps receiver on the roof of the shed and the earlier ntp experiments aren't running at the moment I decided to stop ntp and log the $GPGSV GPS satellites in view messages from the gps unit. My idea is that the radio signals from GPS satellites get obstructed by houses at least the same as UHF signals, so a GPS satellite reception plot will be interesting. Something like the VisualGPS plot I made at a previous house with a different GPS unit. Note that the plotted satellite tracks are way outside the plotted contour which I recall was a nice approximation of the view during the test.

Now to get this data plotted with gnuplot in a polar plot. I found out the orientation of $GPGSV messages (true north is 0 degrees, east is 90 degrees, south is 180 degrees, west is 270 degrees) does not match the azimuth range available by the polar plot in gnuplot (0 degrees is to the right, 90 degrees is up, 180 degrees is to the left). And the horizon is 0 in $GPGSV messages and maximum range in gnuplot. Time for some perl massaging of the $GPGSV lines to gnuplot orientation:
#!/usr/bin/perl -w

use strict;

while (<>){
    chomp;
    if (/^\$GPGSV,\d+,\d+,\d+,([\d,]+)\*[0-9A-Z]{2}$/){
        my @fields=split(/,/,$1);
        while ($#fields>0){
            my $sv=shift @fields;
            my $elevation=shift @fields;
            my $azimuth=shift @fields;
            my $signal=shift @fields;
            if ($signal){
                warn sprintf "SV %d elevation %d azimuth %d signal %d\n",$sv,$elevation,$azimuth,$signal;
                $azimuth=90-$azimuth;
                if ($azimuth<0) {
                    $azimuth+=360;
                }
                printf "%3d %3d\n",$azimuth,90-$elevation;
            }                   
        }               
    }           
}       
And indeed we have data:
SV 33 elevation 27 azimuth 205 signal 38
SV 29 elevation 83 azimuth 100 signal 44
SV 31 elevation 48 azimuth 227 signal 45
SV 21 elevation 47 azimuth 169 signal 44
SV 25 elevation 29 azimuth 122 signal 41
And azimuth/elevation in a file that gnuplot can handle:
245  63
  8   9
283  40
226  44
326  64
The azimuth/elevation data, modified for gnuplot. And the next step is a gnuplot plotscript:
set size square
set angles degrees
set polar
set grid polar 30
set xtics 30
unset border
unset param
set xrange[-90:90]
set yrange[-90:90]
set rrange[0:360]
set trange[0:90]

set title "GPS satellite tracks"
set xlabel "Azimuth"
set ylabel "Elevation"
set terminal png size 600,600
set output "gpsazel.png"
plot "gpsazel.dat" using 1:2 notitle
Which indeed gives a nice plot of some recent data.

Main conclusion: this sirf star II gps is 'too good' for this application. For example, one measurement:
SV 5 elevation 4 azimuth 86 signal 37
Satellite 5 seen at an elevation of 4 degrees above the horizon in easterly direction with a signal/noise ratio of 37 dB. There are high buildings (4 floors) in the easterly direction so I think I'm seeing the gps receiver being way too good at this.

The good part is that I'm not the first one to think of this: GPS Skyline: A Panorama in 1.6GHz Microwave-"Light" which suggests I need to find the right cutoff value for my type of GPS unit.

Tags: , ,
2014-02-18 (#) 3 years ago
After getting the gps running in the shed I noticed a bit of variation in the output location as logged from the NMEA $GPGGA strings in the clockstats file. And reading Tom van Baak testing the MG1613S GPS Receiver noting the variation in location made me decide to do a bit of plotting of location on my own. As Tom notes, plotting distance in meters gives a better idea of scale. So I wrote a bit of perl to massage the lat/long pairs into X/Y meters from a starting point. I was lazy: I used the first measurement as starting point. The resulting X/Y pairs are graphed using gnuplot.

Update: I'm a security specialist, not a programmer: I found some errors in the routines that convert output from the GPS to degrees to meters. Fixed them, so the first graph has been redrawn using data from 17 and 18 Februari.

Tags: , ,
2013-12-24 (#) 3 years ago
Modelled after the ntp server statistics at cs.uu.nl I created years ago I recently started gathering stats on my own. Today I had some time to spare to actually create some graphs from those ntp stats: NTP server stats.

Tags: , ,
2013-12-17 (#) 3 years ago
I looked up the details of the right configuration for ntpd to allow a reset of the packet counters without restarting ntpd for the ntp server project. Relevant part of /etc/ntp.conf:
keys /etc/ntp/ntp.keys
trustedkey 10
requestkey 10
controlkey 10
And in /etc/ntp/ntp.keys is one key 10. And it works:
ntpdc> syss
time since restart:     12
time since reset:       12
packets received:       10
packets processed:      8
current version:        8
previous version:       0
..
ntpdc> reset sys
Keyid: 10
MD5 Password: 
done!
ntpdc> syss
time since restart:     19
time since reset:       3
packets received:       2
packets processed:      1
current version:        1
previous version:       0
..
Learned from How do I configure remote administration - ntp faq and miscellaneous commands and options - ntpd.

Tags: , ,
2013-07-19 (#) 3 years ago
My speedtest on T-Mobile umts The predicted change from KPN to T-Mobile took a bit longer than predicted but it has finally happened. Network speed is now 1 mbit down and 32 kbit up according to speedtest.

Somebody I spoke about it wondered whether there was a data subscription included at all or this was the rate at which things could get expensive fast but the T-Mobile business website confirms that this is the slowest data subscription available from T-Mobile NL.

Tags: , , ,
2013-02-19 (#) 4 years ago
Lots of problems have already been solved by people willing to share the solution. So I wasn't surprised somebody already learned Zabbix to work with HP UPS units.

I found Template gallery - Zabbix forums japan with a template for an HP UPS. It took some translating since "Panne d'alimentation" (literally translated: malnourishment, not enough food. Meaning: low input voltage) is too much thinking for me. But it works now and I have all the data, triggers and graphs I want.

Tags: , ,
  Older news items for tag statistics ⇒
, reachable as koos+website@idefix.net. PGP encrypted e-mail preferred.

PGP key 2C66 3B5D F0D7 C263 local copy PGP key 2C66 3B5D F0D7 C263 via keyservers pgp key statistics for 0x2C663B5DF0D7C263 Koos van den Hout
RSS
Other webprojects: Camp Wireless, wireless Internet access at campsites, The Virtual Bookcase, book reviews, Weather maps