News items for tag statistics - Koos van den Hout

2015-07-05 Moving the lightning detector to the shed and testing 3 years ago
A year in lightning strikes and activity on 20M PSK31
The last year in counted lightning strikes, showing clearly that I got active on 20meter psk31 in October 2014. The 'blips' before that are real thunderstorms.
I thought of this ages ago: Moving the lightning strike detector to the shed but only today got around to it because we had some serious chances of a thunderstorm earlier today which showed up on the lightning strike detector but the graph was completely screwed up again after I tried some psk31 digital mode transmissions on the 20 meter amateur band (14.000 to 14.250 MHz for me).

So now it is in the shed. Moving it to a lower position does mean I will not get readings for thunderstorms as far away as I used to but I'd rather have usable readings at this moment. First tests with transmitting psk31/psk63 on 20 the meter amateur band after I changed it look like it doesn't count the transmissions anymore. Now to wait for the next good thunderstorm to see how that gets counted. Some blips are showing up.

Update 2015-07-14: The first result seems to be that using the lights in the shed (tubelights with starters) shows up clearly. Using the radio still has no effect. I now await the first thunderstorm for more results.

Update 2015-07-28: No thunderstorm has been reported by the KNMI weather institute thunderstorm archive within a short distance of my sensor. I guess the maximum range is quite limited now.

Tags: , , ,
2015-06-16 SSL implementation on the fritzbox isn't secure enough 3 years ago
The latest OpenSSL updates cause me a new problem:
Connecting to fritz.koos.koffie.dot (fritz.koos.koffie.dot)|192.168.178.1|:49443... connected.
OpenSSL: error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small
Unable to establish SSL connection.
Which means the script to fetch the dsl status from the fritzbox can't connect until I find out how to convince wget how to negotiate a non-standard cipher set. Or switch to curl.

Getting the right answers with curl isn't working out either. I can get the SSL working and do a POST to the right URL but the 'best' thing I get back is:
<errorCode>502</errorCode>
<errorDescription>XML error</errorDescription>

Update: The solution was to keep using wget but disable(!) SSL, using the non-SSL port for upnp. The command now is:
wget --user=$FRITZUSER --password=$FRITZPASS --post-file=linkstatusrequest.xml \
--header="Content-Type: text/xml" \
--header="SOAPAction: \"urn:dslforum-org:service:WANCommonInterfaceConfig:1#GetCommonLinkProperties\"" \
http://192.168.178.1:49000/upnp/control/wancommonifconfig1 -O linkstatusanswer.xml
VDSL downstream speed 20150618 And now the data is available again and the graph is updated.

So the recent upgrade in OpenSSL which disabled less secure Diffie-Hellman key negotiation results in having to disable all encryption on the connection with the fritzbox. A security update on the fritzbox may solve this.

Tags: , , , ,
2015-03-27 Overly interested Amazon EC2 nodes 3 years ago
On Camp Wireless and The Virtual Bookcase I see the following pattern in the access logs:
2620:108:700f::36bc:aade - - [27/Mar/2015:13:27:11 +0100] "GET / HTTP/1.1" 302 298 "-" "curl/7.36.0"
2406:da00:ff00::36e2:d963 - - [27/Mar/2015:13:27:38 +0100] "GET / HTTP/1.1" 302 298 "-" "curl/7.36.0"
Constant requests, 2 or 3 per minute from Amazon EC2 IPv6 addresses just requesting the / using curl. Over the day I now see 1334 unique addresses with at most 5 requests from one url.

The same pattern as described in Stange stream of HTTP GET requests in apache logs, from amazon ec2 instances - Server Fault with no real answer to the why.

It's not a problematic amount of traffic, I'd just like to understand what is happenning!

Tags: , , , , ,
2014-11-28 Moving the lightning strike detector to the shed 4 years ago
I have noticed the lightning strike detection in Weather station Utrecht Overvecht goes completely mad when I transmit on the 20 meter amateur band. With the detector being quite close to the antenna I can understand this.

The solution will be to find a place to mount the detector in the shed. It will be lower (less reception of the radio energy of the strikes) but it will also be further away from my interference.

That also means the reading of the detector will have to be done using w1retap since that is what I use on the shed weatherstation computer. I was a bit confused whether w1retap supports this counter but I found out it's based on the DS2423 counter chip which is supported in w1retap, as part of a wind speed meter in a TAI8515 weather station, but w1retap will give the count on readout and the conversion is up to the user.

Tags: , , , ,
2014-11-26 Getting the DSL linespeed from the Fritz!Box 7360 4 years ago
The fact I couldn't get the DSL linespeed from the Fritz!Box 7360 annoyed me a lot, especially since there is a new telephone wiring cabinet which should raise VDSL2 speeds. I went through a number of websites about getting data out of the Fritz!Box with upnp, and finally I made it work and I get the results I want:
<NewLayer1UpstreamMaxBitRate>1480000</NewLayer1UpstreamMaxBitRate>
<NewLayer1DownstreamMaxBitRate>23144000</NewLayer1DownstreamMaxBitRate>
The hint that worked for me was at MRTG en Fritz!Box 7360 (firmware 124.06.05) - tweakers (in Dutch) where it mentiones the changeover to TR-064 protocol which should be reachable over the http://192.168.178.1/tr064/tr64desc.xml url which will ask for authentication with the root username and the Fritz!Box password. More about the Fritz!Box TR-064 implementation at Schnittstellen für Entwickler | AVM Deutschland (in German) which has more documentation at AVM TR-064 – First Steps (pdf, English). This made me end up at doing a SOAP request (post) to http://192.168.178.1/tr064/upnp/control/wancommonifconfig1 which failed. All SOAP requests fail with an HTTP error code 500, but there is a separate SOAP error set in the HTTP status 500 body. I used tcpdump to look at the SOAP error body and found:
<errorCode>504</errorCode> <errorDescription>SSL needed</errorDescription></UPnPError>
The SSL port is (according to the TR-064 first steps document above) 49443 and the URL is over SSL: https://192.168.179.1:49443/upnp/control/wancommonifconfig1 and this works, giving the answers I want.
Read the rest of Getting the DSL linespeed from the Fritz!Box 7360

Tags: , ,
2014-05-25 (#) 4 years ago
After testing the gps sky view it's now time to test with ntpd. First step was to recompile ntpd because the debian default package had no pps support. Recompiling on a 500 MHz AMD Geode takes a bit of time.

Results look ok for a first test:
root@ritchie:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+greenblatt.idef 131.211.8.244    2 u   17   64  377    1.177  101.732  63.403
*metronoom.dmz.c .PPS.            1 u   17   64  377   19.499  100.512   6.722
+auth1.xs4all.nl 193.67.79.202    2 u   11   64  377   18.403  104.008   3.669
oGPS_NMEA(0)     .GPS.            0 l    6    8  377    0.000  114.073   7.364
root@ritchie:~# ntpdc -c loopi
offset:               0.001986 s
frequency:            94.434 ppm
poll adjust:          -30
watchdog timer:       342 s
root@ritchie:~# ntpdc -c kerni
pll offset:           3.3e-08 s
pll frequency:        94.434 ppm
maximum error:        0.175258 s
estimated error:      2e-06 s
status:               2007  pll ppsfreq ppstime nano
pll time constant:    3
precision:            1e-09 s
frequency tolerance:  500 ppm
root@ritchie:~# ntpdc -c sysi
system peer:          GPS_NMEA(0)
system peer mode:     client
leap indicator:       00
stratum:              1
precision:            -19
root distance:        0.00000 s
root dispersion:      0.00749 s
reference ID:         [GPS]
reference time:       d72cb010.dc91595e  Sun, May 25 2014 20:08:16.861
system flags:         auth monitor ntp kernel stats pps 
jitter:               0.006714 s
stability:            0.000 ppm
broadcastdelay:       0.000000 s
authdelay:            0.000000 s
It will need some more calibration probably.

Update: It keeps looking nice after some calibration. Stats gathered at NTP server ritchie.idefix.net stats.

This does mean one of the old project sundial goals has been met: the weather station computer in the shed is now also a time server.

Tags: , , ,
2014-05-24 (#) 4 years ago
I was able to buy a real Garmin GPS 18 LVC secondhand. It's now on the roof of our shed. The first thing I want to do is repeat my plotting of GPS satellite positions from $GPGSV messages and plotting of GPS satellite positions and signal strengths from $GPGSV messages measurements with data from this unit. After that has run for a while I'll configure ntpd to get the correct time from the GPS unit and the PPS signal.

And again, the resulting plot of gps satellite positions versus signal strength is not very helpful in finding out which part of the sky is obscured.

Tags: , ,
2014-04-18 (#) 4 years ago
A bit of searching later found the right incantation to make gnuplot adjust color based on a third value (signal level in my case). It isn't very complicated:
set size square
set angles degrees
set polar
set grid polar 30
set xtics axis 0,30
set ytics axis 0,30
unset border
unset param
set xrange[-90:90]
set yrange[-90:90]
set rrange[0:360]
set trange[0:90]

set title "GPS satellite tracks"
set xlabel "Azimuth"
set ylabel "Elevation"
set terminal png size 600,600
set output "gpsazelsig.png"
plot "gpsazelsig.dat" using 1:2:3 palette notitle
But the resulting plot isn't very helpful for my original question: in which direction radio signals are obstructed. There are some obstructions in the Southwest, but they are comparable to what is in the Northeast.

Tags: , ,
2014-04-16 (#) 4 years ago
I want to get an idea of the 'radio shadow' around our backyard to get a better idea of the minimum elevation to receive from and transmit to amateur radio satellites. Since there still is a gps receiver on the roof of the shed and the earlier ntp experiments aren't running at the moment I decided to stop ntp and log the $GPGSV GPS satellites in view messages from the gps unit. My idea is that the radio signals from GPS satellites get obstructed by houses at least the same as UHF signals, so a GPS satellite reception plot will be interesting. Something like the VisualGPS plot I made at a previous house with a different GPS unit. Note that the plotted satellite tracks are way outside the plotted contour which I recall was a nice approximation of the view during the test.

Now to get this data plotted with gnuplot in a polar plot. I found out the orientation of $GPGSV messages (true north is 0 degrees, east is 90 degrees, south is 180 degrees, west is 270 degrees) does not match the azimuth range available by the polar plot in gnuplot (0 degrees is to the right, 90 degrees is up, 180 degrees is to the left). And the horizon is 0 in $GPGSV messages and maximum range in gnuplot. Time for some perl massaging of the $GPGSV lines to gnuplot orientation:
#!/usr/bin/perl -w

use strict;

while (<>){
    chomp;
    if (/^\$GPGSV,\d+,\d+,\d+,([\d,]+)\*[0-9A-Z]{2}$/){
        my @fields=split(/,/,$1);
        while ($#fields>0){
            my $sv=shift @fields;
            my $elevation=shift @fields;
            my $azimuth=shift @fields;
            my $signal=shift @fields;
            if ($signal){
                warn sprintf "SV %d elevation %d azimuth %d signal %d\n",$sv,$elevation,$azimuth,$signal;
                $azimuth=90-$azimuth;
                if ($azimuth<0) {
                    $azimuth+=360;
                }
                printf "%3d %3d\n",$azimuth,90-$elevation;
            }                   
        }               
    }           
}       
And indeed we have data:
SV 33 elevation 27 azimuth 205 signal 38
SV 29 elevation 83 azimuth 100 signal 44
SV 31 elevation 48 azimuth 227 signal 45
SV 21 elevation 47 azimuth 169 signal 44
SV 25 elevation 29 azimuth 122 signal 41
And azimuth/elevation in a file that gnuplot can handle:
245  63
  8   9
283  40
226  44
326  64
The azimuth/elevation data, modified for gnuplot. And the next step is a gnuplot plotscript:
set size square
set angles degrees
set polar
set grid polar 30
set xtics 30
unset border
unset param
set xrange[-90:90]
set yrange[-90:90]
set rrange[0:360]
set trange[0:90]

set title "GPS satellite tracks"
set xlabel "Azimuth"
set ylabel "Elevation"
set terminal png size 600,600
set output "gpsazel.png"
plot "gpsazel.dat" using 1:2 notitle
Which indeed gives a nice plot of some recent data.

Main conclusion: this sirf star II gps is 'too good' for this application. For example, one measurement:
SV 5 elevation 4 azimuth 86 signal 37
Satellite 5 seen at an elevation of 4 degrees above the horizon in easterly direction with a signal/noise ratio of 37 dB. There are high buildings (4 floors) in the easterly direction so I think I'm seeing the gps receiver being way too good at this.

The good part is that I'm not the first one to think of this: GPS Skyline: A Panorama in 1.6GHz Microwave-"Light" which suggests I need to find the right cutoff value for my type of GPS unit.

Tags: , ,
2014-02-18 (#) 5 years ago
After getting the gps running in the shed I noticed a bit of variation in the output location as logged from the NMEA $GPGGA strings in the clockstats file. And reading Tom van Baak testing the MG1613S GPS Receiver noting the variation in location made me decide to do a bit of plotting of location on my own. As Tom notes, plotting distance in meters gives a better idea of scale. So I wrote a bit of perl to massage the lat/long pairs into X/Y meters from a starting point. I was lazy: I used the first measurement as starting point. The resulting X/Y pairs are graphed using gnuplot.

Update: I'm a security specialist, not a programmer: I found some errors in the routines that convert output from the GPS to degrees to meters. Fixed them, so the first graph has been redrawn using data from 17 and 18 Februari.

Tags: , ,
2013-12-24 (#) 5 years ago
Modelled after the ntp server statistics at cs.uu.nl I created years ago I recently started gathering stats on my own. Today I had some time to spare to actually create some graphs from those ntp stats: NTP server stats.

Tags: , ,
2013-12-17 (#) 5 years ago
I looked up the details of the right configuration for ntpd to allow a reset of the packet counters without restarting ntpd for the ntp server project. Relevant part of /etc/ntp.conf:
keys /etc/ntp/ntp.keys
trustedkey 10
requestkey 10
controlkey 10
And in /etc/ntp/ntp.keys is one key 10. And it works:
ntpdc> syss
time since restart:     12
time since reset:       12
packets received:       10
packets processed:      8
current version:        8
previous version:       0
..
ntpdc> reset sys
Keyid: 10
MD5 Password: 
done!
ntpdc> syss
time since restart:     19
time since reset:       3
packets received:       2
packets processed:      1
current version:        1
previous version:       0
..
Learned from How do I configure remote administration - ntp faq and miscellaneous commands and options - ntpd.

Tags: , ,
2013-07-19 (#) 5 years ago
My speedtest on T-Mobile umts The predicted change from KPN to T-Mobile took a bit longer than predicted but it has finally happened. Network speed is now 1 mbit down and 32 kbit up according to speedtest.

Somebody I spoke about it wondered whether there was a data subscription included at all or this was the rate at which things could get expensive fast but the T-Mobile business website confirms that this is the slowest data subscription available from T-Mobile NL.

Tags: , , ,
2013-02-19 (#) 6 years ago
Lots of problems have already been solved by people willing to share the solution. So I wasn't surprised somebody already learned Zabbix to work with HP UPS units.

I found Template gallery - Zabbix forums japan with a template for an HP UPS. It took some translating since "Panne d'alimentation" (literally translated: malnourishment, not enough food. Meaning: low input voltage) is too much thinking for me. But it works now and I have all the data, triggers and graphs I want.

Tags: , ,
2012-12-18 (#) 6 years ago
I updated the zabbix ssl certificate test script to be able to use starttls services and did some other changes (tests work better in days left). Current version which can also check for smtp tls and returns the certificate time left in days which makes for easier checks:
#!/usr/bin/perl -w

# monitor the number of days left on the SSL certificate on a publicly
# reachable service
#
# usage in zabbix, create an item in a template
# - Type: External check
# - Key:  ssl-expiry-left.monitor[443]
#   change this for other services and use ssl-expiry-left.monitor[587,"-smtp"]
#   for smtp+tls. Yes, you will need to set up a separate item (/template)
#   for each ssl port combination
# - Type of information: Numeric (unsigned)
# - Data type: Decimal
# - Units: Days
# - Update interval (in sec): 43200
# - Application: SSL+service
#
# possible trigger values:
#
# 0: certificate already expired or invalid or not retrievable
# 
# you can add tests for less than 30 or 60 days left

use strict;
use Date::Parse;

my $protoadd="";

if (defined $ARGV[2]){
        if ($ARGV[2] eq "-smtp"){
                $protoadd="-starttls smtp ";
        }
}

my ($host,$port) = ($ARGV[0],$ARGV[1]);

open(SSLINFO,"echo \"\" | openssl s_client -connect $host:$port $protoadd 2>/dev/null | openssl x509 -enddate -noout 2>/dev/null |");

my $expiry=0;

while (<SSLINFO>){
        if (/^notAfter=(.+)\n$/){
                $expiry=str2time($1);
        }
}

if ($expiry>0){
        my $daysleft=($expiry-time())/86400;
        printf "%d\n",$daysleft>=0?$daysleft:0;
} else {
        print "0\n";
}
Assumes a reasonably recent openssl.

And yes, this script has helped me avoid embarrasment over expired certificates.

Tags: , , ,
2012-11-06 (#) 6 years ago
En sommige 'slimme meters' zenden hun actuele gegevens ook continue via radiosignalen uit: Smart meters not so clever about privacy, researchers find - Networkworld (engelstalig artikel).
Researchers at the University of South Carolina have discovered that some types of electricity meter are broadcasting unencrypted information that, with the right software, would enable eavesdroppers to determine whether you're at home.
Met een SDR (software defined radio) is het dus mogelijk de signalen 'gewoon' te ontvangen en met wat zoeken uit te vinden bij welk huis deze horen.

Het bijbehorende research paper Neighborhood Watch: Security and Privacy Analysis of Automatic Meter Reading Systems (pdf) met veel meer technische details zoals frequentie en protocol.

Via Some Smart Meters Broadcast Readings in the Clear - Slashdot met de observatie (voor de US situatie) : Perhaps more distressing, given trends in 4th amendment interpretation, I bet the transmissions are open game for law enforcement.

Tags: , , ,
2012-09-23 (#) 6 years ago
Meer onveiligheid met 'slimme meters', maar dan echt uit de categorie 'privacy gevoelige data delen met iedereen die ervoor wil betalen': Smart meter data shared far and wide - The Age Australia (engelstalig).
DETAILED information about electricity customers' power usage, which gives insights into when a house is occupied, is being shared with third parties including mail houses, debt collectors, data processing analysts and government agencies.
Ik moest even opzoeken wat de term 'mail house' kan betekenen maar dat is denk ik wat we hier een 'marketing bedrijf' noemen, die dus graag zoveel mogelijk gegevens van mensen verzamelen om ze in passende marketing campagnes lastig te vallen.

Dit is wel heel absurd delen van data. En voorzover ik begrijp is de enige manier om achter je eigen gegevens te komen hiermee instemmen. Want je eigen meetgegevens moeten natuurlijk eerst naar een bedrijf wat ze nog net niet publiek post (maar als iemand daarvoor wil betalen..) voordat je ze zelf weer kan opvragen.

Ik blijf erbij: de uitgang voor meetgegevens van 'slimme meters' zit aan de verkeerde kant. Als ik iets wil met deze gegevens wil ik ze aan de binnenkant zelf uitlezen, verwerken, visualiseren en er conclusies uit trekken. Andere statistieken die ik thuis verzamel waar uit af te leiden valt of er mensen thuis zijn kunnen vanwege mijn keuze voor privacy ook niet door derden opgevraagd worden.

Via Australian Smart Meter Data Shared Far and Wide - Slashdot your rights online

Tags: , ,
2012-09-07 (#) 6 years ago
More work in zabbix: we got alerts a few times for load averages > 5. But on a 48-core system in use by people doing calculations that isn't a very useful trigger. My solution is to start monitoring the number of CPUs (a very boring number normally), and create a new trigger
{Template_geo_linux:system.cpu.load[all,avg1].last(0)}/{Template_geo_linux:system.cpu.num.last(0)}>3
This makes a lot more sense: a load of more than 3 times the number of cores is an issue, both on a 1-core (virtual) machine and on a 48-core calculating monster. On some of those calculation servers a load of less than 10 means some model crashed and a scientist will be trying to restart it.

And we can now set a trigger on any change in the number of cores. That would be interesting.
{Template_geo_linux:system.cpu.num.change(0)}>0

Tags: , , ,
2012-09-05 (#) 6 years ago
I want to do some network measurements. Measuring throughput over tcp or udp is easy: use iperf. Although the iperf server process in daemon mode was using 100% cpu for each(!) measurement somehow, so the load on the machine tested ran up to more than 50.

But I am also interested in tcp setup time. It took a bit longer to find something which does that. Searching gave me measure tcp connection setup time: time-gai-connect which was developed to measure differences in IPv4 and IPv6 tcp connection setup times. Interesting answers:
$ ./time-gai-connect.py www.google.com
2a00:1450:400c:c05::63                   19.62209 ms
173.194.78.147                           19.78207 ms
173.194.78.106                           20.66183 ms
173.194.78.105                           19.69004 ms
173.194.78.103                           20.01190 ms
173.194.78.99                            20.47896 ms
173.194.78.104                           21.71206 ms
And this one is really strange to me... on the machine which runs both the IPv4 address and IPv6 address for idefix.net:
$ ./time-gai-connect.py idefix.net
2001:980:14ca:42::18                      0.35095 ms
82.95.196.202                             0.18501 ms
Yes, the IPv6 tcp connect setup takes longer with IPv6 on this local connection.

I added code to measure resolving time when testing it on my laptop, which now shows shorter times for IPv6 than IPv4, one wireless hop away from the same system.

$ ./time-gai-connect.py idefix.net
Resolving                                 3.20196 ms
2001:980:14ca:42::18                      0.96989 ms
82.95.196.202                             1.49107 ms
Measuring this gives more new questions than answers... it seems remote destinations are faster for IPv6 than for IPv4. Example from the same laptop:
$ ./time-gai-connect.py www.xs4all.nl
Resolving                                34.10101 ms
2001:888:0:18::80                        15.76281 ms
194.109.6.92                             15.84506 ms
And from the system running idefix.net:
$ ./time-gai-connect.py www.xs4all.nl
Resolving                                 2.60282 ms
2001:888:0:18::80                        14.32490 ms
194.109.6.92                             15.16104 ms

But anyway, iperf and time-gai-connect will give me the answers I want to measure.


Tags: , , ,
2012-08-17 (#) 6 years ago
At my current work I am also introducing zabbix monitoring. I chose zabbix at my previous work because I like the approach: measure a lot of values and store those, and next you decide whether to draw graphs or run triggers based on those values. Monitoring, graphing and alerting in one system.

The installation of the zabbix agent got puppetized instantly. I found out the rpm from epel leaves a few things to fix, so puppet to the rescue to fix that on installation. By simply configuring those fixes to depend on the package and to notify the service the start of the service will be postponed until those fixes have been done and the agent will start correctly.

Firewall on the monitored machines still needs to be fixed by hand, this is still a problem. Bringing the firewall under puppet control would be great, but that is quite a project.

Tags: , , ,
⇐ Newer news items for tag statistics  Older news items for tag statistics ⇒
, reachable as koos+website@idefix.net. PGP encrypted e-mail preferred.

PGP key 5BA9 368B E6F3 34E4 local copy PGP key 5BA9 368B E6F3 34E4 via keyservers pgp key statistics for 0x5BA9368BE6F334E4 Koos van den Hout
RSS
Other webprojects: Camp Wireless, wireless Internet access at campsites, The Virtual Bookcase, book reviews