News items for tag statistics - Koos van den Hout

2020-09-02 An update to the home 1-wire network 1 month ago
For more than 12 years now(!) the house has temperature sensors using the 1-wire protocol. I recently redid some of the wiring between floors and I finally got around to rerouting the 1-wire network via this new route.

I also added a temperature sensor in the big room in the attic, we are thinking of using that room more often. To get an idea of how good that idea is we wanted to get an idea of the temperatures up there and that's what I have 1-wire sensors for! I soldered an 18b20 sensor to the end of a 4-wire flat phone cable, added it to the network and it's measuring.

So now 12 environmental temperatures are measured every 5 minutes: 9 in the house, one in the weather hut, one in the shed and one on the roof of the shed.

I also updated the 1-wire projects overview with how I use 4-wire flat phone cable in RJ45 connectors for 1-wire network. I had to look up how I did that previously before I could start adding new cables!

Tags: , , ,
2020-06-24 Moved the shed roof temperature sensor from under the roof to above the roof 4 months ago
I wanted to do some things in the shed with cabling, some things were not ideal after the solar power installation.

One of the things was that I had a temperature sensor to measure the temperature above the roof, but with all things that happened with cables it ended up hanging below the roof. I moved it back up in the ventilation pipe and with the current temperatures it started measuring 4 degrees Celcius higher immediately.

Tags: , ,
2020-06-15 A new home timeserver on order 4 months ago
After earlier tries to have a nice GPS-based timeserver for my home network I noticed a simple but very effective GPS 'hat' for the Raspberry Pi, the Raspberry Pi GPS Hat from Uputronix. While the Pi's are already taking over the home network just one more could be a nice addition. In the longer run this will probably replace the shed computer.

So I ordered a Pi with an added dual-band WiFi adapter, a case, the GPS hat and a GPS antenna. The GPS hat has PPS support so I will get the time correct. With the instructions from 5 minute guide to making a GPS Locked Stratum 1 NTP Server with a Raspberry Pi it should be easy. If this all works I may even add the resulting Pi to the IPv6 NTP Pool.

Update 2020-06-16: SOS Solutions came back with some bad news: the uputronix Pi GPS Hat isn't available anymore. I'm now looking at the comparable adafruit hardware which is somewhat more expensive, but offers the same options.

Update 2020-06-18: And the adafruit hardware is also not available soon. I cancelled the GPS unit part of the order and I'm looking at sourcing a GPS module for the Pi from another source. The GPS hat which sossolutions no longer sells is originally from uputronics where a newer version of the Raspberry Pi GPS/RTC Expansion Board is listed as available on the site. Based on a ublox chipset which allows me access to a lot of the GPS data.

Tags: , , ,
2020-05-01 Time for a plot of the number of contacts in amateur radio after a busy month 5 months ago
QSO count plot up to April 2020 Time for a new plot of the number of radio contacts. As usual contest months are quite visible and January is for me the month with the most contests.

But April 2020 is also quite visible. This last week I had a lot of time for radio due to holiday and not going anywhere. And other radio amateurs also had the time to be active, so there were a lot of new calls to get in the log. Combined with a good 10 meter band opening this added to a high number of contacts for a month with only one contest.

Tags: , ,
2020-02-16 Reconsidering data gathering and processing 8 months ago
For years and years I've been doing a lot of data gathering and storing the data using rrdtool. Data such as temperatures from lots of places, from mainboard CPU sensors to an outside weather station, other weather data, web traffic data, house electricy and gas usage, solar power. I started doing this with mrtg in 2001 and switched to rrdtool.

There are some improvements to this system, such as maintaining the rrd files on one machine and doing measurements on other machines in the form of timestamped files to be transported to the machine with rrd via rsync-over-ssh. This allows the central database to do a catch-up of decentrally gathered data after an interruption.

All in all there are two disadvantages at the moment: the system isn't very flexible, adding a datasource means making the big decision about how much data to keep how long and what I want to look at.

Diskspace isn't as constrained as it once was, I may want to keep some data forever and I may want to zoom in to a period a bit longer ago. So I'm looking at different solutions. For one dataset I already added an alternate datastore: the electricity and gas meter readings get copied to a postgres database once a day so I can look at the daily readings forever.

So the search is on for the ideal solution. For gathering and transporting data I am looking at mqtt, a lightweight protocol to gather and transmit data. This also makes it easier to have multiple data collectors look at one source so I can test with a few things first before I make a real switch to any new system.

Tags: ,
2020-01-01 Closing 2019 in amateur radio, time to plot the number of contacts and look back 9 months ago
QSO count plot up to December 2019 Time for a new plot of the number of radio contacts. Months with contests are quite visible. After the peak in number of contacts in July there was first a holday and after that no big peaks in number of contacts. December 2019 jumps out a bit again due to the FT8 roundup on 8/9 December in which I made 66 contacts and later in the month the troposperic ducting allowing contacts over interesting distances in the 70 centimeter and 2 meter band added to a sprint at the end. In 2019 I made a few more contacts than in the previous record year 2017.

Looking back at my amateur radio resolutions for 2019 I think most came true.

If I look at them one by one:
  • Keep learning morse! - I'm still working on my morse, but there is measurable improvement. I have learned the full set for the Belgian CW exam and I'm working on accuracy and speed.
  • Get more countries on more HF bands in the log - More countries and more slots on HF are in the log. I also use the club station to achieve that goal. The ARRL DXCC Award shows that I'm getting somewhere.
  • Moonbounce on 2 meter - I've listened on the right frequencies to the moon on 2 meter. Nothing heard.
  • Those digimode contests, and maybe a few phone contests - I participated in two phone contests and a number of digimode contests. No serious improvement in scores.
  • Operate HF outside - I operated HF outside. Not as much as I would like.
  • At least one satellite contact - Multiple satellite contacts have been made!
Now I have to think about 2020, but the year is still young.

Tags: , , , , ,
2019-08-05 Time for a new plot of the number of radio contacts 1 year ago
QSO count plot up to July 2019 Time for a new plot of the number of radio contacts. Months with contest(s) stand out again as they elevate the number of contacts. In July 2019 I participated in the DL-DX RTTY Contest 2019 and the IARU-HF Championship 2019. That last one has added a few countries to my list of countries confirmed in phone modes.

Tags: , ,
2019-07-25 First onewire stats ageing out 1 year ago
I was looking at some onewire temperature stats and noticed the first stats being aged out. I started monitoring temperatures with 1-wire sensors in January 2007 using rrdtool. I set up round robin archives with an expiry in 11 years, and those 11 years have passed now for the first measurements.

Tags: , , ,
2019-07-05 I tested the randomness setup 1 year ago
Doing some more reading on haveged made me decide to test the actual randomness of my setup with haveged and randomsound which I created to fix the lack of entropy for dnssec signing operations so I booted the same testing virtual machine which can tap from the host /dev/random. I ran rngtest until it was time to shut down the laptop which was showing the output. The result:
$ rngtest < /dev/random 
rngtest 2-unofficial-mt.14
Copyright (c) 2004 by Henrique de Moraes Holschuh
This is free software; see the source for copying conditions.  There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

rngtest: starting FIPS tests...
^Crngtest: bits received from input: 4999640
rngtest: FIPS 140-2 successes: 249
rngtest: FIPS 140-2 failures: 0
rngtest: FIPS 140-2(2001-10-10) Monobit: 0
rngtest: FIPS 140-2(2001-10-10) Poker: 0
rngtest: FIPS 140-2(2001-10-10) Runs: 0
rngtest: FIPS 140-2(2001-10-10) Long run: 0
rngtest: FIPS 140-2(2001-10-10) Continuous run: 0
rngtest: input channel speed: (min=303.011; avg=543.701; max=5684.774)bits/s
rngtest: FIPS tests speed: (min=43.251; avg=64.587; max=84.771)Mibits/s
rngtest: Program run time: 9194254192 microseconds
I ratelimited the virtio-rng-pci driver from the host, so the test took a really long time. Given earlier tries with dnssec-signzone this is fast enough.

No need to buy a hardware random generator, although they are way cool and it would be an idea to have a source of correctness (NTP) next to a source of randomness.

Update: I ran rngtest on /dev/urandom and I had to ask for a really big load of blocks to get failures. The first test with 249 blocks gave the same result as above, just a lot higher bit rate. So now I know less about the correct randomness of my setup but at least the test shows that I can safely run dnssec-signzone which was the original idea.

Tags: , , ,
2019-04-01 Plotting the number of radio contacts after varying months 1 year ago
QSO count plot up to March 2019 After a month with a holiday and a month with one contest I redid the QSO count plot to see the development.

before, before, before, before, before

Tags: , ,
2019-04-01 Wat volts extra 1 year ago
UPS invoer voltage laatste jaar Het viel me op in de grafieken van het invoer voltage volgens de UPS dat het voltage vanaf het stroomnet is gestegen tot 238 volt aan het eind van september 2018. Ik vraag me af wat de oorzaak is van deze wijziging. Het kan niet zijn door de toename van zonnepanelen in de omgeving, het gestegen voltage is zowel overdag als 's nachts.

Tags: , ,
2019-02-27 Rare verandering in VDSL upstream snelheid 1 year ago
VDSL upstream snelheid week Ineens is de haalbare VDSL upstream snelheid (engels 'attainable') gezakt naar wat daarvoor ongeveer de huidige VDSL upstream snelheid (engels 'current') was. Een opvallende hik in de grafieken. Ik heb geen idee wat de aanleiding is en of dit weer kan veranderen.

Dit is alleen te zien in het modem, de hele PPP sessie is gewoon in stand gebleven.

Tags: , ,
2019-02-01 Plotting the number of amateur radio contacts after a contesting month 1 year ago
QSO count plot up to January 2019 After a month with three digimode radio contests I plotted the number of amateur radio contacts again. The number of contacts is clearly higher each January as a contest month, with this January a new peak.

The contests were the ARRL RTTY Roundup on 6 and 7 January, the UBA PSK63 prefix contest on 12 and 13 January and the BARTG RTTY Sprint Contest on 26 and 27 January.

Nicer looking font due to the upgrade of "radio workstation" thompson. I guess even gnuplot is coming along with the modern times.

before, before, before, before

Tags: , , ,
2019-01-01 Switching to 1-wire over USB and forwarding a USB device to a guest VM 1 year ago
The new hardware for the homeserver has no external serial ports, so I could not use the old serial / 1-wire interface that has been doing the home monitoring for years. But I had a spare USB DS2490 interface. So I plugged this into the server and wanted to forward the USB device to the guest VM that runs all the monitoring.

First I had to blacklist all the loaded drivers to have the device available to kvm as-is. In /etc/modprobe.d/local-config.conf:
blacklist w1_smem
blacklist ds2490
blacklist wire
Next step was to attach the device to the right vm. I followed the hints at How to auto-hotplug usb devices to libvirt VMs (Update 1) and edited the definition for the vm to get the host device like:
    <hostdev mode='subsystem' type='usb' managed='no'>
        <vendor id='0x04fa'/>
        <product id='0x2490'/>
But that did not get the usb device attached to the running VM and I did not feel like rebooting it. So I created an extra file with the above and did a
root@conway:~# virsh attach-device --live gosper /tmp/onewire.xml 
Device attached successfully
And then I had to do the same blacklisting as above in the virtual machine. After doing that I detached and attached it from the VM without touching it with simply:
root@conway:~# virsh detach-device --live gosper /tmp/onewire.xml 
Device detached successfully

root@conway:~# virsh attach-device --live gosper /tmp/onewire.xml 
Device attached successfully
After that I had to set up rules for the telemetry user to have enough access to the USB device:
SUBSYSTEMS=="usb", GOTO="usb_w1_start"
ATTRS{idVendor}=="04fa", ATTRS{idProduct}=="2490", GROUP="telemetry", MODE="0666"
And now it all works:
telemetry@gosper:~$ digitemp_DS2490 -a
DigiTemp v3.7.1 Copyright 1996-2015 by Brian C. Lane
GNU General Public License v2.0 -
Found DS2490 device #1 at 002/003
Jan 01 21:53:11 Sensor 10A8B16B0108005D C: 9.500000
Jan 01 21:53:12 Sensor 28627F560200002F C: 17.062500
Jan 01 21:53:14 Sensor 10BC428A010800F4 C: 19.562500
Jan 01 21:53:15 Sensor 1011756B010800F1 C: 11.937500
Jan 01 21:53:16 Sensor 10B59F6B01080016 C: 16.312500
Jan 01 21:53:17 Sensor 1073B06B010800AC C: 18.687500
Jan 01 21:53:18 Sensor 102B2E8A010800F0 C: 29.250000
Jan 01 21:53:20 Sensor 28EF71560200002D C: 16.687500
Working house temperatures again!

Tags: , , , ,
2018-11-05 De gevolgen van DSL interleave 1 year ago
Ping stats Gisterenavond dus toegekomen aan het vervangen van het vdsl modem na de eerdere storing en ineens viel me wat op in de ping stats: tijdens het gebruik van de fritzbox was de round trip tijd lager en stabieler, de uitschieters naar boven komen dus puur van de vdsl laag. De fritzbox zet de interleave uit (fast/fast), de huidige modemdriver in de vigor op fast down en interleaved up.

Misschien toch eens een andere vdsl driver proberen op de vigor.
Read the rest of De gevolgen van DSL interleave

Tags: , ,
2018-10-12 Serious slowness with rrdgraph from rrdtool 2 years ago
One of the things still needing migrating is the NTP server stats which obviously uses rrdtool. Because I want to keep the history I migrated the datasets with:
/usr/local/rrdtool/bin/rrdtool dump \
| ssh newhost /usr/bin/rrdtool restore -f -
And then create a graph of the plloffset for example using:
/usr/bin/rrdtool graph /tmp/ \
--title " pll offset (last 24 hours)" --imginfo \
'<img src="tmpgraphs/%s" WIDTH="%lu" HEIGHT="%lu" alt="Graph">' \
--start -24hours --end now --vertical-label="Seconds" --color BACK#0000FF \
--color CANVAS#c0e5ff --color FONT#ffffff --color GRID#ffffff \
--color MGRID#ffffff --alt-autoscale --imgformat PNG --lazy \ \
CDEF:wipeout=offset,UN,INF,UNKN,IF CDEF:wipeoutn=wipeout,-1,* \
LINE1:offset#000000:"Offset\:" \
GPRINT:offset:LAST:"Current\:%.3lf%s" \
GPRINT:offset:MIN:"Min\:%.3lf%S" \
GPRINT:offset:MAX:"Max\:%.3lf%S" \
GPRINT:offset:AVERAGE:"Average\:%.3lf%S" \
AREA:wipeout#e0e0e0 AREA:wipeoutn#e0e0e0
But on the old server this takes 0.026 seconds, on the new server 3 minutes and 47.46 seconds. No idea what is happening, strace shows nothing strange and rrdtool uses 1 cpu at 100% all that time.
Read the rest of Serious slowness with rrdgraph from rrdtool

Tags: , , ,
2018-09-07 Plotting the number of amateur radio contacts 2 years ago
QSL count plot up to August 2018 After the SCC RTTY contest in August I decided to plot the number of amateur radio contacts again. Clearly visible are months with contests I participate in. And the influence of the summer holiday.

before, before, before

Tags: , ,
2018-09-06 Weird interface names in snmp due to virtio driver 2 years ago
I want to measure network traffic so I decided to copy most of my rrdtool setup from the old home server.

But with virtio network cards I have a confused snmpd:
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: Red Hat, Inc Device 0001
IF-MIB::ifDescr.3 = STRING: Red Hat, Inc Device 0001
IF-MIB::ifDescr.4 = STRING: Red Hat, Inc Device 0001
IF-MIB::ifDescr.5 = STRING: dummy0
IF-MIB::ifDescr.6 = STRING: dumhost
IF-MIB::ifDescr.7 = STRING: dumdh6
Fix: go for the IF-MIB::ifName snmp variables, found in oid
IF-MIB::ifName.1 = STRING: lo
IF-MIB::ifName.2 = STRING: eth0
IF-MIB::ifName.3 = STRING: eth1
IF-MIB::ifName.4 = STRING: eth2
IF-MIB::ifName.5 = STRING: dummy0
IF-MIB::ifName.6 = STRING: dumhost
IF-MIB::ifName.7 = STRING: dumdh6
Those are easier to discern, now my snmp scripts are gathering data again.

Tags: , , ,
2018-04-26 More radio maps and their limits 2 years ago
The PE4KH website has maps of the locations where I contacted radio amateurs all over the world. The maps with generated images are created by exporting my locators worked/confirmed from cqrlog and using gcmwin for linux with a whole set of different configurations to plot the results.

But now the 2 meter band has been added and on that band a 'record distance' is not as far as on HF. On HF my current distance record was a contact with Australia at 16581 kilometers. My current distance record on the 2 meter band is 363 kilometer in Germany. Quite a different scale!

So the maps part has been enhanced with a 2 meter contacts map, but gcmwin can't use 6-position maidenhead locators so the map is quite coarse compared to what I want. I don't know the solution at the moment to improve this. The recent qso map PE4KH does show the more precise gridsquares when available in the log, so maybe that page needs distances added.
Read the rest of More radio maps and their limits

Tags: , ,
2018-03-01 An interesting bug in age-old Perl Net::SNMP code 2 years ago
I recently noticed the network traffic statistics weren't updated correctly for the LAN interface of my Draytek Vigor 130 modem. These statistics were extracted using code that I originally started using at the computer science systems group somewhere in the previous decade. It's all Perl Net::SNMP and not very efficient. I don't know if I wrote it myself or copied from somewhere else, I do know a new bug was introduced.

To understand the code it is important to realize that interface index numbers in SNMP are dynamic. Across a reboot a certain number can change. Interface names are static, but those are never used directly in SNMP.

So to get from a static interface name to a dynamic interface index the interfaces.2.1.2 subtree (ifDescr) has to be fetched from the device and checked for the right names. To get the interface index from an snmp object identifier I used to use this bit of code:
# find the current interface indices for the wanted ^ interfaces
foreach my $oid (oid_lex_sort(keys(%table))) {
    if (oid_base_match($ifTable_ifDesc,$oid)){
#        printf("%s => %s\n", $oid, $table{$oid});
        if (defined $wantstuff{$table{$oid}}){
            # I am lazy. I fill a hash with the interface indices so I can
            # use it for lookups
        #    printf "Found ifindex %d for %s\n",$wantstuff{$table{$oid}}{ifindex},$table{$oid};
But note how the current ifDesc subtree is from the modem:
IF-MIB::ifDescr.1 = STRING: LAN
IF-MIB::ifDescr.4 = STRING: VDSL
IF-MIB::ifDescr.5 = STRING: Resrved
IF-MIB::ifDescr.6 = STRING: 
IF-MIB::ifDescr.7 = STRING: 
IF-MIB::ifDescr.8 = STRING: 
IF-MIB::ifDescr.20.101.1 = STRING: WAN1
IF-MIB::ifDescr.21.101.1 = STRING: WAN2
IF-MIB::ifDescr.22.101.1 = STRING: LAN_PORT1
Using that rindex function there are 4 instances of index 1. Which caused the very similar code looking for the ifInOctets, ifOutOctets and other counters to overwrite the result for index 1 with those from WAN1, WAN2 and LAN_PORT1.

So that code is now improved, no more rindex but a well-defined use of length:
# find the current interface indices for the wanted ^ interfaces
foreach my $oid (oid_lex_sort(keys(%table))) {
    if (oid_base_match($ifTable_ifDesc,$oid)){
        #printf("%s => %s\n", $oid, $table{$oid});
        if (defined $wantstuff{$table{$oid}}){
                        my $intindex=substr($oid,length($ifTable_ifDesc)+1);
                        #printf "Submatch found ifindex %d for %s\n",$intindex,$table{$oid};
            # I am lazy. I fill a hash with the interface indices so I can
            # use it for lookups
            #printf "Found ifindex %d for %s\n",$wantstuff{$table{$oid}}{ifindex},$table{$oid};

Tags: , ,
  Older news items for tag statistics ⇒
, reachable as PGP encrypted e-mail preferred.

PGP key 5BA9 368B E6F3 34E4 local copy PGP key 5BA9 368B E6F3 34E4 via keyservers pgp key statistics for 0x5BA9368BE6F334E4 Koos van den Hout
Other webprojects: Camp Wireless, wireless Internet access at campsites, The Virtual Bookcase, book reviews