News items for tag linux - Koos van den Hout

2022-11-03 It seems the rcu_sched messages stopped after I reseated SATA cables
In the beginning of October I shut down the home server conway and reseated the SATA cables in the hopes of having less problems with timeouts. And started the whole system again to also fix other problems.

About a month later I think this worked, I've never seen a rcu_sched message again since doing that reseating.

Tags: , ,
2022-10-12 Peeking a bit at Kea DHCP server
Yesterday I learned that ISC DHCP server will be end of life at the end of this year. For a package I started using around 1998 with one of the first versions I expected a bit more announcement time. At the same time I'm so used to using ISC dhcp server in my home network I never subscribed to any mailing list or other announcements about ISC dhcp server, it's just there, I can configure it to do what I want including supporting pxe booting systems for installation or diagnostics or supporting special dhcp options for APC AP7920 rackmount power distribution units. And all the virtual lans of my home network.

ISC suggests using Kea DHCP server to replace it in most server implementations. Kea DHCP server should be able to get a lot of configuration data from databases and allow for dynamic updates of the configuration. That is an improvement over ISC dhcp as it is at the moment, which needs a full restart for every change.

So time to peek at Kea DHCP server. I don't think ISC dhcp server will be unavailable after 31 December 2022 but I don't expect updates anymore and when a good replacement is normalized I expect ISC dhcp server to slowly fall away from linux distributions.

Currently it's not even available for Debian or Devuan stable or oldstable strangely enough. I wonder what happened there. But there are distribution packages for debian buster at Cloudsmith - Repositories - ISC - Internet Systems Consortium (isc) - kea-2-3 (kea-2-3) - Packages / format:deb.

Time to install the latest and let apt fix the dependencies:
koos@testrouter:~$ sudo dpkg -i isc-kea-dhcp4_2.3.1-isc20220928105532_amd64.deb isc-kea-dhcp6_2.3.1-isc20220928105532_amd64.deb isc-kea-common_2.3.1-isc20220928105532_amd64.deb 
Selecting previously unselected package isc-kea-dhcp4.
(Reading database ... 46609 files and directories currently installed.)
Preparing to unpack isc-kea-dhcp4_2.3.1-isc20220928105532_amd64.deb ...
Unpacking isc-kea-dhcp4 (2.3.1-isc20220928105532) ...
Selecting previously unselected package isc-kea-dhcp6.
Preparing to unpack isc-kea-dhcp6_2.3.1-isc20220928105532_amd64.deb ...
Unpacking isc-kea-dhcp6 (2.3.1-isc20220928105532) ...
Selecting previously unselected package isc-kea-common.
Preparing to unpack isc-kea-common_2.3.1-isc20220928105532_amd64.deb ...
Unpacking isc-kea-common (2.3.1-isc20220928105532) ...
dpkg: dependency problems prevent configuration of isc-kea-dhcp4:
 isc-kea-dhcp4 depends on libboost-system1.67.0; however:
  Package libboost-system1.67.0 is not installed.
koos@testrouter:~$ sudo apt install -f
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  libboost-system1.67.0 liblog4cplus-1.1-9 libmariadb3 libpq5 mariadb-common
The following NEW packages will be installed:
  libboost-system1.67.0 liblog4cplus-1.1-9 libmariadb3 libpq5 mariadb-common
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
3 not fully installed or removed.
Need to get 760 kB of archives.
After this operation, 4,001 kB of additional disk space will be used.
Looking at the sample configuration makes me think I can do this with a text-based configuration (it's actually JSON) and get it going fast. For my home network that is probably the best solution. Kea does have options to use MariaDB or PostgreSQL backends for storage which does look really nice for my home network but at the same time adds a dependency and a layer of complexity.

I can see IPAM systems totally going to Kea DHCP and give a full interface on managing the databases directly including APIs for adding/removing objects as they are added in other systems.

Tags: , , ,
2022-10-09 I moved the 1-wire interface to a Raspberry Pi
After the problems with detaching and attaching the USB 1-wire interface from a kvm virtual machine to fix an interference issue showed up again I decided to move the USB 1-wire interface to a different machine, one where kvm virtualisation isn't in the mix. The closest available machine that can deal with the 1-wire interface is a Raspberry Pi which also has other monitoring tasks.

This move worked fine and the 1-wire temperatures are showing up again in influxdb. I decided not to update the rrdtool temperature database. I will have to find time to migrate the rrdtool history to influxdb. Ideally there will be some aggregation for older measurements but I'd like an "infinite" archive of a daily average.

Tags: , , , ,
2022-09-24 Can't live-attach a USB device to a kvm virtual host after upgrades
I have a DS2490 USB 1-wire interface on the home server conway which is rerouted to one of the virtual machines so that that virtual machine can read the sensors on the 1-wire network. This rerouting works when the machine is started, the DS2490 USB 1-wire shows up in the virtual machine fine. From time to time this DS2490 USB 1-wire interface gets confused when I am transmitting on the radio so the solution is to detach it from the virtual machine, unplug it from the server, plug it in again and attach it to the virtual machine again. Today this had to be done and I got an unexpected error message:
root@conway:~# virsh attach-device --live gosper /etc/onewire-for-gosper.xml
error: Failed to attach device from /etc/onewire-for-gosper.xml
error: internal error: unable to execute QEMU command 'device_add': failed to find host usb device 2:8
In logfile /var/log/libvirt/libvirtd.log:
2022-09-24 21:16:38.655+0000: 10923: error : qemuMonitorJSONCheckError:395 : internal error: unable to execute QEMU command 'device_add': failed to find host usb device 2:8
To be complete about it: usb device 2:8 is exactly the right one!
root@conway:~# lsusb | grep 2490
Bus 002 Device 008: ID 04fa:2490 Dallas Semiconductor DS1490F 2-in-1 Fob, 1-Wire adapter
This seems to be new since I upgraded the homeserver to Devuan beowulf giving me versions:
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                  Version         Architecture Descripti
ii  libvirt-clients                       5.0.0-4+deb10u1 amd64        Programs 
ii  libvirt-daemon                        5.0.0-4+deb10u1 amd64        Virtualiz
un  libvirt-daemon-driver-storage-gluster                  (no descr
un  libvirt-daemon-driver-storage-rbd                      (no descr
un  libvirt-daemon-driver-storage-zfs                      (no descr
ii  libvirt-daemon-system                 5.0.0-4+deb10u1 amd64        Libvirt d
ii  libvirt-glib-1.0-0:amd64              1.0.0-1         amd64        libvirt G
ii  libvirt0:amd64                        5.0.0-4+deb10u1 amd64        library f

First idea: AppArmor

The first search result that came up was Bug #1552241 “libvirt-bin apparmor settings for usb host device” : Bugs : libvirt package : Ubuntu. So I tried changing the /etc/apparmor.d/abstractions/libvirt-qemu file. After a few tries and reading the warnings in the rest of the file I made sure the source was AppArmor by completely disabling it. The error did not go away so I reverted the libvirt-qemu rules to the original settings, restarted AppArmor and kept debugging.

Second idea: usb rights

Based on QEMU USB passthrough broken after Ubuntu 18.04 upgrade I added udev rules to make sure group libvirt-qemu had read and write rights on the usb device, with /lib/udev/rules.d/51-qemu-usb-passthrough.rules containing:
SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", ATTRS{idVendor}=="04fa", ATTRS{idProduct}=="2490", MODE="0664", GROUP="libvirt-qemu"
And doing the
root@conway:~# udevadm control --reload-rules
And verifying the resulting rule:
root@conway:~# udevadm test -a -p  $(udevadm info -q path -n /dev/bus/usb/002/008)
calling: test
version 3.2.9
This program is for debugging only, it does not run any program
specified by a RUN key. It may show incorrect results, because
some values may be different, or not available at a simulation run.


GROUP 110 /lib/udev/rules.d/51-qemu-usb-passthrough.rules:1
MODE 0664 /lib/udev/rules.d/51-qemu-usb-passthrough.rules:1
handling device node '/dev/bus/usb/002/008', devnum=c189:135, mode=0664, uid=0, gid=110
Indeed the right groupid, but still the same error message when trying the attach-device command.

Interesting find: it's specific to the virtual machine that had the device before

Small update: I can attach the USB device to a different host and detach it from that host again. I just can't attach it to the 'original' host again.

I also posted this question on serverfault: Can't live-attach a USB device to a kvm virtual host again after upgrades.

Update: After a complete reboot of the homeserver the USB 1-wire interface worked again (as I could imagine). But after another interference problem it's now in the same state again. I did change the definition in both the virthost configuration and the xml file from managed='no' to managed='yes' before the reboot but that hasn't helped. Contents of the /etc/onewire-for-gosper.xml file now:
    <hostdev mode='subsystem' type='usb' managed='yes'>
        <vendor id='0x04fa'/>
        <product id='0x2490'/>

Tags: , , ,
2022-08-28 Maintenance for the pi4raz igate / learning about esp32 power requirements
Since last Thursday the aprs server at is down. I used that aprs server for the weather station and for the igate.

The change for the weather station was one word in a script, for the igate I had to remember how to change this with the Arduino development environment set up to support the esp32 board. The easiest way seemed to be from the computer, but every time after the igate started the running process after the setup it crashed and rebooted itself. I spent a lot of time looking for the answers, added debug statements all over the code and ended up in the WiFi initialization code as the place of crashing. And that was the hint, according to Crash when trying to connect to wifi - Issue #3935 - espressif/arduino-esp32 this is a sign of a power shortage.

This is purely my fault: the pi4raz igate design calls for an external power supply feeding it.

The solution was to go back to the separate USB power supply and not use a USB hub connected to the computer. Now the igate is started again and visible on the APRS network: track PE4KH-10 on

Tags: , ,
2022-07-07 Upgraded the homeserver OS to devuan beowulf and replaced the UPS battery
A few days ago I noticed some interesting messages in the apcupsd log:
2022-07-04 10:14:15 +0200  Battery disconnected.
2022-07-04 10:16:24 +0200  Battery reattached.
2022-07-04 10:19:53 +0200  Battery disconnected.
2022-07-04 10:20:40 +0200  Battery reattached.
Checking the UPS statistics showed me the battery charge was dropping to about 7 % of the capacity while the mains power was available. Since the battery was over 5 years old I ordered a new one to replace it.

This battery was scheduled to arrive Wednesday at the start of the afternoon and I wanted to do an upgrade of the Linux distribution on the main homeserver conway anyway because devuan ascii is already 'oldoldstable' (but still getting updates).

The homeserver uses 2 disks with the main lvm volume in a raid-1. The /boot and /boot/efi filesystems are mirrored by hand with the idea to end with a working boot even when 1 disk is missing.

After the shutdown and replacing the UPS battery I switched the server on again and I was greeted by a grub prompt and nothing to boot. After a few tries I got the system booting again, after that I went searching for what went wrong. Eventually I found out the file /boot/efi/EFI/devuan/grub.cfg pointed at a missing filesystem. I found out the best way to fix this is with
# dpkg-reconfigure grub-efi-amd64
both with /dev/sda and /dev/sdb filesystems on /boot and /boot/efi.
Read the rest of Upgraded the homeserver OS to devuan beowulf and replaced the UPS battery

Tags: , ,
2022-06-05 Having multiple wsjt-x instances available from CQRLOG
I'm currently also doing some contacts with a special event station call and I wanted to separate the wsjt-x history for my normal call from the history for the special event station call, just like I split the log databases in CQRLOG.

For the non-amateurradio persons: I have my own callsign, PE4KH which is linked to me. It is also possible to have one extra temporary callsign. Those are usually linked to an event or some other reason for a 'special' callsign. Temporary callsigns in the Netherlands have either the digit 6 or more than one digit.

There is an option for multiple profiles in wsjt-x but those are just for the settings (including callsign) but not for the logging location. This means all different profiles share the same history and will show the same countries as 'new' or 'already contacted'.

When I was looking at the options for starting wsjt-x with different settings I noticed the -r --rig-name <rig-name> Where is for multi-instance support. option in the help. With this option, all the logging is in ~/.local/share/WSJT-X - <rig-name>/ which is what I want.

The next challenge is to start wsjt-x with the extra commandline paramater from CQRLOG. It seems the 'path to wsjt-x' setting doesn't accept commandline parameters. So I created a script ~/bin/ses-wsjtx with:

/usr/bin/wsjtx -r ses
Changed the 'path to wsjt-x' setting to /home/koos/bin/ses-wsjtx and now I get what I want.

Tags: , ,
2022-03-18 Using grafana for alerting too
I've been playing with grafana for about a year since starting with updating my statistics gathering and I keep seeing new options and updates in grafana.

Grafana recently got some new options for alerting and I am trying a few of those. Alerts for things that are a real problem and can cause other problems are a good start. Based on some earlier problems I keep an eye on some filesystems that are over 90% full.

Today I read Three DDoS attacks on my personal website found via Three DDoS attacks on my personal website : r/homelab reddit and this made me wonder about overloads on my webserver. The easiest way to detect problems with web serving I could think of is to look at the queue size in haproxy which is monitored in influxdb/grafana anyway for nice graphs of website traffic.

I did have a time with too high queues for backend webservers. But that was when the backend server was completely broken due to a filesystem problem so that was a logical reason.

It would be nice if I could iterate alerts, like 'for the root filesystem of every monitored system'. Or at least copy them changing only the system name in the rules and alerts.

Tags: ,
2022-03-10 Dear linux kernel, I know what I want with nomodeset
Just noted on bootup of a virtual machine:
Mar 10 19:42:14 turing kernel: [    0.181861] You have booted with nomodeset. This means your GPU drivers are DISABLED
Mar 10 19:42:14 turing kernel: [    0.181862] Any video related functionality will be severely degraded, and you may not even be able to suspend the system properly
Mar 10 19:42:14 turing kernel: [    0.181862] Unless you actually understand what nomodeset does, you should reboot without enabling it
It's a virtual machine which does server tasks. Anything more than 80x25 VGA text mode is pure overkill. It's currently the default card in qemu (Cirrus CLGD 5446 PCI VGA card), I could try the virtio VGA card to see if that saves on memory/cpu.

Tags: , ,
2022-02-23 Filtering logs to only get relevant reports
I want to know if something goes wrong but with the number of (virtual) servers here at home it is not possible to check all logs constantly. So the main machines use logcheck to find the interesting error messages and the rest gets filtered out.

Ideally that leaves no messages, but I do want to know about patterns that indicate attacks so I do get messages constantly about ssh attack attempts and weird nameserver requests or misconfigured nameserver responses.

Recently I've been checking the resulting reports again carefully and noticed some more patterns that could be filtered. And I found two misconfigurations that I solved. Normally those misconfigurations would drown in the noise of the log, only to be found if I was looking for something else. Now it started to stand out after filtering out a lot of messages that are to be expected.

Tags: , ,

IPv6 check

Running test...
, reachable as PGP encrypted e-mail preferred. PGP key 5BA9 368B E6F3 34E4 local copy PGP key 5BA9 368B E6F3 34E4 via keyservers

Meningen zijn die van mezelf, wat ik schrijf is beschermd door auteursrecht. Sommige publicaties bevatten een expliciete vermelding dat ze ongevraagd gedeeld mogen worden.
My opinions are my own, what I write is protected by copyrights. Some publications contain an explicit license statement which allows sharing without asking permission.
Other webprojects: Camp Wireless, wireless Internet access at campsites, The Virtual Bookcase, book reviews
This page generated by $Id: newstag.cgi,v 1.39 2022/11/18 15:23:48 koos Exp $ in 0.043686 seconds.