News items for tag linux - Koos van den Hout

Yesterday I learned that ISC DHCP server will be end of life at the end of this year. For a package I started using around 1998 with one of the first versions I expected a bit more announcement time. At the same time I'm so used to using ISC dhcp server in my home network I never subscribed to any mailing list or other announcements about ISC dhcp server, it's just there, I can configure it to do what I want including supporting pxe booting systems for installation or diagnostics or supporting special dhcp options for APC AP7920 rackmount power distribution units. And all the virtual lans of my home network.

ISC suggests using Kea DHCP server to replace it in most server implementations. Kea DHCP server should be able to get a lot of configuration data from databases and allow for dynamic updates of the configuration. That is an improvement over ISC dhcp as it is at the moment, which needs a full restart for every change.

So time to peek at Kea DHCP server. I don't think ISC dhcp server will be unavailable after 31 December 2022 but I don't expect updates anymore and when a good replacement is normalized I expect ISC dhcp server to slowly fall away from linux distributions.

Currently it's not even available for Debian or Devuan stable or oldstable strangely enough. I wonder what happened there. But there are distribution packages for debian buster at Cloudsmith - Repositories - ISC - Internet Systems Consortium (isc) - kea-2-3 (kea-2-3) - Packages / format:deb.

Time to install the latest and let apt fix the dependencies:

koos@testrouter:~$ sudo dpkg -i isc-kea-dhcp4_2.3.1-isc20220928105532_amd64.deb isc-kea-dhcp6_2.3.1-isc20220928105532_amd64.deb isc-kea-common_2.3.1-isc20220928105532_amd64.deb 
Selecting previously unselected package isc-kea-dhcp4.
(Reading database ... 46609 files and directories currently installed.)
Preparing to unpack isc-kea-dhcp4_2.3.1-isc20220928105532_amd64.deb ...
Unpacking isc-kea-dhcp4 (2.3.1-isc20220928105532) ...
Selecting previously unselected package isc-kea-dhcp6.
Preparing to unpack isc-kea-dhcp6_2.3.1-isc20220928105532_amd64.deb ...
Unpacking isc-kea-dhcp6 (2.3.1-isc20220928105532) ...
Selecting previously unselected package isc-kea-common.
Preparing to unpack isc-kea-common_2.3.1-isc20220928105532_amd64.deb ...
Unpacking isc-kea-common (2.3.1-isc20220928105532) ...
dpkg: dependency problems prevent configuration of isc-kea-dhcp4:
 isc-kea-dhcp4 depends on libboost-system1.67.0; however:
  Package libboost-system1.67.0 is not installed.
[..]
koos@testrouter:~$ sudo apt install -f
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  libboost-system1.67.0 liblog4cplus-1.1-9 libmariadb3 libpq5 mariadb-common
  mysql-common
The following NEW packages will be installed:
  libboost-system1.67.0 liblog4cplus-1.1-9 libmariadb3 libpq5 mariadb-common
  mysql-common
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
3 not fully installed or removed.
Need to get 760 kB of archives.
After this operation, 4,001 kB of additional disk space will be used.
[..]

Looking at the sample configuration makes me think I can do this with a text-based configuration (it's actually JSON) and get it going fast. For my home network that is probably the best solution. Kea does have options to use MariaDB or PostgreSQL backends for storage which does look really nice for my home network but at the same time adds a dependency and a layer of complexity.

I can see IPAM systems totally going to Kea DHCP and give a full interface on managing the databases directly including APIs for adding/removing objects as they are added in other systems.

After the problems with detaching and attaching the USB 1-wire interface from a kvm virtual machine to fix an interference issue showed up again I decided to move the USB 1-wire interface to a different machine, one where kvm virtualisation isn't in the mix. The closest available machine that can deal with the 1-wire interface is a Raspberry Pi which also has other monitoring tasks.

This move worked fine and the 1-wire temperatures are showing up again in influxdb. I decided not to update the rrdtool temperature database. I will have to find time to migrate the rrdtool history to influxdb. Ideally there will be some aggregation for older measurements but I'd like an "infinite" archive of a daily average.

I have a DS2490 USB 1-wire interface on the home server conway which is rerouted to one of the virtual machines so that that virtual machine can read the sensors on the 1-wire network. This rerouting works when the machine is started, the DS2490 USB 1-wire shows up in the virtual machine fine. From time to time this DS2490 USB 1-wire interface gets confused when I am transmitting on the radio so the solution is to detach it from the virtual machine, unplug it from the server, plug it in again and attach it to the virtual machine again. Today this had to be done and I got an unexpected error message:

root@conway:~# virsh attach-device --live gosper /etc/onewire-for-gosper.xml
error: Failed to attach device from /etc/onewire-for-gosper.xml
error: internal error: unable to execute QEMU command 'device_add': failed to find host usb device 2:8

In logfile /var/log/libvirt/libvirtd.log:

2022-09-24 21:16:38.655+0000: 10923: error : qemuMonitorJSONCheckError:395 : internal error: unable to execute QEMU command 'device_add': failed to find host usb device 2:8

To be complete about it: usb device 2:8 is exactly the right one!

root@conway:~# lsusb | grep 2490
Bus 002 Device 008: ID 04fa:2490 Dallas Semiconductor DS1490F 2-in-1 Fob, 1-Wire adapter

This seems to be new since I upgraded the homeserver to Devuan beowulf giving me versions:

| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                  Version         Architecture Descripti
+++-=====================================-===============-============-=========
ii  libvirt-clients                       5.0.0-4+deb10u1 amd64        Programs 
ii  libvirt-daemon                        5.0.0-4+deb10u1 amd64        Virtualiz
un  libvirt-daemon-driver-storage-gluster                  (no descr
un  libvirt-daemon-driver-storage-rbd                      (no descr
un  libvirt-daemon-driver-storage-zfs                      (no descr
ii  libvirt-daemon-system                 5.0.0-4+deb10u1 amd64        Libvirt d
ii  libvirt-glib-1.0-0:amd64              1.0.0-1         amd64        libvirt G
ii  libvirt0:amd64                        5.0.0-4+deb10u1 amd64        library f

First idea: AppArmor

The first search result that came up was Bug #1552241 “libvirt-bin apparmor settings for usb host device” : Bugs : libvirt package : Ubuntu. So I tried changing the /etc/apparmor.d/abstractions/libvirt-qemu file. After a few tries and reading the warnings in the rest of the file I made sure the source was AppArmor by completely disabling it. The error did not go away so I reverted the libvirt-qemu rules to the original settings, restarted AppArmor and kept debugging.

Second idea: usb rights

Based on QEMU USB passthrough broken after Ubuntu 18.04 upgrade I added udev rules to make sure group libvirt-qemu had read and write rights on the usb device, with /lib/udev/rules.d/51-qemu-usb-passthrough.rules containing:

SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", ATTRS{idVendor}=="04fa", ATTRS{idProduct}=="2490", MODE="0664", GROUP="libvirt-qemu"

And doing the

root@conway:~# udevadm control --reload-rules

And verifying the resulting rule:

root@conway:~# udevadm test -a -p  $(udevadm info -q path -n /dev/bus/usb/002/008)
calling: test
version 3.2.9
This program is for debugging only, it does not run any program
specified by a RUN key. It may show incorrect results, because
some values may be different, or not available at a simulation run.

[..]

GROUP 110 /lib/udev/rules.d/51-qemu-usb-passthrough.rules:1
MODE 0664 /lib/udev/rules.d/51-qemu-usb-passthrough.rules:1
handling device node '/dev/bus/usb/002/008', devnum=c189:135, mode=0664, uid=0, gid=110
[..]

Indeed the right groupid, but still the same error message when trying the attach-device command.

Interesting find: it's specific to the virtual machine that had the device before

Small update: I can attach the USB device to a different host and detach it from that host again. I just can't attach it to the 'original' host again.

I also posted this question on serverfault: Can't live-attach a USB device to a kvm virtual host again after upgrades.

Update: After a complete reboot of the homeserver the USB 1-wire interface worked again (as I could imagine). But after another interference problem it's now in the same state again. I did change the definition in both the virthost configuration and the xml file from managed='no' to managed='yes' before the reboot but that hasn't helped. Contents of the /etc/onewire-for-gosper.xml file now:

    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x04fa'/>
        <product id='0x2490'/>
      </source>
    </hostdev>

Since last Thursday the aprs server at aprs.pa4tw.nl is down. I used that aprs server for the weather station and for the igate.

The change for the weather station was one word in a script, for the igate I had to remember how to change this with the Arduino development environment set up to support the esp32 board. The easiest way seemed to be from the computer, but every time after the igate started the running process after the setup it crashed and rebooted itself. I spent a lot of time looking for the answers, added debug statements all over the code and ended up in the WiFi initialization code as the place of crashing. And that was the hint, according to Crash when trying to connect to wifi - Issue #3935 - espressif/arduino-esp32 this is a sign of a power shortage.

This is purely my fault: the pi4raz igate design calls for an external power supply feeding it.

The solution was to go back to the separate USB power supply and not use a USB hub connected to the computer. Now the igate is started again and visible on the APRS network: track PE4KH-10 on aprs.fi.

A few days ago I noticed some interesting messages in the apcupsd log:

2022-07-04 10:14:15 +0200  Battery disconnected.
2022-07-04 10:16:24 +0200  Battery reattached.
2022-07-04 10:19:53 +0200  Battery disconnected.
2022-07-04 10:20:40 +0200  Battery reattached.

Checking the UPS statistics showed me the battery charge was dropping to about 7 % of the capacity while the mains power was available. Since the battery was over 5 years old I ordered a new one to replace it.

This battery was scheduled to arrive Wednesday at the start of the afternoon and I wanted to do an upgrade of the Linux distribution on the main homeserver conway anyway because devuan ascii is already 'oldoldstable' (but still getting updates).

The homeserver uses 2 disks with the main lvm volume in a raid-1. The /boot and /boot/efi filesystems are mirrored by hand with the idea to end with a working boot even when 1 disk is missing.

After the shutdown and replacing the UPS battery I switched the server on again and I was greeted by a grub prompt and nothing to boot. After a few tries I got the system booting again, after that I went searching for what went wrong. Eventually I found out the file /boot/efi/EFI/devuan/grub.cfg pointed at a missing filesystem. I found out the best way to fix this is with

# dpkg-reconfigure grub-efi-amd64

both with /dev/sda and /dev/sdb filesystems on /boot and /boot/efi.

I'm currently also doing some contacts with a special event station call and I wanted to separate the wsjt-x history for my normal call from the history for the special event station call, just like I split the log databases in CQRLOG.

For the non-amateurradio persons: I have my own callsign, PE4KH which is linked to me. It is also possible to have one extra temporary callsign. Those are usually linked to an event or some other reason for a 'special' callsign. Temporary callsigns in the Netherlands have either the digit 6 or more than one digit.

There is an option for multiple profiles in wsjt-x but those are just for the settings (including callsign) but not for the logging location. This means all different profiles share the same history and will show the same countries as 'new' or 'already contacted'.

When I was looking at the options for starting wsjt-x with different settings I noticed the -r --rig-name <rig-name> Where is for multi-instance support. option in the help. With this option, all the logging is in ~/.local/share/WSJT-X - <rig-name>/ which is what I want.

The next challenge is to start wsjt-x with the extra commandline paramater from CQRLOG. It seems the 'path to wsjt-x' setting doesn't accept commandline parameters. So I created a script ~/bin/ses-wsjtx with:

#!/bin/sh

/usr/bin/wsjtx -r ses

Changed the 'path to wsjt-x' setting to /home/koos/bin/ses-wsjtx and now I get what I want.

I've been playing with grafana for about a year since starting with updating my statistics gathering and I keep seeing new options and updates in grafana.

Grafana recently got some new options for alerting and I am trying a few of those. Alerts for things that are a real problem and can cause other problems are a good start. Based on some earlier problems I keep an eye on some filesystems that are over 90% full.

Today I read Three DDoS attacks on my personal website found via Three DDoS attacks on my personal website : r/homelab reddit and this made me wonder about overloads on my webserver. The easiest way to detect problems with web serving I could think of is to look at the queue size in haproxy which is monitored in influxdb/grafana anyway for nice graphs of website traffic.

I did have a time with too high queues for backend webservers. But that was when the backend server was completely broken due to a filesystem problem so that was a logical reason.

It would be nice if I could iterate alerts, like 'for the root filesystem of every monitored system'. Or at least copy them changing only the system name in the rules and alerts.

Just noted on bootup of a virtual machine:

Mar 10 19:42:14 turing kernel: [    0.181861] You have booted with nomodeset. This means your GPU drivers are DISABLED
Mar 10 19:42:14 turing kernel: [    0.181862] Any video related functionality will be severely degraded, and you may not even be able to suspend the system properly
Mar 10 19:42:14 turing kernel: [    0.181862] Unless you actually understand what nomodeset does, you should reboot without enabling it

It's a virtual machine which does server tasks. Anything more than 80x25 VGA text mode is pure overkill. It's currently the default card in qemu (Cirrus CLGD 5446 PCI VGA card), I could try the virtio VGA card to see if that saves on memory/cpu.

I want to know if something goes wrong but with the number of (virtual) servers here at home it is not possible to check all logs constantly. So the main machines use logcheck to find the interesting error messages and the rest gets filtered out.

Ideally that leaves no messages, but I do want to know about patterns that indicate attacks so I do get messages constantly about ssh attack attempts and weird nameserver requests or misconfigured nameserver responses.

Recently I've been checking the resulting reports again carefully and noticed some more patterns that could be filtered. And I found two misconfigurations that I solved. Normally those misconfigurations would drown in the noise of the log, only to be found if I was looking for something else. Now it started to stand out after filtering out a lot of messages that are to be expected.

After fixing the server hardware I had some time due to the Christmas holidays to look at my laptop, a Dell. It's getting a bit aged (originally from January 2016) and especially the disk is getting slow. Due to the upgrade of SSD storage in the homeserver I still have two 240 gigabyte solid state drives. So I tried to migrate the laptop to one of those solid state drives. Which was interesting in a number of ways: there are two operating systems to migrate: Linux and Windows 10 and the harddisk is 500 gigabyte, so 240 gigabyte would need an amount of cleanup before all could be moved.

I thought the harddisk was 320 gigabyte, so the downgrade from 500 to 240 gigabyte was worse than I expected.

I did some reading on migrating Windows 10 to an SSD and found out I needed a cloning tool. Navigating between subscriptions and expensive versions I found Macrium Reflect which according to How to Copy Your Windows Installation to an SSD - PCMag should be able to do this.

I have an external USB to IDE/SATA interface which is great for this kind of work. So the SSD started in that slot.

First windows didn't want to delete the EFI partition from the GPT partition table. Since the original disk has an msdos partition table and the laptop doesn't have UEFI firmware I booted linux and created partitions as I wanted them with the right type.

After that I created the Linux swapspace and filesystem and copied all Linux data to the filesystem.

After that the Macrium Reflect tool would not copy Windows 10 partitions to existing partitions so I had to delete the two Windows 10 partitions. I have no idea why, but this laptop has a Dell partition, a windows partition named RECOVERY and a windows partition named OS. Deleting the two windows partitions on the target disk also made the linux swap and root filesystem disappear without any questions whether that was a good idea.

After that it was several hours to copy the windows filesystems. After that was done I used the windows disk and partition manager to resize the big partition to leave space for the linux installation.

I booted Linux again, created the swap partitions and root filesystem again and copied the data again. At least rsync with the right options is faster than Macrium Reflect.

After that I tried to install grub on the new disk with the right options and did the first test boot of the new disk. Open laptop underside, take out disk carrier, swap disk, put the disk carrier back in and close the laptop again.

No dice: grub stopped really early. I did more searching and found I needed to use grub-install /dev/sdb --skip-fs-probe --boot-directory=/mnt/newinstall/boot so time to remove the new drive again, revert to the old, rerun grub with those options, remove old drive, insert new drive and try again. This time the menu showed that I wanted but I got an error about accessing the disk by uuid.

After that I also tried windows on the SSD but that gave an error it needed the Windows recovery boot.

So again back to the old disk and looking at options for creating a recovery boot USB stick. The 'Create recovery disk' program was busy with disk i/o for about 15 minutes and reported the USB stick for recovery has to be at least 16 Gigabytes which I didn't have available.

At this point I gave up. This process took most of the afternoon and it started to feel frustrating.