News items for tag linux - Koos van den Hout

Ages ago I added a way to get access to my google contacts as a thunderbird address book. But on installation of thunderbird on a new laptop I couldn't find a simple answer to "how did I do that again?!?".

With access to the old laptop I was able to reconstruct my steps, so I'll note them here:

• Install cardbook as add-on in Thunderbird
• Go to this add-in in the Thunderbird userinterface
• From the top left 'hamburger' menu, select 'Address book', 'New address book'
• A window pops up asking 'Address book location', select 'Remote'
• The next window asks 'type of your address book' and gives google as default selection
• As username enter the standard address used for your google account. This doesn't have to end in @gmail.com.
• After entering the address, click 'Validate' and a window pops up with a minimal browser to log into your google account. Do this.
• After logging in the browser window will ask for permission for Cardbook to access your google contacts.
• After selecting a colour for this new address book you can use it.
• In the process google will probably send you alerts about this new login and permission.

And now my contacts are synchronized between android phone, google contacts web interface and thunderbird!

On my todo-list was a postgresql upgrade from 9.6 to 11, a lingering item from the earlier devuan upgrades from ascii to beowulf.

This is one of those upgrades where I am very happy to have lvm snapshots so I know I can get back to a working state if something really goes wrong.

With that snapshot and the instructions from From Stretch to Buster : How to upgrade a 9.6 PostgreSQL cluster to 11 ? - Samuel Forestier it all went fine. After the upgrade I tested all my database-driven websites and local tools to see if they worked. All worked fine so I could stop and delete the 9.6 main postgres cluster and continue running 11.

Final cleanup was deleting the snapshot. Which used as much space as the size of the database! Not unsurprising when thinking about how the upgrade works, but think about the snapshot size.

I run a desktop and a laptop with Ubuntu and both were at Ubuntu 20.04. The desktop is mostly used for things with amateur radio so I wanted to check whether anything broke on that upgrade. With the 18.04 to 20.04 upgrade I had to do some recovery to get the databases behind cqrlog working again,

Time to upgrade the laptop first with the same amateur radio software installed, configured and tested: cqrlog, wsjt-x, fldigi.

The whole do-release-upgrade took more than an hour. And it's still possible that somewhere during the upgrade process the user gets prompted whether or not to change a configuration file, so I came back after a few hours to a system with a prompt and not finished with the upgrade.

The upgrade told me firefox would be changed from an installed package to a 'snap'. The downside for me was that after the first start firefox thought it was a completely new browser with no history/bookmarks/settings. Maybe this was because the start of firefox was triggered by thunderbird starting and wanting to show me a page about donating. Restarting firefox didn't make the old profile show up again. With a bit of searching I found that firefox should import old non-snap settings when started as a 'snap' for the first time. So I stopped firefox, threw out the whole ~/snap/firefox directory and started it again. This time settings/bookmarks/cookies/history were imported.

Next step was to test cqrlog. There is no cqrlog build for ubuntu 22.04 yet, but the build for 20.04 works. All previously logged data was available fine. The upgrade of ubuntu has upgraded hamlib which means the radio IDs got renumbered, I had to update the settings to the new radio ID.

Silencing Ubuntu Pro adverts

In regular maintenance I noticed this gem:

$ sudo apt dist-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following security updates require Ubuntu Pro with 'esm-apps' enabled:
  libimage-magick-perl imagemagick libjs-jquery-ui libopenexr25
  libmagick++-6.q16-8 libmagickcore-6.q16-6-extra libimage-magick-q16-perl
  libmagickwand-6.q16-6 imagemagick-6.q16 libmagickcore-6.q16-6
  imagemagick-6-common
Learn more about Ubuntu Pro at https://ubuntu.com/pro
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

which is no better than an advertisment for Ubuntu Pro. Which is a new service by Canonical offering longer term support (10 years) and support for not just "Main" (which is what you got with Ubuntu before) but "Main" and "Universe". Ubuntu Pro costs a registration for private use at the moment. So 'The following security updates require Ubuntu Pro' isn't completely honest. But then again, it's advertising.

Anyway, I don't want to see this every time I check for updates. I searched for a solution, The following security updates require Ubuntu Pro with 'esm-apps' enabled - reddit.com r/linux

$ cd /etc/apt/conf.d
$ sudo mv 20apt-esm-hook.conf 20apt-esm-hook.conf.disabled
$ sudo touch 20apt-esm-hook.conf
$ sudo chattr +i 20apt-esm-hook.conf

I noticed lots of kernel modules for filesystem support were loaded after running update-grub. This was caused by running os-prober which searches for possible operating system installations on all partitions of the system.

On virtual and physical machines that only run linux and will never run anything else unless I am really changing something this only takes time and uses resources, so I searched for how to disable this. So now there is a line in /etc/default/grub:

# don't look high and low for other operating systems
GRUB_DISABLE_OS_PROBER=true

But now update-grub thinks it is necessary to warn me every time...

# update-grub
Generating grub configuration file ...
[..]
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
done

I know it will not be executed, I added it on purpose. It's not very likely I added GRUB_DISABLE_OS_PROBER=true by accident not knowing what I was doing. Stop nagging me about it. If I didn't know what I was doing on a computer I wouldn't be configuring linux distributions.

Eight and a half years and over 14000 contacts after I bought a Yaesu FT-857D I thought it was time to upgrade. The basic requirements haven't changed a lot: HF, 2 meter, 70 centimeter bands, SSB, Morse, FM, support for computer control. What I wanted to improve on is noise filtering, handling of strong adjacent signals and a waterfall display.

So the choice is the Yaesu FT-991A although I also looked at HF-only radios from Yaesu but decided on this one in the end. This will be the base station radio for a while and I will only use the FT-857D for operating away from home.

The basic installation went fine and I think this is a great amateur radio and good value for money. It is an advanced technological device so I had to dig into manuals and on-line documentation several times to get things set up the way I wanted it.

The good innovation is that the Yaesu FT-991A has an USB port on the back. This USB connection gives the computer 2 serial ports and audio over USB. The first serial port is for Computer Aided Tuning (CAT) control which can control the radio from the computer.

I directly wanted to set up an udev rule to map this to a fixed symlink so I can start rigtctld easily. The new rule:

SUBSYSTEM=="tty", ENV{ID_MODEL}=="CP2105_Dual_USB_to_UART_Bridge_Controller", \
        ATTRS{idVendor}=="10c4", ATTRS{idProduct}=="ea70", \
        ENV{ID_USB_INTERFACE_NUM}=="00", \
        SYMLINK+="ttyCATya"

The ENV{ID_USB_INTERFACE_NUM}=="00", filter only makes this rule activate on the first of the serial ports offered by the CP2105 chip.

My current experience is that the noise filtering is indeed better which helps a lot in the noisy RF environment at home.

Recently I wanted to have the option to install ubuntu on a PC so I created a USB stick with dd. It worked fine and in the end the existing ubuntu on the PC worked ok and could be upgraded and made available again.

So I wanted to revert this USB stick to the normal filesystem that both Windows and Linux can read and write. This turned out to be more difficult than I expected! First I thought Windows could revert the USB stick to a usable state but this turned out to be impossible. I tried on three Windows 10 systems with admin accounts, but none of them were able to create a usable partition and filesystem! The best result I could get was an error something couldn't be started to format the partition, but without any explanation what couldn't be started. Things that were once perfectly doable under MS-DOS are now impossible.

Back to linux to try and find the right partition type and filesystem options to get access again. I could do a lot of things in linux, but I failed to find the right settings that Windows would see as usable storage.

I shared my problems on irc and someone there had the following list of commands to fix this problem:

dd if=/dev/zero of=/dev/sdd bs=1M count=1
parted /dev/sdd mklabel msdos
parted /dev/sdd mkpart primary fat32 1 100%
mkdosfs /dev/sdd1

Which needs to be adjusted for the right device node. Use at your own risk! But indeed after these commands both Windows and Linux were perfectly capable of writing and reading the USB stick.

In the beginning of October I shut down the home server conway and reseated the SATA cables in the hopes of having less problems with timeouts. And started the whole system again to also fix other problems.

About a month later I think this worked, I've never seen a rcu_sched message again since doing that reseating.

Yesterday I learned that ISC DHCP server will be end of life at the end of this year. For a package I started using around 1998 with one of the first versions I expected a bit more announcement time. At the same time I'm so used to using ISC dhcp server in my home network I never subscribed to any mailing list or other announcements about ISC dhcp server, it's just there, I can configure it to do what I want including supporting pxe booting systems for installation or diagnostics or supporting special dhcp options for APC AP7920 rackmount power distribution units. And all the virtual lans of my home network.

ISC suggests using Kea DHCP server to replace it in most server implementations. Kea DHCP server should be able to get a lot of configuration data from databases and allow for dynamic updates of the configuration. That is an improvement over ISC dhcp as it is at the moment, which needs a full restart for every change.

So time to peek at Kea DHCP server. I don't think ISC dhcp server will be unavailable after 31 December 2022 but I don't expect updates anymore and when a good replacement is normalized I expect ISC dhcp server to slowly fall away from linux distributions.

Currently it's not even available for Debian or Devuan stable or oldstable strangely enough. I wonder what happened there. But there are distribution packages for debian buster at Cloudsmith - Repositories - ISC - Internet Systems Consortium (isc) - kea-2-3 (kea-2-3) - Packages / format:deb.

Time to install the latest and let apt fix the dependencies:

koos@testrouter:~$ sudo dpkg -i isc-kea-dhcp4_2.3.1-isc20220928105532_amd64.deb isc-kea-dhcp6_2.3.1-isc20220928105532_amd64.deb isc-kea-common_2.3.1-isc20220928105532_amd64.deb 
Selecting previously unselected package isc-kea-dhcp4.
(Reading database ... 46609 files and directories currently installed.)
Preparing to unpack isc-kea-dhcp4_2.3.1-isc20220928105532_amd64.deb ...
Unpacking isc-kea-dhcp4 (2.3.1-isc20220928105532) ...
Selecting previously unselected package isc-kea-dhcp6.
Preparing to unpack isc-kea-dhcp6_2.3.1-isc20220928105532_amd64.deb ...
Unpacking isc-kea-dhcp6 (2.3.1-isc20220928105532) ...
Selecting previously unselected package isc-kea-common.
Preparing to unpack isc-kea-common_2.3.1-isc20220928105532_amd64.deb ...
Unpacking isc-kea-common (2.3.1-isc20220928105532) ...
dpkg: dependency problems prevent configuration of isc-kea-dhcp4:
 isc-kea-dhcp4 depends on libboost-system1.67.0; however:
  Package libboost-system1.67.0 is not installed.
[..]
koos@testrouter:~$ sudo apt install -f
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  libboost-system1.67.0 liblog4cplus-1.1-9 libmariadb3 libpq5 mariadb-common
  mysql-common
The following NEW packages will be installed:
  libboost-system1.67.0 liblog4cplus-1.1-9 libmariadb3 libpq5 mariadb-common
  mysql-common
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
3 not fully installed or removed.
Need to get 760 kB of archives.
After this operation, 4,001 kB of additional disk space will be used.
[..]

Looking at the sample configuration makes me think I can do this with a text-based configuration (it's actually JSON) and get it going fast. For my home network that is probably the best solution. Kea does have options to use MariaDB or PostgreSQL backends for storage which does look really nice for my home network but at the same time adds a dependency and a layer of complexity.

I can see IPAM systems totally going to Kea DHCP and give a full interface on managing the databases directly including APIs for adding/removing objects as they are added in other systems.

After the problems with detaching and attaching the USB 1-wire interface from a kvm virtual machine to fix an interference issue showed up again I decided to move the USB 1-wire interface to a different machine, one where kvm virtualisation isn't in the mix. The closest available machine that can deal with the 1-wire interface is a Raspberry Pi which also has other monitoring tasks.

This move worked fine and the 1-wire temperatures are showing up again in influxdb. I decided not to update the rrdtool temperature database. I will have to find time to migrate the rrdtool history to influxdb. Ideally there will be some aggregation for older measurements but I'd like an "infinite" archive of a daily average.

I have a DS2490 USB 1-wire interface on the home server conway which is rerouted to one of the virtual machines so that that virtual machine can read the sensors on the 1-wire network. This rerouting works when the machine is started, the DS2490 USB 1-wire shows up in the virtual machine fine. From time to time this DS2490 USB 1-wire interface gets confused when I am transmitting on the radio so the solution is to detach it from the virtual machine, unplug it from the server, plug it in again and attach it to the virtual machine again. Today this had to be done and I got an unexpected error message:

root@conway:~# virsh attach-device --live gosper /etc/onewire-for-gosper.xml
error: Failed to attach device from /etc/onewire-for-gosper.xml
error: internal error: unable to execute QEMU command 'device_add': failed to find host usb device 2:8

In logfile /var/log/libvirt/libvirtd.log:

2022-09-24 21:16:38.655+0000: 10923: error : qemuMonitorJSONCheckError:395 : internal error: unable to execute QEMU command 'device_add': failed to find host usb device 2:8

To be complete about it: usb device 2:8 is exactly the right one!

root@conway:~# lsusb | grep 2490
Bus 002 Device 008: ID 04fa:2490 Dallas Semiconductor DS1490F 2-in-1 Fob, 1-Wire adapter

This seems to be new since I upgraded the homeserver to Devuan beowulf giving me versions:

| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                  Version         Architecture Descripti
+++-=====================================-===============-============-=========
ii  libvirt-clients                       5.0.0-4+deb10u1 amd64        Programs 
ii  libvirt-daemon                        5.0.0-4+deb10u1 amd64        Virtualiz
un  libvirt-daemon-driver-storage-gluster                  (no descr
un  libvirt-daemon-driver-storage-rbd                      (no descr
un  libvirt-daemon-driver-storage-zfs                      (no descr
ii  libvirt-daemon-system                 5.0.0-4+deb10u1 amd64        Libvirt d
ii  libvirt-glib-1.0-0:amd64              1.0.0-1         amd64        libvirt G
ii  libvirt0:amd64                        5.0.0-4+deb10u1 amd64        library f

First idea: AppArmor

The first search result that came up was Bug #1552241 “libvirt-bin apparmor settings for usb host device” : Bugs : libvirt package : Ubuntu. So I tried changing the /etc/apparmor.d/abstractions/libvirt-qemu file. After a few tries and reading the warnings in the rest of the file I made sure the source was AppArmor by completely disabling it. The error did not go away so I reverted the libvirt-qemu rules to the original settings, restarted AppArmor and kept debugging.

Second idea: usb rights

Based on QEMU USB passthrough broken after Ubuntu 18.04 upgrade I added udev rules to make sure group libvirt-qemu had read and write rights on the usb device, with /lib/udev/rules.d/51-qemu-usb-passthrough.rules containing:

SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", ATTRS{idVendor}=="04fa", ATTRS{idProduct}=="2490", MODE="0664", GROUP="libvirt-qemu"

And doing the

root@conway:~# udevadm control --reload-rules

And verifying the resulting rule:

root@conway:~# udevadm test -a -p  $(udevadm info -q path -n /dev/bus/usb/002/008)
calling: test
version 3.2.9
This program is for debugging only, it does not run any program
specified by a RUN key. It may show incorrect results, because
some values may be different, or not available at a simulation run.

[..]

GROUP 110 /lib/udev/rules.d/51-qemu-usb-passthrough.rules:1
MODE 0664 /lib/udev/rules.d/51-qemu-usb-passthrough.rules:1
handling device node '/dev/bus/usb/002/008', devnum=c189:135, mode=0664, uid=0, gid=110
[..]

Indeed the right groupid, but still the same error message when trying the attach-device command.

Interesting find: it's specific to the virtual machine that had the device before

Small update: I can attach the USB device to a different host and detach it from that host again. I just can't attach it to the 'original' host again.

I also posted this question on serverfault: Can't live-attach a USB device to a kvm virtual host again after upgrades.

Update: After a complete reboot of the homeserver the USB 1-wire interface worked again (as I could imagine). But after another interference problem it's now in the same state again. I did change the definition in both the virthost configuration and the xml file from managed='no' to managed='yes' before the reboot but that hasn't helped. Contents of the /etc/onewire-for-gosper.xml file now:

    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x04fa'/>
        <product id='0x2490'/>
      </source>
    </hostdev>