News items for tag linux - Koos van den Hout

2018-11-28 Using mice adopted to my hands 1 week ago
The old rsi problem was acting up again, just like I had RSI in 1999.

One of the things I now did was add a left-side mouse on the linux desktop at home. I have used a left-side mouse for a number of years on a linux desktop and used the instructions from the xmodmap manpage:
       Many  pointers are designed such that the first button is pressed using
       the index finger of the right hand.  People who  are  left-handed  fre‐
       quently  find  that  it is more comfortable to reverse the button codes
       that get generated so that the primary  button  is  pressed  using  the
       index  finger  of  the  left  hand.   This  could be done on a 3 button
       pointer as follows:
       %  xmodmap -e "pointer = 3 2 1"
But I now have two USB mice, one with a forward/backward button and a clearly right-handed design and one simple one on the left. And it is possible to selectively swap mouse buttons on only one input device with xinput.

The list of all inputs:
koos@thompson:~$ xinput list
⎡ Virtual core pointer                          id=2    [master pointer  (3)]
⎜   ↳ Virtual core XTEST pointer                id=4    [slave  pointer  (2)]
⎜   ↳ Logitech USB-PS/2 Optical Mouse           id=9    [slave  pointer  (2)]
⎜   ↳ Logitech Optical USB Mouse                id=10   [slave  pointer  (2)]
⎣ Virtual core keyboard                         id=3    [master keyboard (2)]
    ↳ Virtual core XTEST keyboard               id=5    [slave  keyboard (3)]
    ↳ Power Button                              id=6    [slave  keyboard (3)]
    ↳ Power Button                              id=7    [slave  keyboard (3)]
    ↳ Burr-Brown from TI               USB Audio CODEC  id=8    [slave  keyboard (3)]
    ↳ VIA Technologies Inc. USB Audio Device    id=11   [slave  keyboard (3)]
    ↳ daskeyboard                               id=12   [slave  keyboard (3)]
    ↳ daskeyboard                               id=13   [slave  keyboard (3)]
    ↳ Dell WMI hotkeys                          id=14   [slave  keyboard (3)]
Setting the button order happens with xinput set-button-map which needs an ID. Solution in .xsession:
xinput set-button-map $(xinput list --id-only "Logitech Optical USB Mouse") 3 2 1

Oh, and in that other operating system I use (Windows) one of the problems is the user can't set mouse button order per device. And technical specifications of left-handed mice do not list whether the buttons are swapped in hardware.

Tags: , ,
2018-11-23 Automatic ls colours can be slow 2 weeks ago
I noticed certain commands taking a while to start, including a simple ls. At last I got annoyed enough to diagnose the whole situation and found out the problem is the combination of symbolic links in the listed directory pointing to filesystems behind automounter, one mounted filesystem coming from a NAS with sleeping disk and ls --color doing a stat() on the target of a symbolic link to find the type of the target file to be able to select a colour.

My solution: find the source of the alias and disable it.

Tags: , ,
2018-11-16 Changing the way I listen to podcasts 3 weeks ago
I bought the iRiver ifp-795 in May 2005 to listen to podcasts, mostly while cycling to and from work.

But I need to find time to download new episodes on the laptop and copy them in the right order to the storage of the mp3 player. There is an another device which can do all this and can play the mp3 files too: my android smartphone.

So I looked for an Android podcast player which can deal with podcast feeds not in its own directory. After reading an overview article and browsing the play store I found RadioPublic and managed to add my favourite podcasts.

Adding a feed it didn't know was a bit harder than expected. I want to listen to The ICQ Amateur / Ham Radio Podcast but it wasn't listed. So I tried to add the RSS feed myself by typing the URL which failed. Adding it only worked out after I opened the RSS feed in my browser on android and copied and pasted the url to the 'search' field.

The application has a nice playlist and I can order the downloaded episodes in such a way that I don't get several episodes from the same show in a row.

Ok, I found one downside: it seems impossible to add an mp3 downloaded via the browser to the RadioPublic playlist.

Tags: , ,
2018-10-12 Serious slowness with rrdgraph from rrdtool 1 month ago
One of the things still needing migrating is the NTP server stats which obviously uses rrdtool. Because I want to keep the history I migrated the datasets with:
/usr/local/rrdtool/bin/rrdtool dump \
| ssh newhost /usr/bin/rrdtool restore -f -
And then create a graph of the plloffset for example using:
/usr/bin/rrdtool graph /tmp/ \
--title " pll offset (last 24 hours)" --imginfo \
'<img src="tmpgraphs/%s" WIDTH="%lu" HEIGHT="%lu" alt="Graph">' \
--start -24hours --end now --vertical-label="Seconds" --color BACK#0000FF \
--color CANVAS#c0e5ff --color FONT#ffffff --color GRID#ffffff \
--color MGRID#ffffff --alt-autoscale --imgformat PNG --lazy \ \
CDEF:wipeout=offset,UN,INF,UNKN,IF CDEF:wipeoutn=wipeout,-1,* \
LINE1:offset#000000:"Offset\:" \
GPRINT:offset:LAST:"Current\:%.3lf%s" \
GPRINT:offset:MIN:"Min\:%.3lf%S" \
GPRINT:offset:MAX:"Max\:%.3lf%S" \
GPRINT:offset:AVERAGE:"Average\:%.3lf%S" \
AREA:wipeout#e0e0e0 AREA:wipeoutn#e0e0e0
But on the old server this takes 0.026 seconds, on the new server 3 minutes and 47.46 seconds. No idea what is happening, strace shows nothing strange and rrdtool uses 1 cpu at 100% all that time.
Read the rest of Serious slowness with rrdgraph from rrdtool

Tags: , , ,
2018-10-03 Seeing the same names in logcheck mails every hour 2 months ago
I use the logcheck package to monitor for unexpected log entries. Since upgrading to the new homeserver conway I noticed DNSSEC failures coming back regularly, even at weird times of the night while the domain names seemed related to services we sometimes interact with during the day. To search deeper I enabled query logging on DNS (with a short retention period) in order to find the source.

Eventually I found it: the DNSSEC failures came at the time the mail from logcheck was delivered, because it mentioned domain names that cause a DNSSEC failure. So the way to 'fix' this problem and avoid similar other problems was to whitelist logcheck mail.

Update 2018-10-05: That only helps when enabling the Mail::SpamAssassin::Plugin::Shortcircuit plugin and enabling the USER_IN_WHITELIST shortcircuit.

Update 2018-10-07: Even with whitelist and shortcircuit I still see queries for domain names in the logcheck mails. Call to spamassassin is now changed...

Now, once again...this time with FEEwing

Tags: , ,
2018-09-26 Made the big bang to the new homeserver 2 months ago
So for months and months I had hardware ready for the new homeserver, I was testing bits and pieces in the new environment and I still did not get around to making the big bang. Part of the time the new system was running and using electricity.

And a few weeks ago I had time for the big bang and forgot to mention it!

So one free day I just did the last sync of homedirectories and started migrating all services in a big bang. No more but, if, when, is it done yet. It's a homeserver, not a complete operational datacenter. Although with everything running it sometimes does look that way!

The new setup, more completely documented at Building - and maintaining home server conway 2017 is now running almost all tasks. The main migration was homedirectories, mail, news, webservers. Things are now split over several virtual machines and the base virtual machine running kvm virtual machines is as minimal as possible.

One thing I just noticed is that the new virtual machine with pppoe kernel mode drivers and updated software is doing great: the bigger MTU is working by default and kernel mode pppoe does not show up as using CPU when a 50 mbit download is active. I looked at CPU usage with htop and at the network traffic with iptraf and the result was that iptraf was using the most cpu.

There are still some things left to migrate, including a few public websites that currently give 50x errors. But I will find the time eventually.

Tags: , , ,
2018-09-24 After 25 years with sendmail there was still something to improve 2 months ago
I still like running sendmail on my own systems. But sendmail evolves with time and my configuration does improve slightly sometimes, such as on the introduction of authenticated smtp with secondary passwords.

After the recent upgrades to the home server there is a new mail server with some other new details and suddenly other systems at home could not relay. A bit of searching found Best practice: sendmail and SMTP auth with the right flags for the DAEMON_OPTIONS to only offer authentication on port 587 (submission).

I noticed the local systems tried relaying via port 587 so I changed this to port 25 where IP-based relaying is allowed. No idea why I set this up to use the port 587 when I set it up previously.

And yes, I checked it, I started with sendmail in 1993, so 25 years of sendmail on port 25. I did start with writing my own rules but I switched to .mc based configurations.

Tags: , , ,
2018-09-21 Setting my bash prompt PS1 to remind me I'm in screen 2 months ago
With some systems constantly running screen and others not I started to get confused. Solution: change the visual indications in the prompt inside screen.

I decided to just change the username color in PS1 when I'm in screen. So now:
PS1='${STY:+\[\e[1;36m\]}\u${STY:+\[\e[0m\]}@\h:\w\$ '
In bash, ${STY:+..} gives output when shell variable STY is set. So I add the color set/unset commands to the prompt when STY, a typical screen variable is set. The result is dark cyan, a color that works (for me) on my normal light-grey background xterm/putty sessions.

Oh, and for root things are different:
PS1='\[\e[1;91m\]\u@\h\[\e[0m\]:\w\$ '
Which gives a light red user@hostname.

In the above \e causes an escape to be printed. Wrapping parts of the prompt between \[ and \] causes bash to ignore those for counting the length of the prompt so it doesn't get confused on redrawing the prompt when editing the commandline.

Samples of colours and other formatting at FLOZz' MISC » bash:tip_colors_and_formatting.

Tags: , ,
2018-09-06 Weird interface names in snmp due to virtio driver 3 months ago
I want to measure network traffic so I decided to copy most of my rrdtool setup from the old home server.

But with virtio network cards I have a confused snmpd:
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: Red Hat, Inc Device 0001
IF-MIB::ifDescr.3 = STRING: Red Hat, Inc Device 0001
IF-MIB::ifDescr.4 = STRING: Red Hat, Inc Device 0001
IF-MIB::ifDescr.5 = STRING: dummy0
IF-MIB::ifDescr.6 = STRING: dumhost
IF-MIB::ifDescr.7 = STRING: dumdh6
Fix: go for the IF-MIB::ifName snmp variables, found in oid
IF-MIB::ifName.1 = STRING: lo
IF-MIB::ifName.2 = STRING: eth0
IF-MIB::ifName.3 = STRING: eth1
IF-MIB::ifName.4 = STRING: eth2
IF-MIB::ifName.5 = STRING: dummy0
IF-MIB::ifName.6 = STRING: dumhost
IF-MIB::ifName.7 = STRING: dumdh6
Those are easier to discern, now my snmp scripts are gathering data again.

Tags: , , ,
2018-08-17 Trying (and failing) to correlate security logs 3 months ago
Since activating sendmail authentication with secondary passwords I see a number of attempts to guess credentials to send mail via my system. This is not very surprising, given the constant attack levels on the wider Internet.

For work I am looking at log correlation and monitoring and with that in mind I noted that finding the right information from sendmail where and when the attempt came from is quite hard since there are several processes busy and it's hard to correlate the logging. The failed attempt is logged by saslauthd in /var/log/auth.log:
Aug 16 12:28:57 greenblatt saslauthd[32648]: pam_unix(smtp:auth): check pass; user unknown
Aug 16 12:28:57 greenblatt saslauthd[32648]: pam_unix(smtp:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=
Aug 16 12:28:59 greenblatt saslauthd[32648]: do_auth         : auth failure: [user=monster] [service=smtp] [] [mech=pam] [reason=PAM auth error]
Aug 16 12:29:00 greenblatt saslauthd[32649]: pam_unix(smtp:auth): check pass; user unknown
Aug 16 12:29:00 greenblatt saslauthd[32649]: pam_unix(smtp:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=
Aug 16 12:29:02 greenblatt saslauthd[32649]: do_auth         : auth failure: [user=monster] [service=smtp] [realm=] [mech=pam] [reason=PAM auth error]
This is probably related to this sendmail log information:
Aug 16 12:28:56 greenblatt sm-mta[20716]: STARTTLS=server, [] (may be forged), version=TLSv1/SSLv3, verify=NO, cipher=DHE-RSA-AES256-SHA, bits=256/256
Aug 16 12:29:02 greenblatt sm-mta[20716]: w7GASspx020716: [] (may be forged) did not issue MAIL/EXPN/VRFY/ETRN during connection to MSP-v6
But I can't be sure as there are multiple 'did not issue MAIL/EXPN/VRFY/ETRN' messages in the logs. So I can't build a fail2ban rule based on this.

Tags: , , ,
2018-07-27 Automating Let's Encrypt certificates with DNS-01 protocol 4 months ago
Encrypt all the things meme After thoroughly automating Let's Encrypt certificate renewal and installation I wanted to get the same level of automation for systems that do not expose an http service to the outside world. So that means the DNS-01 challenge within the ACME protocol has to be used.

I found out dehydrated Let's Encrypt certificate management supports DNS-01 and I found a sample on how to do this with bind9 at Example hook script using Dynamic DNS update utility for dns-01 challenge which looks like it can do the job.

It took me a few failed tries to find out that if I want a certificate for the name that it will request the TXT record for to make me prove that I have control over the right bit of DNS. I first assumed something in which turned out wrong. So the bind9 config in /etc/bind/named.conf.local has:
zone "" {
        type master;
        file "/var/cache/bind/";
        masterfile-format text;
        allow-update { key "acmekey-turing"; };
        allow-query { any; };
        allow-transfer {
And in the zone there is just one delegation:
_acme-challenge.turing  IN      NS      ns2
I created and used a dnskey with something like:
# dnssec-keygen -r /dev/random -a hmac-sha512 -b 128 -n HOST acmekey-turing
This gives 2 files, both with the right secret:
# ls Kacmekey-turing.+157+53887.*
Kacmekey-turing.+157+53887.key  Kacmekey-turing.+157+53887.private
# cat Kacmekey-turing.+157+53887.key
acmekey-turing. IN KEY 512 3 157 c2V0ZWMgYXN0cm9ub215
and configured it in /etc/bind/named.conf.options:
key "acmekey-turing" {
        algorithm hmac-md5;
        secret "c2V0ZWMgYXN0cm9ub215";
And now I can request a key for and use it to generate sendmail certificates. And the net result:
        (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256          
SMTP between systems with TLS working and good certificates.

Tags: , , ,
2018-07-19 Configuring sendmail authentication like imaps access to allow secondary passwords 4 months ago
I needed to configure sendmail authenticated access because I want a strict SPF record for which means I always have to make outgoing mail originate from the right server.

For the sendmail authenticated smtp bit I used How to setup and test SMTP AUTH within Sendmail with some configuration details from Setting up SMTP AUTH with sendmail and Cyrus-SASL. To get this running saslauthd is needed to get authentication at all and I decided to let it use the pam authentication mechanism. The relevant part of
define(`confAUTH_OPTIONS', `A p')dnl
And now I can login to sendmail only in an encrypted session. And due to sendmail and other services now having valid certificates I can set up all devices to fully check the certificate so I make it difficult to intercept this password.

And after I got that working I decided I wanted 'secondary passwords' just like I configured extra passwords for IMAPS access so I set up /etc/pam.d/smtp to allow other passwords than the unix password and restrict access to the right class of users.
auth    required quiet user ingroup users
auth    [success=1 default=ignore] nullok_secure
auth    sufficient db=/etc/courier/extrausers crypt=crypt use_first_pass
# here's the fallback if no module succeeds
auth    requisite             
Now I can set up my devices that insist on saving the password for outgoing smtp and if it ever gets compromised I just have to change that password without it biting me too hard.

Tags: , , ,
2018-07-08 Automating Let's Encrypt certificates further 5 months ago
Encrypt all the things meme Over two years ago I started using Let's Encrypt certificates. Recently I wanted to automate this a step further and found dehydrated automated certificate renewal which helps a lot in automating certificate renewal with minimal hassle.

First thing I fixed was http-based verification. The webserver has been set up to make all .well-known/acme-challenge directories end up in one place on the filesystem and it turns out this works great with dehydrated.

I created a separate user for dehydrated, gave that user write permissions for the /home/httpd/html/.well-known/acme-challenge directory. It also needs write access to /etc/dehydrated for its own state. I changed /etc/dehydrated/config with:
Now it was possible to request certificates based on a .csr file. I used this to get a new certificate for the home webserver, and it turned out to be easier than the previous setup based on letsencrypt-nosudo.
Read the rest of Automating Let's Encrypt certificates further

Tags: , , , ,
2018-06-23 SMART can be wrong 5 months ago
Someone brought me a 'WD My cloud' that does not respond at all. So I took it apart and found out how to access the disk in an i386 Linux system: mount the 4th partition as ext4. When the disk was available I did a smart test:
SMART overall-health self-assessment test result: PASSED
But while trying to find out how much data is actually on the disk, I get:
[  866.165641] Sense Key : Medium Error [current] [descriptor]
[  866.165645] Descriptor sense data with sense descriptors (in hex):
[  866.165647]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[  866.165659]         b0 90 ea 60 
[  866.165664] sd 2:0:0:0: [sda]  
[  866.165668] Add. Sense: Unrecovered read error - auto reallocate failed
So the disk isn't very healthy. But rerunning the smart check still shows nothing is wrong. It is a Western Digital 'RED' harddisk especially for NAS systems so it should return errors earlier to the operating system but this disk is bad, which is probably related to why the 'my cloud' enclosure isn't working.
Read the rest of SMART can be wrong

Tags: ,
2018-06-17 Apache 2.2 Proxy and default block for everything but the .well-known/acme-challenge urls 5 months ago
I'm setting up a website on a new virtual machine on the new homeserver and I want a valid letsencrypt certificate. It's a site I don't want to migrate so I'll have to use the Apache proxy on the 'old' server to allow the site to be accessed via IPv4/IPv6 (for consistency I am now setting up everything via a proxy).

So first I set up a proxy to pass all requests for the new server to the backend, something like:
        ProxyPass /
        ProxyPassReverse /
But now the requests for /.well-known/acme-challenge also go there and they are blocked needing a username/password since the new site is not open yet.

So to set up the proxy correctly AND avoid the username checks for /.well-known/acme-challenge the order has to be correct. In the ProxyPass rules the rule for the specific URL has to come first and in the Location setup it has to come last.
        ProxyPass /.well-known/acme-challenge !
        ProxyPass /
        ProxyPassReverse /

        <Location />
        Deny from all
        AuthName "Site not open yet"

        <Location /.well-known/acme-challenge>
            Order allow,deny
            Allow from all
And now the acme-challenge is done locally on the server and all other requests get forwarded to the backend after authentication.

Tags: , , ,
2018-05-03 The preferring IPv6 policy is working 7 months ago
Yesterday I changed some IPv4 addresses on virtual machines on the new homeserver to make autofs work. This is a known issue with autofs: autofs does not appear to support IPv6 hostname lookups for NFS mounts - Debian Bug #737679 and for me the easy solution is to do NFS mounts over rfc1918 ipv4 addresses. I prefer autofs over 'fixed' NFS mounts for those filesystems that are nice to be available but aren't needed constantly.

It took about 9 hours before arpwatch on the central router noticed the new activity. I guess the policy to try to do everything over IPv6 is working.

Tags: , , ,
2018-04-24 KVM and os-specific defaults 7 months ago
Today I wanted to install a new virtual machine on the new homeserver and virt-install gave me a new warning:
WARNING  No operating system detected, VM performance may suffer. Specify an OS with --os-variant for optimal results.
According to the virt-install manpage the --os-variant can be found with osinfo-query os which I can't find in Devuan jessie. But the same information is available via Installing Virtual Machines with virt-install, plus copy pastable distro install one-liners.

I chose debian7 as that is probably the closest to Devuan Jessie to be upgraded to Devuan ascii immediately.

The interesting change is that the resulting linux suddenly has virtio networkcards and a disk /dev/vda. That last bit is quite different from earlier virtual machines.

Tags: , ,
2018-04-06 Keeping squid webproxy running for network mismatches 8 months ago
I considered stopping using squid when upgrading to the new homeserver but I have now changed that decision: I need to keep it running for applications that want to do http connections to IPv6-only systems but can't handle those. There are some old scripts running that need it but it's also the way to fix the problem I noticed with linuxcounter.

Tags: , ,
2018-04-06 25 years of Linux use 8 months ago
Powered by Linux In looking at a problem with the linuxcounter script I noticed I am now passing the 25 years with Linux mark. I first saw it in the beginning of 1993 when part of my internship happened at the 'expa' lab (if I recall correctly) of Hogeschool Utrecht with SLS Linux.

Anyway, still using Linux a lot. It's been an interesting 25 years!

Tags: , ,
2018-01-27 I caused an interesting problem with the VDSL pppoe session 10 months ago
Normally being active on certain HF bands causes one-time VDSL disconnects but what I have currently done seems to have triggered something else. After the connection dropped it refuses to come back at the moment. The entire session looks like:
22:49:28.466922 PPPoE PADI [Service-Name]
22:49:28.490394 PPPoE PADO [AC-Name "dr12.d12"] [Service-Name] [AC-Cookie 0xA3FE109A222CE73945C23FCE85E03F83] [EOL]
22:49:28.490603 PPPoE PADR [Service-Name] [AC-Cookie 0xA3FE109A222CE73945C23FCE85E03F83]
22:49:28.517063 PPPoE PADS [ses 0x40c] [Service-Name] [AC-Name "dr12.d12"] [AC-Cookie 0xA3FE109A222CE73945C23FCE85E03F83] [EOL]
22:49:28.575266 PPPoE  [ses 0x40c] LCP, Conf-Request (0x01), id 72, length 16
22:49:28.575776 PPPoE  [ses 0x40c] LCP, Conf-Request (0x01), id 99, length 22
22:49:28.575798 PPPoE  [ses 0x40c] LCP, Conf-Reject (0x04), id 72, length 10
22:49:28.589161 PPPoE  [ses 0x40c] LCP, Conf-Ack (0x02), id 99, length 22
22:49:28.589164 PPPoE  [ses 0x40c] LCP, Conf-Request (0x01), id 73, length 12
22:49:28.589666 PPPoE  [ses 0x40c] LCP, Conf-Ack (0x02), id 73, length 12
22:49:28.589682 PPPoE  [ses 0x40c] LCP, Echo-Request (0x09), id 0, length 10
22:49:28.589693 PPPoE  [ses 0x40c] CCP, Conf-Request (0x01), id 89, length 17
22:49:28.589702 PPPoE  [ses 0x40c] IPCP, Conf-Request (0x01), id 89, length 18
22:49:28.589711 PPPoE  [ses 0x40c] IP6CP, Conf-Request (0x01), id 89, length 16
22:49:28.603265 PPPoE  [ses 0x40c] LCP, Echo-Reply (0x0a), id 0, length 10
22:49:28.603267 PPPoE  [ses 0x40c] LCP, Term-Request (0x05), id 74, length 6
22:49:28.604033 PPPoE  [ses 0x40c] LCP, Term-Ack (0x06), id 74, length 6
22:49:31.623454 PPPoE PADT [ses 0x40c] [Generic-Error "RP-PPPoE: System call error: Input/output error"] [AC-Cookie 0xA3FE109A222CE73945C23FCE85E03F83]
So in the end the router at my ISP decides to terminate the connection. On the connection failing I decided to change the configuration to use the kernel mode pppoe driver but after this started showing I reverted that change. Which made no difference, the connection is still not coming up.

Update: I went looking at other changes I made to enable the pppoe server test and reverting the /etc/ppp/pap-secrets file to its original format fixed the problem. I guess I somehow started to authenticate the remote end.

And changing from user-mode pppoe to kernel-mode pppoe does lower the MTU to 1492, so that test is also finished. Back to user-mode pppoe.

Tags: , , ,
  Older news items for tag linux ⇒
, reachable as PGP encrypted e-mail preferred.

PGP key 5BA9 368B E6F3 34E4 local copy PGP key 5BA9 368B E6F3 34E4 via keyservers pgp key statistics for 0x5BA9368BE6F334E4 Koos van den Hout
Other webprojects: Camp Wireless, wireless Internet access at campsites, The Virtual Bookcase, book reviews