News items for tag linux - Koos van den Hout

I like nice images, usually photographs I made myself or interesting images from the Transmission Gallery which has a great set of transmission wallpapers.

I collect a few in a directory, and there is an easy way to show these on an X background when you use xscreensaver anyway:

$ xscreensaver-getimage -directory background -root

This will select a random image from directory 'background' and display it on the root window.

With ubuntu 8.04 server on the home server greenblatt I got a daily mail:

Subject: Cron  test -x /usr/sbin/anacron || ( cd / &&          
        run-parts --report /etc/cron.daily )                                    

/etc/cron.daily/logrotate:                                                      
Re-opening all log files                                   
Re-opening all log files                                                        
Re-opening all log files                                               

And I couldn't really find the source. But a google search for logrotate mail 're-opening' helps: It is caused by logrotate and mailman, filed as Bug #244233 in mailman (Ubuntu): “Logrotate is noisy with: Re-opening all log files. The fix is simple: make mailman be quiet in /etc/logrotate.d/mailman. A patch is attached to the ubuntu bug.

As part of the work on system monitoring I am looking into monitoring RAID units. The beta-ict department uses a number of raid units and data gets replicated between buildings.

I want a warning when a disk goes down. The 3ware disk controller has a nice webinterface but I can't integrate that (easily..) into zabbix. What I did was install the tw_cli command line utility from the 3ware LSI raid controller site (lookup your type of controller, find 'support and downloads' and you will see cli utils for lots of unix versions), which makes life easy:

# tw_cli show

Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
------------------------------------------------------------------------
c0    9650SE-16ML  16        15       1       1       1       1      OK

What I want to know is the number of not-optimal disks (yes, indeed one is broken at the moment and needs replacement). That I can monitor in zabbix, when I pick up the value with a script:

#!/bin/sh
# /etc/zabbix/external/3ware.okdisk

sudo /usr/local/sbin/tw_cli show | awk ' /^c0/ { print $6 } '

Root access via sudo which means a line in /etc/sudoers which allows /usr/local/sbin/tw_cli from the zabbix user, and the right setting in zabbix_agentd.conf to bind this script to a user parameter:

UserParameter=3ware.okdisk,/etc/zabbix/external/3ware.okdisk

Now I can program a trigger on the output: 0 is ok, 1 is warning, > 1 is disaster. I added an extra action on the trigger to mail the output of tw_cli '/c0 show' to the admins so we know which disk is broken.

Now to do the same for adaptec (aacraid) based raids.

I wanted to install an extra package on the wardriving box but found out that the choice of distribution: Debian etch is not available anymore, not even as 'oldstable'. A bit of searching finds that I need to look in the Debian distribution archives.

I'm not sure whether I'll keep using Debian versions for the wardrive-box. I want something nice and small and manageable, and the option for a custom kernel (no initrd, preferably no udev).

The interesting bit is that I built the wardriving box in January - February 2008 and it basically ran regularly since that time without software problems.

The extra package I wanted to try is lm_sensors, for the other project: Sundial. I was wondering how high/low the system temperature would get, and whether it would stay within the 0 - 50 ⁰C range. An IP55 rated case might be a good idea for use in the garden shed (which is semi-outdoors). The question is will the mainboard stay above 0 ⁰C when it is -15 ⁰C outside. I know from the wardriving box the Alix board generates some heat, but is it enough to keep itself warm.

The Alix.1c/1d have a temperature sensor, according to Getting started with voyage linux it should work with the w83627hf driver which indeed loads and gives a readout.

I came across the alix.1 series hardware while looking for something low-power for project sundial. Later calculations showed the 'powered by the sun or wind' part of that project would be too expensive compared to just using a plug.

Update: The archived etch works, but lm-sensors wants perl, which is not part of the stripped down debian on the wardrive box. For as far as I can see that is because there is one perl script included. Time to rebuild from source with that script removed.

Update 2010-08-03: Looking where you are going helps too: Voyage Linux is Debian-based but optimized for embedded apps. With debootstrap under Ubuntu or Debian I could set up a newer development environment for the wardrive box and test Voyage Linux.

After a discussion in which I got to quote the Reply-To munging considered harmful I upgraded Ubuntu on my laptop and noticed Thunderbird 3.0.4 recognizes mailing list headers and gives 'Reply', 'Reply all' and 'Reply list' headers depending on what would be correct. Finally!

Ok, this one was new to me:

-bash: ./storscript: /bin/bash: bad interpreter: Text file busy

How? the script was copied using scp and there was a hanging sshd (something about a not 100% reliable network).

I like htop as a nicer looking replacement for top. And on one machine I recently noticed this little gem: the uptime has an exclamation mark when it runs over 100 days. And a bit of searching confirms: htop has a real easter egg, confirmed by the author.

A kernel panic because of some tweaking in the ISDN driver made it time for event-driven maintainance on the home server greenblatt. So I shut down the server, did the last syncs and removed the remaining two parallel ata disks and the promise IDE controller which was no longer needed. I re-enabled the "Cool'n'quiet" bios option so linux power saving works again, which should result in a drop in power use.
Update: As the graph shows removing the disks and re-enabling powersave has helped reducing power use.

Werkend IPv6 via zowel de tunnel als via de native aansluiting. Ik hou nog even de tunnel aan omdat die een voorspelbare v6 range heeft en ik daarin services heb die extern bereikbaar moeten zijn zoals deze website:

$ dig +short idefix.net aaaa
2001:888:1011::694

En nog veel lastiger om te wijzigen:

   Server Name: NS2.IDEFIX.NET
   IP Address: 2001:888:1011:0:0:0:0:694
   IP Address: 82.95.196.202
   Registrar: EPAG DOMAINSERVICES GMBH

Ik heb er dus voor gekozen wireless clients adressen uit de native range te geven en te routeren via die link, en de server en de wired aansluitingen nog via de v6 tunnel. Daar is rulebased routing voor nodig, en omdat ik verbindingen vanaf de server nog via de tunnel wil is 'native' de uitzondering voor mij. Eerst probeerde ik 'niet tunneladres' als regel:

ip -6 rule add type unicast not from 2001:888:1011::/48 priority 32765 table 17

En dan de routing in /etc/ppp/ipv6-up.d/default :

ip -6 route add default dev ${PPP_IFACE} table 17

Maar dat werkte niet: verkeer vanaf de server naar de wireless clients werd ook de ppp0 link opgestuurd, dat werkte dus niet. Dat was weer te omzeilen door een regel voor het wireless netwerk in table 17 te dupliceren, maar dat is niet handig bij eventuele wijzigingen van de IPv6 range.
Beter: een filter maken wat 'native' verkeer herkent. Een beetje voorkennis helpt: in de xs4all native dsl proef krijgen klanten een /48 uit 2001:980::/32. Dus dan in /etc/ppp/ipv6-up.d/default :

ip -6 rule add type unicast from 2001:980::/32 priority 32765 table 17
ip -6 route add default dev ${PPP_IFACE} table 17 || true

De gewone ipv6 default route komt uit /etc/network/interfaces :

auto xs4allipv6
iface xs4allipv6 inet6 v4tunnel
    endpoint 194.109.5.241
    address 2001:888:10:11::2
    netmask 64
    up ip tunnel change xs4allipv6 ttl 64
    up ip -6 addr add 2001:888:1011::13/128 dev xs4allipv6
    up ip -6 route add unreachable 2001:888:1011::/48
    # rest /48 nullrouten, specifiekere routes hebben voorrang
    up ip -6 route add default via 2001:888:10:11::1 src 2001:888:1011::13
    up ip -6 addr change 2001:888:10:11::2/64 dev xs4allipv6 preferred_lft 0
	# tunneladres nooit als uitgaand adres gebruiken
    #
    down ip -6 route del unreachable 2001:888:1011::/48
    # maar dan moet ik ook die nullroute weer weghalen :)

En nu werkt het voor beide IPv6 ranges. Native:

$ traceroute6 abaris
traceroute to abaris.idefix.net (2001:470:1f15:db:131:211:84:204) from 2001:980:111b:2:21f:e1ff:fe45:2894, port 33434, from port 63747, 30 hops max, 60 byte packets
 1  2001:980:111b:2:21f:c6ff:fe59:76f6 (2001:980:111b:2:21f:c6ff:fe59:76f6)  1.517 ms  1.773 ms  5.713 ms 
 2  lo1.dr4.1d12.xs4all.net (2001:888:0:4401::1)  37.795 ms  39.791 ms  20.754 ms 
 3  2001:888:0:4403::2 (2001:888:0:4403::2)  30.975 ms  17.414 ms  20.025 ms 
 4  0.ge-1-2-0.xr1.sara.xs4all.net (2001:888:2:2::1)  19.104 ms  21.797 ms  26.274 ms 
 5  10gigabitethernet3-3.core1.ams1.he.net (2001:470:0:e8::1)  24.385 ms  17.858 ms  17.255 ms 
 6  gige-gbge0.tserv11.ams1.ipv6.he.net (2001:470:0:7d::2)  24.295 ms  24.621 ms  23.004 ms 
 7  abaris.idefix.net (2001:470:1f15:db:131:211:84:204)  30.448 ms  27.849 ms  21.872 ms 

En tunnel:

$ traceroute6 abaris
traceroute to abaris.idefix.net (2001:470:1f15:db:131:211:84:204) from 2001:888:1011::13, port 33434, from port 53569, 30 hops max, 60 byte packets
 1  xs4all17.ipv6.xs4all.nl (2001:888:10:11::1)  17.177 ms  16.589 ms  17.447 ms 
 2  104.ae0.xr4.1d12.xs4all.net (2001:888:0:3::1)  25.176 ms  22.599 ms  15.982 ms 
 3  0.ge-0-2-0.xr1.sara.xs4all.net (2001:888:2:1::1)  22.124 ms  19.112 ms  19.868 ms 
 4  10gigabitethernet3-3.core1.ams1.he.net (2001:470:0:e8::1)  19.501 ms  16.943 ms  15.742 ms 
 5  gige-gbge0.tserv11.ams1.ipv6.he.net (2001:470:0:7d::2)  24.070 ms  20.348 ms  19.108 ms 
 6  abaris.idefix.net (2001:470:1f15:db:131:211:84:204)  25.500 ms  24.720 ms  21.881 ms 

De juiste data in radvd.conf komt uit een slim scriptje wat net nadat wide-dhcp6c de interfaces geconfigureerd heeft toeslaat. Op het moment dat het script /etc/wide-dhcpv6/dhcp6c-script uitgevoerd wordt is iets te vroeg, dus ik heb daar in staan:

RADVDCONF=/etc/radvd.conf

genradvdconf ()
{
    echo > $RADVDCONF
    for IFACE in "$@"
    do  
        /etc/wide-dhcpv6/iface2radvd.sh $IFACE >> $RADVDCONF
    done
}

( sleep 5 ; genradvdconf eth0.1 eth0.3 ; /etc/init.d/radvd reload ) &

En even doordenkend: als ik ooit iets probeer te benaderen bij een andere xs4all-v6-native klant heb ik inderdaad een probleem (met het ontvangen van de antwoorden). Deze oplossing is dus niet blijvend. Maar als IPv6 over dsl gewoon wordt dan hoef ik niet meer met die tunnel bezig te zijn.

With all the IPv6 configuration work something bothers me in ifconfig, it can still decide to stop showing IPv6 (inet6) addresses:

# ifconfig eth0.1
eth0.1    Link encap:Ethernet  HWaddr 00:1f:c6:59:76:f6  
          inet addr:10.42.2.1  Bcast:10.42.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:17245335 errors:0 dropped:0 overruns:0 frame:0
          TX packets:17342308 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:10403720060 (9.6 GB)  TX bytes:17783892122 (16.5 GB)

The addresses are there:

# ip -6 addr ls dev eth0.1
3: eth0.1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 
    inet6 2001:888:1011::694/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::21f:c6ff:fe59:76f6/64 scope link 
       valid_lft forever preferred_lft forever

Filed as Ubuntu bug 560807: ifconfig does not display inet6 addresses.
Update 2010-04-13: Problem source found, diff added to the bugreport.

Ok, sendmail locally on the laptop now works, but the next step is to get this to work from Thunderbird. I want Thunderbird to present the certificate to use the outgoing relay. To import a certificate into Thunderbird, I need to convert it to pkcs#12 format:

$ openssl pkcs12 -export -inkey thunderbird.pem -in machiavelli.idefix.net.crt -out thunderbird.p12 -name "Thunderbird op Machiavelli"
Enter pass phrase for thunderbird.pem:
Enter Export Password:
Verifying - Enter Export Password:

This gives me a .p12 file that Thunderbird understands so I can import it. But even when I set up Thunderbird to send the mail to the correct server on port 587 with TLS enabled it does not present its own certificate, resulting in a 'relaying denied' message (which is exactly what I want when the certificate is missing!). Time to search further. Maybe something in the certificate needs to match, but searching the Thunderbird help and on-line doesn't give hints.
As a backup I can let Thunderbird send to localhost:smtp and use the sendmail installation there to send it to the server (which it can) but then I can't see whether that worked when I close Thunderbird and disconnect the laptop. I'd rather have some visual conformation that mail is out the door.

Mirjam recently bought a new laptop and installed Linux on it (sofar nothing special) but we thought it would be nice if mail from the laptop would work from anywhere in the world. Using the information from Relaying with TLS in Sendmail, ubuntu sendmail and a bit of my own thinking this was not very hard. By default ubuntu hides the entire sendmail certificate creation and signing process, and I needed 'better' certificates signed by my own certificate authority. For the client side:

root@machiavelli:/etc/mail/tls# openssl req -new -key sendmail-common.key -out sendmail-client.csr 
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [NL]:
State or Province Name (full name) [Utrecht]:
Locality Name (eg, city) [Utrecht]:
Organization Name (eg, company) [idefix.net]:
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:machiavelli.idefix.net
Email Address []:koos@machiavelli.idefix.net

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Next I signed this csr using the idefix.net CA, and put the resulting client certificate back in /etc/mail/tls/sendmail-client.crt. On the client, /etc/mail/submit.mc had to be changed to use tls and talk directly to the right machine:

FEATURE(`msp', `postbode.idefix.net', `MSA')
include(`/etc/mail/tls/starttls.m4')dnl

Now for the server side I also generated a csr for the name postbode.idefix.net and signed it. I changed /etc/mail/sendmail.mc to do this correctly:

include(`/etc/mail/tls/starttls.m4')dnl
dnl #
dnl # fix debian weird choice

define(`confTLS_SRV_OPTIONS', `')dnl

And updated the /etc/mail/access map to relay based on the data from the idefix.net certificate:

# SSL magic
CERTIssuer:/C=NL/ST=Utrecht/L=Utrecht/O=idefix.net/OU=Certificate+20Authority/CN=idefix.net+20CA/emailAddress=hostmaster@idefix.net     RELAY

Testing it was harder from home which is normally a trusted network.. it just lost that role for a few minutes. And I noticed that when I use mail -v it will ask the upstream mailhost to also be verbose. As noted in the linked article logging is sparse. One hint in the headers of the relayed mail is:

Received: from machiavelli.idefix.net (wireless-machiavelli.idefix.net [IPv6:2001:888:1011:1:21f:e1ff:fe45:2894])
        by kzdoos.xs4all.nl (8.14.2/8.14.2/Debian-2build1) with ESMTP id o2UKFH9
X002890
        (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK)
        for XXXXXXXXXXXXXXX; Tue, 30 Mar 2010 22:15:18 +0200

Teken dat het echt slecht gaat met de accu van mijn Dell D630 laptop:

[   20.075680] ACPI: Battery Slot [BAT0] (battery absent)
[   20.075762] ACPI: Battery Slot [BAT1] (battery absent)

Volgens acpi (en volgens de dell setup) zit er helemaal geen accu in. En dat terwijl ik nog een screenshot wilde maken van de gnome battery applet die beweerde dat de accu aan vervanging toe was.

I walked in this morning at work with some people looking at me expectingly. About the third person was nice enough to explain: home directories and mail were unavailable. A quick look showed me that the home directory server was waiting for the ldap server and the ldap server showed a kernel panic on the console. Strangely enough the root ldap object was still available so the monitoring system did not notice it.
Anyway, server systems should not wait for the systems administrator after a panic in my opinion, they should be available. So I looked it up, and indeed: Linux Kernel panic reboot explains how simple it is to change this setting. So I changed all servers at work to give up after a panic and reboot. That should help availability. I'm not interested in the intimate details of a panic, I want working ldap. Yes, as several people noted to me, there are ways in which this can lead to a reboot-loop, for example when the panic is file-system related. I'll take that risk when it will 'fix' all other problems.

unix - linux - storage unixfoo is good at linux and netapp knowledge. I browsed it for a while and found lots of interesting stuff.

Handy unix utility which I had a hard time remembering today: watch. For some reason this is part of procps /proc file system utilities. I used watch to keep an eye on the number of USB storage devices seen by a computer because I was busy hooking up 28 of them at a time to 4 USB hubs and sometimes things were flaky, resulting in the famous usb 7-3.3: device not accepting address 83, error -32. The solution was to unplug and replug the USB device. Trying another hub helped too.

$ watch 'lsusb -t 2>&1| grep -c 0x090c'

This showed the number of USB storage devices (of the type I used) detected so I could plug them in and see whether detection went right.

After a bit of searching I managed to get my Dell Latitude D630 laptop to use the audio buttons in fvwm.

I found the probable cause of the not so great power saving: when I installed the first new disk I also updated the bios. And the message I get when trying to load the powernow-k8 cpu driver is:

powernow-k8: Found 1 AMD Athlon(tm) Dual Core Processor 4850e processors (2 cpu cores) (version 2.20.00)
powernow-k8: MP systems not supported by PSB BIOS structure
powernow-k8: MP systems not supported by PSB BIOS structure

So the cpu keeps running at maximum speed without throttling. Searching for the error message finds Ubuntu Bug #33116: powernow-k8 refuses to load and Ubuntu Bug #398109: powernow-k8: Your BIOS does not provide ACPI _PSS objects in a way that Linux understands suggests that I need to check the bios settings to enable "Cool'n'Quiet", enable ACPI APIC and disable MCP61 ACPI HPET Table. That's planned for the next hardware changes.

I noticed that the new Western Digital WD15EADS disk spun down way too fast. After some serious testing I found: when I set the "Advanced Power Management" level (using hdparm -B) to 127 or less the "standby (spindown) timeout" (set using hdparm -S) is ignored and the drive spins down after about 5 8 seconds of inactivity. Way too soon when playing a movie, with mplayer the movie stalls about every 10 seconds because a new bit of movie has to be read from disk which causes another start/stop. The smartctl start/stop counter goes up at the same rate. Feels like a firmware bug to me or a difference of opinion between hdparm and the disk. But the hdparm report suggests that these settings should work on the disk:

ATA device, with non-removable media
        Model Number:       WDC WD15EADS-00S2B0                     
        Firmware Revision:  04.05G04

        Standby timer values: spec'd by Standard, with device specific minimum
        Advanced power management level: 126

           *    Power Management feature set

I asked Western Digital customer help about this but the first (standard?) answer is from Support for WD products in LINUX or UNIX which comes down to "we don't support anything else than jumper settings for these operating systems".

A lot of further searching with google suggests to me that the 'IntelliPark' feature is causing the drive to park its heads after 8 seconds of inactivity which is not a useful default when streaming video from it with a reasonable cache. And the 'Load Cycle Count' will go up fast, which may result in the drive reaching the 'suggested maximum' within a year. I don't need to test the warranty that fast.

As a workaround I set the Advanced Power Management level back to 128 and installed spindown which is a utility which watches the disk activity from userspace and issues a spindown command when no activity (from /proc/diskstats, so for linux at the device level) was measured over the configured period of time. Now it spins down when the filesystems have been idle for 10 minutes which is a lot more usable.
Update: Official answer from Western Digital customer help is that it's not possible to change this 8 second timeout. So I'll stick to the spindown solution.

Filesystems have been moved to the new huge sata disk in home server greenblatt and I found time this evening to remove three old ones. There may be a race condition in the startup scripts where lvm2 is not completely up and running when the filesystems are mounted from the fstab but I saw that happen only once.

The new disk in the homeserver greenblatt was another case of a disk not wanting to go to sleep after the set period. Some searching found two answers: spindown, a daemon to monitor disks for inactivity and spin them down with sg_start --stop or hdparm -y. But the other answer was a better answer: hdparm standby timeout not working for WD raptors? has as answer:

* I also know of quite a number of drives where hdparm -B settings override the -S settings, even if you set the -S settings after the hdparm -B settings. You could try combinations with various values of hdparm -B, especially 1 and 255.

And the manpage of hdparm has this bit:

-B Set Advanced Power Management feature, if the drive supports it. A low value means aggressive power management and a high value means better performance. Possible settings range from values 1 through 127 (which permit spin-down), and values 128 through 254 (which do not permit spin-down). The highest degree of power management is attained with a setting of 1, and the highest I/O performance with a setting of 254. A value of 255 tells hdparm to disable Advanced Power Management altogether on the drive (not all drives support disabling it, but most do).

Default on the WD drives is indeed 128, which does not permit spindown on idle. I changed it to 127, see if that helps. I prefer it if the drives decide for themselves when to spin down.
Update : Yes, the changed advanced power management setting helps, now the drive spins down when not in use.