News items for tag linux - Koos van den Hout

I walked in this morning at work with some people looking at me expectingly. About the third person was nice enough to explain: home directories and mail were unavailable. A quick look showed me that the home directory server was waiting for the ldap server and the ldap server showed a kernel panic on the console. Strangely enough the root ldap object was still available so the monitoring system did not notice it.
Anyway, server systems should not wait for the systems administrator after a panic in my opinion, they should be available. So I looked it up, and indeed: Linux Kernel panic reboot explains how simple it is to change this setting. So I changed all servers at work to give up after a panic and reboot. That should help availability. I'm not interested in the intimate details of a panic, I want working ldap. Yes, as several people noted to me, there are ways in which this can lead to a reboot-loop, for example when the panic is file-system related. I'll take that risk when it will 'fix' all other problems.

unix - linux - storage unixfoo is good at linux and netapp knowledge. I browsed it for a while and found lots of interesting stuff.

Handy unix utility which I had a hard time remembering today: watch. For some reason this is part of procps /proc file system utilities. I used watch to keep an eye on the number of USB storage devices seen by a computer because I was busy hooking up 28 of them at a time to 4 USB hubs and sometimes things were flaky, resulting in the famous usb 7-3.3: device not accepting address 83, error -32. The solution was to unplug and replug the USB device. Trying another hub helped too.

$ watch 'lsusb -t 2>&1| grep -c 0x090c'

This showed the number of USB storage devices (of the type I used) detected so I could plug them in and see whether detection went right.

After a bit of searching I managed to get my Dell Latitude D630 laptop to use the audio buttons in fvwm.

I found the probable cause of the not so great power saving: when I installed the first new disk I also updated the bios. And the message I get when trying to load the powernow-k8 cpu driver is:

powernow-k8: Found 1 AMD Athlon(tm) Dual Core Processor 4850e processors (2 cpu cores) (version 2.20.00)
powernow-k8: MP systems not supported by PSB BIOS structure
powernow-k8: MP systems not supported by PSB BIOS structure

So the cpu keeps running at maximum speed without throttling. Searching for the error message finds Ubuntu Bug #33116: powernow-k8 refuses to load and Ubuntu Bug #398109: powernow-k8: Your BIOS does not provide ACPI _PSS objects in a way that Linux understands suggests that I need to check the bios settings to enable "Cool'n'Quiet", enable ACPI APIC and disable MCP61 ACPI HPET Table. That's planned for the next hardware changes.

I noticed that the new Western Digital WD15EADS disk spun down way too fast. After some serious testing I found: when I set the "Advanced Power Management" level (using hdparm -B) to 127 or less the "standby (spindown) timeout" (set using hdparm -S) is ignored and the drive spins down after about 5 8 seconds of inactivity. Way too soon when playing a movie, with mplayer the movie stalls about every 10 seconds because a new bit of movie has to be read from disk which causes another start/stop. The smartctl start/stop counter goes up at the same rate. Feels like a firmware bug to me or a difference of opinion between hdparm and the disk. But the hdparm report suggests that these settings should work on the disk:

ATA device, with non-removable media
        Model Number:       WDC WD15EADS-00S2B0                     
        Firmware Revision:  04.05G04

        Standby timer values: spec'd by Standard, with device specific minimum
        Advanced power management level: 126

           *    Power Management feature set

I asked Western Digital customer help about this but the first (standard?) answer is from Support for WD products in LINUX or UNIX which comes down to "we don't support anything else than jumper settings for these operating systems".

A lot of further searching with google suggests to me that the 'IntelliPark' feature is causing the drive to park its heads after 8 seconds of inactivity which is not a useful default when streaming video from it with a reasonable cache. And the 'Load Cycle Count' will go up fast, which may result in the drive reaching the 'suggested maximum' within a year. I don't need to test the warranty that fast.

As a workaround I set the Advanced Power Management level back to 128 and installed spindown which is a utility which watches the disk activity from userspace and issues a spindown command when no activity (from /proc/diskstats, so for linux at the device level) was measured over the configured period of time. Now it spins down when the filesystems have been idle for 10 minutes which is a lot more usable.
Update: Official answer from Western Digital customer help is that it's not possible to change this 8 second timeout. So I'll stick to the spindown solution.

Filesystems have been moved to the new huge sata disk in home server greenblatt and I found time this evening to remove three old ones. There may be a race condition in the startup scripts where lvm2 is not completely up and running when the filesystems are mounted from the fstab but I saw that happen only once.

The new disk in the homeserver greenblatt was another case of a disk not wanting to go to sleep after the set period. Some searching found two answers: spindown, a daemon to monitor disks for inactivity and spin them down with sg_start --stop or hdparm -y. But the other answer was a better answer: hdparm standby timeout not working for WD raptors? has as answer:

* I also know of quite a number of drives where hdparm -B settings override the -S settings, even if you set the -S settings after the hdparm -B settings. You could try combinations with various values of hdparm -B, especially 1 and 255.

And the manpage of hdparm has this bit:

-B Set Advanced Power Management feature, if the drive supports it. A low value means aggressive power management and a high value means better performance. Possible settings range from values 1 through 127 (which permit spin-down), and values 128 through 254 (which do not permit spin-down). The highest degree of power management is attained with a setting of 1, and the highest I/O performance with a setting of 254. A value of 255 tells hdparm to disable Advanced Power Management altogether on the drive (not all drives support disabling it, but most do).

Default on the WD drives is indeed 128, which does not permit spindown on idle. I changed it to 127, see if that helps. I prefer it if the drives decide for themselves when to spin down.
Update : Yes, the changed advanced power management setting helps, now the drive spins down when not in use.

I played with temporary IPv6 addresses recently, the privacy extension where the right half of the address isn't always the same address derived from the ethernet mac address but a random address. I noticed when I set Linux to use the temporary address as preferred address it was listed as 'secondary':

# ip -6 addr ls
1: lo:  mtu 16436 
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
22: wlan0:  mtu 1500 qlen 1000
    inet6 2001:888:1011:1:10f3:2799:3587:237e/64 scope global secondary dynamic 
       valid_lft 604544sec preferred_lft 85544sec
    inet6 2001:888:1011:1:21f:e1ff:fe45:2894/64 scope global dynamic 
       valid_lft 2591744sec preferred_lft 604544sec
    inet6 fe80::21f:e1ff:fe45:2894/64 scope link 
       valid_lft forever preferred_lft forever

I thought maybe I can use this to fix my outgoing IPv6 address selection problem. Searching for clues how to change the status of an IPv6 address using the ip command I found: IPv6 Source Address Selection on Linux which answers my question completely, and now I can 'block' the tunnel address completely for outgoing connections:

# ip -6 addr ls dev xs4allipv6
7: xs4allipv6@NONE:  mtu 1480 
    inet6 2001:888:1011::13/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 2001:888:10:11::2/64 scope global deprecated 
       valid_lft forever preferred_lft forever
    inet6 fe80::a2a:1401/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::525f:c4ca/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::a2a:201/64 scope link 
       valid_lft forever preferred_lft forever

The tunnel address is 'deprecated' so it will not be used for outgoing connections but the system still responds to it so routing works. Now the wanted address is chosen when I connect to a system 'nearby' in IPv6 address terms:

tcp6       0      0 2001:888:1011::13:41041 2001:888:0:311:194::119 ESTABLISHED

Power failure this morning at work.. which left us not in the dark (enough emergency lighting) but with a completely silent serverroom. When the power came back we had some hours of work to get everything up and running again. Worst problem was with a number of Xen based virtualhosts, some centos upgrade had suddenly created a network device virbr0 which uses NAT and a local dhcp pool and enslaved all xen domU network interfaces under that bridge with no access to the 'real' network because NAT was not set up so their NFS root mount failed. The details on virbr0:

virbr0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet addr:192.168.122.1  Bcast:192.168.122.255

A bit hard to disable, but at the end ifconfig virbr0 down ; brctl delbr virbr0 helps to get rid of the weird bridge, and all domUs will start after that.

Going old-school today: I wrote a sed script to massage grub.conf to add a windows partition on a second disk. Searching google for has this been done before yields loads of page with handholding on how to add windows by hand to a grub.conf generated by anaconda but no simple 'automated' solution. I am always in favor of letting the computer do the boring work. But a bit of thinking and testing and now sed does the job:

if [ -b /dev/sdb1 ]; then

        cp /boot/grub/grub.conf /boot/grub/grub.conf.pre

        sed -e 's/timeout=5/timeout=30/' -e '/hiddenmenu/a\
title Windows XP (Service Pack 3)\
        rootnoverify (hd1,0)\
        map (hd0) (hd1)\
        map (hd1) (hd0)\
        makeactive\
        chainloader (hd1,0)+1
' -e '/hiddenmenu/d' < /boot/grub/grub.conf.pre > /boot/grub/grub.conf
fi

Everybody knows sed -e 's/../../' but I had to look up 'insert', 'append' and 'delete'.
Update 2009-11-12: Changed insert to append because the previous version inserted windows multiple times with multiple linux kernels. Once is enough. Also moved it from the post-install instructions to the post-reboot script so linux is fully configured before windows gets booted.

I brought some more USB sticks to test with and tested the filler script with 4 sticks. Interesting new problem: some USB sticks are partitioned like a harddisk and some aren't, now to find what to mount. Trying to mount everything gives a lot of kernel error messages. Using vol_id was the way to find the valid filesystems. The writing speed is still at maximum when I write 4 in parallel and no USB errors happen.

Ok, discovering 'all USB storage' is not that hard:

ALLSTICKS=`/bin/ls /dev/disk/by-path/*usb*part1 2>/dev/null`

Now for the choice whether to fill them in parallel or serially. With two sticks (the amount I have available at the moment for testing) running two rsync processes in parallel makes the whole script (discover, mount, fill with rsync, unmount) take 27 seconds, waiting for the first rsync to finish before starting the second one takes 35 seconds. Interesting will be how these numbers look when I add more USB sticks.

An interesting project at work: copying a given set of data to as big a number of USB storage devices as possible. So we buy 4 USB hubs, which got delivered today. Connecting them to the 4 different external USB ports on my laptop shows an interesting result:

 lsusb -t
Bus#  7
`-Dev#   1 Vendor 0x0000 Product 0x0000
  |-Dev#  35 Vendor 0x2001 Product 0xf103
  | `-Dev#  36 Vendor 0x0718 Product 0x0075
  |-Dev#  34 Vendor 0x2001 Product 0xf103
  |-Dev#  33 Vendor 0x2001 Product 0xf103
  `-Dev#  32 Vendor 0x2001 Product 0xf103
Bus#  6
`-Dev#   1 Vendor 0x0000 Product 0x0000
Bus#  5
`-Dev#   1 Vendor 0x0000 Product 0x0000
  `-Dev#  44 Vendor 0x0b97 Product 0x7761
    `-Dev#  45 Vendor 0x0b97 Product 0x7772
Bus#  4
`-Dev#   1 Vendor 0x0000 Product 0x0000
Bus#  3
`-Dev#   1 Vendor 0x0000 Product 0x0000
Bus#  2
`-Dev#   1 Vendor 0x0000 Product 0x0000
Bus#  1
`-Dev#   1 Vendor 0x0000 Product 0x0000
  `-Dev#  24 Vendor 0x413c Product 0x8140

Notice it? No? All the high-speed USB hubs (Vendor 0x2001 Product 0xf103) are behind the same root USB hub. Interesting USB congestion problems ahead probably.

My next step will be to discover all attached usb storage (probably thanking udev a lot in the process) and filling that storage with the wanted set of data.

Seen today: #twatch Open Hardware Networked LCD Screen - Slashdot Hardware. Quite tempting! Affordable (USD 45) and it can do cool stuff. By default it displays real-time topic trends from Twitter on an LCD. But with a bit of playing with LCDproc it can also show system statistics, RSS feeds, mail notifications, and more such as ... METARs.

Hope someone of the LCDproc development team picks this up soon: Although the page assumes there is a way to redirect a serial port to a network stream in linux this is a bit harder.

Radio commercials for Linux: yes, it can be done, and they are now airing in Austin, Texas, United States of America. Tux Takes To The Air, with in true open source style an open source radio ad: one can use it and remix it. I like the line:

Why are you still paying for the privilege of using your computer? There's a better way: Linux.

Found via Slashdot: Forkable Linux Radio Ad Now On the Air In Texas

The tapedrive-with-changer on the homeserver found itself in a wedged state with at the bottom of the dmesg output:

[105715.017656] ch 0:0:1:1: Attempting to queue a TARGET RESET message
[105715.017658] CDB: 0x1b 0x20 0x0 0x0 0x2 0x0
[105715.017663] ch 0:0:1:1: Command not found
[105715.017664] aic7xxx_dev_reset returns 0x2002
[105718.936191]  target0:0:1: FAST-10 SCSI 10.0 MB/s ST (100 ns, offset 15)

And still no access to the scsi tape drive. But, there is a bigger hammer nowadays named sg_reset which can fix this:

# mt -f /dev/nst0 status
/dev/nst0: Input/output error
# sg_reset -b /dev/sg0
sg_reset: starting bus reset        
sg_reset: completed bus reset
# mt -f /dev/nst0 status
SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 0 bytes. Density code 0x25 (DDS-3).
Soft error count since last status=0
General status bits on (41010000):
 BOT ONLINE IM_REP_EN

and it's back, not needing a reboot. The list of options says it all:

Usage: sg_reset  [-b] [-d] [-h] [-V] DEVICE
  where: -b       attempt a SCSI bus reset
         -d       attempt a SCSI device reset
         -h       attempt a host adapter reset
         -V       print version string then exit

   {if no switch given then check if reset underway}
To reset use '-d' first, if that is unsuccessful, then use '-b', then '-h'

Today I decided to put some time in trying to use a different driver for using the openvox B200P card in the ubuntu installed asterisk in home server greenblatt. mISDN has a few stability issues in this setup (ubuntu 8.04 LTS amd64, asterisk 1.4.17 from ubuntu source recompiled for misdn support) where I can't unload the driver (instant kernel warning message and module system wedged) and sometimes after a long while the internal channel got confused and rejected calls, needing a restart of Asterisk.
So I tried the other route: the qozap driver with bristuff patches which comes with ubuntu as package zaptel-source. Configuring this driver was a bit of a puzzle. Step one: don't load ztdummy because that will confuse the channel ordering. TE / NT choice is done by the module parameter ports for the module, the bitmapped value of the TE / NT configuration. In my case I created /etc/modprobe.d/zaptel with:

 
options qozap ports=2

So the second port is switched to NT mode, which matches the jumper setup. The working samples are documented at ZaptelBRI - voip-info.org.
The qozap driver sort of works on the TE side (the side facing the phone company). Diving deeper into docs from openvox showed that the openvox cards need a slightly changed qozap driver source from http://downloads.openvox.cn/pub/drivers/bristuff/patches/. It helps to find that the version of bristuffed in Ubuntu 8.04 is probably 0.4.0.
The current state is working up to a level: I can dial numbers from the internal isdn phone to the external isdn line and I can accept calls, but trying to get dialtone from asterisk results in:

    -- Extension 's' in context 'internte' from 'xxxxxxx' does not exist.  Rejecting call on channel 0/2, span 2

which looks like the problem mentioned in Re: [Asterisk-Users] zaphfc problem: overlapdial don't work after update bristuff but that change (and a complete recompile of my asterisk package) did not do the work. Browsing the source of chan_zap shows a lot of places where it can decide to switch to 's'.

I updated my PXElinux boot menu with a few changes to the pxelinux.cfg/default so there are a number of handy extra booting options:

DEFAULT vesamenu.c32
PROMPT 0

TIMEOUT 100

MENU TITLE Heavy Duty Boot Service
MENU BACKGROUND boot.jpg
# http://www.flickr.com/photos/bottinex/2948585175/

LABEL local
MENU LABEL ^Boot from local disk (default)
        localboot 0

LABEL memtest
MENU LABEL ^Memory test
                kernel memtest86

LABEL pldrescue
MENU LABEL ^PLD Linux rescue
        KERNEL pld-20090221/boot/isolinux/vmlinuz
        APPEND initrd=pld-20090221/rescue.cpi,pld-20090221/custom/custom.cpi root=/dev/ram0 CONF=”`/dev/fd0:/rescue`;;;;;;;;;;;”
        IPAPPEND 1

LABEL ubuntu-i386
MENU LABEL Ubuntu i386 menu
        KERNEL menu.c32
        APPEND ubuntu-installer/ubuntu-i386.cfg

LABEL ubuntu-amd64
MENU LABEL Ubuntu amd64 menu
        KERNEL menu.c32
        APPEND ubuntu-installer/ubuntu-amd64.cfg

LABEL cpuid
MENU LABEL ^Identify Processor
        KERNEL cpuidtest.c32 

The ubuntu submenus are copied from the ubuntu CD's and slightly adjusted to the local situation:

MENU TITLE Ubuntu i386 menu
DISPLAY ubuntu-installer/i386/boot-screens/boot.txt

LABEL install
        kernel ubuntu-installer/i386/linux
        append vga=normal initrd=ubuntu-installer/i386/initrd.gz -- 
LABEL linux
        kernel ubuntu-installer/i386/linux
        append vga=normal initrd=ubuntu-installer/i386/initrd.gz -- 
LABEL cli
        kernel ubuntu-installer/i386/linux
        append tasks=standard pkgsel/language-pack-patterns= pkgsel/install-language-support=false vga=normal initrd=ubuntu-installer/i386/initrd.gz -- 

LABEL expert
        kernel ubuntu-installer/i386/linux
        append priority=low vga=normal initrd=ubuntu-installer/i386/initrd.gz -- 
LABEL cli-expert
        kernel ubuntu-installer/i386/linux
        append tasks=standard pkgsel/language-pack-patterns= pkgsel/install-language-support=false priority=low vga=normal initrd=ubuntu-installer/i386/initrd.gz -- 

LABEL rescue
        kernel ubuntu-installer/i386/linux
        append vga=normal initrd=ubuntu-installer/i386/initrd.gz rescue/enable=true -- 

LABEL mainmenu
        MENU LABEL Return to main menu
        KERNEL vesamenu.c32
        APPEND ~

The PLD Linux rescue CD is ofcourse based on PXE remote boot for your home/work lab. I tried the Ubuntu 'rescue' mode once but the PLD linux rescue environment works a lot better for me.

A bit of reading on pxelinux and syslinux and it looks easy to set up my own pxeboot server at home so I don't have to fiddle with floppies, cd-roms or other media anymore.

First of all I wanted the dhcp server to only respond with pxelinux to real pxe clients. There is other stuff which uses dhcp and tftp and might get confused by pxelinux.0 being offered as boot image such as VoIP phones. That can be done with the following in the right context in dhcpd.conf:

    if substring (option vendor-class-identifier, 0, 9) = "PXEClient" {
        filename "/pxelinux.0";
        next-server 10.42.2.1;
    }

Next a tftp server is needed, I installed atftpd serving /tftpboot.

The next part is the menu structure for pxelinux. The following files are in /tftpboot:

root@greenblatt:/tftpboot# ls -lR
.:
total 628
-rw-r--r-- 1 root root  30920 2009-09-01 21:20 boot.jpg
-rw-r--r-- 1 root root 307200 2009-09-01 20:06 boot.png
-rw-r--r-- 1 root root 103204 2009-09-01 21:23 memtest86
-rw-r--r-- 1 root root  35732 2009-09-01 18:29 menu.c32
-rw-r--r-- 1 root root  14146 2009-09-01 17:57 pxelinux.0
drwxr-xr-x 2 root root   4096 2009-09-01 21:34 pxelinux.cfg
-rw-r--r-- 1 root root 122988 2009-09-01 18:29 vesamenu.c32

./pxelinux.cfg:
total 8
-rw-r--r-- 1 root root 117 2009-09-01 18:02 bootmenu.txt
-rw-r--r-- 1 root root 463 2009-09-01 21:34 default

And the content of pxelinux.cfg/default is:

DISPLAY bootmenu.txt

DEFAULT vesamenu.c32
PROMPT 0

TIMEOUT 100

MENU TITLE Remote Boot Services
MENU BACKGROUND boot.jpg
# http://www.flickr.com/photos/bottinex/2948585175/

LABEL local
MENU LABEL ^Boot from local disk
        localboot 0

LABEL memtest
MENU LABEL ^Memory test
                kernel memtest86

Note that the memtest86+.bin image is renamed memtest86: the original filename did not want to boot.

The image of the heavy duty boot is rotated and scaled to 640x480 and brought down in quality. I will probably add the PLD linux rescue CD image and ubuntu so I won't be bothered by boot medium problems anymore. An environment for a heavy duty boot, hence the image.

Friday afternoon project: trying to make my Palm Tungsten E2 and my Ubuntu laptop talk IP to eachother. It took a bit of searching, but now it is working. It's one of those areas where one should not be afraid of a reboot, the incoming 'LAN access using PPP' kept being rejected until I rebooted the machine. Details added at The Dell Latitude D630 laptop and linux. The PalmOS webbrowser, Blazer, does show up with a weird user-agent: "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; PalmSource/Palm-Zir4; Blazer/4.0) 16;320x320"