News items for tag computersarebetterat - Koos van den Hout

2011-03-28 (#) 8 years ago
Bit of irritation at work today: several servers to be decommissioned so I wanted to wipe the disks. Armed with 2 dban cd's and one dban bootable usb stick I thought I had everything. But the end result was 0, due to cd players with trouble reading the cd's, drivers missing for newer harddisk controllers and 3 systems not wanting to boot from usb. The newest server saw usb for boot but gave a boot error when trying. Time to bring out the pxeboot pxelinux setup (aka heavy duty boot service). But with a twist: on the big client vlan at work there is a pxe/dhcp setup for centrally managed windows desktop PCs which responds to *all* pxe dhcp requests and not just those from the managed PCs. So I had to move systems to a vlan where this doesn't happen.

It all worked although 2 systems didn't want to boot the dban pxe image. Workaround: Boot the PLD Linux rescue CD which I added to the Heavy Duty Boot Services (screenshot of bootmenu). The PLD Linux rescue cd includes the wipe program.

Tags: , ,
2011-02-22 (#) 8 years ago
Little sysadmin trick: you can prepare modules which are separate from the default modules and everything to be ready when you reboot into your new kernel so everything should be up and running right after the reboot. At least in the Debian / Ubuntu ecosystem.

I just upgraded to kernel version 2.6.24-28-server and before the upcoming reboot I did the following:

# module-assistant -l 2.6.24-28-server prepare

# module-assistant -l 2.6.24-28-server build zaptel

# dpkg -i /usr/src/zaptel-modules-2.6.24-28-server_1.4.10~dfsg-1+2.6.24-28.81_amd64.deb
So post-reboot asterisk will have everything available again. For mISDN I did:
$ make clean

$ KVERS=2.6.24-28-server make

$ sudo KVERS=2.6.24-28-server make install

# depmod modules-2.6.24-28-server -A
And everything is ready for the upcoming reboot. I hope (can't check without said reboot).
Update: Yes, it all worked.

Tags: , ,
2010-12-21 (#) 8 years ago
SSL certificates are one of the more complicated things to keep an eye on: they work fine for 1, 2 or 3 years and suddenly all your users get confronted with very weird errors (which you want to be an error). So this is an ideal candidate for monitoring in zabbix. It is not a check which has to be done every 5 minutes, but even at every 12 hours (the zabbix maximum) I can get enough advance warning about a certificate which is going to expire. Using an external check and a simple script:
#!/usr/bin/perl -w

use strict;
use Date::Parse;

my ($host,$port) = ($ARGV[0],$ARGV[1]);

open(SSLINFO,"echo \"\" | openssl s_client -connect $host:$port 2>/dev/null | openssl x509 -enddate -noout 2>/dev/null |");

my $expiry=0;

while (<SSLINFO>){
        if (/^notAfter=(.+)\n$/){

if ($expiry>0){
        printf "%d\n",($expiry-time())/86400;
} else {
        print "0\n";
The port as parameter allows me to define multiple items, one for https and one for ldaps. The SSL on port 443 check calls external check ssl-expiry-left.monitor[443] which results in a call to /etc/zabbix/externalscripts/ssl-expiry-left.monitor 443. The first call to openssl is to connect to the service and request the certificate and the second one is to parse the certificate and fetch the enddate from the certificate.

I first tried to do this in seconds but three years worth of seconds gave problems. So all value fetching and testing had to be adjusted to work in days. We want an alert when there are less than 30 days left.

Interesting in the result display was that the expiry for some really fresh certificates is displayed as '1.06 kDays' (for 1060 days).

Tags: , ,
2010-11-18 (#) 8 years ago
I installed a Digi CM console server today and the initial setup has two approaches:
From the CD, run the Advance Device Discovery Protocol application to locate the Digi CM on the network. In firmware 1.8 and greater, the CM has SNMP, SSHv1, and Telnet to the command line disabled by default, and http is automatically redirected to https. To perform initial device configuration, use the discovery tool on the CD
or ..
Are you a POWER USER? Use the configuration menu: 1. Establish a serial connection through the console port (9600, 8, N, 1). 2. Login as root and password dbps. 3. Type the command configmenu. Select option 1 - Network Configuration and follow the prompts.
Hardware with the bofh nature. I like it.

Tags: , ,
2010-11-05 (#) 8 years ago
A new trick to learn to our zabbix monitoring setup at work is a check for SSL certificate lifetime. Something you don't have to check very often but it gives a lot of support effort when you are too late.

Based on Monitoring of SSL certificate expiration time - Zabbix forums I did some work but decided that perl was my weapon of choice. Resulting test script:

Script deleted, there is now a better version in updated the zabbix ssl certificate test script to be able to use starttls services.

Which I would like to run once per week. But zabbix wants to run every test at least once every 12 hours. Ok. The test is simple: start warning when the value drops below 2592000 (one month worth of seconds).

The next trick is to use the zabbix api access to get the list of hosts which are monitored for SSL certificates and publish the result in our internal documentation mediawiki. Still working on that bit: selecting hosts in a group and selecting the templates for each host. I want templates per host, because that will show which SSL port is tested.

That is also my first perl JSON programming. Which is just a lot of use of Data::Dumper to figure out what kind of interesting objects are returned and picking out the details I want.

Tags: , ,
2010-11-02 (#) 8 years ago
Now I use the home workstation jobs more for audio work I also want that data in the backup cycle on a regular basis. But backups happen when I sleep and normally the system is off when not in use.

Solution: kick the workstation awake using wake-on-lan. That is easy when I know the ethernet address for the active network card and set the system up right (which does mean the power use is non-zero).

18 5 * * Mon,Fri /usr/sbin/etherwake -i eth0.1 jobs
Nicely in sync with the backup scheme.

But that does mean the system is running which is not needed after the backup is done. But sometimes I leave it running overnight for some computation task(s). Solution, a script on the workstation which runs when the backup should be done at 07:10, somewhat less than 2 hours after the wake:

# check if I'm awaken for backup, in that case: shutdown when there is no
# amanda process

if [ "`awk ' $1 < 7200 { print "ja" } ' < /proc/uptime`" = "ja" ]; then
        if [ "`pidof amandad`" = "" ]; then
                /sbin/shutdown -h now
Only when the number of seconds uptime is less than 7200 (2 hours) and there is no amanda daemon left, the system will shutdown. Scheduled backups working, no needless running of the system and when it has been started with other reasons it won't shut down.
The detour via awk is because the -lt option of test doesn't like fractional numbers.

Tags: , ,
2010-10-27 (#) 8 years ago
I introduced a MediaWiki at work (science ict department) to use for internal documentation. One of the things I wanted to try is pages in the wiki created or maintained from other sources.

I created a special namespace for pages with information from other sources, where normal users have no rights to edit pages. This is to make sure nobody tries to edit something which is maintained by a script from another source.

I started with something simple: the list of printers. The windows printserver is leading, so I want to fetch the list there and massage it to generate a list of printers and comments. The weapon of choice is perl and MediaWiki::Bot. The output of smbclient -N -L printserver takes one regexp to find printqueuenames and descriptions. For the overview of cups queues I can parse the output of lpstat -a. With a bit more digging into IPP it should also be possible to get a list of details of printers to link cups queues and their windows counterparts.

I can run this script from crontab each day and the history tracking in MediaWiki will start to help document when something changed. Another thing which we can stop worrying about.

I have visions of the future of automatically linking zabbix (which has a json interface) and mediawiki and maybe a further future with a good database of stuff which is a source of entries in zabbix and the wiki. Double work is unneeded, computers are much better at working with one canonical source and importing that in a lot of places.

Tags: , , ,
2010-10-22 (#) 8 years ago
If you want to look at your apache logfiles in Hollywood-style, try logstalgia which makes apache look like a sort of pong game. Needs a videocard with OpenGL. And access to the access_log of a busy server.

Tags: , , , ,
2010-10-04 (#) 8 years ago
Implementing a blacklist check in zabbix was a bit more complicated than I originally thought. Searching for it found Monitor DNS blacklist entries - zabbix forums which points at Monitor DNS blacklist entries with Zabbix - Penumbra where shell scripts are suggested. I decided to rewrite stuff in perl using Net::DNS because I wanted a bit more robustness.

Well, as the docs for Net::DNS say:

       "Net::DNS" is slow.
checking 89 blacklists took about 25 seconds. Default external checks in zabbix need to be done in 3 seconds.

As a crummy workaround I now run the checks from cron every 15 minutes which dump the result in a tempfile and the check is on the content of the tempfile, also every 15 minutes. Crude, but effective.

Tags: , ,
2010-09-27 (#) 8 years ago
ISC dhcpd can do interesting stuff, but some options can be quite obscure. Today we suddenly got flooded by alcatel telephones doing DHCP requests in the wrong VLAN. Probably having to do something with moving phones between buildings with different voice vlans. I don't want them to use up dynamic IPs in a production network so I searched a bit and found DHCP for specific range of mac addresses? - Networking. Simple bit of config:
    if binary-to-ascii (16, 8, "-", substring (hardware, 0, 4)) = "1-0-80-9f" {
        deny booting;
Which will deny those 00:80:9f addresses. The first 1 is for hardware class ethernet.

Tags: , , ,
2010-09-20 (#) 8 years ago
I wanted to run Ubuntu desktop from the heavy duty boot service to skip the problems with making CDs. There seems to be no full pxeboot environment for Ubuntu desktop but with the recipe from PXE boot Ubuntu -10.04 - LinuxReaders it was easy to set up a pxeboot + nfs environment. It works, I can boot a system to run the ubuntu livecd or prepare for an installation.

Tags: , , ,
2010-09-17 (#) 8 years ago
I built my first RPM package today. Well ok, I adjusted the specfile and patchfile of an existing RPM package. But now it does what I want and I understood what I was doing.

I needed to write installation instructions for adding servers to zabbix. And the whole configure ; make ; make install ; fix this, fix that, fix ... thing is not very co-worker friendly.

So I did a bit of searching and found Andrew Farley's rpms from a forum post on the zabbix forums: CentOS 5 (and RHEL) RPMs for Zabbix 1.8. I changed a few things, removed the build of server and proxy (not needed in my setup) and started doing runs with rpmbuild. After a few tries (and lots of time for tea waiting for the complete rebuild every time) I had a working setup which would deploy to a server in minutes. A simple 'how to configure and activate' was added in the instructions. And now it saves us work. On to the next project!

In general, having to document something, writing down exactly how something fits together is a great way to start thinking about it, and sometimes finding a lot of room for improving things.

Tags: , ,
2010-08-26 (#) 8 years ago
I like nice images, usually photographs I made myself or interesting images from the Transmission Gallery which has a great set of transmission wallpapers.

I collect a few in a directory, and there is an easy way to show these on an X background when you use xscreensaver anyway:

$ xscreensaver-getimage -directory background -root
This will select a random image from directory 'background' and display it on the root window.

Tags: , ,
2010-05-12 (#) 9 years ago
After a discussion in which I got to quote the Reply-To munging considered harmful I upgraded Ubuntu on my laptop and noticed Thunderbird 3.0.4 recognizes mailing list headers and gives 'Reply', 'Reply all' and 'Reply list' headers depending on what would be correct. Finally!

Tags: , , ,
2010-05-04 (#) 9 years ago
Ok, this one was new to me:
-bash: ./storscript: /bin/bash: bad interpreter: Text file busy
How? the script was copied using scp and there was a hanging sshd (something about a not 100% reliable network).

Tags: , , ,
2010-03-30 (#) 9 years ago
Mirjam recently bought a new laptop and installed Linux on it (sofar nothing special) but we thought it would be nice if mail from the laptop would work from anywhere in the world. Using the information from Relaying with TLS in Sendmail, ubuntu sendmail and a bit of my own thinking this was not very hard. By default ubuntu hides the entire sendmail certificate creation and signing process, and I needed 'better' certificates signed by my own certificate authority. For the client side:
root@machiavelli:/etc/mail/tls# openssl req -new -key sendmail-common.key -out sendmail-client.csr 
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
Country Name (2 letter code) [NL]:
State or Province Name (full name) [Utrecht]:
Locality Name (eg, city) [Utrecht]:
Organization Name (eg, company) []:
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []
Email Address []

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
Next I signed this csr using the CA, and put the resulting client certificate back in /etc/mail/tls/sendmail-client.crt. On the client, /etc/mail/ had to be changed to use tls and talk directly to the right machine:
FEATURE(`msp', `', `MSA')
Now for the server side I also generated a csr for the name and signed it. I changed /etc/mail/ to do this correctly:
dnl #
dnl # fix debian weird choice

define(`confTLS_SRV_OPTIONS', `')dnl
And updated the /etc/mail/access map to relay based on the data from the certificate:
# SSL magic
CERTIssuer:/C=NL/ST=Utrecht/L=Utrecht/     RELAY
Testing it was harder from home which is normally a trusted network.. it just lost that role for a few minutes. And I noticed that when I use mail -v it will ask the upstream mailhost to also be verbose. As noted in the linked article logging is sparse. One hint in the headers of the relayed mail is:
Received: from ( [IPv6:2001:888:1011:1:21f:e1ff:fe45:2894])
        by (8.14.2/8.14.2/Debian-2build1) with ESMTP id o2UKFH9
        (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK)
        for XXXXXXXXXXXXXXX; Tue, 30 Mar 2010 22:15:18 +0200

Tags: , ,
2010-03-19 (#) 9 years ago
I walked in this morning at work with some people looking at me expectingly. About the third person was nice enough to explain: home directories and mail were unavailable. A quick look showed me that the home directory server was waiting for the ldap server and the ldap server showed a kernel panic on the console. Strangely enough the root ldap object was still available so the monitoring system did not notice it.
Anyway, server systems should not wait for the systems administrator after a panic in my opinion, they should be available. So I looked it up, and indeed: Linux Kernel panic reboot explains how simple it is to change this setting. So I changed all servers at work to give up after a panic and reboot. That should help availability. I'm not interested in the intimate details of a panic, I want working ldap. Yes, as several people noted to me, there are ways in which this can lead to a reboot-loop, for example when the panic is file-system related. I'll take that risk when it will 'fix' all other problems.

unix - linux - storage unixfoo is good at linux and netapp knowledge. I browsed it for a while and found lots of interesting stuff.

Tags: , ,
2010-03-18 (#) 9 years ago
Handy unix utility which I had a hard time remembering today: watch. For some reason this is part of procps /proc file system utilities. I used watch to keep an eye on the number of USB storage devices seen by a computer because I was busy hooking up 28 of them at a time to 4 USB hubs and sometimes things were flaky, resulting in the famous usb 7-3.3: device not accepting address 83, error -32. The solution was to unplug and replug the USB device. Trying another hub helped too.
$ watch 'lsusb -t 2>&1| grep -c 0x090c'
This showed the number of USB storage devices (of the type I used) detected so I could plug them in and see whether detection went right.

Tags: , ,
2010-03-04 (#) 9 years ago
After a bit of searching I managed to get my Dell Latitude D630 laptop to use the audio buttons in fvwm.

Tags: , ,
2010-02-09 (#) 9 years ago
I did some serious web services programming (in perl) and updated the scripts powering the Friday Afternoon URL page to post new urls via Twitter on Friday. You can follow @fridayaftURL to get a weekly dose of Friday afternoon stuff from all over the web. The urls are now stored in a (postgresql) database and on Friday a script runs which searches for new urls and posts them to Twitter using the Twitter api. When urls need to be shortened it uses the ln -s_ web service to shorten them.

Tags: , ,
⇐ Newer news items for tag computersarebetterat  Older news items for tag computersarebetterat ⇒
, reachable as PGP encrypted e-mail preferred.

PGP key 5BA9 368B E6F3 34E4 local copy PGP key 5BA9 368B E6F3 34E4 via keyservers pgp key statistics for 0x5BA9368BE6F334E4 Koos van den Hout
Other webprojects: Camp Wireless, wireless Internet access at campsites, The Virtual Bookcase, book reviews