Building - and maintaining - a new home server

Recently during the building of a new home server I decided to make a todo list of things still needing to be done. As the todo list grew and grew, I decided to make a page about my experiences. These are just my experiences and ramblings. Your server will look different.

The server is also known as gosper 3.0. The servername is gosper (this fits with the machine names theme at home and this is the third version. The first one was a 486 dx/2 66, the second one a Pentium-90.

First of all.. the 'new' means I could build, test, build, test and test the new server before I moved the services from the 'old' server. This means I didn't have the problem of having to finish everything on short notice because I wanted to get on-line again. This also means I was able to postpone bits way too much (the net result is that between delivery of hardware and switchover was about half a year).


What functions does my homeserver have

And a bunch of other tasks all having to do with my home network, functions for my home environment or stuff I just like to have.

The hardware

It's all PC hardware. Other hardware (Sun..) is nice, but x86 hardware gives me the most value for money.
First of all I decided on what to reuse and what to buy new.

The operating system

Linux 2.4.x kernel with a Debian 2.2 install. My previous homeserver runs Redhat 5.2 (no, all security leaks have been fixed) but I have grown to like Debian more (I just avoid dselect). Interesting kernel oddities though.. I need at least 2.4.12 to have a working driver for the FA312 and above 2.4.15 I lose the wake-on-lan option.

First.. building the base machine

Just a load of screws.. making the motherboard fit. Adding disk drives. The new case has a handy bay for internal 3,5" harddisks which hangs from the top of the case (nicely ventilated, and accessible without having to disassemble the entire system). The towercase fits into the bottom part of the 19" rack together with the UPS.

Net result after the big move: the case emits noise. More noise then I expected. At the moment there are 4 scsi disks and 2 ide disks in there (old /home is also living in the machine so I can access it if needed). The big scsi cable just fits (it's a bit stretched..). Next on the wishlist is a new scsi disk for /home I guess. All scsi disks in use are 'of age'. I also had to swap the scsi controller with the previous gosper to have an external HD50 connector available for the cd changer. Maybe I need to see if it can run with 2 scsi controllers.

Also, I managed to run out of power connectors. Connected the extra fan (cooling the harddisk drive bay) to a power connector that seems to be for powering video cards or other powerhogs.

Other hardware note: The Teles ISA isdn card only works on irq 10. And that irq is only available when the bios is convinced there is no plug-and-play OS on the machine.

The tapedrive decided to die. So I got a different DDS-2 tapedrive (yes, I am one of the rare people who make backups at home). But that one is giving weird scsi errors. It's probably a bus thing, time for a second SCSI card so I have a separate bus for tapedrive (and I'll probably move the cdrom drives to that SCSI bus too).

One step further (I should have put dates to this whole story) : A second scsi controller, an Adaptec 1542. This means a different driver so I can unload and load this driver when the tape drive is completely wedged without having to reboot the computer.

And /home is now a stripe over 2 4.5G UW disks. I bought them as a pair and this made for a real 2-disk stripe which should hopefully speed things up.


Divide and conquer: partitions .. merge and add: logical volumes

Partition sizes are always wrong (too big or too small).
I used a 4 Gb scsi disk as system disk and divided it into: 230M /, 128M swap, 1.6G /var, 1.4G /usr, 30M /tmp, 470M /usr/local/squid. The separate filesystems give me maximum control over the mount flags for each filesystem. The changed mount flags are: /var (nodev), /tmp (nosuid, nodev), /usr/local/squid/cache (noexec, nosuid, nodev, noatime). This is somewhat restrictive, but on the other hand, there is no need for suid stuff in /tmp so why let all the stupid sploits who create /tmp/rootsh work at all.
The system does not boot from this disk because there is also an IDE disk in this system and both LILO and the bios of the system are very sure that IDE comes before SCSI in the boot order. I decided to give in to this and create a 15M /boot partition on the ide disk.
The rest of the ide disk is a type 8e partition so I can manage that space using logical volume manager. The ide space is all in one logical volume named vgi (I first named it ide but when I started testing with devfs I noticed that /dev/ide/ is for.. ide disks). I then made a logical volume /dev/vgi/scratch with all the available space in it, formatted it as ReiserFS filesystem and mounted it as /scratch. That's where I store stuff that doesn't need to be backed up. The choice for ReiserFS is simple: I need it to be resizable (that's why I'm using lvm) and I don't want to wait several ages for fsck's to complete. ReiserFS is a journalling filesystem which helps to avoid inconsistencies and it's fully resizable. Soon I'll be able to put this to the test when I add another ide disk and add that space to the logical volume.
Disk /dev/sda: 255 heads, 63 sectors, 527 cylinders
Units = cylinders of 16065 * 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/sda1             1       527   4233096    5  Extended
/dev/sda5             1        30    240912   83  Linux
/dev/sda6            31        61    248976   82  Linux swap
/dev/sda7            62       277   1734988+  83  Linux
/dev/sda8           278       461   1477948+  83  Linux
/dev/sda9           462       465     32098+  83  Linux
/dev/sda10          466       527    497983+  83  Linux

Disk /dev/hda: 255 heads, 63 sectors, 7473 cylinders
Units = cylinders of 16065 * 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/hda1   *         1         2     16033+  83  Linux
/dev/hda2             3      7473  60010807+  8e  Unknown
Slight update: added a 80G disk (ide diskspace is dirt cheap these days), tried to create a partition on it using fdisk (which did not believe the size of the disk), switched to cfdisk, created a partition for lvm, added it to the vgi volumegroup and resized the scratch volume to 140G. It took a bit of browsing in the lvm-howto to get the right commands, but I got it to work.

Before:

Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/vgi/scratch      60004564  47071500  12933064  78% /scratch
After:
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/vgi/scratch     140037960  47071500  92966460  34% /scratch
Woohoo! ;-)
The disk according to fdisk:

Disk /dev/hdc: 16 heads, 63 sectors, 27744 cylinders
Units = cylinders of 1008 * 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/hdc1             1    158816  80043232+  8e  Unknown
But according to the startup messages:
hdc: 160086528 sectors (81964 MB) w/2048KiB Cache, CHS=158816/16/63, UDMA(33)
There is a slight difference in number of Cylinders. Cfdisk had no problem.

The good results with lvm and reiserfs made me convert the /home partition to lvm+reiser too. Where I found out the following things: the lvm tools seem to dislike the fact that I use devfs but it's not mounted on /dev (on /devices, because I also work with Solaris). And ReiserFS does not support quota in the stable versions.

Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/vghm/home         4194172   3285772    908400  78% /home
The latest here is that I bought two 4.5G IBM UW-scsi harddisk to form a stripe for the new /home.
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/vgsw/home         8789740     32840   8756900   1% /mnt/home

A new network trick: vlans

I got a 3com 3300XM switch for the very agreeable price of EUR 0.00. The switch supports vlans with 802.1q tagging. And for having more networks in this server and making the home router machine obsolete (one machine less that heats up the room) I started to experiment with the 8021q.o module from Linux on the second network card. I had to patch the source of the netgear module to allow for this.

The latest news: the server is now also home router and talks to the dsl modem and the wireless base station via vlans on the second network card.


Base operating system install

I didn't have a working CD player available to boot the Debian CD from so I created floppy images and booted from those. After that I installed the rest from a local webserver containing the Debian CD's. Although that rest wasn't very big since I just did a base install. Because the Netgear FA312 card didn't work with the Debian installer (kernel revision too old) I used a ne2k-pci card.

After the base install I just started adding packages I need like ..

less, vim, latex, procmail, mgetty-fax, the tex packages, the X client libraries and programs.

But I also compiled stuff myself like squid, sendmail, acpid, php, apache, ntp, mrtg, reiserfsprogs, amanda. Debian versions would be stable (as in 'secured, but not the latest and greatest') and from some things I want 'latest and greatest' and/or be able to tune packages exactly like I want them.


New ways of authorization

I'm experimenting with ldap as way of logging in. So I have set up an ldap server (openldap) and the pam and nss client libraries for ldap userinfo and authentication. I also compiled the ldap, pam and nss client libraries on a Solaris 2.6 system which now uses the same ldap data for authentication.

The 'load' of normal users (the mortal users ;) are now only in ldap and not in /etc/passwd.


Services, services and more services

  • DHCP Most other machines configure themselves using DHCP. And a pool of IP numbers for those visiting laptops or test systems.
  • DNS Resolver for the house, primary for the internal zones.
  • ISDN It logs phone numbers. The plan is to be able to log anyone who calls me, maybe looking up numbers in a database. I might set up ipppd some day but at the moment just logging is enough. Using isdnrep I can create nice overviews of who called when (incoming or outgoing).
  • Sensors Using the lm_sensors package I measure the temperature of the system board and the cpu.
  • Statistics Statistics, and more statistics. MRTG measures network traffic using snmp calls, temperatures using the results from the sensors package and makes nice pages on the webserver. Most stuff is now migrated to rrdtool which offers a lot more flexibility. Graphs from environment sensors at home are available publicly. I also gather the DSL line quality statistics using a bunch of scripts I wrote and create nice 3D graphs.
  • Mail Sendmail as the mta and mutt as mailreader.
  • News inn2 as the newsserver, several newsreaders. And active nntp traffic in a select private hierarchy.
  • ntpd I like my clocks very synched.
  • Fax Just the fax, Ma'am. Using mgetty-fax I have an analog dialin port (connected to an analog port of the ISDN line) which can also detect and receive faxes and spool them to a local dir. I have written a small php script so I can view the faxes (even when "on the road"). Conversion happens using a bit of netpbm trickery.
  • Dialin/dialout that is also possible on the analog modem and on the isdn line.
  • Sms The machine can sms me. It uses sms_client which had to be patched binary because Libertel has been rebranded Vodafone.
  • Web Using php and apache I have a webserver which serves pages for my own use (such as the stats) and pages you can view extern. I am still pondering what to make available as I don't want to diminish my own privacy (that's why the network stats aren't available without authorization.. otherwise it is quite easy to see when I am home). I also share the Debian CD's using http so I can easily use the Debian http access on new installs.
  • NFS/Samba The /home, /camera and /scratch filesystems and all cd's are available to all systems at home.
  • X I have a HP X Terminal which needs the server to boot from and manage it.
  • Backup (amanda) Using amanda, I backup the server itself and all systems on the network that need backups.

    The visible results

    For results I can make visible are
  • The running webserver
  • A picture of the rack with new server (at the bottom) and the old server (the open case next to it)
  • The webcam webserver is also hosted on this server but the camera image is picked up somewhere else at the moment
  • Graphs from several sensors

    History

    I am not fast at fixing things and making them run.. ;)
    Koos van den Hout e-mail: koos@kzdoos.xs4all.nl
    Other webprojects: Camp Wireless The Virtual Bookcase webcam.idefix.net Weather maps