Building - and maintaining - a new home server

Recently during the building of a new home server I decided to make a todo list of things still needing to be done. As the todo list grew and grew, I decided to make a page about my experiences. These are just my experiences and ramblings. Your server will look different.

The server is also known as gosper 3.0. The servername is gosper (this fits with the machine names theme at home and this is the third version. The first one was a 486 dx/2 66, the second one a Pentium-90.

First of all.. the 'new' means I could build, test, build, test and test the new server before I moved the services from the 'old' server. This means I didn't have the problem of having to finish everything on short notice because I wanted to get on-line again. This also means I was able to postpone bits way too much (the net result is that between delivery of hardware and switchover was about half a year).

What functions does my homeserver have

File server .. my home dir, my other shared files (mp3, movies) are on it
Mail server .. mail for kzdoos.xs4all.nl ends up there
News server
Web server
It's the fax
And a load of other stuff

And a bunch of other tasks all having to do with my home network, functions for my home environment or stuff I just like to have.

The hardware

It's all PC hardware. Other hardware (Sun..) is nice, but x86 hardware gives me the most value for money.
First of all I decided on what to reuse and what to buy new.

The motherboard.. a 500 MHz Pentium-III board with 128 Mb memory which came from a workstation which got upgraded in the process.
Later I upgraded from 128 Mb to 256 Mb because the machine sometimes got a bit sluggish when doing a lot at once.
The case.. a brand new bigtower case with a good powersupply and enough drive bays. And hopefully not too noisy. And it looks good too. I decided to buy a bigtower in 'aqua' style (yes, it has a translucent blue bit) which was advertised as being 'noise reducing'.
Diskspace. Reused a 4gig scsi disk I still had, bought a new 60G ide disk. More about that later. Reused other disks.
scsi controller.. Adaptec AHA2940UW also from that workstation. It will need a controller with an external HD50 or C50 connector for the cd-roms and I might want to have 2 scsi controllers in that machine (one for disks and one for tape/cdrom) depending on slot availability when I'm done.
Network cards. Bought 2 Netgear FA312 cards to get maximum speed on a switched 100 mbit network. One is in the new server, one is for the workstation.
Later result was that the workstation got a 3com card from the old homeserver (because the Netgear driver for some other OS did not work very well) and now the server has both Netgear cards.
Keyboard. An old keyboard swiped from a dead machine. Keyboards don't become obsolete and old keyboards aren't cluttered with windows keys.

The operating system

Linux 2.4.x kernel with a Debian 2.2 install. My previous homeserver runs Redhat 5.2 (no, all security leaks have been fixed) but I have grown to like Debian more (I just avoid dselect). Interesting kernel oddities though.. I need at least 2.4.12 to have a working driver for the FA312 and above 2.4.15 I lose the wake-on-lan option.

First.. building the base machine

Just a load of screws.. making the motherboard fit. Adding disk drives. The new case has a handy bay for internal 3,5" harddisks which hangs from the top of the case (nicely ventilated, and accessible without having to disassemble the entire system). The towercase fits into the bottom part of the 19" rack together with the UPS.

Net result after the big move: the case emits noise. More noise then I expected. At the moment there are 4 scsi disks and 2 ide disks in there (old /home is also living in the machine so I can access it if needed). The big scsi cable just fits (it's a bit stretched..). Next on the wishlist is a new scsi disk for /home I guess. All scsi disks in use are 'of age'. I also had to swap the scsi controller with the previous gosper to have an external HD50 connector available for the cd changer. Maybe I need to see if it can run with 2 scsi controllers.

Also, I managed to run out of power connectors. Connected the extra fan (cooling the harddisk drive bay) to a power connector that seems to be for powering video cards or other powerhogs.

Other hardware note: The Teles ISA isdn card only works on irq 10. And that irq is only available when the bios is convinced there is no plug-and-play OS on the machine.

The tapedrive decided to die. So I got a different DDS-2 tapedrive (yes, I am one of the rare people who make backups at home). But that one is giving weird scsi errors. It's probably a bus thing, time for a second SCSI card so I have a separate bus for tapedrive (and I'll probably move the cdrom drives to that SCSI bus too).

One step further (I should have put dates to this whole story) : A second scsi controller, an Adaptec 1542. This means a different driver so I can unload and load this driver when the tape drive is completely wedged without having to reboot the computer.

And /home is now a stripe over 2 4.5G UW disks. I bought them as a pair and this made for a real 2-disk stripe which should hopefully speed things up.

Divide and conquer: partitions .. merge and add: logical volumes

Partition sizes are always wrong (too big or too small).
I used a 4 Gb scsi disk as system disk and divided it into: 230M /, 128M swap, 1.6G /var, 1.4G /usr, 30M /tmp, 470M /usr/local/squid. The separate filesystems give me maximum control over the mount flags for each filesystem. The changed mount flags are: /var (nodev), /tmp (nosuid, nodev), /usr/local/squid/cache (noexec, nosuid, nodev, noatime). This is somewhat restrictive, but on the other hand, there is no need for suid stuff in /tmp so why let all the stupid sploits who create /tmp/rootsh work at all.
The system does not boot from this disk because there is also an IDE disk in this system and both LILO and the bios of the system are very sure that IDE comes before SCSI in the boot order. I decided to give in to this and create a 15M /boot partition on the ide disk.
The rest of the ide disk is a type 8e partition so I can manage that space using logical volume manager. The ide space is all in one logical volume named vgi (I first named it ide but when I started testing with devfs I noticed that /dev/ide/ is for.. ide disks). I then made a logical volume /dev/vgi/scratch with all the available space in it, formatted it as ReiserFS filesystem and mounted it as /scratch. That's where I store stuff that doesn't need to be backed up. The choice for ReiserFS is simple: I need it to be resizable (that's why I'm using lvm) and I don't want to wait several ages for fsck's to complete. ReiserFS is a journalling filesystem which helps to avoid inconsistencies and it's fully resizable. Soon I'll be able to put this to the test when I add another ide disk and add that space to the logical volume.

Disk /dev/sda: 255 heads, 63 sectors, 527 cylinders
Units = cylinders of 16065 * 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/sda1             1       527   4233096    5  Extended
/dev/sda5             1        30    240912   83  Linux
/dev/sda6            31        61    248976   82  Linux swap
/dev/sda7            62       277   1734988+  83  Linux
/dev/sda8           278       461   1477948+  83  Linux
/dev/sda9           462       465     32098+  83  Linux
/dev/sda10          466       527    497983+  83  Linux

Disk /dev/hda: 255 heads, 63 sectors, 7473 cylinders
Units = cylinders of 16065 * 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/hda1   *         1         2     16033+  83  Linux
/dev/hda2             3      7473  60010807+  8e  Unknown

Slight update: added a 80G disk (ide diskspace is dirt cheap these days), tried to create a partition on it using fdisk (which did not believe the size of the disk), switched to cfdisk, created a partition for lvm, added it to the vgi volumegroup and resized the scratch volume to 140G. It took a bit of browsing in the lvm-howto to get the right commands, but I got it to work.

Before:

Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/vgi/scratch      60004564  47071500  12933064  78% /scratch

After:

Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/vgi/scratch     140037960  47071500  92966460  34% /scratch

Woohoo! ;-)
The disk according to fdisk:


Disk /dev/hdc: 16 heads, 63 sectors, 27744 cylinders
Units = cylinders of 1008 * 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/hdc1             1    158816  80043232+  8e  Unknown

But according to the startup messages:

hdc: 160086528 sectors (81964 MB) w/2048KiB Cache, CHS=158816/16/63, UDMA(33)

There is a slight difference in number of Cylinders. Cfdisk had no problem.

The good results with lvm and reiserfs made me convert the /home partition to lvm+reiser too. Where I found out the following things: the lvm tools seem to dislike the fact that I use devfs but it's not mounted on /dev (on /devices, because I also work with Solaris). And ReiserFS does not support quota in the stable versions.

Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/vghm/home         4194172   3285772    908400  78% /home

The latest here is that I bought two 4.5G IBM UW-scsi harddisk to form a stripe for the new /home.

Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/vgsw/home         8789740     32840   8756900   1% /mnt/home

A new network trick: vlans

I got a 3com 3300XM switch for the very agreeable price of EUR 0.00. The switch supports vlans with 802.1q tagging. And for having more networks in this server and making the home router machine obsolete (one machine less that heats up the room) I started to experiment with the 8021q.o module from Linux on the second network card. I had to patch the source of the netgear module to allow for this.

The latest news: the server is now also home router and talks to the dsl modem and the wireless base station via vlans on the second network card.

Base operating system install

I didn't have a working CD player available to boot the Debian CD from so I created floppy images and booted from those. After that I installed the rest from a local webserver containing the Debian CD's. Although that rest wasn't very big since I just did a base install. Because the Netgear FA312 card didn't work with the Debian installer (kernel revision too old) I used a ne2k-pci card.

After the base install I just started adding packages I need like ..

less, vim, latex, procmail, mgetty-fax, the tex packages, the X client libraries and programs.

But I also compiled stuff myself like squid, sendmail, acpid, php, apache, ntp, mrtg, reiserfsprogs, amanda. Debian versions would be stable (as in 'secured, but not the latest and greatest') and from some things I want 'latest and greatest' and/or be able to tune packages exactly like I want them.

New ways of authorization

I'm experimenting with ldap as way of logging in. So I have set up an ldap server (openldap) and the pam and nss client libraries for ldap userinfo and authentication. I also compiled the ldap, pam and nss client libraries on a Solaris 2.6 system which now uses the same ldap data for authentication.

The 'load' of normal users (the mortal users ;) are now only in ldap and not in /etc/passwd.

Services, services and more services

DHCP Most other machines configure themselves using DHCP. And a pool of IP numbers for those visiting laptops or test systems.

DNS Resolver for the house, primary for the internal zones.

ISDN It logs phone numbers. The plan is to be able to log anyone who calls me, maybe looking up numbers in a database. I might set up ipppd some day but at the moment just logging is enough. Using isdnrep I can create nice overviews of who called when (incoming or outgoing).

Sensors Using the lm_sensors package I measure the temperature of the system board and the cpu.

Statistics Statistics, and more statistics. MRTG measures network traffic using snmp calls, temperatures using the results from the sensors package and makes nice pages on the webserver. Most stuff is now migrated to rrdtool which offers a lot more flexibility. Graphs from environment sensors at home are available publicly. I also gather the DSL line quality statistics using a bunch of scripts I wrote and create nice 3D graphs.

Mail Sendmail as the mta and mutt as mailreader.

News inn2 as the newsserver, several newsreaders. And active nntp traffic in a select private hierarchy.

ntpd I like my clocks very synched.

Fax Just the fax, Ma'am. Using mgetty-fax I have an analog dialin port (connected to an analog port of the ISDN line) which can also detect and receive faxes and spool them to a local dir. I have written a small php script so I can view the faxes (even when "on the road"). Conversion happens using a bit of netpbm trickery.

Dialin/dialout that is also possible on the analog modem and on the isdn line.

Sms The machine can sms me. It uses sms_client which had to be patched binary because Libertel has been rebranded Vodafone.

Web Using php and apache I have a webserver which serves pages for my own use (such as the stats) and pages you can view extern. I am still pondering what to make available as I don't want to diminish my own privacy (that's why the network stats aren't available without authorization.. otherwise it is quite easy to see when I am home). I also share the Debian CD's using http so I can easily use the Debian http access on new installs.

NFS/Samba The /home, /camera and /scratch filesystems and all cd's are available to all systems at home.

X I have a HP X Terminal which needs the server to boot from and manage it.

Backup (amanda) Using amanda, I backup the server itself and all systems on the network that need backups.

The visible results

For results I can make visible are

The running webserver

A picture of the rack with new server (at the bottom) and the old server (the open case next to it)

The webcam webserver is also hosted on this server but the camera image is picked up somewhere else at the moment

Graphs from several sensors

History

I am not fast at fixing things and making them run.. ;)

September 1999: Bought a PIII-500 PC as workstation
August 2001: 2 Netgear FA312 cards
September 2001: Case, 60G ide disk. I found a good-looking tower case in the comptechworld webshop.
October 2001: AHA2940UW scsi controller, 4G scsi disk for the /, /var, /usr and /tmp filesystems, 2G scsi disk for /home. Base installation, tuning, more tuning, system takes over as primary home server. Volume group vgi for ide space.
February 2002: 80G ide disk added to vgi volume group
July 2002: 2*4.5 GB scsi disk, set up as logical volume vghm in stripe for /home.
October 2002: Server also becomes router and firewall
December 2002: 160G ide disk, started new logical volume group vgu with bigger extend size, removed unused 2G scsi disk from case.
September 2003: 250G ide disk gets added to the vgu volume group and 60G is removed to keep the whole at 3 ide devices.
December 2004: Netgear network card replaced with a higher revision because of the famous short cable problem.
July 2005: 300G ide disk added with its own ide controller (promise ultra tx2) and added to volume group vgu, 30G becomes /camera (for the archives from my digital camera) and the rest goes to /scratch (now 712G).
December 2005: Added a ups to orderly shutdown the system on power failure.
February 2006: 80G ide disk added to the UDMA100 controller, which takes over as disk for the base os (/, /var, /usr, /tmp) and /home. SCSI disks and controller removed. Power usage goes down, cpu usage (and temperature!) goes up on heavy filesystem access. I also converted /home to ext3 filesystem. All the stories about reiserfs problems have made me decide to go to ext3.
July 2006: redid cooling with a fan sucking out air from around the harddisk bays and another fan sucking in air in the low front. Temperatures in the system were getting too high.
March 2007: Added a second tapedrive. I kept running to the attic to change tapes, so I decided to rescue a tapedrive from a server that was being thrown out at work and see if I could get amanda to recognize and use it. So, the AHA2940UW is back in (the newer tapedrive is ultrawide scsi) to drive both tapedrives (and nothing else: I don't use the cd-changer anymore). The AHA1542 is removed.
April 2007: The ide disk from September 2003 develops a bad sector. I manage to repair it (with a 34 hour reiserfsck). Decided to buy a new disk anyway: the nearest Mycom shop had a 400G disk cheap. Getting the system to use the new disk as boot disk was quite an operation.
May/June 2007: The 160G ide disk from December 2002 develops problems. I bought a 320G disk to replace it. In trying to move data from this disk to other disks and ending reiserfs at the same time the whole logical volume set became corrupt, so I redid the whole thing with old-fashioned partitions.
August 2007: Removed the 160G ide disk from December 2002 and the 80G ide disk from February 2002. The second one because 80G doesn't add much anymore and just uses power.
August 2007: Bought a bunch of dds-3 tapes so the newer tapedrive can do all the work. Even less changing of tapes.
December 2007: Netgear networkcards replaced by an Intel E1000/pro gigabit card. All traffic now goes via vlans.
January 2008: Change of mainboard to an AMD Athlon 1400. More speed and more memory (although the 4th dimm slot on the mainboard seems bad, so 'only' 1536 Mb memory). Removed the old dds-2 tapedrive.
June 2008: Moved postgresql from /scratch to /var which made another disk sleep a lot more.
September 2008: New fan to suck in air at the front, monitored through the sensors system. Notable effect: a colder mainboard. Cable on ide0 channel replaced from a 40-lead to an 80-lead ide cable.

Koos van den Hout e-mail: koos+website@idefix.net
Other webprojects: Camp Wireless