Spammers and dealing with the aftermath

Today I got an e-mail from a guy somewhere "I got this spamcop report and
I'm not really sure what this says.. looks like our webserver is sending
spam!"

So, I logged into that machine, became root (something with old accounts
never getting cleaned up) and looked in the cgi-bin directory for my
favorite web-spamming script:

---------- 1 root other 13559 May 19 1999 FormMail.pl

That's after I did a 'chmod 0 FormMail.pl'. Indeed, that was the culprit
of the webserver starting to spam. After that I tried to get an idea of
the health of the mail-queue. It took the 'mailq' command way too long to
get any idea of the state of the mail-queue, so I did

:/var/spool/mqueue# echo * | wc -w

which came somewhere over 6700. I stopped the mail daemons and tried mailq
again, this time just to get an idea whether there was any legitimate
mail in there (from or to the owners of the machine). There wasn't.
The directory entry for the mail-queue had grown to 300 kilobyte. This on
a Solaris 2.6 machine with UFS which means that that directory entry
is hosed.. or at least very slow. I did an rm -rf on the directory mqueue,
remade it after that was done and restarted the mail system.

After mail was flowing again, I had a look in the web-logs. I decided to get
an idea of the IP numbers using FormMail.

apache/logs(584)# grep FormMail access_log | awk ' { print $1 } ' | sort | uniq -c
2 130.13.117.86
62 161.142.100.81
1 161.142.100.85
62 166.114.127.6
55 194.133.172.118
178 195.39.134.85
70 196.40.23.26
123 202.138.155.4
61 203.227.45.253
64 203.82.192.2
58 206.49.58.75
49 211.6.228.50
3911 212.29.68.133
1139 212.29.74.132
62 212.38.131.225
53 212.38.133.69
64 212.67.117.136
430 213.152.93.3
60 216.120.157.114
55 63.167.108.38

A few nslookups showed me that most of those are not in the DNS.
Interesting. A few hours later I decided to start digging for the
abuse addresses for all those networks to report the abuse. I first
cut out the ones with less then 10 entries and then started finding
the corresponding addresses using whois.arin.net, whois.ripe.net,
whois.apnic.net, whois.nic.ad.jp, whois.krnic.net. I do this kind
of searching so often I have a complete set of aliases just for
doing this work. Now for a smarter whois client that can find the
abuse@ address just given an IP.
The addresses turned out to be in the Middle East (Jordania and
Kuwait), Asia (Japan, Korea, Bangladesh) and something close-by
(cybercomm.nl). Most interesting address was a 'broadband isp'
in Bangladesh giving a hotmail address for a contact address.
In the case of the .jp nic I could not get the right name for the
net because it thought the syntax of the query was incorrect. So I
tried to find upstream providers using traceroute in those cases.

So, all abuse addresses got a nice standard mail telling them that
a FormMail script was abused for spamming and that the FormMail
script was now closed but please flog the spammer accordingly.

Now, half an hour after sending all that I have only 6 auto-replies.
And I'm not sure I'll ever hear from some of the smaller ISP's.

Spammers have no problem abusing a small ISP somewhere in a country
they can't even find on a map. They may even have used a proxy at
that ISP or a hacked machine from a totally different location.

Now I have a bill for 2 hours of my time in cleaning up this mess.
Where do I send it ?