News items for tag work - Koos van den Hout

Since activating sendmail authentication with secondary passwords I see a number of attempts to guess credentials to send mail via my system. This is not very surprising, given the constant attack levels on the wider Internet.

For work I am looking at log correlation and monitoring and with that in mind I noted that finding the right information from sendmail where and when the attempt came from is quite hard since there are several processes busy and it's hard to correlate the logging. The failed attempt is logged by saslauthd in /var/log/auth.log:

Aug 16 12:28:57 greenblatt saslauthd[32648]: pam_unix(smtp:auth): check pass; user unknown
Aug 16 12:28:57 greenblatt saslauthd[32648]: pam_unix(smtp:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=
Aug 16 12:28:59 greenblatt saslauthd[32648]: do_auth         : auth failure: [user=monster] [service=smtp] [realm=idefix.net] [mech=pam] [reason=PAM auth error]
Aug 16 12:29:00 greenblatt saslauthd[32649]: pam_unix(smtp:auth): check pass; user unknown
Aug 16 12:29:00 greenblatt saslauthd[32649]: pam_unix(smtp:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=
Aug 16 12:29:02 greenblatt saslauthd[32649]: do_auth         : auth failure: [user=monster] [service=smtp] [realm=] [mech=pam] [reason=PAM auth error]

This is probably related to this sendmail log information:

Aug 16 12:28:56 greenblatt sm-mta[20716]: STARTTLS=server, relay=62.82.128.182.static.user.indesat.com [62.82.128.182] (may be forged), version=TLSv1/SSLv3, verify=NO, cipher=DHE-RSA-AES256-SHA, bits=256/256
Aug 16 12:29:02 greenblatt sm-mta[20716]: w7GASspx020716: 62.82.128.182.static.user.indesat.com [62.82.128.182] (may be forged) did not issue MAIL/EXPN/VRFY/ETRN during connection to MSP-v6

But I can't be sure as there are multiple 'did not issue MAIL/EXPN/VRFY/ETRN' messages in the logs. So I can't build a fail2ban rule based on this.

At work we (indirectly) get the scanning results from Shadowserver which now includes open VNC servers which is yet another service we don't really want left open to the Internet in general. A few were found which are now actively chased after to get them firewalled/disabled.

I really like the concept of shadowserver. ISPs that want the information can get full overviews of insecure services and botnet activity on their network. A full overview of what shadowserver looks for can be found at The scannings will continue until the Internet improves - Shadowserver.

Ever since I upgraded VirtualBox to a version 5 on my work laptop I can't start virtual machines which want to use the VirtualBox host-only network. For some communication I'd like to use that host-only network. Diving deep into the possible reasons found me VirtualBox can't find host-only adapters on Windows 10 which tells:

Oh and please double check that when you go to "Network Connections" and open the Properties dialog of the "VirtualBox Host-Only Network" connection, you see "VirtualBox NDIS6 Bridged Networking Driver" in the "This connection uses the following items" list.

This is indeed the case on my work laptop, and the use of the NDIS6 driver is an upgrade compared to VirtualBox 4. But I know the windows policy at work includes "you cannot use bridged networking" so I think I found the reason.

So now the solution is to use the main NAT network device. I was using the host-only network for some communication between the virtual machine and the host system or other virtual machines which should not be influenced by the 'outside world'. Now I seem to be using NAT as a firewall for that, which is wrong.

After taking the course in April and doing the exam in June I am now a Certified Information Systems Security Professional. As part of my job in computer security it is good to have this knowledge, it really helps me understand things.

At work I reviewed something about TLS security I wrote in May 2014 and noticed I had to make some serious adjustments for the May 2015 state. SSLv3 is no longer accepted, SHA1 is no longer an accepted hashing algorithm and other changes.

This week on the home server greenblatt I had two different impacts from the latest OpenSSL update: SSL communications with the Fritz!Box was failing and SSL in sendmail was failing, both due to the latest insights into the security of the Diffie-Hellman key exchange.

These insights are very very new: in April I did a course in the Certified Information Systems Security Professional (CISSP) common body of knowledge and I learned the default Diffie-Hellman parameters were safe. Now we learn to generate them for each individual system at the same strength as the private key. Knowledge of cryptographic quality ages fast at the moment.

So work made a laptop with the standard Windows 7 software image available to me and I noticed when I took it home it doesn't do any IPv6. Which is not what I want. Some searching found How to disable IPv6 or its components in Windows - Microsoft Support which has the right answers which were used by the people creating this software image. I changed the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters\DisabledComponents registry key to 0x01 so I don't get the Isatap/Teredo tunnels.

Interesting remark in that support article:

We do not recommend that you disable IPv6 or its components, or some Windows components may not function. Additionally, system startup will be delayed for 5 seconds if IPv6 is disabled.

I guess I'll have to find another way to disable the Isatap/Teredo tunnels to make the system boot faster. I want IPv6 to work when it's available native or not at all. Some aspects of the work network make things slow when tunneling protocols are tried. Which is probably the reason of disabling it in the first place.

Update 2014-10-01: It seems this setting gets reset somehow: I am at the Surfnet Relatiedagen 2014 and just noticed the laptop has no IPv6 on the network here, which surprised me. But a check of the settings showed no IPv6 addresses at all, not even link-local. A check on my Android phone shows globally routable IPv6 addresses.

I had an appointment today at a to me new address. I have an Android phone so I use the car navigation available from google. The 'Car' application has the (extra?) option to navigate to my next appointment, which made the fact the invitation had the right address information very useful. I used that and found the location fine. After I arrived people asked me if I had any trouble finding it since it is in a somewhat unusual location. I would have had more problems with a map!

HTC One Tip: How to use Car mode video explaining HTC Car mode and navigating to appointments.

A clear sign this week I am not a system administrator anymore: I had no easy answer to "where can we find some Torx screwdrivers to open the dishwasher".

Using contacts that were from when I was a system administrator helped find the right screwdrivers and the dishwasher was opened and repaired.

Using SSL should secure your services, but it needs to be configured correctly to avoid several attacks and keep the chances low that third parties can decrypt the traffic. With the SSL Server test from Qualys SSL Labs you can check the intimate details of your https SSL configuration.

And when you wonder what to use to improve your score when using Apache mod_ssl, here is a configuration snippet shared by the right people at Tilburg University:

SSLEngine on
SSLProtocol all -SSLv2
# advies Wessel Dankers kub
SSLHonorCipherOrder on
SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:AES256-GCM-SHA384:AES256-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES256-SHA:RC4-SHA:AES256-SHA:AES128-SHA:DES-CBC3-SHA

I heard today about Windows 2012 R2 "desired state configuration" which made me think a bit of puppet. The general idea is to get systems configured to a desired state with whichever changes are needed to get to that state. Desired State Configuration in Windows Server 2012 R2 PowerShell - YouTube for a presentation.

But when I see a bit of configuration sample in the above video it makes me think a lot of puppet:

Configuration FourthCoffeeWebsite
{
	Node ("WebServer1","WebServer2")
	{
		# Install the IIS role
		WindowsFeature IIS
		{
			Ensure  = "Present"
			Name    = "Web-Server"
		}
		
		# Install the ASP .NET 4.5 role
		WindowsFeature AspNet45
		{
			Ensure  = "Present"
			Name	= "Web-Server"
		}
	}
}

Funny how system administration in the Windows and Linux/Unix world is converging. Just like Windows PowerShell makes me think of scripting languages and the unix commandline.

Found out a neat option in a modern-ish libc6 getent: you can use services different from those configured in /etc/nsswitch.conf simply by using the -s switch. It took a bit of searching to find the actual way to use ldap and be able to resolve the ldap server:

$ getent -s 'dns ldap' passwd

There are no users in dns in our config, but this enables finding the ldap servers via dns.

The person(s) whose malvertising network got taken down due to the exposure on malekal.com blog is getting desperate. I'm not only getting the malekal joe-job spam myself but it's now also showing up at the computer security contact address at work.

I've been working on managing Linux systems with puppet for a while. Until now puppet was a tool to manage part of the configuration with still work to be done on each host. But the last two weeks I worked on a (test) webserver completely configured from puppet. With a complete separation of configuration (from puppet), input data (web content), output data (logging) and installed applications it is possible to reduce a webserver to a puppet recipe and an amount of storage. This means adding new webservers to a cluster or rebuilding systems in the cluster is easy. As a test I 'broke' the webserver (wiped the disk), reinstalled basic CentOS (nothing configured) and let puppet deliver a running webserver again, all within 15 minutes.

The new bit for me was using puppet templates to write centos ifcfg-ethX files and apache virtualhost configurations. Apache virtualhosts get a number of parameters (the hostname, aliases, directory index settings, needing php, needing ssl). I started with different templates for 'real' virtualhosts and 'special' virtualhosts like a host which gives a 410 Gone error on all urls but I noticed the templates were still mostly the same so now the type of virtualhost is also set using variables and one template has conditional parts depending on the type of virtualhost.

This does mean I'm learning bits of Ruby, Yet Another Scripting Language (for me).

In general, using puppet makes it very easy to install/remove packages, add scripts, schedule tasks, configure the monitoring setup (zabbix) and do other 'checklist' items to each system in a consistent way. Which in my opinion improves security and general quality.

I follow some blogs of people working in the same kind of system administration work as I do. The article How to disable LSI MegaRAID SAS controller's suspend boot on error "feature" by Joshua Hobblit caught my eye because it mentions a different utility from LSI to manage MegaRAID controllers from the commandline.

In the past Supermicro systems with OEM LSI controllers have refused to work with several versions of the MegaCLI software. But storcli works on these systems, enabling me to monitor them fully for harddisk failures and disable the beeper whenever a disk should fail.

Search the LSI website documents & downloads section for keyword storcli.

New release of Repocafe with new features. Most improvements are based on wishes from users. On our wishlist as developers is still MySQL support (and maybe other databases too) and git repo support.

Repocafe project at freecode

We like our Linux kernels chatty during boot, seeing stuff in the startup messages like

serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
00:06: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:07: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A

is perfectly fine with us. Defaults with several linux distributions are going the other way. For CentOS we already disable the plymouth splash screen, but to disable more eyecandy and get real kernel messages the commandline options rhgb and quiet need to be removed from the kernel commandline in the grub config. Option rhgb enables 'red hat graphic boot' and option quiet disables most kernel messages.

Via How do I set the default kernel parameters in CentOS for all existing and future kernels? - Server Fault I found the right way. The next step was to turn this into a puppet recipe so this is done automatically:

class serverpackages::fixgrubconfig {
        exec { "Clean grub default options":
                path => "/sbin:/bin",
                onlyif => 'egrep -c \'(rhgb|quiet)\' /boot/grub/grub.conf',
                command => '/usr/local/sbin/normalizegrubconfig',
                require => file["normalizegrubconfig"];
        }
        file { "normalizegrubconfig":
                path => '/usr/local/sbin/normalizegrubconfig',
                ensure => present,
                owner => 'root',
                group => 'root',
                mode => 0700,
                content => '#!/bin/sh
#
# THIS FILE IS UNDER PUPPET CONTROL
# reset grub config for all kernels
for KERNEL in /boot/vmlinuz-* ; do
        grubby --update-kernel="$KERNEL" --remove-args="rhgb quiet"
done
';
        }
}

Problem solved, yet another thing puppet adds to the baseline configuration. The upside of using grubby to manage this is that 'creating correct grub config files' is builtin into grubby.

With yet another Java update on a computer at work I was reading A close look at how Oracle installs deceptive software with Java updates - ZDNet Ed Bott. I did disable the "install ask.com toolbar" option.

At the end, the java update guides the browser to http://java.com/en/download/installed.jsp?detect=jre which says it is "Verifying Installation". But according to the Java security prompt it needs "unrestricted access which may put your computer and personal information at risk."
I picked "Cancel".

Interesting clash between the bind 9.8.2 package for CentOS 6.4 and puppet: When puppet updates /etc/named.conf it's not visible in the chroot setup for named. The named startup script uses bind mounts to make configuration files visible within the chroot environment.

root@geodns01:~# mount | grep named.conf
/etc/named.conf on /var/named/chroot/etc/named.conf type none (rw,bind)
root@geodns01:~# md5sum /etc/named.conf /var/named/chroot/etc/named.conf
d028cfee6cf1a1f77993da7c769273ad  /etc/named.conf
82d1717bb34db23804f67ad855e090ea  /var/named/chroot/etc/named.conf

I first thought this was some form of caching, but a suggestion was the way the files were replaced by puppet: if puppet creates a new file and then renames the old one and the new one, the file will have a different inode after that action. I tested for this:

root@geodns01:~# mkdir test
root@geodns01:~# touch file.conf
root@geodns01:~# touch /root/test/file.conf
root@geodns01:~# mount --bind file.conf /root/test/file.conf
root@geodns01:~# ls -il /root/file.conf /root/test/file.conf
652873 -rw-r--r-- 1 root root 0 Mar 26 19:20 /root/file.conf
652873 -rw-r--r-- 1 root root 0 Mar 26 19:20 /root/test/file.conf
root@geodns01:~# vim --cmd 'set backup' file.conf
root@geodns01:~# ls -li file.conf* test/file.conf
652876 -rw-r--r-- 1 root root 7 Mar 26 19:25 file.conf
652873 -rw-r--r-- 1 root root 0 Mar 26 19:20 file.conf~
652873 -rw-r--r-- 1 root root 0 Mar 26 19:20 test/file.conf

This confirms that replace-by-rename will clash with bind mounts being actually inode based. The workaround isn't that hard: the startup script for named explicitly tests for an existing non-zero-byte /var/named/chroot/etc/named.conf and will skip the mount --bind part in that case. Time to learn puppet about this feature, puppet now manages both /etc/named.conf and /var/named/chroot/etc/named.conf.

Good use of having learned the Nato phonetic spelling alphabet: today I had a support person on the phone who called from the UK and who needed an e-mail address to exchange some information. Spelling as I learned it in English class didn't work, but the Nato phonetic spelling alphabet helped.

Reading about 0day Linux/CentOS SSHd Spam Exploit — libkeyutils.so.1.9 I decided to actively monitor for the wrong file(s) using zabbix. Using a template linked to our linux template so all linux servers with a working agent are monitored. Developed on zabbix 1.8.16: zbx_template_detect_libkeyutil.xml: zabbix template to detect libkeyutil.so.1.9 files. As noted, if you find such a file and rpm can't give you the package it is from, you have a serious security incident.

Original news about the problems via Linux-servers geroot en geplunderd via 0-day - Webwereld.nl