News items for tag security - Koos van den Hout

Sometimes hobby and work intertwine when I'm not expecting it.

I set up a domainname and added a dummy website for something related to amateur radio. I have no idea if it will go anywhere, but I thought I'd get the web configuration right. The domain name isn't published anywhere.

But, to my surprise:

178.174.174.11 - - [20/May/2020:09:14:35 +0200] "GET /.git/HEAD HTTP/1.0" 404 594 "-" "-"
178.174.174.11 - - [20/May/2020:09:14:35 +0200] "GET /.git/HEAD HTTP/1.0" 404 594 "-" "-"
178.174.174.11 - - [20/May/2020:09:14:53 +0200] "GET /.git/HEAD HTTP/1.0" 404 594 "-" "-"
178.174.174.11 - - [20/May/2020:09:14:53 +0200] "GET /.git/HEAD HTTP/1.0" 404 594 "-" "-"
81.92.203.216 - - [20/May/2020:09:15:12 +0200] "GET /.git/HEAD HTTP/1.0" 404 594 "-" "-"
2a00:d680:30:50::67 - - [24/May/2020:16:54:36 +0200] "GET /wp-login.php HTTP/1.1" 404 594 "http://******.*******.**/wp-login.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"

I added the domain name and requested a LetsEncrypt certificate on 11 May 2020, I set up the webserver correctly on 19 May 2020. The only 'publication' of the name is via the certificate transparancy log. Somehow this is enough for the first probes for possible security issues.

Looking in the haproxy logs finds even more requests on 15 and 18 May 2020. Part of the requests are via http, not https.

Keeping encryption settings correct needs a lot of testing to make sure things are right. With external-facing webservices this is easy with the Qualys SSL scan, but for other services than https or services not facing outward a local tester is needed. This local tester is sslscan, a commandline tool but which depends on the shared openssl libraries which have insecure protocols disabled to helps disabling those deprecated protocols.

But to test services the client needs to support those old protocols to do the test correctly.

So I built a static version of sslscan with static openssl using the instructions at https://github.com/rbsec/sslscan. And that works for the full testing range!

I have a Synology NAS at home running DSM, so I had a look at the certificate options. According to the documentation it can get a LetsEncrypt certificate so I tried that. And it worked... which wasn't what I expected.

Some testing later found out port 80 tcp was open for every IPv6 address at home. That's now fixed and limited to those few IPv6 addresses that need to be reachable from the outside world.

Browsing the opinions about allowing outside access to the webserver on the Synology versus not allowing it showed me some differing opinions, but an article listening some malware and ransomware targetting Synology systems made me decide to close the system. Looking at the nginx configuration on the Synology gives me the idea some of the web-accessible functionality is available via port 80.

I think I found the most probable reason of the earlier problem with DNSSEC signed subzones. I was trying this with a domain for which I don't have control over one of the secondary nameservers.

In one of my showerthought moments I decided to try another domain where I have that full control (just less nameservers) and was able to make it all validate correctly after some tries. Forgetting one or more of all the steps needed to correctly create a domain with DNSSEC and getting the delegation right will give errors.

So I guess running a nameserver with all DNSSEC options disabled hinders validation.

I noticed the news about LetsEncrypt revoking a lot of certificates on 4 March 2020 and did some checking to find out eventually that one of my certificates is in that set. Users have been notified of this problem... when their account had a contact e-mail address. By default dehydrated doesn't set an e-mail address so none of my instances used one. I do like to get informed so I searched how to update the contact info. The data is in /etc/dehydrated/config field CONTACT_EMAIL but I needed some searching before I found the method to get the update passed on to LetsEncrypt.

Some searching later found Update registration email address - Issue #425 dehydrated which shows that a simple dehydrated --account does the magic.

A few days ago I changed the configuration of haproxy to stop accepting TLSv1.0 and TLSv1.1. With the upcoming deprecation of TLSv1.0 and TLSv1.1 this seemed the right SSL configuration. Today I remembered there is one directly reachable Apache server, so I had a look at the settings there and checked the results with the Qualys SSL Labs SSL Server test where I noticed some ciphers listed as 'weak'. And seeing different results between my haproxy and apache servers, which I did not expect as I used the same settings for SSLCipherSuite in Apache and ssl-default-bind-ciphers in haproxy.

The last issue was caused by the fact that Apache2.4.25 in Devuan ascii uses libssl 1.0.2 and haproxy 1.7.5 uses libssl 1.1.0. I'm not sure that's an ideal configuration but it's what I work with.

With the output of openssl ciphers -v I get a list of cipher names. But this is with libssl1.1.0 so the output lists ciphers that Apache doesn't have access to (yet). The good part is that Apache ignores ciphers that aren't available, so the net result is a running and working configuration.

The current result is for Apache 2.4.25:

SSLProtocol All -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLHonorCipherOrder on
SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256

And for haproxy 1.7.5:

ssl-default-bind-options force-tlsv12 no-tls-tickets
ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256

The fun part is that I can test the SSL negotiation with sslscan locally but sslscan is linked against openssl 1.0.2 so it misses some of the newer options. And I also test with the Qualys SSL Labs ssl test but that takes a while.

The too long; didn't read version of finding the right configuration options

And later I found I could have saved a lot of time researching options using the Mozilla SSL Configuration Generator. I don't completely agree with the suggestions there because I want to generate my own dhparams. Using 'well-known Diffie-Hellman paramaters' has security risks. But otherwise all the suggestions for ciphers are very usable and save me a lot of time.

In 2020 the support for TLSv1.0 and TLSv1.1 will end so the famous qualys SSL test is giving capped grades. I decided to get with the times and limit my outside web ports to TLSv1.2 so now I am back at an A+ grade.

Eventually this will start to cause problems as Devuan stable doesn't have an openssl with TLSv1.3 support yet.

Suricata is running and detecting attacks, but it was causing a 100% cpu load after a restart of the ppp connection (the DSL here uses PPP over Ethernet).

The errors point at the problem starting when the ppp connection restarts:

21/1/2020 -- 00:59:36 - <Error> - [ERRCODE: SC_ERR_AFP_READ(191)] - Error reading data from iface 'ppp0': (100u) Network is down
21/1/2020 -- 00:59:37 - <Error> - [ERRCODE: SC_ERR_AFP_CREATE(190)] - Couldn't set fanout mode, error Invalid argument

Which also starts to fill the system log with:

Jan 21 00:59:42 xxxxxxxx kernel: [11347441.726755] device ppp0 left promiscuous mode
Jan 21 01:00:13 xxxxxxxx kernel: [11347472.055712] device ppp0 entered promiscuous mode
Jan 21 01:00:13 xxxxxxxx kernel: [11347472.071533] device ppp0 left promiscuous mode
Jan 21 01:00:13 xxxxxxxx kernel: [11347472.091653] device ppp0 entered promiscuous mode

The interesting part is that this causes higher power usage about five and a half hours later.

Solution: restart suricata in an /etc/ppp/ip-up.d/ script.

Today I needed blocks of random letters to practise sending morse. What better tool to create those blocks than good old pwgen with the right settings:

$ pwgen -0 -A 5 12
ahhud eizaa kuoku ahyoo aequi epiis eiwei eimap sohsh papai ikeit oucho

And the trick for generating groups of five digits is a bit longer:

$ pwgen -r abcdefghijklmnopqrstuvwxyz -A 5 12
97228 85996 98876 38451 06091 98556 53369 73632 29509 29032 89601 16078

Or both letters and digits:

$ pwgen -A 5 12
sa7la oc7ko an5ne axae6 vohz6 aez5i eh3qu sha5m inai8 eor3a fuv1o ro6ha

Use better parameters with pwgen to generate actual passwords.

I tried adding subzones with DNSSEC by adding the DS record to the parent zone, but in both tries I got errors from DNSViz. Different errors even: in one case the signature on the DS record was seen as invalid and in another case there was no signature at all. The errors are reproducable, even after waiting for caches to empty.

I kept seeing warnings about an expired signature when running named-checkzone or dnssec-signzone and it took some searching before I found the reason.

Recently I removed the records with type SPF from my zones since the recommended approach is to use TXT records with SPF data. The RRSIG records for the SPF records were left in the signed zonefile, but not updated so they expired and started to give warnings.

The SPF records were for names that had other data too which seems to trigger this. Removing a record completely (no RRTYPEs left for the name) removes all signatures.

The things in DNSSEC I haven't tested yet are a signed subzone, a ZSK rollover and a KSK rollover. Those will eventually happen too.

Now I have DNSSEC running ok on my domains I can start looking at security innovations that rely on DNSSEC.

The first one is DANE for the mailserver, in which the public key signature is published in DNS record secured with DNSSEC to give a separate path to verify the public key during the SMTP session.

The public key of the mailserver is also signed by LetsEncrypt as described in Automating Let's Encrypt certificates further and Automating Let's Encrypt certificates with DNS-01 protocol so there are two completely independent paths to verify the identity of the mail server.

To find the public key of the mailserver for a given domain:

$ dig +short idefix.net mx
10 postbox.idefix.net.
$ dig +short _25._tcp.postbox.idefix.net tlsa
3 1 2 2B55764A99A47AEC5B66D8EB4E741F2646BF6352CABC9BE3F37D2F42 0BD7EF56B5BE3058E7B10964BA963777364443057E45599E07A82375 7A812F1A7014356A

I found the tlsa tool from package hash-slinger by Paul Wouters to create these records. This can be both from the protocol which has certain risks (if that connection is intercepted) or from the public key file. Or via the web tool Generate TLSA Record by Shumon Huque.

TLSA records are generically linked to a TCP or UDP port. The next step will probably be to start adding records for other public services with TLS like https. There was a time that some people were convinced DANE was going to replace certificate authorities for https, but at this moment it is very limited. I have added TLSA records for https (tcp/443) for camp-wireless.com and www.camp-wireless.com for now and I'm testing with these. For now one of my favourite checkers isn't convinced.

This does increase the chances for things to go wrong. With the tlsa program it is possible to verify records too, so I can use this to verify TLSA records.

$ tlsa --verify -6 --starttls smtp --port 25 postbox.idefix.net
SUCCESS (Usage 3 [DANE-EE]): Certificate offered by the server matches the TLSA record (2001:980:14ca:1::23)

Although this certificate is a valid LetsEncrypt certificate, DNS-based Authentication of Named Entities (DANE) does not support usage 1 (check the certificate public key and verify certificate chain to a known root) for SMTP with STARTTLS, so it is usage 3 (just check the certificate public key). The tlsa program does not check this specifically, but the web checker at DANE TLSA Server checker found the issue, so I corrected that.

I use selector 1 to just check the public key because the complete certificate changes with every LetsEncrypt renewal. My choice for mtype 2 (sha512) is just a wish for a strong hashing algorithm.

This also makes the link between service configuration and DNS contents a lot stronger. Maybe this needs secure automated updates.

Met alle tips voor het maken van veilige wachtwoorden en die alleen beschikbaar hebben vanuit een wachtwoordmanager loop ik nu tegen websites aan die vragen om een wachtwoord maar vervolgens moet je dat wachtwoord ineens op een fysiek andere plek dan achter je eigen computer intikken.

De eerste keer dat ons dat overkwam was bij een camping van staatsbosbeheer op Ameland. We hadden ons via de website ingeschreven en bij het aanmelden op de camping zelf bleek er een aanmeldscherm te zijn waar je met e-mail adres en wachtwoord moest inloggen. Maar we gebruiken voor dat soort websites altijd gegenereerde wachtwoorden die we niet weten. Met veel zoeken naar de hoek van de camping met een beetje mobiele data dekking konden we bij onze wachtwoordkluis en konden we het wachtwoord opzoeken. Want het aanmeldscherm is omdat de beheerder van de camping er ook maar een uur per dag is.

De tweede keer was bij de bibliotheek in Utrecht. Als je daar in de bibliotheek zelf een reservering wilt maken moet je ook inloggen op een computer met gebruikersnaam en wachtwoord. En ook daar konden we het niet snel opzoeken, maar daar konden ze ons helpen aan de hand van de bibliotheekpas.

Some things noticed by Suricata IDS are amusing to me. When looking at lines like:

11/15/2019-13:14:35.001691  [**] [1:2402000:5363] ET DROP Dshield Block Listed Source group 1 [**] [Classification: Misc Attack] [Priority: 2] {TCP} 185.156.73.11:46843 -> 82.95.196.202:41505
11/15/2019-13:15:06.794357  [**] [1:2402000:5363] ET DROP Dshield Block Listed Source group 1 [**] [Classification: Misc Attack] [Priority: 2] {TCP} 88.214.26.102:42131 -> 82.95.196.202:8703
11/15/2019-13:15:06.794357  [**] [1:2403384:53195] ET CINS Active Threat Intelligence Poor Reputation IP group 85 [**] [Classification: Misc Attack] [Priority: 2] {TCP} 88.214.26.102:42131 -> 82.95.196.202:8703
11/15/2019-13:15:20.065796  [**] [1:2403393:53195] ET CINS Active Threat Intelligence Poor Reputation IP group 94 [**] [Classification: Misc Attack] [Priority: 2] {UDP} 93.174.95.106:27221 -> 82.95.196.202:7
11/15/2019-13:15:32.845110  [**] [1:2402000:5363] ET DROP Dshield Block Listed Source group 1 [**] [Classification: Misc Attack] [Priority: 2] {TCP} 185.156.73.21:44503 -> 82.95.196.202:43935
11/15/2019-13:16:23.399397  [**] [1:2402000:5363] ET DROP Dshield Block Listed Source group 1 [**] [Classification: Misc Attack] [Priority: 2] {TCP} 185.175.93.27:58989 -> 82.95.196.202:53166

All 'Dshield Block Listed' and 'Poor Reputation IP' traffic is port scans to ports that are blocked. So it's not a surprise those IPs have a poor reputation or a Dshield listing.

After hearing about intrusion detection systems a few times I decided to give one a try at home. Although a lot of attacks are blocked I sometimes see weird attacks and it would be nice to have a better idea of what exactly the attack was.

Yes, I have weird interests sometimes. I'm glad I have an ISP (xs4all) where I can select the option 'give me the completely unfiltered Internet connection' so I even see SMB protocol attempts.

I first tried 'snort' but that doesn't deal with PPP interfaces by default. It can be recompiled to accept those but I did not want that. The next option I heard about is 'Suricata' which is running at the moment.

I was amused by the reports of DDoS-like NTP traffic. Those are caused by the NTP statistics gathering. I know NTP can be abused for generating DDoS traffic but all security reports about NTP servers I manage have been false positives.

Anyway it's running and complaining a lot about the traffic it sees. For example the IPv6 port scan/network mapping attempts I noticed two months ago are still active.

11/13/2019-15:06:59.703451  [**] [1:2002911:6] ET SCAN Potential VNC Scan 5900-5920 [**] [Classification: Attempted Information Leak] [Priority: 2] {TCP} 240e:00f7:4f01:000c:0000:0000:0000:0003:6050 -> 2001:0980:14ca:0001:020d:56ff:fece:ffe1:5901
11/13/2019-15:08:39.645780  [**] [1:2002911:6] ET SCAN Potential VNC Scan 5900-5920 [**] [Classification: Attempted Information Leak] [Priority: 2] {TCP} 240e:00f7:4f01:000c:0000:0000:0000:0003:5167 -> 2001:0980:14ca:0001:020d:56ff:fece:ffe6:5901

It seems the TCP reflective SYN attacks are continuing. In researching my options I saw the option to use a netmask with the iptables recent module.

This helps a bit with the attacks trying to flood an entire block. I've updated the filtering to work by the /24, start a check on a SYN from such a block, end when an ACK flies by and start dropping when the rate is over 10 per 2 minutes.

iptables -A INPUT -p tcp -m tcp --tcp-flags SYN,ACK SYN -m recent --update --seconds 120 --hitcount 10 --name tcpsyn --mask 255.255.255.0 --rsource -j LOGDROP
iptables -A INPUT -p tcp -m tcp --tcp-flags SYN,ACK SYN -m recent --set --name tcpsyn --mask 255.255.255.0 --rsource
iptables -A INPUT -p tcp -m tcp --tcp-flags SYN,ACK ACK -m recent --remove --name tcpsyn --mask 255.255.255.0 --rsource

LOGDROP is a rule to drop packets and ratelimit the logging of dropped packets, to avoid turning a network attack into a disk attack.

But I have to be careful not to make services hard to reach for legitimate clients. The above is working, and during attacks I don't see a single SYN_RECV socket.

Sometimes you really wonder about the amount of errors made by noisy attacks. I noticed the following pattern in the system logs:

nnrpd[7029]: 189.243.177.73 unrecognized Accept-Encoding: identity
nnrpd[7029]: 189.243.177.73 unrecognized Content-Length: 586
nnrpd[7029]: 189.243.177.73 unrecognized Accept-Language: en-us
nnrpd[7029]: 189.243.177.73 unrecognized Host: 74.219.111.25
nnrpd[7029]: 189.243.177.73 unrecognized Accept: */*
nnrpd[7029]: 189.243.177.73 unrecognized User-Agent: ApiTool
nnrpd[7029]: 189.243.177.73 unrecognized Connection: close
nnrpd[7029]: 189.243.177.73 unrecognized Cache-Control: max-age=0
nnrpd[7029]: 189.243.177.73 unrecognized Content-Type: text/xml
nnrpd[7029]: 189.243.177.73 unrecognized Authorization: Basic YWRtaW46ezEyMjEzQkQ...

With some searching I eventually found exploit code for certain series of digital video recorders which can be anywhere on the wide Internet.

The whole protocol mismatch makes this a lot noisier via the nntp port than via http, but I also see some attempts via the http port.

Update: Suricata doesn't recognize the specific attack, but it does notice the HTTP basic auth in the traffic:

11/13/2019-20:12:33.772828  [**] [1:2006402:11] ET POLICY Incoming Basic Auth Base64 HTTP Password detected unencrypted [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {TCP} 188.59.207.57:43753 -> 82.95.196.202:119

I'm seeing lots of sockets in state SYN_RECV again and noticed this time my earlier iptables rules to not respond to tcp syn packets that don't build up a connection aren't working. Between two syn packets from the same source there is 5 minutes, so my system responds to all of them. Ranges of addresses in the same block are used as source IPv4 addresses. For one address the traffic is very minimal:

22:40:51.600077 IP 112.175.120.39.58275 > 82.95.196.202.22: Flags [S], seq 720891004, win 29200, length 0
22:40:51.600392 IP 82.95.196.202.22 > 112.175.120.39.58275: Flags [S.], seq 1729897232, ack 720891005, win 29200, options [mss 1460], length 0
22:40:52.612035 IP 82.95.196.202.22 > 112.175.120.39.58275: Flags [S.], seq 1729897232, ack 720891005, win 29200, options [mss 1460], length 0
22:40:54.628048 IP 82.95.196.202.22 > 112.175.120.39.58275: Flags [S.], seq 1729897232, ack 720891005, win 29200, options [mss 1460], length 0
22:40:58.660031 IP 82.95.196.202.22 > 112.175.120.39.58275: Flags [S.], seq 1729897232, ack 720891005, win 29200, options [mss 1460], length 0
22:41:06.851865 IP 82.95.196.202.22 > 112.175.120.39.58275: Flags [S.], seq 1729897232, ack 720891005, win 29200, options [mss 1460], length 0
22:41:22.980000 IP 82.95.196.202.22 > 112.175.120.39.58275: Flags [S.], seq 1729897232, ack 720891005, win 29200, options [mss 1460], length 0
22:45:18.565999 IP 112.175.120.39.41767 > 82.95.196.202.465: Flags [S], seq 910623633, win 29200, length 0
22:45:18.566415 IP 82.95.196.202.465 > 112.175.120.39.41767: Flags [S.], seq 3977721413, ack 910623634, win 29200, options [mss 1460], length 0
22:45:19.588000 IP 82.95.196.202.465 > 112.175.120.39.41767: Flags [S.], seq 3977721413, ack 910623634, win 29200, options [mss 1460], length 0
22:45:21.604022 IP 82.95.196.202.465 > 112.175.120.39.41767: Flags [S.], seq 3977721413, ack 910623634, win 29200, options [mss 1460], length 0
22:45:25.667936 IP 82.95.196.202.465 > 112.175.120.39.41767: Flags [S.], seq 3977721413, ack 910623634, win 29200, options [mss 1460], length 0
22:45:33.860000 IP 82.95.196.202.465 > 112.175.120.39.41767: Flags [S.], seq 3977721413, ack 910623634, win 29200, options [mss 1460], length 0
22:45:49.987965 IP 82.95.196.202.465 > 112.175.120.39.41767: Flags [S.], seq 3977721413, ack 910623634, win 29200, options [mss 1460], length 0

But multiply this with several source IPs in the same IPv4 /24 block and a lot of open servers in the world and suddenly you get a lot of return traffic.

I noticed some really unused records in one zone which is now DNSSEC signed. For example I still had gplus.idefix.net to point at my Google+ page.

So I removed them and did the signing after increasing the serial number. Indeed the records that had no update kept their original signature and the records that where changed (such as the SOA because of the serial number) were signed with new signatures.

Today I was reminded of the first zone I signed with DNSSEC and did the check again with DNSViz. And I saw a lot of error messages. Some searching found that I let all the signatures expire (after the default time of 30 days).

Solution: re-sign the zone and have a careful look at when I need to sign the zones again. Officially just in time for expiry time of the signature (default 30 days) minus TTL of the record.

Obviously this process has to be automated. In the first go I decided to force new signatures after 21 days. But I tested some things later and decided to go for more regular checks of the ages of the signatures and refresh the signatures that are about to expire. This is usually reserved for 'big' zones with lots of resolvers querying them but I decided to implement this myself to avoid problems, and learn more about DNSSEC.

The magic signing command is now:

-zone-signedserial:
    named-checkzone $* $^
    ./SOA.pl $^
    dnssec-signzone -S -K /etc/bind/keys -g -a -r /dev/random -D -S -e +2592000 -i 604800 -j 86400 -o $* $^
    rndc reload $*
    touch $@

The expiry is set with -e at 30 days, the checkinterval with -i at 7 days and the jitter factor with -j at 1 day.

Now there is a special part in the Makefile to be called from cron on a regular basis. It won't produce any output when there is nothing to update.

agecheck:
    @for zone in $(SIGNEDZONES); do if [ `find $${zone}-signedserial -mtime +7 -print` ]; then touch $${zone}-zone ; $(MAKE) --no-print-directory $${zone}-signedserial; fi ;done

The Make variable SIGNEDZONES is filled with the zonenames of the zones that have to be kept DNSSEC signed. File structure for each forward zone is as listed in first zone with valid DNSSEC signatures.

So now almost all my domains are DNSSEC signed. A learning experience and a good level of security.