News items for tag security - Koos van den Hout

Suricata is running and detecting attacks, but it was causing a 100% cpu load after a restart of the ppp connection (the DSL here uses PPP over Ethernet).

The errors point at the problem starting when the ppp connection restarts:

21/1/2020 -- 00:59:36 - <Error> - [ERRCODE: SC_ERR_AFP_READ(191)] - Error reading data from iface 'ppp0': (100u) Network is down
21/1/2020 -- 00:59:37 - <Error> - [ERRCODE: SC_ERR_AFP_CREATE(190)] - Couldn't set fanout mode, error Invalid argument

Which also starts to fill the system log with:

Jan 21 00:59:42 xxxxxxxx kernel: [11347441.726755] device ppp0 left promiscuous mode
Jan 21 01:00:13 xxxxxxxx kernel: [11347472.055712] device ppp0 entered promiscuous mode
Jan 21 01:00:13 xxxxxxxx kernel: [11347472.071533] device ppp0 left promiscuous mode
Jan 21 01:00:13 xxxxxxxx kernel: [11347472.091653] device ppp0 entered promiscuous mode

The interesting part is that this causes higher power usage about five and a half hours later.

Solution: restart suricata in an /etc/ppp/ip-up.d/ script.

Today I needed blocks of random letters to practise sending morse. What better tool to create those blocks than good old pwgen with the right settings:

$ pwgen -0 -A 5 12
ahhud eizaa kuoku ahyoo aequi epiis eiwei eimap sohsh papai ikeit oucho

And the trick for generating groups of five digits is a bit longer:

$ pwgen -r abcdefghijklmnopqrstuvwxyz -A 5 12
97228 85996 98876 38451 06091 98556 53369 73632 29509 29032 89601 16078

Use better parameters with pwgen to generate actual passwords.

I tried adding subzones with DNSSEC by adding the DS record to the parent zone, but in both tries I got errors from DNSViz. Different errors even: in one case the signature on the DS record was seen as invalid and in another case there was no signature at all. The errors are reproducable, even after waiting for caches to empty.

I kept seeing warnings about an expired signature when running named-checkzone or dnssec-signzone and it took some searching before I found the reason.

Recently I removed the records with type SPF from my zones since the recommended approach is to use TXT records with SPF data. The RRSIG records for the SPF records were left in the signed zonefile, but not updated so they expired and started to give warnings.

The SPF records were for names that had other data too which seems to trigger this. Removing a record completely (no RRTYPEs left for the name) removes all signatures.

The things in DNSSEC I haven't tested yet are a signed subzone, a ZSK rollover and a KSK rollover. Those will eventually happen too.

Now I have DNSSEC running ok on my domains I can start looking at security innovations that rely on DNSSEC.

The first one is DANE for the mailserver, in which the public key signature is published in DNS record secured with DNSSEC to give a separate path to verify the public key during the SMTP session.

The public key of the mailserver is also signed by LetsEncrypt as described in Automating Let's Encrypt certificates further and Automating Let's Encrypt certificates with DNS-01 protocol so there are two completely independent paths to verify the identity of the mail server.

To find the public key of the mailserver for a given domain:

$ dig +short idefix.net mx
10 postbox.idefix.net.
$ dig +short _25._tcp.postbox.idefix.net tlsa
3 1 2 2B55764A99A47AEC5B66D8EB4E741F2646BF6352CABC9BE3F37D2F42 0BD7EF56B5BE3058E7B10964BA963777364443057E45599E07A82375 7A812F1A7014356A

I found the tlsa tool from package hash-slinger by Paul Wouters to create these records. This can be both from the protocol which has certain risks (if that connection is intercepted) or from the public key file. Or via the web tool Generate TLSA Record by Shumon Huque.

TLSA records are generically linked to a TCP or UDP port. The next step will probably be to start adding records for other public services with TLS like https. There was a time that some people were convinced DANE was going to replace certificate authorities for https, but at this moment it is very limited. I have added TLSA records for https (tcp/443) for camp-wireless.com and www.camp-wireless.com for now and I'm testing with these. For now one of my favourite checkers isn't convinced.

This does increase the chances for things to go wrong. With the tlsa program it is possible to verify records too, so I can use this to verify TLSA records.

$ tlsa --verify -6 --starttls smtp --port 25 postbox.idefix.net
SUCCESS (Usage 3 [DANE-EE]): Certificate offered by the server matches the TLSA record (2001:980:14ca:1::23)

Although this certificate is a valid LetsEncrypt certificate, DNS-based Authentication of Named Entities (DANE) does not support usage 1 (check the certificate public key and verify certificate chain to a known root) for SMTP with STARTTLS, so it is usage 3 (just check the certificate public key). The tlsa program does not check this specifically, but the web checker at DANE TLSA Server checker found the issue, so I corrected that.

I use selector 1 to just check the public key because the complete certificate changes with every LetsEncrypt renewal. My choice for mtype 2 (sha512) is just a wish for a strong hashing algorithm.

This also makes the link between service configuration and DNS contents a lot stronger. Maybe this needs secure automated updates.

Met alle tips voor het maken van veilige wachtwoorden en die alleen beschikbaar hebben vanuit een wachtwoordmanager loop ik nu tegen websites aan die vragen om een wachtwoord maar vervolgens moet je dat wachtwoord ineens op een fysiek andere plek dan achter je eigen computer intikken.

De eerste keer dat ons dat overkwam was bij een camping van staatsbosbeheer op Ameland. We hadden ons via de website ingeschreven en bij het aanmelden op de camping zelf bleek er een aanmeldscherm te zijn waar je met e-mail adres en wachtwoord moest inloggen. Maar we gebruiken voor dat soort websites altijd gegenereerde wachtwoorden die we niet weten. Met veel zoeken naar de hoek van de camping met een beetje mobiele data dekking konden we bij onze wachtwoordkluis en konden we het wachtwoord opzoeken. Want het aanmeldscherm is omdat de beheerder van de camping er ook maar een uur per dag is.

De tweede keer was bij de bibliotheek in Utrecht. Als je daar in de bibliotheek zelf een reservering wilt maken moet je ook inloggen op een computer met gebruikersnaam en wachtwoord. En ook daar konden we het niet snel opzoeken, maar daar konden ze ons helpen aan de hand van de bibliotheekpas.

Some things noticed by Suricata IDS are amusing to me. When looking at lines like:

11/15/2019-13:14:35.001691  [**] [1:2402000:5363] ET DROP Dshield Block Listed Source group 1 [**] [Classification: Misc Attack] [Priority: 2] {TCP} 185.156.73.11:46843 -> 82.95.196.202:41505
11/15/2019-13:15:06.794357  [**] [1:2402000:5363] ET DROP Dshield Block Listed Source group 1 [**] [Classification: Misc Attack] [Priority: 2] {TCP} 88.214.26.102:42131 -> 82.95.196.202:8703
11/15/2019-13:15:06.794357  [**] [1:2403384:53195] ET CINS Active Threat Intelligence Poor Reputation IP group 85 [**] [Classification: Misc Attack] [Priority: 2] {TCP} 88.214.26.102:42131 -> 82.95.196.202:8703
11/15/2019-13:15:20.065796  [**] [1:2403393:53195] ET CINS Active Threat Intelligence Poor Reputation IP group 94 [**] [Classification: Misc Attack] [Priority: 2] {UDP} 93.174.95.106:27221 -> 82.95.196.202:7
11/15/2019-13:15:32.845110  [**] [1:2402000:5363] ET DROP Dshield Block Listed Source group 1 [**] [Classification: Misc Attack] [Priority: 2] {TCP} 185.156.73.21:44503 -> 82.95.196.202:43935
11/15/2019-13:16:23.399397  [**] [1:2402000:5363] ET DROP Dshield Block Listed Source group 1 [**] [Classification: Misc Attack] [Priority: 2] {TCP} 185.175.93.27:58989 -> 82.95.196.202:53166

All 'Dshield Block Listed' and 'Poor Reputation IP' traffic is port scans to ports that are blocked. So it's not a surprise those IPs have a poor reputation or a Dshield listing.

After hearing about intrusion detection systems a few times I decided to give one a try at home. Although a lot of attacks are blocked I sometimes see weird attacks and it would be nice to have a better idea of what exactly the attack was.

Yes, I have weird interests sometimes. I'm glad I have an ISP (xs4all) where I can select the option 'give me the completely unfiltered Internet connection' so I even see SMB protocol attempts.

I first tried 'snort' but that doesn't deal with PPP interfaces by default. It can be recompiled to accept those but I did not want that. The next option I heard about is 'Suricata' which is running at the moment.

I was amused by the reports of DDoS-like NTP traffic. Those are caused by the NTP statistics gathering. I know NTP can be abused for generating DDoS traffic but all security reports about NTP servers I manage have been false positives.

Anyway it's running and complaining a lot about the traffic it sees. For example the IPv6 port scan/network mapping attempts I noticed two months ago are still active.

11/13/2019-15:06:59.703451  [**] [1:2002911:6] ET SCAN Potential VNC Scan 5900-5920 [**] [Classification: Attempted Information Leak] [Priority: 2] {TCP} 240e:00f7:4f01:000c:0000:0000:0000:0003:6050 -> 2001:0980:14ca:0001:020d:56ff:fece:ffe1:5901
11/13/2019-15:08:39.645780  [**] [1:2002911:6] ET SCAN Potential VNC Scan 5900-5920 [**] [Classification: Attempted Information Leak] [Priority: 2] {TCP} 240e:00f7:4f01:000c:0000:0000:0000:0003:5167 -> 2001:0980:14ca:0001:020d:56ff:fece:ffe6:5901

It seems the TCP reflective SYN attacks are continuing. In researching my options I saw the option to use a netmask with the iptables recent module.

This helps a bit with the attacks trying to flood an entire block. I've updated the filtering to work by the /24, start a check on a SYN from such a block, end when an ACK flies by and start dropping when the rate is over 10 per 2 minutes.

iptables -A INPUT -p tcp -m tcp --tcp-flags SYN,ACK SYN -m recent --update --seconds 120 --hitcount 10 --name tcpsyn --mask 255.255.255.0 --rsource -j LOGDROP
iptables -A INPUT -p tcp -m tcp --tcp-flags SYN,ACK SYN -m recent --set --name tcpsyn --mask 255.255.255.0 --rsource
iptables -A INPUT -p tcp -m tcp --tcp-flags SYN,ACK ACK -m recent --remove --name tcpsyn --mask 255.255.255.0 --rsource

LOGDROP is a rule to drop packets and ratelimit the logging of dropped packets, to avoid turning a network attack into a disk attack.

But I have to be careful not to make services hard to reach for legitimate clients. The above is working, and during attacks I don't see a single SYN_RECV socket.

Sometimes you really wonder about the amount of errors made by noisy attacks. I noticed the following pattern in the system logs:

nnrpd[7029]: 189.243.177.73 unrecognized Accept-Encoding: identity
nnrpd[7029]: 189.243.177.73 unrecognized Content-Length: 586
nnrpd[7029]: 189.243.177.73 unrecognized Accept-Language: en-us
nnrpd[7029]: 189.243.177.73 unrecognized Host: 74.219.111.25
nnrpd[7029]: 189.243.177.73 unrecognized Accept: */*
nnrpd[7029]: 189.243.177.73 unrecognized User-Agent: ApiTool
nnrpd[7029]: 189.243.177.73 unrecognized Connection: close
nnrpd[7029]: 189.243.177.73 unrecognized Cache-Control: max-age=0
nnrpd[7029]: 189.243.177.73 unrecognized Content-Type: text/xml
nnrpd[7029]: 189.243.177.73 unrecognized Authorization: Basic YWRtaW46ezEyMjEzQkQ...

With some searching I eventually found exploit code for certain series of digital video recorders which can be anywhere on the wide Internet.

The whole protocol mismatch makes this a lot noisier via the nntp port than via http, but I also see some attempts via the http port.

Update: Suricata doesn't recognize the specific attack, but it does notice the HTTP basic auth in the traffic:

11/13/2019-20:12:33.772828  [**] [1:2006402:11] ET POLICY Incoming Basic Auth Base64 HTTP Password detected unencrypted [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {TCP} 188.59.207.57:43753 -> 82.95.196.202:119

I'm seeing lots of sockets in state SYN_RECV again and noticed this time my earlier iptables rules to not respond to tcp syn packets that don't build up a connection aren't working. Between two syn packets from the same source there is 5 minutes, so my system responds to all of them. Ranges of addresses in the same block are used as source IPv4 addresses. For one address the traffic is very minimal:

22:40:51.600077 IP 112.175.120.39.58275 > 82.95.196.202.22: Flags [S], seq 720891004, win 29200, length 0
22:40:51.600392 IP 82.95.196.202.22 > 112.175.120.39.58275: Flags [S.], seq 1729897232, ack 720891005, win 29200, options [mss 1460], length 0
22:40:52.612035 IP 82.95.196.202.22 > 112.175.120.39.58275: Flags [S.], seq 1729897232, ack 720891005, win 29200, options [mss 1460], length 0
22:40:54.628048 IP 82.95.196.202.22 > 112.175.120.39.58275: Flags [S.], seq 1729897232, ack 720891005, win 29200, options [mss 1460], length 0
22:40:58.660031 IP 82.95.196.202.22 > 112.175.120.39.58275: Flags [S.], seq 1729897232, ack 720891005, win 29200, options [mss 1460], length 0
22:41:06.851865 IP 82.95.196.202.22 > 112.175.120.39.58275: Flags [S.], seq 1729897232, ack 720891005, win 29200, options [mss 1460], length 0
22:41:22.980000 IP 82.95.196.202.22 > 112.175.120.39.58275: Flags [S.], seq 1729897232, ack 720891005, win 29200, options [mss 1460], length 0
22:45:18.565999 IP 112.175.120.39.41767 > 82.95.196.202.465: Flags [S], seq 910623633, win 29200, length 0
22:45:18.566415 IP 82.95.196.202.465 > 112.175.120.39.41767: Flags [S.], seq 3977721413, ack 910623634, win 29200, options [mss 1460], length 0
22:45:19.588000 IP 82.95.196.202.465 > 112.175.120.39.41767: Flags [S.], seq 3977721413, ack 910623634, win 29200, options [mss 1460], length 0
22:45:21.604022 IP 82.95.196.202.465 > 112.175.120.39.41767: Flags [S.], seq 3977721413, ack 910623634, win 29200, options [mss 1460], length 0
22:45:25.667936 IP 82.95.196.202.465 > 112.175.120.39.41767: Flags [S.], seq 3977721413, ack 910623634, win 29200, options [mss 1460], length 0
22:45:33.860000 IP 82.95.196.202.465 > 112.175.120.39.41767: Flags [S.], seq 3977721413, ack 910623634, win 29200, options [mss 1460], length 0
22:45:49.987965 IP 82.95.196.202.465 > 112.175.120.39.41767: Flags [S.], seq 3977721413, ack 910623634, win 29200, options [mss 1460], length 0

But multiply this with several source IPs in the same IPv4 /24 block and a lot of open servers in the world and suddenly you get a lot of return traffic.

I noticed some really unused records in one zone which is now DNSSEC signed. For example I still had gplus.idefix.net to point at my Google+ page.

So I removed them and did the signing after increasing the serial number. Indeed the records that had no update kept their original signature and the records that where changed (such as the SOA because of the serial number) were signed with new signatures.

Today I was reminded of the first zone I signed with DNSSEC and did the check again with DNSViz. And I saw a lot of error messages. Some searching found that I let all the signatures expire (after the default time of 30 days).

Solution: re-sign the zone and have a careful look at when I need to sign the zones again. Officially just in time for expiry time of the signature (default 30 days) minus TTL of the record.

Obviously this process has to be automated. In the first go I decided to force new signatures after 21 days. But I tested some things later and decided to go for more regular checks of the ages of the signatures and refresh the signatures that are about to expire. This is usually reserved for 'big' zones with lots of resolvers querying them but I decided to implement this myself to avoid problems, and learn more about DNSSEC.

The magic signing command is now:

-zone-signedserial:
    named-checkzone $* $^
    ./SOA.pl $^
    dnssec-signzone -S -K /etc/bind/keys -g -a -r /dev/random -D -S -e +2592000 -i 604800 -j 86400 -o $* $^
    rndc reload $*
    touch $@

The expiry is set with -e at 30 days, the checkinterval with -i at 7 days and the jitter factor with -j at 1 day.

Now there is a special part in the Makefile to be called from cron on a regular basis. It won't produce any output when there is nothing to update.

agecheck:
    @for zone in $(SIGNEDZONES); do if [ `find $${zone}-signedserial -mtime +7 -print` ]; then touch $${zone}-zone ; $(MAKE) --no-print-directory $${zone}-signedserial; fi ;done

The Make variable SIGNEDZONES is filled with the zonenames of the zones that have to be kept DNSSEC signed. File structure for each forward zone is as listed in first zone with valid DNSSEC signatures.

So now almost all my domains are DNSSEC signed. A learning experience and a good level of security.

While looking at some network issues at home I noticed some weird traffic coming in from the outside: forged SYN traffic. Fast enough to trigger my iptables rules to stop being part of tcp syn attacks so all traffic gets dropped. Searching for a bit finds Hell of a Handshake: Abusing TCP forReflective Amplification DDoS Attacks - usenix which discusses this kind of attack.

At the moment it's about 1 or 2 packets per second. The traffic itself isn't notable on my connection and even without the firewall rules it still wouldn't impact my system. But do this with a lot of systems on the Internet running some tcp service and quite some traffic will go to the targeted IP address.

I guess someone doesn't like some Maltese Casino website. I don't like casino websites either because they promote addictive behaviour but I'm not about to use a DDoS.

A search for the top 10 tried usernames for ssh gives a nice list:

     52 admin
     23 pi
     19 test
      7 oracle
      6 support
      6 nagios
      5 user
      5 ubnt
      4 ftpuser
      3 virtualbookcase

I noticed some interesting traffic in my home network this morning, an attempt at finding IPv6 systems. Since IPv6 privacy enhancements are enabled on most systems this is exactly like finding a needle in a haystack.

I noticed an amount of outgoing icmpv6 traffic, and looking at the destination addresses and the type of traffic found lots of 'unreachable route' messages to a few Chinese IPv6 addresses. Searching for the netblock '240e:f7:4f01:c' finds more reports of portscanning activity.

10:14:27.761704 IP6 (hlim 239, next-header TCP (6) payload length: 24) 240e:f7:4f01:c::3.12980 > 2001:980:14ca:1:5054:ff:feae:17.902: Flags [S], cksum 0xd0a9 (correct), seq 3726392987, win 29200, options [mss 1460], length 0
10:14:28.278108 IP6 (hlim 239, next-header TCP (6) payload length: 24) 240e:f7:4f01:c::3.19933 > 2001:980:14ca:1:5054:ff:feae:8003.12587: Flags [S], cksum 0xe1cc (correct), seq 95632679, win 29200, options [mss 1460], length 0
10:14:29.219766 IP6 (hlim 239, next-header TCP (6) payload length: 24) 240e:f7:4f01:c::3.41487 > 2001:980:14ca:1:5054:ff:feae:fff2.902: Flags [S], cksum 0x3c31 (correct), seq 500442149, win 29200, options [mss 1460], length 0
10:14:33.637405 IP6 (hlim 239, next-header TCP (6) payload length: 24) 240e:f7:4f01:c::3.35832 > 2001:980:14ca:1:5054:ff:feae:15.902: Flags [S], cksum 0xa6ea (correct), seq 2324914849, win 29200, options [mss 1460], length 0
10:14:34.468975 IP6 (hlim 239, next-header TCP (6) payload length: 24) 240e:f7:4f01:c::3.12470 > 2001:980:14ca:42::ffe8.16992: Flags [S], cksum 0x5a72 (correct), seq 3249792078, win 29200, options [mss 1460], length 0
10:14:34.469038 IP6 (flowlabel 0x63971, hlim 64, next-header ICMPv6 (58) payload length: 72) 2001:980:14ca:61::13 > 240e:f7:4f01:c::3: [icmp6 sum ok] ICMP6, destination unreachable, unreachable route 2001:980:14ca:42::ffe8
10:14:35.230776 IP6 (hlim 239, next-header TCP (6) payload length: 24) 240e:f7:4f01:c::3.63145 > 2001:980:14ca:1:20d:56ff:fece:8006.19: Flags [S], cksum 0xb87b (correct), seq 4259180220, win 29200, options [mss 1460], length 0
10:14:35.952841 IP6 (hlim 239, next-header TCP (6) payload length: 24) 240e:f7:4f01:c::3.9056 > 2001:980:14ca:42::8013.16992: Flags [S], cksum 0xbb3b (correct), seq 2896438720, win 29200, options [mss 1460], length 0
10:14:35.952880 IP6 (flowlabel 0x63971, hlim 64, next-header ICMPv6 (58) payload length: 72) 2001:980:14ca:61::13 > 240e:f7:4f01:c::3: [icmp6 sum ok] ICMP6, destination unreachable, unreachable route 2001:980:14ca:42::8013

My previous test with DNSSEC zone signing showed a problem with entropy in virtual machines. Today I had time to reboot the home server running the virtual machines including the virtual machine with the nameserver, based on bind9.

Now I can create DNSSEC signatures for zonefiles at high speed (0.028 seconds) with enough entropy available. My first test is with camp-wireless.com which is a domainname for redirecting to Camp Wireless but since that variant was mentioned somewhere I had to generate the redirects to the right version.

The next step was to upload the DS records for the zone to my registrar and get them entered into the top level domain. This failed on the first attempt, the DS records have to be entered very carefully at the registrar.

I tested the result with dnsviz for camp-wireless.com and found an error in the first try: I updated the serial after signing the zone. So the soa record wasn't signed correctly anymore.

I updated my zonefile Makefile to do the steps in the right order:

-zone-signedserial:
        named-checkzone $* $^
        ./SOA.pl $^
        dnssec-signzone -S -K /etc/bind/keys -g -a -r /dev/random -D -S -o $* $^
        rndc reload $*
        touch $@

For the zone camp-wireless.com the original data is in camp-wireless.com-zone, the DNSSEC signatures in camp-wireless.com-zone.signed. And make will abort when one of the commands gives an error level, so it will for example stop completely when I make a typo in the zonefile which will make named-checkzone fail. The -D option creates a file to be used with $INCLUDE in the original zonefile. This does create a circular dependency: named-checkzone will fail when the -signedserial file isn't available on the first run. So the first run will have to be manually.

So now the zone is signed correctly. The next developments will be to find out how to monitor this extensively so I won't be surprised by problems and to redo the signing from time to time to make DNSSEC zone walking very hard.

And when I trust all of this I will implement it on other domain names that I manage.

Today some of the letsencrypt certificates were older than 60 days, so the renewal script started to kick in. Last year I completely automated the certificate renewal of letsencrypt certificates with dehydrated and wrote some scripts around the renewal process with hopefully enough error handling.

Today some of the error handling got tested, one renewal gave an error:

  + ERROR: An error occurred while sending post-request to https://acme-v02.api.letsencrypt.org/acme/new-order (Status 500)

And indeed the dehydrated script gave an error level, the resulting (empty!) .crt file wasn't copied and nothing happened. On the next run of the renewal script this certificate will still be older than 60 days and therefore the renewal will be tried again.

IPv6 is growing up: I saw an ssh attempt to an inside machine, reachable only via IPv6. The source was a Chinese IPv6 address which had not tried anything on any other public service.

Jul 30 18:39:02 ritchie sshd[27454]: Bad protocol version identification '\026\003\001' from 240e:d9:d800:200::212 port 44926

A month later I'm still seeing SMTP floods from 185.222.211.11 and adjacent addresses. I activated the sendmail-reject filter ruleset in fail2ban which keeps several addresses in that range blocked most of the time.

Given reports like 185.222.211.238 | Cloud Core LP | AbuseIPDB and 185.222.211.243 | Cloud Core LP | AbuseIPDB I'm not the only one seeing abuse from this range.