News archive November 2021 - Koos van den Hout

Archive by year: 1999 | 2000 | 2001 | 2002 | 2003 | 2004 | 2005 | 2006 | 2007 | 2008 | 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 | 2021

2021-11-22 Resizing a filesystem through several layers
For work I use a supplied laptop with Windows 10. For some of my work I want to have a Linux environment available so I have VirtualBox with a Linux virtual machine running. And because some of the work I do on that Linux virtual machine I use full-disk encryption. And the installation was done with the encrypted lvm setting.

Resizing the filesystem because it was getting full turned out to be a lot of steps! After stopping the virtual machine I wanted to resize the disk from the VirtualBox media manager but that gave an error. After that I tried the commandline, giving about the same error:
> "\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifymedium rotterdam.vdi --resize 32768
0%...
Progress state: VBOX_E_NOT_SUPPORTED
VBoxManage.exe: error: Failed to resize medium
VBoxManage.exe: error: Resizing to new size 34359738368 is not yet supported for medium 'C:\Users\hout0101\VirtualBox VMs\rotterdam\rotterdam.vdi'
VBoxManage.exe: error: Details: code VBOX_E_NOT_SUPPORTED (0x80bb0009), component MediumWrap, interface IMedium
VBoxManage.exe: error: Context: "enum RTEXITCODE __cdecl handleModifyMedium(struct HandlerArg *)" at line 816 of file VBoxManageDisk.cpp
It turns out the .vdi is the wrong type for dynamic resizing. Solution: clone it! The new .vdi will have the dynamic type automatically and there is a "before" .vdi now on disk to revert to if anything goes wrong.
> "\Program Files\Oracle\VirtualBox\VBoxManage.exe" showhdinfo rotterdam.vdi
UUID:           f832b0b4-8738-491d-bd9c-291d755a4af7
Parent UUID:    base
State:          created
Type:           normal (base)
Location:       C:\Users\hout0101\VirtualBox VMs\rotterdam\rotterdam.vdi
Storage format: VDI
Format variant: fixed default
Capacity:       26067 MBytes
Size on disk:   26070 MBytes
Encryption:     disabled
Property:       AllocationBlockSize=1048576
In use by VMs:  rotterdam (UUID: 2454dadb-a82d-4d74-bbea-8dcf2b2d1bf1)
> "\Program Files\Oracle\VirtualBox\VBoxManage.exe" clonehd rotterdam.vdi rotterdam-2.vdi
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Clone medium created in format 'VDI'. UUID: 835e2f75-c19d-4e98-865e-d7acf1359fc7
> "\Program Files\Oracle\VirtualBox\VBoxManage.exe" showhdinfo rotterdam-2.vdi
UUID:           835e2f75-c19d-4e98-865e-d7acf1359fc7
Parent UUID:    base
State:          created
Type:           normal (base)
Location:       C:\Users\hout0101\VirtualBox VMs\rotterdam\rotterdam-2.vdi
Storage format: VDI
Format variant: dynamic default
Capacity:       26067 MBytes
Size on disk:   26069 MBytes
Encryption:     disabled
Property:       AllocationBlockSize=1048576
> "\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifymedium rotterdam-2.vdi --resize 32768
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
I moved the old .vdi out of the way and added the new .vdi to the virtual machine and started it again. This worked fine, but the root volume wasn't any bigger (yet). Next steps: enlarge the extended partition and the Linux partition in it on disk using parted. You really have to know what you are doing here, so I'm not just going to give a cut-and-paste sample.

Now I can resize the encrypted and mounted volume! With the right passphrase.
# cryptsetup resize /dev/mapper/sda5_crypt
And grow the 'physical' (ahem) volume:
# pvresize /dev/mapper/sda5_crypt
Resize the logical volume:
# lvextend /dev/rotterdam-vg/root -l +1674
And finally resize the mounted filesystem:
# resize2fs /dev/mapper/rotterdam--vg-root
And the filesystem has grown, and looks good in a fsck on the next boot.

So solid state disk → Windows filesystem → vdi file → VirtualBox → disk in Linux virtual machine → partition → lukscrypt → logical volume manager → volume → filesystem.

Tags: , ,
2021-11-21 Help in debugging DMARC/SPF/DKIM from xs4all
This morning I had DMARC reports in my mail from xs4all. With all the testing of DMARC, DKIM and SPF I did yesterday I was confused for a while what caused this since my domain names shouldn't be linked to xs4all anymore in any way.

First I thought one of the DMARC testing autoresponders I tried was linked to xs4all or maybe somehow my domains still show some link to xs4all, but after a while it dawned on me that all the testing of DKIM was done via a forward that is still active on my now closed xs4all account.

My DMARC record currently says I want reports of problems, so I am getting those. Anyway, I guess it's all working and I'm seeing no new problems.

Oh and outlook[.]com is still rejecting my e-mail so no progress there. But then again Microsoft and not handling Internet e-mail standards correctly was something I ranted about before: 17 and a half years ago. Not a lot of improvement.

Tags: ,
2021-11-20 Setting the right SPF records
In debugging mail from the shell server I noticed something in the headers:
Authentication-Results: xs4all.nl; spf=none smtp.mailfrom=gosper.idefix.net;
        dkim=pass header.d=idefix.net; dmarc=pass header.from=idefix.net
The shell server sees itself as gosper.idefix.net and uses this on locally generated outgoing mail. I only had an SPF record for idefix.net so setting one up for gosper.idefix.net too can help fix things. I also need a DMARC policy allowing mail from subdomains of idefix.net, with more specific DMARC policies for active subdomains.

Tags: ,
2021-11-20 Publishing the information about using DKIM: dmarc records
After getting DKIM signing running with sendmail and opendkim I generated DKIM keys for idefix.net, configured them in the mailserver with opendkim and published them in DNS. The next thing to publish is a policy record showing that all outgoing mail for these domains should be signed.

I started with a policy that shows mail should be signed but to not reject it when it isn't, but report it to me as unsigned.
;; QUESTION SECTION:
;_dmarc.camp-wireless.com.      IN      TXT

;; ANSWER SECTION:
_dmarc.camp-wireless.com. 86400 IN      TXT     "v=DMARC1;p=none;sp=reject;pct=100;rua=mailto:dmarcreports at camp-wireless.com;"
With a similar policy for idefix.net. Mail with problems shouldn't be rejected yet: DNS propagation isn't instantaneous and testing first.

Tags: ,
2021-11-20 Trying to get DKIM running
My recent issues with getting my e-mail delivered made me look at DKIM signing of outgoing e-mail messages. To not break things I have started testing this with outgoing e-mail from camp-wireless.com which normally publishes it doesn't send mail at all, so the first steps were to change that policy: changing the MX record and SPF record.

I started reading into configuring sendmail with dkim and found OpenDKIM which can work as a sendmail milter.

Based on How to configure DKIM & SPF & DMARC on Sendmail for multiple domains on CentOS 7 I took the same steps for my Devuan installation.

In Devuan (and probably Debian/Ubuntu) there is a opendkim package for the service and a opendkim-tools package for the associated tools. I needed the second one to get the opendkim-genkey command. I can imagine keys being generated/managed on a different system than the actual signing server.

After configuring this for camp-wireless.com including generating a keypair and publishing the public key via DNS I started sending test messages but had no luck. It turned out the sending host has to be in the InternalHosts table of opendkim. I added the address ranges and after that things started to work.

After fixing that I got the results I wanted:
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=camp-wireless.com;
        s=gosper; t=1637408594;
        bh=YewDlohOT9RvALNQw4cVukwSpmAm5tXtGWJxLDUJZa4=;
        h=To:From:Subject:Date:From;
        b=GGMEeCY5xmgFDBQ5NzgZfAVvyr+ctBKOTGpwMqq1W/tgJYMY8WyzaM5XfEiWijGKr
        abBN5WLbiyoXsd62lNVxcDOBUYWzkOnwZCw5WgdlzZJSIxgRdnWMQLxL1E9BJdudwR
        zriX1/vAaR34RFM1kiSVp0dqa98/Kxfdp2DPPRDsAVJ6sdxqz1YHD4odveDcLEQQZv
        jUMNPVmQps90mZORtdKtOOWQP0RYkZvmjNsJZuwIrRkFvUzOmAVT6MDDf4kZ35lbes
        oAp0me8tQgoffNLRQpO7akSKhbh1Kn5fAv50WILhM0rK/ChkWqvOrcfgIwbSSPduzM
        DI1w23jCnwaKQ==
And a verification:
Authentication-Results: xs4all.nl; spf=pass smtp.mailfrom=camp-wireless.com;
dkim=pass header.d=camp-wireless.com
I was wondering about roaming users who authenticate to my mailserver and send messages that way. In a first test those messages get signed too. That means I can start signing mail from idefix.net and other production domain names!
Read the rest of Trying to get DKIM running

Tags: , ,
2021-11-19 Attacks on new sites are fast!
I was working on a new site for a project and requested a certificate for it. The time between the certificate being generated and the first attack was 3 minutes and 7 seconds.

15:12:10 UTC: certificate generated and published on the certificate transparancy log
15:15:17 UTC:
185.67.34.1 - - [19/Nov/2021:16:15:17 +0100] "GET /restapi.php HTTP/1.1" 404 1008 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0"
15:15:18 UTC:
185.67.34.1 - - [19/Nov/2021:16:15:18 +0100] "POST /gate.php HTTP/1.1" 404 1008 "-" "Dalvik/2.1.0 (Linux; U; Android 6.0.1; SM-J700F Build/MMB29K)"

Tags: , ,
2021-11-15 Blocking mail I can't answer
Someone mailed me a few days ago with an interesting question. So I typed a reasonably long answer. But upon sending this answer I received the following error message:
   ----- The following addresses had permanent fatal errors -----
<????????@outlook.com>
    (reason: 550 5.7.1 Unfortunately, messages from [45.83.232.134] weren't sent. Please contact your Internet se...ail.live.com/mail/troubleshooting.aspx#errors. [HE1EUR04FT003.eop-eur04.prod.protection.outlook.com])

   ----- Transcript of session follows -----
... while talking to outlook-com.olc.protection.outlook.com.:
>>> MAIL From:<?????? .at. idefix.net> SIZE=4837 BODY=8BITMIME
<<< 550 5.7.1 Unfortunately, messages from [45.83.232.134] weren't sent. Please contact your Internet service provider since part of their network is on our block list (S3150). You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors. [HE1EUR04FT003.eop-eur04.prod.protection.outlook.com]
554 5.0.0 Service unavailable
Trying to get my IPv4 address allowed didn't work. The form for getting IP addresses whitelisted did not allow for IPv6 addresses, but then again outlook.com has no IPv6 addresses listed for its MX record. I would think microsoft would do something with IPv6 to support innovations in Internet but I guess they only do that to win contracts.

After a while I got a response with a ticket number and an hour later a response that looked like maybe a person had taken a look at it, with "Our investigation has determined that the above IP(s) do not qualify for mitigation." So that leaves me with possible mails from outlook[.]com that I can't answer, making me look bad because I don't seem to reply at all.

I'm convinced the mail setup is correct on my end. The domain idefix.net has an SPF record and the mail was sent out via the approved route.

The only solution I can think of at the moment is blocking mail from outlook.com at the protocol level with an error message pointing at a webpage what the problem is, so when someone sends an e-mail from outlook[.]com to one of my domains they will get an error message with an embedded hint what they should do, namely We cannot reply to your mail, please send us mail from a different domain, see https://idefix.net/mailreject.html for an explanation. About the same as microsoft does, although the careful reader might have noticed the error code S3150 is not mentioned at http://mail.live.com/mail/troubleshooting.aspx#errors.

Tags: , , , ,
2021-11-12 Meer magische getallen in Sunspec Modbus
Na een poosje blijkt ook het getal 65534 (0xFFFE) in sunspec modbus antwoorden een vorm van 'geen geldige uitlezing' te kunnen zijn. Ik heb de scripts die de gegevens ophalen en verwerken richting influxdb hier op aangepast.

Tags: , ,
2021-11-11 De S_OK is weer terug op de SolarEdge omvormer
Na ongeveer 36 uur was de omvormer waarvan ik de netwerkaansluiting had omgezet nog steeds niet aan het communiceren met solaredge. In het netwerkverkeer zag ik wel wat tcp communicatie, maar die leek geen data te bevatten (allemaal packets met 0 bytes data).

Ik zat nog even te zoeken in een handleiding over de SolarEdge communicatie met de monitoring dienst en kwam toen tegen dat expliciet ingesteld moet worden langs welke route de communicatie voor monitoring loopt. En die had ik nog niet expliciet ingesteld op LAN. Nadat ik dat gedaan had stond er gelijk weer een S_OK op het display en ging er ineens wel data stromen en zie ik dingen bijgewerkt worden bij de solaredge monitoring dashboard.

Tags: ,
2021-11-10 Nieuwe firmware geïnstalleerd in SolarEdge omvormers
Het lokaal uitlezen van de SolarEdge omvormers via Modbus/TCP ging op den duur toch weer haperen, zeker over wifi.

Ik begreep uit wat beschrijvingen dat het bijwerken van de firmware in de omvormer hierbij zou moeten helpen dus dat heb ik vandaag gedaan. Ik had eerder naar een van de omvormers ook een netwerkkabel gelegd maar nog niet de omvormer open gemaakt om die van binnen aan te sluiten. Nu moest de omvormer ook open om de firmware bij te werken dus heb ik gelijk het ontbrekende stukje UTP kabel gemaakt en aangesloten, en de wifi module verwijderd.

De SolarEdge handleiding voor het bijwerken van omvormer firmware had na alle stappen om de omvormer veilig open te maken een aantal stappen voor het kiezen van een upgrade maar de omvormer 'zag' gelijk bij het inschakelen dat er een kaartje met firmware in zat en startte vanzelf de upgrade.

Op beide omvormers is de upgrade gelukt. De omvormer waar ik ook de netwerkaansluiting omgezet heb van wifi naar bedraad wil momenteel geen gegevens doorgeven naar SolarEdge monitoring (ik zie wel verkeer naar buiten gaan) en er staat ook geen S_OK op het display. Ik denk dat dat komt door de wijziging netwerkaansluiting. Volgens wat bronnen kan dat even duren, maximaal 2 dagen.

Ondertussen kan ik nu via Modbus/TCP de nodige gegevens goed uitlezen, en ik heb wat veranderingen aan het monitoring script gemaakt zodat er ook data van de afzonderlijke omvormers naar een influxdb database gaat.

Tags: , ,
2021-11-09 Zonnepanelen omvormers lokaal monitoren (2)
Gisterenavond de andere omvormer ingesteld voor Modbus TCP. Ook hier hetzelfde effect, de omvormer reageert eerst niet. Het lijkt dat er een dag/nacht overgang of een aantal uren wachten nodig is voor deze instelling overgenomen is.

Nu met zonlicht:
$ ./sunspec-status -v se-boven -m 0
INVERTER:
             Model: SolarEdge  SE2200
  Firmware version: 3.2434
     Serial Number: xxxxxxxx

            Status: ON (MPPT)

 Power Output (AC):          129 W
  Power Input (DC):          130 W
        Efficiency:        98.51 %
  Total Production:     2952.829 kWh
      Voltage (AC):       238.90 V (49.99 Hz)
      Current (AC):         0.78 A
      Voltage (DC):       380.10 V
      Current (DC):         0.34 A
       Temperature:        30.16 C (heatsink)

Als de omvormer iets meer actief wordt komt er inderdaad een meting uit het DC voltage. In de verwerking moet ik dus echt een uitzondering maken voor 'geen data' bij uitlezing 65535.

Tags: , ,
2021-11-07 Zonnepanelen toch (ook) lokaal monitoren
Onze zonnepanelen met SolarEdge omvormers liggen er al een tijd en na wat nadenken over monitoring heb ik toen toch voor de optie gekozen om de gegevens gewoon aan de solaredge API te vragen en wat te verwerken in rrdtool voor mooie grafiekjes en in een postgresql database voor langdurig bewaren.

Maar de laatste weken krijg ik vrij regelmatig foutcode 429 van de SolarEdge API dat ik teveel queries zou doen. Ik kan niet terugvinden waar dat vandaan komt. Ook na het vervangen van de API key (zodat eventuele andere scripts die ik heb laten slingeren met mijn api key stoppen) blijven deze status 429 resultaten komen:
HTTP Status 429 – Too Many Requests
Message Concurent limit quota exceeded
Description The user has sent too many requests in a given amount of time ("rate limiting").
Tijd om naar de opties te kijken om de omvormers uit te lezen via Modbus over TCP. De code is er: tjko/sunspec-monitor: Monitoring Sunspec (Modbus TCP) compatible Solar Inverters - GitHub maar nu de omvormers zo ver krijgen dat dit werkt.

Update: Handleiding gevonden hoe ik Modbus TCP inschakel op de SolarEdge omvormer. Voorlopig lijk ik er tegenaan te lopen dat de Modbus TCP setting dit maar 2 minuten open stelt en daarna weer afsluit. Terwijl ik dit eigenlijk eens per 5 of 10 minuten wil opvragen. Met een firmware upgrade schijnt dit te verhelpen te zijn.

Update: Een paar uur later snapt de omvormers wel Modbus TCP op poort 502. Vreemd, het lijkt wel alsof het een tijdje duurt voordat de configuratie actief wordt. Hoe dan ook, succes:
$ ./sunspec-status -m 0 -v se-schuur
INVERTER:
             Model: SolarEdge  SE2200
  Firmware version: 3.2434
     Serial Number: xxxxxxxx

            Status: SLEEPING

 Power Output (AC):            0 W
  Power Input (DC):            0 W
        Efficiency:         0.00 %
  Total Production:     3514.256 kWh
      Voltage (AC):       239.30 V (49.99 Hz)
      Current (AC):         0.00 A
      Voltage (DC):      6553.50 V
      Current (DC):         0.00 A
       Temperature:        17.79 C (heatsink)

De 6553.5 volt DC lijkt nog een vreemd iets (uitlezing 65535 keer vermenigvuldigingsfactor -1), ik moet morgen als de omvormer wakker is dit nog eens controleren. Omdat er een aantal andere variabelen ook 65535 zijn lijkt het een geval 'niet actief' of iets dergelijks. Nalezen van de sunspec modbus specificatie geeft bevestiging: waarde 0xFFFF (65535) is voor 'not implemented'.

Ik zie mooie grafiekjes zonnestroom aankomen als ik deze data in influxdb stop en er grafana op loslaat.

Tags: , ,
2021-11-01 I participated in the CQWW DX Contest SSB
Last weekend was the CQ Worldwide DX Contest SSB and I participated Saturday and Sunday. This is a 48-hour contest so I had multiple chances for making radio contacts between other things to do in the weekend.

I was planning to participate in this contest with the idea of getting some new countries in the log, but propagation decided to not cooperate very well. Looking back at the log I see a number of 'well-known' stations: other amateur radio stations that I see active in other contests.

In the end I made 81 contacts on the 20, 15 and 10 meter bands. Overview:
Band   160   80   40   20   15   10
QSO's    0    0    0   75    2    4
Cty      0    0    0   27    2    3
Zone     0    0    0    5    2    2
Pts: 96  Mul: 41 Score: 3936

Tags: , ,

IPv6 check

Running test...
, reachable as koos+website@idefix.net. PGP encrypted e-mail preferred. PGP key 5BA9 368B E6F3 34E4 local copy PGP key 5BA9 368B E6F3 34E4 via keyservers

RSS
Meningen zijn die van mezelf, wat ik schrijf is beschermd door auteursrecht. Sommige publicaties bevatten een expliciete vermelding dat ze ongevraagd gedeeld mogen worden.
My opinions are my own, what I write is protected by copyrights. Some publications contain an explicit license statement which allows sharing without asking permission.
Other webprojects: Camp Wireless, wireless Internet access at campsites, The Virtual Bookcase, book reviews
This page generated by $Id: morenews.cgi,v 1.51 2021/11/09 13:09:49 koos Exp $ in 0.026774 seconds.