News items for tag zigbee - Koos van den Hout

It may seem like a complicated stack compared to monitoring with rrdtool, but the wireless environmental monitoring with zigbee plans are starting to work. The zigbee stick arrived, I found out I needed to upgrade the Raspberry Pi in the utility closet to be able to run zigbee2mqtt so I did that: Raspberry pi monitoring the smart meter is now installed and the zigbee environment sensors arrived and the first one joined the network. After some changes to mqtt-to-influxdb-forwarder I was able to get the data into influxdb.

The final step was to tell grafana where to find the data and create a dashboard visualizing the results, see the screenshot.

I'm also improving small things in using zigbee sensors and improving data collection. After learning about not leaving the network running with permit_join true I had a look at the configuration messages I can send to zigbee2mqtt. It is clear zigbee2mqtt is not just from zigbee sensors to mqtt messages but also the other way around, both to adjust settings in zigbee2mqtt itself and to send commands to zigbee devices.

I did change the friendly_name of the first sensor to the name of the room it is in, and it's now showing up in the statistics under that name. This does break the history, so I should change the name as soon as I add a sensor to the zigbee network.

The recent MicroSD failure in the Raspberry Pi made me look at the logging in zigbee2mqtt as it is running for a long time and default logging includes every received message which would give a lot of wear on the MicroSD in the Raspberry Pi. So I changed the configuration to only log to console.

This is something that can't be changed via an mqtt message, which is logical (otherwise it would have security implications).

I may also look at less system logging to the MicroSD. Someone suggested to have a look at log2ram for this. This creates a ramdisk for logging which is synchronized to persistent storage every day or on shutdown.

I sort of forgot I had zigbee2mqtt running since 18 June and the network enhanced itself with two devices: a 'lidl smart plug' and a 'power supply/relay/dimmer'. The Lidl Silvercrest smart plug (EU, CH, FR, BS, DK) (HG06337) left the network by itself but the Busch-Jaeger Zigbee Light Link power supply/relay/dimmer (6735/6736/6737) was actively reporting and I was able to switch the light on and off.

After resetting it to the state I found it in I tried to remove it from the network (by sending a 'remove' message to zigbee2mqtt) but it came back right away. So I stopped zigbee2mqtt, set permit_join to false and restarted it. After that I gave the 'remove' command again and that worked and it hasn't come back.

Log from zigbee2mqtt with the device id removed:

Zigbee2MQTT:info  2021-07-03 15:59:04: MQTT publish: topic 'zigbee2mqtt/0xd85def11a1004f69', payload '{"brightness_relay":254,"linkquality":33,"state_relay":"OFF"}'
Zigbee2MQTT:info  2021-07-03 15:59:08: Removing '0x****************'
Zigbee2MQTT:info  2021-07-03 15:59:08: Successfully removed 0x****************
Zigbee2MQTT:info  2021-07-03 15:59:08: MQTT publish: topic 'zigbee2mqtt/bridge/log', payload '{"message":"0x****************","type":"device_removed"}'
Zigbee2MQTT:warn  2021-07-03 15:59:08: Device '0x****************' left the network
Zigbee2MQTT:info  2021-07-03 15:59:08: MQTT publish: topic 'zigbee2mqtt/bridge/event', payload '{"data":{"ieee_address":"0x****************"},"type":"device_leave"}'
Zigbee2MQTT:info  2021-07-03 15:59:08: MQTT publish: topic 'zigbee2mqtt/bridge/log', payload '{"message":"left_network","meta":{"friendly_name":"0x****************"},"type":"device_removed"}'
Zigbee2MQTT:warn  2021-07-03 15:59:08: Device '0x****************' left the network
Zigbee2MQTT:info  2021-07-03 15:59:08: MQTT publish: topic 'zigbee2mqtt/bridge/event', payload '{"data":{"ieee_address":"0x****************"},"type":"device_leave"}'
Zigbee2MQTT:info  2021-07-03 15:59:08: MQTT publish: topic 'zigbee2mqtt/bridge/log', payload '{"message":"left_network","meta":{"friendly_name":"0x****************"},"type":"device_removed"}'

Sorry to whoever in the neighbourhood wasn't able to get their new lightswitch/dimmer working with their own hub. It should work now.

I checked the documentation and it's perfectly possible to tell zigbee2mqtt to allow/deny joins (even for a set time) via a message delivered via mqtt: MQTT topics and message structure: zigbee2mqtt/bridge/request/permit_join. I will leave the fixed configuration to joins disabled and will allow a join by hand when there is an actual device to join.

To share my work on mhaas / mqtt-to-influxdb-forwarder : IoT MQTT to InfluxDB forwarder I forked the repository to KHoos / mqtt-to-influxdb-forwarder: zigbee2mqtt to InfluxDB forwarder and committed my changes to the source, and later updated the documentation and committed those changes too.

I thought somebody must have been doing zigbee2mqtt measurements to influxdb before and I was somewhat right: it was mentioned in a bugreport! In issue Can't receive messages from mqtt #3 it shows that someone is already trying to do this but it's not completely there yet. So I cloned mhaas / mqtt-to-influxdb-forwarder : IoT MQTT to InfluxDB forwarder after a few updates it does what I want: correctly use the sensor id as tag and only parse messages that have live sensor data. Zigbee2mqtt likes to publish its internal housekeeping as mqtt messages and I don't need those.

So now I get from

zigbee2mqtt/0x00158d0006fafb00 {"battery":100,"humidity":37.86,"linkquality":84,"pressure":1032,"temperature":28.67,"voltage":3045}

To

DEBUG:forwarder.InfluxStore:Writing InfluxDB point: {'fields': {u'linkquality': 84.0, u'temperature': 28.67, u'battery': 100.0, u'humidity': 37.86, u'pressure': 1032.0, u'voltage': 3045.0}, 'tags': {'sensor_address': u'0x00158d0006fafb00'}, 'measurement': 'environment'}

And available as:

$ influx -database environment -precision rfc3339
Connected to http://localhost:8086 version 1.6.4
InfluxDB shell version: 1.6.4
> select * from environment;
name: environment
time                           battery humidity linkquality pressure sensor_address     temperature voltage
----                           ------- -------- ----------- -------- --------------     ----------- -------
[..]
2021-06-22T21:05:10.227577886Z 100     37.86    84          1032     0x00158d0006fafb00 28.67       3045

I will need to add something with friendly names, but this is a nice start. Data flows!

And regex101 regular expression tester and debugger saved the day in finding how to change the Python regular expression to only accept data from zigbee messages with a sensor address.

The page How to improve network range and stability - zigbee2mqtt.io has using an USB extension cable as the first solution to improve communications. And indeed my earlier problems with zigbee communications now seem solved and I get good quality readings from longer distances.

After reinstalling the Raspberry Pi in the utility closet so it can run newer software I did the steps to install zigbee2mqtt on it which was quite possible this time.

I migrated the settings and the database from my first run of zigbee2mqtt on a linux laptop and on the first try no communication started with the zigbee dongle.

On the second try (different usb port) things started working. The zigbee dongle is currently plugged directly into the Raspberry and as several manuals say this is not an ideal configuration. Those manuals are right: suddenly the sensor about 4 meters away isn't seen. I will need to improve the situation and move the dongle or its antenna to a better location.

Zigbee2MQTT:info  2021-06-18 22:28:15: MQTT publish: topic 'zigbee2mqtt/0x00158d0006fafb00', payload '{"battery":100,"humidity":62.52,"linkquality":147,"pressure":1023,"temperature":27.77,"voltage":3175}'

Because the installation of zigbee2mqtt was not possible on the Raspberry pi in the utility closet I decided to do a reinstallation with Raspbian buster. According to some on-line opinions reinstalling is better than a distro upgrade on a microsd card.

I was lucky to have a spare MicroSD card and a spare Raspberry Pi available. I did the whole installation on the spare and made sure to set up everything already so things would start running. And the thing that I missed was fixed easily with the old configuration still available. The Raspberry Pi in the utility closet was only running the smart meter monitoring, but now it is upgraded it can do more things, and running zigbee2mqtt for checking on wireless sensors in the house is next.

The package with the zigbee environment sensors I ordered arrived this morning and I had to get the first test done right away.

Joining the network/resetting the sensor is easy with a long press of the button and it showed up:

Zigbee2MQTT:info  2021-06-12 11:53:17: Device '0x00158d0006fafb00' joined
Zigbee2MQTT:info  2021-06-12 11:53:17: Starting interview of '0x00158d0006fafb00'
gbee2MQTT:info  2021-06-12 11:53:52: Successfully interviewed '0x00158d0006fafb00', device has successfully been paired
Zigbee2MQTT:info  2021-06-12 11:53:52: Device '0x00158d0006fafb00' is supported, identified as: Xiaomi Aqara temperature, humidity and pressure sensor (WSDCGQ11LM)
Zigbee2MQTT:info  2021-06-12 11:53:52: Configuring '0x00158d0006fafb00'
Zigbee2MQTT:info  2021-06-12 11:53:52: Successfully configured '0x00158d0006fafb00'

And now it gives me data!

Zigbee2MQTT:info  2021-06-12 12:15:53: MQTT publish: topic 'zigbee2mqtt/0x00158d0006fafb00', payload '{"battery":100,"humidity":45.77,"linkquality":153,"pressure":1033,"temperature":24.9,"voltage":3055}'
Zigbee2MQTT:info  2021-06-12 12:15:53: MQTT publish: topic 'zigbee2mqtt/0x00158d0006fafb00', payload '{"battery":100,"humidity":45.85,"linkquality":156,"pressure":1033,"temperature":24.9,"voltage":3055}'
Zigbee2MQTT:info  2021-06-12 12:15:53: MQTT publish: topic 'zigbee2mqtt/0x00158d0006fafb00', payload '{"battery":100,"humidity":45.85,"linkquality":156,"pressure":1033,"temperature":24.9,"voltage":3055}'

Also visible via mqtt:

koos@testrouter:~$ mosquitto_sub -t zigbee2mqtt/0x00158d0006fafb00 -v
zigbee2mqtt/0x00158d0006fafb00 {"battery":100,"humidity":45.77,"linkquality":153,"pressure":1033,"temperature":24.9,"voltage":3055}
zigbee2mqtt/0x00158d0006fafb00 {"battery":100,"humidity":45.85,"linkquality":156,"pressure":1033,"temperature":24.9,"voltage":3055}
zigbee2mqtt/0x00158d0006fafb00 {"battery":100,"humidity":45.85,"linkquality":156,"pressure":1033,"temperature":24.9,"voltage":3055}

I started looking at the instructions for running zigbee2mqtt and the instructions for installing npm/nodejs gave me a lot of error messages on the raspberrypi running in the utility closet and checking the smart meter.

It turns out it needs an upgrade from Raspbian jessie. This Raspberry Pi is dedicated to reading the smart meter since August 2016 and it has been running fine gathering the smart meter data.

The raspbian forums state that it is better to upgrade by reinstallation on a different SD card. So I guess it's time to rebuild the smartmeter Pi if I want it to run the zigbee sensor network.

Update:
I installed all the software on a linux laptop and now I have a running zigbee2mqtt.