2020-02-16 Reconsidering data gathering and processing
For years and years I've been doing a lot of data gathering and storing the data using rrdtool. Data such as temperatures from lots of places, from mainboard CPU sensors to an outside weather station, other weather data, web traffic data, house electricy and gas usage, solar power. I started doing this with mrtg in 2001 and switched to rrdtool. There are some improvements to this system, such as maintaining the rrd files on one machine and doing measurements on other machines in the form of timestamped files to be transported to the machine with rrd via rsync-over-ssh. This allows the central database to do a catch-up of decentrally gathered data after an interruption. All in all there are two disadvantages at the moment: the system isn't very flexible, adding a datasource means making the big decision about how much data to keep how long and what I want to look at. Diskspace isn't as constrained as it once was, I may want to keep some data forever and I may want to zoom in to a period a bit longer ago. So I'm looking at different solutions. For one dataset I already added an alternate datastore: the electricity and gas meter readings get copied to a postgres database once a day so I can look at the daily readings forever. So the search is on for the ideal solution. For gathering and transporting data I am looking at mqtt, a lightweight protocol to gather and transmit data. This also makes it easier to have multiple data collectors look at one source so I can test with a few things first before I make a real switch to any new system.