Reconsidering data gathering and processing / 2020-02-16

2020-02-16 Reconsidering data gathering and processing 1 year ago
For years and years I've been doing a lot of data gathering and storing the data using rrdtool. Data such as temperatures from lots of places, from mainboard CPU sensors to an outside weather station, other weather data, web traffic data, house electricy and gas usage, solar power. I started doing this with mrtg in 2001 and switched to rrdtool.

There are some improvements to this system, such as maintaining the rrd files on one machine and doing measurements on other machines in the form of timestamped files to be transported to the machine with rrd via rsync-over-ssh. This allows the central database to do a catch-up of decentrally gathered data after an interruption.

All in all there are two disadvantages at the moment: the system isn't very flexible, adding a datasource means making the big decision about how much data to keep how long and what I want to look at.

Diskspace isn't as constrained as it once was, I may want to keep some data forever and I may want to zoom in to a period a bit longer ago. So I'm looking at different solutions. For one dataset I already added an alternate datastore: the electricity and gas meter readings get copied to a postgres database once a day so I can look at the daily readings forever.

So the search is on for the ideal solution. For gathering and transporting data I am looking at mqtt, a lightweight protocol to gather and transmit data. This also makes it easier to have multiple data collectors look at one source so I can test with a few things first before I make a real switch to any new system.

Tags: ,

IPv6 check

Running test...
, reachable as koos+website@idefix.net. PGP encrypted e-mail preferred. PGP key 5BA9 368B E6F3 34E4 local copy PGP key 5BA9 368B E6F3 34E4 via keyservers

RSS
Meningen zijn die van mezelf, wat ik schrijf is beschermd door auteursrecht. Sommige publicaties bevatten een expliciete vermelding dat ze ongevraagd gedeeld mogen worden.
My opinions are my own, what I write is protected by copyrights. Some publications contain an explicit license statement which allows sharing without asking permission.
Other webprojects: Camp Wireless, wireless Internet access at campsites, The Virtual Bookcase, book reviews
This page generated by $Id: newsitem.cgi,v 1.54 2020/12/31 15:36:31 koos Exp $ in 0.031912 seconds.