News items for tag statistics - Koos van den Hout

2021-09-11 Adding physical hardware temperatures in telegraf/influxdb/grafana 5 days ago
Grafana dashboard with host cpu temperatures After starting the collection of a lot of the system data I wanted with telegraf/influxdb/grafana one small part was missing: the temperature sensors. I like these, so I had a look and found the inputs.temp plugin in telegraf which is normally disabled.

Enabling it on hosts that have actual hardware to measure worked ok. On the Raspberry Pi systems it gives one temperature:
> SHOW TAG VALUES ON "telegraf" WITH key="sensor" WHERE host='joy'
name: temp
key    value
---    -----
sensor cpu_thermal_input
On the home server conway it gives quite a lot of temperatures:
> SHOW TAG VALUES ON "telegraf" WITH key="sensor" WHERE host='conway'
name: temp
key    value
---    -----
sensor coretemp_core0_crit
sensor coretemp_core0_critalarm
sensor coretemp_core0_input
sensor coretemp_core0_max
sensor coretemp_core1_crit
sensor coretemp_core1_critalarm
sensor coretemp_core1_input
sensor coretemp_core1_max
sensor coretemp_core2_crit
sensor coretemp_core2_critalarm
sensor coretemp_core2_input
sensor coretemp_core2_max
sensor coretemp_core3_crit
sensor coretemp_core3_critalarm
sensor coretemp_core3_input
sensor coretemp_core3_max
sensor coretemp_core4_crit
sensor coretemp_core4_critalarm
sensor coretemp_core4_input
sensor coretemp_core4_max
sensor coretemp_core5_crit
sensor coretemp_core5_critalarm
sensor coretemp_core5_input
sensor coretemp_core5_max
sensor coretemp_physicalid0_crit
sensor coretemp_physicalid0_critalarm
sensor coretemp_physicalid0_input
sensor coretemp_physicalid0_max
For the dashboard showing all relevant temperatures for a system this is a bit overkill and makes the dashboard hard to read. Solution: go for all the temperature sensors that end in 'input', with the variable in the dashboard defined as 'ending in input':
> SHOW TAG VALUES ON "telegraf"  WITH key="sensor" WHERE host='conway' AND sensor=~/input$/
name: temp
key    value
---    -----
sensor coretemp_core0_input
sensor coretemp_core1_input
sensor coretemp_core2_input
sensor coretemp_core3_input
sensor coretemp_core4_input
sensor coretemp_core5_input
sensor coretemp_physicalid0_input
So far this works with all physical systems.

Tags: , ,
2021-09-09 Collecting more system data with Telegraf for Influxdb/Grafana 1 week ago
Grafana host dashboard with telegraf data including entropy
Grafana host dashboard with telegraf data including entropy. The dip in entropy is caused by the dnssec-signzone process
I have been collecting certain system data for ages with rrdtool, but now I see what is possible with Telegraf collecting agent and after some initial attempts I'm all in favour and data is flowing.

All the data I collected is already standard in telegraf, including entropy! Other data is also collected that is good to keep an eye on for performance.

I made some tweaks to the standard telegraf configuration: collect every 5 minutes, not exactly on the clock since I read The mystery of load average spikes which reminded me of my own experience Be very careful of what you measure. I also avoid gathering data on nfs filesystems (which come and go thanks to autofs).

I rolled out telegraf over all systems at home, and now there is a nice 'System info' dashboard in Grafana.

Tags: , ,
2021-09-05 Network traffic statistics in Influxdb/Grafana 1 week ago
Grafana dashboard with network traffic I continued my slow migration of statistics to Influxdb/Grafana and added the network traffic. I've been gathering this for ages in rrdtool, my earlier view was that I've been using rrdtool for network and other statistics since October 2002 so it is a bit of a change.

I updated the perl scripts that fetch network traffic statistics over SNMP to also add the data to influxdb. And it was simple to create a dashboard with that data. The overview pages with data for all interfaces for one measured host also link to detail pages per interface which also show the number of errors.

Tags: ,
2021-07-29 Zigbee - zigbee2mqtt - mqtt-to-influxdb-forwarder - influxdb - grafana working 1 month ago
It may seem like a complicated stack compared to monitoring with rrdtool, but the wireless environmental monitoring with zigbee plans are starting to work. The zigbee stick arrived, I found out I needed to upgrade the Raspberry Pi in the utility closet to be able to run zigbee2mqtt so I did that: Raspberry pi monitoring the smart meter is now installed and the zigbee environment sensors arrived and the first one joined the network. After some changes to mqtt-to-influxdb-forwarder I was able to get the data into influxdb.

The final step was to tell grafana where to find the data and create a dashboard visualizing the results, see the screenshot.

I'm also improving small things in using zigbee sensors and improving data collection. After learning about not leaving the network running with permit_join true I had a look at the configuration messages I can send to zigbee2mqtt. It is clear zigbee2mqtt is not just from zigbee sensors to mqtt messages but also the other way around, both to adjust settings in zigbee2mqtt itself and to send commands to zigbee devices.

I did change the friendly_name of the first sensor to the name of the room it is in, and it's now showing up in the statistics under that name. This does break the history, so I should change the name as soon as I add a sensor to the zigbee network.

Tags: , , ,
2021-07-02 Be very careful of what you measure 2 months ago
Screenshot change in ping time I realized the high maximum IPv4 ping times never showed up when I redid the measurement by hand, and IPv4 was the first test in the crontab line.

So I guess a lot of scripts ping ping.xs4all.nl every 5 minutes and it's a bit congested at exactly that time. I added a small delay before the start of the measurement and suddenly the strange peaks are gone.

Tags: ,
2021-06-07 Wireless enviromental monitoring with zigbee 3 months ago
Since January 2008 I measure temperatures and other environmental data in and around the house with 1-wire sensors and adaptors. These work fine but need wires between the sensors and that isn't ideal for quick spot measurements.

So I looked into other options recently, and found affordable zigbee temperature/air pressure/humidity sensors. And an USB zigbee interface which works with linux and with a lot of the available application software. Because the next problem is going sensor - zigbee network - zigbee usb interface - some magic - database of measurements.

Because I see myself wanting long series of measurements from a number of places in the house and testing without breaking those series I ordered two USB zigbee interfaces and eight environmental sensors. I guess I want production and development enviromental monitoring.

The zigbee stick is not the cheapest solution but it is documented to work with zigbee2mqtt and buying one supports work on that software.

Tags: , ,
2021-03-01 Updating my statistics gathering 6 months ago
Grafana dashboard sample For years and years I have been using rrdtool to gather and graph statistics at home. I started gathering home temperatures around 2008 but I see NTP statistics gathering from 2003 and my last mrtg graphs were created in October 2002. So that suggests I've been using rrdtool since that date.

Anyway, I'm looking at newer options. After some asking around I installed influxdb and started gathering data. I adjusted some of my data gathering scripts around rrdtool to also put the data in influxdb.

The easiest data to gather and graph was the load average, available entropy and number of processes for a number of systems at home. So that dashboard has been built and allows selection of the wanted computer.

My first conclusion is just collecting data and thinking what kind of graphs to create later is a lot easier with influxdb. With rrdtool the round robin database is designed around the graphs you want. In this case I just start gathering data and when data has come in start playing with possible graphs from that data.

The next challenge is to set the rules for maintaining the old data. One of the triggers to look at other options was that I was at the end of a nearly 11-year cycle of stored temperatures in rrdtool, and I wanted to keep that history if possible.

I don't have to keep every measurement forever, but with storage being cheap I think I will keep daily averages forever when this is 'production'.

Tags: , ,
2021-01-01 Closing 2020 in amateur radio with a plot of the number of contacts and a review 8 months ago
QSO count plot up to December 2020 Time to plot the number of contacts in 2020 and a review. I made no specific resolutions for 2020 but looking back there were positive developments.
  • The Kenwood TS480-SAT is at a remote location with good antennas for most of the HF bands. This enabled me to work new countries and get more voice and morse contacts in the log.
  • I was active on amateur satellites a few times, including from Austria.
  • The morse speed improved and I got on the air more with morse. Including a few morse contests.
  • I tried to follow the Bulgarian Saints 2020 stations and I had at least one contact with one of the stations in 10 out of the 12 months of 2020. In 8 months I had at least one contact in morse with the station of that month. So I earned the Bulgarian Saints diploma 2020.
  • In general I made more contacts in this year than in any other year. The endfed antenna is now mounted outside in such a way I can leave it there, which makes getting on the radio for a few contacts easier. There were also more special event stations active this year.
  • I had radio contacts with several new countries.
  • The box with outgoing QSL cards is now empty!
  • I'm active as QSL manager for my local club, this is fun and my part of keeping the club running.
Plans for 2021:
  • Keep practising morse, try to pass the morse exam.
  • More satellite contacts. Weather permitting...
  • Morse and phone in contests.
  • Order new QSL cards and keep on sending them.
Read the rest of Closing 2020 in amateur radio with a plot of the number of contacts and a review

Tags: , , , ,
2020-11-28 Getting the DSL linespeed with both current and attainable speed from the Fritz!Box 7360/7590 9 months ago
I noticed I documented my original script to fetch the upstream and downstream DSL speed from the Fritz!Box 7360 but never documented the additional steps I took later to add the attainable upstream and downstream speeds to the current upstream and downstream speeds.

After switching to the Fritz!Box 7590 I missed my VDSL statistics so I dug up the scripts I had for the 7360 and tested whether they still work. And yes they do, so no changes there. The complete script:
#!/usr/bin/perl -w

use strict;

my ($fritzuser,$fritzpass);

$fritzuser="root";
$fritzpass="VerySecretPassword";

system("wget --user=$fritzuser --password=$fritzpass --post-file=wanifcfgrequest.xml --header=\"Content-Type: text/xml\" --header=\"SOAPAction: \\\"urn:dslforum-org:service:WANDSLInterfaceConfig:1#GetInfo\\\"\" --no-check-certificate http://fritz.koos.koffie.dot:49000/upnp/control/wandslifconfig1 -O wanifcfganswer.xml -o getfritz.log");

open(STATUSFILE,"){
        if (/(\d+)<\/NewUpstreamMaxRate>/){
                $upstream = $1;
        }
        if (/(\d+)<\/NewDownstreamMaxRate>/){
                $downstream = $1;
        }
        if (/(\d+)<\/NewUpstreamCurrRate>/){
                $upstreamcur = $1;
        }
        if (/(\d+)<\/NewDownstreamCurrRate>/){
                $downstreamcur = $1;
        }
}

if (defined $upstream and defined $downstream){
        my $line=sprintf("N:%d:%d:%d:%d",$downstream*1000,$upstream*1000,$downstreamcur*1000,$upstreamcur*1000);
        print $line."\n";
}
This does need the wanifcfgrequest.xml file:
<?xml version="1.0" encoding="utf-8"?>
<s:Envelope
   xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"
   s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
   <s:Body>
      <u:GetInfo xmlns:u="urn:dslforum-org:service:WANDSLInterfaceConfig1">
      </u:GetInfo>
   </s:Body>
</s:Envelope>
And I get a usable wanifcfganswer.xml:
<?xml version="1.0"?>
 <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<s:Body>
<u:GetInfoResponse xmlns:u="urn:dslforum-org:service:WANDSLInterfaceConfig:1">
<NewEnable>1</NewEnable>
<NewStatus>Initializing</NewStatus>
<NewDataPath>Fast</NewDataPath>
<NewUpstreamCurrRate>33032</NewUpstreamCurrRate>
<NewDownstreamCurrRate>111049</NewDownstreamCurrRate>
<NewUpstreamMaxRate>35725</NewUpstreamMaxRate>
<NewDownstreamMaxRate>117129</NewDownstreamMaxRate>
<NewUpstreamNoiseMargin>50</NewUpstreamNoiseMargin>
<NewDownstreamNoiseMargin>60</NewDownstreamNoiseMargin>
<NewUpstreamAttenuation>80</NewUpstreamAttenuation>
<NewDownstreamAttenuation>110</NewDownstreamAttenuation>
<NewATURVendor>41564d00</NewATURVendor>
<NewATURCountry>0400</NewATURCountry>
<NewUpstreamPower>498</NewUpstreamPower>
<NewDownstreamPower>513</NewDownstreamPower>
</u:GetInfoResponse>
</s:Body>
</s:Envelope>
This works without any change both on the Fritz!Box 7360 and the Fritz!Box 7590.

Tags: , ,
2020-09-02 An update to the home 1-wire network 1 year ago
For more than 12 years now(!) the house has temperature sensors using the 1-wire protocol. I recently redid some of the wiring between floors and I finally got around to rerouting the 1-wire network via this new route.

I also added a temperature sensor in the big room in the attic, we are thinking of using that room more often. To get an idea of how good that idea is we wanted to get an idea of the temperatures up there and that's what I have 1-wire sensors for! I soldered an 18b20 sensor to the end of a 4-wire flat phone cable, added it to the network and it's measuring.

So now 12 environmental temperatures are measured every 5 minutes: 9 in the house, one in the weather hut, one in the shed and one on the roof of the shed.

I also updated the 1-wire projects overview with how I use 4-wire flat phone cable in RJ45 connectors for 1-wire network. I had to look up how I did that previously before I could start adding new cables!

Tags: , , ,

IPv6 check

Running test...
, reachable as koos+website@idefix.net. PGP encrypted e-mail preferred. PGP key 5BA9 368B E6F3 34E4 local copy PGP key 5BA9 368B E6F3 34E4 via keyservers

RSS
Meningen zijn die van mezelf, wat ik schrijf is beschermd door auteursrecht. Sommige publicaties bevatten een expliciete vermelding dat ze ongevraagd gedeeld mogen worden.
My opinions are my own, what I write is protected by copyrights. Some publications contain an explicit license statement which allows sharing without asking permission.
Other webprojects: Camp Wireless, wireless Internet access at campsites, The Virtual Bookcase, book reviews
This page generated by $Id: newstag.cgi,v 1.34 2020/12/31 15:36:31 koos Exp $ in 0.020815 seconds.