In the previous post I talked about how this project got started. A customer had a question, and I have an affinity for building systems, exploring data, and wandering down rabbit-holes. My goal is to learn about data analytics platforms by re-creating a dashboard based on data from my cable modem. I am starting with InfluxDB for the data storage, and Grafana for visualization.
This is a brute-force iterative attempt to build a dashboard using InfluxDB and Grafana – two tools that I am not familiar with as I get started. I am going to be building a prototype to determine what is possible and the level of effort that may be required. I am doing this to learn things which I did not know before and produce a (hopefully) useful blog series. With the disclaimer and “I’m still learning” statements out of the way, onward we go!
Before this project, I have never used InfluxDB. As a time series database, it speaks to my weird little log processing soul. Other people like working on cars, building stuff with Legos, or whatever, I like messing around with logs. A time series database is like other databases in that it stores data. However, a time series database organizes that data based on time, then by other keys. A regular database organizes data based on keys, and then maybe then by time, if you tell it to.
As it turns out, InfluxDB is part of a technology stack called TICK (Telegraf, InfluxDB, Chronograf, Kapacitor). This is the part where I would usually consider tweaking the requirements to add another system to try. Finding new toys to play with is always nice, it’s like Christmas. Except you must build the toys yourself and then figure out how to use them. Resisting temptation this time, I am sticking to the original goal, InfluxDB and Grafana. I also make a note to myself to come back and try out the full stack, once I figure out what those other tools do. Temptation is hard to resist, but I have deadlines to meet, and my poor laptop is melting the stickers on the lid.
Installing InfluxDB is simple. I fire up a Linux VM, and follow the handy instructions from the documentation. After a few minutes I have a shiny new database system to work with. I really appreciate a good getting started guide when I am learning a new system, and InfluxDB has a good one. The getting started guide tells you what you need to know without a lot of fluff. Over a couple cups of coffee, I run through some simple exercises and gather enough knowledge to be dangerous to my VM. I am now an InfluxDB master, right? 
When you jump into a new system, even with good introductions and getting started information, you are likely to encounter the learning curve. Some products have a smooth steady ascension to mastery, like hiking the rolling foothills of a mountain range. After working through a couple of use cases, you will have learned everything that you need to know to get stuff done.
Other products have a learning cliff. If you have your climbing gear and oxygen handy, and some stubbornness (or a good professional service team… hint…hint) you can make it up the sheer face to prosperity. In my experience, you will not know what kind of learning curve you are going to encounter until you start implementing a use case.
InfluxDB is the gentle rolling hill kind of learning curve. InfluxDB has some of its own terminology such as measurements, tags, fields. I usually read documentation by scanning for pictures and code blocks. I skip all the filler text. It is this habit that causes me to bang my head into my keyboard on occasion. While the ah-hah discovery moments are more rewarding when I just figure it out, it is usually faster and less painful to just read the documentation. I have not learned this lesson myself, but I am aware of it.
In the case of InfluxDB the cranial-percussive-typing moment came when I tried to understand the format of the statement to add data to the database. As shown in the InfluxDB getting started guide, the insert statement is supposed to look like this:
<measurement>[,<tag-key>=<tag-value>…] <field-key>=<field-value>[,<field2-key>=<field2-value>…] [unix-nano-timestamp]
This looks a lot like I would have expected, except when I tried to build this with my own data I failed repeatedly. Since I only look at code blocks and pictures by default, I missed something important. The documentation looks like this:
I see only the purple bit when I first look at it, then try it for myself, and fail. The answer is right there, at the end of the line directly beneath the code block. This statement is white-space sensitive – spaces matter.
CPUTemp, LaptopName=GregsLaptop CPU1=1000000 1598286793
Is not the same thing as:
CPUTemp,LaptopName=GregsLaptop CPU1=1000000 1598286793
Ah, hah! I spent at least 30 minutes trying to smash my way through this, only to be reminded that reading is good, then solve my challenge in a couple short keystrokes.
I have now learned how to store data in InfluxDB, and that data looks like the data I want to send it eventually. I just need to decide on some measurement string values, decide what keys I need, and then put it all together. Not a bad result. I go brew another pot of coffee and get ready for the fun part.
Being able to insert data to a database on the server command line is a couple steps from success. It only shows that it is possible. A pot of gold, at the end of the rainbow. The method to send data from another application to a database usually requires a few extra steps. InfluxDB has an API, so I send curl requests to get some data into the database, that works well. Then I start trying to do the same type of simple requests with the Python Requests library. It works too, my head hasn’t hit the keyboard in like an hour, this is kind of awesome.
Instead of creating my own client, there is a maintained InfluxDB client on PyPi.org. In some cases, using 3rd party client saves a lot of time, and headache. In a production deployment, I will want to review this library in more detail and do more testing. A dependency can turn into a risk rather quickly if one is not careful. Since I am just prototyping, I pull the library down and update my code to use it. A few moments later, I can see my data is in the database. Neat!
With a little effort, and a bare-minimum amount of reading, I have a database with my modem data in it. I can query the data and see the results that I expect. My scripts are going to keep adding new data every 10-15 minutes in the background. It is time to move onto making charts and dashboards.
Making data pretty, and useful.
Grafana is a flexible open source monitoring and visualization tool. I think its strengths are simplicity in configuration, and the number of data platforms that it can talk to. With a simple point and click interface, I can connect to a data source and start creating charts and dashboards very quickly. I install Grafana on the same server with InfluxDB. After a couple of minutes, I have Grafana up and running.
Adding the data source for InfluxDB to Grafana is easy. Grafana and InfluxDB are on the same box, and because I am building a simple prototype on a VM on my laptop, there is zero security involved. I type in some values and test the configuration. A pretty green box tells me that I am a freakin’ wizard.
It is things like this that make me happy. These small moments are a big part of why I do what I do. Sure, this is not actually difficult. This green banner message not a sign of anything other than I can follow instructions occasionally, but my mom thinks it super-impressive. Little wins like this stack up, each success gives a little spark to push a little further and do something more interesting. For me, the problems, challenges, and troubleshooting have a similar effect. If I just stick with it, I will figure it out. Occasionally, I do have to go read some documentation, after every other option fails, but that is ok too. Someone is probably tracking visits and clicks on documentation sites, I might as well give them some data to play with too.
Creating a chart in Grafana is easy. As I said before, Grafana is all about point and click.
And a nice empty dashboard shows up. The hits just keep on coming, I literally haven’t screwed anything up in several minutes. Granted, I have not done much of anything in several minutes, but the point still stands.
The handy blue button there “Add new panel” brings me to another screen that lets me define a chart for my panel
With Grafana working, and data flowing into InfluxDB I can build the dashboard based on the example shown below. I choose the Median Receive SNR (Signal to Noise) for the first panel to build. It does not matter what order I build the panels, and this one involves some math.
After a few minutes, I have built my first panel. This is a simple “Stat” panel, with options to display the trendline for the previous hours, and I added the 2 decimal precision instead of the default rounded integers. Grafana did most of the heavy lifting populating the choice lists. All I had to do was choose the fields for the measurement “downstreamchannels” and change the aggregator from mean to median and time grouping to 1 hour instead of 1 second.
The rest of the dashboard comes together pretty much just as easily as this panel, except the channel counts panels. I thought these would be the easiest, but they took a bit of extra experimenting to get them running.
These two panels are just counts the distinct values for the index channel id number, over a period of time. The problem is that the value that needs to be counted, the index of the channel, is a tag, not a field in InfluxDB. Tags are meta data, fields are numeric values. The tags are indexed, but there doesn’t seem to be a way to query index values, they can only be used to query field values. To get around this, we could re-structure the data. It is an odd limitation if you are familiar with databases and other systems, but it is not a deal breaker, because I can get around it in a couple of different ways.
I figure out that I can add my own query for Grafana to use instead of using the point-and-click easy buttons. Thanks to the getting started guide, I know that InfluxDB provides a command to get the tag values. This command is not time-bound, but that is ok for my purposes. The command is SHOW TAG VALUES FROM <measurement> WITH key = “tagname”. I can put this into Grafana as a new query, and add a transformation to count the results.
Now that I know about this bump in the road I can design around it in the future. No project ever goes 100% to plan. In the larger context, this this is a win.
Finishing up the dashboard, took a couple more hours. I stumbled here and there trying to find different options to make a panel look the way I wanted, but it came together. I think it is nice.
Throughout the process, I was impressed how well InfluxDB and Grafana behaved. The documentation was solid, I only had to read a few parts. This speaks to the intuitive functionality of the products. Both InfluxDB and Grafana just work.
As I write this, there is more than a month of data in InfluxDB. The dashboard is still working correctly, and everything is still very quick. I still don’t really know what the data really means, and that is still just fine with me. The charts give an impression of what is going on, and because I have a history, if I ever need to know what these values mean, I will have a lot of historical data to review. I have not had any problems with the database or the VM it is sitting on, despite a few rounds of updates, and rebooting my laptop periodically.
A working prototype does not mean the project is ready for production deployment. I would still have a fair amount of research and testing to do if I were going to pitch this to a customer. With that said, a prototype that stays functional for a month without any care and feeding is a good start.
InfluxDB is a product that I will keep in mind for future metrics and time-series data projects. Grafana’s performance and simplicity reinforced why it is a go-to tool for dashboarding for many projects. I spent more time trying to get a hold of the data, and parse it into usable structures, than I did loading the data into the database and visualizing it.
That is one system down, three left to go in the series. In my next post I will be building this dashboard using the same data feed, but I will be using Elasticsearch for the storage, and Kibana for visualization.
***
This blog was written by Greg Porterfield, Senior Security Consultant at Set Solutions.