How to use Prometheus to monitor the memo key-value store

Prometheus is an open-source systems monitoring and alerting platform used by many organizations and companies such as Docker, maintained by an active developer and user community.

Prometheus gives knowledge to the humans. And to developers.

The basic idea is that Prometheus pulls quantitative measurements (aka "metrics") from processes and/or machines, and presents them to its user, typically via a web interface. Metrics can be queried with a rich language, exposed in a tabular form, or graphically. You can even use Grafana to build beautiful interactive dashboards.

This allows for instance to monitor the resources consumed by services in a Docker swarm, and even to trigger alarms when the cloud is too... cloudy.

memo shares knowledge between humans. And containers.

Infinit's "memo" key-value store now exposes metrics to Prometheus, for instance the number of accessible blocks, the number of peers, etc. This allows to monitor the global health of a cluster of memo nodes. For instance, the following screenshot shows how we monitor the CPU load of a four-machine cluster that we stress by reads and writes in Memo.

But it's even more helpful (and fun) to watch a memo cluster heal itself following churn and failures. Below, one can notice, through the Grafana GUI, that the under-replicated blocks get recreated on other nodes so as to maintain the redundancy factor at all times.

Improving memo

Prometheus is not only a tool for you to monitor your memo cluster, it's also for us a means to measure our performances, and to improve them. Be sure that each time a metrics is added, optimization will follow!

Running Prometheus on your side

Using Prometheus goes in two steps. First, run infinit with the option --prometheus to chose the port on which metrics are published. If you don't specify anything, the default port is 8080.

infinit volume run --mountpoint /Volumes/Shared --fetch --cache --name Shared --prometheus=8080  

Then tell Prometheus where to pull Infinit for metrics. This is done with a simple configuration file, e.g.:

$> cat infinit-prometheus.yml
scrape_configs:  
  - job_name: 'infinit'
    static_configs:
      - targets: ['localhost:8080']

See the Prometheus Configuration Documentation for more details.

Then run Prometheus

$> prometheus -config.file infinit-prometheus.yml

And open your navigator to http://localhost:9090, and explore the different metrics we expose, all starting with infinit_.


Get started with Infinit in less than 10 minutes! Test it!

Akim Demaille

Software engineer at Docker.

Paris

Subscribe to Infinit's Blog

Get the latest posts delivered right to your inbox.

or subscribe via RSS with Feedly!