Connecting Prometheus with PuppetDB

Puppet is one of the first infrastructure automation tools we used at Inuits. Our Puppet git repositories have accumulated ten years of commits.

In parallel, we are constantly bringing new tools to our infra, and over the last years, Prometheus usage has grown both internally and in our customers' infrastructure.

It was time for the two projects to meet each other - and therefore, I have contributed native PuppetDB service discovery support to the Prometheus server.

Puppet and Prometheus

Puppet and Prometheus are two popular Open Source projects, and they have users in common. It does not come as a surprise then that the community already bridged the gap between both tools.

prometheus-puppetdb-sd

Camptocamp’s prometheus-puppetdb-sd was, to date the go-to approach for Service Discovery between Puppet and Prometheus. That project was popular and integrated with VoxPupuli’s Prometheus Puppet module.

Prometheus service discoveries

One important part of Prometheus service discoveries is that we should implement it in a way that we do not have business logic inside the discovery.

In Puppet, it translates as, you should be able to use the service discovery with PuppetDB, not requiring any specific Puppet module to be installed. Therefore, I could not integrate prometheus-puppetdb-sd as-is. I had to start from scratch and make it very generic.

From scratch is a big word. I took some inspiration from Camptocamp’s approach, and I even contacted them before writing the first line of code. They provided me with excellent insights over PuppetDB APIs that I could use to start the service discovery with the correct endpoints.

The result is a very simplified architecture by connecting Prometheus directly with PuppetDB. Any resource will work, so there is no need for specific puppet modules either.

prometheus-puppetdb-sd

The PuppetDB Service Discovery is using the Puppet Query Language - PQL. Any resource or node that is in PuppetDB can be discovered by Prometheus

Example: monitoring Node Exporters

Relabeling is a crucial part of Prometheus configuration. It enables you to take the output from service discoveries and turn that into the desired target.

The following simple query will start monitoring all the servers that have the package node_exporter defined:

scrape_configs:
- job_name: node
  puppetdb_sd_configs:
  - url: http://puppetdb.local:8080
    query: 'resources { type = "Package" and title = "node_exporter" }'
    port: 9100
  relabel_configs:
  - source_labels: [__meta_puppetdb_certname]
    target_label: instance
  - source_labels: [__meta_puppetdb_environment]
    `target_label: environment

Example: Apache vhosts

Now, let’s do something different.

Instead of monitoring an exporter, let’s use the Blackbox Exporter to automatically verify the availability of all the deployed vhosts.

scrape_configs:
- job_name: apache_vhosts
  metrics_path: /probe
  params:
    module: [http_2xx]
  puppetdb_sd_configs:
  - url: http://puppetdb.local:8080
    query: 'resources { type = "Apache::Vhost" }'
    include_parameters: true
  relabel_configs:
    - source_labels: [__meta_puppetdb_parameter_servername]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: 127.0.0.1:9115

Note that this example is using include_parameters. The discovery would then try to include resource parameters as meta labels. You should only do this if your Prometheus web UI is protected enough or you do not have secrets in the results of your query.

Replacing prometheus-puppetdb-sd

If you use the prometheus-puppetdb-sd project and your Scrape::Job resources only have one target, you can easily switch to the PuppetDB native service discovery:

scrape_configs:
- job_name: 'puppetdb-scrape-jobs'
  puppetdb_sd_configs:
  # This example uses the Prometheus::Scrape_job exported resources.
  # https://github.com/camptocamp/prometheus-puppetdb-sd
  # This example is compatible with Prometheus-puppetdb-sd,
  # if the exported Prometheus::Scrape_job resources have at most one target.
  - url: https://puppetdb.example.com
    query: 'resources { type = "Prometheus::Scrape_job" and exported = true }'
    include_parameters: true
    tls_config:
      cert_file: prometheus-public.pem
      key_file: prometheus-private.pem
      ca_file: ca.pem
  relabel_configs:
  - source_labels: [__meta_puppetdb_certname]
    target_label: certname
  - source_labels: [__meta_puppetdb_parameter_targets]
    regex: '(.+),?.*'
    replacement: $1
    target_label: __address__
  - source_labels: [__meta_puppetdb_parameter_job_name]
    target_label: job
  - regex: '__meta_puppetdb_parameter_labels_(.+)'
    replacement: '$1'
    action: labelmap

Conclusion

Connecting Prometheus natively with PuppetDB enables you to discover any resource you have defined in your Puppet environment. One of the advantages is that you do not need to instrument your Puppet modules - it works directly with any resources.