ICINGA2 – API and PASSIVE CHECKS

What is Icinga2?

The Icinga2 configuration options are rich and provide you with a lot of ways to reuse what you already have. They serve well, but once your monitoring becomes big enough they just don’t cut it anymore. Reducing the load on the monitoring hosts and the servers you check becomes a high priority. One way to solve this is to use the API and create passive services where applicable.
At the time of writing this blog post, there isn’t enough information for people who are trying to learn what passive checks are and how to implement them. I’m writing this with the hope that I will make it easier for people who want to know how to get started.

Passive Checks

In their essence, these are nothing more than services that don’t trigger any commands. Instead, they rely on getting their states, results, and performance data injected from another source. The best way to do this right now is by using the Icinga API.
The most common use case is when you’re able to gather information from a large number of servers or services by running a single script. This script can either be run by Icinga as a normal check command which returns a combined state for all the services, or it can be triggered in any other way.
The plugin gathers data for each service it checks and then injects it directly into an existing service using the Icinga API.
A passive service would either use the “dummy” command and have “active_checks” disabled or have them enabled and rely on the dummy default text output which should only be served if the API injection is delayed and exceeds the check interval for the service.

apply Service "ServiceName" {
        max_check_attempts = 1
        retry_interval = 5m
        check_interval = 5m

        enable_active_checks = false

        check_command = "dummy"
        vars.dummy_text = "No Passive Check Result Received"
        vars.dummy_state = '3'

        assign where host.vars.passive_host
}

In order to avoid the frustration of manually creating each service before importing its state, you can use the API for spawning services as well. They can be assigned to any existing host, or if you desire the host(s) they are assigned to can also be created using the API.
I don’t recommend mixing API and manual configuration on the same host objects unless you have implemented a good logic for discovering and getting rid of obsolete(zombie) services that no longer exist and therefore don’t receive status updates. One way to do this is to use the API and filter and remove services with the ‘dummy’ default output, which should be present when the service no longer receives external injections.
If your host only contains API-generated services it can be easily removed together with everything attached to it and then rebuilt the next time your script is triggered.

ICINGA API

The Icinga API is a powerful tool that provides access to the entire Icinga configuration, objects, and states. Most of the examples found online use the Linux shell command curl. If shell scripts suit you, that is the simplest way of interacting with it, however, I’ll provide examples for both Python and curl that do the same thing. Python is more flexible and it also runs on Windows, which unfortunately is required sometimes.
Examples of other languages can be found here.

Creating a Host

curl

curl -k -s -u root:icinga -H 'Accept: application/json' -X PUT 
'https://localhost:5665/v1/objects/hosts/example.localdomain' 
-d '{ "templates": [ "generic-host" ], 
"attrs": { "address": "192.168.1.1", "check_command": "hostalive", "vars.os" : "Linux" } }' 
| python -m json.tool

Python

import requests, json

req_url = 'https://localhost:5665/v1/objects/hosts/example.localdomain
headers = { 'Accept': 'application/json', 'X-HTTP-Method-Override': 'PUT' }
data = { "templates": [ 'generic-host' ],
             "attrs": { 'address': '8.8.8.8' }
       }
resp = requests.get(req_url, headers=headers, auth=(username, password), data=json.dumps(data), verify=False)

if (resp.status_code == 200):
    print "Result: " + json.dumps(resp.json(), indent=4, sort_keys=True)
else:
    print resp.text

CREATING A SERVICE

curl

curl -k -s -u root:icinga -H 'Accept: application/json' -X PUT 
'https://localhost:5665/v1/objects/services/example.localdomain!passive-service' 
-d '{ "templates": [ "generic-service" ], 
"attrs": { "check_command": "dummy", "check_interval": '300' ,"retry_interval": '300' } }'

Python

import requests, json

req_url = 'https://localhost:5665/v1/objects/services/example.localdomain!passive-service
headers = { 'Accept': 'application/json',
            'X-HTTP-Method-Override': 'PUT' }
data = { "templates": [ 'generic-service' ],
             "attrs": { 'display_name': passive-service,
                        'check_command': 'dummy',
                        'enable_active_checks': '1',
                        'vars.dummt_text': 'No passive check result received',
                        'vars.dummy_state': '3',
                        'max_check_attempts': '1',
                        'retry_interval': '300',
                        'check_interval': '300',
                        'host_name': example.localdomain }
               }
resp = requests.get(req_url, headers=headers, auth=(username, password), data=json.dumps(data), verify=False)

if (resp.status_code == 200):
    print "Result: " + json.dumps(resp.json(), indent=4, sort_keys=True)
else:
    print resp.text               

INJECTING THE STATE OF A SERVICE

curl

curl -k -s -u root:icinga -H 'Accept: application/json' -X POST 
'https://localhost:5665/v1/actions/process-check-result?service=example.localdomain!passive-ping6' 
-d '{ "exit_status": 2, "plugin_output": "PING CRITICAL - Packet loss = 100%", 
"performance_data": [ "rta=5000.000000ms;3000.000000;5000.000000;0.000000", "pl=100%;80;100;0" ], 
"check_source": "example.localdomain" }' | python -m json.tool

Python

import requests, json

req_url = 'https://localhost:5665/v1/actions/process-check-result?service=example.localdomain!passive-ping6
headers = { 'Accept': 'application/json', 'X-HTTP-Method-Override': 'POST' }
data = { "exit_status": 0,
         "plugin_output": 'OK: the service is OK',
         "performance_data": [ val1=5%;80;90;0;100 ],
         "check_source": example.localdomain }

resp = requests.get(req_url, headers=headers, auth=(username, password), data=json.dumps(data), verify=False)

if (resp.status_code == 200):
    print "Result: " + json.dumps(resp.json(), indent=4, sort_keys=True)
else:
    print resp.text

Importing PerfData variables is easily done with an array. Otherwise, Python doesn’t parse the required quotes correctly which leads to Icinga not recognizing the Data. Here is an example of how that’s done:

perf_arr.append('Heap=' + str(heap_curr) + 'b;' + str(heap_warn) + ';' + str(heap_crit) + ';0;' + str(srv_heap_max[cnt]))
perf_arr.append('JVM_Threads=' + str(srv_act_thr[cnt]) + ';')
perf_arr.append('Physical_Memory=' + str(srv_phys_mem[cnt]) + 'b;')
perf_arr.append('JVM_Load=' + str(srv_jvm_load[cnt]) + ';')
perf_arr.append('ExitStatus=' + str(exit_status) + ';')
perf_data = perf_arr

data = { "exit_status": service_state,
...
"performance_data":  perf_data,
...
}

DELETING A HOST OR SERVICE AND EVERYTHING ATTACHED TO IT RECURSIVELY

curl

curl curl -k -s -u root:icinga -H 'Accept: application/json' -X DELETE 
'https://localhost:5665/v1/objects/hosts/example.localdomain?cascade=1' | python -m json.tool

Python

import requests, json

req_url = 'https://localhost:5665/v1/objects/hosts/example.localdomain?cascade=1
headers = { 'Accept': 'application/json', 'X-HTTP-Method-Override': 'DELETE' }

resp = requests.get(req_url, headers=headers, auth=(username, password), verify=False)
   
if (resp.status_code == 200):
    print "Result: " + json.dumps(resp.json(), indent=4, sort_keys=True)
else:
    print resp.text

GETTING THE CURRENT STATE OF A SERVICE

curl

/usr/bin/curl -k -s -u root:icinga 'https://localhost:5665/v1/objects/services?service=api_generated_host_1!13&attrs=name&attrs=last_check_result'|jq

Python

import requests, json

req_url = 'https://localhost:5665/v1/objects/services?service=api_generated_host_1!13&attrs=name&attrs=last_check_result'

resp = requests.get(req_url, auth=(username, password), verify=False)

if (resp.status_code == 200):
    print "Result: " + json.dumps(resp.json(), indent=4, sort_keys=True)
else:
    print resp.text

Automation and Autodiscovery

With the ability to create, destroy, and update objects automatically the Icinga API allows the implementation of flexible automation. A well-thought-out script can make services appear and disappear seamlessly as they are implemented or removed from your environment.
Autodiscovery is only limited by the complexity of the script you use.
Here I will show an example of listing all the services on a specific host, checking their current state and output, and if both the state and the output match with the ones set by the dummy command – Deleting them.
Normally the dummy command should only give its own output if the external injection is delayed longer than the check_interval of the service.

Python

req_url = 'https://localhost:5665/v1/objects/services?filter=match("' + args.host_name + '",host.name)'
resp = requests.get(req_url, auth=(args.username, args.password), verify=False)

json_str = json.dumps(resp.json())
j = json.loads(json_str)

if args.actionDebug == True:
    if (resp.status_code == 200):
        print "Dynamic result: 200"
    else:
        print "Dynamic result:" + resp.status_code

z_svc = []

for x in j.keys():
    if x == "results":
        for i in j[x]:
            if i['attrs']['last_check_result']['exit_status'] == 3.0:
                if i['attrs']['last_check_result']['output'] == "UNKNOWN: No passive check result received":
                    z_svc.append(i['attrs']['__name'])
            else:
                continue

cnt = 0
while (cnt < len(z_svc)):
    req_url = 'https://' + args.api_address + ':' + args.api_port + '/v1/objects/services/' + z_svc[cnt] + '?cascade=1'
    headers = { 'Accept': 'application/json',
                'X-HTTP-Method-Override': 'DELETE' }

    resp = requests.post(req_url, headers=headers, auth=(args.username, args.password), verify=False)

    if args.actionDebug == True:
        print "Deleting Zombie Service " + z_svc[cnt]
        if (resp.status_code == 200):
            print "Result: " + json.dumps(resp.json(), indent=2, sort_keys=True)
        else:
            print resp.text

    cnt = cnt + 1

Performance Data

Every good monitoring goes hand in hand with a well-organized Graphic view of the event history. To achieve this all your monitoring scripts should return properly structured performance data values. The usual monitoring script output contains a text message and a check status. For example(python):

print 'OK: This service is funcitoning as expected'
exit(0)

This output, however, contains no performance data. It can be used for basic checks, but if you ever need to dig into the history of what happened to the service in the past day/month/year it just isn’t very useful.
In Icinga2 performance data is delivered by the printed output message. After the status text there needs to be a pipe symbol | and everything after it is considered by Icinga as performance data.
Here is an example of the richest way of providing performance data to the check result.

print 'OK: This service is ok | disk_space_percent=25%;40;60;0;100 disk_space_mb=25GB;40;60;0;100'

Everything before the ‘=’ is the name of the value. For example ‘disk_space_percent’
The first field after the ‘=’ is meant to be the current state of the service always followed by ‘;’. That’s the least amount of information required to receive performance data that can be interpreted by the monitoring. Additionally, you can add the type of value to make sure the monitoring understands what the value represents. MB,GB,% etc.
The second and third sections are the predefined Warning and Critical states for the service. If the service ever exceeds them it would change its state accordingly. They are also separated by ‘;’
The fourth and fifth numbers can be useful in services that return a multitude of values as performance data. They determine the minimum and maximum amount possible for this performance number. The last value is unintuitively not followed by a semicolon. This allows Icinga to interpret the current state in a nice Pie chart embedded in the icingaweb2 interface.
For example, the free disk space check usually monitors all partitions on your machine. It returns a value for each of them, but if one exceeds the warning/critical threshold, The entire check changes its state. In order to find what the issue is as fast as possible the above-shown pie charts shuffle in a way that always the one with the worst state will be the first you see. Mouse over it and you will be able to see enough information to know what you’re dealing with.