ICINGA2 Fine-Tuning

What is Icinga2?

Icinga2 is a great tool built upon the foundation of the well-known Nagios monitoring, inheriting
all the pros it has to offer. With many plugins available in your repository and thousands more in the community-driven Nagios-exchange website, icinga2 is a very good choice for your infrastructure monitoring.
I will cover some of the optional features that can be tweaked to suit your needs, once the main configuration has been set up.

Some of the points are:

• Taking full advantage of the variables defined in your Host file.
• Setting up custom scheduled downtimes.
• Changing the timeout for specific checks.
• Using a different user for your checks.
• Writing your own plugin.

TAKING FULL ADVANTAGE OF VARIABLES DEFINED IN YOUR HOST FILE

Host variables are a great way to utilize your commands and services to their full potential.
A single command definition can be reused for many different services and all the variables
prone to change can be put directly in your host file where you can easily edit them without
risking breaking the entire service.
Somewhere in your host file, you can have a variable defined like:

  vars.website_port = "33333"
                    vars.console_port = "44444"
                    vars.agent = "ssh"
                    

Here is an example of how an “by_ssh” “http” check makes use of such variables:

 object CheckCommand "ssh_http" {
                       import "by_ssh"
                        vars.by_ssh_command = PluginDir + "/check_http -H localhost $uri$ -p $port$ $secondary_port$ $extra_port$ $cust_string$"
                          vars.uri = ""
                          vars.port = ""
                          vars.website_port = ""
                          vars.console_port = ""
                          vars.cust_string = ""
                    }
                    

There is no need to write empty variable definitions here in this case, but I do to make it clearer.
Here I define the command template, intentionally leaving the “-u” in front of the $uri$
variable empty, because I do not wish to use it every time I call this command. The “cust_string” can be used
for any parameter you wish to add to the http check: For example ” -R something” will search for “something”
in the curl -s -i http://127.0.0.1:PORT/ output and return critical if it doesn’t find it. There are three
definitions for port but I use only one at a time. The reason is that I can have the ports defined
in the host file and reuse the command on many machines.
For example docker container ports may be different on every server:

    apply Service "website_http" {
                        import "generic-service"
                        check_command = "ssh_http"
                            vars.uri = "-u /some/address"
                            vars.port = ""
                            #vars.website_port = "" Defined in Host file
                            vars.console_port = ""
                            vars.cust_string = ""
                        assign where host.vars.agent == "ssh"
                    }

The weight of the variable definition is as follows:
Command > Host File > Service
The command can be overwritten by a variable defined in the host file
And the var in the host file can be overwritten by a var defined in the service.
That’s why in the service, the variable you wish to be taken from the host file is commented out,
while the other three options are defined as empty variables to overwrite anything else that may be
taken from the same host file by mistake. This enables you to call the same command multiple times
Making use of different variables defined in each host file. The other more common use for host.vars is grouping servers by variable. As you can see in the service above,
the service is going to run on all servers with host variable agent with value “ssh”… (assign where host.vars.agent == “ssh”)

SETTING UP CUSTOM SCHEDULED DOWNTIMES

Scheduled downtimes can be set in the downtimes.conf file located in /etc/icinga2/conf.d/ directory.
The predefined method “ScheduledDowntime” is used to create a service for a specific server or
a range of machines. For example:

 apply ScheduledDowntime "YourServiceName" to Service {
                      author = "icingaadmin"
                      comment = "Insert Your Comment Here"<br> <br>  ranges = {
                          monday = "00:00-01:45,14:00-15:00",
                          tuesday = "13:40-14:00,15:34-15:45",

                                       - - -  

                          sunday = "17:00-17:30,18:50-19:00,23:00-00:00"
                        }
                        assign where host.name == "yourHostName"
                    }
                    

The downtimes can also be assigned to a service group instead of a host. There can be as many per day as you like, separated with commas in quotation marks. You can also have as many downtime services as you need, as long as each one has a unique name.
Properly set up downtimes are a very nice thing to have, because otherwise, scheduled restarts of services will flood your monitoring notifications with unnecessary mail.

CHANGING THE TIMEOUT FOR SPECIFIC CHECKS

USING A DIFFERENT USER FOR YOUR CHECKS

WRITING YOUR OWN PLUGIN