Blog

Icinga 2 - monitor AWS EC2 instances dynamically

Icinga 2 - monitor AWS EC2 instances dynamically

Now days it is a challenge to monitor instances which are created/deleted on the fly (instances in auto scaling group for example), so how we could accomplish this with Icinga 2? Well, in this blog we will provide some hints, how to have such configuration without writing complicated AWS CLI/API calls or lambda functions, but configuring mainly icinga.

The first and most important step is to have Icinga installed and configured. Additionally, Icinga Director, which will be the main responsible for taking care about checking if there are newly created/deleted EC2 instances and if so to add/remove them from Icinga accordingly. Last, but not least, AWS module for Icinga Web 2 (it requires AWS PHP SDK). Let's get started.

In Director create a host template, which will be applied when there are new EC2 instances to be added for monitoring. It will be inherited by all your virtual AWS instances. You could also create service templates, and add them to the host template.

zones.d/director-global/host_templates.conf

template Host "example" {
    check_command = "custom_health"
    notes = "This is EC2 instance"
}




With AWS module, you would define first an Import Source, specifying AWS region, AWS access key and Object type in our case we picked up 'EC2 Instances'.


The most powerful tool are the modifiers, from here you can filter/narrow your instances by certain criteria. Because we are interested in our production instances, we set a filter by auto scaling group in which are all EC2 instances related to that environment.


From the preview, you can get an overview of the instances that you are matching with this rule.

Next Step is to define Sync Rule, which is responsible for creating real Icinga objects based on data imported through one or more Import Source. Sync Properties allow you to specify how to treat the various properties in a granular way.



After you have set up all of above, you need a way to automate it, here comes the Director jobs, you need one job for each import source -> sync rule -> deploy changes job.




In order to run all those jobs, you could create a OS service, managed by SysV which executes 'icingacli director jobs run forever', and in order to have all detected changes deployed add another script as cronjob running every few hours doing 'director config deploy'.

cat /usr/lib/systemd/system/director-jobs.service
[Unit]
Description=Director Job runner

[Service]
Type=simple
ExecStart=/usr/bin/icingacli director jobs run --forever --verbose
Restart=on-success

[Install]
WantedBy=multi-user.target


Something which needs to be noted is that you need some scripts which are doing the real checks, in normal set ups, you would have them on the monitored hosts uploaded somehow, but in this case who/what will do that? One approach would be to tweak a bit the normal bash nagios scripts so they are executed trough ssh.

And voala, you have a icinga monitoring AWS hosts, which might be changed because of autoscaling policy.

What do you think? Leave a comment below! ;)