Blog

SaltStack - Configuration Management and Remote Execution

SaltStack - Configuration Management and Remote Execution


What is the purpose of a tool like SaltStack and a better question, what problem does it solve ?

The two main purposes of SaltStack are configuration management and remote execution. You have probably heard or used one of the more popular alternatives of SaltStack - Ansible, Puppet, or Chef. All of them pretty much accomplish the same goal. I like Salt in particular because it is written in Python and it is relatively lightweight. It uses ZeroMQ's communication layer which makes it really fast and it also uses the PyYAML for its configuration management recipes (called states).

In a nutshell if you manage any number of servers and you need to do something on them, you would have to log in to each one at a time and do your task. Even if that task is a small one like restarting an instance or checking their uptime, or a larger one like doing an installation and configuration of something, you would still have to do it one server at a time. If you manage a lot of servers you will need to do a lot of manual work to accomplish your tasks. This is where Salt Stack can be applied to automate your work and provide the ability to remotely execute commands on any number of machines.

Salt works using either Master/Minion setup where you have a master node from which you execute commands to the minion nodes, or using salt-ssh which is pretty much what it sounds like, it allows you to execute anything that you normally would to a configured minion on any machine over ssh, no matter if Salt is installed there.

 

Remote Execution

Communication between the Master node and its minions is done using public key encryption. First when configuring minions you need to tell your minion where the master node is located and that is accomplished via the /etc/salt/minion conf file. There is a line where you can specify the Master's location with the format - master : 127.0.0.1.

After a new minion is added we need to accept that minion's encryption keys on the master node. 

To see what keys are available and pending acceptance we can use :

 # salt-key -L
 Accepted Keys:
 Denied Keys:
 Unaccepted Keys:
      new_minion
      test_minion
      test_minion2
 Rejected Keys:

Then accept the ones we want with :

 # salt-key -a <minion name>

Now we can test if communication from master with minions works, using this command :
 
# salt '*' test.ping 
Webserver1: True
Webserver2: True
Webserver3: True

In the same way we can execute any command we want on any minion :

 # salt '*' cmd.run "service httpd restart"

And for that matter it is exactly the same using salt-ssh :

 # salt-ssh '*' cmd.run "uptime"
 Webserver1:       05:19am up 18 days 21:05, 1 user, load average: 0.03, 0.01, 0.00  Webserver2:       05:19am up 30 days 17:05, 1 user, load average: 0.05, 0.01, 0.00  Webserver3:       05:19am up 30 days 17:05, 1 user, load average: 0.04, 0.01, 0.00

Installating a package on all destination servers:

# salt '*' pkg.install vim

Every single destination server that you want to execute commands on needs to be added to the Salt Roster file located in the master's /etc/salt/roster. It will look something like this :

 # cat /etc/salt/roster
 Webserver1:
    host: webserver1.hostname.com
    priv: /root/.ssh/id_rsa
 Webserver2:
    host: webserver1.hostname.com
    priv: /root/.ssh/id_rsa
 Webserver3:
    host: webserver1.hostname.com
    priv: /root/.ssh/id_rsa

I personally prefer using salt-ssh for all of my configuration management and remote execution since I do not have to maintain Salt minion installations on all servers. I just maintain a master node and a roster file with all destination servers I want to manage. Then create a central configuration management repository and do remote execution on any server I like from the master.

Here is a visualization of the remote execution functionality of Salt:


Configuration Management

Salt uses template configuration files ending in .sls called states usually written in YAML format but many other formats are available as well.

Here is a very simple example of a Salt State that just installs a package on any OS (Salt determines the package manager needed to do the installation)

 mariadb-client:
   pkg:
     - installed

And a basic installation of apache looks like this :

 apache:
   pkg.installed: []
   service.running:
     - watch:
       - pkg: apache
       - file: /etc/httpd/conf/httpd.conf
       - user: apache
   user.present:
     - uid: 87
     - gid: 87
     - home: /var/www/html
     - shell: /bin/nologin
     - require:
       - group: apache
   group.present:
     - gid: 87
     - require:
       - pkg: apache
 /etc/httpd/conf/httpd.conf:
   file.managed:
     - source: salt://apache/httpd.conf
     - user: root
     - group: root
     - mode: 644

At the very bottom of this article I will put a more complicated example (since it is relatively long) where we automate the complete installation of Apache and make certain changes to its configuration, such as copying a virtual host template, making some environment changes, changing its port, starting the webserver and running some smoke tests on it to make sure the configured virtual host works.

Applying a Salt state to any server is as simple as :

# salt-ssh Webserver1 state.apply apache2/apache2

For debugging, you can also do a test run to see what it will do before actually doing anything, using :

# salt-ssh Webserver1 state.apply apache2/apache2 test=True -l debug

If you need to apply all of your states to a new minion you can just execute the hightstate command on it, which will apply your collection of .sls files (called a state tree) on the new minion.

# salt 'Webserver3' state.highstate

And here is the longer Apache installation and configuration example I mentioned earlier :

# Set variables for your environment
{% set envString = 'prod' %}
# Vhost configuration
{% set vhostFileName = 'itgix_vhost.conf' %}
{% set vhostServerName = 'itgix.com' %}
# Paths
{% set dataPath = '/opt/data/itgix/HTTPServer' %}
{% set logPath = '/opt/logs/itgix/HTTPServer' %}
{% set apacheInstallDir = '/opt/app/HTTPServer/prod/conf' %}
# Get server IP of eth0 interface - used in Smoke Tests only
{% set serverIPAddress = salt['network.interfaces']()['eth0']['inet'][0]['address'] %}
# Get hostname
{% set serverHostname = salt['grains.get']('host') %}
# Set ENVIRONMENT variable
setEnvVariable:
   environ.setenv:
     - name: ENVIRONMENT
     - value: {{ envString }}
     - update_minion: True
# Install apache
#apache2:
#  pkg:
#    - name: apache2
#    - installed
# Add Group equivalent to - groupadd -g 5000 itgix0000
webserverGroup:
  group.present:
    - name: itgix0000
    - gid: 5000
    - addusers:
      - itgix0000
# Add User equivalent to - itgix0000:x:5000:5000::/home/ihs00000:/bin/false
webserverUser:
  user.present:
    - name: itgix0000
    - shell: /bin/false
    - home: /home/itgix0000
    - uid: 5000
    - gid: 5000
    - groups:
      - itgix0000
# Create env dir
envDir:
  file.directory:
    - name: {{ dataPath }}/{{ envString }}/htdocs/infra
    - user: root
    - group: root
    - dir_mode: 755
    - file_mode: 644
    - makedirs: True
    - recurse:
      - user
      - group
      - mode
# Create logs dir
logPath:
  file.directory:
    - name: {{ logPath }}/{{ envString }}
    - user: ihs00000
    - group: ihs00000
    - dir_mode: 755
    - file_mode: 644
    - makedirs: True
    - recurse:
      - user
      - group
      - mode
# Create or replace uid.conf with itgix0000 user and group name
uid.conf:
  file.managed:
    - name: {{ apacheInstallDir }}/uid.conf
    - user: root
    - group: root
    - replace: True
    - contents:
      - User itgix0000
      - Group itgix0000
# Create or replace lbtestfile.html with: environment - hostname
lbtestfile.html:
  file.managed:
    - name: {{ dataPath }}/{{ envString }}/htdocs/infra/lbtestfile.html
    - user: root
    - group: root
    - replace: True
    - contents: |
        <html>
          <head>
              <title> {{ envString }} - {{ serverHostname }} </title>
          </head>
          <body>
              <pre>
                  {{ envString }} - {{ serverHostname }}
                  Status: OK
              </pre>
          </body>
        </html>
# Create or replace WhereAmI.Info with env - hostname
WhereAmI.info:
  file.managed:
    - name: {{ dataPath }}/{{ envString }}/htdocs/infra/WhereAmI.info
    - user: root
    - group: root
    - replace: True
    - contents:
      - {{ envString }} - {{ serverHostname }}
# Create or replace listen.conf - Listen port
listen.conf:
  file.managed:
    - name: {{ apacheInstallDir }}/listen.conf
    - user: root
    - group: root
    - replace: True
    - contents:
      - Listen {{ serverIPAddress }}:40200
# Copy Virtual host - itgix_vhost.conf (force: True is used to overwrite file if exists)
meapp.conf:
  file.managed:
    - name: {{ apacheInstallDir }}/vhosts.d/{{ vhostFileName }}
    - source: salt://apache2_4/files/{{ vhostFileName }}
    - force: True
    - makedirs: True
    - user: root
    - group: root
    - mode: 644
# Copy Virtual host - 00_default (force: True is used to overwrite file if exists)
00_default.conf:
  file.managed:
    - name: {{ apacheInstallDir }}/vhosts.d/00_default.conf
    - source: salt://apache2_4/files/00_default.conf
    - force: True
    - user: root
    - group: root
    - mode: 644
# Find and replace all occurances of <env> with the current environment
replaceEnvInVhost:
  file.replace:
    - name: {{ apacheInstallDir }}/vhosts.d/{{ vhostFileName }}
    - backup: original
    - pattern: <env>
    - repl: {{ envString }}
# Find and replace all occurances of <env> with the current environment
replaceEnvInDefaultVhost:
  file.replace:
    - name: {{ apacheInstallDir }}/vhosts.d/00_default.conf
    - backup: original
    - pattern: <env>
    - repl: {{ envString }}
# Add ServerName to vhost
replaceSrvNameInVhost:
  file.replace:
    - name: {{ apacheInstallDir }}/vhosts.d/{{ vhostFileName }}
    - backup: original
    - pattern: <server_name>
    - repl: {{ vhostServerName }}
# Enable apache2 service to be started at runtime and start apache2 now.
#enableAndStartApache:
#  service.running:
#    - name: apache2
#    - enable: True
# Get IP of eth0 and execute smoke tests towards it
smokeTestWithIP:
  cmd.run:
    - name: curl -v http://{{ serverIPAddress }}:40200/infra/lbtestfile.html
smokeTestWithHostAndIP:
  cmd.run:
    - name: curl -v -H "host:itgix.com" http://{{ serverIPAddress }}:40200/infra/lbtestfile.html