CtrlAltUpgrade
← All writing

Ansible · · Timothy Alexander

One Ansible playbook to baseline every Linux server

For a long time, setting up a new Linux server in our district meant following a checklist from memory: add the admin accounts, lock down SSH, point time at our internal servers, trust our certificate authority, turn on security updates, install the monitoring agents. Each server came out a little different, and nobody could say for sure what "set up" meant.

Now it's one Ansible playbook run from Semaphore. It's safe to run again at any time, and running it again is how we check for drift.

What you need

  • Ansible, and ideally a UI to run it from (we use Semaphore, self-hosted in Docker)
  • An inventory of your servers, grouped however makes sense for you
  • Secrets (API tokens, enrollment keys) stored in your runner, never in the repo

One playbook made of small ones

The baseline itself has no tasks. It just imports a series of focused playbooks, each of which can also be run on its own:

---
# Universal Ubuntu server baseline. Safe to re-run; also used for drift checks.
- ansible.builtin.import_playbook: configure-certificates.yml
- ansible.builtin.import_playbook: configure-ssh.yml
- ansible.builtin.import_playbook: configure-time.yml
- ansible.builtin.import_playbook: configure-updates.yml
- ansible.builtin.import_playbook: provision-monitoring-agent.yml
- ansible.builtin.import_playbook: provision-security-agent.yml

Splitting it up this way means each piece can be fixed or re-run on its own. If a server only needs its certificates refreshed or its time settings corrected, that's one small job, not the whole baseline.

What each piece does

  • Certificates. Installs our internal root CA into the system trust store, then verifies against it before moving on, so internal HTTPS services work from the server.
  • SSH. Key-based access only. The new settings go in a drop-in file, the whole SSH configuration is validated with sshd -t, and only then is SSH reloaded. A typo can't lock you out.
  • Time. Sets the time zone, installs chrony, points it at our internal time servers, and then waits until the clock actually syncs instead of assuming it will.
  • Updates. Turns on unattended security upgrades, but never reboots on its own. Reboots happen when we choose.
  • Monitoring and security agents. Installs and enrolls the Zabbix and Wazuh agents (and our endpoint protection), using keys that come from Semaphore's secret store at run time.

Patterns worth stealing

Validate before you apply. Anything that could break access gets tested first. SSH config goes through sshd -t; Zabbix config goes through zabbix_agent2 -T. If validation fails, the play stops before anything changes.

Assert what you expect. The monitoring maintenance playbook checks that each server's agent config files exist and aren't empty before it touches anything, and fails with a message that says exactly which step to run first:

- name: Require existing configuration and enrollment
  ansible.builtin.assert:
    that:
      - item.stat.exists
      - (item.stat.size | default(0)) > 0
    fail_msg: "Missing or empty {{ item.item }}. Run the install/enrollment playbook first."
  loop: "{{ monitoring_files.results }}"

Pin and hold versions. Agents are installed at an approved version and held with dpkg_selections, so a routine apt upgrade can't move an agent ahead of its server.

Go one at a time when it matters. Playbooks that restart services use serial: 1. If something goes wrong, it goes wrong on one server.

Running it

In Semaphore, the baseline is a template with an inventory and a variable group holding the secrets. A brand-new server gets one earlier job that sets up administrator access; after that, it's "run the baseline" and wait for green. Because every step is safe to repeat, re-running it doubles as a drift check: anything that reports changed on a server that was already baselined is worth a look.

I've put a sanitized copy of the playbooks, with organization-specific values moved into an example variables file, on GitHub: Linux Server Baseline.

The biggest win isn't speed, though that's nice. It's that "a properly set up server" now has a definition, and the definition lives in git where anyone on the team can read it.