Ansible Playbook for Updating Linux (Debian/Ubuntu)

Video on using Ansible to Update Linux

The three steps to update a machine with Ansible

  1. Create Ansible Inventory/Hosts file
  2. Create Playbook
  3. Run Playbook

Create Inventory

The first thing we need to do is create an inventory file. This will contain a list of our servers along with the credentials.

touch hosts.txt

Now let’s encrypt the file with Ansible Vault.

ansible-vault encrypt hosts.txt

The file is now encrypted. To edit the file, we need to use `ansible-vault edit`.
If you want to, you can configure the hosts.txt file and then encrypt it when you are finished.

ansible-vault edit hosts.txt

Now add some hosts. In this example we add the local Kali machine, because why not. If you have Ubuntu servers, replace debian with ubuntu.

[debian]
kali ansible_host=127.0.0.1 ansible_ssh_user=kali ansible_ssh_port=22 ansible_ssh_password='kali pass' ansible_become_pass='kali sudo pass'

Add as many hosts as you need. For sake of simplicity, we are only adding one, and it is our localhost.

Create Playbook

Create a new playbook.

vi debian_update.yml

Put the following into the playbook. Edit as desired. Change hosts to match the above hosts in the inventory/hosts file.

---
- name: OS update
  hosts: debian
  gather_facts: yes
  become: yes

  tasks:
    - name: dist-upgrade
      ansible.builtin.apt:
        upgrade: dist
        update_cache: yes
      register: upgrade_result

    - name: Check if a reboot is required
      ansible.builtin.stat:
        path: /var/run/reboot-required
        get_checksum: no
      register: reboot_required_file

    - name: Reboot the server (if required).
      ansible.builtin.reboot:
      when: reboot_required_file.stat.exists
      register: reboot_result

    - name: Remove unneeded dependencies
      ansible.builtin.apt:
        autoremove: yes
      register: autoremove_result

    - name: Print errors if upgrade failed
      ansible.builtin.debug:
        msg: |
          Upgrade Result: {{ upgrade_result }}
          Reboot Result: {{ reboot_result }}
          Autoremove Result: {{ autoremove_result }}

A couple of notes

  1. On the 3rd line it defines which group to run this playbook against. In this case debian.
  2. This will check if a reboot is needed and reboot the machine. Reboots are usually needed when the kernel is updated
  3. The 5th line contains `become: yes` this means that the playbook will use sudo. You can specify the sudo password in the hosts file `ansible_become_pass=sudopass` or with the -k or –ask-become options
  4. The update and reboot are natively built into Ansible. Hence the ansible.builtin.

Run Playbook

Now that we have our inventory and playbook, we can upgrade our machines.

ansible-playbook debian_update.yml -i hosts.ini --ask-vault-password

Tip! If you have not specified a “ansible_ask_become” password (that is the sudo password), you can specify it with the -k or –ask-become options.

Run sudo Command over SSH. Single line.

When running an SSH command that uses sudo, something like

ssh admin@192.168.1.20 "sudo apt -y update && sudo apt -y upgrade"

You may receive the following error.

sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: a password is required

To work around this, you can use the -t option. -q is not needed, but makes thing quieter.

ssh -qt admin@192.168.1.20 "sudo apt -y update && sudo apt -y upgrade "

The sudo password will also be hidden.

https://unix.stackexchange.com/questions/134155/how-do-you-keep-the-password-hidden-when-invoked-during-the-su-command

https://stackoverflow.com/questions/233217/how-to-pass-the-password-to-su-sudo-ssh-without-overriding-the-tty

Install and Setup Tailscale on Ubuntu

Add the Tailscale package

curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/lunar.noarmor.gpg | sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null
curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/lunar.tailscale-keyring.list | sudo tee /etc/apt/sources.list.d/tailscale.list

Update and install Tailscale

sudo apt update && sudo apt upgrade
sudo apt install tailscale
sudo tailscale up

You’ll be given a link to visit to authenticate the device.

You can check the Tailscale IP address with

tailscale ip -4

https://tailscale.com/kb/1275/install-ubuntu-2304

Can’t log into NixOS after Install!

You thought everything went well with your NixOS install, you reboot, enter your username and password, and bam! Login incorrect.

Okay, try it again.

Login incorrect

Hmm…

Let’s try root. Nope, same thing…

If you don’t enter the password in correctly for root at the end of an installation, there will not be a root password, hence you can not log in.

Best way to keep this from happening is to make sure the password is set up before rebooting.

If you are one of those unfortunate souls who entered the wrong root password and missed the warning at the end of the installation

Try the following.

nixos-enter --root '/mnt'

Note: If you already rebooted, boot up on the minimum USB drive, mount the root partition, then run the nixos-enter command.

mount /dev/disk/by-label/nixos /mnt
nixos-enter --root '/mnt'

passwd to set the root password. You can also set your user password with

passwd username

Change username to your username.

Reboot and login!

How To Install NixOS Minimum from USB drive

You can make the USB drive by downloading the image off of nixos.org and then use Etcher, dd, or your favorite iso to USB drive utility.

The minimum version of NixOS does not come with a GUI installer.

https://nixos.org/manual/nixos/stable/#ch-installation

The manual contains all the info needed. For a minimum install, there are a couple of steps that you need to perform, before you can install.

  1. Format hard disk
  2. Create config file
  3. Install

Format Hard Disk

We’ll assume that /dev/sda is our target disk. This will overwrite the disk. Make sure you don’t need anything on it.

parted /dev/sda -- mklabel gpt
parted /dev/sda -- mkpart root ext4 512MB -8GB
parted /dev/sda -- mkpart swap linux-swap -8GB 100%
parted /dev/sda -- mkpart ESP fat32 1MB 512MB
parted /dev/sda -- set 3 esp on

Format the partitions

mkfs.ext4 -L nixos /dev/sda1
mkswap -L swap /dev/sda2
mkfs.fat -F 32 -n boot /dev/sda3
mount /dev/disk/by-label/nixos /mnt
mkdir -p /mnt/boot
mount /dev/disk/by-label/boot /mnt/boot
swapon /dev/sda2

Create a basic config file

nixos-generate-config --root /mnt

You can edit the config to make any changes you need. You may want to uncomment the user lines to setup a new user.

nano /mnt/etc/nixos/configuration.nix

Install NixOS

nixos-install

Last step is to setup the root and user password.

passwd

Reboot the machine once the password is set.

After you log in, set the user password.

passwd username

Extract part of a tar archive

You can use tar -tvf to show the contents of a tar file.

tar -tvf  filename.tgz

You can extract a portion of the archive with

tar -zxvf filename.tgz path/inside/file -C destination/path

For instance, if I have a tar backup of my /home directory, and I need to extract a file out of the tarred Downloads to my current Downloads directory, I can do

tar -zxvf home.tgz home/incredigeek/Downloads/slack.deb ~/Downloads

https://www.cyberciti.biz/faq/list-the-contents-of-a-tar-or-targz-file/

https://stackoverflow.com/questions/24057301/bash-extract-only-part-of-tar-gz-archive

Copy SSH Keys to Server with SFTP

These steps assume you already have a public SSH key, if not, create one

SSH-Copy-Id is an easier way to upload ssh keys, however, it does not work on all devices.

ssh to the remote server using your password.

If it is not already created, create the authorized_keys file under the .ssh folder

touch ~/.ssh/authorized_keys

chmod 600 ~/.ssh/authorized_keys

vi ~/.ssh/authorized_keys

Add your public key to the end of the authorized_keys file

Ensure that the correct owner and permissions are on the files.

The .ssh directory should be

chmod 700 .ssh

And the authorized_keys file should be 600

chmod 600 ~/.ssh/authorized_keys

Both should be owned by the user. Change username to your username.

sudo chown -R username:username .ssh/authorized_keys

Helpful links

https://blog.tinned-software.net/setup-sftp-only-account-using-openssh-and-ssh-key/

https://blog.tinned-software.net/ssh-passwordless-login-with-ssh-key/

How To Check if RHEL/AlmaLinux needs a reboot after an update

Typically you’ll need to reboot a server after an update if the Linux Kernel was updated. It is possible that services need to be restarted.

There is some good information here https://serverfault.com/questions/122178/how-can-i-check-from-the-command-line-if-a-reboot-is-required-on-rhel-or-centos

Using Yum Utilities needs-restarting

Install the needs-restarting utility

sudo dnf install -y yum-utils

Once installed, we can check if we need to reboot with

sudo needs-restarting -r

The -r option only reports if a reboot is required.

If we wanted to automatically check and reboot, we could do

sudo needs-restarting -r || sudo shutdown -r

Alternative way

We could alternatively just check the kernel version and if it is different, manually reboot the machine. Note that there could be a couple cases where the kernel didn’t update, but you still need a reboot, or services needed to be restarted View links below for more information.

LAST_KERNEL=$(rpm -q --last kernel | perl -pe 's/^kernel-(\S+).*/$1/' | head -1)
CURRENT_KERNEL=$(uname -r)

test $LAST_KERNEL = $CURRENT_KERNEL || shutdown -r

How to determine if Ubuntu Needs a Reboot after an update

Typically after a Linux Kernel update, you will want to reboot your machine to take advantage of the new kernel. But how do you know if you need to reboot?

Fortunately, there is a simple way to check.

cat /var/run/reboot-required

If it returns

*** System restart required ***

Then we should reboot the machine.

https://www.cyberciti.biz/faq/how-to-find-out-if-my-ubuntudebian-linux-server-needs-a-reboot/

Using Auditd to monitor changes to Linux

Install and enable auditd with

sudo dnf install auditd
sudo systemctl enable auditd
sudo systemctl start auditd

Add a file or directory to monitor with

auditctl -w /etc/passwd -k password

-w is watch path
-k is a filter key we can use later to search through logs

Now we can search with ausearch

ausearch -k password

Using Preconfigured Rules

There are already some preconfigured rules in /usr/share/audit/sample-rules/

We can copy those to /etc/auditd/rules.d/ and use them.

cd /usr/share/audit/sample-rules/
cp 10-base-config.rules 30-stig.rules 31-privileged.rules 99-finalize.rules /etc/audit/rules.d/
augenrules --load

Note on the 31-privileged.rules file. You’ll need to run the commands in the file which will create a new file. Then we can copy that to “/etc/auditd/rules.d/”

find /bin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' > priv.rules
#find /sbin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' >> priv.rules
#find /usr/bin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' >> priv.rules
#find /usr/sbin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' >> priv.rules
#filecap /bin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules
#filecap /sbin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules
#filecap /usr/bin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules
#filecap /usr/sbin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules

And Copy priv.rules to /etc/audit/rules.d/31-privileged.rules. Overwrite the file there if needed.

cp ./priv.rules /etc/audit/rules.d/31-privileged.rules

Load the rules.

augenrules --load

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/auditing-the-system_security-hardening