Migrate CentOS 8 Stream to AlmaLinux 8

https://wiki.almalinux.org/documentation/migration-guide

Update CentOS 8 Stream

sudo dnf update -y

Download and run the almalinux-deploy script

curl -O https://raw.githubusercontent.com/AlmaLinux/almalinux-deploy/master/almalinux-deploy.sh
sudo bash almalinux-deploy.sh -d

You’ll need to run with the -d “downgrade” option if you are migrating from CentOS 8 Stream. https://github.com/AlmaLinux/almalinux-deploy/tree/master?tab=readme-ov-file#roadmap

You may need to remove packages if there are conflicts. On one instance, there were issues and I needed to remove grafana and llvm-compat-libs.

sudo yum remove grafana llvm-compat-libs

After those errors are fixed, rerun.

sudo bash almalinux-deploy.sh -d

Once the script finishes

sudo reboot

Once it comes back up, check the Linux version

cat /etc/*release

Example output

AlmaLinux release 8.9 (Midnight Oncilla)
AlmaLinux release 8.9 (Midnight Oncilla)
NAME="AlmaLinux"
VERSION="8.9 (Midnight Oncilla)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.9"
PLATFORM_ID="platform:el8"
PRETTY_NAME="AlmaLinux 8.9 (Midnight Oncilla)"

Install Node.js 18 on AlmaLinux 8

List available Node.js versions available.

dnf module list nodejs
AlmaLinux 8 - AppStream
Name           Stream           Profiles                                     Summary
nodejs         10 [d][x]        common [d], development, minimal, s2i        Javascript runtime
nodejs         12 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         14 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         16 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         18 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         20 [x]           common [d], development, minimal, s2i        Javascript runtime

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

As we can see above, nodejs 18 is disabled. Enable it with

sudo dnf module enable nodejs:18

Now we can install with

sudo dnf install nodejs

You may need to uninstall older versions.

Ansible Playbook for Linux SNMP

This playbook is for installing and configuring SNMP on Ubuntu or RedHat machines

Change the snmp_location and snmp_contact etc. variables. Or define them in the inventory file, or pass them in as –extra-vars. –extra-vars=”snmp_location=’location address’ snmpv3_user=incredigeek …etc”

Couple of notes

  • We check to see if a read only SNMPv3 user has been created. If so, we don’t create a new one.
  • The snmpd service is stopped and started each time this is run
  • You will still need to allow SNMP through the firewall. Ubuntu or Fedora
---
- name: Linux SNMP Config
  hosts: all
  gather_facts: yes
  become: yes

# Install SNMPv3 on RHEL or Debian/Ubuntu
# Disable SNMP v1 and v2 on RHEL
# Configure SNMPv3 user

  vars:

    # Change these!
    snmp_location: My SNMP location
    snmp_contact: My SNMP contact info
    snmpv3_pass: mypassword
    snmpv3_user: incredigeek

    # These are used to disable the default public community.
    cmnt: '#'
    cmnt_lines:
      - com2sec notConfigUser
      - group   notConfigGroup
      - view    systemview
      - access  notConfigGroup 


  tasks: 

    - name: Check if SNMPv3 user exists
      ansible.builtin.lineinfile:
        path: /etc/snmp/snmpd.conf
        regexp: '^rouser'
        state: absent
      check_mode: yes
      changed_when: false
      register: snmpv3_user_exists

    - name: Stop SNMPD Service
      ansible.builtin.service:
        name: snmpd
        state: stopped

    - name: RHEL SNMP Config
      block:
        - name: Install SNMP RHEL
          ansible.builtin.dnf:
            name:
              - net-snmp
              - net-snmp-utils
            state: present

        - name: Disable public snmp community RHEL
          replace:
            path: /etc/snmp/snmpd.conf
            regexp: '^{{ item }}'
            replace: '{{ cmnt }} {{ item }}'
          loop: "{{ cmnt_lines }}"
        - name: Set SNMP Location
          ansible.builtin.lineinfile:
            path: /etc/snmp/snmpd.conf
            regexp: '^syslocation.*'
            line: "syslocation {{ snmp_location }}"

        - name: Set SNMP Contact
          ansible.builtin.lineinfile:
            path: /etc/snmp/snmpd.conf
            regexp: '^syscontact.*'
            line: "syscontact {{ snmp_contact }}"
        - name: Setup SNMPv3 user for RHEL
          shell: net-snmp-create-v3-user -ro -a SHA -A '{{ snmpv3_pass }}' -x '{{ snmpv3_pass }}' -X AES {{ snmpv3_user }}
          when: not snmpv3_user_exists.found

      when: ansible_os_family == "RedHat"

    - name: Debian SNMP Config
      block:
        - name: Install SNMP on Debian
          ansible.builtin.apt:
            pkg:
            - snmp
            - snmpd
            - libsnmp-dev

        - name: Modify available from address
          ansible.builtin.lineinfile:
            path: /etc/snmp/snmpd.conf
            regexp: '^agentAddress udp:127\.0\.0\.1:161'
            line: 'agentAddress udp:161,udp6:[::1]:161'

        - name: Set SNMP Location
          ansible.builtin.lineinfile:
            path: /etc/snmp/snmpd.conf
            regexp: '^sysLocation.*'
            line: "sysLocation {{ snmp_location }}"

        - name: Set SNMP Contact
          ansible.builtin.lineinfile:
            path: /etc/snmp/snmpd.conf
            regexp: '^sysContact.*'
            line: "sysContact {{ snmp_contact }}"
        - name: Setup SNMPv3 user for Debian
          shell: net-snmp-config --create-snmpv3-user -ro -a SHA -A '{{ snmpv3_pass }}' -x '{{ snmpv3_pass }}' -X AES {{ snmpv3_user }}
          when: not snmpv3_user_exists.found

      when: ansible_os_family == "Debian"

    - name: Enable SNMPD Service
      ansible.builtin.service:
        name: snmpd
        enabled: true

    - name: Start SNMPD Service
      ansible.builtin.service:
        name: snmpd
        state: started

Ansible Playbook to Detect OS version

This playbook can be used to report the Linux Distribution, OS Family, Distribution Version, and Distribution Major Version. This can be helpful for verifying all operating systems are up to date, or for working out what to use in other playbooks.

You will need to already have an inventory file.

Playbook yaml file

The playbook is very simple. Copy and paste the following contents into a file named “os_info.yaml”

---
- hosts: all
  gather_facts: yes
  become: false
  tasks:
  - name: Distribution
    debug: msg=" distribution {{ ansible_distribution }} - os_family {{ ansible_os_family}} - distribution_version {{ansible_distribution_version}} - distribution_major_version {{ ansible_distribution_major_version }}"

If we wanted to, we could break out each Ansible variable in its own debug line. I prefer having them all on a single line.

Running the Playbook

Run the playbook like any other playbook. Change inventory.ini to your inventory file. If your inventory file is encrypted, use the –ask-vault-pass option.

ansible-playbook -i inventory.ini os_info.yaml 

Results

Here are some example results.

 ---------------------
< TASK [Distribution] >
 ---------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||


ok: [almalinux_server01] => {
    "msg": " distribution AlmaLinux - os_family RedHat - distribution_version 9.3 - distribution_major_version 9"
}
ok: [fedora_server01] => {
    "msg": " distribution Fedora - os_family RedHat - distribution_version 39 - distribution_major_version 39"
}
ok: [centos_server] => {
    "msg": " distribution CentOS - os_family RedHat - distribution_version 7.9 - distribution_major_version 7"
}
ok: [ubuntu_serevr01] => {
    "msg": " distribution Ubuntu - os_family Debian - distribution_version 20.04 - distribution_major_version 20"
}

Ansible Playbook to upgrade Linux Servers (Debian, Ubuntu, RedHat, Fedora, CentOS)

This is an Ansible playbook that can upgrade all your Linux machines! Or at least most of them. No openSUSE support yet.

Copy the playbook below, and put all your servers into an inventory file and run with

ansible-playbook -i hosts.ini master_update.yaml --ask-vault-pass

Couple of notes.

  1. This will do a full update automatically reboot your servers if needed.
  2. There is a special section for RHEL, CentOS 7 servers. If a server is running say CentOS 7, it will default to using YUM instead of DNF.
  3. You need sudo or become: yes to reboot and install upgrades.

Linux OS Upgrade Playbook

---
- name: Linux OS Upgrade
  hosts: all
  gather_facts: yes
  become: yes

  tasks:
    - name: Upgrade Debian and Ubuntu systems with apt
      block: 
        - name: dist-upgrade
          ansible.builtin.apt:
            upgrade: dist
            update_cache: yes 
          register: upgrade_result

        - name: Debain check if reboot is required
          shell: "[ -f /var/run/reboot-required ]"
          failed_when: False
          register: debian_reboot_required
          changed_when: debian_reboot_required.rc == 0
          notify:
            - Reboot server 

        - name: Debian remove unneeded dependencies
          ansible.builtin.apt:
            autoremove: yes
          register: autoremove_result 

        - name: Debian print errors if upgrade failed
          ansible.builtin.debug:
            msg: | 
              Upgrade Result: {{ upgrade_result }}
              Autoremove Result: {{ autoremove_result }}
      when: ansible_os_family == "Debian"
    
    - name: Upgrade RHEL systems with DNF
      block:
        - name: Get packages that can be upgraded with DNF
          ansible.builtin.dnf:
            list: upgrades
            state: latest
            update_cache: yes 
          register: reg_dnf_output_all

        - name: List packages that can be upgraded with DNF
          ansible.builtin.debug: 
            msg: "{{ reg_dnf_output_all.results | map(attribute='name') | list }}"

        - name: Upgrade packages with DNF
          become: yes
          ansible.builtin.dnf:
            name: '*'
            state: latest
            update_cache: yes
            update_only: no
          register: reg_upgrade_ok

        - name: Print DNF errors if upgrade failed
          ansible.builtin.debug:
            msg: "Packages upgrade failed"
          when: reg_upgrade_ok is not defined

        - name: Install dnf-utils
          become: yes
          ansible.builtin.dnf:
            name: 'dnf-utils'
            state: latest
            update_cache: yes
          when: reg_dnf_output_all is defined

      when: ansible_os_family == "RedHat" and not (ansible_distribution_major_version == "7")

    - name: Upgrade legacy RHEL systems with YUM
      block:
        - name: Get packages that can be upgraded with YUM
          ansible.builtin.yum:
            list: upgrades
            state: latest
            update_cache: yes 
          register: reg_yum_output_all
            

        - name: List packages that can be upgraded with YUM
          ansible.builtin.debug: 
            msg: "{{ reg_yum_output_all.results | map(attribute='name') | list }}"

        - name: Upgrade packages with YUM
          become: yes
          ansible.builtin.yum:
            name: '*'
            state: latest
            update_cache: yes
            update_only: no
          register: reg_yum_upgrade_ok

        - name: Print YUM errors if upgrade failed
          ansible.builtin.debug:
            msg: "Packages upgrade failed"
          when: reg_yum_upgrade_ok is not defined
            
        - name: Check legacy RHEL system if a reboot is required
          become: yes
          command: needs-restarting -r
          register: reg_reboot_required
          ignore_errors: yes
          failed_when: false
          changed_when: reg_reboot_required.rc != 0
          notify:
            - Reboot server 
      when: ansible_os_family == "RedHat" and ansible_distribution_major_version == "7"


  handlers:
    - name : Reboot server
      ansible.builtin.reboot:
        msg: "Reboot initiated by Ansible after OS update"
        reboot_timeout: 3600
        test_command: uptime

Helpful links

https://github.com/simeononsecurity/ansible_linux_update/tree/main
https://simeononsecurity.com/guides/automate-linux-patching-and-updates-with-ansible/
https://thenathan.net/2020/07/16/yum-and-dnf-update-and-reboot-with-ansible/

More space needed on the /boot filesystem. RHEL / Fedora / Alma / Rocky

Error Summary
-------------
Disk Requirements:
   At least 28MB more space needed on the /boot filesystem.

The above error is due to the /boot partition being out of space. We can fix this issue by removing older unused Linux kernels. You could also increase the disk space, but that is a little more involved.

First we need to list which kernels we have installed.

rpm -qa | grep kernel

Example output

[incredigeek@apache ~]$ rpm -qa | grep kernel
kernel-core-4.18.0-522.el8.x86_64
kernel-tools-4.18.0-529.el8.x86_64
kernel-modules-4.18.0-526.el8.x86_64
kernel-4.18.0-526.el8.x86_64
kernel-modules-4.18.0-529.el8.x86_64
kernel-4.18.0-522.el8.x86_64
kernel-4.18.0-529.el8.x86_64
kernel-core-4.18.0-529.el8.x86_64
kernel-devel-4.18.0-522.el8.x86_64
kernel-core-4.18.0-526.el8.x86_64
kernel-devel-4.18.0-529.el8.x86_64
kernel-tools-libs-4.18.0-529.el8.x86_64
kernel-devel-4.18.0-526.el8.x86_64
kernel-headers-4.18.0-529.el8.x86_64
kernel-modules-4.18.0-522.el8.x86_64

The kernel in bold is the one we will remove.

Next we remove erase the old kernel(s)/items.

sudo rpm -e kernel-4.18.0-522.el8.x86_64 kernel-core-4.18.0-522.el8.x86_64 kernel-devel-4.18.0-522.el8.x86_64 kernel-modules-4.18.0-522.el8.x86_64

And now we continue with our update

sudo dnf update

Helpful links.

https://www.cyberciti.biz/faq/installing-kernel-2-6-32-131-2-1-el6-x86_64-needs-8mb-on-boot-filesystem/

Ansible Playbook for Updating Linux (Debian/Ubuntu)

Video on using Ansible to Update Linux

The three steps to update a machine with Ansible

  1. Create Ansible Inventory/Hosts file
  2. Create Playbook
  3. Run Playbook

Create Inventory

The first thing we need to do is create an inventory file. This will contain a list of our servers along with the credentials.

touch hosts.txt

Now let’s encrypt the file with Ansible Vault.

ansible-vault encrypt hosts.txt

The file is now encrypted. To edit the file, we need to use `ansible-vault edit`.
If you want to, you can configure the hosts.txt file and then encrypt it when you are finished.

ansible-vault edit hosts.txt

Now add some hosts. In this example we add the local Kali machine, because why not. If you have Ubuntu servers, replace debian with ubuntu.

[debian]
kali ansible_host=127.0.0.1 ansible_ssh_user=kali ansible_ssh_port=22 ansible_ssh_password='kali pass' ansible_become_pass='kali sudo pass'

Add as many hosts as you need. For sake of simplicity, we are only adding one, and it is our localhost.

Create Playbook

Create a new playbook.

vi debian_update.yml

Put the following into the playbook. Edit as desired. Change hosts to match the above hosts in the inventory/hosts file.

---
- name: OS update
  hosts: debian
  gather_facts: yes
  become: yes

  tasks:
    - name: dist-upgrade
      ansible.builtin.apt:
        upgrade: dist
        update_cache: yes
      register: upgrade_result

    - name: Check if a reboot is required
      ansible.builtin.stat:
        path: /var/run/reboot-required
        get_checksum: no
      register: reboot_required_file

    - name: Reboot the server (if required).
      ansible.builtin.reboot:
      when: reboot_required_file.stat.exists
      register: reboot_result

    - name: Remove unneeded dependencies
      ansible.builtin.apt:
        autoremove: yes
      register: autoremove_result

    - name: Print errors if upgrade failed
      ansible.builtin.debug:
        msg: |
          Upgrade Result: {{ upgrade_result }}
          Reboot Result: {{ reboot_result }}
          Autoremove Result: {{ autoremove_result }}

A couple of notes

  1. On the 3rd line it defines which group to run this playbook against. In this case debian.
  2. This will check if a reboot is needed and reboot the machine. Reboots are usually needed when the kernel is updated
  3. The 5th line contains `become: yes` this means that the playbook will use sudo. You can specify the sudo password in the hosts file `ansible_become_pass=sudopass` or with the -k or –ask-become options
  4. The update and reboot are natively built into Ansible. Hence the ansible.builtin.

Run Playbook

Now that we have our inventory and playbook, we can upgrade our machines.

ansible-playbook debian_update.yml -i hosts.ini --ask-vault-password

Tip! If you have not specified a “ansible_ask_become” password (that is the sudo password), you can specify it with the -k or –ask-become options.

Run sudo Command over SSH. Single line.

When running an SSH command that uses sudo, something like

ssh admin@192.168.1.20 "sudo apt -y update && sudo apt -y upgrade"

You may receive the following error.

sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: a password is required

To work around this, you can use the -t option. -q is not needed, but makes thing quieter.

ssh -qt admin@192.168.1.20 "sudo apt -y update && sudo apt -y upgrade "

The sudo password will also be hidden.

https://unix.stackexchange.com/questions/134155/how-do-you-keep-the-password-hidden-when-invoked-during-the-su-command

https://stackoverflow.com/questions/233217/how-to-pass-the-password-to-su-sudo-ssh-without-overriding-the-tty

LibreNMS Error “.sock: rrd_fetch_r failed:”

The following error showed up after adding a new device to LibreNMS.

.sock:rrd_fetch_r failed: 

It was not displaying any graph data, but the device was up and connected.

Looks like the error is SELinux related. You can fix the error by resetting the security context with the following command.

sudo restorecon -RFv /opt/librenms

If that does not work, try running all the following

Running the following commands will fix the issue most of the time:

sudo chown -R librenms:librenms '/opt/librenms'

sudo setfacl -d -m g::rwx /opt/librenms/bootstrap/cache /opt/librenms/storage /opt/librenms/logs /opt/librenms/rrd

sudo chmod -R ug=rwX /opt/librenms/bootstrap/cache /opt/librenms/storage /opt/librenms/logs /opt/librenms/rrd

sudo semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/bootstrap/cache(/.*)?'

sudo semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/storage(/.*)?'

sudo semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/logs(/.*)?'

sudo semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/rrd(/.*)?'

sudo restorecon -RFv /opt/librenms

Install and Setup Tailscale on Ubuntu

Add the Tailscale package

curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/lunar.noarmor.gpg | sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null
curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/lunar.tailscale-keyring.list | sudo tee /etc/apt/sources.list.d/tailscale.list

Update and install Tailscale

sudo apt update && sudo apt upgrade
sudo apt install tailscale
sudo tailscale up

You’ll be given a link to visit to authenticate the device.

You can check the Tailscale IP address with

tailscale ip -4

https://tailscale.com/kb/1275/install-ubuntu-2304