Migrate CentOS 8 Stream to AlmaLinux 8

https://wiki.almalinux.org/documentation/migration-guide

Update CentOS 8 Stream

sudo dnf update -y

Download and run the almalinux-deploy script

curl -O https://raw.githubusercontent.com/AlmaLinux/almalinux-deploy/master/almalinux-deploy.sh
sudo bash almalinux-deploy.sh -d

You’ll need to run with the -d “downgrade” option if you are migrating from CentOS 8 Stream. https://github.com/AlmaLinux/almalinux-deploy/tree/master?tab=readme-ov-file#roadmap

You may need to remove packages if there are conflicts. On one instance, there were issues and I needed to remove grafana and llvm-compat-libs.

sudo yum remove grafana llvm-compat-libs

After those errors are fixed, rerun.

sudo bash almalinux-deploy.sh -d

Once the script finishes

sudo reboot

Once it comes back up, check the Linux version

cat /etc/*release

Example output

AlmaLinux release 8.9 (Midnight Oncilla)
AlmaLinux release 8.9 (Midnight Oncilla)
NAME="AlmaLinux"
VERSION="8.9 (Midnight Oncilla)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.9"
PLATFORM_ID="platform:el8"
PRETTY_NAME="AlmaLinux 8.9 (Midnight Oncilla)"

Install Node.js 18 on AlmaLinux 8

List available Node.js versions available.

dnf module list nodejs
AlmaLinux 8 - AppStream
Name           Stream           Profiles                                     Summary
nodejs         10 [d][x]        common [d], development, minimal, s2i        Javascript runtime
nodejs         12 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         14 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         16 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         18 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         20 [x]           common [d], development, minimal, s2i        Javascript runtime

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

As we can see above, nodejs 18 is disabled. Enable it with

sudo dnf module enable nodejs:18

Now we can install with

sudo dnf install nodejs

You may need to uninstall older versions.

Ansible Playbook to upgrade Linux Servers (Debian, Ubuntu, RedHat, Fedora, CentOS)

This is an Ansible playbook that can upgrade all your Linux machines! Or at least most of them. No openSUSE support yet.

Copy the playbook below, and put all your servers into an inventory file and run with

ansible-playbook -i hosts.ini master_update.yaml --ask-vault-pass

Couple of notes.

  1. This will do a full update automatically reboot your servers if needed.
  2. There is a special section for RHEL, CentOS 7 servers. If a server is running say CentOS 7, it will default to using YUM instead of DNF.
  3. You need sudo or become: yes to reboot and install upgrades.

Linux OS Upgrade Playbook

---
- name: Linux OS Upgrade
  hosts: all
  gather_facts: yes
  become: yes

  tasks:
    - name: Upgrade Debian and Ubuntu systems with apt
      block: 
        - name: dist-upgrade
          ansible.builtin.apt:
            upgrade: dist
            update_cache: yes 
          register: upgrade_result

        - name: Debain check if reboot is required
          shell: "[ -f /var/run/reboot-required ]"
          failed_when: False
          register: debian_reboot_required
          changed_when: debian_reboot_required.rc == 0
          notify:
            - Reboot server 

        - name: Debian remove unneeded dependencies
          ansible.builtin.apt:
            autoremove: yes
          register: autoremove_result 

        - name: Debian print errors if upgrade failed
          ansible.builtin.debug:
            msg: | 
              Upgrade Result: {{ upgrade_result }}
              Autoremove Result: {{ autoremove_result }}
      when: ansible_os_family == "Debian"
    
    - name: Upgrade RHEL systems with DNF
      block:
        - name: Get packages that can be upgraded with DNF
          ansible.builtin.dnf:
            list: upgrades
            state: latest
            update_cache: yes 
          register: reg_dnf_output_all

        - name: List packages that can be upgraded with DNF
          ansible.builtin.debug: 
            msg: "{{ reg_dnf_output_all.results | map(attribute='name') | list }}"

        - name: Upgrade packages with DNF
          become: yes
          ansible.builtin.dnf:
            name: '*'
            state: latest
            update_cache: yes
            update_only: no
          register: reg_upgrade_ok

        - name: Print DNF errors if upgrade failed
          ansible.builtin.debug:
            msg: "Packages upgrade failed"
          when: reg_upgrade_ok is not defined

        - name: Install dnf-utils
          become: yes
          ansible.builtin.dnf:
            name: 'dnf-utils'
            state: latest
            update_cache: yes
          when: reg_dnf_output_all is defined

      when: ansible_os_family == "RedHat" and not (ansible_distribution_major_version == "7")

    - name: Upgrade legacy RHEL systems with YUM
      block:
        - name: Get packages that can be upgraded with YUM
          ansible.builtin.yum:
            list: upgrades
            state: latest
            update_cache: yes 
          register: reg_yum_output_all
            

        - name: List packages that can be upgraded with YUM
          ansible.builtin.debug: 
            msg: "{{ reg_yum_output_all.results | map(attribute='name') | list }}"

        - name: Upgrade packages with YUM
          become: yes
          ansible.builtin.yum:
            name: '*'
            state: latest
            update_cache: yes
            update_only: no
          register: reg_yum_upgrade_ok

        - name: Print YUM errors if upgrade failed
          ansible.builtin.debug:
            msg: "Packages upgrade failed"
          when: reg_yum_upgrade_ok is not defined
            
        - name: Check legacy RHEL system if a reboot is required
          become: yes
          command: needs-restarting -r
          register: reg_reboot_required
          ignore_errors: yes
          failed_when: false
          changed_when: reg_reboot_required.rc != 0
          notify:
            - Reboot server 
      when: ansible_os_family == "RedHat" and ansible_distribution_major_version == "7"


  handlers:
    - name : Reboot server
      ansible.builtin.reboot:
        msg: "Reboot initiated by Ansible after OS update"
        reboot_timeout: 3600
        test_command: uptime

Helpful links

https://github.com/simeononsecurity/ansible_linux_update/tree/main
https://simeononsecurity.com/guides/automate-linux-patching-and-updates-with-ansible/
https://thenathan.net/2020/07/16/yum-and-dnf-update-and-reboot-with-ansible/

Using Auditd to monitor changes to Linux

Install and enable auditd with

sudo dnf install auditd
sudo systemctl enable auditd
sudo systemctl start auditd

Add a file or directory to monitor with

auditctl -w /etc/passwd -k password

-w is watch path
-k is a filter key we can use later to search through logs

Now we can search with ausearch

ausearch -k password

Using Preconfigured Rules

There are already some preconfigured rules in /usr/share/audit/sample-rules/

We can copy those to /etc/auditd/rules.d/ and use them.

cd /usr/share/audit/sample-rules/
cp 10-base-config.rules 30-stig.rules 31-privileged.rules 99-finalize.rules /etc/audit/rules.d/
augenrules --load

Note on the 31-privileged.rules file. You’ll need to run the commands in the file which will create a new file. Then we can copy that to “/etc/auditd/rules.d/”

find /bin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' > priv.rules
#find /sbin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' >> priv.rules
#find /usr/bin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' >> priv.rules
#find /usr/sbin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' >> priv.rules
#filecap /bin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules
#filecap /sbin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules
#filecap /usr/bin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules
#filecap /usr/sbin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules

And Copy priv.rules to /etc/audit/rules.d/31-privileged.rules. Overwrite the file there if needed.

cp ./priv.rules /etc/audit/rules.d/31-privileged.rules

Load the rules.

augenrules --load

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/auditing-the-system_security-hardening

How to Undelete Files on XFS Filesystem

There are a couple different options for undeleting files for XFS filesystems.

TestDisk

TestDisk is a great command line recovery tool. Unfortunately, it can be slightly more difficult on systems using XFS compared to EXT4 systems. TestDisk does not support undeleting a file in place on XFS.

You can still recover files using TestDisk, you just need to recover the whole drive and dig through the recovery results to find the files you want.

xfs_undelete

There is also another utility that can be helpful. xfs_undelete

https://github.com/ianka/xfs_undelete

It allows for a little more flexibility in recovering files. For instance, you can specify to recover the files from the past hour to recover.

Download prerequisites

sudo dnf install tcllib
wget https://raw.githubusercontent.com/ianka/xfs_undelete/master/xfs_undelete
chmod u+x ./xfs_undelete

./xfs_undelete

Example of running xfs_undelete

./xfs_undelete -t -1hour ./dev/sda2

You will need a different filesystem to save the files to. Otherwise you will receive the following error.

Your output directory is  /home/bob/recovery/
That is within the filesystem  /  you want to recover files
from. This isn't feasible as it would overwrite the deleted files you wanted to
recover. Please specify the option -o /path/to/output_directory on another (rw
mounted) filesystem or run xfs_undelete from within a directory on that
filesystem so the recovered files could be written there. They cannot be
recovered in place.

It’s not the greatest idea to recover on the system while running. Ideally, shut the system down, plug the drive into another machine as read only, and copy the files off.

You could also boot up in single user mode or a live Linux iso/thumbdrive and mount another recovery drive. Should work for both physical and virtual environments.

Oxidized Error “OpenSSL::PKey::PKeyError with msg “dh#set_pqg= is incompatible with OpenSSL 3.0”

Looks like the issue has something to do with net-ssh. There are some other similar errors people were having.

https://github.com/ytti/oxidized/issues/2642
https://github.com/ytti/oxidized/pull/2570

The easy way to resolve the issue is to install oxidized using git.

Prerequisites

Make sure rake is installed

sudo dnf install rake
 or 
sudo apt install rake

Install Oxidized from Git

Steps were copied from here. https://github.com/ytti/oxidized#build-from-git

You should be able to copy and paste these commands in the users home directory.

git clone https://github.com/ytti/oxidized.git
cd oxidized/
gem install bundler
rake install

After it is installed, restart the service

systemctl restart oxidized

Or continue on installing and with LibreNMS

CentOS – This system is not registered with an entitlement server. You can use subscription-manager to register.

If you are getting the following response when trying to use the yum or dnf command,

This system is not registered with an entitlement server. You can use subscription-manager to register.

Try editing the subscription-manager.conf file, and disable it by changing enable=1 to enable=0

sudo nano /etc/yum/pluginconf.d/subscription-manager.conf

After you may run

yum clean

That should take care of the problem.

https://serverfault.com/questions/764900/how-to-remove-this-warning-this-system-is-not-registered-to-red-hat-subscriptio

https://sahlitech.com/entitlement-server-fix/

Hardening SNMP on CentOS/RedHat/Fedora Etc.

These steps should be similar across Red Hat type distros.

Before we proceed, lets stop SNMP

sudo systemctl stop snmpd

Disable SNMP Versions 1 and 2c

First we are going to disable SNMP v1 and v2c

You can manually edit the /etc/snmp/snmpd.conf file and comment out or delete every line starting with com2sec, group, access. Or you can run the following sed commands to change it for you.

sudo sed -i 's/^com2sec/# com2sec/g' /etc/snmp/snmpd.conf
sudo sed -i 's/^group/# group/g' /etc/snmp/snmpd.conf
sudo sed -i 's/^access/# access/g' /etc/snmp/snmpd.conf

https://serverfault.com/questions/376688/how-to-disable-version-1-and-version-2c-in-snmpd

Create SNMP Version 3 User

Follow the prompts to create a SNMP v3 user.

sudo net-snmp-create-v3-user -ro -a SHA -x AES

Start SNMP

sudo systemctl start snmpd

You should be good to go.

If you are running a firewall, you will need to allow an exception for SNMP, UDP port 161. You may also need to allow an SELinux exception. Check out the last portion of both these articles.

Allowing SNMP Through Firewall

Enable Syslog for PowerDNS Recursor

  1. Enable Logging in PowerDNS Recursor Config
  2. Edit Systemd Unit File for PowerDNS to Allow Syslog
  3. Enable Logging in rsyslog Config File

The following links were helpful in setting things up.

https://doc.powerdns.com/recursor/running.html
https://www.reddit.com/r/linuxadmin/comments/9lc4jl/logging_queries_in_pdnsrecursor/

Enable logging in PowerDNS Recursor Config

First we need to find the line that says “disable-syslog” and uncomment/change it to

disable-syslog=no

Next find the line that says “quiet” and uncomment/change it to

quiet=no

Some other lines you may want to check and change

logging-facality=1
loglevel=6

Edit Systemd Unit File for PowerDNS to allow Syslog

Next we need to modify the Systemd unit file to allow PowerDNS Recursor to log to syslog.

systemctl edit --full pdns-recursor.service

On the ExecStart Line, remove the part that says

--disable-syslog

The resulting line should look something like

[Service]
ExecStart=/usr/sbin/pdns_recursor --socket-dir=%t/pdns-recursor --socket-dir=%t/pdns-recursor --daemon=no --write-pid=no --log-timestamp=no

Save the file.

Enable Logging in rsyslog Config File

Edit the rsyslog file

sudo vim /etc/rsyslog.conf

Add the following line

local1.*        /var/log/pdns_recursor.log

This should now log all of the PowerDNS Recursor log info to “/var/log/pdns_recursor.log”

Restart the rsyslog and PowerDNS Recursor service

sudo systemctl restart rsyslog
sudo systemctl restart pdns-recursor

You should now see DNS request in the log file.

tail /var/log/pdns_recursor.log

They should also show up in the “/var/log/messages”