More space needed on the /boot filesystem. RHEL / Fedora / Alma / Rocky

Error Summary
-------------
Disk Requirements:
   At least 28MB more space needed on the /boot filesystem.

The above error is due to the /boot partition being out of space. We can fix this issue by removing older unused Linux kernels. You could also increase the disk space, but that is a little more involved.

First we need to list which kernels we have installed.

rpm -qa | grep kernel

Example output

[incredigeek@apache ~]$ rpm -qa | grep kernel
kernel-core-4.18.0-522.el8.x86_64
kernel-tools-4.18.0-529.el8.x86_64
kernel-modules-4.18.0-526.el8.x86_64
kernel-4.18.0-526.el8.x86_64
kernel-modules-4.18.0-529.el8.x86_64
kernel-4.18.0-522.el8.x86_64
kernel-4.18.0-529.el8.x86_64
kernel-core-4.18.0-529.el8.x86_64
kernel-devel-4.18.0-522.el8.x86_64
kernel-core-4.18.0-526.el8.x86_64
kernel-devel-4.18.0-529.el8.x86_64
kernel-tools-libs-4.18.0-529.el8.x86_64
kernel-devel-4.18.0-526.el8.x86_64
kernel-headers-4.18.0-529.el8.x86_64
kernel-modules-4.18.0-522.el8.x86_64

The kernel in bold is the one we will remove.

Next we remove erase the old kernel(s)/items.

sudo rpm -e kernel-4.18.0-522.el8.x86_64 kernel-core-4.18.0-522.el8.x86_64 kernel-devel-4.18.0-522.el8.x86_64 kernel-modules-4.18.0-522.el8.x86_64

And now we continue with our update

sudo dnf update

Helpful links.

https://www.cyberciti.biz/faq/installing-kernel-2-6-32-131-2-1-el6-x86_64-needs-8mb-on-boot-filesystem/

How To Check if RHEL/AlmaLinux needs a reboot after an update

Typically you’ll need to reboot a server after an update if the Linux Kernel was updated. It is possible that services need to be restarted.

There is some good information here https://serverfault.com/questions/122178/how-can-i-check-from-the-command-line-if-a-reboot-is-required-on-rhel-or-centos

Using Yum Utilities needs-restarting

Install the needs-restarting utility

sudo dnf install -y yum-utils

Once installed, we can check if we need to reboot with

sudo needs-restarting -r

The -r option only reports if a reboot is required.

If we wanted to automatically check and reboot, we could do

sudo needs-restarting -r || sudo shutdown -r

Alternative way

We could alternatively just check the kernel version and if it is different, manually reboot the machine. Note that there could be a couple cases where the kernel didn’t update, but you still need a reboot, or services needed to be restarted View links below for more information.

LAST_KERNEL=$(rpm -q --last kernel | perl -pe 's/^kernel-(\S+).*/$1/' | head -1)
CURRENT_KERNEL=$(uname -r)

test $LAST_KERNEL = $CURRENT_KERNEL || shutdown -r

Using Auditd to monitor changes to Linux

Install and enable auditd with

sudo dnf install auditd
sudo systemctl enable auditd
sudo systemctl start auditd

Add a file or directory to monitor with

auditctl -w /etc/passwd -k password

-w is watch path
-k is a filter key we can use later to search through logs

Now we can search with ausearch

ausearch -k password

Using Preconfigured Rules

There are already some preconfigured rules in /usr/share/audit/sample-rules/

We can copy those to /etc/auditd/rules.d/ and use them.

cd /usr/share/audit/sample-rules/
cp 10-base-config.rules 30-stig.rules 31-privileged.rules 99-finalize.rules /etc/audit/rules.d/
augenrules --load

Note on the 31-privileged.rules file. You’ll need to run the commands in the file which will create a new file. Then we can copy that to “/etc/auditd/rules.d/”

find /bin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' > priv.rules
#find /sbin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' >> priv.rules
#find /usr/bin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' >> priv.rules
#find /usr/sbin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' >> priv.rules
#filecap /bin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules
#filecap /sbin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules
#filecap /usr/bin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules
#filecap /usr/sbin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules

And Copy priv.rules to /etc/audit/rules.d/31-privileged.rules. Overwrite the file there if needed.

cp ./priv.rules /etc/audit/rules.d/31-privileged.rules

Load the rules.

augenrules --load

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/auditing-the-system_security-hardening

Repairing Fedora grub from chroot

Notes from repairing a Fedora drive.

Mount system in chroot.

If the system is a raid drive and your not able to access it refer to this post.  May just need to install the raid utilities.

For mounting the chroot environment refer to this post

Repairing grub

yum install grub2-efi-*

Install grub.  Change /sda to your drive, may need to specify the efi partition.

grub2-install /dev/sda

If your boot and efi partitions are mounted.

grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg

Reboot.

DNF/YUM not working in chroot environment

The reason is probably because the chrooted environment can’t resolve DNS.

Test it with

ping incredigeek.com

If it is not resolving, edit “/etc/resolv.conf” and change/add your nameserver.  Or just replace everything in it with

echo "nameserver 4.2.2.2" > /etc/resolv.conf

It should now be able to resolve and you should be able to use yum, or dnf.

yum update

Add Self Signed SSL certificate to LibreNMS in CentOS

Install mod_ssl

yum install mod_ssl -y

Create Directory for SSL key.

mkdir /etc/ssl/key
chmod 700 /etc/ssl/key

Create certificate.

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/pki/tls/private/localhost.key -out /etc/pki/tls/certs/localhost.crt

Fill out the info or what is applicable.

Now edit the LibreNMS Apache config file /etc/httpd/conf.d/librenms.conf

All you have to do is add the following three lines under the VirtualHost and change *:80 to *:443.

SSLEngine on
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSL CertificateKeyFile /etc/pki/tls/private/localhost.key

So when your finished the file should look like this.

<VirtualHost *:443>
 DocumentRoot /opt/librenms/html/
 ServerName server_hostname_or_IP
 SSLEngine on
 SSLCertificateFile /etc/pki/tls/certs/localhost.crt
 SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
 CustomLog /opt/librenms/logs/access_log combined
 ErrorLog /opt/librenms/logs/error_log
 AllowEncodedSlashes NoDecode
 <Directory "/opt/librenms/html/">
 Require all granted
 AllowOverride All
 Options FollowSymLinks MultiViews
 </Directory>
</VirtualHost>

Don’t forget to allow https/port 443 traffic through the firewall.  Guide here

If you have any issues, you may need to chmod the key and crt file.

chmod 644 /etc/pki/tls/certs/localhost.crt
chmod 644 /etc/pki/tls/private/localhost.key

You should now be able to access LibreNMS using https.  Note, you’ll need to allow an exception in your browser for your self signed certificate.

https://LibreNMS_IP_Address

Recovering LibreNMS from crashed XenServer VM

Had a LibreNMS instance crash, or the VM crashed, not bootable anymore.  Was able to boot it up on a CentOS iso with rescue mode, which gave me access to the files.  So the idea is to manually copy off the LibreNMS files and LibreNMS database and import them to a new LibreNMS instance

Issues

  • Need to access the the system files
  • MySQL doesn’t start in a chroot environment, so no way to do a mysqldump —  (Has to do with systemv or something)

Steps

  1. Create new LibreNMS VM with a new instance of LibreNMS installed
  2. Gain access to the crashed system
  3. Copy over LibreNMS MySQL databases to new LibreNMS instance
  4. Copy over LibreNMS files “/opt/librenms” to new LibreNMS instance
  5. Clean up.  Set users on directories, check SELinux etc.

1. Installing New LibreNMS VM

  • Guide for Ubuntu/Debian distro’s here
  • Guide for for Fedora/CentOS/RedHat here

2. Gaining access to crashed VM

There can be a couple of ways to gain access to a crashed XenServer VM.  One of the easiest ways is to boot up in recovery mode and go through the installers rescue mode.  Guide to boot up XenServer VM in recovery mode here.

The installer rescue mode should detect the OS and mount everything.  If not you should be able to mount the root partition manually.

Once booted up, you’ll need to enable network access if your going to use sftp or scp to copy files.  There are a few different ways to do this

  • Run “dhclient” to pull an address via DHCP
  • Set a static IP address
    • Guide for Ubuntu/Debian distro’s here
    • Guide for fedora/CentOS/RedHat here

3.Copy LibreNMS Mysql Database

Backup the LibreNMS MySQL database directory

tar czvf librenms_mysql.tgz /var/lib/mysql

Use scp or sftp to copy it to the new LibreNMS instance

scp librenms_mysql.tgz user@new_LibreNMS_ip

Now on the new LibreNMS instance we need to run the following few commands

systemctl stop mariadb
rm -rf /var/lib/mysql/*
tar xzvf librenms_mysql.tgz -C /
chown -R mysql:mysql /var/lib/mysql

4. Copy LibreNMS directory

Backup the LibreNMS directory

tar czvf librenms.tgz /opt/librenms

Use scp or sftp to copy it to the new LibreNMS instance

scp librenms.tgz user@new_LibreNMS_ip

Now on the new LibreNMS instance we need to run the following few commands

rm -rf /opt/librenms/*
tar xzvf librenms.tgz -C /
chown -R librenms:librenms /opt/librenms

5. Clean up

Disable SELinux if you have not already.  Guide here

Restart apache, and start mysql.  If your on Ubuntu, the services are named apache2 and mysql

systemctl restart httpd
systemctl restart mariadb

That should get it working, if not try a reboot.

Special notes

The whole MySQL directory needs to be copied, there are innodb files that will keep MySQL from starting if they are not copied.

There is some good info here
https://stackoverflow.com/questions/1795176/how-to-change-mysql-data-directory