Show Down Devices From SSH – LibreNMS

Fortunately, it is super easy to get a list of down devices in LibreNMS. All we need to do is a dump of the devices from MySQL.

You can use the following command to print which devices are down. Should be the same devices that show down on your dashboard.

mysql -u librenms -p librenms -e 'use librenms ; select hostname,sysName,status from devices where status=0'

Tip: You can change 0 to 1 to see all up devices.

Show down devices via SSH in LibreNMS

Migrate LibreNMS to new Server

Some quick notes on manually moving LibreNMS to a new server.  For more detailed instructions you can refer to the following post

Recovering LibreNMS from crashed XenServer VM

Steps

  1. Backup /opt/librenms file
  2. Backup librenms database
  3. Install LibreNMS on new server
  4. Import /opt/librenms files
  5. Import librenms database

Backing up LibreNMS files

LibreNMS stores all it’s files in /opt/librenms
Quick and easy way to back up the LibreNMS files is to use tar.

tar zcvf librenms_backup.tgz /opt/librenms

Migrating Database

I believe you can copy the whole /var/lib/mysql directory.  Should be able to use the above command.  If not you can do a mysqldump and then import it on the new system.

LibreNMS upgrade PHP from 7.2 to 7.4 – CentOS 7

First you’ll need to install the remi repo

sudo wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo wget http://rpms.remirepo.net/enterprise/remi-release-7.rpm
rpm -Uvh remi-release-7.rpm epel-release-latest-7.noarch.rpm

You’ll need to install yum-utils, disable all old php versions, and enable the remi-php74

sudo yum install -y yum-utils
sudo yum-config-manager --disable remi-php56
sudo yum-config-manager --disable remi-php71
sudo yum-config-manager --disable remi-php72
sudo yum-config-manager --disable remi-php73
sudo yum-config-manager --enable remi-php74

If you run into issues you may want to see if you have the webtatic repo installed. I had to disable it to get php to update.

vi /etc/yum.repos.d/webtatic
[webtatic]
name=Webtatic Repository EL7 - $basearch
baseurl=https://repo.webtatic.com/yum/el7/$basearch/
mirrorlist=https://mirror.webtatic.com/yum/el7/$basearch/mirrorlist
failovermethod=priority
enabled=0   <- Disabled by changing to a 0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-webtatic-el7

And running the yum update should update it to php 7.4

sudo yum update

More information available here.

LibreNMS update to Python 3

You may get the following alert in LibreNMS. Basically you need to install python 3 to keep things up to date.

Python 3 is required to run LibreNMS as of May, 2020. You need to install Python 3 to continue to receive updates. If you do not install Python 3 and required packages, LibreNMS will continue to function but stop receiving bug fixes and updates.

Install Python 3

Install Python 3 with yum, or apt if you are on a Debian based distro.

sudo yum install python3
sudo pip3 install -r /opt/librenms/requirements.txt

Verify LibreNMS is updated and working

Run the following commands to make sure that LibreNMS is working correctly and is up to date.

cd /opt/librenms
sudo ./validate.php
sudo ./daily.sh

LibreNMS backup script

You should be able to copy and paste the following in a backup.sh file and then execute from cron. Should work out of the box, but you can change the backup directory and the teams.sh path if needed/wanted.

#!/bin/bash

# LibreNMS backup script
# Jan 1, 2019

lDate=`date +%Y%m%d-%H%M`       # local date + hour minute
dDate=`date +%Y%m%d`            # todays date

# If you have the teams.sh script, you can trigger a backup notification
ALERT="/home/admin/teams.sh -b"

# Directory to backup to
bDir="/backup"
bName="librenms_backup"

# MySQL settings for tar and sqldump
sqlDir="/var/lib/mysql"
sqlDB="librenms"
sqlUN="root"
sqlPW=""
LOG="${bDir}/${lDate}-${bName}.log"

# Directory that contains data
dDir="/opt/librenms"

# tar LibreNMS dir
# tar SQL dir "the whole thing with the innode files
# sql dump of the db for extra redundancy

if [ -d ${bDir} ]; then
echo "backup dir exist, starting to backup"
else
        echo "backup dir not available.  Quiting"
        exit 1
fi

${ALERT} "Starting backup for ${bName} - `date`"

systemctl stop mariadb httpd
# LibreNMS data backup
tar -zcvf ${bDir}/${lDate}-${bName}.tgz ${dDir}
if [ $? -eq 0 ]; then
        echo "Tar succesfully backed up ${bDir}"
else
        echo "Tar failed while trying to backup ${dDir}"
        echo " ${lDate} - Tar failed while trying to backup ${dDir}" >> ${LOG}
        ${ALERT} "${lDate} - Tar failed while trying to backup ${dDir}"
fi

# MySQL data backup
tar -zcvf ${bDir}/${lDate}-${bName}-mysql.tgz ${sqlDir}
if [ $? -eq 0 ]; then
        echo "Tar succesfully backed up ${sqlDir}"
else
        echo "Tar failed while trying to backup ${sqlDir}"
        echo " ${lDate} - Tar failed while trying to backup ${sqlDir}" >> ${LOG}
        ${ALERT} "${lDate} - Tar failed while trying to backup ${sqlDir}"
fi

systemctl start mariadb httpd
sleep 5

 # SQL dump
mysqldump -u ${sqlUN} -p'4rfvBHU8!' ${sqlDB} > ${bDir}/${lDate}-${bName}.sql
if [ $? -eq 0 ]; then
        echo "MySQL DB dumped"
else
        echo "Ran into error while doing sql dump"
        echo "${lDate} - Ran into error while doing sql dump" >> ${LOG}
        ${ALERT} "${lDate} - Ran into error while doing sql dump"
fi

echo "Removing old backups"
if ( ls ${bDir} | grep -q ${dDate} );then
        find ${bDir}/* -prune -mtime +31 -exec rm {} \;
else
        echo "Looks like there are no backup files!  Aborting!!!"
        ${ALERT} "${lDate} - Error: find failed to find any backup files in backup dir.  Aborting!!!"
fi

${ALERT} "Finished backup for ${bName} - `date`"

LibreNMS ./validate.sh – Some folders have incorrect file permissions, this may cause issues.

Helpful article here. https://wifitechtalk.com/librenms-permissions-error/

When running the following command,

sudo /opt/librenms/validate.sh

I’ve been getting the following errors.

[FAIL]  Some folders have incorrect file permissions, this may cause issues.
    [FIX]: 
    sudo chown -R librenms:librenms /opt/librenms
    sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
    sudo chmod -R ug=rwX /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
    Files:
     /opt/librenms/storage/framework/views/adc52b677409cdba8d8e89dc

You can run the commands and fix the problem, but they pop up later and it does the same thing.

Create Script to fix permissions

Work around is to have a script run these commands every few minutes.

vi /root/librenms_fix_permissions.sh

Add the following

#!/bin/bash 
sudo chown -R librenms:librenms /opt/librenms 
sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/ 
sudo chmod -R ug=rwX /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/

Make the script executable

sudo chmod +x librenms_fix_permissions.sh

Add to Crontab

sudo crontab -e 

Add the following entry to Cron to run every 30 minutes

*/30 * * * * /root/librenms_fix_permissions.sh

LibreNMS – Daily.sh failed

All the following commands are run from the “/opt/librenms” directory

$ sudo ./daily.sh 
Re-running /opt/librenms/daily.sh as librenms user
Updating to latest codebase                        FAIL
error: Your local changes to the following files would be overwritten by merge:
        html/js/lang/de.js
        html/js/lang/en.js
Please, commit your changes or stash them before you can merge.
...
$

Checking with validate.php shows that you can run githup-remove to fix it

$ sudo ./validate.php
[WARN]  Your install is over 24 hours out of date, last update: Mon, 23 Feb 2020 05:43:12 +0000
         [FIX]: 
         Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.
 [WARN]  Your local git contains modified files, this could prevent automatic updates.
         [FIX]: 
         You can fix this with ./scripts/github-remove
         Modified Files:
          html/js/lang/de.js
...
$

Run “sudo ./scripts/github-remove -d”

$ sudo ./scripts/github-remove -d
Are you sure you want to delete all modified and untracked files? [y/N] y
$

You should also be able to remove the files individually if the above command does not work.

Run validate again to make sure it checks out good.

Now run ./daily.sh

LibreNMS – Setting up Alerts to monitor Voltage on Mikrotik Routers

Create Alert Rule

Go to “Alerts -> Alert Rules” Then “Create rule from Collection”

Search for “Sensor under limit” and Select

Setup Sensor under limit

Configure Alert Rule

Add another rule to limit to just the Voltage Sensors by using the “sensors.sensor_class = Voltage”

Select your Groups and Transports and Save.

Modify Alert Rule

Disable Alerts for Unused Ports

You may get some alerts because there are some ports that do not have any any voltage on them. You can disable them on a per device basis by going to the “device -> Edit -> Health” and turning alerts off

Turn off voltage alerts on port that has no PoE