LibreNMS backup script

You should be able to copy and paste the following in a backup.sh file and then execute from cron. Should work out of the box, but you can change the backup directory and the teams.sh path if needed/wanted.

#!/bin/bash

# LibreNMS backup script
# Jan 1, 2019

lDate=`date +%Y%m%d-%H%M`       # local date + hour minute
dDate=`date +%Y%m%d`            # todays date

# If you have the teams.sh script, you can trigger a backup notification
ALERT="/home/admin/teams.sh -b"

# Directory to backup to
bDir="/backup"
bName="librenms_backup"

# MySQL settings for tar and sqldump
sqlDir="/var/lib/mysql"
sqlDB="librenms"
sqlUN="root"
sqlPW=""
LOG="${bDir}/${lDate}-${bName}.log"

# Directory that contains data
dDir="/opt/librenms"

# tar LibreNMS dir
# tar SQL dir "the whole thing with the innode files
# sql dump of the db for extra redundancy

if [ -d ${bDir} ]; then
echo "backup dir exist, starting to backup"
else
        echo "backup dir not available.  Quiting"
        exit 1
fi

${ALERT} "Starting backup for ${bName} - `date`"

systemctl stop mariadb httpd
# LibreNMS data backup
tar -zcvf ${bDir}/${lDate}-${bName}.tgz ${dDir}
if [ $? -eq 0 ]; then
        echo "Tar succesfully backed up ${bDir}"
else
        echo "Tar failed while trying to backup ${dDir}"
        echo " ${lDate} - Tar failed while trying to backup ${dDir}" >> ${LOG}
        ${ALERT} "${lDate} - Tar failed while trying to backup ${dDir}"
fi

# MySQL data backup
tar -zcvf ${bDir}/${lDate}-${bName}-mysql.tgz ${sqlDir}
if [ $? -eq 0 ]; then
        echo "Tar succesfully backed up ${sqlDir}"
else
        echo "Tar failed while trying to backup ${sqlDir}"
        echo " ${lDate} - Tar failed while trying to backup ${sqlDir}" >> ${LOG}
        ${ALERT} "${lDate} - Tar failed while trying to backup ${sqlDir}"
fi

systemctl start mariadb httpd
sleep 5

 # SQL dump
mysqldump -u ${sqlUN} -p'4rfvBHU8!' ${sqlDB} > ${bDir}/${lDate}-${bName}.sql
if [ $? -eq 0 ]; then
        echo "MySQL DB dumped"
else
        echo "Ran into error while doing sql dump"
        echo "${lDate} - Ran into error while doing sql dump" >> ${LOG}
        ${ALERT} "${lDate} - Ran into error while doing sql dump"
fi

echo "Removing old backups"
if ( ls ${bDir} | grep -q ${dDate} );then
        find ${bDir}/* -prune -mtime +31 -exec rm {} \;
else
        echo "Looks like there are no backup files!  Aborting!!!"
        ${ALERT} "${lDate} - Error: find failed to find any backup files in backup dir.  Aborting!!!"
fi

${ALERT} "Finished backup for ${bName} - `date`"

LibreNMS ./validate.sh – Some folders have incorrect file permissions, this may cause issues.

Helpful article here. https://wifitechtalk.com/librenms-permissions-error/

When running the following command,

sudo /opt/librenms/validate.sh

I’ve been getting the following errors.

[FAIL]  Some folders have incorrect file permissions, this may cause issues.
    [FIX]: 
    sudo chown -R librenms:librenms /opt/librenms
    sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
    sudo chmod -R ug=rwX /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
    Files:
     /opt/librenms/storage/framework/views/adc52b677409cdba8d8e89dc

You can run the commands and fix the problem, but they pop up later and it does the same thing.

Create Script to fix permissions

Work around is to have a script run these commands every few minutes.

vi /root/librenms_fix_permissions.sh

Add the following

#!/bin/bash 
sudo chown -R librenms:librenms /opt/librenms 
sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/ 
sudo chmod -R ug=rwX /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/

Make the script executable

sudo chmod +x librenms_fix_permissions.sh

Add to Crontab

sudo crontab -e 

Add the following entry to Cron to run every 30 minutes

*/30 * * * * /root/librenms_fix_permissions.sh

LibreNMS – Daily.sh failed

All the following commands are run from the “/opt/librenms” directory

$ sudo ./daily.sh 
Re-running /opt/librenms/daily.sh as librenms user
Updating to latest codebase                        FAIL
error: Your local changes to the following files would be overwritten by merge:
        html/js/lang/de.js
        html/js/lang/en.js
Please, commit your changes or stash them before you can merge.
...
$

Checking with validate.php shows that you can run githup-remove to fix it

$ sudo ./validate.php
[WARN]  Your install is over 24 hours out of date, last update: Mon, 23 Feb 2020 05:43:12 +0000
         [FIX]: 
         Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.
 [WARN]  Your local git contains modified files, this could prevent automatic updates.
         [FIX]: 
         You can fix this with ./scripts/github-remove
         Modified Files:
          html/js/lang/de.js
...
$

Run “sudo ./scripts/github-remove -d”

$ sudo ./scripts/github-remove -d
Are you sure you want to delete all modified and untracked files? [y/N] y
$

You should also be able to remove the files individually if the above command does not work.

Run validate again to make sure it checks out good.

Now run ./daily.sh

LibreNMS – Setting up Alerts to monitor Voltage on Mikrotik Routers

Create Alert Rule

Go to “Alerts -> Alert Rules” Then “Create rule from Collection”

Search for “Sensor under limit” and Select

Setup Sensor under limit

Configure Alert Rule

Add another rule to limit to just the Voltage Sensors by using the “sensors.sensor_class = Voltage”

Select your Groups and Transports and Save.

Modify Alert Rule

Disable Alerts for Unused Ports

You may get some alerts because there are some ports that do not have any any voltage on them. You can disable them on a per device basis by going to the “device -> Edit -> Health” and turning alerts off

Turn off voltage alerts on port that has no PoE

Manually create graph of device in LibreNMS

In LibreNMS, you can hit the “Show RRD Command” to give you the command to manually create a graph.  Copy the command.

In the command below, the path shows the IP address of the device to generate the graph from.  Theoretically you can change that to a different IP to generate graphs from other devices.

SSH into server server and run the command, change /tmp/randomtext to something like /root/mikrotik_092619.png or whatever you want.  You can also adjust the size, color etc of the graph.

 rrdtool graph /tmp/randomtext --alt-y-grid --alt-autoscale-max --rigid -E --start 1515946500 --end 1516551300 --width 1712.7 --height 483 -c BACK#EEEEEE00 -c SHADEA
#EEEEEE00 -c SHADEB#EEEEEE00 -c FONT#000000 -c CANVAS#FFFFFF00 -c GRID#a5a5a5 -c MGRID#FF9999 -c FRAME#5e5e5e -c ARROW#5e5e5e -R normal --font LEGEND:8:DejaVuSansMono --font AXIS:7:DejaVuSansMono -
-font-render-mode normal COMMENT:'Volts Cur Min Max\n' DEF:sensor495=/opt/librenms/rrd/192.168.88.1/sensor-voltage-routeros-0.rrd:sensor:AVERAGE LINE1:sensor495#CC0000:'Voltage 0
 ' GPRINT:sensor495:LAST:%5.1lfV GPRINT:sensor495:MIN:%5.1lfV GPRINT:sensor495:MAX:%5.1lfV\l

LibreNMS manually adjust GPS coordinates for a Geo Location

The locations are stored in the LibreNMS locations table

mysql -u librenms -p librenms

To list all the locations

select * from locations;

Set the specific coordinates for a location using the following command.
Replace the lat and lng, also the id should match the line your wanting to update.

update locations set lat = "33.5140",lng = "-86.8178" where id=25;

LibreNMS dump devices in MySQL database

All the devices are stored in the librenms datebase devices table.

You can dump all of them with the following command from a shell prompt.

sudo mysql -u librenms  -p librenms -e "select hostname,sysName from devices"

Example output

[jim@localhost ~]$ sudo mysql -p librenms -e "select hostname,sysName from devices"
 Enter password:
 +----------------------------------------+--------------------------------+
 | hostname                               | sysName                        |
 +----------------------------------------+--------------------------------+
 | 192.168.1.1                            | linksys                        |
 | 192.168.1.9                            | wifi-unifi                     |
 +----------------------------------------+--------------------------------+

Install Oxidized on LibreNMS

Install Oxidized

Install Instructions from here

Install prerequisites

For CentOS, RockyLinux, and Alma Linux.

sudo yum -y install make cmake which sqlite-devel openssl-devel libssh2-devel ruby gcc ruby-devel libicu-devel gcc-c++

Install Ruby Gems

If you run into issues with installing the oxidized-web gem because of a ruby version, try installing ruby 2.6.1 via rvm the docs say install 2.1.2, but was having issues with it. 2.6.1 seems fine.

Install Oxidized

Note: If you run into issues with oxidized not being able to ssh into devices, and showing “OpenSSL::PKey::PKeyError with msg “dh#set_pqg= is incompatible with OpenSSL 3.0″” in the log, try installing from git.

gem install oxidized 
gem install oxidized-script
gem install oxidized-web 

Run Oxidized twice to generate the config

oxidized

If you have issues running oxidized due to an invalid or self signed ssl cert, you can either fix the cert or ignore it in the oxidized config. https://github.com/ytti/oxidized/pull/618/files

Modify the Oxidized config file in “/root/.config/oxidized/config” to look like this.

Modify the http portion of the config file to look like the following.

http:
url: https://localhost/api/v0/oxidized
scheme: https
secure: false

Setup Config

Put the config file in the users home directory.

vi .config/oxidized/config

Setup the config, Basic config below, edit as needed

---
username: admin
password: password
model: airos
resolve_dns: true
interval: 3600
use_syslog: false
debug: false
threads: 30
timeout: 20
retries: 3
prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/
rest: 127.0.0.1:8888
next_adds_job: false
vars: {}
groups:
airos:
username: ubnt
password: ubnt
routeros:
username: admin
password:
vars:
ssh_port: 2222
models: {}
pid: /home/incredigeek/.config/oxidized/pid
crash:
directory: /home/incredigeek/.config/oxidized/crashes
hostnames: false
stats:
history_size: 10
input:
default: ssh, telnet
debug: false
ssh:
secure: false
ftp:
passive: true
utf8_encoded: true
output:
default: file
file:
directory: /home/incredigeek/.config/oxidized/configs
source:
default: http
debug: false
http:
secure: false
scheme: https
url: https://localhost/api/v0/oxidized
map:
name: hostname
model: os
group: group
headers:
X-Auth-Token: '3wq2b87fj4e6fb5987b5812t6ej9709g'
model_map:
cisco: ios
juniper: junos
mikrotik: routeros

Setup as system service

Find the “oxidized.service” file, should be in

  /usr/local/rvm/gems/ruby-2.6.1/gems/oxidized-0.25.1/extra/oxidized.service

or

/usr/local/share/gems/gems/oxidized-0.28.0/extra/oxidized.service

and copy it to

  /usr/lib/systemd/system/

Edit it and change the user and try to launch it

systemctl start oxidized

If it fails, you may need to substitute the ExecStart= variable to the following more info

ExecStart=/usr/local/rvm/gems/ruby-2.6.1/wrappers/oxidize

Change the ruby number if yours is different.

Enable the service on system boot up and start

systemctl enable --now oxidized

Check that oxidized is running

systemctl status oxidized

LibreNMS config

Add the following to your /opt/librenms/config.php config file.

# Oxidized configuration
 $config['oxidized']['enabled']                  = TRUE;
 $config['oxidized']['url']                      = 'http://127.0.0.1:8888';
 $config['oxidized']['features']['versioning']   = true;
 $config['oxidized']['group_support']            = true;
 $config['oxidized']['default_group']            = 'default';
 $config['oxidized']['reload_nodes']             = true;
 $config['oxidized']['ignore_os'] = array('linux','windows');
 $config['oxidized']['ignore_types'] = array('server','power');

LibreNMS should now feed Oxidized the devices.

You can check in the LibreNMS interface to see if it is getting the configs. https://librenms/oxidized

Where librenms is your LibreNMS servers ip/hostname.