Bulk Update SNMP v3 Settings for Devices in LibreNMS

With support for DES being dropped, you may be faced with having to upgrade device settings to AES. In this post we’ll explore changing the settings in LibreNMS for all Mikrotik devices and then touch on making changes to a group of Mikrotik devices.

Upgrading SNMP Settings for Devices in LibreNMS

In LibreNMS, we can go to Device -> Device Settings (Gear on the right hand side) -> SNMP, to set the SNMP settings for that device.

Since this would get rather boring to change on multiple devices, and these settings are all in a MySQL database, we can skip using the mouse and use a few MySQL commands to update multiple devices at once.

Log into the LibreNMS server over ssh and then connect to the MySQL database

mysql -u librenms -p librenms

First we can get a list of all the devices (Mikrotik routers in this example) and show the hostname with the SNMP authentication and cryptography algorithms.

select hostname,authalgo,cryptoalgo from devices where os="routeros";

Now if we want to update the cryptography settings for all of our Mikorotik devices, we can do the following.

update devices cryptoalgo set cryptoalgo="AES"  where os="routeros";

This will set all of the devices to use AES for the cryptography algorithm.

We can also change the authentication algorithm to SHA with this

update devices authalgo set authalgo="SHA"  where os="routeros";
LibreNMS update device SNMP settings

Bulk updating of Network Devices

The bottom “script” can be used for changing SNMP settings on multiple Mikrotik devices.

Create a mikrotik.lst file with all the IP addresses of all the devices you need to update. Can you use the above MySQL commands to get a list from LibreNMS.

Change the following options in the script

  • routerpassword to the Mikrotik password
  • admin to your username
  • encryptionpassword to your SNMP encryption password
  • authpassword to your authentication password
  • addresses=192.168.0.0/16 to the list of IP addresses that should be able to access SNMP info on the mikrotik device. AKA your LibreNMS server.
  • SNMPname to your SNMP username
for ip in `cat mikrotik.lst` 
do 
echo $ip 
timeout 15 sshpass -p 'routerpassword' ssh -o StrictHostKeyChecking=no admin@${ip} -p1022 '/snmp community set addresses=192.168.0.0/16 authentication-protocol=SHA1 authentication-password=authpassword encryption-protocol=AES encryption-password=encryptionpassword security=private read-access=yes write-access=no SNMPname'
done

Copy and paste the above “code” in a shell script file.

nano mikrotik.sh
chmod +x mikrotik.sh 
./mikrotik.sh

The script should run and update all the SNMP settings on all the devices in mikrotik.lst

Add Custom OID to device in LibreNMS

In the post, we’ll be adding a custom OID for a Ubiquiti Solar Charge Controller.

  • Find the device OID
  • Add the Custom OID in LibreNMS
  • Check the graph

Find the custom OID

Check out the following post if you are trying to add a Ubiquiti Solar Charge controller graph to LibreNMS. Otherwise you may need to do some googling around looking for the OID.

Add Custom OID in LibreNMS

Go to your device -> Settings(Little Gear) -> Custom OID -> +Add New OID

Adding Custom OID in LibreNMS for Ubiquiti UI Charge Controller

Couple notes about the information.
– There needs to be a 0 after the end of the OID.
– Data Type needs to be Gauge, Not Counter. A gauge can go up and down. A counter just counts up.

Hit “Test OID” and you should get a little notification saying it got a value for the OID. In this case 25572.

Test Custom OID in LibreNMS for Ubiquiti UI Charge Controller

Now Hit “Save OID”

Check the Graph

Now LibreNMS should start graphing our Custom OID. You may need to give it a minute to start showing data.

The graphs show up under Graphs -> Custom OID

Where are Custom OID Graphs?
LibreNMS Custom OID Graph

Setting up Grafana on LibreNMS

Thanks to the guys who put together the information at the following links.

https://wadman.co.nz/2021/01/02/Viewing-LibreNMS-data-in-Grafana/
https://www.reddit.com/r/LibreNMS/comments/ojc8cc/how_to_almost_natively_integrate_librenms_and/

I ran into some issues trying to get this to work. So here are some of my notes. I already had a LibreNMS installation set up.

  1. Install RRDReST
    1. Install Docker
    2. Configure Docker compose file
  2. Configure LibreNMS API User and Key
  3. Set up and Configure Grafana
    1. Install Grafana
    2. Install JSON Data Source Plugin
    3. Add LibreNMS API Data Source
    4. Add RRDReST Data Source
    5. Import Dashboard into Grafana
  4. View graphs

Installing RRDReST

NOTE FOR CENTOS 8, ALMALINUX 8 and 9
The steps for installing RRDReST are slightly different. Check out the following post.
https://www.incredigeek.com/home/setting-up-rrdrest-on-centos-8-or-almalinux-9/

I had issues installing RRDReST. I am guessing it had to do with it accessing files. I was able to install it in a docker container.

  1. Install Docker
  2. Create a compose file
  3. Install container

Install docker

sudo yum install -y docker docker-compose
sudo systemctl enable docker

Create docker compose file with the following options

vi docker-compose.yml

Change the TZ to your time zone. If you have issues with the graphs, most likely something is off with the time zone between this container and Grafana/LibreNMS server

version: "3.5"

services:

  rrdrest:
    image: michaelwadman/rrdrest:latest
    container_name: rrdrest
    restart: always
    volumes:
      - "/opt/librenms:/opt/librenms"
    environment:
      - TZ=America/Denver

Save the file and start and setup the container with

sudo docker-compose up -d

You will need your docker container IP address to setup the connection in Grafana

sudo docker exec -it rrdrest ip addr | grep eth0

Congratulations. You should now have a RRDReST docker container that will auto start on system boot and has the correct time zone.

Configure LibreNMS API User and Key

  1. Create a Grafana user in LibreNMS. (Settings Gear -> Manage Users -> Add Users)
    You could technically skip this step and use an existing user.
  2. Create API token for the newly created user (Setting Gear -> API -> API Settings)
Create a Grafana user in LibreNMS
Setup API Key for Grafana in LibreNMS

Set up and Configure Grafana

Basic steps are as follows

  1. Install Grafana
  2. Install JSON Data Source Plugin
  3. Configure Data Sources
    1. LibreNMS API
    2. RRDRest API
  4. Import Dashboard into Grafana

Install Grafana

There is not anything special with installing Grafana on the same server as LibreNMS. You can follow the official guide to install it

https://grafana.com/docs/grafana/latest/installation/

After Grafana is installed, install the JSON API data source. You can do this using the grafana-cli

grafana-cli plugins install marcusolsson-json-datasource

A note on SSL/TLS certificates. If you have an SSL certificate for LibreNMS, you can use it for grafana. If you run into issues, try copying the cert (fullchain.pem, privkey.pem) to /etc/grafana/

You’ll most likely need to change owner

sudo chown root:grafana /etc/grafana/*.pem

And maybe the file permissions.

sudo chmod 640 /etc/grafana/*.pem

Install JSON Data Source Plugin

This is fairly straight forward.

grafana-cli plugins install marcusolsson-json-datasource

Add LibreNMS API Data Source

In Grafana, go to Configuration -> Data Sources -> Add data source

  • Set Name for Data Source
  • URL should be https://your_librenms_url/api/v0
  • Add Custom HTTP Header
    • Header field should be “X-Auth-Token”
    • Value field should contain the API token we created in LibreNMS
  • Save and Test
    If you receive any errors, refer to the Troubleshooting part at the end.
Adding LibreNMS API Data Source in Grafana

Add RRDReST Data Source

In Grafana, go to Configuration -> Data Sources -> Add data source

  • Set Name for Data Source
  • URL needs to be your docker container IP address (Steps above)
  • Save and Test (Should return “Unprocessable Entity”)
Adding RRDReST API Data Source in Grafana

Import Dashboard into Grafana

Now we need a dashboard to present our data.

  • Go to Create -> Import
  • Upload JSON file (Download from here or PasteBin )
  • Under RRDReST API , select our RRDReST Data Source
  • Under LibreNMS API , select our LibreNMS Data Source
  • Click Import

You should now be able to view your dashboard and use the drop down menus to select devices

Grafana viewing bandwidth on device being monitored by LibreNMS

Troubleshooting

There were a couple of issues I ran into while trying to get everything working together.

RRDReST shows 404 Not Found

Issue: When trying to run RRDReST with uvicorn, I was never able to access the rrd files, even the test rrd files that are included when installing RRDReST. I am guessing it is either a permisions issue, or something unable to access the files.
Work around: Install RRDReST via Docker container.

Error Running uvicorn RRDReST

Error Adding LibreNMS API

Issue: Get a “JSON API: Bad Request” when trying to set up the LibreNMS API Data Source in Grafana.

Work around: Install a valid SSL Certificate and set up a DNS record so you can access LibreNMS at librenms.yourdomain.com.

More info: I would assume that “Skip TLS Verify” would work with or without a valid certificate, but it would not work for me. There are potentially some other options with modifying how Nginx or Apache is set up that would get this working. If you setup Grafana to use a SSL certificate, you may need to copy the certificate files (fullchain.pem, privkey.pem) to /etc/grafana/ and run “chown root:grafana *.pem” to let grafana have access to the files.

Grafana LibreNMS API JSON API: Bad Request

LibreNMS – Package not found: The ‘command_runner>=’

The Problem

Running the ./validate.php script returns the following error

[FAIL]  Python3 module issue found: 'Required packages: ['PyMySQL!=1.0.0', 'python-dotenv', 'redis>=3.0', 'setuptools', 'psutil>=5.6.0', 'command_runner>=1.3.0']
Package not found: The 'command_runner>=1.3.0' distribution was not found and is required by the application
'
        [FIX]:
        pip3 install -r /opt/librenms/requirements.txt

Running the [FIX] throws an error saying gcc failed with exit status 1.

The Solution

Fortunately this issue is easy to resolve.

First we need to install python3-devel

sudo yum install python3-devel

Next, as the librenms user, run the pip command to install the requirements.

pip3 install --user -U -r /opt/librenms/requirements.txt

Run ./validate.php to verify that everything is working.

librenms validate.php results

Get list of IP DHCP Leases from Mikrotik RouterOS Command Line

Sometimes it is nice to have a list of just the IP addresses and the hostname or mac address. Can be especially helpful when you are trying to get a list of devices to do a bulk update in LibreNMS. Helpful LibreNMS links for renaming and showing down devices

we are going to run the command to pull the DHCP information and then use Unix utilities to parse the data out.

Replace dhcp8 with the correct dhcp server in the mikrotik.

First, SSH into your Mikrotik router.

ssh user@mikrotik

Next we’ll want to go to the ip dhcp-server lease directory

/ip dhcp-server lease

Now we can run the following command to loop through and pull out all the info we need.
You can change the info like “mac-address” for something like “status”. You can use Tab to see what options are available. Just delete mac-address and hit tab twice.

:foreach i in=[find] do={ :put ([get $i address].",".[get $i mac-address].",".[get $i server].",".[get $i host-name ])}

Found the above helpful info from the following pdf file.

https://mum.mikrotik.com/presentations/PH18/presentation_5113_1516612695.pdf

LibreNMS Error Running ./validate.php – Database: incorrect column…

LibreNMS error while running ./validate.php

Recently I started getting the following error while running ./validate.php.

$ sudo -u librenms ./validate.php
====================================
Component | Version
--------- | -------
LibreNMS  | 21.8.0-41-g0a76ca4
DB Schema | 2021_08_26_093522_config_value_to_medium_text (217)
PHP       | 7.4.20
Python    | 3.6.8
MySQL     | 10.5.10-MariaDB
RRDTool   | 1.4.8
SNMP      | NET-SNMP 5.7.2
====================================

[OK]    Composer Version: 2.1.6
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[FAIL]  Database: incorrect column (notifications/datetime)
[FAIL]  Database: incorrect column (users/created_at)
[FAIL]  We have detected that your database schema may be wrong, please report the following to us on Discord (https://t.libren.ms/discord) or the community site (https://t.libren.ms/5gscd):
        [FIX]:
        Run the following SQL statements to fix.
        SQL Statements:
         SET TIME_ZONE='+00:00';
         ALTER TABLE `notifications` CHANGE `datetime` `datetime` timestamp NOT NULL DEFAULT '1970-01-02 00:00:00' ;
         ALTER TABLE `users` CHANGE `created_at` `created_at` timestamp NOT NULL DEFAULT '1970-01-02 00:00:01' ;

Reading online it sounds like some of the recent changes are causing the issue. Looks fairly easy to resolve though.

First we’ll need to get a MySQL prompt. We’ll do that by running

mysql -u librenms -p librenms

It’ll ask use for the librenms user’s mysql password.

Once we have the MySQL prompt we can just copy and paste the commands in.

SET TIME_ZONE='+00:00';
ALTER TABLE `notifications` CHANGE `datetime` `datetime` timestamp NOT NULL DEFAULT '1970-01-02 00:00:00' ;
ALTER TABLE `users` CHANGE `created_at` `created_at` timestamp NOT NULL DEFAULT '1970-01-02 00:00:01' ;

Type ‘quit’ to exit MySQL and lets run the validate script again.

sudo -u librenms ./validate.php

Everything should check out OK.

LibreNMS Error – port_groups Base table or view already exists

port_groups issue upgrading LibreNMS

The above error can be resolved by dropping the port_groups table in mysql. You may double check that there is no information in the table. I checked this instance and it didn’t have any data in the table so I dropped it with

drop table port_groups;

I then ran the lnms command again and it created the table and started migrating.

Show Down Devices From SSH – LibreNMS

Fortunately, it is super easy to get a list of down devices in LibreNMS. All we need to do is a dump of the devices from MySQL.

You can use the following command to print which devices are down. Should be the same devices that show down on your dashboard.

mysql -u librenms -p librenms -e 'use librenms ; select hostname,sysName,status from devices where status=0'

Tip: You can change 0 to 1 to see all up devices.

Show down devices via SSH in LibreNMS

Migrate LibreNMS to new Server

Some quick notes on manually moving LibreNMS to a new server.  For more detailed instructions you can refer to the following post

Recovering LibreNMS from crashed XenServer VM

Steps

  1. Backup /opt/librenms file
  2. Backup librenms database
  3. Install LibreNMS on new server
  4. Import /opt/librenms files
  5. Import librenms database

Backing up LibreNMS files

LibreNMS stores all it’s files in /opt/librenms
Quick and easy way to back up the LibreNMS files is to use tar.

tar zcvf librenms_backup.tgz /opt/librenms

Migrating Database

I believe you can copy the whole /var/lib/mysql directory.  Should be able to use the above command.  If not you can do a mysqldump and then import it on the new system.

Cannot load Zend OPcache – it was already loaded

Started getting the following error while running ./validate.php for LibreNMS

Cannot load Zend OPcache - it was already loaded

Looks like the problem arises out of PHP trying to load two ini files for OPcache. Was trying to enable OPcache for LibreNMS so I created an opcache.ini file and put the settings in it. I missed the default 10-opcache.ini file.

Moving all the settings into the 10-opcache.ini file and deleting the created opcache.ini file resolved the issue for me.

https://unix.stackexchange.com/questions/253448/php-7-install-throws-cannot-load-zend-opcache-it-was-already-loaded-error/253484