As we can see from above, we are getting random blank spaces in the graph. Possible causes for this are
Using SNMPv1 doesn’t support more then 125Mbps.
https://community.librenms.org/t/librenms-gaps-in-graphs/5209/2
Could be the server is not able to consistently complete polling of devices.
https://community.librenms.org/t/gaps-in-data-gaphs/469/2
Prerequisites
In this example, the server is already using Let’s Encrypt to create the certificate for a LibreNMS server. So all we are doing is copying the certificate to a Grafana directory, putting the correct permissions on it, and updating the Grafana config file to use the certificate.
Steps
In the following commands, change librenms.incredigeek.com to the directory that Let’s Encrypt is using for your fully qualified domain name (FQDN). Usually it is just your FQDN, but could also have -0001 or something appended to the end.
cp -f /etc/letsencrypt/live/librenms.incredigeek.com/privkey.pem /etc/grafana/ cp -f /etc/letsencrypt/live/librenms.incredigeek.com/fullchain.pem /etc/grafana/ chown root:grafana /etc/grafana/*.pem chmod 640 /etc/grafana/*.pem Enable grafana on system bootup
In the above, we are copying the privkey.pem and fullchain.pem to /etc/grafana. We are then setting the correct owner/permissions on the files so that the Grafana service can read the certificate.
This is super easy. Open up the Grafana config file in /etc/grafana.ini
vi /etc/grafana.ini
Find the following variables and configure them like so
protocol = https
cert_file = /etc/grafana/fullchain.pem
cert_key = /etc/grafana/privkey.pem
Restart Grafana
systemctl restart grafana-server.service
You should now have a working SSL certificate for the site.
Let’s Encrypt certificates need to be updated frequently. This means that we should automate the above steps to avoid any down time. After all, a monitoring tool with down time defeats the purpose of monitoring.
We’ll need to create a root crontab
sudo crontab -e
Add the following changing out the FQDN to your FQDN.
0 0 1 * * cp -f /etc/letsencrypt/live/librenms.incredigeek.com/privkey.pem /etc/grafana/ && cp -f /etc/letsencrypt/live/librenms.incredigeek.com/fullchain.pem /etc/grafana/ && chown root:grafana /etc/grafana/*.pem && chmod 640 /etc/grafana/*.pem
This is set to run once a month. Change if desired. Also change out librenms.incredigeek.com with your FQDN.
Note about domain name and IP addresses. Let’s Encrypt will not create a certificate for an IP address. You should be using a domain name instead (i.e. networkmonitoring.yourdomain.com) If the certificate is installed, and you access it via the IP address, you will receive a HTTPS error in your browser.
LibreNMS uses fping to check if devices are up or not. So if something is broken with fping, say a SELinux permission, you can receive the “Could not ping” error, while trying to add a new device.
First we need to verify that fping is working. SSH into the LibreNMS server and try pinging an address.
fping 192.168.1.20
There was an issue with fping working if ipv6 was disabled. If fping is not working at all, check out this thread.
If you get an alive or unreachable message, then we know fping is working and can move on to the next stage of troubleshooting.
If you are using SELinux, then there is a good chance the problems has to do with that. You can try rerunning all the SELinux commands from the install guide. Note that it has a specific portion for fping.
https://docs.librenms.org/Installation/Install-LibreNMS/#selinux
If it is still not working, we can take a look at the issue with the audit2why command and feed in the audit log.
audit2why < /var/log/audit/audit.log
Here is some example output.
[root@librenms ~]#
type=AVC msg=audit(1676192040.183:404404): avc: denied { bind } for pid=128555 comm="fping" lport=1 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=rawip_socket permissive=0
Was caused by:
Missing type enforcement (TE) allow rule.
[root@librenms ~]#
Another, perhaps more effective way to check the log is to follow it using the “tail -f” command.
tail -f /var/log/audit/audit.log | grep denied
And then in the web browser, try adding a new device. If SELinux is blocking it, it should throw a denied entry.
Example output
type=AVC msg=audit(1676192040.183:404404): avc: denied { bind } for pid=128555 comm="fping" lport=1 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=rawip_socket permissive=0
Now we have verified that the issue is SELinux permissions related. We can create a module to allow it.
audit2allow -a -M fping_http < /var/log/audit/audit.log
And apply the module with
semodule -i fping_http.pp
You may need to do this a couple times. Check the audit log again to see if anything new shows up. Notice the slight difference in this error compared to the above error.
# tail -f /var/log/audit/audit.log | grep denied
type=AVC msg=audit(1676192613.121:404409): avc: denied { node_bind } for pid=153257 comm="fping" scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=rawip_socket permissive=0
We’ll create a new module for this and apply it
audit2allow -a -M node_http < /var/log/audit/audit.log semodule -i node_http.pp
Not sure that is the best way to fix the problem. But it appears that SELinux is keeping Apache “httpd” from running fping which is why we need to create and load the modules.
Renaming of 192.168.1.20 failed . Does your web server have permission to modify the rrd files?
First thing to check is verify that the IP address is not already being monitored.
If you are getting the above error while trying to rename a device in LibreNMS, you may need to rerun some of the SELinux commands from the installation.
SSH into the server and run
restorecon -RFvv /opt/librenms
Now try renaming the device. Note that you can’t rename the device if the name/ip to a name that is being used by a different device.
If you continue to have issues, check the permissions from the installation guide (Official guide here)
You can also check for SELinux errors with
audit2why < /var/log/audit/audit.log
More SELinux info here
Looks like the issue has something to do with net-ssh. There are some other similar errors people were having.
https://github.com/ytti/oxidized/issues/2642
https://github.com/ytti/oxidized/pull/2570
The easy way to resolve the issue is to install oxidized using git.
Make sure rake is installed
sudo dnf install rake or sudo apt install rake
Steps were copied from here. https://github.com/ytti/oxidized#build-from-git
You should be able to copy and paste these commands in the users home directory.
git clone https://github.com/ytti/oxidized.git cd oxidized/ gem install bundler rake install
After it is installed, restart the service
systemctl restart oxidized
Or continue on installing and with LibreNMS
There are some differences on setting up RRDReST on CentOS 8, Almalinux 9 vs CentOS 7
If you are setting this up to use with LibreNMS and Grafana, check out the rest of the this article. https://www.incredigeek.com/home/setting-up-grafana-on-librenms/
All the docker commands have been swapped out for podman.
Podman is default on CentOS 8 and later and is for the most part a drop in replacement for Docker.
sudo yum install -y podman podman-compose sudo systemctl enable podman
Create docker compose file with the following options
vi podman-compose.yml
Change the TZ to your time zone. If you have issues with the graphs, most likely something is off with the time zone between this container and Grafana/LibreNMS server
version: "3.5"
services:
rrdrest:
image: michaelwadman/rrdrest:latest
container_name: rrdrest
restart: always
volumes:
- "/opt/librenms/rrd:/opt/librenms/rrd:Z"
environment:
- TZ=America/Denver
Note that the :Z is needed for SELinux to allow RRDReST to access the sub folders. AKA. the rrd files.
Save the file and start and setup the container with
sudo podman-compose up -d
You will need your docker container IP address to setup the connection in Grafana
sudo docker exec -it rrdrest ip addr | grep eth0
The “restart: always” option does not appear to work on systems with podman. We can create a systemd service instead.
Use the following command to automatically create a
podman generate systemd rrdrest
Copy the contents to a new file in
/etc/systemd/system/rrdrest.service
Enable the new service with
systemctl enable rrdrest
Congratulations. RRDReST is now setup and running.
With support for DES being dropped, you may be faced with having to upgrade device settings to AES. In this post we’ll explore changing the settings in LibreNMS for all Mikrotik devices and then touch on making changes to a group of Mikrotik devices.
In LibreNMS, we can go to Device -> Device Settings (Gear on the right hand side) -> SNMP, to set the SNMP settings for that device.
Since this would get rather boring to change on multiple devices, and these settings are all in a MySQL database, we can skip using the mouse and use a few MySQL commands to update multiple devices at once.
Log into the LibreNMS server over ssh and then connect to the MySQL database
mysql -u librenms -p librenms
First we can get a list of all the devices (Mikrotik routers in this example) and show the hostname with the SNMP authentication and cryptography algorithms.
select hostname,authalgo,cryptoalgo from devices where os="routeros";
Now if we want to update the cryptography settings for all of our Mikorotik devices, we can do the following.
update devices cryptoalgo set cryptoalgo="AES" where os="routeros";
This will set all of the devices to use AES for the cryptography algorithm.
We can also change the authentication algorithm to SHA with this
update devices authalgo set authalgo="SHA" where os="routeros";
The bottom “script” can be used for changing SNMP settings on multiple Mikrotik devices.
Create a mikrotik.lst file with all the IP addresses of all the devices you need to update. Can you use the above MySQL commands to get a list from LibreNMS.
Change the following options in the script
for ip in `cat mikrotik.lst`
do
echo $ip
timeout 15 sshpass -p 'routerpassword' ssh -o StrictHostKeyChecking=no admin@${ip} -p1022 '/snmp community set addresses=192.168.0.0/16 authentication-protocol=SHA1 authentication-password=authpassword encryption-protocol=AES encryption-password=encryptionpassword security=private read-access=yes write-access=no SNMPname'
done
Copy and paste the above “code” in a shell script file.
nano mikrotik.sh chmod +x mikrotik.sh ./mikrotik.sh
The script should run and update all the SNMP settings on all the devices in mikrotik.lst
In the post, we’ll be adding a custom OID for a Ubiquiti Solar Charge Controller.
Check out the following post if you are trying to add a Ubiquiti Solar Charge controller graph to LibreNMS. Otherwise you may need to do some googling around looking for the OID.
Go to your device -> Settings(Little Gear) -> Custom OID -> +Add New OID
Couple notes about the information.
– There needs to be a 0 after the end of the OID.
– Data Type needs to be Gauge, Not Counter. A gauge can go up and down. A counter just counts up.
Hit “Test OID” and you should get a little notification saying it got a value for the OID. In this case 25572.
Now Hit “Save OID”
Now LibreNMS should start graphing our Custom OID. You may need to give it a minute to start showing data.
The graphs show up under Graphs -> Custom OID
Thanks to the guys who put together the information at the following links.
https://wadman.co.nz/2021/01/02/Viewing-LibreNMS-data-in-Grafana/
https://www.reddit.com/r/LibreNMS/comments/ojc8cc/how_to_almost_natively_integrate_librenms_and/
I ran into some issues trying to get this to work. So here are some of my notes. I already had a LibreNMS installation set up.
NOTE FOR CENTOS 8, ALMALINUX 8 and 9
The steps for installing RRDReST are slightly different. Check out the following post.
https://www.incredigeek.com/home/setting-up-rrdrest-on-centos-8-or-almalinux-9/
I had issues installing RRDReST. I am guessing it had to do with it accessing files. I was able to install it in a docker container.
sudo yum install -y docker docker-compose
sudo systemctl enable docker
Create docker compose file with the following options
vi docker-compose.yml
Change the TZ to your time zone. If you have issues with the graphs, most likely something is off with the time zone between this container and Grafana/LibreNMS server
version: "3.5"
services:
rrdrest:
image: michaelwadman/rrdrest:latest
container_name: rrdrest
restart: always
volumes:
- "/opt/librenms:/opt/librenms"
environment:
- TZ=America/Denver
Save the file and start and setup the container with
sudo docker-compose up -d
You will need your docker container IP address to setup the connection in Grafana
sudo docker exec -it rrdrest ip addr | grep eth0
Congratulations. You should now have a RRDReST docker container that will auto start on system boot and has the correct time zone.
Basic steps are as follows
There is not anything special with installing Grafana on the same server as LibreNMS. You can follow the official guide to install it
https://grafana.com/docs/grafana/latest/installation/
After Grafana is installed, install the JSON API data source. You can do this using the grafana-cli
grafana-cli plugins install marcusolsson-json-datasource
A note on SSL/TLS certificates. If you have an SSL certificate for LibreNMS, you can use it for grafana. If you run into issues, try copying the cert (fullchain.pem, privkey.pem) to /etc/grafana/
You’ll most likely need to change owner
sudo chown root:grafana /etc/grafana/*.pem
And maybe the file permissions.
sudo chmod 640 /etc/grafana/*.pem
This is fairly straight forward.
grafana-cli plugins install marcusolsson-json-datasource
In Grafana, go to Configuration -> Data Sources -> Add data source
In Grafana, go to Configuration -> Data Sources -> Add data source
You should now be able to view your dashboard and use the drop down menus to select devices
There were a couple of issues I ran into while trying to get everything working together.
Issue: When trying to run RRDReST with uvicorn, I was never able to access the rrd files, even the test rrd files that are included when installing RRDReST. I am guessing it is either a permisions issue, or something unable to access the files.
Work around: Install RRDReST via Docker container.
Issue: Get a “JSON API: Bad Request” when trying to set up the LibreNMS API Data Source in Grafana.
Work around: Install a valid SSL Certificate and set up a DNS record so you can access LibreNMS at librenms.yourdomain.com.
More info: I would assume that “Skip TLS Verify” would work with or without a valid certificate, but it would not work for me. There are potentially some other options with modifying how Nginx or Apache is set up that would get this working. If you setup Grafana to use a SSL certificate, you may need to copy the certificate files (fullchain.pem, privkey.pem) to /etc/grafana/ and run “chown root:grafana *.pem” to let grafana have access to the files.
Running the ./validate.php script returns the following error
[FAIL] Python3 module issue found: 'Required packages: ['PyMySQL!=1.0.0', 'python-dotenv', 'redis>=3.0', 'setuptools', 'psutil>=5.6.0', 'command_runner>=1.3.0']
Package not found: The 'command_runner>=1.3.0' distribution was not found and is required by the application
'
[FIX]:
pip3 install -r /opt/librenms/requirements.txt
Running the [FIX] throws an error saying gcc failed with exit status 1.
Fortunately this issue is easy to resolve.
First we need to install python3-devel
sudo yum install python3-devel
Next, as the librenms user, run the pip command to install the requirements.
pip3 install --user -U -r /opt/librenms/requirements.txt
Run ./validate.php to verify that everything is working.
