The problem: Linux servers have been configured to send their local syslogs to LibreNMS, but are not showing up under the LibreNMS -> DEVICE -> Logs-> Syslog
After a bit of troubleshooting, found that the issue is the hostname being sent with the logs is different than what LibreNMS has for the device. It appears that some Linux distributions will or can use an abbreviated system hostname. There is a section in the LibreNMS docs about this
In this example, the server is already using Let’s Encrypt to create the certificate for a LibreNMS server. So all we are doing is copying the certificate to a Grafana directory, putting the correct permissions on it, and updating the Grafana config file to use the certificate.
Steps
Copy Certificate to Grafana Directory
Configure Grafana Config File
Automate Certificate Copy to Grafana Directory
Copy Certificate files
In the following commands, change librenms.incredigeek.com to the directory that Let’s Encrypt is using for your fully qualified domain name (FQDN). Usually it is just your FQDN, but could also have -0001 or something appended to the end.
cp -f /etc/letsencrypt/live/librenms.incredigeek.com/privkey.pem
/etc/grafana/
cp -f /etc/letsencrypt/live/librenms.incredigeek.com/fullchain.pem /etc/grafana/
chown root:grafana /etc/grafana/*.pem
chmod 640 /etc/grafana/*.pem Enable grafana on system bootup
In the above, we are copying the privkey.pem and fullchain.pem to /etc/grafana. We are then setting the correct owner/permissions on the files so that the Grafana service can read the certificate.
Configure Grafana Config File
This is super easy. Open up the Grafana config file in /etc/grafana.ini
vi /etc/grafana.ini
Find the following variables and configure them like so
You should now have a working SSL certificate for the site.
Automate Certificate Copy
Let’s Encrypt certificates need to be updated frequently. This means that we should automate the above steps to avoid any down time. After all, a monitoring tool with down time defeats the purpose of monitoring.
We’ll need to create a root crontab
sudo crontab -e
Add the following changing out the FQDN to your FQDN.
This is set to run once a month. Change if desired. Also change out librenms.incredigeek.com with your FQDN.
Note about domain name and IP addresses. Let’s Encrypt will not create a certificate for an IP address. You should be using a domain name instead (i.e. networkmonitoring.yourdomain.com) If the certificate is installed, and you access it via the IP address, you will receive a HTTPS error in your browser.
LibreNMS uses fping to check if devices are up or not. So if something is broken with fping, say a SELinux permission, you can receive the “Could not ping” error, while trying to add a new device.
LibreNMS unable to ping device
First we need to verify that fping is working. SSH into the LibreNMS server and try pinging an address.
fping 192.168.1.20
There was an issue with fping working if ipv6 was disabled. If fping is not working at all, check out this thread.
If you get an alive or unreachable message, then we know fping is working and can move on to the next stage of troubleshooting.
If you are using SELinux, then there is a good chance the problems has to do with that. You can try rerunning all the SELinux commands from the install guide. Note that it has a specific portion for fping.
Now we have verified that the issue is SELinux permissions related. We can create a module to allow it.
audit2allow -a -M fping_http < /var/log/audit/audit.log
And apply the module with
semodule -i fping_http.pp
You may need to do this a couple times. Check the audit log again to see if anything new shows up. Notice the slight difference in this error compared to the above error.
audit2allow -a -M node_http < /var/log/audit/audit.log
semodule -i node_http.pp
Not sure that is the best way to fix the problem. But it appears that SELinux is keeping Apache “httpd” from running fping which is why we need to create and load the modules.
Renaming of 192.168.1.20 failed . Does your web server have permission to modify the rrd files?
First thing to check is verify that the IP address is not already being monitored.
If you are getting the above error while trying to rename a device in LibreNMS, you may need to rerun some of the SELinux commands from the installation.
SSH into the server and run
restorecon -RFvv /opt/librenms
Now try renaming the device. Note that you can’t rename the device if the name/ip to a name that is being used by a different device.
If you continue to have issues, check the permissions from the installation guide (Official guide here)
Create docker compose file with the following options
vi podman-compose.yml
Change the TZ to your time zone. If you have issues with the graphs, most likely something is off with the time zone between this container and Grafana/LibreNMS server
With support for DES being dropped, you may be faced with having to upgrade device settings to AES. In this post we’ll explore changing the settings in LibreNMS for all Mikrotik devices and then touch on making changes to a group of Mikrotik devices.
Upgrading SNMP Settings for Devices in LibreNMS
In LibreNMS, we can go to Device -> Device Settings (Gear on the right hand side) -> SNMP, to set the SNMP settings for that device.
Since this would get rather boring to change on multiple devices, and these settings are all in a MySQL database, we can skip using the mouse and use a few MySQL commands to update multiple devices at once.
Log into the LibreNMS server over ssh and then connect to the MySQL database
mysql -u librenms -p librenms
First we can get a list of all the devices (Mikrotik routers in this example) and show the hostname with the SNMP authentication and cryptography algorithms.
select hostname,authalgo,cryptoalgo from devices where os="routeros";
Now if we want to update the cryptography settings for all of our Mikorotik devices, we can do the following.
update devices cryptoalgo set cryptoalgo="AES" where os="routeros";
This will set all of the devices to use AES for the cryptography algorithm.
We can also change the authentication algorithm to SHA with this
update devices authalgo set authalgo="SHA" where os="routeros";
LibreNMS update device SNMP settings
Bulk updating of Network Devices
The bottom “script” can be used for changing SNMP settings on multiple Mikrotik devices.
Create a mikrotik.lst file with all the IP addresses of all the devices you need to update. Can you use the above MySQL commands to get a list from LibreNMS.
Change the following options in the script
routerpassword to the Mikrotik password
admin to your username
encryptionpassword to your SNMP encryption password
authpassword to your authentication password
addresses=192.168.0.0/16 to the list of IP addresses that should be able to access SNMP info on the mikrotik device. AKA your LibreNMS server.
SNMPname to your SNMP username
for ip in `cat mikrotik.lst`
do
echo $ip
timeout 15 sshpass -p 'routerpassword' ssh -o StrictHostKeyChecking=no admin@${ip} -p1022 '/snmp community set addresses=192.168.0.0/16 authentication-protocol=SHA1 authentication-password=authpassword encryption-protocol=AES encryption-password=encryptionpassword security=private read-access=yes write-access=no SNMPname'
done
Copy and paste the above “code” in a shell script file.