LibreNMS enable RRDCached on CentOS 7

Commands from https://docs.librenms.org/Extensions/RRDCached/

Create System Service

sudo vi /etc/systemd/system/rrdcached.service
[Unit]
Description=Data caching daemon for rrdtool
After=network.service
[Service]
Type=forking
PIDFile=/run/rrdcached.pid
ExecStart=/usr/bin/rrdcached -w 1800 -z 1800 -f 3600 -s librenms -U librenms -G librenms -B -R -j /var/tmp -l unix:/run/rrdcached.sock -t 4 -F -b /opt/librenms/rrd/
[Install]
WantedBy=default.target

Save with esc + :wq

Enable rrdcached service on boot up and start the service now

systemctl enable --now rrdcached.service

Enable RRDcached in librenms config

vi /opt/librenms/config.php

add/edit the following line. If it is commented out, uncomment it

$config['rrdcached'] = "unix:/run/rrdcached.sock";

Save with esc + :wq

Check the disk IO after a little bit to see if it went down.

LibreNMS manually clean up MySQL Syslog Database

Adding the following option to the config.php file is supposed to delete anything over 30 days.

$config['syslog_purge']                                 = 30;

You can also manually delete the entries out of the MySQL database by logging into MySQL, selecting the librenms database, and running the command below.

The command deletes all entries older than 12/9/2018 at 08:00. Change the date and time as needed.

DELETE FROM syslog WHERE timestamp < '2018-12-9 08:00:00';

The following link has some more clean up options.
https://docs.librenms.org/Support/Cleanup-options/

LibreNMS bulk delete

There is a php script in /opt/librenms/ that lets you delete a host from the command line.

sudo /opt/librenms/delhost.php 192.168.1.20

Replace 192.168.1.20 with the hostname/ip address of the host you want to delete.

Delete Multiple Hosts

First you’ll need to get a list of devices you want to remove.  You can do this by viewing the devices in the LibreNMS MySQL database;

Example:

$ mysql -u librenms -p librenms
MariaDB [librenms]> select hostname from devices;
+----------------------------------------+
| hostname |
+----------------------------------------+
| 192.168.88.1 |
| 192.168.1.20 |
| 192.168.1.12 |
| 192.168.88.5 |
4 rows in set (0.00 sec)
MariaDB [librenms]> exit

Put all the IP addresses you want to remove into a file and run the following for loop.  Replace “remove_ip.lst” with the name of your ip list file.

for i in `cat ~/remove_ip.lst`; do sudo /opt/librenms/delhost.php $i; done

LibreNMS Error: Missing dependencies! Run the following command to fix:

Received this error while trying to access LibreNMS in web browser.

Error: Missing dependencies! Run the following command to fix:

./scripts/composer_wrapper.php install –no-dev

 

Tried sshing into the LibreNMS server and running the script, but that didn’t fix the issue.

Ran ./validate.php and it came up with the some of the following errors.

[FAIL] Missing PHP extension: mbstring
[FIX] Please install mbstring
[FAIL] The poller (localhost.localdomain) has not completed within the last 5 minutes, check the cron job.
[WARN] Your install is over 24 hours out of date, last update: Thu, 7 May 2018 19:18:35 +0000
[FIX] Make sure your daily.sh cron is running and run ./daily.sh by hand to see if there are any errors.

Installed mbstring.  You may need to change the number if yum is complaining about package conflicts.  Use “yum search mbstring” to show all the versions available.

yum install php71w-mbstring

Ran ./validate.php again and it wanted this script run

./scripts/composer_wrapper.php install --no-dev

Restart Apache

systemctl restart httpd

Add Self Signed SSL certificate to LibreNMS in CentOS

Install mod_ssl

yum install mod_ssl -y

Create Directory for SSL key.

mkdir /etc/ssl/key
chmod 700 /etc/ssl/key

Create certificate.

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/pki/tls/private/localhost.key -out /etc/pki/tls/certs/localhost.crt

Fill out the info or what is applicable.

Now edit the LibreNMS Apache config file /etc/httpd/conf.d/librenms.conf

All you have to do is add the following three lines under the VirtualHost and change *:80 to *:443.

SSLEngine on
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSL CertificateKeyFile /etc/pki/tls/private/localhost.key

So when your finished the file should look like this.

<VirtualHost *:443>
 DocumentRoot /opt/librenms/html/
 ServerName server_hostname_or_IP
 SSLEngine on
 SSLCertificateFile /etc/pki/tls/certs/localhost.crt
 SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
 CustomLog /opt/librenms/logs/access_log combined
 ErrorLog /opt/librenms/logs/error_log
 AllowEncodedSlashes NoDecode
 <Directory "/opt/librenms/html/">
 Require all granted
 AllowOverride All
 Options FollowSymLinks MultiViews
 </Directory>
</VirtualHost>

Don’t forget to allow https/port 443 traffic through the firewall.  Guide here

If you have any issues, you may need to chmod the key and crt file.

chmod 644 /etc/pki/tls/certs/localhost.crt
chmod 644 /etc/pki/tls/private/localhost.key

You should now be able to access LibreNMS using https.  Note, you’ll need to allow an exception in your browser for your self signed certificate.

https://LibreNMS_IP_Address

Recovering LibreNMS from crashed XenServer VM

Had a LibreNMS instance crash, or the VM crashed, not bootable anymore.  Was able to boot it up on a CentOS iso with rescue mode, which gave me access to the files.  So the idea is to manually copy off the LibreNMS files and LibreNMS database and import them to a new LibreNMS instance

Issues

  • Need to access the the system files
  • MySQL doesn’t start in a chroot environment, so no way to do a mysqldump —  (Has to do with systemv or something)

Steps

  1. Create new LibreNMS VM with a new instance of LibreNMS installed
  2. Gain access to the crashed system
  3. Copy over LibreNMS MySQL databases to new LibreNMS instance
  4. Copy over LibreNMS files “/opt/librenms” to new LibreNMS instance
  5. Clean up.  Set users on directories, check SELinux etc.

1. Installing New LibreNMS VM

  • Guide for Ubuntu/Debian distro’s here
  • Guide for for Fedora/CentOS/RedHat here

2. Gaining access to crashed VM

There can be a couple of ways to gain access to a crashed XenServer VM.  One of the easiest ways is to boot up in recovery mode and go through the installers rescue mode.  Guide to boot up XenServer VM in recovery mode here.

The installer rescue mode should detect the OS and mount everything.  If not you should be able to mount the root partition manually.

Once booted up, you’ll need to enable network access if your going to use sftp or scp to copy files.  There are a few different ways to do this

  • Run “dhclient” to pull an address via DHCP
  • Set a static IP address
    • Guide for Ubuntu/Debian distro’s here
    • Guide for fedora/CentOS/RedHat here

3.Copy LibreNMS Mysql Database

Backup the LibreNMS MySQL database directory

tar czvf librenms_mysql.tgz /var/lib/mysql

Use scp or sftp to copy it to the new LibreNMS instance

scp librenms_mysql.tgz user@new_LibreNMS_ip

Now on the new LibreNMS instance we need to run the following few commands

systemctl stop mariadb
rm -rf /var/lib/mysql/*
tar xzvf librenms_mysql.tgz -C /
chown -R mysql:mysql /var/lib/mysql

4. Copy LibreNMS directory

Backup the LibreNMS directory

tar czvf librenms.tgz /opt/librenms

Use scp or sftp to copy it to the new LibreNMS instance

scp librenms.tgz user@new_LibreNMS_ip

Now on the new LibreNMS instance we need to run the following few commands

rm -rf /opt/librenms/*
tar xzvf librenms.tgz -C /
chown -R librenms:librenms /opt/librenms

5. Clean up

Disable SELinux if you have not already.  Guide here

Restart apache, and start mysql.  If your on Ubuntu, the services are named apache2 and mysql

systemctl restart httpd
systemctl restart mariadb

That should get it working, if not try a reboot.

Special notes

The whole MySQL directory needs to be copied, there are innodb files that will keep MySQL from starting if they are not copied.

There is some good info here
https://stackoverflow.com/questions/1795176/how-to-change-mysql-data-directory

Install LibreNMS on CentOS

Install prerequisites

yum install -y epel-release 
rpm -Uvh https://mirror.webtatic.com/yum/el7/webtatic-release.rpm
yum install -y policycoreutils-python cronie fping git httpd ImageMagick jwhois mariadb mariadb-server mtr MySQL-python net-snmp net-snmp-utils nmap php71w php71w-cli php71w-common php71w-curl php71w-gd php71w-mcrypt php71w-mysql php71w-process php71w-snmp php71w-xml php71w-zip python-memcached rrdtool

Setup LibreNMS user

useradd librenms -d /opt/librenms -M -r
usermod -a -G librenms apache

Install LibreNMS

cd /opt
git clone https://github.com/librenms/librenms.git librenms

Configure MySQL

Start MySQL

systemctl start mariadb
mysql -u root

Create the Database and database user. Change password to a different password

CREATE DATABASE librenms CHARACTER SET utf8 COLLATE utf8_unicode_ci;
CREATE USER 'librenms'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON librenms.* TO 'librenms'@'localhost';
FLUSH PRIVILEGES;
exit

Disable MySQL strict mode

vi /etc/my.cnf

Add the following in the [mysqld] section

innodb_file_per_table=1
sql-mode=""
lower_case_table_names=0

Enable and restart mariadb i.e.(MySQL)

systemctl enable mariadb
systemctl restart mariadb

Configure PHP.  Set Timezone

You can go here to figure out which time zone you need.

vi  /etc/php.ini

Find the [Date] Section and change the timezone

Example:

date.timezone = America/New_York

Configure Apache

vi /etc/httpd/conf.d/librenms.conf

Add the following, change the server name.

<VirtualHost *:80>
  DocumentRoot /opt/librenms/html/
  ServerName  librenms.example.com

  AllowEncodedSlashes NoDecode
  <Directory "/opt/librenms/html/">
    Require all granted
    AllowOverride All
    Options FollowSymLinks MultiViews
  </Directory>
</VirtualHost>

Enable and restart apache

systemctl enable httpd
systemctl restart httpd

Allow fping

cd /opt/librenms
vi http_fping.tt

Add the following

module http_fping 1.0;

require {
type httpd_t;
class capability net_raw;
class rawip_socket { getopt create setopt write read };
}

#============= httpd_t ==============
allow httpd_t self:capability net_raw;
allow httpd_t self:rawip_socket { getopt create setopt write read };

Configure SELinux

You can disable SELinux completely in the /etc/selinux/config file or run these commands to make LibreNMS and SELinux play together.

semanage fcontext -a -t httpd_sys_content_t '/opt/librenms/logs(/.*)?'
semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/logs(/.*)?'
restorecon -RFvv /opt/librenms/logs/
semanage fcontext -a -t httpd_sys_content_t '/opt/librenms/rrd(/.*)?'
semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/rrd(/.*)?'
restorecon -RFvv /opt/librenms/rrd/
setsebool -P httpd_can_sendmail=1
checkmodule -M -m -o http_fping.mod http_fping.tt
semodule_package -o http_fping.pp -m http_fping.mod
semodule -i http_fping.pp

Configure Firewall

firewall-cmd --zone public --add-service http
firewall-cmd --permanent --zone public --add-service http
firewall-cmd --zone public --add-service https
firewall-cmd --permanent --zone public --add-service https

Configure snmpd

cp /opt/librenms/snmpd.conf.example /etc/snmp/snmpd.conf

Edit the snmpd file and change RANDOMSTRINGGOESHERE to your community string

vi /etc/snmp/snmpd.conf

Enable and restart snmpd

curl -o /usr/bin/distro https://raw.githubusercontent.com/librenms/librenms-agent/master/snmp/distro
chmod +x /usr/bin/distro
systemctl enable snmpd
systemctl restart snmpd

Setup Cron Job and Cycle logs

cp /opt/librenms/librenms.nonroot.cron /etc/cron.d/librenms
cp /opt/librenms/misc/librenms.logrotate /etc/logrotate.d/librenms

Set Permisions

chown -R librenms:librenms /opt/librenms
setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs
setfacl -R -m g::rwx /opt/librenms/rrd /opt/librenms/logs

Finish Install

Finish the install by

http://your-server-ip/install.php

LibreNMS not able to ping device and can’t change device name

If your having issues in the LibreNMS web interface changing a devices IP address or you can’t add a new one because it says it can’t ping the device, it is probably because SELinux is causing issues.

As far as changing a devices ip address, it looks like SELinux is not allowing Apache write access to the /opt/librenms/rrd directory

Temporary fix is to run

setenforce 0

You can permenatly disable SElinux by opening up “/etc/selinux/config” and changing “SELINUX=enforcing” to “SELINUX=disabled” and rebooting.

 

Renaming of x.x.x.x failed . Does your web server have permission to modify the rrd files?

It looks like there may be a bug where you’ll try to change the IP of a device and get the above error.  It looks like the issue has something to do with the ip address.  If your trying to change the IP on a router that has multiple IP addresses, try a different address in LibreNMS.

 

Install LibreNMS on Ubuntu with HTTPS

The goal of this guide is to install LibreNMS on an Ubuntu Server with a self signed certificate.  Most of the steps are copied out of the LibreNMS Documentation found here.

Install required packages

sudo apt install apache2 composer fping git graphviz imagemagick libapache2-mod-php7.0 mariadb-client mariadb-server mtr-tiny nmap php7.0-cli php7.0-curl php7.0-gd php7.0-json php7.0-mcrypt php7.0-mysql php7.0-snmp php7.0-xml php7.0-zip python-memcache python-mysqldb rrdtool snmp snmpd whois

Create LibreNMS user

sudo useradd librenms -d /opt/librenms -M -r
sudo usermod -a -G librenms www-data

Install LibreNMS

cd /opt
sudo git clone https://github.com/librenms/librenms.git librenms

Configure MySQL

sudo systemctl restart mysql
sudo mysql -uroot -p

Run the following MySQL commands to create the LibreNMS user. Change password to your own password.

CREATE DATABASE librenms CHARACTER SET utf8 COLLATE utf8_unicode_ci;
CREATE USER 'librenms'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON librenms.* TO 'librenms'@'localhost';
FLUSH PRIVILEGES;
exit

Edit following file

sudo vi /etc/mysql/mariadb.conf.d/50-server.cnf

Add the following inside the [mysqld] section

innodb_file_per_table=1
sql-mode=""
lower_case_table_names=0

Restart MySQL

sudo systemctl restart mysql

Configure PHP

Edit the two files and set the time zone, date.timezone.  Example “America/New_York”

sudo vi /etc/php/7.0/apache2/php.ini
sudo vi /etc/php/7.0/cli/php.ini

Then run these commands

sudo a2enmod php7.0
sudo a2dismod mpm_event
sudo a2enmod mpm_prefork
sudo phpenmod mcrypt

Generate Self signed certificate

Enable ssl in apache

sudo a2enmod ssl

Generate Cert

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/localhost.key -out /etc/ssl/certs/localhost.crt

Configure Apache

Edit the following config file

sudo vi /etc/apache2/sites-available/librenms.conf

Add the following

<VirtualHost *:443>
 DocumentRoot /opt/librenms/html/
 ServerName librenms.example.com
 SSLEngine on
 SSLCertificateFile /etc/ssl/certs/localhost.crt
 SSLCertificateKeyFile /etc/ssl/private/localhost.key
 CustomLog /opt/librenms/logs/access_log combined
 ErrorLog /opt/librenms/logs/error_log
 AllowEncodedSlashes NoDecode
 <Directory "/opt/librenms/html/">
 Require all granted
 AllowOverride All
 Options FollowSymLinks MultiViews
 </Directory>
</VirtualHost>

Run the following commands

sudo a2ensite librenms.conf
sudo a2enmod rewrite
sudo systemctl restart apache2

Configure snmpd

sudo cp /opt/librenms/snmpd.conf.example /etc/snmp/snmpd.conf

Set the SNMP community string in the following file

sudo vi /etc/snmp/snmpd.conf

Then run these commands

sudo curl -o /usr/bin/distro https://raw.githubusercontent.com/librenms/librenms-agent/master/snmp/distro
sudo chmod +x /usr/bin/distro
sudo systemctl restart snmpd

Setup Crontab

sudo cp /opt/librenms/librenms.nonroot.cron /etc/cron.d/librenms

Copy logrotate config

sudo cp /opt/librenms/misc/librenms.logrotate /etc/logrotate.d/librenms

Set permissions

mkdir -p /opt/librenms/rrd /opt/librenms/logs
sudo chown -R librenms:librenms /opt/librenms
sudo setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs
sudo setfacl -R -m g::rwx /opt/librenms/rrd /opt/librenms/logs

Web Installer

Restart apache

sudo systemctl restart apache2

Finish the install by going to

https://your-server/install.php

Change “your-server” to your server’s ip address, or hostname.  Since we created a self signed certificate, you’ll need to accept the https error.

Validate

Back on the command line run the php validation script

sudo /opt/librenms/validate.php

Finally log into your new LibreNMS instance by going to

https://your-server

Change “your-server” to your server’s IP address or hostname.