Recovering LibreNMS from crashed XenServer VM

Had a LibreNMS instance crash, or the VM crashed, not bootable anymore.  Was able to boot it up on a CentOS iso with rescue mode, which gave me access to the files.  So the idea is to manually copy off the LibreNMS files and LibreNMS database and import them to a new LibreNMS instance

Issues

  • Need to access the the system files
  • MySQL doesn’t start in a chroot environment, so no way to do a mysqldump —  (Has to do with systemv or something)

Steps

  1. Create new LibreNMS VM with a new instance of LibreNMS installed
  2. Gain access to the crashed system
  3. Copy over LibreNMS MySQL databases to new LibreNMS instance
  4. Copy over LibreNMS files “/opt/librenms” to new LibreNMS instance
  5. Clean up.  Set users on directories, check SELinux etc.

1. Installing New LibreNMS VM

  • Guide for Ubuntu/Debian distro’s here
  • Guide for for Fedora/CentOS/RedHat here

2. Gaining access to crashed VM

There can be a couple of ways to gain access to a crashed XenServer VM.  One of the easiest ways is to boot up in recovery mode and go through the installers rescue mode.  Guide to boot up XenServer VM in recovery mode here.

The installer rescue mode should detect the OS and mount everything.  If not you should be able to mount the root partition manually.

Once booted up, you’ll need to enable network access if your going to use sftp or scp to copy files.  There are a few different ways to do this

  • Run “dhclient” to pull an address via DHCP
  • Set a static IP address
    • Guide for Ubuntu/Debian distro’s here
    • Guide for fedora/CentOS/RedHat here

3.Copy LibreNMS Mysql Database

Backup the LibreNMS MySQL database directory

tar czvf librenms_mysql.tgz /var/lib/mysql

Use scp or sftp to copy it to the new LibreNMS instance

scp librenms_mysql.tgz user@new_LibreNMS_ip

Now on the new LibreNMS instance we need to run the following few commands

systemctl stop mariadb
rm -rf /var/lib/mysql/*
tar xzvf librenms_mysql.tgz -C /
chown -R mysql:mysql /var/lib/mysql

4. Copy LibreNMS directory

Backup the LibreNMS directory

tar czvf librenms.tgz /opt/librenms

Use scp or sftp to copy it to the new LibreNMS instance

scp librenms.tgz user@new_LibreNMS_ip

Now on the new LibreNMS instance we need to run the following few commands

rm -rf /opt/librenms/*
tar xzvf librenms.tgz -C /
chown -R librenms:librenms /opt/librenms

5. Clean up

Disable SELinux if you have not already.  Guide here

Restart apache, and start mysql.  If your on Ubuntu, the services are named apache2 and mysql

systemctl restart httpd
systemctl restart mariadb

That should get it working, if not try a reboot.

Special notes

The whole MySQL directory needs to be copied, there are innodb files that will keep MySQL from starting if they are not copied.

There is some good info here
https://stackoverflow.com/questions/1795176/how-to-change-mysql-data-directory

Install LibreNMS on CentOS

Install prerequisites

yum install -y epel-release 
rpm -Uvh https://mirror.webtatic.com/yum/el7/webtatic-release.rpm
yum install -y policycoreutils-python cronie fping git httpd ImageMagick jwhois mariadb mariadb-server mtr MySQL-python net-snmp net-snmp-utils nmap php71w php71w-cli php71w-common php71w-curl php71w-gd php71w-mcrypt php71w-mysql php71w-process php71w-snmp php71w-xml php71w-zip python-memcached rrdtool

Setup LibreNMS user

useradd librenms -d /opt/librenms -M -r
usermod -a -G librenms apache

Install LibreNMS

cd /opt
git clone https://github.com/librenms/librenms.git librenms

Configure MySQL

Start MySQL

systemctl start mariadb
mysql -u root

Create the Database and database user. Change password to a different password

CREATE DATABASE librenms CHARACTER SET utf8 COLLATE utf8_unicode_ci;
CREATE USER 'librenms'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON librenms.* TO 'librenms'@'localhost';
FLUSH PRIVILEGES;
exit

Disable MySQL strict mode

vi /etc/my.cnf

Add the following in the [mysqld] section

innodb_file_per_table=1
sql-mode=""
lower_case_table_names=0

Enable and restart mariadb i.e.(MySQL)

systemctl enable mariadb
systemctl restart mariadb

Configure PHP.  Set Timezone

You can go here to figure out which time zone you need.

vi  /etc/php.ini

Find the [Date] Section and change the timezone

Example:

date.timezone = America/New_York

Configure Apache

vi /etc/httpd/conf.d/librenms.conf

Add the following, change the server name.

<VirtualHost *:80>
  DocumentRoot /opt/librenms/html/
  ServerName  librenms.example.com

  AllowEncodedSlashes NoDecode
  <Directory "/opt/librenms/html/">
    Require all granted
    AllowOverride All
    Options FollowSymLinks MultiViews
  </Directory>
</VirtualHost>

Enable and restart apache

systemctl enable httpd
systemctl restart httpd

Allow fping

cd /opt/librenms
vi http_fping.tt

Add the following

module http_fping 1.0;

require {
type httpd_t;
class capability net_raw;
class rawip_socket { getopt create setopt write read };
}

#============= httpd_t ==============
allow httpd_t self:capability net_raw;
allow httpd_t self:rawip_socket { getopt create setopt write read };

Configure SELinux

You can disable SELinux completely in the /etc/selinux/config file or run these commands to make LibreNMS and SELinux play together.

semanage fcontext -a -t httpd_sys_content_t '/opt/librenms/logs(/.*)?'
semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/logs(/.*)?'
restorecon -RFvv /opt/librenms/logs/
semanage fcontext -a -t httpd_sys_content_t '/opt/librenms/rrd(/.*)?'
semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/rrd(/.*)?'
restorecon -RFvv /opt/librenms/rrd/
setsebool -P httpd_can_sendmail=1
checkmodule -M -m -o http_fping.mod http_fping.tt
semodule_package -o http_fping.pp -m http_fping.mod
semodule -i http_fping.pp

Configure Firewall

firewall-cmd --zone public --add-service http
firewall-cmd --permanent --zone public --add-service http
firewall-cmd --zone public --add-service https
firewall-cmd --permanent --zone public --add-service https

Configure snmpd

cp /opt/librenms/snmpd.conf.example /etc/snmp/snmpd.conf

Edit the snmpd file and change RANDOMSTRINGGOESHERE to your community string

vi /etc/snmp/snmpd.conf

Enable and restart snmpd

curl -o /usr/bin/distro https://raw.githubusercontent.com/librenms/librenms-agent/master/snmp/distro
chmod +x /usr/bin/distro
systemctl enable snmpd
systemctl restart snmpd

Setup Cron Job and Cycle logs

cp /opt/librenms/librenms.nonroot.cron /etc/cron.d/librenms
cp /opt/librenms/misc/librenms.logrotate /etc/logrotate.d/librenms

Set Permisions

chown -R librenms:librenms /opt/librenms
setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs
setfacl -R -m g::rwx /opt/librenms/rrd /opt/librenms/logs

Finish Install

Finish the install by

http://your-server-ip/install.php

Change timezone in CentOS, Fedora, RedHat

The file /etc/localtime is a symbolic link to the timezone.  All the timezones are listed in /usr/share/zoneinfo/

Replace America/New_York with the appropriate timezone.

ln -sf /usr/share/zoneinfo/America/New_York /etc/localtime

You can view the current time zone with the following command

timedatectl

or with date

date +"%Z %z"

Install SSH Server on Linux (Debian, Ubuntu, Fedora, CentOS, RedHat)

Debian / Ubuntu

sudo apt-get install -y openssh-server

RPM based Distros, Fedora / CentOS / RedHat

sudo dnf install -y openssh-server

or use yum

sudo yum install -y openssh-server

Start ssh service

sudo systemctl start sshd

By default the SSH service should start when the system starts, but if not try the following command to enable the service on boot up.

Debian / Ubuntu

systemctl enable ssh

Fedora, CentOS, RedHat

systemctl enable sshd

Change SSH port

Not necessary, but it is a good idea to change the default ssh port.  To change the port edit the sshd file.

vi /etc/ssh/sshd_config

If you change the port, you’ll need to allow it in the firewall (firewalld, iptables) and if SELinux is enabled, semanage.

Allow SSH access from a specific host using hosts.allow and hosts.deny on Linux

This is just a quick write on the hosts.allow and deny files.  You can lookup “spawn” and/or “twist” for some advanced usage.

 

So to limit an IP address, or a IP range access to SSH, do the following

Deny all incoming request for SSH

Edit the “hosts.deny” file

vi /etc/hosts.deny

add the following line

sshd : ALL

Now edit “hosts.allow” and allow the client IP, or IP range to access SSH

vi /etc/hosts.allow

add the following line to allow a single IP

sshd : 192.168.1.182

If you want to allow the whole subnet, then replace the above line with this one

sshd : 192.168.1.

hosts.allow overrides hosts.deny.  So you deny everything and then allow exceptions.

How to Install ZenPack on Zenoss 5.1

To install a ZenPack on Zenoss 5.1 and later, you are going to need to stop the zenoss services, restart a couple services that are needed to install the ZenPack, install the ZenPack, and then restart the Zenoss service.

Stop Zenoss.Core :

serviced service stop 

Make sure the service is stopped by running the following command.  When it reports back the Zenoss is Stopped, then continue.

 if ( serviced service status Zenoss.Core | awk '{print $3}' | grep -q Stopped) ; then echo "Zenoss is not running!" ; else echo "Zenoss is not Stopped!" ; fi

Or you can run this command, just make sure it says it is stopped.

serviced service status zenoss.core

Create a Snapshot

serviced service snapshot Zenoss.core

Start the following three services

serviced service start Infrastructure
zeneventserver
Zope


serviced service start Infrastructure
serviced service start zeneventserver
serviced service start Zope

Change directory to tmp, make sure you put the zenpack in /tmp

cd /tmp

Install the Zenpack

serviced service run zope zenpack-manager install *.egg

Example:

serviced service run zope zenpack-manager install ZenPacks.zenoss.MikroTik-1.1.1.egg

Restart the Zenoss Service

serviced service restart zenoss.core

Log into Zenoss and check it.

How To change the Hostname in CentOS 7

Check current hostname with the hostname command.

hostname

Change hostname with hostnamectl

hostnamectl set-hostname newhostname

 

Example:

Changing hostname from “localhost” to “newhost”.

[root@localhost ~]# hostname
localhost
[root@localhost ~]# hostnamectl set-hostname newhost
[root@localhost ~]# hostname
newhost
[root@localhost ~]# 

 

 

How To Enable/Disable SELinux

Disable SELinux

You can manually edit the SELinux config file in /etc/selinux/config and change the variable SELINUX=enforcing to disabled

vi /etc/selinux/config
...
SELINUX=disabled
...

or you can use this little command

sed -i.bak -e 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config

Enable SELinux

You can enable SELinux manually the same as above but set SELINUX=disabled to SELINUX=enforcing

vi /etc/selinux/config
...
SELINUX=enforcing
...

or

sed -i.bak -e 's/^SELINUX=.*/SELINUX=enforcing/g' /etc/selinux/config

How to View Installed Programs in linux

Debian Based Distros

The below command should work for Debain, Ubuntu, and the Raspberry Pi Raspbian.

dpkg --get-selections

 

RPM Based Distros

The following works on Fedora, CentOS, ReHat.

rpm -qa

 

The above commands return all of the packages installed on a system.  If you want to look for a specific program or package you can use grep to filter the results.

rpm -qa | grep program

or

dpkg --get-selections | grep program