Recovering LibreNMS from crashed XenServer VM

Had a LibreNMS instance crash, or the VM crashed, not bootable anymore.  Was able to boot it up on a CentOS iso with rescue mode, which gave me access to the files.  So the idea is to manually copy off the LibreNMS files and LibreNMS database and import them to a new LibreNMS instance

Issues

  • Need to access the the system files
  • MySQL doesn’t start in a chroot environment, so no way to do a mysqldump —  (Has to do with systemv or something)

Steps

  1. Create new LibreNMS VM with a new instance of LibreNMS installed
  2. Gain access to the crashed system
  3. Copy over LibreNMS MySQL databases to new LibreNMS instance
  4. Copy over LibreNMS files “/opt/librenms” to new LibreNMS instance
  5. Clean up.  Set users on directories, check SELinux etc.

1. Installing New LibreNMS VM

  • Guide for Ubuntu/Debian distro’s here
  • Guide for for Fedora/CentOS/RedHat here

2. Gaining access to crashed VM

There can be a couple of ways to gain access to a crashed XenServer VM.  One of the easiest ways is to boot up in recovery mode and go through the installers rescue mode.  Guide to boot up XenServer VM in recovery mode here.

The installer rescue mode should detect the OS and mount everything.  If not you should be able to mount the root partition manually.

Once booted up, you’ll need to enable network access if your going to use sftp or scp to copy files.  There are a few different ways to do this

  • Run “dhclient” to pull an address via DHCP
  • Set a static IP address
    • Guide for Ubuntu/Debian distro’s here
    • Guide for fedora/CentOS/RedHat here

3.Copy LibreNMS Mysql Database

Backup the LibreNMS MySQL database directory

tar czvf librenms_mysql.tgz /var/lib/mysql

Use scp or sftp to copy it to the new LibreNMS instance

scp librenms_mysql.tgz user@new_LibreNMS_ip

Now on the new LibreNMS instance we need to run the following few commands

systemctl stop mariadb
rm -rf /var/lib/mysql/*
tar xzvf librenms_mysql.tgz -C /
chown -R mysql:mysql /var/lib/mysql

4. Copy LibreNMS directory

Backup the LibreNMS directory

tar czvf librenms.tgz /opt/librenms

Use scp or sftp to copy it to the new LibreNMS instance

scp librenms.tgz user@new_LibreNMS_ip

Now on the new LibreNMS instance we need to run the following few commands

rm -rf /opt/librenms/*
tar xzvf librenms.tgz -C /
chown -R librenms:librenms /opt/librenms

5. Clean up

Disable SELinux if you have not already.  Guide here

Restart apache, and start mysql.  If your on Ubuntu, the services are named apache2 and mysql

systemctl restart httpd
systemctl restart mariadb

That should get it working, if not try a reboot.

Special notes

The whole MySQL directory needs to be copied, there are innodb files that will keep MySQL from starting if they are not copied.

There is some good info here
https://stackoverflow.com/questions/1795176/how-to-change-mysql-data-directory

Setup SNMP on XenServer

log into the Xenserver via SSH, or get a console through XenCenter.

Open up the snmpd.conf file with your favorite text editor, nano, vi etc.

vi /etc/snmp/snmpd.conf

Find the line the following line

com2sec notConfigUser default public

and change the source “default” to the allowed IP ranges and the community “public” to whatever you want your new community string to be.

Example:

com2sec notConfigUser 192.168.1.0/24 privateString

Save, exit and start snmpd

service snmpd start

Add the following line in your iptables config.  It needs to be above the COMMIT line.

-A RH-Firewall-1-INPUT -s 0.0.0.0/0 -m udp -p udp --dport 161 -j ACCEPT

Restart iptables

service iptables restart

You should be good to go.

Increase hard disk size on Windows VM in XenServer

To increase a Windows VM disk size you’ll first need to shut down the VM and then increase the disk size from XenCenter.  When you boot back up into the VM, the extra space shows up as unallocated.  All that needs to be done is the main partition needs to be extended.

Open up the Windows Disk Management Utility.  You can do this by clicking on the start button, typing in disk management and hit enter.  This will bring up the Disk Management program.

In disk manager, you should see the Unallocated space at the end of Disk 0, i.e. your main drive.  Right click on your main partition (C:) and click Extend Volume.  You will be presented with a wizard that will walk you through extending your disk.

Example:

Before increasing the VM’s disk size.

0

The unallocated space is the extra space that we added via XenCenter.  Right click on the middle partition (C:), and click extend.

1

After were are finished extending the partition, we see that our main partition is about 8GB larger.

3

How to Boot a XenServer VM into Recovery Mode

Recovery Mode can be helpful if you need to do certain task like operate on the VM with the disk unmounted or you need to reset a root password.

All of the following is done through XenCenter

To boot the VM into recovery mode all you have to do is

  1. Shutdown the VM
  2. Insert boot media i.e. (DVD, cd, iso image, etc.) into DVD Drive 1:
  3. Start VM in Recovery Mode.  In the main menu click on VM –> Start/Shut Down –> Start in Recovery Mode

When the VM boots up it will boot off of whatever is in DVD Drive 1.

 

 

XenServer Unable to Import OVF

Check to see if you have an import template.

In the Main Menu click “View” and check “Hidden Objects”, in the Server View you should see a Template named something like “XenServer Transfer VM 6.2.0-123456c (hidden)”

If you don’t have one, then run the following command on the XenServer to create a template.

/opt/xensource/packages/files/transfer-vm/install-transfer-vm.sh

If the command fails double check that you have a default storage repository selected and try again.

Once the template is created import the OVF.

How to Create a Local Storage Repository for XenServer

Find your hard drive and or partition.

cat /proc/partitions

The above command should give you a list of all the HDD’s and partitions on the system.

Find the UUID of the disk with the following command

ll /dev/disk/by-id

Find the UUID of your XenServer.

xe host-list

Create the storage repository.

 xe sr-create content-type=user device-config:device=/dev/disk/by-id/"Disk UUID" host-uuid="Host UUID" name-label=”Local Storage 2” shared=false type=lvm

example:

[root@XenServer ~]# cat /proc/partitions
major minor #blocks name

3 0 22378 loop0
6 0 285474816 sda   <--- Main HHD is is here, shows total size of disk
8 1 4193297 sda1 
8 2 4193297 sda2       
8 3 244085167 sda3  <--- This is the partition we need
252 0 4096 dm-0
252 1 105070592 dm-1
253 0 104857600 tda
253 1 4363264 tdb
[root@XenServer ~]# ll /dev/disk/by-id/
total 0
lrwxrwxrwx 1 root root 9 Oct 13 20:38 scsi-56234c232c92f371217ce8549f51e0nf1 -> ../../sda
lrwxrwxrwx 1 root root 10 Oct 13 20:38 scsi-56234c232c92f371217ce8549f51e0ff1-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Oct 13 20:38 scsi-56234c232c92f371217ce8549f51e0ff1-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Oct 13 20:38 scsi-56234c232c92f371217ce8549f51e0ff1-part3 -> ../../sda3   <-- This is the UUID for the partition
lrwxrwxrwx 1 root root 9 Oct 13 20:38 usb-Dell_Virtual_CDROM_1028_123456 -> ../../sr1
[root@XenServer ~]# xe host-list
uuid ( RO)                : dce4e7fe-8342-bd11-12c8-83caac124a4b   <--- UUID for XenServer
          name-label ( RW): XenServer001
    name-description ( RW): Default install of XenServer

[root@XenServer ~]# xe sr-create content-type=user device-config:device=/dev/disk/by-id/scsi-56234c232c92f371217ce8549f51e0ff1-part3 host-uuid=dce4e7fe-8342-bd11-12c8-83caac124a4b name-label="Local Storage" shared=false type=lvm

 

How to export/import a XenServer VM form the Command Line

We can list the VM’s by running

xe vm-list

You can export a VM using either the name of the VM or by using the uuid.  The above command list both so you can use which ever one you want.

You will also need to shutdown the VM your going to export.

xe vm-shutdown vm=ubuntu

 

Export by Name
The name of the VM is”ubuntu”.

xe vm-export vm=ubuntu filename=/backup/ubuntu.xva

Export using uuid

xe vm-export uuid=b24dcd65-5e12-4576-2f39-46ecab9362ab filename=/backup/ubuntu.xva

 

Importing VM

xe vm-import vm=centos5 filename=/backup/ubuntu.xva

How to add a NFS ISO Repository to XenServer

This assumes you have a NFS Share setup up.  If you do not you can follow this link.

Open up XenCenter click on add storage device button on the top.

Select NFS iso and hit next.

Connect XenServer to NFS

Enter in the name you want to call your repository
Connect XenServer to NFS 2

Next we add the share location which in my case is 192.168.200.250:/nfs  and hit Finish.

Connect XenServer to NFS 3

If all went well you should have a new NFS ISO repository

Connect XenServer to NFS 5

Connect XenServer to NFS 6