Troubleshooting Backup Errors on WHM / cPanel

Below are some helpful locations of files, logs etc for troubleshooting backup errors on WHM

WHM backup logs
Change date to the correct date. Should be one log per day or I guess every time a backup runs.

/usr/local/cpanel/logs/cpbackup/{date}.log

View WHM backup config

cat /var/cpanel/backups/config

View WHM remote destination config(s)
Replace *** with the appropriate name.

cat /var/cpanel/backups/***.backup_destination

Rysnc.pm file

May need to modify this file to increase time out limits if you are having issues with time out errors for backups.

/usr/local/cpanel/Cpanel/Transport/Files/Rsync.pm

This link has some more info https://forums.cpanel.net/threads/what-commands-does-cpanel-run-over-ssh-to-do-rsync-backups.671777/

Unable to prune transport Rsync Incremental Backup – WHM/cPanel

For some reason I keep getting an alert about the transport failing to prune the incremental backups. Shows “ssh slave failed: timed out”

Going to the backup server shows that the directories have been pruned. This makes the alert a bit confusing.

It appears that others are experiencing the same problem.

https://forums.cpanel.net/threads/remote-incremental-backups-timeouts.606911/page-3

https://forums.cpanel.net/threads/error-pruning-ssh-slave-failed-timed-out.627691/

You can check the backup log to see if it gives you any errors or ideas on what the problem is. Replace {currentdate} with the date of the log file you want.

/usr/local/cpanel/logs/cpbackup/{currentdate}.log

One thing to try is to increase the time out on the

In WHM, go to Backup -> Backup Configuration -> Additional Destinations -> Your Destination
Scroll down to the bottom and enter a higher timeout value

Setting cPanel remote transport timeout

One user said they patched the rsync.pm file. Looks like there may be a 30 second timeout for rsync, so maybe increasing that would help.

/usr/local/cpanel/Cpanel/Transport/Files/Rsync.pm

Backup Matrix Synapse PostgreSQL Database

This is part of a series of posts on backing up and restoring a backup for Matrix Synapse server. Synapse was installed using the matrix-docker-ansible deployment which while a little complicated can greatly ease management later on down the road. All the main components are in docker containers so we need to use docker to access.

https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/maintenance-postgres.md#backing-up-postgresql

As the root user run

docker exec --env-file=/matrix/postgres/env-postgres-psql matrix-postgres pg_dumpall -h matrix-postgres | gzip -c > /matrix/postgres.sql.gz

This will dump the Postgres database in /matrix/postgres.sql.gz
We can use this later to restore to a new server or keep as a backup.

Configure rsnapshot on Ubuntu Server

rsnapshot is a utility that uses rsync to backup files locally or it can backup files from a remote server.

While trying to figure out a good solution for backing up an Ubuntu Server I decided to try rsnapshot, however since it can either create a local backup or pull a remote backup it needs to be configured to do that on the backup server side. It does not “push” a backup to a backup server.

Some helpful snippits from the man file.

rsnapshot will typically be invoked as root by a cron job, or series of cron jobs. It is possible, however, to run as any    arbitrary user with an alternate configuration file.
...
USAGE
        rsnapshot can be used by any user, but for system-wide backups you will probably want to run it as root.
...
NOTES
        Make sure your /etc/rsnapshot.conf file has all elements separated by tabs.  See
        /usr/share/doc/rsnapshot/examples/rsnapshot.conf.default.gz for a working example file.
    Make sure you put a trailing slash on the end of all directory references.  If you don't, you may have extra directories    created in your snapshots.  For more information on how the trailing slash is handled, see the rsync(1) manpage.

Overview

Scenario

Host A runs xyz application and host B is the backup server. We create a backup user on host A, host B then uses that user to ssh and rsync backups to itself.

  1. Create backup user
  2. Configure rysnc to be used without a password
  3. Setup SSH Key, aka Passwordless authentication (On backup server)
  4. Setup rsnapshot config (On backup server)
  5. Configure rsnapshot in crontab (On backup server)
  6. Final Testing

Create backup user

The following commands are fairly straight forward. Change backupuser to whatever you want to call your backup user.

sudo useradd -m backupuser
passwd backupuser
sudo usermod -a -G sudo backupuser

Configure rysnc to be used without a password

We need to setup the backup user to be able to use “sudo rsync” without having to input the user password. If we don’t use sudo we can’t access system files for backups. And if we have to manually input the password every time rsync runs, then the backups would not be automatic. The following link was helpful.

https://unix.stackexchange.com/questions/325100/proper-way-to-set-up-rsnapshot-over-ssh

All we need to do is create a file in /etc/sudoers.d/username and then tell it we don’t need to enter a password when “sudo rsync” is run.

sudo tee /etc/sudoers.d/backupuser <<<'backupuser ALL = (root) NOPASSWD: /usr/bin/rsync'

Setup SSH Key, aka Passwordless authentication (On backup server)

Log into the backup server

Create SSH keys. Note that since rsnapshot wants to run as root, we create the key and copy it as the root user.

sudo ssh-keygen

Accept all the defaults so we can login from the backup server without having to enter in a password.

Copy ssh key to the server we are wanting to back up

sudo ssh-copy-id backupuser@ip

enter in the password and the the key should get copied it over. Once complete, verify that you can login without having to enter in a password.

Setup rsnapshot config (On backup server)

Open up the rsnapshot config file and modify where appropriate. /etc/rsnapshot.conf

Change the path to where the snapshots are stored. By default it stores them under /.snapshots. I moved it under a local user as I am not needing to use rsnapshot to backup the local backup server files.


# SNAPSHOT ROOT DIRECTORY
snapshot_root /home/user/rsnapshot/snapshots/

Add a daily backup option under Backup levels

# BACKUP LEVELS / INTERVAL #
retain daily 6

Setup remote server to get a backup from. Replace ipaddress and directories as needed. hostname is the sever name. You can change to whatever you want.

### BACKUP POINTS / SCRIPTS ###
# LOCALHOST
# Comment or delete entries unless you want to backup those as well
# EXAMPLE.COM
backup  backupuser@ipaddress:/home/     hostname/       +rsync_long_args=--rsync-path="sudo rsync"

If you would like to back up multiple locations you can create multiple entries with different remote paths. Example locations to add

backup  backupuser@ipaddress:/etc/     hostname/       +rsync_long_args=--rsync-path="sudo rsync"
backup  backupuser@ipaddress:/usr/local/     hostname/       +rsync_long_args=--rsync-path="sudo rsync"

Verify that the config is good with

sudo rsnapshot configtest

It should return Syntax OK

Setup Crontab

sudo crontab -e

Add the following line to run rsnapshot at 3AM every day. More information about crontab can be found here.

0 3 * * * /usr/bin/rsnapshot daily

Final Testing

Manually run a backup to verify everything is set up correctly.

sudo rsnapshot daily

After it runs you can check the directory you specified in the config file to verify that the files did get copied.

Migrate LibreNMS to new Server

Some quick notes on manually moving LibreNMS to a new server.  For more detailed instructions you can refer to the following post

Recovering LibreNMS from crashed XenServer VM

Steps

  1. Backup /opt/librenms file
  2. Backup librenms database
  3. Install LibreNMS on new server
  4. Import /opt/librenms files
  5. Import librenms database

Backing up LibreNMS files

LibreNMS stores all it’s files in /opt/librenms
Quick and easy way to back up the LibreNMS files is to use tar.

tar zcvf librenms_backup.tgz /opt/librenms

Migrating Database

I believe you can copy the whole /var/lib/mysql directory.  Should be able to use the above command.  If not you can do a mysqldump and then import it on the new system.

LibreNMS backup script

You should be able to copy and paste the following in a backup.sh file and then execute from cron. Should work out of the box, but you can change the backup directory and the teams.sh path if needed/wanted.

#!/bin/bash

# LibreNMS backup script
# Jan 1, 2019

lDate=`date +%Y%m%d-%H%M`       # local date + hour minute
dDate=`date +%Y%m%d`            # todays date

# If you have the teams.sh script, you can trigger a backup notification
ALERT="/home/admin/teams.sh -b"

# Directory to backup to
bDir="/backup"
bName="librenms_backup"

# MySQL settings for tar and sqldump
sqlDir="/var/lib/mysql"
sqlDB="librenms"
sqlUN="root"
sqlPW=""
LOG="${bDir}/${lDate}-${bName}.log"

# Directory that contains data
dDir="/opt/librenms"

# tar LibreNMS dir
# tar SQL dir "the whole thing with the innode files
# sql dump of the db for extra redundancy

if [ -d ${bDir} ]; then
echo "backup dir exist, starting to backup"
else
        echo "backup dir not available.  Quiting"
        exit 1
fi

${ALERT} "Starting backup for ${bName} - `date`"

systemctl stop mariadb httpd
# LibreNMS data backup
tar -zcvf ${bDir}/${lDate}-${bName}.tgz ${dDir}
if [ $? -eq 0 ]; then
        echo "Tar succesfully backed up ${bDir}"
else
        echo "Tar failed while trying to backup ${dDir}"
        echo " ${lDate} - Tar failed while trying to backup ${dDir}" >> ${LOG}
        ${ALERT} "${lDate} - Tar failed while trying to backup ${dDir}"
fi

# MySQL data backup
tar -zcvf ${bDir}/${lDate}-${bName}-mysql.tgz ${sqlDir}
if [ $? -eq 0 ]; then
        echo "Tar succesfully backed up ${sqlDir}"
else
        echo "Tar failed while trying to backup ${sqlDir}"
        echo " ${lDate} - Tar failed while trying to backup ${sqlDir}" >> ${LOG}
        ${ALERT} "${lDate} - Tar failed while trying to backup ${sqlDir}"
fi

systemctl start mariadb httpd
sleep 5

 # SQL dump
mysqldump -u ${sqlUN} -p'4rfvBHU8!' ${sqlDB} > ${bDir}/${lDate}-${bName}.sql
if [ $? -eq 0 ]; then
        echo "MySQL DB dumped"
else
        echo "Ran into error while doing sql dump"
        echo "${lDate} - Ran into error while doing sql dump" >> ${LOG}
        ${ALERT} "${lDate} - Ran into error while doing sql dump"
fi

echo "Removing old backups"
if ( ls ${bDir} | grep -q ${dDate} );then
        find ${bDir}/* -prune -mtime +31 -exec rm {} \;
else
        echo "Looks like there are no backup files!  Aborting!!!"
        ${ALERT} "${lDate} - Error: find failed to find any backup files in backup dir.  Aborting!!!"
fi

${ALERT} "Finished backup for ${bName} - `date`"

Extract Android backup

https://stackoverflow.com/questions/18533567/how-to-extract-or-unpack-an-ab-file-android-backup-file

Install Android Backup Toolkit

wget https://downloads.sourceforge.net/project/adbextractor/android-backup-tookit-20180521.zip
unzip android-backup-tookit-20180521.zip
cd android-backup-tookit/android-backup-extractor/android-backup-extractor-20180521-bin

Unpack backup. Changes it from an ab to a tar file

java -jar abe.jar unpack ~/path/to/backup.ab ~/path/to/backup.tar

After thats complete, you can untar it. Change Desktop to the path you want to extract to.

tar -xvf ~/path/to/backup.tar -C ~/Desktop/

Extract encrypted Signal backup

https://github.com/xeals/signal-back
More information at the above link

Installation of precompiled binary for Signal-Back

wget https://github.com/xeals/signal-back/releases/download/v0.1.7-alpha.2/signal-back_linux_amd64
chmod +x signal-back_linux_amd64
./signal-back_linux_amd64

Help info

Usage: signal-back_linux_amd64 COMMAND [OPTION…] BACKUPFILE
   --help, -h     show help
   --version, -v  print the version
 Commands: 
   format   Read and format the backup file
   analyse  Information about the backup file
   extract  Retrieve attachments from the backup
   check    Verify that a backup is readable
   help     Shows a list of commands or help for one command

Extract attachments from backup

Replace ~/Desktop/signal…backup with the path to your Signal backup.

./signal-back_linux_amd64 extract ~/Desktop/signal-2019-09-30-01-43-21.backup

It’ll ask for the password and then start pulling out the attachments and putting them in the current directory.

Export messages to CSV file

Default option is XML. CSV will let you open it up in Excel.

./signal-back_linux_amd64 format -f CSV --output signal-output.csv ~/Desktop/signal-2019-09-30-01-43-21.backup

Specify password to use

You can use the -p option to specify the password. It does not matter if there are spaces every 5 numbers or not. Also the -P option is suppose to let you use a file.

./signal-back_linux_amd64 extract -p "48294 55709 09123 94563 74662 12800" ~/Desktop/signal-2019-09-30-01-43-21.backup

Other help options

bob@localhost:~/Downloads/signal-back$ ./signal-back_linux_amd64 help
 Usage: signal-back_linux_amd64 COMMAND [OPTION…] BACKUPFILE
 --help, -h     show help
   --version, -v  print the version
 Commands:
   format   Read and format the backup file
   analyse  Information about the backup file
   extract  Retrieve attachments from the backup
   check    Verify that a backup is readable
   help     Shows a list of commands or help for one command
bob@localhost:~/Downloads/signal-back$ ./signal-back_linux_amd64 help format
 Usage: signal-back_linux_amd64 format [OPTION…] BACKUPFILE
 Parse and transform the backup file into other formats.
 Valid formats include: CSV, XML, RAW.
 --format FORMAT, -f FORMAT  output the backup as FORMAT (default: "xml")
   --message TYPE, -m TYPE     format TYPE messages (default: "sms")
   --output FILE, -o FILE      write decrypted format to FILE
   --password PASS, -p PASS    use PASS as password for backup file
   --pwdfile FILE, -P FILE     read password from FILE
   --verbose, -v               enable verbose logging output
bob@localhost:~/Downloads/signal-back$ ./signal-back_linux_amd64 help extract
 Usage: signal-back_linux_amd64 extract [OPTION…] BACKUPFILE
 Decrypt files embedded in the backup.
 --outdir DIRECTORY, -o DIRECTORY  output attachments to DIRECTORY
   --password PASS, -p PASS          use PASS as password for backup file
   --pwdfile FILE, -P FILE           read password from FILE
   --verbose, -v                     enable verbose logging output
bob@localhost:~/Downloads/signal-back$