Backup Matrix Synapse PostgreSQL Database

This is part of a series of posts on backing up and restoring a backup for Matrix Synapse server. Synapse was installed using the matrix-docker-ansible deployment which while a little complicated can greatly ease management later on down the road. All the main components are in docker containers so we need to use docker to access.

https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/maintenance-postgres.md#backing-up-postgresql

As the root user run

docker exec --env-file=/matrix/postgres/env-postgres-psql matrix-postgres pg_dumpall -h matrix-postgres | gzip -c > /matrix/postgres.sql.gz

This will dump the Postgres database in /matrix/postgres.sql.gz
We can use this later to restore to a new server or keep as a backup.

Configure rsnapshot on Ubuntu Server

rsnapshot is a utility that uses rsync to backup files locally or it can backup files from a remote server.

While trying to figure out a good solution for backing up an Ubuntu Server I decided to try rsnapshot, however since it can either create a local backup or pull a remote backup it needs to be configured to do that on the backup server side. It does not “push” a backup to a backup server.

Some helpful snippits from the man file.

rsnapshot will typically be invoked as root by a cron job, or series of cron jobs. It is possible, however, to run as any    arbitrary user with an alternate configuration file.
...
USAGE
        rsnapshot can be used by any user, but for system-wide backups you will probably want to run it as root.
...
NOTES
        Make sure your /etc/rsnapshot.conf file has all elements separated by tabs.  See
        /usr/share/doc/rsnapshot/examples/rsnapshot.conf.default.gz for a working example file.
    Make sure you put a trailing slash on the end of all directory references.  If you don't, you may have extra directories    created in your snapshots.  For more information on how the trailing slash is handled, see the rsync(1) manpage.

Overview

Scenario

Host A runs xyz application and host B is the backup server. We create a backup user on host A, host B then uses that user to ssh and rsync backups to itself.

  1. Create backup user
  2. Configure rysnc to be used without a password
  3. Setup SSH Key, aka Passwordless authentication (On backup server)
  4. Setup rsnapshot config (On backup server)
  5. Configure rsnapshot in crontab (On backup server)
  6. Final Testing

Create backup user

The following commands are fairly straight forward. Change backupuser to whatever you want to call your backup user.

sudo useradd -m backupuser
passwd backupuser
sudo usermod -a -G sudo backupuser

Configure rysnc to be used without a password

We need to setup the backup user to be able to use “sudo rsync” without having to input the user password. If we don’t use sudo we can’t access system files for backups. And if we have to manually input the password every time rsync runs, then the backups would not be automatic. The following link was helpful.

https://unix.stackexchange.com/questions/325100/proper-way-to-set-up-rsnapshot-over-ssh

All we need to do is create a file in /etc/sudoers.d/username and then tell it we don’t need to enter a password when “sudo rsync” is run.

sudo tee /etc/sudoers.d/backupuser <<<'backupuser ALL = (root) NOPASSWD: /usr/bin/rsync'

Setup SSH Key, aka Passwordless authentication (On backup server)

Log into the backup server

Create SSH keys. Note that since rsnapshot wants to run as root, we create the key and copy it as the root user.

sudo ssh-keygen

Accept all the defaults so we can login from the backup server without having to enter in a password.

Copy ssh key to the server we are wanting to back up

sudo ssh-copy-id backupuser@ip

enter in the password and the the key should get copied it over. Once complete, verify that you can login without having to enter in a password.

Setup rsnapshot config (On backup server)

Open up the rsnapshot config file and modify where appropriate. /etc/rsnapshot.conf

Change the path to where the snapshots are stored. By default it stores them under /.snapshots. I moved it under a local user as I am not needing to use rsnapshot to backup the local backup server files.


# SNAPSHOT ROOT DIRECTORY
snapshot_root /home/user/rsnapshot/snapshots/

Add a daily backup option under Backup levels

# BACKUP LEVELS / INTERVAL #
retain daily 6

Setup remote server to get a backup from. Replace ipaddress and directories as needed. hostname is the sever name. You can change to whatever you want.

### BACKUP POINTS / SCRIPTS ###
# LOCALHOST
# Comment or delete entries unless you want to backup those as well
# EXAMPLE.COM
backup  backupuser@ipaddress:/home/     hostname/       +rsync_long_args=--rsync-path="sudo rsync"

If you would like to back up multiple locations you can create multiple entries with different remote paths. Example locations to add

backup  backupuser@ipaddress:/etc/     hostname/       +rsync_long_args=--rsync-path="sudo rsync"
backup  backupuser@ipaddress:/usr/local/     hostname/       +rsync_long_args=--rsync-path="sudo rsync"

Verify that the config is good with

sudo rsnapshot configtest

It should return Syntax OK

Setup Crontab

sudo crontab -e

Add the following line to run rsnapshot at 3AM every day. More information about crontab can be found here.

0 3 * * * /usr/bin/rsnapshot daily

Final Testing

Manually run a backup to verify everything is set up correctly.

sudo rsnapshot daily

After it runs you can check the directory you specified in the config file to verify that the files did get copied.

Migrate LibreNMS to new Server

Some quick notes on manually moving LibreNMS to a new server.  For more detailed instructions you can refer to the following post

Recovering LibreNMS from crashed XenServer VM

Steps

  1. Backup /opt/librenms file
  2. Backup librenms database
  3. Install LibreNMS on new server
  4. Import /opt/librenms files
  5. Import librenms database

Backing up LibreNMS files

LibreNMS stores all it’s files in /opt/librenms
Quick and easy way to back up the LibreNMS files is to use tar.

tar zcvf librenms_backup.tgz /opt/librenms

Migrating Database

I believe you can copy the whole /var/lib/mysql directory.  Should be able to use the above command.  If not you can do a mysqldump and then import it on the new system.

LibreNMS backup script

You should be able to copy and paste the following in a backup.sh file and then execute from cron. Should work out of the box, but you can change the backup directory and the teams.sh path if needed/wanted.

#!/bin/bash

# LibreNMS backup script
# Jan 1, 2019

lDate=`date +%Y%m%d-%H%M`       # local date + hour minute
dDate=`date +%Y%m%d`            # todays date

# If you have the teams.sh script, you can trigger a backup notification
ALERT="/home/admin/teams.sh -b"

# Directory to backup to
bDir="/backup"
bName="librenms_backup"

# MySQL settings for tar and sqldump
sqlDir="/var/lib/mysql"
sqlDB="librenms"
sqlUN="root"
sqlPW=""
LOG="${bDir}/${lDate}-${bName}.log"

# Directory that contains data
dDir="/opt/librenms"

# tar LibreNMS dir
# tar SQL dir "the whole thing with the innode files
# sql dump of the db for extra redundancy

if [ -d ${bDir} ]; then
echo "backup dir exist, starting to backup"
else
        echo "backup dir not available.  Quiting"
        exit 1
fi

${ALERT} "Starting backup for ${bName} - `date`"

systemctl stop mariadb httpd
# LibreNMS data backup
tar -zcvf ${bDir}/${lDate}-${bName}.tgz ${dDir}
if [ $? -eq 0 ]; then
        echo "Tar succesfully backed up ${bDir}"
else
        echo "Tar failed while trying to backup ${dDir}"
        echo " ${lDate} - Tar failed while trying to backup ${dDir}" >> ${LOG}
        ${ALERT} "${lDate} - Tar failed while trying to backup ${dDir}"
fi

# MySQL data backup
tar -zcvf ${bDir}/${lDate}-${bName}-mysql.tgz ${sqlDir}
if [ $? -eq 0 ]; then
        echo "Tar succesfully backed up ${sqlDir}"
else
        echo "Tar failed while trying to backup ${sqlDir}"
        echo " ${lDate} - Tar failed while trying to backup ${sqlDir}" >> ${LOG}
        ${ALERT} "${lDate} - Tar failed while trying to backup ${sqlDir}"
fi

systemctl start mariadb httpd
sleep 5

 # SQL dump
mysqldump -u ${sqlUN} -p'4rfvBHU8!' ${sqlDB} > ${bDir}/${lDate}-${bName}.sql
if [ $? -eq 0 ]; then
        echo "MySQL DB dumped"
else
        echo "Ran into error while doing sql dump"
        echo "${lDate} - Ran into error while doing sql dump" >> ${LOG}
        ${ALERT} "${lDate} - Ran into error while doing sql dump"
fi

echo "Removing old backups"
if ( ls ${bDir} | grep -q ${dDate} );then
        find ${bDir}/* -prune -mtime +31 -exec rm {} \;
else
        echo "Looks like there are no backup files!  Aborting!!!"
        ${ALERT} "${lDate} - Error: find failed to find any backup files in backup dir.  Aborting!!!"
fi

${ALERT} "Finished backup for ${bName} - `date`"

Extract Android backup

https://stackoverflow.com/questions/18533567/how-to-extract-or-unpack-an-ab-file-android-backup-file

Install Android Backup Toolkit

wget https://downloads.sourceforge.net/project/adbextractor/android-backup-tookit-20180521.zip
unzip android-backup-tookit-20180521.zip
cd android-backup-tookit/android-backup-extractor/android-backup-extractor-20180521-bin

Unpack backup. Changes it from an ab to a tar file

java -jar abe.jar unpack ~/path/to/backup.ab ~/path/to/backup.tar

After thats complete, you can untar it. Change Desktop to the path you want to extract to.

tar -xvf ~/path/to/backup.tar -C ~/Desktop/

Extract encrypted Signal backup

https://github.com/xeals/signal-back
More information at the above link

Installation of precompiled binary for Signal-Back

wget https://github.com/xeals/signal-back/releases/download/v0.1.7-alpha.2/signal-back_linux_amd64
chmod +x signal-back_linux_amd64
./signal-back_linux_amd64

Help info

Usage: signal-back_linux_amd64 COMMAND [OPTION…] BACKUPFILE
   --help, -h     show help
   --version, -v  print the version
 Commands: 
   format   Read and format the backup file
   analyse  Information about the backup file
   extract  Retrieve attachments from the backup
   check    Verify that a backup is readable
   help     Shows a list of commands or help for one command

Extract attachments from backup

Replace ~/Desktop/signal…backup with the path to your Signal backup.

./signal-back_linux_amd64 extract ~/Desktop/signal-2019-09-30-01-43-21.backup

It’ll ask for the password and then start pulling out the attachments and putting them in the current directory.

Export messages to CSV file

Default option is XML. CSV will let you open it up in Excel.

./signal-back_linux_amd64 format -f CSV --output signal-output.csv ~/Desktop/signal-2019-09-30-01-43-21.backup

Specify password to use

You can use the -p option to specify the password. It does not matter if there are spaces every 5 numbers or not. Also the -P option is suppose to let you use a file.

./signal-back_linux_amd64 extract -p "48294 55709 09123 94563 74662 12800" ~/Desktop/signal-2019-09-30-01-43-21.backup

Other help options

bob@localhost:~/Downloads/signal-back$ ./signal-back_linux_amd64 help
 Usage: signal-back_linux_amd64 COMMAND [OPTION…] BACKUPFILE
 --help, -h     show help
   --version, -v  print the version
 Commands:
   format   Read and format the backup file
   analyse  Information about the backup file
   extract  Retrieve attachments from the backup
   check    Verify that a backup is readable
   help     Shows a list of commands or help for one command
bob@localhost:~/Downloads/signal-back$ ./signal-back_linux_amd64 help format
 Usage: signal-back_linux_amd64 format [OPTION…] BACKUPFILE
 Parse and transform the backup file into other formats.
 Valid formats include: CSV, XML, RAW.
 --format FORMAT, -f FORMAT  output the backup as FORMAT (default: "xml")
   --message TYPE, -m TYPE     format TYPE messages (default: "sms")
   --output FILE, -o FILE      write decrypted format to FILE
   --password PASS, -p PASS    use PASS as password for backup file
   --pwdfile FILE, -P FILE     read password from FILE
   --verbose, -v               enable verbose logging output
bob@localhost:~/Downloads/signal-back$ ./signal-back_linux_amd64 help extract
 Usage: signal-back_linux_amd64 extract [OPTION…] BACKUPFILE
 Decrypt files embedded in the backup.
 --outdir DIRECTORY, -o DIRECTORY  output attachments to DIRECTORY
   --password PASS, -p PASS          use PASS as password for backup file
   --pwdfile FILE, -P FILE           read password from FILE
   --verbose, -v                     enable verbose logging output
bob@localhost:~/Downloads/signal-back$ 

Install Oxidized on LibreNMS

Install Oxidized

Install Instructions from here

Install prerequisites

sudo yum -y install make cmake which sqlite-devel openssl-devel libssh2-devel ruby gcc ruby-devel libicu-devel gcc-c++

Install Ruby Gems

gem install oxidized
gem install oxidized-script oxidized-web

If you run into issues with installing the oxidized-web gem because of a ruby version, try installing ruby 2.6.1 via rvm the docs say install 2.1.2, but was having issues with it. 2.6.1 seems fine.

Install Oxidized

gem install oxidized 
gem install oxidized-script
gem install oxidized-web

Run Oxidized twice to generate the config

oxidized

If you have issues running oxidized due to an invalid or self signed ssl cert, you can either fix the cert or ignore it in the oxidized config. https://github.com/ytti/oxidized/pull/618/files

Modify the Oxidized config file in “/root/.config/oxidized/config” to look like this.

Modify the http portion of the config file to look like the following.

http:
url: https://localhost/api/v0/oxidized
scheme: https
secure: false

Setup Config

vi .config/oxidized/config

Setup the config, Basic config below, edit as needed

---
username: admin
password: password
model: airos
resolve_dns: true
interval: 3600
use_syslog: false
debug: false
threads: 30
timeout: 20
retries: 3
prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/
rest: 127.0.0.1:8888
next_adds_job: false
vars: {}
groups:
airos:
username: ubnt
password: ubnt
routeros:
username: admin
password:
vars:
ssh_port: 2222
models: {}
pid: /home/incredigeek/.config/oxidized/pid
crash:
directory: /home/incredigeek/.config/oxidized/crashes
hostnames: false
stats:
history_size: 10
input:
default: ssh, telnet
debug: false
ssh:
secure: false
ftp:
passive: true
utf8_encoded: true
output:
default: file
file:
directory: /home/incredigeek/.config/oxidized/configs
source:
default: http
debug: false
http:
secure: false
scheme: https
url: https://localhost/api/v0/oxidized
map:
name: hostname
model: os
group: group
headers:
X-Auth-Token: '3wq2b87fj4e6fb5987b5812t6ej9709g'
model_map:
cisco: ios
juniper: junos
mikrotik: routeros

Setup as system service

Find the “oxidized.service” file, should be in

/usr/local/rvm/gems/ruby-2.6.1/gems/oxidized-0.25.1/extra/oxidized.service

and copy it to

/usr/lib/systemd/system/

Edit it and change the user and try to launch it

systemctl start oxidized

If it fails, you may need to substitute the ExecStart= variable to the following more info

ExecStart=/usr/local/rvm/gems/ruby-2.6.1/wrappers/oxidize

Change the ruby number if yours is different.

Enable the service on system boot up and start

systemctl enable --now oxidized

Check that oxidized is running

systemctl status oxidized

LibreNMS config

Add the following to your /opt/librenms/config.php config file.

# Oxidized configuration
$config['oxidized']['enabled'] = TRUE;
$config['oxidized']['url'] = 'http://127.0.0.1:8888';
$config['oxidized']['features']['versioning'] = true;
$config['oxidized']['group_support'] = true;
$config['oxidized']['default_group'] = 'default';
$config['oxidized']['reload_nodes'] = true;# Oxidized configuration
$config['oxidized']['enabled'] = TRUE;
$config['oxidized']['url'] = 'http://127.0.0.1:8888';
$config['oxidized']['features']['versioning'] = true;
$config['oxidized']['group_support'] = true;
$config['oxidized']['default_group'] = 'default';
$config['oxidized']['reload_nodes'] = true;
$config['oxidized']['ignore_os'] = array('linux','windows');
$config['oxidized']['ignore_types'] = array('server','power');

LibreNMS should now feed Oxidized the devices.

You can check in the LibreNMS interface to see if it is getting the configs. https://librenms/oxidized

where librenms is your LibreNMS servers ip/hostname.