Setting up Graphana on LibreNMS

Thanks to the guys who put together the information at the following links.

https://wadman.co.nz/2021/01/02/Viewing-LibreNMS-data-in-Grafana/
https://www.reddit.com/r/LibreNMS/comments/ojc8cc/how_to_almost_natively_integrate_librenms_and/

I ran into some issues trying to get this to work. So here are some of my notes. I already had a LibreNMS installation set up.

  1. Install RRDReST
    1. Install Docker
    2. Configure Docker compose file
  2. Configure LibreNMS API User and Key
  3. Set up and Configure Grafana
    1. Install Grafana
    2. Install JSON Data Source Plugin
    3. Add LibreNMS API Data Source
    4. Add RRDReST Data Source
    5. Import Dashboard into Grafana
  4. View graphs

Installing RRDReST

I had issues installing RRDReST. I am guessing it had to do with it accessing files. I was able to install it in a docker container.

  1. Install Docker
  2. Create a compose file
  3. Install container

Install docker

sudo yum install -y docker docker-compose
sudo systemctl enable docker

Create docker compose file with the following options

vi docker-compose.yml

Change the TZ to your time zone. If you have issues with the graphs, most likely something is off with the time zone between this container and Grafana/LibreNMS server

version: "3.5"

services:

  rrdrest:
    image: michaelwadman/rrdrest:latest
    container_name: rrdrest
    restart: always
    volumes:
      - "/opt/librenms:/opt/librenms"
    environment:
      - TZ=America/Denver

Save the file and start and setup the container with

sudo docker-compose up -d

You will need your docker container IP address to setup the connection in Grafana

sudo docker exec -it rrdrest ip addr | grep eth0

Congratulations. You should now have a RRDReST docker container that will auto start on system boot and has the correct time zone.

Configure LibreNMS API User and Key

  1. Create a Grafana user in LibreNMS. (Settings Gear -> Manage Users -> Add Users)
    You could technically skip this step and use an existing user.
  2. Create API token for the newly created user (Setting Gear -> API -> API Settings)
Create a Grafana user in LibreNMS
Setup API Key for Grafana in LibreNMS

Set up and Configure Grafana

Basic steps are as follows

  1. Install Grafana
  2. Install JSON Data Source Plugin
  3. Configure Data Sources
    1. LibreNMS API
    2. RRDRest API
  4. Import Dashboard into Grafana

Install Grafana

There is not anything special with installing Grafana on the same server as LibreNMS. You can follow the official guide to install it

https://grafana.com/docs/grafana/latest/installation/

After Grafana is installed, install the JSON API data source. You can do this using the grafana-cli

grafana-cli plugins install marcusolsson-json-datasource

A note on SSL/TLS certificates. If you have an SSL certificate for LibreNMS, you can use it for grafana. If you run into issues, try copying the cert (fullchain.pem, privkey.pem) to /etc/grafana/

You’ll most likely need to change owner

sudo chown root:grafana /etc/grafana/*.pem

And maybe the file permissions.

sudo chmod 640 /etc/grafana/*.pem

Install JSON Data Source Plugin

This is fairly straight forward.

grafana-cli plugins install marcusolsson-json-datasource

Add LibreNMS API Data Source

In Grafana, go to Configuration -> Data Sources -> Add data source

  • Set Name for Data Source
  • URL should be https://your_librenms_url/api/v0
  • Add Custom HTTP Header
    • Header field should be “X-Auth-Token”
    • Value field should contain the API token we created in LibreNMS
  • Save and Test
    If you receive any errors, refer to the Troubleshooting part at the end.
Adding LibreNMS API Data Source in Grafana

Add RRDReST Data Source

In Grafana, go to Configuration -> Data Sources -> Add data source

  • Set Name for Data Source
  • URL needs to be your docker container IP address (Steps above)
  • Save and Test (Should return “Unprocessable Entity”)
Adding RRDReST API Data Source in Grafana

Import Dashboard into Grafana

Now we need a dashboard to present our data.

  • Go to Create -> Import
  • Upload JSON file (Download from here or PasteBin )
  • Under RRDReST API , select our RRDReST Data Source
  • Under LibreNMS API , select our LibreNMS Data Source
  • Click Import

You should now be able to view your dashboard and use the drop down menus to select devices

Grafana viewing bandwidth on device being monitored by LibreNMS

Troubleshooting

There were a couple of issues I ran into while trying to get everything working together.

RRDReST shows 404 Not Found

Issue: When trying to run RRDReST with uvicorn, I was never able to access the rrd files, even the test rrd files that are included when installing RRDReST. I am guessing it is either a permisions issue, or something unable to access the files.
Work around: Install RRDReST via Docker container.

Error Running uvicorn RRDReST

Error Adding LibreNMS API

Issue: Get a “JSON API: Bad Request” when trying to set up the LibreNMS API Data Source in Grafana.

Work around: Install a valid SSL Certificate and set up a DNS record so you can access LibreNMS at librenms.yourdomain.com.

More info: I would assume that “Skip TLS Verify” would work with or without a valid certificate, but it would not work for me. There are potentially some other options with modifying how Nginx or Apache is set up that would get this working. If you setup Grafana to use a SSL certificate, you may need to copy the certificate files (fullchain.pem, privkey.pem) to /etc/grafana/ and run “chown root:grafana *.pem” to let grafana have access to the files.

Grafana LibreNMS API JSON API: Bad Request

Backup Matrix Synapse PostgreSQL Database

This is part of a series of posts on backing up and restoring a backup for Matrix Synapse server. Synapse was installed using the matrix-docker-ansible deployment which while a little complicated can greatly ease management later on down the road. All the main components are in docker containers so we need to use docker to access.

https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/maintenance-postgres.md#backing-up-postgresql

As the root user run

docker exec --env-file=/matrix/postgres/env-postgres-psql matrix-postgres pg_dumpall -h matrix-postgres | gzip -c > /matrix/postgres.sql.gz

This will dump the Postgres database in /matrix/postgres.sql.gz
We can use this later to restore to a new server or keep as a backup.

Install UniFi 5.6.42 in Docker

Pull UniFi 5.6.42 docker image

sudo docker pull jacobalberty/unifi:5.6.42

Install image

sudo docker run -d --init --restart=unless-stopped --name=unifi-controller --net=host --volume=/docker/unifi:/var/lib/unifi -p 8080:8080/tcp -p 8081:8081/tcp -p 8443:8443/tcp -p 8843:8843/tcp -p 8880:8880/tcp -p 8883:8883/tcp -p 3478:3478/udp jacobalberty/unifi:5.6.42

Access and setup the unifi controller from a browser.

https://ip-address:8443

Auto renew SSL Cert with UniFi running in Docker

Setting up the SSL cert for UniFi service when running in docker is fairly easy to do. All you have to do is modify the UniFi SSL renew script to use the UniFi Docker directory and change the start and stop service to start and stop the Docker container. The script below should be ready to go.

Download, chmod +x it, and run, drop it in cron to auto renew.

In the below script, change (unifiDir=”/docker/unifi”) to your UniFi directory.

Note: this triggers calling the teams.sh script that will send an update to Microsoft Teams to let you know that the certs should be renewed. Check here for more info.

#!/usr/bin/env bash
# Added support to do UniFi and UniFi controllers at the same time using the same cert.
# Original script from https://git.sosdg.org/brielle/lets-encrypt-scripts/raw/branch/master/gen-unifi-cert.sh
# More info here https://www.reddit.com/r/Ubiquiti/comments/43v23u/using_letsencrypt_with_the_unifi_controller/ 
# And here https://www.reddit.com/r/Ubiquiti/comments/43v23u/using_letsencrypt_with_the_unifi_controller/
# Modified script from here: https://github.com/FarsetLabs/letsencrypt-helper-scripts/blob/master/letsencrypt-unifi.sh
# Modified by: Brielle Bruns <bruns@2mbit.com>
# Download URL: https://source.sosdg.org/brielle/lets-encrypt-scripts
# Version: 1.7
# Last Changed: 04/10/2020
# 04/10/2020: Changed directories and commands to work with a UniFi Docker install
# 02/02/2016: Fixed some errors with key export/import, removed lame docker requirements
# 02/27/2016: More verbose progress report
# 03/08/2016: Add renew option, reformat code, command line options
# 03/24/2016: More sanity checking, embedding cert
# 10/23/2017: Apparently don't need the ace.jar parts, so disable them
# 02/04/2018: LE disabled tls-sni-01, so switch to just tls-sni, as certbot 0.22 and later automatically fall back to http/80 for auth
# 05/29/2018: Integrate patch from Donald Webster <fryfrog[at]gmail.com> to cleanup and improve tests
# 09/26/2018: Change from TLS to HTTP authenticator

# Location of LetsEncrypt binary we use.  Leave unset if you want to let it find automatically
# LEBINARY="/usr/src/letsencrypt/certbot-auto"

# Change to your UniFi Docker directory
unifiDir="/docker/unifi"

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

function usage() {
  echo "Usage: $0 -d <domain> [-e <email>] [-r] [-i]"
  echo "  -d <domain>: The domain name to use."
  echo "  -e <email>: Email address to use for certificate."
  echo "  -r: Renew domain."
  echo "  -i: Insert only, use to force insertion of certificate."
}

while getopts "hird:e:" opt; do
  case $opt in
    i) onlyinsert="yes";;
    r) renew="yes";;
    d) domains+=("$OPTARG");;
    e) email="$OPTARG";;
    h) usage
       exit;;
  esac
done

DEFAULTLEBINARY="/usr/bin/certbot /usr/bin/letsencrypt /usr/sbin/certbot
  /usr/sbin/letsencrypt /usr/local/bin/certbot /usr/local/sbin/certbot
  /usr/local/bin/letsencrypt /usr/local/sbin/letsencrypt
  /usr/src/letsencrypt/certbot-auto /usr/src/letsencrypt/letsencrypt-auto
  /usr/src/certbot/certbot-auto /usr/src/certbot/letsencrypt-auto
  /usr/src/certbot-master/certbot-auto /usr/src/certbot-master/letsencrypt-auto"

if [[ ! -v LEBINARY ]]; then
  for i in ${DEFAULTLEBINARY}; do
    if [[ -x ${i} ]]; then
      LEBINARY=${i}
      echo "Found LetsEncrypt/Certbot binary at ${LEBINARY}"
      break
    fi
  done
fi

# Command line options depending on New or Renew.
NEWCERT="--renew-by-default certonly"
RENEWCERT="-n renew"

# Check for required binaries
if [[ ! -x ${LEBINARY} ]]; then
  echo "Error: LetsEncrypt binary not found in ${LEBINARY} !"
  echo "You'll need to do one of the following:"
  echo "1) Change LEBINARY variable in this script"
  echo "2) Install LE manually or via your package manager and do #1"
  echo "3) Use the included get-letsencrypt.sh script to install it"
  exit 1
fi

if [[ ! -x $( which keytool ) ]]; then
  echo "Error: Java keytool binary not found."
  exit 1
fi

if [[ ! -x $( which openssl ) ]]; then
  echo "Error: OpenSSL binary not found."
  exit 1
fi

if [[ ! -z ${email} ]]; then
  email="--email ${email}"
else
  email=""
fi

shift $((OPTIND -1))
for val in "${domains[@]}"; do
        DOMAINS="${DOMAINS} -d ${val} "
done

MAINDOMAIN=${domains[0]}

if [[ -z ${MAINDOMAIN} ]]; then
  echo "Error: At least one -d argument is required"
  usage
  exit 1
fi

if [[ ${renew} == "yes" ]]; then
  LEOPTIONS="${RENEWCERT}"
else
  LEOPTIONS="${email} ${DOMAINS} ${NEWCERT}"
fi

if [[ ${onlyinsert} != "yes" ]]; then
  echo "Firing up standalone authenticator on TCP port 80 and requesting cert..."
  ${LEBINARY} --server https://acme-v01.api.letsencrypt.org/directory \
              --agree-tos --standalone --preferred-challenges http ${LEOPTIONS}
fi

if [[ ${onlyinsert} != "yes" ]] && md5sum -c "/etc/letsencrypt/live/${MAINDOMAIN}/cert.pem.md5" &>/dev/null; then
  echo "Cert has not changed, not updating controller."
  exit 0
else
  echo "Cert has changed or -i option was used, updating controller..."
  TEMPFILE=$(mktemp)
  CATEMPFILE=$(mktemp)

  # Identrust cross-signed CA cert needed by the java keystore for import.
  # Can get original here: https://www.identrust.com/certificates/trustid/root-download-x3.html
  cat > "${CATEMPFILE}" <<'_EOF'
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----
_EOF

  md5sum "/etc/letsencrypt/live/${MAINDOMAIN}/cert.pem" > "/etc/letsencrypt/live/${MAINDOMAIN}/cert.pem.md5"
  echo "Using openssl to prepare certificate..."
  cat "/etc/letsencrypt/live/${MAINDOMAIN}/chain.pem" >> "${CATEMPFILE}"
  openssl pkcs12 -export  -passout pass:aircontrolenterprise \
          -in "/etc/letsencrypt/live/${MAINDOMAIN}/cert.pem" \
          -inkey "/etc/letsencrypt/live/${MAINDOMAIN}/privkey.pem" \
          -out "${TEMPFILE}" -name unifi \
          -CAfile "${CATEMPFILE}" -caname root

  docker container stop ${dockerContainerId}
  sleep 10
  dockerContainerId=$(sudo docker container list | grep unifi-controller | awk '{print $1}')
  echo "Removing existing certificate from Unifi protected keystore..."
  keytool -delete -alias unifi -keystore ${unifiDir}/keystore -deststorepass aircontrolenterprise

  echo "Inserting certificate into Unifi keystore..."
  keytool -trustcacerts -importkeystore \
          -deststorepass aircontrolenterprise \
          -destkeypass aircontrolenterprise \
          -destkeystore ${unifiDir}/keystore \
          -srckeystore "${TEMPFILE}" -srcstoretype PKCS12 \
          -srcstorepass aircontrolenterprise \
          -alias unifi

  sleep 2
  echo "Starting Unifi controllers..."
  docker container start ${dockerContainerId}
  ./teams.sh -b "$(hostname) - UniFi service is restarting, ssl cert should be renewed."

  echo "Done!"
fi

Basic Docker commands

In the following commands, 367c7a1465ec = Docker container ID

Start/stop Docker service

systemctl start docker
systemctl stop docker

Automatically start docker on system boot

systemctl enable docker

List docker containers

sudo docker container list
sudo docker container list
367c7a1465ec        jacobalberty/unifi:latest   "/usr/loca/bin/dock…" 15 minutes ago      Up 14 minutes (healthy) unifi-controller

The bold part is your Docker container ID

List docker images on system

sudo docker images
sudo docker images
jacobalberty/unifi latest baebbe301633 9 days ago 711MB

Stop container

sudo docker stop 367c7a1465ec

Start container

sudo docker stop 367c7a1465ec

Installing UniFi controller in Docker Container

Instructions followed from here. Some changes were made
Commands work on Ubuntu 16.04

Install Docker

sudo apt install -y docker docker.io
systemctl enable docker
systemctl start docker

Install UniFi

sudo docker pull jacobalberty/unifi:latest

Setup Docker Image

The following command sets up a container which we can later manipulate to start and stop the “service”
You can specify where you want the UniFi files to reside if desired.

sudo docker run -d --init --restart=unless-stopped --name=unifi-controller --net=host --volume=/docker/unifi:/var/lib/unifi -p 8080:8080/tcp -p 8081:8081/tcp -p 8443:8443/tcp -p 8843:8843/tcp -p 8880:8880/tcp -p 8883:8883/tcp -p 3478:3478/udp jacobalberty/unifi:latest

Docker commands

List docker containers

sudo docker container list
367c7a1465ec        jacobalberty/unifi:latest   "/usr/loca/bin/dock…" 15 minutes ago      Up 14 minutes (healthy) unifi-controller

List docker images on system

sudo docker images
jacobalberty/unifi latest baebbe301633 9 days ago 711MB

Stop container. Also stops the UniFi service. Change the ID to your container ID.

sudo docker stop 367c7a1465ec

Other notes

When setting up the Docker image, the directory specified was “/docker/unifi” so all the UniFi files are in there and it looks like if you manipulate the files, it makes the changes fine. At least for setting up the SSL certificates.

OpenDroneMap – Process drone images

Install docker

sudo apt install docker

Create a new folder for OpenDroneMap to use

mkdir ODM 
cd ODM
mkdir images

Put all your drone images in the new images directory

Run command to process images.

sudo docker run -ti --rm -v ./ODM:/datasets/code opendronemap/odm --project-path /datasets

Note: ./ODM is the folder that the output will be stored. You need your drone images inside an images folder inside your working directory. i.e. Drone images go inside ODM/images

Zenoss 5 RabbitMQ not starting

The following is some commands to try and troubleshoot RabbitMQ not starting in Zenoss 5.

Connecting to the RabbitMQ container

serviced service attach $(serviced service list | grep -i rabbitmq | awk '{print $2}')

Check the service

You can check the RabbitMQ service by running “rabbitmqctl status”

[root@764399e5hhba /]# rabbitmqctl status
Status of node rabbit@rbt0 ...
Error: unable to connect to node rabbit@rbt0: nodedown

DIAGNOSTICS
===========

attempted to contact: [rabbit@rbt0]

rabbit@rbt0:
  * unable to connect to epmd (port 4369) on rbt0: nxdomain (non-existing domain)


current node details:
- node name: rabbitmqct22222@764399e5hhba
- home dir: /var/lib/rabbitmq
- cookie hash: yy3+awwOpeaaaa12wdf42ff==

[root@764399e5hhba /]# 

As you can see the node is down so RabbitMQ is not able to start.

Try to ping rbt0 (RabbitMQ connects to rbt0 so if it can’t resolve, then it can’t start)

ping rbt0

If you get a “ping: unknown host rbt0” then add the following to /etc/hosts.  Change the IP address to the IP address of the container.  You can run “ip addr” or ifconfig to get the IP.

172.20.0.11  rbt0

Or if your interface is eth0, you can run this command.

echo "$(ifconfig eth0 | grep "inet " | awk '{print $2}')  rbt0"  >> /etc/hosts

Then run “rabbitmqctl status” again.  The service auto starts, so it may take a minute, but you should see something similar to the following. (Not that I cut some of the text out.)

[root@764399e5hhba /]# rabbitmqctl status
 Status of node rabbit@rbt0 ...
 [{pid,4629},
 {running_applications,
 ... CUT TEXT ...
 {uptime,5}]
 ...done.
 [root@764399e5hhba /]#

 

Checking vhost

List the RabbitMQ vhosts with “rabbitmqcl list_vhosts”

[root@764399e5hhba /]# rabbitmqctl list_vhosts
 Listing vhosts ...
 /
 /zenoss
 ...done.

If you run rabbitmqctl list_vhosts and don’t see /zenoss, then add it

rabbitmqctl add_vhost /zenoss
rabbitmqctl set_permissions -p /zenoss zenoss '.*' '.*' '.*'

If the vhost is up then you can try deleting the zenoss vhosts and readding it.

[root@764399e5hhba /]#
rabbitmqctl delete_vhost /zenoss
rabbitmqctl add_vhost /zenoss
rabbitmqctl set_permissions -p /zenoss zenoss '.*' '.*' '.*'

The only downside to theses changes is that once the service gets restarted the changes you made inside the container will be lost, There should be a way to update the container so that the changes are persistent.

Set up ssh keys in zenoss 5

Here are the basic steps of how to setup ssh keys for zenoss.

  1. Log into the zenoss server
  2. Open up docker container
  3. Change to zenoss user
  4. Generate ssh keys
  5. Upload ssh keys
  6. Exit out of container
  7. Commit container

Example:

[root@zenoss ~]# serviced service shell -s AddingSSHkey -i zope bash
I0709 3:02:47.791788 01773 server.go:341] Connected to the control center at port 192.168.1.10:4979
I0709 3:02:48.127949 01773 server.go:435] Acquiring image from the dfs...
I0709 3:02:48.131438 01773 server.go:437] Acquired!  Starting shell
Trying to connect to logstash server... 127.0.0.1:5042
Connected to logstash server.
[root@321feeg2253a /]# su zenoss
[zenoss@321feeg2253a /]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/zenoss/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/zenoss/.ssh/id_dsa.
Your public key has been saved in /home/zenoss/.ssh/id_dsa.pub.
The key fingerprint is:
12:ab:14:d5:54:09:d3:1f:f7:12:21:ae:hd:16:a5:1b zenoss@321feeg2253a
The key's randomart image is:
+--[ DSA 2048]----+
|     =====F      |
|    S+== + AA    |
|  A=+=++  +      |
| AB= .. + =      |
|  ++S S  +.      |
|   ..    -       |
|                 |
|                 |
|                 |
+-----------------+
[zenoss@321feeg2253a /]# ssh-copy-id admin@192.168.1.10
The authenticity of host '192.168.1.10 (192.168.1.10)' can't be established.
RSA key fingerprint is 12:ab:14:d5:54:09:d3:1f:f7:12:21:ae:hd:16:a5:1b.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
admin@192.168.1.10's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'admin@192.168.1.10'"
and check to make sure that only the key(s) you wanted were added.

[zenoss@321feeg2253a /]# exit
exit
[root@321feeg2253a /]# exit
exit
[root@zenoss ~]# serviced snapshot commit AddingSSHkey
0sdj2jj412waawjideow120x_isjriw19-121200
[root@zenoss ~]# exit