the “magic bytes” are the first few bytes of a file that can tell you what format the file is.
https://en.wikipedia.org/wiki/List_of_file_signatures
Mikrotik Backup file magic bytes
| File type | Magic bytes |
| RC4 Encrypted | EFA89172 |
| AES Encrypted | EFA89173 |
| Not Encrypted | 88ACA1B1 |
Using the above list, we can view a Mikrotik .backup file in a hex editor like GHex or dump it with xxd.
Thanks to the guys who put together the information at the following links.
https://wadman.co.nz/2021/01/02/Viewing-LibreNMS-data-in-Grafana/
https://www.reddit.com/r/LibreNMS/comments/ojc8cc/how_to_almost_natively_integrate_librenms_and/
I ran into some issues trying to get this to work. So here are some of my notes. I already had a LibreNMS installation set up.
NOTE FOR CENTOS 8, ALMALINUX 8 and 9
The steps for installing RRDReST are slightly different. Check out the following post.
https://www.incredigeek.com/home/setting-up-rrdrest-on-centos-8-or-almalinux-9/
I had issues installing RRDReST. I am guessing it had to do with it accessing files. I was able to install it in a docker container.
sudo yum install -y docker docker-compose
sudo systemctl enable docker
Create docker compose file with the following options
vi docker-compose.yml
Change the TZ to your time zone. If you have issues with the graphs, most likely something is off with the time zone between this container and Grafana/LibreNMS server
version: "3.5"
services:
rrdrest:
image: michaelwadman/rrdrest:latest
container_name: rrdrest
restart: always
volumes:
- "/opt/librenms:/opt/librenms"
environment:
- TZ=America/Denver
Save the file and start and setup the container with
sudo docker-compose up -d
You will need your docker container IP address to setup the connection in Grafana
sudo docker exec -it rrdrest ip addr | grep eth0
Congratulations. You should now have a RRDReST docker container that will auto start on system boot and has the correct time zone.
Basic steps are as follows
There is not anything special with installing Grafana on the same server as LibreNMS. You can follow the official guide to install it
https://grafana.com/docs/grafana/latest/installation/
After Grafana is installed, install the JSON API data source. You can do this using the grafana-cli
grafana-cli plugins install marcusolsson-json-datasource
A note on SSL/TLS certificates. If you have an SSL certificate for LibreNMS, you can use it for grafana. If you run into issues, try copying the cert (fullchain.pem, privkey.pem) to /etc/grafana/
You’ll most likely need to change owner
sudo chown root:grafana /etc/grafana/*.pem
And maybe the file permissions.
sudo chmod 640 /etc/grafana/*.pem
This is fairly straight forward.
grafana-cli plugins install marcusolsson-json-datasource
In Grafana, go to Configuration -> Data Sources -> Add data source
In Grafana, go to Configuration -> Data Sources -> Add data source
You should now be able to view your dashboard and use the drop down menus to select devices
There were a couple of issues I ran into while trying to get everything working together.
Issue: When trying to run RRDReST with uvicorn, I was never able to access the rrd files, even the test rrd files that are included when installing RRDReST. I am guessing it is either a permisions issue, or something unable to access the files.
Work around: Install RRDReST via Docker container.
Issue: Get a “JSON API: Bad Request” when trying to set up the LibreNMS API Data Source in Grafana.
Work around: Install a valid SSL Certificate and set up a DNS record so you can access LibreNMS at librenms.yourdomain.com.
More info: I would assume that “Skip TLS Verify” would work with or without a valid certificate, but it would not work for me. There are potentially some other options with modifying how Nginx or Apache is set up that would get this working. If you setup Grafana to use a SSL certificate, you may need to copy the certificate files (fullchain.pem, privkey.pem) to /etc/grafana/ and run “chown root:grafana *.pem” to let grafana have access to the files.
Running the ./validate.php script returns the following error
[FAIL] Python3 module issue found: 'Required packages: ['PyMySQL!=1.0.0', 'python-dotenv', 'redis>=3.0', 'setuptools', 'psutil>=5.6.0', 'command_runner>=1.3.0']
Package not found: The 'command_runner>=1.3.0' distribution was not found and is required by the application
'
[FIX]:
pip3 install -r /opt/librenms/requirements.txt
Running the [FIX] throws an error saying gcc failed with exit status 1.
Fortunately this issue is easy to resolve.
First we need to install python3-devel
sudo yum install python3-devel
Next, as the librenms user, run the pip command to install the requirements.
pip3 install --user -U -r /opt/librenms/requirements.txt
Run ./validate.php to verify that everything is working.
Some of the UniFi UNVR’s have system files on a USB drive. There seem to be a number of the drives failing recently, rendering the UNVR inoperable. Fortunately it is easy to replace. The following steps should preserve the video recordings.
Note on Recovery
You could potentially mount the failed or failing USB drive on a Linux machine copy off a UniFi backup. Unfortunately, the UniFi Protect backup does not preserve the users. Just the video groups. You will probably have to resend invites to users.
It sounds like the UniFi Protect system will try to read the drives and if it can preserve the data or read the raid information it will try to use that. That is what it sounds like at least from the forums. More info on drive management.
The following links were helpful in setting things up.
https://doc.powerdns.com/recursor/running.html
https://www.reddit.com/r/linuxadmin/comments/9lc4jl/logging_queries_in_pdnsrecursor/
First we need to find the line that says “disable-syslog” and uncomment/change it to
disable-syslog=no
Next find the line that says “quiet” and uncomment/change it to
quiet=no
Some other lines you may want to check and change
logging-facality=1
loglevel=6
Next we need to modify the Systemd unit file to allow PowerDNS Recursor to log to syslog.
systemctl edit --full pdns-recursor.service
On the ExecStart Line, remove the part that says
--disable-syslog
The resulting line should look something like
[Service]
ExecStart=/usr/sbin/pdns_recursor --socket-dir=%t/pdns-recursor --socket-dir=%t/pdns-recursor --daemon=no --write-pid=no --log-timestamp=no
Save the file.
Edit the rsyslog file
sudo vim /etc/rsyslog.conf
Add the following line
local1.* /var/log/pdns_recursor.log
This should now log all of the PowerDNS Recursor log info to “/var/log/pdns_recursor.log”
Restart the rsyslog and PowerDNS Recursor service
sudo systemctl restart rsyslog sudo systemctl restart pdns-recursor
You should now see DNS request in the log file.
tail /var/log/pdns_recursor.log
They should also show up in the “/var/log/messages”
Edit the rsyslog file
sudo vi /etc/rsyslog.conf
add the following. Change out the IP address for your remote syslog server IP address.
The syntax for this has changes. Newer recommended way is
*.* action(type="omfwd" target="192.0.10.5" port="514" protocol="tcp"
Old Syntax
## rsyslog Server *.* @10.0.10.5:514
https://www.rsyslog.com/sending-messages-to-a-remote-syslog-server/
Restart the rsyslog service and check your remote log server.
systemctl restart rsyslog
If you are having issues viewing logs in LibreNMS, try adding the hostname in the /etc/rsyslog.conf file
$LocalHostName server.hostname.com
Upgrading is fairly straightforward.
cd into the git directory and run
git pull
That will download any new files.
Run the ansible command, you can have it ask you for the password to use.
ansible-playbook -i inventory/hosts setup.yml --tags=setup-all --ask-pass --ask-become-pass
It should give you a report if anything failed.
https://github.com/spantaleev/matrix-docker-ansible-deploy
When using the above to run a matrix server, it can be confusing how to verify and check which version of Postgres you are running. Fortunately this is really easy to check.
Run
sudo matrix-postgres-cli --version
And you should get the version of Postgres that is being used.
psql (PostgreSQL) 9.6
You will need a hash of the password to put into the database. We don’t cover that in this post. You could copy the password from a different user account or use a different UniFi instance to change the password and then check the DB to find the hash.
ssh unifiadmin@unifiserver
Connect to Mongo by typing in the following.
mongo -port 27117
Then select the ace database by typing
use ace
The admins are in the admin collection/table. Use the following command to list all the users and their name, email, and password hash.
db.admin.find({ }, { name:"" , email : "", "x_shadow" : "" })
The following looks complex. Fortunately though you should be able to copy and paste. You should only need to change the
– ObjectId to your User Id
– Password Hash to your password hash
db.admin.update({"_id" : ObjectId("223abc5489de0a93be758493")}, {$set: { "x_shadow" : "$6$nwpi7.q2$OuD9/UZGZt5cD739Dt7j8Gb1uPtfU99p0DeDSurSNBZVizieUrFVFbRufiZMgOk2IaaDZN9BVmL9yUwQ2mC8f."}});
Note: The hash above is password. Not recommended for use.
You should receive a confirmation that it succeeded. Test the new password by logging into the UniFi Controller.
For more MongoDB commands, check out this post.
Manually adding the UniFi Protect self signed certificate can be helpful if you are accessing the controller over the local network. It has the benefit of letting your browser remember the password if wanted and not having to verify the certificate.
First we need to acquire the certificate.
In a Chrome based web browser, go to the IP address of the UniFi Protect NVR
Click in the URL bar where is says “Not secure” -> Certificate -> Details -> Copy to file
Follow the export Wizard. You can use the default settings.
You can now import the certificate by double clicking on the cert.
Note that it says to install the cert in the “Trusted Root Certification Authorities” store
Now we can move on to Configuring the hosts file.
We need to add an entry for unifi.local as the cert if for unifi.local hostname and not the controllers IP address.
We’ll essentially be following the same steps outlined in the following post.
C:\Windows\System32\drivers\etc\
192.168.1.20 unifi.local
You should now be set. Open a browser and got to https://unifi.local to access the UniFi Protect Controller.