Install Oxidized on LibreNMS

Install Oxidized

Install Instructions from here

Install prerequisites

For CentOS, RockyLinux, and Alma Linux.

sudo yum -y install make cmake which sqlite-devel openssl-devel libssh2-devel ruby gcc ruby-devel libicu-devel gcc-c++

Install Ruby Gems

If you run into issues with installing the oxidized-web gem because of a ruby version, try installing ruby 2.6.1 via rvm the docs say install 2.1.2, but was having issues with it. 2.6.1 seems fine.

Install Oxidized

Note: If you run into issues with oxidized not being able to ssh into devices, and showing “OpenSSL::PKey::PKeyError with msg “dh#set_pqg= is incompatible with OpenSSL 3.0″” in the log, try installing from git.

gem install oxidized 
gem install oxidized-script
gem install oxidized-web 

Run Oxidized twice to generate the config

oxidized

If you have issues running oxidized due to an invalid or self signed ssl cert, you can either fix the cert or ignore it in the oxidized config. https://github.com/ytti/oxidized/pull/618/files

Modify the Oxidized config file in “/root/.config/oxidized/config” to look like this.

Modify the http portion of the config file to look like the following.

http:
url: https://localhost/api/v0/oxidized
scheme: https
secure: false

Setup Config

Put the config file in the users home directory.

vi .config/oxidized/config

Setup the config, Basic config below, edit as needed

---
username: admin
password: password
model: airos
resolve_dns: true
interval: 3600
use_syslog: false
debug: false
threads: 30
timeout: 20
retries: 3
prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/
rest: 127.0.0.1:8888
next_adds_job: false
vars: {}
groups:
airos:
username: ubnt
password: ubnt
routeros:
username: admin
password:
vars:
ssh_port: 2222
models: {}
pid: /home/incredigeek/.config/oxidized/pid
crash:
directory: /home/incredigeek/.config/oxidized/crashes
hostnames: false
stats:
history_size: 10
input:
default: ssh, telnet
debug: false
ssh:
secure: false
ftp:
passive: true
utf8_encoded: true
output:
default: file
file:
directory: /home/incredigeek/.config/oxidized/configs
source:
default: http
debug: false
http:
secure: false
scheme: https
url: https://localhost/api/v0/oxidized
map:
name: hostname
model: os
group: group
headers:
X-Auth-Token: '3wq2b87fj4e6fb5987b5812t6ej9709g'
model_map:
cisco: ios
juniper: junos
mikrotik: routeros

Setup as system service

Find the “oxidized.service” file, should be in

  /usr/local/rvm/gems/ruby-2.6.1/gems/oxidized-0.25.1/extra/oxidized.service

or

/usr/local/share/gems/gems/oxidized-0.28.0/extra/oxidized.service

and copy it to

  /usr/lib/systemd/system/

Edit it and change the user and try to launch it

systemctl start oxidized

If it fails, you may need to substitute the ExecStart= variable to the following more info

ExecStart=/usr/local/rvm/gems/ruby-2.6.1/wrappers/oxidize

Change the ruby number if yours is different.

Enable the service on system boot up and start

systemctl enable --now oxidized

Check that oxidized is running

systemctl status oxidized

LibreNMS config

Add the following to your /opt/librenms/config.php config file.

# Oxidized configuration
 $config['oxidized']['enabled']                  = TRUE;
 $config['oxidized']['url']                      = 'http://127.0.0.1:8888';
 $config['oxidized']['features']['versioning']   = true;
 $config['oxidized']['group_support']            = true;
 $config['oxidized']['default_group']            = 'default';
 $config['oxidized']['reload_nodes']             = true;
 $config['oxidized']['ignore_os'] = array('linux','windows');
 $config['oxidized']['ignore_types'] = array('server','power');

LibreNMS should now feed Oxidized the devices.

You can check in the LibreNMS interface to see if it is getting the configs. https://librenms/oxidized

Where librenms is your LibreNMS servers ip/hostname.

Install Ruby 2.6.1 via RVM on CentOS

Install Prerequisites

yum install -y curl gcc-c++ patch readline readline-devel zlib zlib-devel libyaml-devel libffi-devel openssl-devel make cmake bzip2 autoconf automake libtool bison libssh2-devel libicu-devel

Install RVM

curl -L get.rvm.io | bash -s stable

Setup RVM, install Ruby 2.6.1 and set to default.

source /etc/profile.d/rvm.sh 
rvm install 2.6.1
rvm use --default 2.6.1

You may need to add root or the user your using to the rvm group

sudo usermod -G rvm username

Pulled some of the info from here https://github.com/ytti/oxidized#installing-ruby-212-using-rvm

Bash ANSI-C Quoting

Pulled from https://www.gnu.org/software/bash/manual/bashref.html#ANSI_002dC-Quoting

As a side note there is a lot of good bash info out here. https://www.gnu.org/software/bash/manual/bashref.html

3.1.2.4 ANSI-C Quoting

Words of the form $'string' are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard. Backslash escape sequences, if present, are decoded as follows: \a

alert (bell) \b

backspace \e\E

an escape character (not ANSI C) \f

form feed \n

newline \r

carriage return \t

horizontal tab \v

vertical tab \\

backslash \'

single quote \"

double quote \?

question mark \nnn

Some examples

echo Hello $'\t' World

Returns “Hello World” with a tab space between both words.

echo Hello $'\n' World

Returns Hello on one line and World on the second

echo "\"Hello World\""

Returns “Hello World” inside double quotes

Bash script to send messages to Microsoft Teams

Copy and save into teams.sh file. Make executable. Change web hook, Run!

The script is a modified Slack script from off the web.

#!/bin/bash
# bash script to send messages to Microsoft Teams.
function usage {
echo "HELP ME!"
echo "description: send messages to Microsoft Teams channels"
echo "special notes: You'll need to change the teamsUrl variable to contain your webhook from Teams."
echo "usage: ${0} -b \"Message contents\""
echo " -m Message body"
echo " -h This help info"
exit 1
}
while getopts "m:h" opt; do
case ${opt} in
m) msgBody="$OPTARG"
;;
h) usage
;;
\?) echo "Invalid option -$OPTARG" >&2
;;
esac
done
# Add/Change the webhook to one you created in Teams
teamsUrl="https://teamswebhook"
if [[ ! "${msgBody}" ]]; then
echo "You didn't specify a message!"
usage
fi
read -d '' payLoad << EOF
{
"text": "${msgBody}",
}
EOF
statusCode=$(curl \
--write-out %{http_code} \
--silent \
--output /dev/null \
-X POST \
-H 'Content-type: application/json' \
--data "${payLoad}" ${teamsUrl})
echo ${statusCode}

Bash script to monitor system service

This bash script runs and checks to see if a service like httpd, or mysql is running and alerts if it is not.

Script Usage

servicemonitor.sh httpd mariadb

Where httpd and mariadb are the services you want to monitor/check.

Setup Script

Create servicemonitor.sh file and paste the following contents in.

#!/bin/bash

timeHour=`date +%H` # date/time just shows the hour
quietHour="02"    # If it is this hour, then exit program, useful if services are expected to go down during a particular time for maintenance
if ( echo ${timeHour} | grep ${quietHour}); then
         echo "Is during quiet time.  Quiting."
         exit
fi

 function ALERT {
 msg="~/teams.sh -b"  # Sends a message to Microsoft Teams channel.  Needs the teams.sh script in the users home directory.
 ${msg} "$1"
 }
 function SERVICECHECK {
 serviceName="${1}"
 if (systemctl status ${serviceName} | grep Active | grep inactive); then
         ALERT "ERROR: $(hostname) - ${serviceName} - ${0} is inactive"
         echo "ERROR: ${serviceName} is inactive!"
 else
         echo "Running!"
 fi
 }
 for i in $@
 do
 echo Checking ${i}
 SERVICECHECK ${i}
 done

Note the teams.sh script that is called is another script that is called that sends an alert to Microsoft Teams. Is not needed for this script to run, but allows for remote alerting.

Save file and make it executable

chmod +x servicemonitor.sh

Add script to crontab (Optional)

crontab -e

The following runs the script every 5 minutes. Can change the 5 to 1 to run every minute. Change httpd and mariadb to the service you want to monitor.

*/5 * * * * /home/UserName/servicemonitor.sh httpd mariadb

Delete anonymous MySQL user

Log into mysql

mysql -u root -p

List users

select User,Host from mysql.user;

Should return something like the following

MariaDB [mysql]> select User,Host from user;
+----------+-----------------------+
| User | Host |
+----------+-----------------------+
| root | 127.0.0.1 |
| librenms | localhost |
| | localhost.localdomain |
+----------+-----------------------+
3 rows in set (0.00 sec)
MariaDB [mysql]>

Delete anonymous user

Note that there are two single quotes ‘ before the @ sign, not a double quote “

drop user ''@'localhost.localdomain';

LibreNMS enable RRDCached on CentOS 7

Commands from https://docs.librenms.org/Extensions/RRDCached/

Create System Service

sudo vi /etc/systemd/system/rrdcached.service
[Unit]
Description=Data caching daemon for rrdtool
After=network.service
[Service]
Type=forking
PIDFile=/run/rrdcached.pid
ExecStart=/usr/bin/rrdcached -w 1800 -z 1800 -f 3600 -s librenms -U librenms -G librenms -B -R -j /var/tmp -l unix:/run/rrdcached.sock -t 4 -F -b /opt/librenms/rrd/
[Install]
WantedBy=default.target

Save with esc + :wq

Enable rrdcached service on boot up and start the service now

systemctl enable --now rrdcached.service

Enable RRDcached in librenms config

vi /opt/librenms/config.php

add/edit the following line. If it is commented out, uncomment it

$config['rrdcached'] = "unix:/run/rrdcached.sock";

Save with esc + :wq

Check the disk IO after a little bit to see if it went down.

VIM/SED Search and replace lines that do not contain numbers

Objective: Find all lines in a file that only contain alpha characters and delete or replace.

Sample contents of file.

 Z2j2NH23
VTQnPwSS
hFbxgvFt
VSVR8v3F
GPrP4zo

The following sed command works for our objective.

sed s/[[:alpha:]]\{8\}/ALPHAONLY/g file.txt

The part in the [] tells sed to search for any alpha characters a-Z, the part in bold \{8\} tells it to search 8 spaces out (Change if needed) and ALPHAONLY is what alpha line will get substituted to.

sed s/[[:alpha:]]\{8\}/ALPHAONLY/g



Returns

Z2j2NH23
ALPHAONLY
ALPHAONLY
VSVR8v3F
GPrP4zo9

You can run the same basic syntax in VI

Search and replace

:%s/[[:alpha:]]\{8\}/ALPHAONLY/g 

Or to delete the lines

:%d/[[:alpha:]]\{8\}/d

You can also change [[:alpha:]] for [[:digit:]] if you want to search for numbers instead.