Server Logs not Showing up in LibreNMS

The problem: Linux servers have been configured to send their local syslogs to LibreNMS, but are not showing up under the LibreNMS -> DEVICE -> Logs-> Syslog

After a bit of troubleshooting, found that the issue is the hostname being sent with the logs is different than what LibreNMS has for the device. It appears that some Linux distributions will or can use an abbreviated system hostname. There is a section in the LibreNMS docs about this

https://docs.librenms.org/Extensions/Syslog/#matching-syslogs-to-hosts-with-different-names

We can either do what the docs say, or we can set the host name in the rsyslog.conf file on each of the servers.

Log into the server and open up

sudo vi /etc/rsyslog.conf

At the very top, add the following line to set the hostname

$LocalHostName host.server_name_fqdn.com

Save the file and restart rsyslog

sudo systemctl restart rsyslog

Refresh the page to verify the logs are showing up in LibreNMS.

If you are still having issues, you may want to check the following

  1. SELinux on LibreNMS SELinux Audit Commands and Links, Setup LibreNMS as Syslog Server
  2. Firewall on LibreNMS FrDual Zones in Firewalld (Public/Private or External/Internal), Install LibreNMS on CentOS
  3. Read the documentation entirely through

Setup LibreNMS as Syslog Server

Using the LibreNMS documentation for setting up syslog-ng so LibreNMS can ingest logs from Cisco, Mikrotik, Ubiquiti etc. equipment.

https://docs.librenms.org/Extensions/Syslog/

Enable Syslog in LibreNMS settings

First thing we need to do is enable syslog for LibreNMS. Edit the /opt/librenms/config.php and add or enable

$config['enable_syslog'] = 1;

Install and Configure syslog-ng

Install syslog-ng with dnf or yum.

sudo dnf install -y syslog-ng

Create a config file for LibreNMS

vi /etc/syslog-ng/conf.d/librenms.conf

Put the following in the config file

source s_net {
        tcp(port(514) flags(syslog-protocol));
        udp(port(514) flags(syslog-protocol));
};

destination d_librenms {
        program("/opt/librenms/syslog.php" template ("$HOST||$FACILITY||$PRIORITY||$LEVEL||$TAG||$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC||$MSG||$PROGRAM\n") template-escape(yes));
};

log {
        source(s_net);
        source(s_sys);
        destination(d_librenms);
};

Restart and enable syslog-ng

sudo systemctl restart syslog-ng
sudo systemctl enable syslog-ng

SELinux

If we are running SELinux, we’ll need to make and apply a module to let the logs show up in the web interface.

vi librenms-rsyslog.te

Put the following in the file

module mycustom-librenms-rsyslog 1.0;

require {
        type syslogd_t;
        type httpd_sys_rw_content_t;
        type ping_exec_t;
        class process execmem;
        class dir { getattr search write };
        class file { append getattr execute open read };
}

#============= syslogd_t ==============
allow syslogd_t httpd_sys_rw_content_t:dir { getattr search write };
allow syslogd_t httpd_sys_rw_content_t:file { open read append getattr };
allow syslogd_t self:process execmem;
allow syslogd_t ping_exec_t:file execute;

Now run the following commands to make and apply our SELinux module.

checkmodule -M -m -o librenms-rsyslog.mod librenms-rsyslog.te
semodule_package -o librenms-rsyslog.pp -m librenms-rsyslog.mod
sudo semodule -i librenms-rsyslog.pp

Setup Remote Syslog on Cisco

Configure Logging

First we need to drop into configuration mode

conf t

Now we run the following command. Change ip-address to the address of you remote syslog server.

logging host ip-address

You will want to make sure that your time/timezone is correct.

https://community.cisco.com/t5/networking-knowledge-base/how-to-configure-logging-in-cisco-ios/ta-p/3132434

Set timezone

Change UTC and 0 to your your timezone and how many hours off UTC you are. For example for EST you would do EST -5

clock timezone UTC 0

Enable Remote Logging in Rsyslog on Linux (Debian, Ubuntu, CentOS, Fedora)

Edit the rsyslog file

sudo vi /etc/rsyslog.conf

add the following. Change out the IP address for your remote syslog server IP address.

The syntax for this has changes. Newer recommended way is

*.*  action(type="omfwd" target="192.0.10.5" port="514" protocol="tcp"

Old Syntax

## rsyslog Server
*.*     @10.0.10.5:514

https://www.rsyslog.com/sending-messages-to-a-remote-syslog-server/

Restart the rsyslog service and check your remote log server.

systemctl restart rsyslog

If you are having issues viewing logs in LibreNMS, try adding the hostname in the /etc/rsyslog.conf file

$LocalHostName server.hostname.com

LibreNMS manually clean up MySQL Syslog Database

Adding the following option to the config.php file is supposed to delete anything over 30 days.

$config['syslog_purge']                                 = 30;

You can also manually delete the entries out of the MySQL database by logging into MySQL, selecting the librenms database, and running the command below.

The command deletes all entries older than 12/9/2018 at 08:00. Change the date and time as needed.

DELETE FROM syslog WHERE timestamp < '2018-12-9 08:00:00';

The following link has some more clean up options.
https://docs.librenms.org/Support/Cleanup-options/