Exim Troubleshooting Email Logs

The following is a great article explaining the main Exim log.

https://forums.cpanel.net/resources/reading-and-understanding-the-exim-main_log.383/

The following are some helpful tips from the post.

Search log by email address

You can search for specific addresses with the exigrep. Replace email@address with the email address of interest.

exigrep email@address /var/log/exim_mainlog

Message Direction

Looking at entries in the main log, some of the messages will have an indicator from the following table that tell us the status of the message and/or where it came from or went.

<=Indicates the arrival of a message to Exim for handling
=>Shows a normal message delivery
->Additional address for the same delivery, i.e. an Email forwarder.
>>cutthrough is a router precondition
This option requests delivery be attempted while the item is being received. It is usable in the RCPT ACL and valid only for single-recipient mails forwarded from one SMTP connection to another. If a recipient-verify callout connection is requested in the same ACL it is held open and used for the data, otherwise one is made after the ACL completes.
*>delivery suppressed by -N
**delivery failed; address bounced
==delivery deferred; temporary problem
<>For “<>” from the exim manual; Additionally, you will often find A bounce message is shown with the sender address “<>”, and if it is locally generated, this is followed by an item of the form
R=<message id>

Some other posts that may be helpful while troubleshooting mail deliveries.

View messages by ID

Bulk Delete Messages in Queue

Unable to launch Flatpaks on Fedora using Hardened Kernel

If you have installed the hardened Linux Kernel on Fedora, you may have encountered the following error when trying to launch Flatpak applications.

bwrap: No permissions to creating new namespace, likely because the kernel does not allow non-privileged user namespaces. On e.g. debian this can be enabled with 'sysctl kernel.unprivileged_userns_clone=1'.
error: Failed to sync with dbus proxy

https://security.stackexchange.com/questions/209529/what-does-enabling-kernel-unprivileged-userns-clone-do

https://github.com/containers/bubblewrap/issues/324

The issue looks to arise from the fact that the hardened Linux Kernel disables unprivileged name space and Fedora does not have setuid on by default on the bubblewrap executable.

Enabling setuid on bubblewrap

You can set the setuid permission on the bubblewrap executable with

sudo chmod u+s /usr/bin/bwrap

Allow Unprivileged Name Space (Alternative work around)

You could also allow unprivileged name space by running

sysctl kernel.unprivileged_userns_clone=1

Note that setting the setuid seems the safer/recommended option.

It looks like using the setuid binary for bubblewrap would be better to use then enabling unprivileged user space.

https://madaidans-insecurities.github.io/guides/linux-hardening.html#sysctl-kernel

Remove setuid on bubblewrap

If you would like to remove the setuid permission for any reason, you can with the following command.

sudo chmod u-s /usr/bin/bwrap

Dual Zones in Firewalld (Public/Private or External/Internal)

In Firewalld we can use multiple zones for different types of traffic. For instance, we can setup an “internal” zone with our local IP addresses that are trusted, and then setup the public facing interface to the “drop” or “block” zone to block everything not from our internal network.

  1. Setup trusted IP addresses in the “internal” zone
  2. Configure services/ports that should be allowed on our “internal” zone
  3. Set “drop” zone as the default for all other traffic
  4. Reload firewall

1. Setup trusted IP addresses in “internal” zone

Add all of our trusted IP addresses to the internal zone. The following example adds all of the private IP addresses “RFC 1918” to the internal zone. Change as needed.

firewall-cmd --zone=internal --add-source=192.168.0.0/16 --add-source=172.16.0.0/12 --add-source=10.0.0.0/8 --permanent

2. Configure services/ports that should be allowed on our “internal” zone

Next we need to specify which services or ports should be accessible in our trusted zone.

Here is an example to allow https, ssh, and cockpit services

firewall-cmd --zone=internal --add-service=https --add-service=ssh --add-service=cockpit --permanent 

Here is an example to allow port 8080 tcp

firewall-cmd --zone=internal --add-port=8080/tcp --permanent

3. Set “drop” zone as the default for all other traffic

The final configuration piece we need to do is set the default zone. Anything not specified in other zones will get processed by the default zone.

firewall-cmd --set-default-zone=drop

The drop zone drops everything.

4. Reload firewall

Reload the firewall with

firewall-cmd --reload


Verifying changes

Let’s verify the changes with the firewall-cmd –get-active-zones command

# firewall-cmd --get-active-zones
drop
  interfaces: en0
internal
  sources: 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8

You can also use

firewall-cmd --list-all-zones

to list all the zones. Active zones show (active) next to them.

You can verify that your changes worked by doing an internal and external nmap scan.

If you have issues with services still being accessible from the outside, try disabling Network Manager for that specific interface

You can edit the ifcfg-eth0 file and add

NM_CONTROLLED=no

Invalid privacy protocol specified after -3x flag: DES

With REHL 9 and AlmaLinux 9 and presumably other RedHat derivative, DES is no longer available for net-snmp communication. DES or Data Encryption Standard is an old encryption standard and has been superseded by AES.

So when you try running snmpwalk with “-x DES” option, you get the following error

Invalid privacy protocol specified after -3x flag: DES
USAGE: snmpwalk [OPTIONS] AGENT [OID]

  Version:  5.9.1

To fix the issue, you’ll need to upgrade your devices to AES.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/considerations_in_adopting_rhel_9/assembly_shells-and-command-line-tools_considerations-in-adopting-rhel-9#ref_changes-to-system-management_assembly_shells-and-command-line-tools

https://serverfault.com/questions/1119288/snmp-des-algorithm-no-longer-working

Installing Basic Linux tools on AlmaLinux 9 (tar, wget, htop)

The local team wizard Mark, ran into some issues while trying to setup a system with AlmaLinux 9. Tar wasn’t installed! What?! No worries. We can solve this by just installing tar with dnf. While we are at it, lets install some other helpful utilities.

sudo dnf install -y tar wget htop

Tada! We are back in business.

CentOS – This system is not registered with an entitlement server. You can use subscription-manager to register.

If you are getting the following response when trying to use the yum or dnf command,

This system is not registered with an entitlement server. You can use subscription-manager to register.

Try editing the subscription-manager.conf file, and disable it by changing enable=1 to enable=0

sudo nano /etc/yum/pluginconf.d/subscription-manager.conf

After you may run

yum clean

That should take care of the problem.

https://serverfault.com/questions/764900/how-to-remove-this-warning-this-system-is-not-registered-to-red-hat-subscriptio

https://sahlitech.com/entitlement-server-fix/

Named Error “option ‘dnssec-enable’ no longer exists”

After a recent update, the named service stopped working. When manually trying to restart the service it would just fail.

Running the named-checkconf command

sudo /usr/bin/named-checkconf -z /etc/named.conf

Returned the following error.

/etc/named.conf:37: option 'dnssec-enable' no longer exists

Looks like what happened is they removed the dnssec-option, but enabled it by default

The dnssec-enable option has been obsoleted and no longer has any effect. DNSSEC responses are always enabled if signatures and other DNSSEC data are present. [GL #866]

https://bind9.readthedocs.io/en/v9_16/notes.html#id118

So the option is invalid now and not needed.

To fix the issue, simply open up the /etc/named.conf file and delete the line that has ‘dnssec-enable’

Install and Configure Fail2ban on Fedora/CentOS/RedHat

The following is a very basic guide for setting up Fail2ban for SSH.

Install and basic config

Install Fail2ban

sudo dnf install fail2ban

You may need to install the epel repo

sudo yum install epel-release

Configure to run on system boot

sudo systemctl enable fail2ban

Start Fail2ban service

sudo systemctl start fail2ban

Copy config file with

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Modify the config file

nano /etc/fail2ban/jail.local

Uncomment the following line and add any IPs that need to be whitelisted

ignoreip = 127.0.0.1/8 ::1 192.168.1.20

Save the file and restart Fail2Ban

sudo systemctl restart fail2ban

Configuring Fail2Ban for SSH

Create a new jail file in /etc/fail2ban/jail.d/ called sshd.local

nano /etc/fail2ban/fail.d/sshd.local

Add the following. Note: if you are using a custom ssh port, change “port = ssh” to “port = portnumber”

[sshd]
enabled = true
port = ssh
action = iptables-multiport
logpath = /var/log/secure
maxretry = 5
bantime = 300

Restart Fail2ban

sudo systemctl restart Fail2ban

You can list the firewall rules to verify that an IP gets banned.

iptables -S | grep ipaddress

Unbanning an IP Address

You can unban an IP address with the following command.

sudo fail2ban-client set sshd unbanip 192.168.1.100

You can check out the following link for more information

https://www.redhat.com/sysadmin/protect-systems-fail2ban

Fresh Install of Ubuntu Server Doesn’t use all Disk Space

For some reason Ubuntu server didn’t use all the available disk space while installing. Thankfully this is an easy fix.

sudo lvextend –resizefs -l +100%FREE ubuntu-vg/ubuntu-lv

If the Volume Group or LVM Volume is different, you will need to change the name in the above command. You can use the “sudo pvdisplay” and “sudo lvdisplay” to show you details about your volumes.

https://unix.stackexchange.com/questions/664486/lvm-root-partition-only-uses-half-the-volume-size

SFTP Server – Configure Folder to be used by two users

Bob is the companies local Linux administrator. He has been tasked with creating a secure shared SFTP folder so members in the R&D department can securely collaborate on “The New Project”.

Bob immediately recognizes a potential difficulty. If Steve and John are working on a prototype, how will John be able to edit Steve’s file if the user permissions are set to only allow John to read?

Bob first goes to the break room to locate a coffee mug.

After consulting Google and the man pages for sftp, sftp-server, sshd_config, sshd he found out what he needed to do.

  1. Create directory for the share
  2. Create a user group
  3. Create the individual users and add them to the user group
  4. Modify the sshd_config
  5. Restart the SSHD service and verify that it works

Create Directory for SFTP Share Directory

First Bob needed a directory to hold the R&D files.

mkdir /sftp/rdshare
mkdir /sftp/rdshare/files/
chown 755 /sftp/rdfiles

For some reason, he ran into issues with the folder getting set to the 775 permission which caused issues with logging in. Manually changing it to 755 fixed that issue.

Create User Group

Now Bob needs a user group to add everyone to.

sudo groupadd rdsftp

Now on to creating the users. Since we are just using the accounts for SFTP, we are setting the nologin option. None of these users will be able to use ssh to log on to the server.

sudo useradd -g rdsftp -s /sbin/nologin -M sftpadmin
passwd sftpadmin

Repeat for John, Steve, Jill, etc…

Use the sftpadmin user as an “admin” user and change the “home” directory permissions

chown -R adminuser:rdsftp /sftp/rdfiles

Modify sshd_config file

There are a couple things that need to be changed in the sshd_config file to make this all work.

sudo vi /etc/sshd_config

At the bottom of the file, Bob adds

# R&D SFTP share settings
Match Group rdsftp
        ChrootDirectory /sftp/rdshare/          # <- chroots the users into this directory
        ForceCommand internal-sftp -u 0002      # <- -u for umask.  Needed so users have write permissions for all files

This will chroot all the users into the /sftp/rdshare directory which makes /sftp/rdshare the users / directory.

The -u umask option is the secret for getting all the users to manage all the files. Without it, John would not be able to update Steve’s inventory file.

Restart services and test

Now we can restart the ssh server

sudo systemctl resart sshd

And verify that john can log in.

sftp john@localhost

Any existing sessions will need to be terminated for the changes to take effect.

Further reading.

https://askubuntu.com/questions/982123/multiple-owner-of-same-folder
https://www.tothenew.com/blog/how-to-set-up-shared-folderrepository-between-two-or-more-users-on-linux/
https://medium.com/linuxstories/linux-how-to-setup-an-sftp-server-37e6fb91649b
https://linuxandevops.wordpress.com/2017/07/30/ssh-scp-sftp-connections-and-file-permissions-part-2/