How to enable Ping Watchdog on Ubiquiti AirOS from Command Line

Ping Watchdog is a feature that will automatically reboot a device if the specified IP address is unreachable.

Here is a quick run down on enabling Ping Watchdog on Ubiquiti Radios from the command line.

1. SSH into the radio

ssh ubnt@192.168.1.20

2. Edit the config file

vi /tmp/system.cfg

Find the lines that start with pwdog

At a minimum, you will need to change the following two options:

  • pwdog.status to enabled
  • pwdog.host to the IP you want to ping. Add this line if it does not exist.
pwdog.delay=300
pwdog.host=192.168.1.1
pwdog.period=300
pwdog.retry=3
pwdog.status=enabled

You can adjust the other options to your desired taste.

Exit vi by hitting esc then typing in wq then hit Enter

3. Save configuration

Finally, save the configuration changes with

/usr/etc/rc.d/rc.softrestart save

How to Set up a PowerDNS Recursor

The following are the steps needed to install a PowerDNS recursor on RHEL, Fedora, Rocky Linux, or AlmaLinux

Install from package manager with

yum install pdns-recursor

Allow DNS through Firewall

sudo firewall-cmd --add-service=dns --permanent

Configure the `/etc/pdns-recursor/recursor.conf` file. The local-address is the DNS recursor, the allow-from, are the addresses you would like to allow access to

local-address=192.0.1.2
allow-from=192.0.0.0/16, 10.0.0.0/8

Start and enable the `pdns-recursor` service

systemctl enable --now pdns-recursor

https://doc.powerdns.com/recursor/getting-started.html

Ansible not working on RockyLinux 8, AlmaLinux 8, RHEL 8

[WARNING]: Unhandled error in Python interpreter discovery for host localhost: Expecting value: line 1
column 1 (char 0)

https://github.com/ansible/ansible/issues/83357

Ansible 2.17 moved to using Python 3.7. This causes issues with systems that use Python 3.6 (i.e., RHEL 8 based distros). Unfortunately, you can’t just upgrade Python either, as 3.6 is used in system tools such as DNF/YUM.

There are two options.

  1. Upgrade to a RHEL 9 based distribution
  2. Use Ansible 2.16

Ansible 2.16 should be the default installed version on RHEL 8 based distros.

Migrate CentOS 7 to AlmaLinux 8

The steps are taken from this page https://wiki.almalinux.org/elevate/ELevating-CentOS7-to-AlmaLinux-9.html

To upgrade to AlmaLinux 9, you will need to migrate to AlmaLinux 8 first.

sudo curl -o /etc/yum.repos.d/CentOS-Base.repo https://el7.repo.almalinux.org/centos/CentOS-Base.repo
sudo yum upgrade -y

Once yum finishes, reboot

sudo reboot

Now install elevate-release and leapp packages

sudo yum install -y http://repo.almalinux.org/elevate/elevate-release-latest-el$(rpm --eval %rhel).noarch.rpm
sudo yum install -y leapp-upgrade leapp-data-almalinux

Let’s run a pre upgrade check to see if there are any errors.

sudo leapp preupgrade

If everything checked out ok, proceed with the upgrade.

sudo leapp upgrade

Once finished, reboot.

sudo reboot

Once the system is booted, verify it upgraded.

cat /etc/*release

Problems

https://wiki.almalinux.org/elevate/ELevate-frequent-issues

LUKS

If you are using LUKS and encounter an error, check out the following link.

https://www.it-hure.de/2024/02/update-alma-rhel-with-leapp-and-luks/

You can disable the check with the following command.

rm -rf /usr/share/leapp-repository/repositories/system_upgrade/common/actors/inhibitwhenluks

More space needed on / filesystem

Try setting LEAPP_OVL_SIZE to 4096

export LEAPP_OVL_SIZE=4096

Then launch the upgrade with

sudo --preserve-env leapp upgrade

https://forums.almalinux.org/t/at-least-48mb-more-space-needed-on-the-filesystem-running-leapp-upgrade/3808/2

You can also try removing files to create more free space.

https://github.com/oamg/leapp/issues/778

Upgrading to 8.8 (Or 8.4)

For the same reason, we recommend upgrading your CentOS 7 machine to AlmaLinux OS version 8.8. To do so, you need to navigate to the /etc/leapp/files/ directory and edit the leapp_upgrade_repositories.repo to lower the AlmaLinux version in baseurl/mirror to 8.8.

The 8.8 repositories are archived. To upgrade to AlmaLinux 8.8, change ‘/etc/leapp/leapp_upgrade_repositories.repo” to the following.

[almalinux8-BaseOS]
name=AlmaLinux 8 - BaseOS
baseurl=https://vault.almalinux.org/8.8/BaseOS/$basearch/os/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-AlmaLinux

[almalinux8-AppStream]
name=AlmaLinux 8 - AppStream
baseurl=https://vault.almalinux.org/8.8/AppStream/$basearch/os/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-AlmaLinux

[almalinux8-PowerTools]
name=AlmaLinux 8 - PowerTools
baseurl=https://vault.almalinux.org/8.8/PowerTools/$basearch/os/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-AlmaLinux

[almalinux8-HighAvailability]
name=AlmaLinux 8 - HighAvailability
baseurl=https://vault.almalinux.org/8.8/HighAvailability/$basearch/os/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-AlmaLinux

[almalinux8-ResilientStorage]
name=AlmaLinux 8 - ResilientStorage
baseurl=https://vault.almalinux.org/8.8/ResilientStorage/$basearch/os/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-AlmaLinux

[almalinux8-Extras]
name=AlmaLinux 8 - Extras
baseurl=https://vault.almalinux.org/8.8/extras/$basearch/os/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-AlmaLinux

If you have issues, you may consider upgrading to 8.4 first. To do that, change 8.8 to 8.4, and comment out the ResilientStorage section.

GPG Key failing after upgrade

Try importing the AlmaLinux 8 GPG key

rpm --import https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux

https://almalinux.org/blog/2023-12-20-almalinux-8-key-update

Differences between RTO, RPO, MTBF, and MTFF

Here is a quick overview of the differences between, RTO, RPO, MTBF, and MTFF.

NameMeaning
RTO (Recovery Time Objective)Time it takes to recover from a disruption, system failure, data loss etc.
RPO (Recovery Point Objective)How much data can you afford to loose? If RPO is 24 hours, then backups need to be performed daily.
MTBF (Mean Time Between Failures)Time between failures. Use for repairable systems
MTTF (Mean Time to Failure)Time before system fails. Use for nor repairable systems.

http://techtarget.com/whatis/definition/recovery-point-objective-RPO

http://rubrik.com/insights/rto-rpo-whats-the-difference

http://en.m.wikipedia.org/wiki/Mean_time_between_failures

Securely Delete Files on Linux

We can use srm to securely delete files on Linux.

Install srm with

sudo apt install secure-delete

We can now securely delete files by running

srm filetodelete.txt

# srm --help
srm v3.1 (c) 1997-2003 by van Hauser / THC <vh@thc.org>

Syntax: srm [-dflrvz] file1 file2 etc.

Options:
-d ignore the two dot special files "." and "..".
-f fast (and insecure mode): no /dev/urandom, no synchronize mode.
-l lessens the security (use twice for total insecure mode).
-r recursive mode, deletes all subdirectories.
-v is verbose mode.
-z last wipe writes zeros instead of random data.

srm does a secure overwrite/rename/delete of the target file(s).
Default is secure mode (38 writes).
You can find updates at http://www.thc.org

Other links for securely erasing drives.
https://www.tomshardware.com/how-to/secure-erase-ssd-or-hard-drive

Top 8 Nmap options

Here are 8 excellent Nmap options, what they do, and why you would use them.

Most of the options can be run together. You will normally want to perform scans with administrator or root privileges.

OptionWhat is doesWhy you would use
1.-snNo port scanHelpful for quickly discovering hosts that are up
2.-iL file.lstScan IP addresses in file.lstHelpful if you already have a list IP addresses to scan
3. -nSkip reverse DNS lookupThis can help speed up scanning
4.-PnPretend host is upUse when hosts have Ping disabled. e.g. Windows
5. -OOS detectionUse to detect OS version
6.-T4Speed up scanIncreases scan speed (Default is -T3)
7.-AAggressive scan optionsShorthand option. Enables OS detection (-O), version Scanning (-sV), script scanning (-sC), and runs a traceroute
8.-oA filenameSave output to ALL formatsThis saves the output to separate files for XML and grepable formats
Nmap table: 8 common options.

A Quick Overview of SAML

SAML stands for Security Assertion Markup Language. It allows for Single Sign On or SSO to a service.

There are three entities or roles involved when using SAML to sign into a service.

  1. Principal or Subject: a.k.a. you, or the person or service logging in.
  2. Service Provider (SP): This is the service you are accessing. It could be email, a website, etc.
  3. Identity Provider (IdP): This is the entity response for authenticating the Principal.

As an example, let’s say you want to log into a new website utilizing your email SSO credentials. You click the SSO login button, you are redirected to the IdP to login. Once authenticated, your device will receive a token which is then passed back to the Service Provider and allows you access to the new website.

This is a very simplified version of what happens when you login using SAML. It may be helpful to know that the Service Provider and the Identify Provider will have needed to be configured to work together before the user attempts to log in.

https://auth0.com/blog/how-saml-authentication-works

https://infosec.mozilla.org/guidelines/iam/saml.html

Locate large files on Linux

Show size of directories. The -h option prints the size in human readable format.

du -h --max=1 ./

We can use sort and tail to filter and only show the 10 largest files and directories. The -a option shows all files and directories.

du -ah ./ | sort -h | tail -n10

We can use the find command to show all files over xMB. In this case 100MB

fine . -type f -size +100M -print

https://linuxhandbook.com/find-biggest-files-linux
https://linuxize.com/post/find-large-files-in-linux/

Hacking Hashed SSH known_hosts file

On Ubuntu, by default, the hosts in .ssh/known_hosts are hashed. This can theoretically help with security. If an attacker compromises a host, they will not be able to tell the IP addresses of other hosts in the known_hosts file.

https://security.stackexchange.com/questions/56268/ssh-benefits-of-using-hashed-known-hosts

Anatomy of the hashed known_hosts

Here is an example of a hashed entry in the known_hosts file.

|1|ma8KL2XrNYkNnknf68N4IuZ+c+I=|PmR+n2i0/epUGZZh2S+LB6OaowQ= ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEjqG8/el8c669FxcvEw5mMfDRTDxsjgLiz44dCTtchs

There are three main parts.

The first part ma8KL2XrNYkNnknf68N4IuZ+c+I= is the salt to use.

PmR+n2i0/epUGZZh2S+LB6OaowQ= This is our hashed IP address/hostname

ssh-ed25519 is the key type

AAAAC3NzaC1lZDI1NTE5AAAAIEjqG8/el8c669FxcvEw5mMfDRTDxsjgLiz44dCTtchs Is the public SSH key of the remote host.

SSH-KEYSCAN

We can use ssh-keyscan to check the keys of hosts. The -t ssh-ed25519 option only shows ed25519 keys. Remove or change to show all key types e.g. RSA/DSA

For example:

┌──(kali㉿localhost)-[~]
└─$ ssh-keyscan -t ssh-ed25519 localhost
# localhost:22 SSH-2.0-OpenSSH_9.6p1 Debian-3
localhost ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEjqG8/el8c669FxcvEw5mMfDRTDxsjgLiz44dCTtchs

We can compare the SSH public key with the one in our known_hosts file to verify we have the correct host.

As a side note, we can also use the -H option to show us a hashed version. The salt changes each time it is run, so it is not useful for comparing the hashed IP address.

Example:

┌──(kali㉿localhost)-[~]
└─$ ssh-keyscan -H -t ssh-ed25519 localhost
# localhost:22 SSH-2.0-OpenSSH_9.6p1 Debian-3
|1|j2j9iv/GkPfnG9Yv4WzJsy/L1pc=|wethKgsGBH0Mi+rFW3zSNSWiGso= ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEjqG8/el8c669FxcvEw5mMfDRTDxsjgLiz44dCTtchs

Hacking known_hosts hashes

There are a few different techniques that can be used to identify known hosts IP addresses even if they are hashed.

https://stackoverflow.com/questions/427979/how-do-you-extract-ip-addresses-from-files-using-a-regex-in-a-linux-shell

Search through bash history

history | egrep '([0-9]{1,3}\.){3}[0-9]{1,3}'

Example output

┌──(kali㉿localhost)-[~]
└─$ history | egrep '([0-9]{1,3}\.){3}[0-9]{1,3}' | head -n2
1 ssh kali@127.0.0.1

Check if SSH Public Key is on Shodan

Since the SSH public key is um, well, public, we can search for it on Shodan to see if it’s a known public server. https://www.shodan.io

Copy the public ssh key from the known_hosts file. It is the last portion of the line i.e.
AAAAC3NzaC1lZDI1NTE5AAAAIEjqG8/el8c669FxcvEw5mMfDRTDxsjgLiz44dCTtchs

|1|ma8KL2XrNYkNnknf68N4IuZ+c+I=|PmR+n2i0/epUGZZh2S+LB6OaowQ= ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEjqG8/el8c669FxcvEw5mMfDRTDxsjgLiz44dCTtchs

Brute force

Since the address space for IPv4 is fairly small, and the private IP address space even smaller, brute forcing all the addresses is perfectly feasible.

Here is a quick example on how you would hash an IP address. Commands are taken from the above Stack Exchange link.

Take the salt and put it into a variable

key=`echo j2j9iv/GkPfnG9Yv4WzJsy/L1pc= | base64 -d | xxd -p`

Next we can run the following command to hash the result. The IP (127.0.0.1) is where we would want to enumerate the IP address.

echo -n "127.0.0.1" | openssl sha1 -mac HMAC -macopt hexkey:$key|awk '{print $2}' | xxd -r -p | base64

The output is PmR+n2i0/epUGZZh2S+LB6OaowQ= which is the correct hash.

Automating should be fairly simple.

A note on SSH ports. If the host is using a non standard ssh port, you will need to update the above command with the port, but the address needs to be wrapped in square brackets []

echo -n "[127.0.0.1]:2222" | openssl sha1 -mac HMAC -macopt hexkey:$key|awk '{print $2}' | xxd -r -p | base64

Use ssh-keyscan

A final way we can discover known-hosts, is by using ssh-keyscan. The man page says the following

ssh-keyscan is a utility for gathering the public SSH host keys of a number of hosts. It was designed to aid in building and verifying ssh_known_hosts files

ssh-keyscan uses non-blocking socket I/O to contact as many hosts as possible in parallel, so it is very efficient. The keys from a domain of 1,000 hosts can be collected in tens of seconds, even when some of those hosts are down or do not run sshd(8). For scanning, one does not need login access to the machines that are being scanned, nor does the scanning process involve any encryption.

Hosts to be scanned may be specified by hostname, address or by CIDR network range (e.g. 192.168.16/28). If a network range is specified, then all addresses in that range will be scanned.

This makes it super convenient to do a network scan using ssh-keyscan and then compare the public ssh keys with those in the known_hosts file.

Example:

ssh-keyscan 192.168.0.0/16

To scan all private IP ranges (RFC1912), we just run the scan with all three IP ranges

ssh-keyscan 192.168.0.0/16
ssh-keyscan 172.12.0.0/12
ssh-keyscan 10.0.0.0/8