Troubleshooting DNS CPU Usage on Mikrotik Router

Problem : Lots of CPU utilization. Profile shows a good bit of it is DNS related.

DNS eating CPU on Router

The router is setup to allow DNS to pass through to web servers so rDNS and other records can be looked up and resolved. This is a specific IP block that gets it’s addresses from the router. The firewall rules explicitly allow this address range. We’ll say 192.168.88.0/24, and blocks everything else. This works for the web servers. But why are we still getting a bunch of CPU utilization with DNS?

As it turns out, the firewall rule that allows the server address range also includes routers own address! So we have unintentionally whitelisted DNS access to our router.

To resolve the issue we can add another firewall rule that explicitly blocks DNS traffic to the routers IP address. We are using two rules, one to block TCP and the other UDP.

ip firewall filter add chain=input dst-address=192.168.88.1 protocol=6 dst-port=53 in-interface-list=WAN action=drop
ip firewall filter add chain=input dst-address=192.168.88.1 protocol=17 dst-port=53 in-interface-list=WAN action=drop

Rules 6 & 7 are the two new rules we just applied. 14 & 15 block input to the router, however rules 8 & 9 inadvertently allowed access to the router’s public IP.

Firewall Rules for Router

The Result? Our CPU usage dropped!

CPU Usage dropped after adding DNS firewall rules.

Quite dramatically too as the following LibreNMS screenshot shows.

LibreNMS CPU graph showing the overall CPU utilization improvement

For more information about DNS Amplification attacks, refer to the following links.

https://ask.wireshark.org/question/6865/dns-amplification-attack/
https://security.stackexchange.com/questions/237127/why-would-hackers-attack-a-dns-server-with-a-dos

Named Debugging Levels

In the following command, x should be the debug level number

named -g -d x

Example,

named -g -d 3

Following info taken from here.
https://docstore.mik.ua/orelly/networking/dnsbind/ch12_01.htm

12.1.1 What Information Is at Each Level?

Here is a list of the information that each debugging level will give. The debugging information is cumulative; for example, level 2 includes all level 1’s debugging information. The data are divided into the following basic areas: starting up, updating the database, processing queries, and maintaining zones. We won’t cover updating the name server’s internal database – problems always occur elsewhere. However, what the name server adds or deletes from its internal database can be a problem, as you’ll see in Chapter 13, Troubleshooting DNS and BIND .

Level 1

The information at this level is necessarily brief. Name servers can process lots of queries, which can create lots of debugging output. Since the output is condensed, you can collect data over long periods. Use this debugging level for basic startup information and for watching query transactions. You’ll see some errors logged at this level, including syntax errors and DNS packet formatting errors. This level will also show referrals.

Level 2

Level 2 provides lots of useful stuff: it lists the IP addresses of remote name servers that are used during a lookup, along with their round trip time values; it calls out bad responses; and it tags a response as to which type of query it is answering, a SYSTEM (sysquery) or a USER query. When you are tracking down a problem with a secondary server loading a zone, this level shows you the zone values – serial number, refresh time, retry time, expire time, and time left – as the secondary checks if it is up-to-date with its master.

Level 3

Level 3 debugging becomes much more verbose because it generates lots of messages about updating the name server database. Make sure you have enough disk space if you are going to collect debugging output at level 3 or above. At level 3, you’ll also see: duplicate queries called out, system queries generated (sysquery), the names of the remote name servers used during a lookup, and the number of addresses found for each server.

Level 4

Use level 4 debugging when you want to see the query and response packets received by the name server. This level also shows the credibility level for cached data.

Level 5

There are a variety of messages at level 5, but none of them are particularly useful for general debugging. This level includes some error messages, for example, when a malloc() fails, and a message when the name server gives up on a query.

Level 6

Level 6 shows you the response sent to the original query.

Level 7

Level 7 shows you a few configuration and parsing messages.

Level 8

There is no significant debugging information at this level.

Level 9

There is no significant debugging information at this level.

Level 10

Use level 10 debugging when you want to see the query and response packets sent by the name server. The format of these packets is the same format used in level 4. You wouldn’t use this level very often, since you can see the name server response packet with nslookup .

Level 11

There are only a couple of debugging messages at this level, and they are in seldom-traversed code.

UBNTMOD check range of IP addresses and see if they resolve DNS

The following uses the ubntmod.sh script to check a device then report if it is resolving DNS or not. ip=”192.168.1.” specifies the first part of the ip, the “for ((i=1; i<=254;i++))” tells it to go from 192.168.1.1 – 192.168.1.254, change the beginning and ending number if you want to change the ip range.

ip="192.168.1." && for ((i=10; i<=30;i++)); do if ( fping ${ip}$i -r1 | grep alive); then ./ubntmod.sh -i ${ip}${i} -e ; else echo ${ip}$i not alive; fi ; done 

Broken out for easier reading.

ip="192.168.1." 
for ((i=10; i<=30;i++))
do
if ( fping ${ip}$i -r1 | grep alive); then
./ubntmod.sh -i ${ip}${i} -e
else echo ${ip}$i not alive
fi
done

If the script is able to log into the device and resolve DNS you should get

192.168.1.1 Resolves DNS

Check if Mikrotik is an Open DNS Resolver

https://www.openresolver.com

You can test if a router is acting as an open DNS resolver by running the following command from a Linux terminal. If you need to install dig, refer to here for Debian/Ubuntu and here for RPM/CentOS/Fedora Distros.

Replace 192.168.88.1 with the host you want to test against.

dig +short test.openresolver.com TXT @192.168.88.1

If you receive the following

"open-resolver-detected"

The router is acting as an open resolver.

If you get

;; connection timed out; no servers could be reached

Then you are unable to use that router to resolve DNS.

Example running the command against a Mikrotik router with Remote DNS turned on Then adding a firewall rule to block unwanted request.

bob@localhost:~$ dig +short test.openresolver.com TXT @192.168.88.1
"open-resolver-detected"
bob@localhost:~$ 
<<-- Put firewall rule on router -->>
bob@localhost:~$ dig +short test.openresolver.com TXT @192.168.88.1
;; connection timed out; no servers could be reached  
bob@localhost:~$ 

Extra notes

If you have firewall rules allowing your IP address to use the router for DNS, then the above command to test will show it as an Open Resolver. Ideally you would want a connection from the outside to test. Or you can use this link and test it from the website. https://www.openresolver.com

cPanel unable to access locally hosted domains

Check and verify that DNS is not being blocked upstream by a firewall. Behavior is weird, the server can get out to the internet, access to the servers IP address is available, but can’t ping domains that are locally hosted. Are also unable to ping the domains from the internet in.

To resolve the issue either disable the DNS firewall rules, or better yet add some rules to allow access to the cPanel server.

Install dig on Ubuntu, Debian or Kali Linux

install dig
Help options for dig


Dig is a DNS lookup utility.  It is included in most Linux distributions by default, but if it isn’t you can easily install dig with the following command.

The dig utility is apart of the dnsutils package

sudo apt-get install dnsutils -y

After it is installed, we can verify that it is working with

dig -v

For more information on how to use dig, refer to the following link.

https://www.howtogeek.com/663056/how-to-use-the-dig-command-on-linux/

The following is copied and pasted from the dig man page.

NAME
       dig - DNS lookup utility

SYNOPSIS
       dig [@server] [-b address] [-c class] [-f filename] [-k filename] [-m] [-p port#] [-q name]
           [-t type] [-v] [-x addr] [-y [hmac:]name:key] [[-4] | [-6]] [name] [type] [class]
           [queryopt...]

       dig [-h]

       dig [global-queryopt...] [query...]

DESCRIPTION
       dig is a flexible tool for interrogating DNS name servers. It performs DNS lookups and
       displays the answers that are returned from the name server(s) that were queried. Most DNS
       administrators use dig to troubleshoot DNS problems because of its flexibility, ease of use
       and clarity of output. Other lookup tools tend to have less functionality than dig.

       Although dig is normally used with command-line arguments, it also has a batch mode of
       operation for reading lookup requests from a file. A brief summary of its command-line
       arguments and options is printed when the -h option is given. Unlike earlier versions, the
       BIND 9 implementation of dig allows multiple lookups to be issued from the command line.

       Unless it is told to query a specific name server, dig will try each of the servers listed
       in /etc/resolv.conf. If no usable server addresses are found, dig will send the query to the
       local host.

       When no command line arguments or options are given, dig will perform an NS query for "."
       (the root).

       It is possible to set per-user defaults for dig via ${HOME}/.digrc. This file is read and
       any options in it are applied before the command line arguments. The -r option disables this
       feature, for scripts that need predictable behaviour.

       The IN and CH class names overlap with the IN and CH top level domain names. Either use the
       -t and -c options to specify the type and class, use the -q the specify the domain name, or
       use "IN." and "CH." when looking up these top level domains.

SIMPLE USAGE
       A typical invocation of dig looks like:

            dig @server name type

       where:

       server
           is the name or IP address of the name server to query. This can be an IPv4 address in
           dotted-decimal notation or an IPv6 address in colon-delimited notation. When the
           supplied server argument is a hostname, dig resolves that name before querying that name
           server.

           If no server argument is provided, dig consults /etc/resolv.conf; if an address is found
           there, it queries the name server at that address. If either of the -4 or -6 options are
           in use, then only addresses for the corresponding transport will be tried. If no usable
           addresses are found, dig will send the query to the local host. The reply from the name
           server that responds is displayed.

       name
           is the name of the resource record that is to be looked up.

       type
           indicates what type of query is required — ANY, A, MX, SIG, etc.  type can be any valid
           query type. If no type argument is supplied, dig will perform a lookup for an A record.


					

Modifying DNS Entry for Domain

Search /etc/named.conf to find the zone file for the domain.

Find the domain name and see where the zone file is.  Example zone block.

zone "incredigeek.com" in {
 type master;
 file "/var/named/mzones/incredigeek.com.hosts";
 allow-query { any; };
 forwarders {};
};

The file is /var/named/mzones/incredigeek.com.hosts

Edit your zone file by opening it up in a text editor.

Example.  Text in bold added for comments.

$TTL 21600
$ORIGIN com.
incredigeek IN SOA dns1.dns-server.com. dns2.dns-server.com.(
 0000147 ; serial  <- This needs to be incremented so it is greater than the previous version of this file
 43200 ; refresh (12 hours)
 7200 ; retry (2 hours)
 604800 ; expire (7 days)
 21600 ) ; minimum
 NS dns1.dns-server.com.
 NS dns2.dns-server.com.
 300 A 10.0.0.11   <- A record for root domain
$ORIGIN incredigeek.com.
localhost IN A 127.0.0.1
www 300 IN A 10.0.0.11  <- www subdomain A record
login 300 IN A 10.0.0.12 <- another subdomain A record

Save file and reload Bind

On FreeBSD

rndc reload incredigeek.com

you can reload everything with

rndc reload

On Fedora/CentOS/REHL

service named reload

On Ubuntu/Debian

service bind9 restart

You may need to reload Bind on any slave servers

Add local DNS entry in Windows

Windows has a similar file like the Linux /etc/hosts file.  It is located in

Windows\System32\drivers\etc\hosts

You’ll need to have administrator privileges to modify the file.  So you can launch notepad as Administrator then open the file.  If you don’t see the file make sure the drop down across from “File name:” is set to “All Files”.

You can add entry’s just like you would on Linux

192.168.22.12   hostname