How to speed up the scanning of large networks ranges

The article “How to find out local IP addresses of ISP” discusses scanning ISP users on local networks such as 10.0.0.0/8, 172.16.0.0/12, 100.64.0.0/10. This is the only possible way to scan for hosts that are behind NAT. But the problem is that these networks are large – they have a lot of IP addresses and scanning, especially over Wi-Fi (and not over the wire), takes days. The article linked to shows some techniques for identifying the ranges of user congestion, which can speed up the process quite well.

In the article “Nmap usage tips” I added a new recipe “Fast scanning of huge networks in nmap”. The method source is referenced, of course I checked it before adding it. And it showed unexpectedly good results. I used it in tandem with Router Scan – so we'll compare the scan time with just one Router Scan versus preparatory steps with Nmap scans.

This option, in fact, is not always suitable – if you have a low quality Internet connection, then such an intensive scan can completely clog the channel (the Internet will be lost and, most likely, many online hosts will be missed).

I'll show you how I used this method with Router Scan, but you can use it to explore local and global subnets for any purpose (search for web servers, search for SSH servers, search for computers with RDP, etc.).

Suppose I want to scan for routers on the 100.64.0.0/10 subnet. If you just enter this range into Router Scan and start scanning, it will take several days.

Note: Router Scan works fine in Linux through Wine in terms of finding vulnerable routers, but if you are using Router Scan on Windows, then you can use the commands below in WSL – they will work fine there.

The main idea behind fast scanning is to ping with a high level of concurrency:

sudo nmap -v -sn -PE -n --min-hostgroup 1024 --min-parallelism 1024 -oX nmap_output.xml 100.64.0.0/10

Options meaning:

  • -sn: do not scan ports, only ping the host
  • -PE: determines if the host is alive via ICMP echo
  • -n: Does not convert IP addresses to domain names.
  • --min-hostgroup 1024: Adjust parallel scan group sizes. The minimum grouping is set to 1024 IP addresses. Nmap has the ability to port scan or version scan multiple hosts in parallel. Nmap does this by dividing the target IP space into groups and then scanning one group at a time. In general, larger groups are more efficient. The downside is that host results can't be provided until the whole group is finished. So if Nmap started out with a group size of 50, the user would not receive any reports (except for the updates offered in verbose mode) until the first 50 hosts are completed. By default, Nmap takes a compromise approach to this conflict. It starts out with a group size as low as five so the first results come quickly and then increases the groupsize to as high as 1024. The exact default numbers depend on the options given. For efficiency reasons, Nmap uses larger group sizes for UDP or few-port TCP scans.
  • --min-parallelism 1024: Control parallelism of queries. This parameter is very important. These options control the total number of probes that may be outstanding for a host group. They are used for port scanning and host discovery. By default, Nmap calculates an ever-changing ideal parallelism based on network performance. If packets are being dropped, Nmap slows down and allows fewer outstanding probes. The ideal probe number slowly rises as the network proves itself worthy. These options place minimum or maximum bounds on that variable. By default, the ideal parallelism can drop to one if the network proves unreliable and rise to several hundred in perfect conditions.
  • -oX nmap_output.xml: Output the result in XML format, filename is nmap_output.xml

After the scan is complete, you can parse the XML document to find out which IP addresses are up.

You can decrease the value of the --min-parallelism option if you notice symptoms such as loss of Internet connection. Wired networks can usually be set to higher values than Wi-Fi. The worse the Wi-Fi signal, the lower the value must be set.

Look at the following scan result data:

Nmap done: 4012855 IP addresses (140507 hosts up) scanned in 2590.61 seconds
           Raw packets sent: 7897197 (221.122MB) | Rcvd: 160371 (6.456MB)

4012855 addresses were scanned in 2590 seconds – that's 43 minutes. Found 140507 hosts online.

Now, let's extract all the IP addresses that are online (status “up”) and save them in the hosts.txt file:

grep -A 2 'up' nmap_output.xml | grep -E -o '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' > hosts.txt

Now we run another nmap scan: we will take all the hosts that are online and check if there are open ports 80, 443, 8080, 1080.

sudo nmap -v -PE -n --min-hostgroup 1024 --min-parallelism 1024 -p 80,443,8080,1080 --open -iL hosts.txt -oX nmap_routers.xml

Result:

Nmap done: 140509 IP addresses (121710 hosts up) scanned in 1044.75 seconds
           Raw packets sent: 1100582 (45.478MB) | Rcvd: 208169 (8.408MB)

The scan was completed even faster – in 17 minutes. The disadvantage of this method is clearly visible here: of the previously identified as working 140509 hosts, 121710 turned out to be online. Due to the features of fast scanning, some packets were lost, and we did not find out that the hosts were on the network.

Let's extract the IP addresses that are online from the file obtained as a result of the second scan:

grep -A 2 'up' nmap_routers.xml | grep -E -o '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' > routers.txt

Let's count the number of hosts that have open at least one of the ports we are interested in:

cat routers.txt | wc -l

I got 3409.

I loaded the IP addresses from the routers.txt file into Router Scan. The scan took 18 minutes, and a total of 1,870 vulnerable routers were found.

Of these, 1135 were previously unknown.

It turned out that I spent a little over an hour (43 + 17 + 18 minutes) instead of waiting for the scan results for several days. This efficiency gain was well worth the effort.

Recommended for you:

Leave a Reply

Your email address will not be published. Required fields are marked *