How to speed up the scanning of numerous web sites with Interlace

How to accelerate web application scanning via multi-threading

Typical web application scanners that are used security audit of sites consume quite a few resources: I mean CPU, RAM, and even bandwidth. However, they work for a long time – the bottleneck is the speed of the response of the remote host. That is, it turns out that we can not particularly affect the scanning speed.

If you need to use, for example, Nikto to scan a few hundred sites (hosts), then this is guaranteed to take a very long time. If you run each scan after the previous one, then you will need to take a vacation at work. You can run them simultaneously in different terminals, but manually, it is also quite tedious and it is important not to get confused yet with the results obtained – this is very easy with this amount of tasks.

As a better solution, the idea comes to write a script to automate the process. But not everyone can write scripts. In addition, you need not just write commands that start simultaneously (most likely, it will just cause the operating system to freeze) – you need some kind of control over the number of tasks performed and the “starting up” of new ones as their previous ones complete. That is, you need not just to write a script – you need to write a GOOD script.

Fortunately for us, there is already a ready tool that does exactly what you need. Moreover, there are even two such tools:

• Interlace is a tool created by a penetration tester for simultaneous scanning of numerous targets. Due to its purpose, it is very easy to use just for scanning a very large number of targets.
• Parallel is a universal program for running in parallel any utilities and tools. It is even more flexible, but it also requires a little more effort to study its syntax.

This instruction is divided into two parts – and the first part is devoted to Interlace. In it, we will learn how to use Interlace, see examples of computer resource consumption and consider several practical examples of scanning in many threads and subsequent analysis of the collected data.

How to install Interlace

To install Interlace on Kali Linux

git clone https://github.com/codingo/Interlace
cd Interlace/
sudo python3 setup.py install

To install Interlace on BlackArch

sudo pacman -S interlace

Interlace tutorial

The essence of Interlace work is that it runs a command many times using a given template. This template contains the executable file, as well as replacement strings, which in the real command will be replaced, for example, with a target.

We begin with an introduction to variables (replacement strings) that can be used in a command template. In the left column is a variable that will be replaced in a real command. And in the right column its description and where its values come from:

Run web application scanners in many threads

So, we already got acquainted with the options from the table, and to run interlace, in addition we need to know about the -c option (after which the command template comes) and the -cL option (with which the file containing many command templates is specified).

Suppose we want to run Nikto for several purposes.

The syntax for running nikto itself in one thread:

nikto --host SITE

For the results not to be lost, save them for each scan in a separate file:

nikto --host SITE > ./SITE-nikto.txt

To build an interlace program template, we need to use the _target_ replacement string (which denotes sites from the list of targets), we get:

nikto --host _target_ > ./_target_-nikto.txt

Now we combine the interlace command itself:

interlace -tL ./targets.txt -threads 5 -c "nikto --host _target_ > ./_target_-nikto.txt" -v

Where:

• interlace – Interlace executable
• -tL ./targets.txt – list of files with targets
• -threads 5 – means to run scans in 5 threads
• -c "nikto --host _target_ > ./_target_-nikto.txt" – the command template we made up a little earlier
• -v – increases the verbality of the output (more information is shown)

Suppose in the targets.txt file we have only two targets:

bugcrowd.com
hackerone.com

Then, as a result of the work of the previous command, two scans will be launched:

nikto --host hackerone.com > ./hackerone.com-nikto.txt
nikto --host bugcrowd.com > ./bugcrowd.com-nikto.txt

Consider another simple example, and then move on to more complex cases. I’ll use WhatWeb as a scanning program – just to not wait for a very slow Nikto.

I'll start by compiling a list of targets:

curl -s https://top.mail.ru/Rating/Computers-Programming/Today/Visitors/{1..10}.html | grep -E '><a class="t90.*</a>' | sed 's/<br \/><a class="t90 t_grey" href="//' | grep -E -o '>.*<' | sed 's/>//' | sed 's/<//' > prog.txt
cat prog.txt | sed 's/https:\/\///' | sed 's/http:\/\///' | sed 's/\///' > prog_sites.txt

There are 300 websites on my list:

cat prog_sites.txt | wc -l
300

And I want to check each of these sites using WhatWeb (collects information about the technologies used).

I want to save the results in a whatweb-results folder. I create this folder:

mkdir ./whatweb-results

So, my launch command:

interlace -tL ./prog_sites.txt -threads 100 -c "whatweb _target_ --colour never > ./whatweb-results/_target_-whatweb.txt" -v

I run 100 processes, so the CPU (finally!) on my computer is also fully loaded:

Very quickly – in less than a minute, everything is ready!

Now in the folder whatweb-results we have 300 files – to speed up the analysis, you can, for example, use the search for certain words. In general, the analysis depends on the tool used to collect information. For example, if you used WPScan, you can search for the word ‘vulnerability’ or ‘identified’.

Consider some examples of analysis with grep.

Find a server with old versions of PHP:

grep -E -H 'PHP\[5.1' whatweb-results/*

Output:

whatweb-results/jedit.ru-whatweb.txt:http://jedit.ru [200 OK] Apache[2.2.3], Cookies[PHPSESSID], Country[RUSSIAN FEDERATION][RU], Email[--Rating@Mail.ru], HTTPServer[CentOS][Apache/2.2.3 (CentOS)], IP[188.127.229.153], PHP[5.1.6], PasswordField[edit[pass]], PoweredBy[-drupal,drupal], Script[text/javascript], X-Powered-By[PHP/5.1.6]

There is the only one site power by PHP 5.1. And what about PHP 5.2?

grep -E -H 'PHP\[5.2' whatweb-results/*

Output:

whatweb-results/apps-oracle.ru-whatweb.txt:http://apps-oracle.ru [200 OK] Country[RUSSIAN FEDERATION][RU], Email[--Rating@Mail.ru,Rating@Mail.ru], Google-Analytics [UA-25693489-1], HTTPServer[nginx-reuseport/1.13.4], IP[5.101.152.75], JQuery[1.11.3], MetaGenerator[WordPress 4.4.4], Open-Graph-Protocol[blog], PHP[5.2.17], Script[text/javascript], Title[APPS-ORACLE.RU], UncommonHeaders[link], WordPress[4.4.4], X-Powered-By[PHP/5.2.17], X-UA-Compatible[IE=EmulateIE7]
 
whatweb-results/blog.webitdesign.ru-whatweb.txt:http://blog.webitdesign.ru [301 Moved Permanently] Country[RUSSIAN FEDERATION][RU], HTTPServer[Gentoo Linux][nginx/1.15.8], IP[77.222.56.7], PHP[5.2.17-pl0-gentoo], RedirectLocation[http://www.blog.webitdesign.ru/], X-Powered-By[PHP/5.2.17-pl0-gentoo], nginx[1.15.8]
 
whatweb-results/blog.webitdesign.ru-whatweb.txt:http://www.blog.webitdesign.ru/ [200 OK] All-in-one-SEO-Pack[2.3.13.2], Country[RUSSIAN FEDERATION][RU], Email[--Rating@Mail.ru,Rating@Mail.ru], Frame, HTML5, HTTPServer[Gentoo Linux][nginx/1.15.8], IP[77.222.56.7], JQuery[1.12.4], PHP[5.2.17-pl0-gentoo], Script[javascript,javascript1.1,javascript1.2,javascript1.3,text/javascript], Title[Блог разработчика | Разработка на Oracle, Python], UncommonHeaders[link], WordPress, X-Powered-By[PHP/5.2.17-pl0-gentoo], X-UA-Compatible[IE=edge], nginx[1.15.8]
 
whatweb-results/delphimaster.net-whatweb.txt:http://delphimaster.net [200 OK] Cookies[PHPSESSID], Country[RUSSIAN FEDERATION][RU], Email[Rating@Mail.ru,xayam@yandex.ru], HTTPServer[openresty/1.13.6.2], IP[195.208.1.104], JQuery[1.3.2], OpenSearch[http://delphimaster.net/opensearch.xml], PHP[5.2.17], Script[JavaScript,text/javascript], Title[DMSearch 2.9 - поиск на форумах delphimaster за период 2002.01.08 - 2019.03.10], X-Powered-By[PHP/5.2.17]
 
whatweb-results/delphisite.ru-whatweb.txt:http://delphisite.ru [200 OK] Cookies[SESS05472d46f4cbfa316697cfba96fb5b1d], Country[RUSSIAN FEDERATION][RU], probably Drupal, DublinCore, Email[--Rating@Mail.ru], HTTPServer[nginx], IP[195.114.3.241], PHP[5.2.17], Script[text/javascript], Title[Всё для программирования в среде Delphi | DelphiSite], X-Powered-By[PHP/5.2.17], nginx
 
whatweb-results/gamebox-alex.pp.ua-whatweb.txt:http://gamebox-alex.pp.ua [200 OK] Abyss-Web-Server[2.5.0.0-X2-Win32], Country[RUSSIAN FEDERATION][RU], Email[--Rating@Mail.ru], Google-Analytics[Universal] [UA-65855299-1], HTTPServer[Windows (32 bit)][Abyss/2.5.0.0-X2-Win32 AbyssLib/2.5.0.0], IP[37.61.183.193], PHP[5.2.6], Script[text/javascript], X-Powered-By[PHP/5.2.6]
 
whatweb-results/grafika.me-whatweb.txt:http://grafika.me [200 OK] Apache[2.2.9][mod_ssl/2.2.9], Cookies[SESSb8f3849a2a2de4b277ddee5ed791d52f], Country[ESTONIA][EE], Drupal, Email[Rating@Mail.ru], HTML5, HTTPServer[Debian Linux][Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny9 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g], IP[5.45.114.12], JQuery, OpenSSL[0.9.8g], PHP[5.2.6-1+lenny9][Suhosin-Patch], PasswordField[pass], Script[text/javascript], Title[Компьютерная графика и вычислительная геометрия | Компьютерная графика], X-Powered-By[PHP/5.2.6-1+lenny9]
 
whatweb-results/snakeproject.ru-whatweb.txt:http://snakeproject.ru [200 OK] Apache[2.2.22][mod_fcgid/2.3.8-dev], Cookies[PHPSESSID], Country[RUSSIAN FEDERATION][RU], Email[Rating@Mail.ru], Frame, Google-Analytics[Universal] [UA-36867057-1], HTML5, HTTPServer[Unix][Apache/2.2.22 (Unix) mod_fcgid/2.3.8-dev], IP[81.177.143.36], JQuery, PHP[5.2.17], Script[text/javascript], Title[snakeproject.ru - Python уроки, Python и Django, запросы SQL и T-SQL, настройка cisco, asterisk, freebsd, серверов], Via-Proxy[1.0 1gb-proxy-l24-aux7 (squid/3.1.16)], X-Cache[1gb-proxy-l24-aux7,1gb-proxy-l24-aux7:80], X-Powered-By[PHP/5.2.17]
 
whatweb-results/www.softcraft.ru-whatweb.txt:http://www.softcraft.ru [200 OK] Country[RUSSIAN FEDERATION][RU], Email[Rating@Mail.ru], HTTPServer[openresty/1.13.6.2], IP[195.208.1.100], PHP[5.2.17], Script[text/javascript], Title[SoftCraft: отправная точка], X-Powered-By[PHP/5.2.17]

Let’s find all the sites with WordPress:

grep -E -H 'WordPress' whatweb-results/* | grep -o -E ':[a-z0-9.:/-]+ ' | sed 's/://'

Oh, right now I got an idea… let's save them to a file:

grep -E -H 'WordPress' whatweb-results/* | grep -o -E ':[a-z0-9.:/-]+ ' | sed 's/://' > wordpress.txt
cat wordpress.txt | sed 's/https:\/\///' | sed 's/http:\/\///' | sed 's/\///g' > wordpress_sites.txt

How many did I get?

cat wordpress_sites.txt | wc -l
55

Create a wordpress-results folder:

mkdir wordpress-results

Run the WPScan scan in 20 threads:

interlace -tL ./wordpress_sites.txt -threads 20 -c "wpscan --url _target_ > ./wordpress-results/_target_.txt" -v

Initially CPU was consumed completely, but only for short time. RAM is always enough:

Analysis:

grep -E -H -A 5 'vulnerab' wordpress-results/*

Some results:

Here remarkable is the fact that I need not much time from launch to the get of results!

Scanning targets on multiple ports

If there are several ports in addition to the targets, then all possible combinations of target-port commands will be composed. For example (there are two targets in the targets.txt file):

interlace -tL ./targets.txt -threads 5 -c "nikto --host _target_:_port_ > ./_target_-_port_-nikto.txt" -p 80,443 -v

Will trigger commands:

nikto --host hackerone.com:80 > ./hackerone.com-80-nikto.txt
nikto --host bugcrowd.com:80 > ./bugcrowd.com-80-nikto.txt
nikto --host bugcrowd.com:443 > ./bugcrowd.com-443-nikto.txt
nikto --host hackerone.com:443 > ./hackerone.com-443-nikto.txt

Running multiple commands against multiple targets

Sometimes it is necessary to use more than one tools against multiple targets, therefore you may need to execute several commands. And again interlace hurries to help, due to it, we can launch multiple commands against multiple targets via a single command.

Suppose the list of commands includes testssl.sh, nikto and sslscan, then you can create a file with templates for each command, save it in commands.txt:

nikto --host _target_:_port_ > _output_/_target_-nikto.txt
sslscan _target_:_port_ > _output_/_target_-sslscan.txt
testssl.sh _target_:_port_ > _output_/_target_-testssl.txt

As usual, you can specify a file with targets and each program will be scanned with each program. In the following example, there is only one target (example.com) and two ports – as a result, 6 scanning processes will be launched: two for nikto, sslscan and testssl.sh on ports 80 and 443 for the example.com:

interlace -t example.com -o ~/Engagements/example/ -cL ./commands.txt -p 80,443

The results will be saved in the ~/Engagements/example/ folder.

Adding CIDR support to any program

Some programs do not support specifying IP address ranges – this can be very inconvenient. Thanks to Interlace, this problem can be solved for any tool!

It is enough as a target to indicate the range in the form 192.168.12.0/24 or 192.168.12.1-255 or 192.168.12.*. Interlace understands these notations and automatically expands them into a list of IP addresses:

interlace -t 192.168.12.0/24 -c "vhostscan _target_ -oN _output_/_target_-vhosts.txt" -o ~/scans/ -threads 50

The VHostScan program itself does not support ranges (CIDR notation). But since Interlace will expand the range entry and make a queue of VHostScan commands, each of which will be given one of the IP for scanning, this will make it possible for VHostScan to scan on a range of addresses.

Add support for asterisks in IP address ranges

This is called Glob notation. Suppose we need to start scanning virtual hosts for each target in 192.168.12.*, This can be done with one command:

interlace -t 192.168.12.* -c "vhostscan _target_ -oN _output_/_target_-vhosts.txt" -o ~/scans/ -threads 50

Add support for writing IP ranges using dash

Yes, again. VHostScan does not support dash notation in ranges. Interlace acts like the examples described above, so if you want to scan virtual hosts for all targets in the range 192.168.12.1-15, the following command will do this:

interlace -t 192.168.12.1-15 -c "vhostscan _target_ -oN _output_/_target_-vhosts.txt" -o ~/scans/ -threads 50

How to make a command line utility multi-threaded

It is necessary to clarify that multi-threading is meant not for one target. In fact, such multi-threading is already present in many scanners and brute-force programs. As mentioned above, the speed still rests on the speed of the response of the remote server. This means multi-threading when you need to scan multiple targets.

That is, if you need to scan one target, the scan time of which is 1 minute, then Interlace will not help you to speed up this process. But if you need to scan 10 targets, the scan time is 1 minute each, then Interlace instead of the logical 1 * 10 = 10 minutes scan can make the scan complete in 1 minute.

How many threads will be launched is set with the -threads option. You need to determine a specific value specifically for your computer and for the program that you are multi-threaded – do a few tests and look at the resource consumption.

Exclude hosts from scanning

Using the -e or -eL options, you can make Interlace exclude some targets. These arguments are also compatible with the range notations mentioned above (CIDR, asterisk, and dash).

To start scanning virtual hosts for each target within the range 192.168.12.0/24 but not within the range 192.168.12.0/26, run the command like this:

interlace -t 192.168.12.0/24 -e 192.168.12.0/26 -c "vhostscan _target_ -oN _output_/_target_-vhosts.txt" -o ~/scans/ -threads 50

Running a single command simultaneously on multiple proxies

Note: it does not mean that the first request will be sent through one proxy, the second request through another proxy, etc. It means that the same requests will be sent to the same goal, but through different proxies.

In the following example several independent scans will be run through a proxy for the same targets:

interlace -tL ./targets.txt -pL ./proxies.txt -threads 5 -c "nikto --host _target_:_port_ -useproxy _proxy_ > ./_target_-_port_-nikto.txt" -p 80,443 -v

Note that every use of a repacement string (such as _target_, _port_, and _proxy_) causes commands to be created with all possible combinations.

That is, the total number of launches will be equal to: the number of targets (_target_) * the number of ports (_port_) * the number of proxy (_proxy_).

Search for a domain name on a large IP range

Such speed and comfort of launch lead to the idea of attacks and search methods that seemed impossible before.

For example, quite a life case. Situation: the site behind CloudFlare and search methods (1, 2, 3) did not lead to the disclosure of the real IP.

The action plan is as follows:

1) Given the nature of the site, you can assume the owner and, knowing his other sites, suggest the selected hosting. In the worst situation, you can simply take a list of hosting providers which ignores abuses.

2) (Optional step) If the ranges are very large, then you can scan hosts to find which of them are actually online

3) Using VHostScan with Interlace boost, scan all these ranges for the desired domain name.

Suppose that with this method I need to find the real IP of the hackware.ru site (let's say that it is hidden behind CloudFlare) and there is an assumption that Hostland LTD is used as the hosting. With the help of the All IP of ISPs (it is enough to know any of its IP or just the website address) I get the whole range of IP addresses of the hoster:

185.26.120.0/22
185.37.62.0/24
185.221.212.0/24

I copy them to the hostland.txt file.

I have only three small ranges and the second stage can be skipped. But if the IP range is very wide, then if you wish, you can scan and keep only those that respond to the 80th port. Of course, you can use Nmap, but for you to think about other interesting ways to use Interlace, I will go a little different way. In the following example, I selected only hosts that responded to ping:

interlace -tL hostland.txt -c "timeout 5 ping -c 4 _target_ >/dev/null && echo _target_ >> hostland_live.txt 2>/dev/null" -threads 100

How many IP do I have now:

cat hostland_live.txt | wc -l
391

I talked about VHostScan in the article “How to find web server virtual host”. The most important thing that we need to remember from there is that if you need to find one specific host, then you must also indicate several obviously wrong hosts in the dictionary so that VHostScan has the material for self-learning.

So, I make this file vhosts.dic:

11111111fdsfghfghdfgjggh.ru
hackware.ru
dkdkgfg-df.com
hgjlk-hgmhgj-fdgdfgafdh.org
tttteessst435345.net
dddlsllstlsdfsdfsdhy.site
sssssssssssiiiiiiiittttttt.ru

I create the vhostscan-results folder – in it I will save the results:

mkdir vhostscan-results

I make up the VHostScan command to search the site for all the live IP hoster:

interlace -tL hostland_live.txt -c "vhostscan -t _target_ -w vhosts.dic -oN _output_/_target_-vhosts.txt" -o ./vhostscan-results -threads 100

If the host is found, then the following is displayed:

[+] Most likely matches with a unique count of 1 or less:
    [>] hackware.ru

I will search for the keyword "most likely matches":

grep -E -H 'Most likely matches' vhostscan-results/*

The real IP was found, although there were a lot of false positives.

In general, this is more of a proof-of-concept, as there is still a lot to be corrected – adjust the VHostScan accuracy with options, you may need to add a scan via SSL or scan it in several steps to eliminate false positives. It is likely that instead of this program it is better to use some other or own small script to increase the level of accuracy.

How to convert a range of IP to a list of 1 IP per line

Suppose we have several ranges of IP addresses:

• 185.26.120.0/22
• 185.37.62.0/24
• 185.221.212.0/24

And we just need to create a file in which each IP address from this range will be written on a separate line. With Interlace, this can be done very easy:

interlace -t 185.26.120.0/22,185.37.62.0/24,185.221.212.0/24 -c "echo _target_ >> ip.txt" -threads 100

Check if anyone is lost:

cat ip.txt | wc -l
1536

All Interlace options

Usage:

interlace [-h] (-t TARGET | -tL FILE) [-e EXCLUSIONS | -eL FILE]
                 [-threads THREADS] [-timeout TIMEOUT] (-c COMMAND | -cL FILE)
                 [-o OUTPUT] [-p PORT] [--proto PROTO] [-rp REALPORT]
                 [--no-cidr] [--no-color] [-v | --silent]

Options:

  -h, --help        show this help message and exit
  -t TARGET         Specify a target or domain name either in comma format,
                    CIDR notation, glob notation, or a single target.
  -tL FILE          Specify a list of targets or domain names.
  -e EXCLUSIONS     Specify an exclusion either in comma format, CIDR
                    notation, or a single target.
  -eL FILE          Specify a list of exclusions.
  -threads THREADS  Specify the maximum number of threads to run (DEFAULT:5)
  -timeout TIMEOUT  Command timeout in seconds (DEFAULT:600)
  -c COMMAND        Specify a single command to execute.
  -cL FILE          Specify a list of commands to execute
  -o OUTPUT         Specify an output folder variable that can be used in
                    commands as _output_
  -p PORT           Specify a port variable that can be used in commands as
                    _port_
  --proto PROTO     Specify protocols that can be used in commands as _proto_
  -rp REALPORT      Specify a real port variable that can be used in commands
                    as _realport_
  --no-cidr         If set then CIDR notation in a target file will not be
                    automatically be expanded into individual hosts.
  --no-color        If set then any foreground or background colours will be
                    stripped out.
  -v, --verbose     If set then verbose output will be displayed in the
                    terminal.
  --silent          If set only findings will be displayed and banners and
                    other information will be redacted.

That’s all about Interlace. But usage cases of the program is not limited to the examples shown. Surely you can come up with even more interesting use cases.

The next part is devoted to Parallel: ‘How to accelerate web application scanning via multi-threading with Parallel’.