Perimeter analysis for website security audit

Analysis of the infrastructure of the attacked site

This article focuses on the XRay tool, designed to identify subdomains and collect information about them.

There are plenty tools for all kind of passive and active information gathering, including many already reviewed on the miloserdov.org website. Tools for enumerating subdomains are also provided – only in the articles of miloserdov.org lately three excellent tools have been considered, each with its own special functions and advantages:

Nevertheless, the XRay program is unique and has several amazing advantages:

  • information collection continues for found subdomains, including using the Shodan and ViewDNS services
  • the found data is provided with a very convenient web interface with the ability to search through any fields of information collected

How to install Xray in Kali Linux

You must first install the Go compiler. To do this, go to the article “How to install Go (compiler and tools) on Linux” and select “Manual installation of the latest version of the Go compiler”.

After installing the Go compiler, just run the commands:

go get github.com/evilsocket/xray
rm -rf /home/git/go/pkg/dep/sources/https---go.googlesource.com-text/
cd $GOPATH/src/github.com/evilsocket/xray/
make
mv build/xray /usr/bin/
xray -h

API keys for XRay

The XRay tool can be used without any API keys at all, but I would highly recommend getting them for Shodan, since a lot of functionality depends on them. In addition, Shodan API keys can be used for free – for the purposes of this article, a paid tariff plan is not needed.

Shodan API Key

The parameter specifying the shodan.io API key is optional (-shodan-key KEY), however, if you do not specify it, service fingerprints will not be collected and much less information will be displayed (in fact, the XRay functions in this case will be reduced to DNS enumeration of subdomains).

ViewDNS API keys

If the parameter with the ViewDNS API key (-viewdns-key KEY) is specified, then historical data about the hosts will also be received.

APIs for only 250 requests are free – I have this limit expired during the first launch of XRay. Moreover, the limit is not renewable – that is, it is not a limit for 1 day or 1 month – when 250 requests end, then they end forever (or you need to switch to a paid tariff plan).

Anonymity Issues

To list subdomains, this program uses your main DNS resolver. Also, to capture banners from open ports, several connections can be established from your host to computers on the network that you are scanning. Technically, you simply connect to public addresses on open ports (there are no port scans, information about open ports is taken directly from Shodan using the API of this service), but it is possible that someone may not like this behavior anyway.

Perhaps you should consider proxying the entire process to hide your IP.

Network mapping of target websites

To start XRay, you must specify two required options:

  • -domain with the name of the initial domain for which subdomain data will be collected
  • -wordlist with the name of the dictionary file, which contains a list of subdomains for brute force

XRay comes with several dictionaries, you will find them in the /home/git/go/src/github.com/evilsocket/xray/wordlists/ folder:

  • all.lst
  • default.lst
  • top1mil-20000.lst
  • top1mil-5000.lst
  • top1mil.lst

The same dictionaries are available online: https://github.com/evilsocket/xray/tree/master/wordlists

Copy the dictionary of interest to the current folder:

cp /home/git/go/src/github.com/evilsocket/xray/wordlists/default.lst .

Or just download the dictionary you need if you cannot find it on the local system (the following command downloads the default.lst dictionary with a list of subdomains):

wget https://raw.githubusercontent.com/evilsocket/xray/master/wordlists/default.lst

Run the program as follows:

sudo xray -domain ebay.com -wordlist default.lst -shodan-key SHODAN_API_KEY -viewdns-key VIEWDNS_API_KEY

The following keys are optional, but the -shodan-key option is recommended:

  • -shodan-key SHODAN_API_KEY
  • -viewdns-key VIEWDNS_API_KEY

Immediately after starting the program, it reports that:

@ Saving session to ebay.com-xray-session.json
@ Web UI running on http://127.0.0.1:8080/

This means that the progress of the current session is saved to the ebay.com-xray-session.json file.

It also says that the web-based graphical user interface is located at http://127.0.0.1:8080/. We will return to it, but let's examine the following information that the program displays:

An example of such a line:

50.81 % completed, 11.15 req/s, 503 unique targets found so far ...

It means that 50.81% of all work has been completed, an average of 11.15 queries per second are being performed, and 503 unique targets have already been found.

Website perimeter analysis before hacking

Now let's move on to the web interface. To do this, open http://127.0.0.1:8080/ in any web browser.

For each subdomain found, information such as:

  • IP address
  • hostname that is bound to this IP address
  • subdomain (often does not match the host name)
  • open ports
  • Additional Information

Additional information may include fields such as:

  • amazon:bucket
  • html:title (title)
  • http:disallow (files and folders disallowed in the robots.txt file)
  • http:poweredby (software running the server)
  • http:redirect (address where the page redirects)
  • http:server (server, examples: cloudflare, Microsoft-IIS/10.0, nginx, Apache, envoy
  • Apache-Coyote/1.1, openresty)
  • https:chain (SSL certificate information)

Example of additional information output (paths disallowed from indexing were found, an approximate version of the server is shown):

http:disallow 	*rt=nc, /ebayadvsearch/
http:server 	Apache-Coyote/1.1
https:chain[0] 	C=US/O=eBay, Inc./OU=Site Operations/L=San Jose/P=California/CN=bulksell.ebay.com
https:chain[1] 	C=US/O=DigiCert Inc/OU=/L=/P=/CN=DigiCert SHA2 Secure Server CA

In the following example, a number of paths are also closed from indexing, a redirect to another address is revealed, an approximate version of the server is shown)

http:disallow 	/, /help/confidence/, /help/policies/, /disney/, ...
http:redirect 	http://pages.ebay.com/messages/page_not_responding.html?eBayErrorEventName=p4ewgjflhso%3F%3Cumjg%7D34%2B630%3F42%3D-2019.11.05.03.26.00.766.MST
http:server 	Apache-Coyote/1.1

In this sample output, the used ASP.NET technology is found:

http:poweredby 	ASP.NET
http:redirect 	https://admin.manage.microsoft.com/
http:server 	Microsoft-IIS/10.0
https:chain[0] 	C=US/O=Microsoft Corporation/OU=Microsoft Corporation/L=Redmond/P=WA/CN=manage.microsoft.com
https:chain[1] 	C=US/O=Microsoft Corporation/OU=Microsoft IT/L=Redmond/P=Washington/CN=Microsoft IT TLS CA 2

The web interface is very convenient for searching and analyzing information. You can search by any field.

An example of a search for the word ‘poweredby’ to show detailed information about the technologies used, if it was found:

Search for the word ‘Forbidden’:

Search for the word ‘login’:

Search by name or by part of the subdomain name:

You can search by IP address or by part of IP address:

You can even use * (asterisk) as a wildcard:

IP addresses are links to shodan.io, where you can get more information about this host.

Even subdomains were found at local addresses of the form 10.*.*.*:

Among the subdomains there are those where the SSH server is running:

By the way, you can search by the open port number.

Remember that the files with the results are saved in the current working directory with the names of the form *-xray-session.json.

How to continue session in Xray

The collection of information when using a large dictionary of subdomains is delayed for a long time. If you interrupted the process before it was completed, then you can continue it again at any time. To do this, simply run the command again:

sudo xray -domain DOMAIN -wordlist default.lst -shodan-key SHODAN_API_KEY -viewdns-key VIEWDNS_API_KEY

How to launch the XRay web UI without restarting brute force

If the XRay command has completed its work, a file with names of the form *-xray-session.json will be created. This file stores all the data collected. Therefore, if you saved this file and again want to continue the analysis using the XRay web interface, then simply run the original command:

sudo xray -domain DOMAIN -wordlist default.lst -shodan-key SHODAN_API_KEY -viewdns-key VIEWDNS_API_KEY

The XRay program will verify that all the data has already been collected and will not launch brute force, but at the same time it will launch the web interface.

Xray Options

You can change the IP address and port of the web user interface, you can change the location of the file where the data is saved using the XRay options.

All options of the XRay tool:

  -address string
    	IP address to bind the web ui server to. (default "127.0.0.1")
  -consumers int
    	Number of concurrent consumers to use for subdomain enumeration. (default 16)
  -domain string
    	Base domain to start enumeration from.
  -port int
    	TCP port to bind the web ui server to. (default 8080)
  -preserve-domain
    	Do not remove subdomain from the provided domain name.
  -session string
    	Session file name. (default "<domain-name>-xray-session.json")
  -shodan-key string
    	Shodan API key.
  -viewdns-key string
    	ViewDNS API key.
  -wordlist string
    	Wordlist file to use for enumeration. (default "wordlists/default.lst")

Last Updated on

Recommended for you:

Leave a Reply

Your email address will not be published.