Perimeter analysis for website security audit
Analysis of the infrastructure of the attacked site
This article focuses on the XRay tool, designed to identify subdomains and collect information about them.
There are plenty tools for all kind of passive and active information gathering, including many already reviewed on the miloserdov.org website. Tools for enumerating subdomains are also provided – only in the articles of miloserdov.org lately three excellent tools have been considered, each with its own special functions and advantages:
- The fastest subdomains enumeration and monitoring (also works on Windows!)
- How to discover subdomains without brute-force
- How to search subdomains and build graphs of network structure with Amass
Nevertheless, the XRay program is unique and has several amazing advantages:
- information collection continues for found subdomains, including using the Shodan and ViewDNS services
- the found data is provided with a very convenient web interface with the ability to search through any fields of information collected
How to install Xray in Kali Linux
You must first install the Go compiler. To do this, go to the article “How to install Go (compiler and tools) on Linux” and select “Manual installation of the latest version of the Go compiler”.
After installing the Go compiler, just run the commands:
go get github.com/evilsocket/xray rm -rf /home/git/go/pkg/dep/sources/https---go.googlesource.com-text/ cd $GOPATH/src/github.com/evilsocket/xray/ make mv build/xray /usr/bin/ xray -h
API keys for XRay
The XRay tool can be used without any API keys at all, but I would highly recommend getting them for Shodan, since a lot of functionality depends on them. In addition, Shodan API keys can be used for free – for the purposes of this article, a paid tariff plan is not needed.
Shodan API Key
The parameter specifying the shodan.io API key is optional (-shodan-key KEY), however, if you do not specify it, service fingerprints will not be collected and much less information will be displayed (in fact, the XRay functions in this case will be reduced to DNS enumeration of subdomains).
ViewDNS API keys
If the parameter with the ViewDNS API key (-viewdns-key KEY) is specified, then historical data about the hosts will also be received.
APIs for only 250 requests are free – I have this limit expired during the first launch of XRay. Moreover, the limit is not renewable – that is, it is not a limit for 1 day or 1 month – when 250 requests end, then they end forever (or you need to switch to a paid tariff plan).
Anonymity Issues
To list subdomains, this program uses your main DNS resolver. Also, to capture banners from open ports, several connections can be established from your host to computers on the network that you are scanning. Technically, you simply connect to public addresses on open ports (there are no port scans, information about open ports is taken directly from Shodan using the API of this service), but it is possible that someone may not like this behavior anyway.
Perhaps you should consider proxying the entire process to hide your IP.
Network mapping of target websites
To start XRay, you must specify two required options:
- -domain with the name of the initial domain for which subdomain data will be collected
- -wordlist with the name of the dictionary file, which contains a list of subdomains for brute force
XRay comes with several dictionaries, you will find them in the /home/git/go/src/github.com/evilsocket/xray/wordlists/ folder:
- all.lst
- default.lst
- top1mil-20000.lst
- top1mil-5000.lst
- top1mil.lst
The same dictionaries are available online: https://github.com/evilsocket/xray/tree/master/wordlists
Copy the dictionary of interest to the current folder:
cp /home/git/go/src/github.com/evilsocket/xray/wordlists/default.lst .
Or just download the dictionary you need if you cannot find it on the local system (the following command downloads the default.lst dictionary with a list of subdomains):
wget https://raw.githubusercontent.com/evilsocket/xray/master/wordlists/default.lst
Run the program as follows:
sudo xray -domain ebay.com -wordlist default.lst -shodan-key SHODAN_API_KEY -viewdns-key VIEWDNS_API_KEY
The following keys are optional, but the -shodan-key option is recommended:
- -shodan-key SHODAN_API_KEY
- -viewdns-key VIEWDNS_API_KEY
Immediately after starting the program, it reports that:
@ Saving session to ebay.com-xray-session.json @ Web UI running on http://127.0.0.1:8080/
This means that the progress of the current session is saved to the ebay.com-xray-session.json file.
It also says that the web-based graphical user interface is located at http://127.0.0.1:8080/. We will return to it, but let's examine the following information that the program displays:
An example of such a line:
50.81 % completed, 11.15 req/s, 503 unique targets found so far ...
It means that 50.81% of all work has been completed, an average of 11.15 queries per second are being performed, and 503 unique targets have already been found.
Website perimeter analysis before hacking
Now let's move on to the web interface. To do this, open http://127.0.0.1:8080/ in any web browser.
For each subdomain found, information such as:
- IP address
- hostname that is bound to this IP address
- subdomain (often does not match the host name)
- open ports
- Additional Information
Additional information may include fields such as:
- amazon:bucket
- html:title (title)
- http:disallow (files and folders disallowed in the robots.txt file)
- http:poweredby (software running the server)
- http:redirect (address where the page redirects)
- http:server (server, examples: cloudflare, Microsoft-IIS/10.0, nginx, Apache, envoy
- Apache-Coyote/1.1, openresty)
- https:chain (SSL certificate information)
Example of additional information output (paths disallowed from indexing were found, an approximate version of the server is shown):
http:disallow *rt=nc, /ebayadvsearch/ http:server Apache-Coyote/1.1 https:chain[0] C=US/O=eBay, Inc./OU=Site Operations/L=San Jose/P=California/CN=bulksell.ebay.com https:chain[1] C=US/O=DigiCert Inc/OU=/L=/P=/CN=DigiCert SHA2 Secure Server CA
In the following example, a number of paths are also closed from indexing, a redirect to another address is revealed, an approximate version of the server is shown)
http:disallow /, /help/confidence/, /help/policies/, /disney/, ... http:redirect http://pages.ebay.com/messages/page_not_responding.html?eBayErrorEventName=p4ewgjflhso%3F%3Cumjg%7D34%2B630%3F42%3D-2019.11.05.03.26.00.766.MST http:server Apache-Coyote/1.1
In this sample output, the used ASP.NET technology is found:
http:poweredby ASP.NET http:redirect https://admin.manage.microsoft.com/ http:server Microsoft-IIS/10.0 https:chain[0] C=US/O=Microsoft Corporation/OU=Microsoft Corporation/L=Redmond/P=WA/CN=manage.microsoft.com https:chain[1] C=US/O=Microsoft Corporation/OU=Microsoft IT/L=Redmond/P=Washington/CN=Microsoft IT TLS CA 2
The web interface is very convenient for searching and analyzing information. You can search by any field.
An example of a search for the word ‘poweredby’ to show detailed information about the technologies used, if it was found:
Search for the word ‘Forbidden’:
Search for the word ‘login’:
Search by name or by part of the subdomain name:
You can search by IP address or by part of IP address:
You can even use * (asterisk) as a wildcard:
IP addresses are links to shodan.io, where you can get more information about this host.
Even subdomains were found at local addresses of the form 10.*.*.*:
Among the subdomains there are those where the SSH server is running:
By the way, you can search by the open port number.
Remember that the files with the results are saved in the current working directory with the names of the form *-xray-session.json.
How to continue session in Xray
The collection of information when using a large dictionary of subdomains is delayed for a long time. If you interrupted the process before it was completed, then you can continue it again at any time. To do this, simply run the command again:
sudo xray -domain DOMAIN -wordlist default.lst -shodan-key SHODAN_API_KEY -viewdns-key VIEWDNS_API_KEY
How to launch the XRay web UI without restarting brute force
If the XRay command has completed its work, a file with names of the form *-xray-session.json will be created. This file stores all the data collected. Therefore, if you saved this file and again want to continue the analysis using the XRay web interface, then simply run the original command:
sudo xray -domain DOMAIN -wordlist default.lst -shodan-key SHODAN_API_KEY -viewdns-key VIEWDNS_API_KEY
The XRay program will verify that all the data has already been collected and will not launch brute force, but at the same time it will launch the web interface.
Xray Options
You can change the IP address and port of the web user interface, you can change the location of the file where the data is saved using the XRay options.
All options of the XRay tool:
-address string IP address to bind the web ui server to. (default "127.0.0.1") -consumers int Number of concurrent consumers to use for subdomain enumeration. (default 16) -domain string Base domain to start enumeration from. -port int TCP port to bind the web ui server to. (default 8080) -preserve-domain Do not remove subdomain from the provided domain name. -session string Session file name. (default "<domain-name>-xray-session.json") -shodan-key string Shodan API key. -viewdns-key string ViewDNS API key. -wordlist string Wordlist file to use for enumeration. (default "wordlists/default.lst")
Related articles:
- How to discover subdomains without brute-force (82.2%)
- The fastest subdomains enumeration and monitoring (also works on Windows!) (64.2%)
- FinalRecon: a simple and fast tool to gather information about web sites, works on Windows (61.5%)
- How to search subdomains and build graphs of network structure with Amass (56.2%)
- How to find out hostnames for many IP addresses (56%)
- sqlmap usage guide. Part 1: Basic web-site checks (GET) (RANDOM - 26.5%)