Monitoring the emergence of new subdomains in real time

While penetration testing of a web resource, one of the strategies is cracking the perimeter. Large companies with a developed web structure may have network nodes that are not actively used or supported – simply put, they are forgotten. On the one hand, such nodes may be more vulnerable to attacks due to outdated software or simply because they were not intended to work with external users (in production). But at the same time, they can be connected to the local network and using the network pivoting methods, you can continue collecting information and attacking nodes through the local network.

Related article: Network pivoting: concept, examples, techniques, tools

Finding subdomains is one way to explore the perimeter. There are already three articles devoted to this issue and three tools that use a variety of techniques for identifying subdomains:

• The fastest subdomains enumeration and monitoring (also works on Windows!)
• How to discover subdomains without brute-force
• How to search subdomains and build graphs of network structure with Amass

Some of the tools discussed in those articles use Certificate Transparency log analysis. In addition to finding long-standing subdomains, Certificate Transparency can be used to detect subdomains as they appear in near real-time! This is exactly what this article is about – we will look at how to learn about subdomains as soon as they appear, what tools are there for this, and in what cases this technique will work.

Let's start by getting to know Certificate Transparency. The next section contains information from Wikipedia.

Certificate Transparency

What is Certificate Transparency

Certificate Transparency (CT) is an Internet security standard and open source framework for monitoring and auditing digital certificates. The standard creates a system of public logs that seek to eventually record all certificates issued by publicly trusted certificate authorities, allowing efficient identification of mistakenly or maliciously issued certificates.

Certificate Transparency is described in an experimental RFC 6962.

As of 2021, Certificate Transparency is mandatory for all TLS certificates, but not other types of certificates.

One of the problems with digital certificate management is that fraudulent certificates take a long time to be spotted, reported and revoked by the browser vendors. Certificate Transparency would help by making it impossible for a certificate to be issued for a domain without the domain owner knowing.

Certificate Transparency does not require side channel communication to validate certificates as do some competing technologies such as Online Certificate Status Protocol (OCSP) and Convergence. Certificate Transparency also operates without the need to trust a third party.

Certificate Transparency logs

Certificate Transparency depends on verifiable Certificate Transparency logs. A log appends new certificates to an ever-growing Merkle hash tree.[1]:Section 3 To be seen as behaving correctly, a log must:

• Verify that each submitted certificate or precertificate has a valid signature chain leading back to a trusted root certificate authority certificate.
• Refuse to publish certificates without this valid signature chain.
• Store the entire verification chain from the newly accepted certificate back to the root certificate.
• Present this chain for auditing upon request.

A log may accept certificates that are not yet fully valid and certificates that have expired.

Certificate Transparency monitors

Monitors act as clients to the log servers. Monitors check logs to make sure they are behaving correctly. An inconsistency is used to prove that a log has not behaved correctly, and the signatures on the log's data structure (the Merkle tree) prevent the log from denying that misbehavior.

Certificate Transparency auditors

Auditors also act as clients to the log servers. Certificate Transparency auditors use partial information about a log to verify the log against other partial information they have.

Certificate Transparency in plain language

1. All issued TLS certificates must be logged in the Certificate Transparency. If you do not log the certificate, it will not be considered valid even if all other conditions are met.

2. The journal of issued TLS certificates is public for everyone.

It follows from this that as soon as I issue the certificate for the project2.hackware.ru subdomain,

information about this certificate immediately falls into a publicly accessible journal

and information about the project2.hackware.ru subdomain becomes publicly available.

As a result, as soon as I received a certificate for project2.hackware.ru, even if I did not manage to configure anything at project2.hackware.ru, then anyone who wants to know about this subdomain already knows about it.

This is the whole essence of the technique considered in this article.

Limitations of this technique

Obviously, for subdomains operating over the HTTP protocol (that is, for which a TLS certificate is not issued at all), this method of searching for subdomains is inapplicable. It is applicable only for subdomains using the HTTPS protocol.

Also, if separate certificates are not registered for subdomains, and the work is done with a certificate with a wildcard character (for example, *.hackware.ru), then this technique will also fail.

Why do companies issue certificates for subdomains instead of using a wildcard?

By the way, the question may arise, why issue separate certificates for each subdomain if you can issue one certificate with a wildcard character (for example, *.hackware.ru) that will cover all subdomains at once?

You can see in the properties of the certificate of the HackWare.ru website that it is the certificate with a wildcard character that is used here.

Why, for example, are the system administrators of yandex.ru not so “smart” and “didn’t think of it” to issue one *.yandex.ru certificate and use it for everything?

The fact is that in large organizations it is difficult to guarantee the absence of leaks, and if the private key from the developer responsible for the secondary direction of the third-rate service for some domain food.movie.delivery.yandex.ru leaks, then this is one thing, and if the private key for *.yandex.ru, including subdomains passport.yandex.ru, mail.yandex.ru and others, then this is much more serious. Leakage of the private key for the *.yandex.ru certificate will mean the ability to perform man-in-the-middle attacks for the main domain and all subdomains, while the web browser will show the connection as secure (green lock)!

For this reason, organizations always try to issue separate certificates for subdomains. Information about these subdomains is recorded in the public log. Which means for us that we can find out about them exactly at the moment when the certificate is issued (the subdomain is created).

Where to View Certificate Transparency Logs

The official Certificate Transparency website https://certificate.transparency.dev/ has a list of monitoring companies as well as a list of those providing access to the logs.

A couple of addresses where you can enter a domain name and get a complete list of certificates ever issued for it:

• https://crt.sh/
• https://transparencyreport.google.com/https/certificates

Tool for monitoring new subdomains (issuing new TLS certificates)

The list of issued certificates for a DOMAIN can be obtained by parsing a link like https://crt.sh/?q=DOMAIN. If you prefer to work with the JSON format, then use a link like https://crt.sh/?q=DOMAIN&output=json.

The algorithm is very simple:

1. We get a list of subdomains at the link https://crt.sh/?q=DOMAIN and save them to a file
2. From time to time, we get the list of subdomains again at the link https://crt.sh/?q=DOMAIN and compare them with the existing ones
3. If new subdomains are found, then display them on the screen
4. If there are several domains, then we repeat this procedure for each of them.

This is a simple task and to automate it, you can write your own Bash script in several lines. But I found a ready-made tool called Sublert, unfortunately, it was not so easy with it. It simply does not work in its original form, I had to spend quite a lot of time on it, but I managed to figure it out.

Installing Sublert on Kali Linux

sudo apt install git python3 python3-termcolor python3-tld python3-requests python3-dnspython python3-psycopg2 python3-tld
git clone https://github.com/yassineaboukir/sublert && cd sublert

If an error occurs

Traceback (most recent call last):
  File "/home/mial/bin/sublert/sublert.py", line 16, in <module>
    from tld.utils import update_tld_names
ImportError: cannot import name 'update_tld_names' from 'tld.utils' (/usr/lib/python3/dist-packages/tld/utils.py)

Then run the following commands:

sudo apt install python3-pip
sudo pip3 install tld==0.9.6

Installing Sublert in BlackArch

sudo pacman -S sublert

Sublert patch

In its original form, Sublert sends new subdomain alerts through Slack when it finds new subdomains. I don’t know what it is and I don’t like the idea of using third-party services, so now we will change the code of the program so that it outputs the obtained results directly to standard output. If you are satisfied with Slack, then refer to the official instructions: https://medium.com/@yassineaboukir/automated-monitoring-of-subdomains-for-fun-and-profit-release-of-sublert-634cfc5d7708

Open config.py file – for Kali Linux users this file is in the same folder where you downloaded Sublert, for BlackArch users this file is located at /usr/share/sublert/config.py:

sudo gedit /usr/share/sublert/sublert.py

In this file find the snippet

            url = "https://" + url.replace('+ ', '')
            rev_url.append(get_fld(url))
            data = "{}:new: {}".format(at_channel(), url)
            slack(data)

and replace it all with

            print (url)

Next problem: Sublert uses two ways to get data from the log:

1) connecting to the database

2) if the first method fails, then an address like https://crt.sh/?q=DOMAIN&output=json is parsed

So, when connecting to the database, the program DOES NOT SEE new subdomains. I don't know what the problem is, but without the next change, the program will work, but even when subdomains appear, it will not show you anything.

So in the sublert.py file find and completely remove the following snippet:

        try:
            #connecting to crt.sh postgres database to retrieve subdomains.
            unique_domains = set()
            domain = domain.replace('%25.', '')
            conn = psycopg2.connect("dbname={0} user={1} host={2}".format(DB_NAME, DB_USER, DB_HOST))
            conn.autocommit = True
            cursor = conn.cursor()
            cursor.execute("SELECT ci.NAME_VALUE NAME_VALUE FROM certificate_identity ci WHERE ci.NAME_TYPE = 'dNSName' AND reverse(lower(ci.NAME_VALUE)) LIKE reverse(lower('%{}'));".format(domain))
            for result in cursor.fetchall():
                matches = re.findall(r"\'(.+?)\'", str(result))
                for subdomain in matches:
                    try:
                        if get_fld("https://" + subdomain) == domain:
                            unique_domains.add(subdomain.lower())
                    except: pass
            return sorted(unique_domains)
        except:

In Kali Linux, to start, specify the path to the sublert.py file:

python3 sublert.py -u DOMAIN

Run in BlackArch like this:

sudo sublert -u DOMAIN

To add new domains to monitoring use the -u option, for example:

python3 sublert.py -u yandex.ru
sudo sublert -u yandex.ru

You will be asked if you want to view the list of found subdomains (enter “Y” for “Yes” or “N” for “No”):

If you do not want to be prompted to list subdomains each time, then use the -q false option, for example:

sudo sublert -u wise.com -q false
sudo sublert -u hackware.ru -q false
sudo sublert -u paysend.com -q false

To display a list of all domains to monitor the issuance of new certificates, run the command

sublert -a

To check the issuance of new certificates for subdomains, run the command:

sudo sublert

If no new subdomains are found, it will show:

[!] Done.

If new subdomains are found, the following line will be displayed (although nothing is actually sent to Slack):

[!] Exporting the result to Slack. Please don't interrupt!

The found subdomains will also be shown.

As you can see, the project2reloaded.hackware.ru subdomain was found, for which I created a subdomain and received a certificate just before running the script.

You can manually run the script, or set up automatic launch using Cron and sending notifications to you by email or in another way.

On the card page of the Sublert program https://en.kali.tools/?p=1580 you will find a complete list of options. Of these, note the following:

  -u TARGET, --url TARGET
                        Domain to monitor. E.g: yahoo.com (default: None)
  -q true|false, --question true|false
                        Disable user input questions (default: True)
  -d DOMAIN, --delete DOMAIN
                        Domain to remove from the monitored list. E.g: yahoo.com (default: None)
  -t THREADS, --threads THREADS
                        Number of concurrent threads to use. (default: 10)
  -a [LISTING], --list [LISTING]
                        Listing all monitored domains.
  -m, --reset
                        Reset everything.