How to speed up the generation of dictionaries with passwords
In one of the comments under the article on creating dictionaries, I was asked how to make the program use GPU in order to create passwords even faster. But this is the wrong question, the fact is that the speed of creating passwords is primarily limited by the speed of the storage, that is, the speed of the solid-state or hard drive. You may notice that even if the creation of the dictionary takes a long time, the CPU is not very busy – that is, the computation speed is not a bottleneck anyway.
In fact, the method shown below will already start to run into the performance of one processor core (or the speed of RAM), but it is many times faster than the usual method of creation and if the speed is still not enough for you, then I don't know how to fix it yet.
Actually
I wanted to put this section in the Conclusion, but decided to move it to the very beginning – so that you do not say that I deceived you Yes, dictionaries will be created faster, but immediately after creation they will be in RAM. Then, if you want to save the dictionary after restarting your computer, you need to copy it to permanent storage (solid state or hard drive). So, the time of copying the created dictionary to disk will be exactly the same as if we originally created the dictionary on disk. Well, that is, there is no gain in time. Therefore, this article is:
1. Proof that the bottleneck in the creation of dictionaries is the disk, and not the computing device
2. Another example of using the tmpfs file system
3. It is quite possible that you will find an application for the shown method, as it reduces the load on persistent storage
And you can also achieve acceleration if you need to make changes to data that are already in tmpfs. For example, you need to sort a large dictionary, or change it based on the Rules, or combine it with other dictionaries. That is, if you need to do several operations with the dictionary, then tmpfs will help speed up all the processes, since at each stage time will be saved that would be spent on a regular disk for reading and writing.
Creating dictionaries in tmpfs
tmpfs is an in-memory virtual file system.
The tmpfs file system has the following features:
- The file system can use swap space when the physical load on memory requires it.
- The file system consumes as much physical memory and swap space as is required to store the current contents of the file system.
- During a remount operation (mount -o remount), the file system can be resized (without losing the existing contents of the file system).
If the tmpfs filesystem is unmounted, its contents are lost (deleted). This is very important – copy all the data from it before turning off the computer, otherwise everything will be lost.
While this is obvious, just in case: since all data resides in RAM, your computer must have enough RAM to store the data you want to put in tmpfs.
Let's create a mount point:
mkdir /tmp/mytmpfs
Let's create a virtual file system of 20 Gigabytes in RAM:
sudo mount -t tmpfs -o size=20g tmpfs /tmp/mytmpfs
We pass into it:
cd /tmp/mytmpfs
To create a dictionary, we will use the maskprocessor program. With the following command, we create a dictionary of 5 characters (uppercase and lowercase letters, numbers):
time maskprocessor -1 ?l?u?d ?1?1?1?1?1 > dic.txt
There is one more command before maskprocessor – time, it measures the program running time.
It took about 6 seconds to create the dictionary.
In the System Monitor, you can see that while the program was running to create a dictionary, one of the processor cores was fully loaded. You can also see that the consumption of RAM has increased.
Now, for example, let's create the same dictionary on the hard disk:
time maskprocessor -1 ?l?u?d ?1?1?1?1?1 > dic.txt
On a hard disk, this operation takes from 20 seconds to one minute (apparently, it depends on how busy the disk itself is).
As an experiment, let's take the command for converting the dictionary according to the Rule from this article:
john --rules=lud5 --wordlist=dic.txt --stdout > lud5.txt
On the hard drive, it completed in 43 seconds, and in RAM in 41 seconds. The reason for these results is that in this case the bottleneck is the performance of one processor core (the process of creating and converting dictionaries is performed in one thread).
Related articles:
- How to create dictionaries that comply with specific password strength policies (using Rule-based attack) (100%)
- How to generate candidate passwords that match password strength policies (filtering words with grep) (100%)
- Advanced wordlist generating techniques (100%)
- How to generate dictionaries by any parameters with pydictor (83.4%)
- Programs for generating wordlists (77.8%)
- Comprehensive Guide to John the Ripper. Part 1: Introducing and Installing John the Ripper (RANDOM - 50%)