Comprehensive Guide to John the Ripper. Part 6: How to brute force non-standard hashes

Table of contents

1. Introducing and Installing John the Ripper

2. Utilities for extracting hashes

3. How to start cracking passwords in John the Ripper (how to specify masks, dictionaries, hashes, formats, modes)

4. Practical examples of John the Ripper usage

5. Rule-based attack

6. How to brute force non-standard hashes

6.1 How to crack iterated, salted and arbitrary hashes based on MD5, SHA1 and other raw hashes

6.2 Hash types

6.3 Dynamic in John the Ripper

6.4 Dynamic mode format syntax

6.5 Examples of using custom dynamic formats in john

6.6 Constants and Salts

6.7 How to specify custom hash in format dynamic on command line

6.8 How to properly write hash with salt and username for John the Ripper

6.9 Built-in dynamic formats

6.10 How to store custom hash format dynamic in config file

6.11 How to write salts with special characters

7. 

8.


How to crack iterated, salted and arbitrary hashes based on MD5, SHA1 and other raw hashes

John the Ripper and Hashcat support a large number of password hashes to brute-force. The list of hashes supported in John the Ripper can be viewed with the command:

john --list=formats

To see the supported hashes in Hashcat, you can go to the page http://en.kali.tools/?p=155 or refer to the program's help:

hashcat -h

The lists of supported hashes are impressive for both programs.

Hash types

Hashes can be divided into the following groups:

  • Raw (md2, md4, md5, sha1, sha224, sha256, sha384, sha512, tiger, sha3_224, sha3_256, sha3_384, sha3_512, keccak_256, keccak_512, ripemd128, ripemd160, ripemd256, ripemd320, whirlpool, gost, skein224, skein256, skein384, skein512, panama, haval_128_3, haval_128_4, haval_128_5, haval_160_3, haval_160_4, haval_160_5, haval_192_3, haval_192_4, haval_192_5, haval_256_3, haval_256_4, haval_256_5). These are independent hashing algorithms, checksum calculations
  • Salted hashes. They are based on a raw hash, but adding salt to the password string, e.g. md5($pass.$salt)
  • Iterated. They are also based on raw hashes, but the hash result is then hashed again - this can happen many times. Example: md5($salt.sha1($salt.$pass))

In addition to these main groups, hashes can be divided according to their area of application:

  • Authentication
  • Network protocols
  • Operating Systems
  • Database servers
  • FTP, HTTP, SMTP, LDAP servers
  • Enterprise Application Software (EAS)
  • Full disk encryption
  • Documents
  • Password managers
  • Archives
  • Web applications (forums, CMS, E-Commerce)

In these groups, some hashes are independent, that is, they are calculated using their own algorithms), but also quite often, especially in web applications, operating systems that store passwords in a hashed form, salted and iterated hashes are used, which are based on raw hashes, for example:

  • sha1($s.sha1($s.sha1($p)))
  • sha1($s.sha1($p))
  • sha256(sha256($p).$s)

If it's a popular application, then support for its hash type is usually added to John the Ripper and Hashcat, and we don't really need to know which algorithm is used to calculate the hash.

But imagine the situation, you received a hash from a highly specialized program, figured out that the hash is calculated, for example, according to the following algorithm: md5(md5(md5($p).$s).$s2). But support for this algorithm (this type of hash) is absent in both John the Ripper and Hashcat, what should be done in this situation?

This part is devoted to the answer to this question - we will learn how to crack hashes calculated using raw, salted and iterated algorithms.

Related:

Dynamic in John the Ripper

A dynamic “self-describing” format (a.k.a. dynamic expression compiler). This is a mode in which the user, without using programming, describes the formula by which passwords are hashed.

Let's say we got the password hash:

838c9416a8d094b7e660a0f3b12e3e543c71f7f4

And its salt:

mial

Through reverse engineering, it was found that the hash is calculated using the following formula:

sha1(md5(sha512($p.$s).$p).$s)

We need to write the hash to a file in the following format:

HASH$SALT

That is, the hash and salt are separated by a dollar sign ($).

Create a hash.txt file and write:

838c9416a8d094b7e660a0f3b12e3e543c71f7f4$mial

Let's create a tiny dictionary, name it wordlist.txt and write to it:

123
hackware
123567890
123456
111111111111111111

Command to launch brute-force with dictionary attack:

john -form=dynamic='sha1(md5(sha512($p.$s).$p).$s)' --wordlist=wordlist.txt hash.txt

Mask attack:

john -form=dynamic='sha1(md5(sha512($p.$s).$p).$s)' --mask='?l?l?l?l?l?l?l?l' hash.txt

The password was successfully cracked, this is “hackware”:

In this example, we used our own formula to calculate the hash, but John the Ripper has many built-in dynamic formats. Moreover, you can write your dynamic format in the config file and use it by the assigned name, instead of having to enter the formula each time. We'll first look at the built-in formats and then return to the dynamic format syntax that you can write yourself.

Although not, to understand the notation of the built-in formats, you need to know the syntax - so let's start with the syntax for writing custom dynamic formats.

Dynamic mode format syntax

The expression MUST be a single crypto hash computation expression. This means the last step is to compute a hash, but not multiple hashes concatenated with each other. That is, md5($p) is valid, but md5($p).md5($p) is NOT valid.

All string concatenations must have a “.” (dot) character between them. So md5($p.$p) is valid and md5($p$s) is not valid.

Variables:

  • $p or $pass is used for password candidates
  • $s or $salt is used as the constant 'salt'
  • $s2 is used for the second salt constant
  • $u is used for user id (which is the first field in the input file)
  • from $c1 to $c8. These variables are 'constants'. REMEMBER, they must be used in order, that is, if you have $c2, then you must already have $c1. So md5($s.$c1.$p.$c1),c1=x gives us md5($s."x".$p."x")

Changing the case of letters (note: changing the case of letters only applies to some things).

  • lc($u) for lower case username (correct lower case Unicode)
  • uc($u) for uppercase username (again, correct uppercase Unicode)
  • lc($p) and uc($p) are for lower/upper case password.

IMPORTANT: If uc($u) or lc($u) are used in an expression, then ALL $u must be done in the same manner. This also applies to the password. So md5($u.$p.$) is valid but md5(uc($u).$p.$u) is not. This is a limitation of the dynamic format in general.

When using constants, they must be added, separated by commas, to the end of the expression. IMPORTANT: if symbols “:” (colon) are used in constants, then they must be written in hexadecimal form. For example: md5($c1.$p),c1=test_\x3a_test. This will yield md5("test_:_test".$p)

The section below will be devoted to symbols causing problems. For now, let's just mention that there is a utility called raw2dyna (included in the John the Ripper package). To convert any string to hexadecimal, use the -2h option, for example, the following command will print the hexadecimal notation for the colon:

raw2dyna -2h=':'
$HEX$3a

It is not necessary to encode the entire string - you can encode individual characters as shown above.

The same program, using the -2r option, can convert hexadecimal notation to raw data:

raw2dyna -2r='3a'

The problematic symbols will be discussed below, and there will be a separate part about the auxiliary utilities - there are a lot of them, even taking into account the fact that I do not refer to the utility programs for extracting hashes, which we have already considered in the second part.

Grouping. All expressions must be a single “outer” expression, but within an outer expression, there can be a lot of grouped data. Functions start with a function name and use ( and ) to group the data present in the hash. So things like md5($p) or md5(md5(md5($s.$p))) or md5(md5($p).$s.md5($s.$p)) are correct because they have a single outer (last) expression and all the necessary symbols for correct syntax.

Hash functions have several "flavors" that do slightly different things. For instance:

  • md5(EXPRESSION)
  • MD5(EXPRESSION)
  • md5_raw(EXPRESSION)
  • md5_64(EXPRESSION)
  • md5_64c(EXPRESSION)

The lowercase variant (such as md5(…)) will return the results as a base16 number written in lowercase. That is, md5("password") will return 5f4dcc3b5aa765d61d8327deb882cf99,

The uppercase variant will return the results as a base16 number written in uppercase. That is, MD5("password") will return 5F4DCC3B5AA765D61D8327DEB882CF99. The hash function to which _raw is appended returns just a raw binary object. That is, md5_raw("password") will give us 16 bytes, the first of which will be '\0x5f', and then 15 others.

md5_64("password") will return the mime base-64 result, that is, X03MO1qnZdYdgyfeuILPmQ (these are raw base64-encoded hash bytes)

md5_64c("password") returns the crypt alphabet base64 string, for our example it is LorACpebNRMRUmTSi69DaE.

Typically _64 and _64c are used as an outer function, informing john what the input hash type “looks like” (ie, the hashes are in base64). It's the same with uppercase. If the external function is uppercase, then only uppercase hex strings of the correct length are valid.

Hash functions (each has a lowercase, uppercase version, plus _raw, _64, and _64c appended):

  • md2
  • md4
  • md5
  • sha1
  • sha224
  • sha256
  • sha384
  • sha512
  • tiger
  • sha3_224
  • sha3_256
  • sha3_384
  • sha3_512
  • keccak_256
  • keccak_512
  • ripemd128
  • ripemd160
  • ripemd256
  • ripemd320
  • whirlpool
  • gost
  • skein224
  • skein256
  • skein384
  • skein512
  • panama
  • haval_128_3
  • haval_128_4
  • haval_128_5
  • haval_160_3
  • haval_160_4
  • haval_160_5
  • haval_192_3
  • haval_192_4
  • haval_192_5
  • haval_256_3
  • haval_256_4
  • haval_256_5

These functions can be mixed in almost any way. Therefore, the sha512(md5(panama($p).$s.ripmd128($s.$p))) is completely correct from a syntax point of view.

Examples of using custom dynamic formats in john

1. The hash is calculated as MD5 from the password:

md5($p)

2. The hash is calculated as MD5 from the password, and then the resulting string is hashed again using MD5:

md5(md5($p))

3. The hash is calculated as MD5 from the password, and then the resulting hash is encoded in Base64 and this new string is hashed again using MD5:

md5(md5_64($p))

4. The salt is added to the password, the MD5 hash is calculated for the resulting concatenated string:

md5($p.$s)

5. The password is hashed using the sha1 algorithm:

sha1($p)

6. The password is hashed with the MD5 algorithm, and then the resulting string is hashed with the sha1 algorithm:

sha1(md5($p))

7. The password is combined with the salt, the resulting string is hashed with the sha512 algorithm, then the salt, the resulting hash and password are taken, and all of them are combined into one string, which is hashed with the sha512 algorithm:

sha512($s.sha512($p.$s).$p)

Constants and Salts

The constants were mentioned above - their functions are similar to the function of salts, then what are they for? Basically, a constant is the salt, but it is passed to John the Ripper in a different way. The salt is specified along with the hash, and the constant is specified in the formula (which is used with the -form option or stored in the config file).

An example of using a constant:

md5($c1.$p),c1=hackware

That is, static, unchanging data should be used as a constant. And the salt is individual for each password.

How to specify custom hash in format dynamic on command line

A custom dynamic format is specified with the -form=dynamic='EXPRESSION' option

For instance:

-form=dynamic='sha1(md5(sha512($p.$s).$p).$s)'

Note that quotes are required and that single quotes must be used!

You can see for yourself - take your expression, put it in double quotes and echo it to the terminal, for example:

echo "sha1(md5(sha512($p.$s).$p).$s)"

The following remains of the expression:

sha1(md5(sha512(.).).)

That is, it is in this form that the program would see it.

How to properly write hash with salt and username for John the Ripper

The general formula for writing hashes for dynamic is as follows:

userID:$dynamic_#$base_16_hash[$salt]

Where:

  • userID is a username (if needed to calculate the hash)
  • $dynamic_# (number of built-in dynamic format - they will be discussed below)
  • $base_16_hash is the hash itself
  • $salt is salt

Apart from the hash, all other data is not required for most hashes.

An example of a salted hash (the salt is separated from the hash by the $ character):

699fce08bf085fb80f5ae1f240cbbe720aa62278$jxV9tvClmWz0

Example hash with username:

user:1c3470194afdc84b90a0781c5e4462fc

Built-in dynamic formats

To see a list of built-in formats, enter the command:

john --list=subformats

An example of the first lines:

Format = dynamic_0 type = dynamic_0: md5($p) (raw-md5)
Format = dynamic_1 type = dynamic_1: md5($p.$s) (joomla)
Format = dynamic_2 type = dynamic_2: md5(md5($p)) (e107)
Format = dynamic_3 type = dynamic_3: md5(md5(md5($p)))
Format = dynamic_4 type = dynamic_4: md5($s.$p) (OSC)
Format = dynamic_5 type = dynamic_5: md5($s.$p.$s)
Format = dynamic_6 type = dynamic_6: md5(md5($p).$s)
Format = dynamic_8 type = dynamic_8: md5(md5($s).$p)

dynamic_* strings are format names and should be used when choosing one format or another. You can also see the formulas by which each format is calculated. For some formats, there is a short commentary.

You may notice that lines with two different beginnings are displayed:

  • "Format = …"
  • "UserFormat = …"

 

They differ in that Format are built-in formats in John the Ripper, and UserFormat, basically, are also built-in formats, but from the community. In the dynamic.conf file (can be located at /usr/share/john/dynamic.conf) you can see and add your own UserFormats.

For Format, names like dynamic_# are used where # is a number. John reserves the numbers dynamic_0 through dynamic_999 for “built-in” formats. Not all of them have been defined yet.

UserFormat also uses names like dynamic_# where # is a number from 1001 to 9999. Again, not all of these formats are defined yet.

The dynamic formats are run like regular hash formats, that is, they are specified with the --format option, you need to specify dynamic_# as the name, replacing # with the number of the format you need, for example:

john --test --format=dynamic_1030

How to store custom hash format dynamic in config file

You can write your own dynamic formats to john.conf (john.ini), dynamic.conf (included in john.conf), or john.local.conf.

It is not enough just to write a formula - a primitive language has been developed, which is already quite close to programming.

You can find the documentation in the files:

  • ./doc/DYNAMIC
  • ./doc/DYNAMIC_SCRIPTING

 

How to write salts with special characters

Salts can contain problematic characters. Some of them can be characters such as: : \r \n \ NULL and so on.

Symbol : (colon) is used in JtR as a field separator. If it exists in salt, then it will break the salt into several fields (which is wrong).

A carriage return or line feed will break the line and JtR will not read it correctly.

NULL bytes are a problem in any C program that uses normal “string” functions.

The \ character is used to escape characters inside dynamic and can cause problems.

Another problem found: if the salt ends with white space, for example “ ” or “\t” (space or Tab), then they are removed during the preparation and clearing of lines (the program does not expect that the trailing space or tab can be part of the string).

Because of all these problems, support for hexadecimal salt was added to dynamic.

$dynamic_#$HASH$HEX$HEX_BYTES_OF_SALT

In this format, if the salt was 1234, then the equivalent value would be$HEX$31323334.

This allows you to encode and use salts like this:

$\x0:\r\nBadSalt

Even this kind of salt CAN be used, after encoding it looks like this:

$HEX$003a0a0d42616453616c74

You can encode characters and convert salts using the already mentioned program raw2dyna. It allows you to make various modifications with hashes - see its help.

The simplest use shown above is to convert single characters:

raw2dyna -2h='STRING'

Recommended for you:

Leave a Reply

Your email address will not be published.