How to hash long passwords (>72 characters) with blowfish

The problem here is basically a problem of entropy. So let's start looking there:

Entropy Per Character

The number of bits of entropy per byte are:

Hex Characters Bits: 4 Values: 16 Entropy In 72 Chars: 288 bits

Bits: 4

Values: 16

Entropy In 72 Chars: 288 bits

Alpha-Numeric Bits: 6 Values: 62 Entropy In 72 Chars: 432 bits

Bits: 6

Values: 62

Entropy In 72 Chars: 432 bits

"Common" Symbols Bits: 6.5 Values: 94 Entropy In 72 Chars: 468 bits

Bits: 6.5

Values: 94

Entropy In 72 Chars: 468 bits

Full Bytes Bits: 8 Values: 255 Entropy In 72 Chars: 576 bits

Bits: 8

Values: 255

Entropy In 72 Chars: 576 bits

So, how we act depends on what type of characters we expect.

The First Problem

The first problem with your code, is that your "pepper" hash step is outputting hex characters (since the fourth parameter to hash_hmac() is not set).

Therefore, by hashing your pepper in, you're effectively cutting the maximum entropy available to the password by a factor of 2 (from 576 to 288 possible bits).

The Second Problem

However, sha256 only provides 256 bits of entropy in the first place. So you're effectively cutting a possible 576 bits down to 256 bits. Your hash step * immediately*, by very definition loses at least 50% of the possible entropy in the password.

You could partially solve this by switching to SHA512, where you'd only reduce the available entropy by about 12%. But that's still a not-insignificant difference. That 12% reduces the number of permutations by a factor of 1.8e19. That's a big number... And that's the factor it reduces it by...

The Underlying Issue

The underlying issue is that there are three types of passwords over 72 characters. The impact that this style system has on them will be very different:

Note: from here on out I'm assuming we're comparing to a pepper system which uses SHA512 with raw output (not hex).

High entropy random passwords These are your users using password generators which generate what amount to large keys for passwords. They are random (generated, not human chosen), and have high entropy per character. These types are using high-bytes (characters > 127) and some control characters. For this group, your hashing function will significantly reduce their available entropy into bcrypt. Let me say that again. For users who are using high entropy, long passwords, your solution significantly reduces the strength of their password by a measurable amount. (62 bits of entropy lost for a 72 character password, and more for longer passwords)

Medium entropy random passwords This group is using passwords containing common symbols, but no high bytes or control characters. These are your typable passwords. For this group, you are going to slightly unlock more entropy (not create it, but allow more entropy to fit into the bcrypt password). When I say slightly, I mean slightly. The break-even occurs when you max out the 512 bits that SHA512 has. Therefore, the peak is at 78 characters. Let me say that again. For this class of passwords, you can only store an additional 6 characters before you run out of entropy.

Low entropy non-random passwords This is the group who are using alpha-numeric characters that are probably not randomly generated. Something like a bible quote or such. These phrases have approximately 2.3 bits of entropy per character. For this group, you can significantly unlock more entropy (not create it, but allow more to fit into the bcrypt password input) by hashing. The breakeven is around 223 characters before you run out of entropy. Let's say that again. For this class of passwords, pre-hashing definitely increases security significantly.

Back To The Real World

These kinds of entropy calculations don't really matter much in the real world. What matters is guessing entropy. That's what directly effects what attackers can do. That's what you want to maximize.

While there's little research that's gone into guessing entropy, there are some points that I'd like to point out.

The chances of randomly guessing 72 correct characters in a row are extremely low. You're more likely to win the Powerball lottery 21 times, than to have this collision... That's how big of a number we're talking about.

But we may not stumble on it statistically. In the case of phrases the chance of the first 72 characters being the same is a whole lot higher than for a random password. But it's still trivially low (you're more likely to win the Powerball lottery 5 times, based on 2.3 bits per character).

Practically

Practically, it doesn't really matter. The chances of someone guessing the first 72 characters right, where the latter ones make a significant difference are so low that it's not worth worrying about. Why?

Well, let's say you're taking a phrase. If the person can get the first 72 characters right, they are either really lucky (not likely), or it's a common phrase. If it's a common phrase, the only variable is how long to make it.

Let's take an example. Let's take a quote from the bible (just because it's a common source of long text, not for any other reason):

You shall not covet your neighbor's house. You shall not covet your neighbor's wife, or his manservant or maidservant, his ox or donkey, or anything that belongs to your neighbor.

That's 180 characters. The 73rd character is the g in the second neighbor's. If you guessed that much, you're likely not stopping at nei, but continuing with the rest of the verse (since that's how the password is likely to be used). Therefore, your "hash" didn't add much.

BTW: I am ABSOLUTELY NOT advocating using a bible quote. In fact, the exact opposite.

Conclusion

You're not really going to help people much who use long passwords by hashing first. Some groups you can definitely help. Some you can definitely hurt.

But in the end, none of it is overly significant. The numbers we are dealing with are just WAY too high. The difference in entropy isn't going to be much.

You're better off leaving bcrypt as it is. You're more likely to screw up the hashing (literally, you've done it already, and you're not the first, or last to make that mistake) than the attack you're trying to prevent is going to happen.

Focus on securing the rest of the site. And add a password entropy meter to the password box on registration to indicate password strength (and indicate if a password is overlong that the user may wish to change it)...

That's my $0.02 at least (or possibly way more than $0.02)...

As Far As Using A "Secret" Pepper:

There is literally no research into feeding one hash function into bcrypt. Therefore, it's unclear at best if feeding a "peppered" hash into bcrypt will ever cause unknown vulnerabilities (we know doing hash1(hash2($value)) can expose significant vulnerabilities around collision resistance and preimage attacks).

Considering that you're already considering storing a secret key (the "pepper"), why not use it in a way that's well studied and understood? Why not encrypt the hash prior to storing it?

Basically, after you hash the password, feed the entire hash output into a strong encryption algorithm. Then store the encrypted result.

Now, an SQL-Injection attack will not leak anything useful, because they don't have the cipher key. And if the key is leaked, the attackers are no better off than if you used a plain hash (which is provable, something with the pepper "pre-hash" doesn't provide).

Note: if you choose to do this, use a library. For PHP, I strongly recommend Zend Framework 2's Zend\Crypt package. It's actually the only one I'd recommend at this current point in time. It's been strongly reviewed, and it makes all the decisions for you (which is a very good thing)...

Something like:

use Zend\Crypt\BlockCipher;

public function createHash($password) {
    $hash = password_hash($password, PASSWORD_BCRYPT, ["cost"=>$this->cost]);

    $blockCipher = BlockCipher::factory('mcrypt', array('algo' => 'aes'));
    $blockCipher->setKey($this->key);
    return $blockCipher->encrypt($hash);
}

public function verifyHash($password, $hash) {
    $blockCipher = BlockCipher::factory('mcrypt', array('algo' => 'aes'));
    $blockCipher->setKey($this->key);
    $hash = $blockCipher->decrypt($hash);

    return password_verify($password, $hash);
}

And it's beneficial because you're using all of the algorithms in ways that are well understood and well studied (relatively at least). Remember:

Anyone, from the most clueless amateur to the best cryptographer, can create an algorithm that he himself can't break.

Bruce Schneier

---

Peppering passwords is surely a good thing to do, but let's see why.

First we should answer the question when exactly a pepper helps. The pepper only protects the passwords, as long as it stays secret, so if an attacker has access to the server itself, it is of no use. A much easier attack though is SQL-injection, which allows read-access to the database (to our hash-values), i prepared a demo of SQL-injection to show how easy it can be (click the next arrow to get a prepared input).

Then what does the pepper actually help? As long as the pepper stays secret, it protects weak passwords from a dictionary attack. The password 1234 would then become something like 1234-p*deDIUZeRweretWy+.O. This password is not only much longer, it contains also special characters and will never be part of any dictionary.

Now we can estimate what passwords our users will use, probably more users will enter weak passwords, as there are users with passwords between 64-72 characters (actually this will be very rare).

Another point is the range for brute-forcing. The sha256 hash function will return 256 bits output or 1.2E77 combinations, that's ways too much for brute-forcing, even for GPU's (if i calculated correctly, this would need about 2E61 years on a GPU in 2013). So we do not get a real disadvantage applying the pepper. Because the hash-values are not systematic you cannot speed up brute-forcing with common patterns.

P.S. As far as i know, the 72 character limit is specific to the algorithm of BCrypt itself. The best answer i found is this.

P.P.S I think your example is flawed, you cannot generate the hash with the full password length, and verify it with a truncated one. You probably meant to apply the pepper the same way for generating the hash and for verification of the hash.

---

Bcrypt uses an algorithm based on the expensive Blowfish key setup algorithm.

The recommended 56 byte password limit (including null termination byte) for bcrypt relates to the 448 bit limit of the Blowfish key. Any bytes beyond that limit are not fully mixed into the resulting hash. The 72 byte absolute limit on bcrypt passwords is therefore less relevant, when you consider the actual effect on the resulting hash by those bytes.

If you think your users would normally choose passwords over 55 bytes in length, remember you can always increase the rounds of password stretching instead, to increase security in the case of a password table breach (although this has to be a lot compared with adding extra characters). If the access rights of users are so critical that users would normally require a massively long password, then the password expiry should also be short, like 2 weeks. This means that a password is much less likely to be remain valid while a hacker invests their resources in defeating the work factor involved in testing each trial password to see if it will produce a matching hash.

Of course, in the case of the password table not being breached, we should only allow hackers, at most, ten attempts to guess a user's 55 byte password, before locking the user's account out ;)

If you do decide to pre-hash a password that is longer than 55 bytes, then you should use SHA-384, as it has the largest output without going over the limit.

---