Why Is OWASP Changing ESAPI Encryption?

Reasons for Change

Symmetric Encryption in ESAPI 1.4

The as-delivered, default ESAPI 1.4 encryption mechanism uses "PBEWithMD5AndDES", a "Password Based Encryption" algorithm. As implemented in ESAPI 1.4, password (or pass phrase), taken from the property MasterPassword, is concatenated with the salt as specified by the property MasterSalt, and then repeated hashed 20 times using the MD5 message digest algorithm. The intent of the salt and repetitive hashing is to slow down dictionary attacks on the password. The result after the 20 iterations of hashing is used as the DES encryption key. The encryption uses Cipher Block Chaining (CBC) cipher mode.

Problems with Symmetric Encryption in ESAPI 1.4

The problems with symmetric encryption in ESAPI 1.4 are many. Here are some of the main issues:

1. The implementation only supported a single encryption key. While this may be sufficient for some uses, clearly there are cases where multiple recipients are involved when you would like to use unique encryption keys for each recipient.

2. ESAPI 1.4 came with a default setting for MasterPassword and MasterSalt, thus not requiring that it be changed. How many of you forgot to change these?

3. Password based encryption is likely to be much weaker randomly choosing a secret key. For instance, if you restrict yourself to printable ASCII characters (which most people will), it would require a pass phrase of roughly 45 characters to be the equivalent of a randomly chosen 128-bit secret key.

4. The default algorithm, DES, only has an effective key length of 56-bits. This key size is too short and is in the realm of brute forcing within a short amount of time for most medium to large companies. (Fortunately, this could be addressed in part by changing the EncryptionAlgorithm property from "PBEWithMD5AndDES" to either "PBEWithHmacSHA1AndDESede" [if you are using JDK 1.5 or later] or to "PBEWithMD5AndTripleDES" [if you are still using JDK 1.4]. Unfortunately, because of the implementation of JavaEncryptor, only "password based encryption" algorithms can be used in ESAPI 1.4.)

5. The symmetric encryption does not provide for any means of confirming message authenticity. Cryptographers consider ensuring message authenticity an important feature of modern cryptosystems.

Symmetric Encryption in ESAPI 2.0rc1 and 2.0rc2

By default, release candidates ESAPI 2.0-rc1 and 2.0-rc2 used the 256-bit AES cipher with the cipher mode Electronic Code Book (ECB) for encryption with Encryptor via the ESAPI reference implementation, JavaEncryptor. ECB cipher mode is the simplest cipher mode to use, but it is also cryptographically very weak. If more than one block of plaintext is encrypted with the same key, those identical blocks of ciphertext always encrypt to the same ciphertext block, thus revealing patterns in the plaintext input. For example, these images from Wikipedia's Block cipher modes of operation illustrate this point well:

Ciphertext encrypted with ECB cipher mode are also subject to "block replay attacks". See Bruce Schneier's Applied Cryptography: protocols, algorithms, and source code for details.

In both ESAPI 2.0-rc1 and 2.0-rc2, one can choose other block ciphers (e.g. Blowfish) or other key sizes (e.g., 512-bit AES), but encryption in ESAPI 2.0-rc1 and 2.0-rc2 only support ECB cipher mode with no padding; it cannot be used to decrypt data encrypted with anything other than an algorithm using ECB cipher mode and no padding. Most encryption (outside of ESAPI) uses some other cipher mode (e.g., Cipher Block Chaining (CBC) mode) and some sort of padding scheme, such as PKCS#5 padding, meaning that they are many situations where ESAPI encryption would not be suitable.

Problems with Symmetric Encryption in ESAPI 2.0-rc1 and 2.0-rc2

The problems with symmetric encryption in ESAPI 2.0-rc1 and 2.0rc2 are many. Here are some of the main issues:

1. The implementation only supported a single encryption key. While this may be sufficient for some uses, clearly there are cases where multiple recipients are involved when you would like to use unique encryption keys for each recipient.

2. ESAPI 2.0-rc1 and 2.0-rc2 came with a default settings for Encryptor.MasterKey and Encryptor.MasterSalt, thus not requiring that these be changed. How many of you forgot to change these?

3. Because the reference implementation in JavaEncryptor allows no means to specify an Initialization Vector (IV), only the weak ECB cipher mode can be used.

4. The symmetric encryption does not provide for any means of confirming message authenticity. Cryptographers consider ensuring message authenticity an important feature of modern cryptosystems.

The Encryption Changes in ESAPI 2.0-rc3 and Later

Briefly speaking, the changes being implemented for ESAPI Java 2.0 are:

1. Starting in ESAPI Java 2.0-rc3, the default cipher transformation is changing to "AES-128/CBC/PKCS5Padding" (i.e., 128-bit AES in CBC cipher mode and PKCS#5 padding) with a random IV. The main reason that 128-bit AES was chosen over 256-bit AES for the default is that 128-bit AES is slightly faster than 256-bit AES, and it does not require you downloading the Unlimited Strength Jurisdiction Policy files from Sun Microsystems. For those believing that 128-bit AES is not sufficiently strong--bruting forcing it (except perhaps with a reasonable quantum computer) is still not feasible--and certain key related attacks (see Biryukov, Dunkelman, Keller, Khovratovich, and Shamir's Cryptology ePrint Archive: Report 2009/374 -- Key Recovery Attacks of Practical Complexity on AES Variants With Up To 10 Rounds for further details) show that longer key sizes in AES may not always be the best.

2. Because cipher modes other than ECB require an Initialization Vector (IV), a mechanism to handle both fixed and random IVs has been added. The use of the “fixed” IV should only be considered if there is a need to be compatible with some legacy software or third party software package. Those cases should be rare.

3. The original attempt to preserve backward compatibility with ESAPI Java 1.4 (via org.owasp.esapi.reference.crypto.LegacyJavaEncryptor) has been removed. This was put to a vote on Jan 31, 2010 before both the ESAPI Users and ESAPI Developers mailing lists and the lack of response was deafening. There literally was but a single response and that was to kill off LegacyJavaEncryptor. (By this time, the two symmetric encryption interfaces in Encryptor had already been deprecated.)

4. The byte-encoding has been changed from native byte encoding to UTF-8 byte-encoding throughout ESAPI 2.0 and not just for encryption. This was needed so that one could (say) encrypt on a Sparc running Solaris and then move the encrypted data to an Intel host running Windows and still be able to decrypt the data. Without this change to use uniform encoding throughout, this could not be guaranteed.

The Good, the Bad, and the Ugly

Or put another way, there are always trade-offs to be made...

The Good

We get improved security by encouraging the use of stronger cipher modes (and ESAPI is all about "enterprise security", right?). Furthermore, it is possible to choose a preferred JCE provider for ESAPI and ESAPI will attempt to use that provider. For example, if you wish to use Bouncy Castle rather than the SunJCE, you can do so. One place where this might be a significant advantage is if you need to use a FIPS 140-2 compliant JCE implementation. Whereas in production environments, you ultimately will want to ensure this by editing your $JAVA_HOME/jre/lib/security/java.security file, this is often inconvenient for developers who simply wish to test because this file often requires super-user access to alter. In such cases, setting the new ESAPI property Encryptor.PreferredJCEProvider to the fully qualified class name of your FIPS 140-2 JCE provider in your ESAPI.properties file and you can test without any other changes to your code or you environment.

We also get message authenticity as long as we don't use any of the deprecated encryption methods or intentionally disable it. (The only sane reason for intentionally disabling the MAC calculation is for FIPS 140-2 compliance. This MAC calculation requires deriving two new SecretKeys from the original key—one to use for message authenticity and a second one to use for message confidentiality. While we believe that this mechanism is secure [it was originally suggested by professional cryptographers], this key derivation would not be using FIPS 140-2 approved code. Therefore if you require FIPS 140-2 compliance, you should set “Encryptor.CipherText.useMAC=false” in your ESAPI.properties file.)

The Bad

With cipher modes that require an IV, the same IV must be used both to encrypt and decrypt. While it is not required that the IV be kept secret from adversaries, there are some attacks that are possible if the adversary is permitted to alter the IV at will and observe the results of the ensuing decryption attempt.

So that leaves two choices for the IV:

• Using a fixed IV: The IV can be agreed upon out-of-band by the parties exchanging encrypted messages. In this way, the encrypted data that is stored and/or transmitted can be limited to only the raw ciphertext. (WARNING: This case should be reserved for situations requiring compatibility with legacy or third party software packages.)

• Using a random IV: Most cryptographers prefer to use a random IV, typically a different one with each message to be encrypted. However, doing so on encrypted data that will be persisted (e.g., to a database) or transmitted to the recipient this random IV must be stored / made known. Therefore, the raw ciphertext can no longer suffice; whatever random IV that was chosen must be communicated.

Likewise, the use of padding is going to add some overhead to the length of the ciphertext.

For example, a short plaintext of 0-15 bytes, this overhead can be around 200% of the total plaintext size. This overhead diminishes (as a percent of the total plaintext size) as the plaintext size increases. Here is a table that illustrates this:

__________________________________

1. That is, the String that is returned by the deprecated Encryptor.decrypt(String) method.

As you can see, since the size of the IV and the amount of padding bytes are fixed at a maximum, this overhead goes down as the length of the plaintext message increases. The IV is always a fixed length for a given cipher; it is always the same as the cipher block size. The padding can vary, but for PKCS5 padding, the padding will be between 1 to the cipher block size (in bits) / 8 bytes. For AES, the cipher block size is 128-bits, but more typically, a cipher's block size is 64-bits so the padding would be between 1 to 16 bytes for AES and 1 to 8 bytes for a 64-bit block size cipher and the IV would be IV would be 16 bytes for AES and 8 bytes for most other ciphers.

The Ugly

Well, so far, this "bad" news may be bad for you but good for your database and/or storage vendor, but you probably can live with it, especially if you are willing to do a little bit of recoding of your database table sizes.

But wait Skippy, don't go running off just quite yet. As Robert Heinlein wrote in his 1966 novel The Moon is a Harsh Mistress "There ain't no such thing as a free lunch". (Some of us more hardened cynics know it more commonly as TANSTAAFL.)

As mentioned earlier, backward compatibility with ESAPI 1.4 (originally planned via LegacyJavaEncryptor) has been removed because of apparent lack of interest. What this means is if you are one of the unlucky stuckeys who are using ESAPI 1.4 symmetric encryption to encrypt data and you have persisted this data (and this would include the use of the org.owasp.esapi.EncryptedProperties class in ESAPI 1.4), you will first have to decrypt this encrypted data using ESAPI 1.4 before using encryption with ESAPI 2.0. As they say, “we are sorry for the inconvenience”, but there was an opportunity to make your voice heard so we refuse to take all the blame.

In addition, all this backward compatibility and flexibility comes at the extra cost of complexity, and not merely additional complexity in the reference implementation. There is also some additional complexity in the ESAPI Encryptor interfaces as well.

In particular, there new classes for plaintext (PlainText) and ciphertext (CipherText) class to coalesce all this complexity of handling the ciphertext result from encryption operations. And then there are new encryption and decryption methods for the Encryptor interface. Specifically, the encrypt and decrypt methods have been generalized as:

CipherText encrypt(SecretKey key, PlainText plaintext)
        throws EncryptionException;

and

PlainText decrypt(SecretKey key, CipherText ciphertext)
        throws EncryptionException

(There are also two corresponding encrypt / decrypt methods that omit the initial SecretKey argument and instead use the SecretKey based on Encryptor.MasterKey.)

The two existing interfaces from ESAPI 1.4 and earlier:

String encrypt(String plaintext) throws EncryptionException

and

String decrypt(String ciphertext) throws EncryptionException

are still supported but have been deprecated, mainly because they do not support integrity / authenticity checks which most cryptographers consider essential. Also these older String-based encrypt() / decrypt() interfaces have slightly different semantics than do their ESAPI 1.4 counterparts in that these methods default to CBC cipher mode, PKCS5Padding, and a random IV. This means that at a minimum, the IV must be included as the resulting base64-encoded “ciphertext”.

However, this difference in cipher mode means that if you have data that was previously encrypted using ESAPI 1.4 or earlier, you will not be able to decrypt it using the ESAPI 2.0

String decrypt(String ciphertext) throws EncryptionException

method. If you require such compatibility, you may contact the author, Kevin W. Wall, for assistance.

Note that while the new String-based encrypt() / decrypt() methods use the stronger CBC cipher mode with a random IV, they still do not ensure authenticity and integrity so the more general encrypt() / decrypt() methods should be preferred over the String-based ones. (See below for examples of how to use these new methods.)

Further Details

Further details are provided in the Encryptor Javadoc and the “Symmetric Encryption” section of the ESAPI User Guide.