Security with HTTPS and SSL

The Secure Sockets Layer (SSL)—now technically known as Transport Layer Security (TLS)—is a common building block for encrypted communications between clients and servers. It's possible that an application might use SSL incorrectly such that malicious entities may be able to intercept an app's data over the network. To help you ensure that this does not happen to your app, this article highlights the common pitfalls when using secure network protocols and addresses some larger concerns about using Public-Key Infrastructure (PKI).

Concepts

In a typical SSL usage scenario, a server is configured with a certificate containing a public key as well as a matching private key. As part of the handshake between an SSL client and server, the server proves it has the private key by signing its certificate with public-key cryptography.

However, anyone can generate their own certificate and private key, so a simple handshake doesn't prove anything about the server other than that the server knows the private key that matches the public key of the certificate. One way to solve this problem is to have the client have a set of one or more certificates it trusts. If the certificate is not in the set, the server is not to be trusted.

There are several downsides to this simple approach. Servers should be able to upgrade to stronger keys over time ("key rotation"), which replaces the public key in the certificate with a new one. Unfortunately, now the client app has to be updated due to what is essentially a server configuration change. This is especially problematic if the server is not under the app developer's control, for example if it is a third party web service. This approach also has issues if the app has to talk to arbitrary servers such as a web browser or email app.

In order to address these downsides, servers are typically configured with certificates from well known issuers called Certificate Authorities (CAs). The host platform generally contains a list of well known CAs that it trusts. As of Android 4.2 (Jelly Bean), Android currently contains over 100 CAs that are updated in each release. Similar to a server, a CA has a certificate and a private key. When issuing a certificate for a server, the CA signs the server certificate using its private key. The client can then verify that the server has a certificate issued by a CA known to the platform.

However, while solving some problems, using CAs introduces another. Because the CA issues certificates for many servers, you still need some way to make sure you are talking to the server you want. To address this, the certificate issued by the CA identifies the server either with a specific name such as gmail.com or a wildcarded set of hosts such as *.google.com.

The following example will make these concepts a little more concrete. In the snippet below from a command line, the openssl tool's s_client command looks at Wikipedia's server certificate information. It specifies port 443 because that is the default for HTTPS. The command sends the output of openssl s_client to openssl x509, which formats information about certificates according to the X.509 standard. Specifically, the command asks for the subject, which contains the server name information, and the issuer, which identifies the CA.

$ openssl s_client -connect wikipedia.org:443 | openssl x509 -noout -subject -issuer
subject= /serialNumber=sOrr2rKpMVP70Z6E9BT5reY008SJEdYv/C=US/O=*.wikipedia.org/OU=GT03314600/OU=See www.rapidssl.com/resources/cps (c)11/OU=Domain Control Validated - RapidSSL(R)/CN=*.wikipedia.org
issuer= /C=US/O=GeoTrust, Inc./CN=RapidSSL CA

You can see that the certificate was issued for servers matching *.wikipedia.org by the RapidSSL CA.

An HTTPS Example

Assuming you have a web server with a certificate issued by a well known CA, you can make a secure request with code as simple this:

URL url = new URL("https://wikipedia.org");
URLConnection urlConnection = url.openConnection();
InputStream in = urlConnection.getInputStream();
copyInputStreamToOutputStream(in, System.out);

Yes, it really can be that simple. If you want to tailor the HTTP request, you can cast to an HttpURLConnection. The Android documentation for HttpURLConnection has further examples about how to deal with request and response headers, posting content, managing cookies, using proxies, caching responses, and so on. But in terms of the details for verifying certificates and hostnames, the Android framework takes care of it for you through these APIs. This is where you want to be if at all possible. That said, below are some other considerations.

Common Problems Verifying Server Certificates

Suppose instead of receiving the content from getInputStream(), it throws an exception:

javax.net.ssl.SSLHandshakeException: java.security.cert.CertPathValidatorException: Trust anchor for certification path not found.
        at org.apache.harmony.xnet.provider.jsse.OpenSSLSocketImpl.startHandshake(OpenSSLSocketImpl.java:374)
        at libcore.net.http.HttpConnection.setupSecureSocket(HttpConnection.java:209)
        at libcore.net.http.HttpsURLConnectionImpl$HttpsEngine.makeSslConnection(HttpsURLConnectionImpl.java:478)
        at libcore.net.http.HttpsURLConnectionImpl$HttpsEngine.connect(HttpsURLConnectionImpl.java:433)
        at libcore.net.http.HttpEngine.sendSocketRequest(HttpEngine.java:290)
        at libcore.net.http.HttpEngine.sendRequest(HttpEngine.java:240)
        at libcore.net.http.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:282)
        at libcore.net.http.HttpURLConnectionImpl.getInputStream(HttpURLConnectionImpl.java:177)
        at libcore.net.http.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:271)

This can happen for several reasons, including:

1. The CA that issued the server certificate was unknown
2. The server certificate wasn't signed by a CA, but was self signed
3. The server configuration is missing an intermediate CA

The following sections discuss how to address these problems while keeping your connection to the server secure.

Unknown certificate authority

In this case, the SSLHandshakeException occurs because you have a CA that isn't trusted by the system. It could be because you have a certificate from a new CA that isn't yet trusted by Android or your app is running on an older version without the CA. More often a CA is unknown because it isn't a public CA, but a private one issued by an organization such as a government, corporation, or education institution for their own use.

Fortunately, you can teach HttpsURLConnection to trust a specific set of CAs. The procedure can be a little convoluted, so below is an example that takes a specific CA from an InputStream, uses it to create a KeyStore, which is then used to create and initialize a TrustManager. A TrustManager is what the system uses to validate certificates from the server and—by creating one from a KeyStore with one or more CAs—those will be the only CAs trusted by that TrustManager.

Given the new TrustManager, the example initializes a new SSLContext which provides an SSLSocketFactory you can use to override the default SSLSocketFactory from HttpsURLConnection. This way the connection will use your CAs for certificate validation.

Here is the example in full using an organizational CA from the University of Washington:

// Load CAs from an InputStream
// (could be from a resource or ByteArrayInputStream or ...)
CertificateFactory cf = CertificateFactory.getInstance("X.509");
// From https://www.washington.edu/itconnect/security/ca/load-der.crt
InputStream caInput = new BufferedInputStream(new FileInputStream("load-der.crt"));
Certificate ca;
try {
    ca = cf.generateCertificate(caInput);
    System.out.println("ca=" + ((X509Certificate) ca).getSubjectDN());
} finally {
    caInput.close();
}

// Create a KeyStore containing our trusted CAs
String keyStoreType = KeyStore.getDefaultType();
KeyStore keyStore = KeyStore.getInstance(keyStoreType);
keyStore.load(null, null);
keyStore.setCertificateEntry("ca", ca);

// Create a TrustManager that trusts the CAs in our KeyStore
String tmfAlgorithm = TrustManagerFactory.getDefaultAlgorithm();
TrustManagerFactory tmf = TrustManagerFactory.getInstance(tmfAlgorithm);
tmf.init(keyStore);

// Create an SSLContext that uses our TrustManager
SSLContext context = SSLContext.getInstance("TLS");
context.init(null, tmf.getTrustManagers(), null);

// Tell the URLConnection to use a SocketFactory from our SSLContext
URL url = new URL("https://certs.cac.washington.edu/CAtest/");
HttpsURLConnection urlConnection =
    (HttpsURLConnection)url.openConnection();
urlConnection.setSSLSocketFactory(context.getSocketFactory());
InputStream in = urlConnection.getInputStream();
copyInputStreamToOutputStream(in, System.out);

With a custom TrustManager that knows about your CAs, the system is able to validate that your server certificate come from a trusted issuer.

Caution: Many web sites describe a poor alternative solution which is to install a TrustManager that does nothing. If you do this you might as well not be encrypting your communication, because anyone can attack your users at a public Wi-Fi hotspot by using DNS tricks to send your users' traffic through a proxy of their own that pretends to be your server. The attacker can then record passwords and other personal data. This works because the attacker can generate a certificate and—without a TrustManager that actually validates that the certificate comes from a trusted source—your app could be talking to anyone. So don't do this, not even temporarily. You can always make your app trust the issuer of the server's certificate, so just do it.

Self-signed server certificate

The second case of SSLHandshakeException is due to a self-signed certificate, which means the server is behaving as its own CA. This is similar to an unknown certificate authority, so you can use the same approach from the previous section.

You can create your own TrustManager, this time trusting the server certificate directly. This has all of the downsides discussed earlier of tying your app directly to a certificate, but can be done securely. However, you should be careful to make sure your self-signed certificate has a reasonably strong key. As of 2012, a 2048-bit RSA signature with an exponent of 65537 expiring yearly is acceptable. When rotating keys, you should check for recommendations from an authority (such as NIST) about what is acceptable.

Missing intermediate certificate authority

The third case of SSLHandshakeException occurs due to a missing intermediate CA. Most public CAs don't sign server certificates directly. Instead, they use their main CA certificate, referred to as the root CA, to sign intermediate CAs. They do this so the root CA can be stored offline to reduce risk of compromise. However, operating systems like Android typically trust only root CAs directly, which leaves a short gap of trust between the server certificate—signed by the intermediate CA—and the certificate verifier, which knows the root CA. To solve this, the server doesn't send the client only it's certificate during the SSL handshake, but a chain of certificates from the server CA through any intermediates necessary to reach a trusted root CA.

To see what this looks like in practice, here's the mail.google.com certificate chain as viewed by the openssl s_client command:

$ openssl s_client -connect mail.google.com:443
---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=mail.google.com
   i:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
 1 s:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
---

This shows that the server sends a certificate for mail.google.com issued by the Thawte SGC CA, which is an intermediate CA, and a second certificate for the Thawte SGC CA issued by a Verisign CA, which is the primary CA that's trusted by Android.

However, it is not uncommon to configure a server to not include the necessary intermediate CA. For example, here is a server that can cause an error in Android browsers and exceptions in Android apps:

$ openssl s_client -connect egov.uscis.gov:443
---
Certificate chain
 0 s:/C=US/ST=District Of Columbia/L=Washington/O=U.S. Department of Homeland Security/OU=United States Citizenship and Immigration Services/OU=Terms of use at www.verisign.com/rpa (c)05/CN=egov.uscis.gov
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 International Server CA - G3
---

What is interesting to note here is that visiting this server in most desktop browsers does not cause an error like a completely unknown CA or self-signed server certificate would cause. This is because most desktop browsers cache trusted intermediate CAs over time. Once a browser has visited and learned about an intermediate CA from one site, it won't need to have the intermediate CA included in the certificate chain the next time.

Some sites do this intentionally for secondary web servers used to serve resources. For example, they might have their main HTML page served by a server with a full certificate chain, but have servers for resources such as images, CSS, or JavaScript not include the CA, presumably to save bandwidth. Unfortunately, sometimes these servers might be providing a web service you are trying to call from your Android app, which is not as forgiving.

There are two approaches to solve this issue:

• Configure the server to include the intermediate CA in the server chain. Most CAs provide documentation on how to do this for all common web servers. This is the only approach if you need the site to work with default Android browsers at least through Android 4.2.
• Or, treat the intermediate CA like any other unknown CA, and create a TrustManager to trust it directly, as done in the previous two sections.

Common Problems with Hostname Verification

As mentioned at the beginning of this article, there are two key parts to verifying an SSL connection. The first is to verify the certificate is from a trusted source, which was the focus of the previous section. The focus of this section is the second part: making sure the server you are talking to presents the right certificate. When it doesn't, you'll typically see an error like this:

java.io.IOException: Hostname 'example.com' was not verified
        at libcore.net.http.HttpConnection.verifySecureSocketHostname(HttpConnection.java:223)
        at libcore.net.http.HttpsURLConnectionImpl$HttpsEngine.connect(HttpsURLConnectionImpl.java:446)
        at libcore.net.http.HttpEngine.sendSocketRequest(HttpEngine.java:290)
        at libcore.net.http.HttpEngine.sendRequest(HttpEngine.java:240)
        at libcore.net.http.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:282)
        at libcore.net.http.HttpURLConnectionImpl.getInputStream(HttpURLConnectionImpl.java:177)
        at libcore.net.http.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:271)

One reason this can happen is due to a server configuration error. The server is configured with a certificate that does not have a subject or subject alternative name fields that match the server you are trying to reach. It is possible to have one certificate be used with many different servers. For example, looking at the google.com certificate with openssl s_client -connect google.com:443 | openssl x509 -text you can see that a subject that supports *.google.com but also subject alternative names for *.youtube.com, *.android.com, and others. The error occurs only when the server name you are connecting to isn't listed by the certificate as acceptable.

Unfortunately this can happen for another reason as well: virtual hosting. When sharing a server for more than one hostname with HTTP, the web server can tell from the HTTP/1.1 request which target hostname the client is looking for. Unfortunately this is complicated with HTTPS, because the server has to know which certificate to return before it sees the HTTP request. To address this problem, newer versions of SSL, specifically TLSv.1.0 and later, support Server Name Indication (SNI), which allows the SSL client to specify the intended hostname to the server so the proper certificate can be returned.

Fortunately, HttpsURLConnection supports SNI since Android 2.3. Unfortunately, Apache HTTP Client does not, which is one of the many reasons we discourage its use. One workaround if you need to support Android 2.2 (and older) or Apache HTTP Client is to set up an alternative virtual host on a unique port so that it's unambiguous which server certificate to return.

The more drastic alternative is to replace HostnameVerifier with one that uses not the hostname of your virtual host, but the one returned by the server by default.

Caution: Replacing HostnameVerifier can be very dangerous if the other virtual host is not under your control, because a man-in-the-middle attack could direct traffic to another server without your knowledge.

If you are still sure you want to override hostname verification, here is an example that replaces the verifier for a single URLConnection with one that still verifies that the hostname is at least on expected by the app:

// Create an HostnameVerifier that hardwires the expected hostname.
// Note that is different than the URL's hostname:
// example.com versus example.org
HostnameVerifier hostnameVerifier = new HostnameVerifier() {
    @Override
    public boolean verify(String hostname, SSLSession session) {
        HostnameVerifier hv =
            HttpsURLConnection.getDefaultHostnameVerifier();
        return hv.verify("example.com", session);
    }
};

// Tell the URLConnection to use our HostnameVerifier
URL url = new URL("https://example.org/");
HttpsURLConnection urlConnection =
    (HttpsURLConnection)url.openConnection();
urlConnection.setHostnameVerifier(hostnameVerifier);
InputStream in = urlConnection.getInputStream();
copyInputStreamToOutputStream(in, System.out);

But remember, if you find yourself replacing hostname verification, especially due to virtual hosting, it's still very dangerous if the other virtual host is not under your control and you should find an alternative hosting arrangement that avoids this issue.

Warnings About Using SSLSocket Directly

So far, the examples have focused on HTTPS using HttpsURLConnection. Sometimes apps need to use SSL separate from HTTP. For example, an email app might use SSL variants of SMTP, POP3, or IMAP. In those cases, the app would want to use SSLSocket directly, much the same way that HttpsURLConnection does internally.

The techniques described so far to deal with certificate verification issues also apply to SSLSocket. In fact, when using a custom TrustManager, what is passed to HttpsURLConnection is an SSLSocketFactory. So if you need to use a custom TrustManager with an SSLSocket, follow the same steps and use that SSLSocketFactory to create your SSLSocket.

Caution: SSLSocket does not perform hostname verification. It is up the your app to do its own hostname verification, preferably by calling getDefaultHostnameVerifier() with the expected hostname. Further beware that HostnameVerifier.verify() doesn't throw an exception on error but instead returns a boolean result that you must explicitly check.

Here is an example showing how you can do this. It shows that when connecting to gmail.com port 443 without SNI support, you'll receive a certificate for mail.google.com. This is expected in this case, so check to make sure that the certificate is indeed for mail.google.com:

// Open SSLSocket directly to gmail.com
SocketFactory sf = SSLSocketFactory.getDefault();
SSLSocket socket = (SSLSocket) sf.createSocket("gmail.com", 443);
HostnameVerifier hv = HttpsURLConnection.getDefaultHostnameVerifier();
SSLSession s = socket.getSession();

// Verify that the certicate hostname is for mail.google.com
// This is due to lack of SNI support in the current SSLSocket.
if (!hv.verify("mail.google.com", s)) {
    throw new SSLHandshakeException("Expected mail.google.com, "
                                    "found " + s.getPeerPrincipal());
}

// At this point SSLSocket performed certificate verificaiton and
// we have performed hostname verification, so it is safe to proceed.

// ... use socket ...
socket.close();

Blacklisting

SSL relies heavily on CAs to issue certificates to only the properly verified owners of servers and domains. In rare cases, CAs are either tricked or, in the case of Comodo or DigiNotar, breached, resulting in the certificates for a hostname to be issued to someone other than the owner of the server or domain.

In order to mitigate this risk, Android has the ability to blacklist certain certificates or even whole CAs. While this list was historically built into the operating system, starting in Android 4.2 this list can be remotely updated to deal with future compromises.

Pinning

An app can further protect itself from fraudulently issued certificates by a technique known as pinning. This is basically using the example provided in the unknown CA case above to restrict an app's trusted CAs to a small set known to be used by the app's servers. This prevents the compromise of one of the other 100+ CAs in the system from resulting in a breach of the apps secure channel.

Client Certificates

This article has focused on the user of SSL to secure communications with servers. SSL also supports the notion of client certificates that allow the server to validate the identity of a client. While beyond the scope of this article, the techniques involved are similar to specifying a custom TrustManager. See the discussion about creating a custom KeyManager in the documentation for HttpsURLConnection.

Nogotofail: A Network Traffic Security Testing Tool

Nogotofail is a tool gives you an easy way to confirm that your apps are safe against known TLS/SSL vulnerabilities and misconfigurations. It's an automated, powerful, and scalable tool for testing network security issues on any device whose network traffic could be made to go through it.

Nogotofail is useful for three main use cases:

• Finding bugs and vulnerabilities.
• Verifying fixes and watching for regressions.
• Understanding what applications and devices are generating what traffic.

Nogotofail works for Android, iOS, Linux, Windows, Chrome OS, OSX, in fact any device you use to connect to the Internet. There’s an easy-to-use client to configure the settings and get notifications on Android and Linux, as well as the attack engine itself which can be deployed as a router, VPN server, or proxy.

You can access the tool at the Nogotofail open source project.