Rubin Shrestha
Rubin Shrestha
Blog

Why am I experiencing TLS handshake failures?

by |Published

For my site, I have a few subdomains, each with separate certificates. I have had no issues until about a month ago. For reference, no Nginx, TLS, firewall, or system configuration changes were made prior to the issue starting.

For a small (but seemingly increasing) portion of users, they get ERR_SSL_PROTOCOL_ERROR when navigating to a specific subdomain (api subdomain) in their browser. All other subdomains work fine.

This appears to only happen for US users, although I can’t be sure.

The server uses nginx for all subdomains. For this post, I have replaced the hostname with example.com.

Symptoms:

  • Browser: ERR_SSL_PROTOCOL_ERROR
  • cURL: curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_INVALID_TOKEN (0x80090308) - The token supplied to the function is invalid
  • Desktop app: Reqwest(reqwest::Error { kind: Request, url: "https://api.example.com/.....", source: hyper_util::client::legacy::Error(Connect, Os { code: -2146893048, kind: Uncategorized, message: "The token supplied to the function is invalid" }) })
  • tcpdump shows a TCP 3 way handshake, ACK packets, and RST packets, but no TLS packets. (for unaffected IPs, TLS packets such as ClientHello and ServerHello are visible)
  • openssl s_client from an affected client shows SSL handshake has read 5 bytes and written 327 bytes, followed by lines such as No ALPN negotiated.

Additionally, an affected user ran openssl s_client -debug. After sending the ClientHello, their client reports reading 5 bytes: ff ff ff ff ff, followed by ssl3_get_record:wrong version number.

One affected user tried visiting another subdomain (analytics.example.com). When making a request to that subdomain (first visit / clicking a button on the page to make a request), there would be a brief window where the user could visit the API subdomain in their browser (but still not with cURL) for a minute or two.

Other users have reported being able to visit the page in their browser, but unable to connect through cURL or a desktop app (which uses the API subdomain).

Steps I’ve taken to attempt to resolve it:

  • Modified nginx config: disabling TLS1.3, disabling HTTP2, changing ciphers, changing other nginx config options such as session cache and ecdh curve.
  • Issuing a new certificate
  • Using RSA certificate / Using ECDSA certificate
  • Issuing a new certificate containing 2 different subdomains, analytics and api: analytics.example.com continues to work fine for all users, while api.example.com continues to be problematic.

nginx -T output:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:


user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;

}

# configuration file /etc/nginx/mime.types:
......

# configuration file /etc/nginx/sites-enabled/analytics.example.com:
server {
    http2 on;
    server_name analytics.example.com;

    location / {
        proxy_pass http://127.0.0.1:10516; 
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection upgrade;
    }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/analytics.example.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/analytics.example.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}


server {
    if ($host = analytics.example.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    listen 80;
    server_name analytics.example.com;
    return 404; # managed by Certbot


}

# configuration file /etc/nginx/sites-enabled/api.example.com:
server {
    http2 on;
    server_name api.example.com;

    location / {
        proxy_pass http://127.0.0.1:7148/id/api/; 
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection upgrade;
    }

    listen 443 ssl default_server; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/api.example.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/api.example.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}


server {
    if ($host = api.example.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80;
    server_name api.example.com;
    return 404; # managed by Certbot
}

# configuration file /etc/letsencrypt/options-ssl-nginx.conf:
# This file contains important security parameters. If you modify this file
# manually, Certbot will be unable to automatically provide future security
# updates. Instead, Certbot will print and log an error message with a path to
# the up-to-date file that you will need to refer to when manually updating
# this file. Contents are based on https://ssl-config.mozilla.org

ssl_session_cache shared:le_nginx_SSL:10m;
#ssl_session_cache off;
#ssl_session_timeout 1440m;
ssl_session_timeout 1d;
ssl_session_tickets off;

ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;
ssl_ecdh_curve X25519:prime256v1:secp384r1;

ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384";

Full openssl output from affected user:

> openssl s_client -connect api.example.com:443 -servername api.example.com -showcerts
CONNECTED(000001E8)
13012:error:1408F10B:SSL routines:ss13_get_record:wrong version number:ss1/record/ss13_record.c:331:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 327 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)

Other information:

  • Server is not behind any proxy such as Cloudflare.
  • Subdomains have only A records, no AAAA records.
  • Nginx is the only process listening on port 443.

You may also like