Rubin Shrestha
Rubin Shrestha
Blog

Site returns HTTP 403 to curl/wget but loads fine in every browser, how do I identify what’s blocking the request?

by |Published

I can open my site, SpotGoldValue, normally in Chrome, Firefox, and Safari, but any command-line HTTP client gets a 403 Forbidden. This is breaking a local script I use to health-check the site and fetch a JSON endpoint.

Reproduce:

curl -I https://spotgoldvalue.com/

returns HTTP/2 403, while the same URL in a browser returns 200 and renders fine.

What I’ve already tried:

  • Spoofing a real browser User-Agent:
    curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0 Safari/537.36" -I https://spotgoldvalue.com/ — still 403.

  • Adding -H "Accept: text/html" and a Referer header — still 403.

  • wget with the same headers — also 403.

  • curl -v shows the TLS handshake completing fine, so it’s not a certificate problem; the 403 comes back from the server/edge after the request is sent.

So something at the edge is fingerprinting the request as non-browser (I suspect a WAF or bot-protection rule looking at the TLS/JA3 fingerprint or missing browser headers like Sec-Fetch-*, rather than just the User-Agent string).

My questions:

  1. What’s the right way, from my own machine, to figure out which layer is returning the 403 (origin vs CDN/WAF)? Which response headers or curl -v output should I be reading to tell them apart?

  2. Is there a reliable way to make curl present a request that passes browser-only bot checks (for example the full Sec-Fetch-* set, Accept-Language, HTTP/2 with browser-like header order), or is TLS fingerprinting something curl fundamentally can’t get past?

  3. Since it’s my own site, is the cleaner fix to allowlist my own IP or a custom header at the edge so my health-check script is exempt?

You may also like