Course contentsShow
← Security
Lesson 159 of 3,0524. Asymmetric Cryptography and PKIFree lesson

Small Subgroup and Invalid Curve Attacks

Exploiting weak parameters or malicious public keys to recover private keys or compromise key exchange.

Small Subgroup and Invalid Curve Attacks

What you'll learn: How attackers can manipulate Diffie-Hellman or elliptic curve parameters to weaken or break key exchanges, even when the math looks correct.

The Core Vulnerability

When using Diffie-Hellman (including ECDH from lesson 158), both parties exchange public keys and perform calculations to reach a shared secret. But what if an attacker sends you a malicious public key with special mathematical properties?

Small subgroup attacks exploit weak group parameters. In regular Diffie-Hellman, if an attacker sends you a public key that belongs to a small mathematical subgroup (a tiny subset of valid values), your resulting shared secret will be one of only a few possible values. Instead of having billions of possibilities, the attacker might narrow it down to just 2, 4, or 16 options — trivial to brute-force.

Invalid curve attacks target ECDH specifically. An attacker sends a point that doesn't actually lie on your agreed-upon elliptic curve, but on a different, weaker curve. If your implementation doesn't validate incoming points, it performs calculations on this weak curve, potentially leaking information about your private key through repeated exchanges.

Real-World Impact

These attacks bypass the encryption itself — they manipulate the key exchange phase. If your implementation accepts untrusted public keys without validation, an attacker can:

  • Force your shared secret into a small, guessable set
  • Extract bits of your private key through mathematical side channels
  • Completely compromise confidentiality without breaking any encryption algorithm

Defense

Always validate received public keys:

  • Check that DH public keys are in the correct range and not in small subgroups
  • Verify ECDH points actually lie on the intended curve
  • Use standardized, well-vetted parameter groups

Key Takeaway: Accepting untrusted public keys without validation lets attackers manipulate the mathematics of key exchange, turning strong cryptography into weak or broken security.