OAEP size-check underflow permits heap writes past tiny RSA frames

The OAEP encoder subtracts its fixed overhead from an unsigned frame length before proving that the overhead fits. A public gcry_pk_encrypt call using a supported 1024-bit RSA key and SHA-512 therefore accepts a one-byte message, allocates 128 bytes, and writes a 64-byte digest starting at offset 65 -- one byte beyond the secure allocation.

Vulnerable code

cipher/rsa-common.c, _gcry_rsa_oaep_encode:

c
  hlen = _gcry_md_get_algo_dlen (algo);

  /* We skip step 1a which would be to check that LABELLEN is not
     greater than 2^61-1.  See rfc-3447 7.1.1. */

  /* Step 1b.  Note that the obsolete rfc-2437 uses the check:
     valuelen > nframe - 2 * hlen - 1 .  */
  if (valuelen > nframe - 2 * hlen - 2 || !nframe)
    {
      /* Can't encode a VALUELEN value in a NFRAME bytes frame. */
      return GPG_ERR_TOO_SHORT; /* The key is too short.  */
    }

  /* Allocate the frame.  */
  frame = xtrycalloc_secure (1, nframe);
  if (!frame)
    return gpg_err_code_from_syserror ();

  /* Step 2a: Compute the hash of the label.  We store it in the frame
     where later the maskedDB will commence.  */
  _gcry_md_hash_buffer (algo, frame + 1 + hlen, label, labellen);

Why it matters

Applications commonly encrypt to externally supplied public keys. Outside FIPS mode, 1024-bit RSA remains accepted, so no malformed tiny modulus is required: OAEP/SHA-512 needs at least 130 frame bytes, but a 1024-bit modulus supplies 128. The audit's public-API ASan reproducer reported a 64-byte heap-buffer-overflow in _gcry_md_hash_buffer; the same test safely returned GPG_ERR_TOO_SHORT after reordering the check.

Proposed fix

Compute overhead = 2 * hlen + 2 with checked size_t arithmetic, reject nframe < overhead, and only then evaluate valuelen > nframe - overhead. Add public-API regression cases for every supported OAEP digest at frame sizes immediately below, at, and above its minimum, including 1024-bit RSA with SHA-512.