NIST HMAC Test vectors
RFC4868 is not the HMAC RFC, which is actually RFC 2104. 4868 refers to the use of HMAC within IPSEC, which is why there is a key length restriction.

The maximum length key that can be used internally with HMAC-SHA256 is equal to the block size, 64 bytes or 512 bits. This can be useful in cases where the key is not full entropy such as the shared secret generated by a Diffie-Hellman key exchange; an ECDH secret of 512 bits has around 256-bits of security.

If the key length exceeds the block length, it must be first hashed to get it to fit within the block, this is the reason for the 70 byte test value, to make sure that implementations can properly handle it. The other lengths in the NIST vectors are one equal to the block length, and 2 arbitrarily shorter lengths. If an implementation can handle all 4 test values, it is generally safe to assume it can handle a 32-byte key as well.