err_set_debug(): Prevent possible recursion on malloc failure

Fixes #19331

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/19332)
