Find hash and signature algorithms used in an X.509 certificate signed with RSASSA-PSS I'm trying to understand the new PSS scheme, but having trouble figuring out how to parse it properly.

When receiving a certificate with OID of RSASSA-PSS, how can I know which hash and signing algorithm was used?
The ASN.1 module

The OID of RSASSA-PSS is specified as below, however, it should only be used as part of an AlgorithmIdentifier, which is a SEQUENCE of the OID, followed by the parameters. These parameters are NULL for PKCS#1 v1.5 signatures, but for PSS they must be present

Here is the OID:

   --
   -- When id-RSASSA-PSS is used in an AlgorithmIdentifier, the
   -- parameters MUST be present and MUST be RSASSA-PSS-params.
   --
   id-RSASSA-PSS    OBJECT IDENTIFIER ::= { pkcs-1 10 }


and here are the configuration parameters of the algorithm:

   -- AlgorithmIdentifier.parameters for id-RSASSA-PSS.
   -- Note that the tags in this Sequence are explicit.
   --
   RSASSA-PSS-params ::= SEQUENCE {
       hashAlgorithm      [0] HashAlgorithm      DEFAULT sha1,
       maskGenAlgorithm   [1] MaskGenAlgorithm   DEFAULT mgf1SHA1,
       saltLength         [2] INTEGER            DEFAULT 20,
       trailerField       [3] TrailerField       DEFAULT trailerFieldBC
   }


where the hashAlgorithm defines the hash over the data, just like for PKCS#1 v1.5 signatures specify a hash over the data within the message.

However, we still have an undefined maskGenAlgorithm of the type with the same name:

   MaskGenAlgorithm ::= AlgorithmIdentifier { {PKCS1MGFAlgorithms} }


... that's not very helpful, so lets dive deeper:

   --
   -- Allowed mask generation function algorithms.
   -- If the identifier is id-mgf1, the parameters are a HashAlgorithm.
   --
   PKCS1MGFAlgorithms    ALGORITHM-IDENTIFIER ::= {
       { OID id-mgf1 PARAMETERS HashAlgorithm },
       ...  -- Allows for future expansion --
   }


ah, another hash function is defined here for use with the only defined Mask Generation Function, MGF1.

This - possibly different - hash function is used within MGF1 and therefore within PSS to provide the security required for MGF1. 

Finally, a boring little constant for trailerField, so that we're complete:

   TrailerField ::= INTEGER { trailerFieldBC(1) }


Conclusion

OK, let's recapitulate: the signing algorithm is of course PSS, which depends on a mask generation function MGF1 as only option and a hash over the message. The MGF1 however also has a hash function algorithm assigned to it. Furthermore, you also can indicate a salt length - defaulting to 20 bytes - and a trailer field which currently just has one option.

Library design options

This message hash defaults to SHA-1, which is not secure for signing data. It also depends on a separate hash function used for MGF1, which also defaults to SHA-1. However, for MGF1 the hash function is actually secure. This unfortunately leads to different options for library designers:


specify one hash, and let this be the hash for both the message and the MGF1;
specify one hash, and let this be the message hash for just the message and keep MGF1 to use SHA-1;
always let the user specify both hashes, ignoring the fact that there is a default hash.


Commonly unfortunately either option 1 or option 2 is taken, leaving the developer guessing what hash is used for MGF1.

Encodings

OK, that was harder than it should be; let instead look at some options for encodings, using the online Lapo ASN.1 parser to decode the manually generated bytes.

First the one specified in the comments of the ASN.1 module within the standard:

   30 0D
      06 09 2A 86 48 86 F7 0D 01 01 0A
      30 00


which parses to:

   SEQUENCE (2 elem)
      OBJECT IDENTIFIER 1.2.840.113549.1.1.10 rsaPSS (PKCS #1)
      SEQUENCE (0 elem)


that's shorter then you'd expect. That's because the PSS parameters consist of a zero element sequence. This may sound weird, but it simply means that the default values are used, i.e. sha1, mgf1SHA1, a saltLength of 20 bytes and of course the only possible trailerField identifier, set to 1.



You can also specify the exact same structure without default values:

  30 3E
     06 09 2A 86 48  86 F7 0D 01 01 0A
     30 31
        A0 0B
           30 09
              06 05 2B 0E 03 02 1A
              05 00
        A1 18
           30 16
              06 09 2A 86 48 86 F7 0D 01 01 08
              30 09
                 06 05 2B 0E 03 02 1A
                 05 00
        A2 03
           02 01 14
        A3 03
           02 01 01


so this explicit encoding shows the actual structure. In above structure it is easy to replace the default parameters by different ones. 

  SEQUENCE (2 elem)
    OBJECT IDENTIFIER 1.2.840.113549.1.1.10 rsaPSS (PKCS #1)
    SEQUENCE (4 elem)
      [0] (1 elem)
        SEQUENCE (2 elem)
          OBJECT IDENTIFIER 1.3.14.3.2.26 sha1 (OIW)
          NULL
      [1] (1 elem)
        SEQUENCE (2 elem)
          OBJECT IDENTIFIER 1.2.840.113549.1.1.8 pkcs1-MGF (PKCS #1)
          SEQUENCE (2 elem)
            OBJECT IDENTIFIER 1.3.14.3.2.26 sha1 (OIW)
            NULL
      [2] (1 elem)
        INTEGER 20
      [3] (1 elem)
        INTEGER 1


WARNING: beware that if a value has a default, and if the default value is being used then DER encoding specifies that it MUST NOT be encoded. So above is valid BER but not valid DER. 

X.509 specifies that the signature field (of type AlgorithmIdentifier is in the TBSCertificate data (TBS means "to be signed", the data that is used in the signature calculation). The TBSCertificate data MUST be DER encoded. So above is not valid in a certificate.



Of course, as the numbering of SEQUENCE with defaults is explicit, generally it would be more logical to just specify the [0] and [1] parameters (tag encoding A0 and A1and leave the rest to the default).

So I would expect API's to generate e.g.:

  30 1E
     06 09 2A 86 48  86 F7 0D 01 01 0A
     30 11
        A0 0F
           30 0D
              06 09 60 86 48 01 65 03 04 02 01
              05 00


which parses as:

  SEQUENCE (2 elem)
    OBJECT IDENTIFIER 1.2.840.113549.1.1.10 rsaPSS (PKCS #1)
    SEQUENCE (1 elem)
      [0] (1 elem)
        SEQUENCE (2 elem)
          OBJECT IDENTIFIER 2.16.840.1.101.3.4.2.1 sha-256 (NIST Algorithm)
          NULL


for SHA-256 as message hash and the default SHA-1 as MGF1 hash; the rest is set to the default values as well.



And, for the ones that uses the same SHA-256 hash for the message and MGF1:

  30 3C
     06 09 2A 86 48  86 F7 0D 01 01 0A
     30 2F
        A0 0F
           30 0D
              06 09 60 86 48 01 65 03 04 02 01
              05 00
        A1 1C
           30 1A
              06 09 2A 86 48 86 F7 0D 01 01 08
              30 0D
                 06 09 60 86 48 01 65 03 04 02 01
                 05 00


which parses as:

SEQUENCE (2 elem)
  OBJECT IDENTIFIER 1.2.840.113549.1.1.10 rsaPSS (PKCS #1)
  SEQUENCE (2 elem)
    [0] (1 elem)
      SEQUENCE (2 elem)
        OBJECT IDENTIFIER 2.16.840.1.101.3.4.2.1 sha-256 (NIST Algorithm)
        NULL
    [1] (1 elem)
      SEQUENCE (2 elem)
        OBJECT IDENTIFIER 1.2.840.113549.1.1.8 pkcs1-MGF (PKCS #1)
        SEQUENCE (2 elem)
          OBJECT IDENTIFIER 2.16.840.1.101.3.4.2.1 sha-256 (NIST Algorithm)
          NULL


this just leaves the salt and trailer field to the default.

Lamentation on default values

It might not come as a surprise that I'm not a fan of default values. They complicate the hell out of structures like this, and once you've chosen a default, such as SHA-1, it is pretty likely that you'll get into trouble as well, as SHA-1 should be considered broken by now. So we now have a provable secure padding mechanism that defaults to an insecure hash.

A slightly larger algorithm identifier that can be canonically encoded (i.e. has just one representation in bytes) and explicitly specifies the algorithms should have been preferred over the ASN.1 hell it is now. As stated above, the DER encoding requires to leave out values if they are identical to the default one.