BIP0016 pay-to-scripthash transaction detection BIP0016 acknowledges it works by "recognizing one 'special' form of scriptPubKey", so P2SH transactions are actually special cases.

What are the exact rules to detect these special cases?
The document you linked to provides all the information you need.

Specifically, for an output to be recognized as pay-to-scripthash, its scriptPubKey must match the format

OP_HASH160 [20-byte-hash-value] OP_EQUAL


The scriptSig of the redeeming input is validated using different rules:


  
  Validation fails if there are any operations other than "push data" operations in the scriptSig.
  Normal validation is done: an initial stack is created from the signatures and {serialized script}, and the hash of the script is computed and validation fails immediately if it does not match the hash in the outpoint.
  {serialized script} is popped off the initial stack, and the transaction is validated again using the popped stack and the deserialized script as the scriptPubKey.
  


Older clients would accept such inputs, but they would not validate the included script. Therefore, the addition of these rules is a soft fork.