Detecting JTAG pinout
You can probably ignore the 'optional' pins except for RTCK, there should be no debug scenario where these pins are necessary unless your part has the JTAG port disabled by being tied-off in a 'reset' state. If the target does implement these optional pins, the PCB is as likely to hard-wire them as to expose them on a connector. If you have a connector identified as a candidate JTAG port, it may also provide a 3v3 reference (target supply rail) which it is expecting your probe to use (and can likely also be ignored).

RTCK may be present on an old Arm target, anything ARM11 and earlier. This is a sampled and delayed version of the TCK input, re-synchronised to the internal core clock. If RTCK is in use, you need to run your JTAG slow, or wait for the RTCK edges. Anything using the CoreSight DAP won't use RTCK (it could conceivably be wired as a loopback of TCK, but serves no purpose).

Bear in mind that 2-pin Serial-Wire debug ports are now common, you'll need a different sequence to detect these (of the order of 100 bits per pin which is tested as data). There is no simple FSM in this case like in JTAG.

I noticed that the current Open-OCD documentation suggests that JTAG clock speed is limited to ~CPU/6. This is only true for the older ARM cores where the RTAG TAP interfaces directly to the processor. Any processor with a CoreSight DAP has the debug clock completely asynchronous to the internal clocks, and can even run with JTAG/SW clock faster than the processor.

See here for the standard connectors - you might also find a supply, a target voltage reference and a target functional reset - but this will depend on the designer's plan for debug of the board.