What is the name of this connector on the end of a serial cable (not DB9)?
That is a 6P4P modular plug (6 position 4 pin) often mistakenly referred to as an RJ11.    An RJ45 was also mentioned in the answers. RJ45 is often incorrectly used for 8P8P (8 position 8 pin)  modular jack.    RJ stands for registered jack and each number used after it refers to the wiring scheme originally used.  RJ11 and RJ45 have been used incorrectly for so long they are now excepted terms  much in the same way "scotch" tape is used for any "cellophane" tape.

You should look at the specs for that piece of hardware and see if they have a wiring scheme for their equipment. It depends on what you are connecting to that serial port and how you plan to use it. Some things to consider; is it DTE or DCE, will you need to used flow control, what is the baud rate going to be, does your cable need to be a "null" for what you are doing such as programming VIA the serial port.