Replacing RJ45 connector with another type connector
You would only be able to answer this by examining the return loss and crosstalk if using duplex mode for the distance you need. Given that RJ-45 come with ferrite common mode filters as an option, you would lose this noise improvment. When choosing a connector, beware that this is one of the most common types and your ability to locate them is limited only by your awareness of distributors.

Characteristic impedance is geometric property of conductor spacing and thickness or in coaxial inner OD/ID for a given dielectric constant.  This defines the distributed inductance to capacitance ratio which is a function for impedance that must be matched to each 110 Ohm line.