CAT5/6 to non RJ45 connector assemblies
Depending on your design, you can try the following:


2-mm (preferably) or 2.54-mm (possible) pitch IDC connection, initially designed for flat ribbon cable but works with stranded CAT5 UTP well.
BNC connection, especially as you noted about instrumentation (measurements?), here soldering is needed.  In that case you can also use RCA and similar ones. 
so called "euroconnection", 3.51-mm (or wider if needed), where you can connect both stranded and/or solid UTP.




Maybe it looks like a hack, i.e. non-standard, but as you mentioned your approach is initially not standard too. All these connection are popular (and therefore cheap) and pluggable many times.