Can't specify rpcport for testnet
For any of addnode, connect, port, bind, rpcport, rpcbind, and wallet you should move them into the [test] and/or [main] sections of your bitcoin.conf. You can also move any other setting that you want only applied to mainnet or testnet into the appropriate section.

Example:

server=1
daemon=1

[main]
prune=1000

[test]
rpcport=55555