Adding 3 way dimmer when two circuits exist
(I am by no means any sort of expert, but I hope this points you in the right direction at least. I'm not an electrician but randomly looked up stuff about switches before because I was curious why they were called &quot;3-way&quot;.)
You mentioned that there are three switches controlling the lights. That means there are two 3-way switches and one 4-way switch. The Wikipedia article on this does a much better job explaining than I can. Essentially you need 3-way switches to have two switches control one light, but you need 4-way switches to allow for more than two.
You said the switch you replaced has 4 wires. Sounds to me like you're looking at the 4-way switch from the middle of the circuit. I would guess you bought three 3-way dimmers and replaced the other two without issue but are confused with this one. I think you simply need a 4-way dimmer.