Which wire should be capped when replacing an old 4 wire smoke alarm with a 3 wire alarm
I'm assuming you also have a ground wire? You should use the orange wire as the interconnect, and cap the brown. You could even go as far as to use red electrical tape to wrap the orange wire near the marrette so it's properly labeled, but it's not necessary. Just make sure you're using the same wire throughout and it will be fine.

However, do be advised that you will need to replace all your smoke alarms.  Interconnected smoke alarms generally do not communicate between brands.  You will usually need to buy multiple of the same model for them to work properly.  This is especially true switching between 4-wire smoke detectors to a 3-wire model.  Is this a condo or apartment building?  If so, the smoke detectors are all interconnected, no?  In which case you probably should not be replacing just one, as that won't trigger the alarm throughout the building.

Lastly, 4-wire smoke detectors are often tied into a security panel.  Do you have a security system installed?  If so, firstly, contact your security company before disconnecting your smoke alarm, or you may get a visit from the fire department.  Secondly, you may not be able to switch to a 3-wire model.  It will prevent your monitored smoke alarm from working.