Can you remove a special ship part and keep it for later?
Page 7 of the official rulebook states in relation to Ancient Ship Parts (which are the special type of parts you are referring to):


  you may place this part in any of your Ships (returning an exist­ing part 
  if needed); you may also keep the part next to your
  board and place it later with the Upgrade action; if you later want to
  replace an Ancient Ship Part, it is discarded.


So to answer your question they are discarded.

Source: [Official English Rulebook]