Do all polymers contain either carbon or silicon?
No, neither C nor Si have to be present in a polymer.
IUPAC defines polymer simply as 


  A substance composed of macromolecules.


In turn, there is no limitations on the elements the macromolecules may consist of.
In fact, there is an extensive class of inorganic polymers free of carbon and silicon.