Is it ethical to remove your co-authorship shortly before submission?
Is it ethical to submit the paper as a co-author and wait for the reviews and trying to improve the paper later 


Submitting a paper with major errors is unethical, whereas submitting a paper that needs polishing isn't.


  or is it more fair to remove my co-authorship before the submission?


You can remove yourself, but the paper would probably benefit from your continued support.




  A colleaque of mine suggested that I should care more for my own papers rather than the papers of others.


This is one of your papers.