Extent of "unscientific" or wrong papers in research mathematics
I would say this is a common, but not fatal, issue.  I think most mathematicians are aware of several papers in their field with serious errors/gaps without retractions/errata, and over time at least the wrong papers which are important get discovered.  (e.g., see this MO post for temporary counterexamples---there aren't too many that went unnoticed for too long).  There are also some grey areas where most people aren't sure if they can trust certain papers or not, and in certain areas/topics it's more of an issue than others.

The culture in mathematics is that retractions are typically for academic dishonesty, and incorrect papers either get corrected (or at least have their flaws exposed) in other papers or with errata.  

What to do:


for an individual paper, check its review on MathSciNet, and check papers that reference that paper: if many people use it freely, it's probably okay, but if someone later found an issue they will often mention it in another paper.  if the paper has a corrigendum, it will be linked in MathSciNet.
if you start reading the paper and for this or other reasons become suspicious, talk to experts---usually if a paper is important and there is a serious issue, experts become aware of the situation relatively quickly 
check if the author has errata/comments on their webpage
some authors are more reliable than others, and they get a reputation for it; this is something you typically learn over time from your own research and talking to experts.
don't panic: if you happen to use a result that was wrong and don't realize it (or publish a paper yourself with a major error), that's okay.  it happens from time to time and you can't check everything.  you might discover the error later or someone might point it out.  in any case, you are often able to then write another paper correcting the situation.
don't contribute to the problem: put errata for your own papers on your website and if there is a major error not corrected in a later paper, publish a corrigendum
(optional) keep track of known issues in a personal notebook along with their resolution.  i do this because there are quite a number of papers in my area with errors/gaps and i have a bad memory.  i contemplated making this public and letting other people contribute to this list, but i haven't so far mainly because i'm not sure i want to put in the effort to update it and make sure it is fair to all parties involved.  however i have shared this list with some people individually.