Is dropping one of my major considered "bad" if I were to study related to that field?
Would this have negative impact in my future career?


Absolutely no one cares what you did in undergrad, except grad school admissions. Further, as Nate Eldredge said in the comments:


  I don't think it would so much be an issue that you dropped the physics major, as that you don't have a physics degree.


So, I think your question is equivalent to:


  Am I less likely to get into grad school for physics/math with only a math degree?


You're certainly less likely to get into physics grad school without a physics degree (though with enough coursework, it's not impossible). For math grad school, I'm less qualified to comment, but I strongly suspect that a dual degree would have more weight than one degree + coursework. So, yes, I think the second degree would help you get into grad school. Only you can weigh this reality against the advantages of self-studying. 

By the way, I also wanted to comment on this line:


  the only reason that I'm doing is so that I can have much more time to self-study by which I learn better.


This has been discussed elsewhere, the consensus being that while many passionate students feel this way, few truly do not benefit from a course's structure and a professor's experience. Just one other factor to consider in your decision.