Am I removing too much paint from exterior when scraping?
This is a subjective question, but ideally any paint that's not well-bonded should be removed. Eventually it'll flake off as well. 

That said, there's always a balancing act to be done. You'll have to assess the situation and proceed accordingly. It may not be economically practical to strip the paint entirely, depending on intended lifetime of the siding, etc.