What other than programming does a computer science degree usually involve?
A computer science education, among answering a lot of "how?" questions, also covers areas whose purpose is to answer most of the "why?" questions. Those are, in my opinion, the most important, mind-opening and motivating for a future computer scientist with a programming background:


Logic
Computer Architecture
Computability 
Complexity
Machine Learning and Artificial Intelligence (which are very closely related and are a part of applied math, which you mentioned. These are of great importance on today's market)


see comments for more examples.

Those will give your programming skills a major boost, enrich you with deeper understanding of what's behind the lines of code and thus give you a great advantage when applying for a future job in this area.

Should you study computer science if you are now interested only in programming? I think absolutely yes. You will not lack programming courses for sure, so that's a plus already. Also, you will learn many more interesting things about programming that maybe you haven't yet encountered to figure out that you are interested in them.