Can the act of studying more be a defense mechanism? Let's say he/she wants to get good grades because unconsciously he/she is afraid that otherwise a bad future awaits.

My question, is the act of studying more a defense mechanism? If it is, under which category we can classify it?

I thing closest is displacement, but as I understood if you are angry you redirect your anger with displacement. We can substitute other emotion for anger. If it is frustration, frustration needs to be redirected. But the person pushes herself more here so she is not redirecting her frustration(you can always clarify if I understood a concept wrong or send me a resource to clarify, I study by myself I just like learning).

Waiting for you answers! Thank you.
Whilst I am a defender of Freudian and Neo-Freudian psychology, please be aware that this covers areas of psychology considered by some here to be pseudoscientific.  With that in mind, you are right to question the hyposthesis that any defence mechanism involved would be displacement.

Ego defence mechanisms theorized by Sigmund Freud, could have a role in the situation you are talking about, but they also may not.  Displacement involves displacing emotions towards an issue onto another issue.

Using a similar scenario to yours for an example, someone who is frustrated with their grades in science education may displace their emotions towards those grades on to their tutor.  In this situation, instead of internalizing the issue with themselves not understanding the subject, they blame the tutor for not teaching them properly.

The defence mechanism you are talking about, is loosely connected to sublimation (satisfying an impulse - e.g. aggression - with a substitute object).  In your scenario, the person is channelling their frustration constructively by trying to improve the situation through more studying.