What should I be aware of when implementing algorithms myself?
Does matching all the test vectors mean my implementations are valid mathematically?


Basically the comments got it, but test vectors are designed to attempt to hit lots of cases, but with high probability will not catch every single mistake. Should you do it? Definitely. Does it mean everything is perfect? No.


  Is implementing mathematics correctly not enough because there are other things I need to plan ahead for?


Like you mention, side channel attacks are of concern. Another concern which has little to do with security is efficiency. That said, if your library is too slow, others might disable it. This could be developers or users (if that option exists in the software).

I am all for doing things for learning purposes. It is a good exercise and can teach a lot. I am a "do it to learn it" kind of person. So I don't want to discourage you from that. If projects you are writing need crypto, and they are to be used by others, you are probably better off using stuff that is out there and has presumably been reviewed by cryptographers.