What should I be aware of when implementing algorithms myself? I plan to build my own crypto library. The project will be primarily for me to learn (and if useful for no other purpose, that is fine). In the past I have implemented a few hashes, and AES quite a few times while learning about it, but those projects were solely for me to learn, not for any field use. With this one I'd like it to work on it as if it were be to used outside of myself.

I understand that using/implementing your own crypto is usually a bad idea (if not always), so the answer to how to approach this is always "just don't". What makes sense in my head is: if all the tests work, the implementations should be valid. So I guess what I mean to ask is, does matching all the test vectors mean my implementations are valid mathematically? Is implementing mathematics correctly not enough because there are other things I need to plan ahead for? I know pretty much nothing about side-channel attacks, so I don't know if that has to do with the primitives in use or the system using them.

tl;dr: I don't intend to put it into production, but I want to build as if I were, to learn.
Does matching all the test vectors mean my implementations are valid mathematically?


Basically the comments got it, but test vectors are designed to attempt to hit lots of cases, but with high probability will not catch every single mistake. Should you do it? Definitely. Does it mean everything is perfect? No.


  Is implementing mathematics correctly not enough because there are other things I need to plan ahead for?


Like you mention, side channel attacks are of concern. Another concern which has little to do with security is efficiency. That said, if your library is too slow, others might disable it. This could be developers or users (if that option exists in the software).

I am all for doing things for learning purposes. It is a good exercise and can teach a lot. I am a "do it to learn it" kind of person. So I don't want to discourage you from that. If projects you are writing need crypto, and they are to be used by others, you are probably better off using stuff that is out there and has presumably been reviewed by cryptographers.