Should I remove these use cases of MD5/SHA1 from my program?
Do you know the article If You’re Typing the Letters A-E-S Into Your Code You’re Doing It Wrong? This is basically the same, but with a focus on hashes.

There is a quite important misconception in your system:


you stated "One of its design goals was high security."
and then you start implementing your selfmade algorithms, even stating: "same key is used for enc and hmac; yes i know it is considered bad practice, but not really serious so far"


So, what is your goal here? Actual high security, in a real application? Here a few pointers, if that is actually the goal:


You should probably start by throwing out all those parts, which you designed yourself, and start utilizing proper crypto libraries and the correct primitives. E.g. key derivation functions exist for a reason. 
Randomness (which you call entropy) is nothing to be taken lightly. Just because you implemented a system, you can't figure out how to break yourself, doesn't mean it is secure. Use a cryptographically secure random number generator and be done with it.
Key management is actually a quite critical topic, which is overlooked quite often. E.g. depending on your application, it can be bad if the keys are stored in the same location as user data, etc. If you can utilize a crypto library to handle the key management, all the better.


Your own suggestion of replacing MD5 and SHA-1 with a truncated version of SHA-256 would be a step in the correction direction (better without truncation), as it would be a lie to call anything based on MD5 and SHA-1 highly secure. However, this is just a minor step, and your actual problems go way deeper than that.