What encryption/decryption system should I use for media server There is a server from which media files will be streamed to cross platform mobile/desktop apps.

The idea is to encrypt the files while uploading using the web browser, store it in the server and when requested, files will be sent over to the front end, decrypted there and media played.

Users can use multiple devices (Android, iOS, Webapp, PC, Mac, Linux) with their account, but at any particular time only one device can play the media. If users, hypothetically, get access to the encrypted files from the server or while downloading to the app (man in the middle), should not be able to decrypt the files. Only the app should decrypt the files under the hood.

I am not sure how or what system of encryption to implement since I barely know cryptography.

A good part of this question is answered here, but it really doesn't discuss the part of users not able to decrypt the files using other tools.
"but it really doesn't discuss the part of users not able to decrypt the files using other tools." ... yes it does, because it discusses key management. As by Kerckhoffs principle, you should not try and secure the algorithm itself, security relies on the keys. Tools that receive the correct keys should be able to decrypt the media.

You can of course embed a secret or private key in the app, and require both the keys from the discussed scheme and the app key. It would however just take a single reverse engineering of the app to nullify the additional protection.

The practical way on how keys can be protected on the specific devices (e.g. using device specific key stores) is off topic for this site.