How to delete downloaded files on Android TV
Although there seemed to be enough space (0.9GB) to install a small file manager app (about 30MB), I had to:
- clear caches from several apps
- deleting stored data from several apps
- uninstall some default app updates
After that I got about 1.6GB of free space, and just then it allowed me to install a File manager, where I was able to browse to the Download folder and remove unnecessary data.

I don't think this is a real solution, there must be another way, but this was the way that worked for me.