Unable to remotely access lightning-charge
After a lot of research I found the solution to my problem. 

I didn't know this, but in nodejs there's a huge difference if your server runs on 127.0.0.1 and on 0.0.0.0. 

It isn't said in the lightning-charge docs, but except the --port attribute you also have --host which you can set.

Starting lightning-charge with this line charged --api-token mySecretToken --host 0.0.0.0 fixed all of my problems. Gosh was that an unpleasant journey!