Ceiling leaking from upstairs bathroom ONLY when water supply is turned OFF
Maybe someone hit a drain pipe when drilling a wall anchor? That it only happens with the water off is odd, I'd just focus on finding the leak. The water is coming from somewhere.

I'd start by opening up any access panels you have to the upstairs shower. You're looking for blackened wood where the pipe enters the floor in the wall.



You're probably going to have to replace the drywall around the ceiling spot, you might as well open that up too. Try to drain the bubble by poking it in case it still has water in it. find the vertical source of the water that way- though occasionally water travels horizontally.

Follow the water. You make enough holes in the wall and you'll find the source.