Why doesn't rigid body physics work properly after I add the Build modifier?
I suggest to add an invisible wall to bypass the problem of the cube physics when the build modifier is attached to it. So the ball will collide and bounce on that wall.

This is a quick experiment where the cube acts the way you need (I don't know why it works):

https://filebin.net/zisptqbp3v67qm9c