iOS beta app got leaked, can I get in trouble with Apple?
If it is your app, you should probably notify Apple. If you are testing for someone else, notify the developer and Apple. It really isn't your fault. To your second question ("Can Apple see the the app got installed on more than the usual 100 devices), I'd say no. Because jailbroken users use methods such as iFunBox or vShare/AppCake, Apple has no way of keeping track of how many times the app was downloaded unless they used the Apple way. Finally, I'd try to find out how your/the app was leaked.