How can I use a mask modifier with a musgrave texture?
The short answer is: sort of

This setup will make the parts of your mesh transparent where the musgrave texture is black. If you want to see the backside, or other objects through it, replace the holdout by transparent. 
However the edges of the mask won't look 3D, the ones in the clip seem to have some depth, this method won't allow that, but maybe it's enough to get started. With animation nodes you can assemble your own meshes, it won't be easy, but in my opinion that would be the proper way to go.