How did they estimate the mass of Sgr A*'s companion G2 without knowing what it was?
To put it simply, you can't. 

Both cases are explained in these two papers:

Binary: https://arxiv.org/pdf/1410.1884.pdf
Just a Cloud: https://arxiv.org/pdf/1112.3264.pdf

In both cases you have to assume some kind of model to account for the detected fluxes in different near-infrared bands. 


In the "Just-a-cloud" case you would need a cloud with 3 solar masses and a temperature of about 550 K.   
In the "binary-merger" case you would need a 2 solar mass main sequence star covered in gas and dust (possibly generated by a recent merger with a companion, which, however, would somewhat distort the 2 solar mass estimate).