What kind of Super Saiyan transformation did Trunks use when fighting Zamasu?
The Transformation is Super Trunks and is only done by Trunks right now. In short it looks like a half super saiyan god transformation where his body is clad with god ki but only halfway which is why in this transformation he has the "two-tone" aura similar to a kaioken ssb. As shown below 



It is very similar to pseudo-super saiyan as described in wiki here wiki link

Here is the link to read about Super Trunks Super Trunks. 

Right now there isn't any real evidence of it being a god-level transformation. But it appears to be powered by the hope of humanity. 

As quoted by the wiki:  


  After gathering the power of light and hope from mankind, Trunks gained a new type of power up in this form, turning his aura completely blue, his body producing blue energy around him.