How can I level a transition in a floor before gluing a vinyl tile?
The simple answer is you don't transition here.  Any type of transition you try to put on that edge will be just as awkward or even more awkward than the already bad transition.  

I would try to get some thin plywood and lay it over existing vinyl and try to match the height that way.  You will need a metal L to protect the edge of the vinyl next to the tile.

(Actually I would never do this.  I would lay tile next to existing tile or take tile out and vinyl the whole floor.  I think you are causing yourself more work by trying to do the job quicker.)