Mac Mini late 2012 second monitor resolution
According to this page on EveryMac.com the resolutions you want to run should be supported.

The problem might be the adaptor. 

I would try a PRAM reset (restart while holding down Command-Option-P-R), or possibly an SMC reset, detailed in Apple's Support system.

If these don't work, I'd try a different adaptor.

Good Luck.