Use a USB Controller with Xoom without adapter
Can you not use a USB OTG (On The Go) Cable. It will have a connection (microusb) at one end to fit to your tablet and a USB connection at the other (for many peripherals). This should surely connect your X Box Controller to your tablet. They are very cheap as well.

Alternatively, and maybe to be used in conjunction with see the following article from Pocketables entitled "Guide to Game Controllers on Android" written by A. Odegard on May 9th 2012

http://www.pocketables.com/2012/05/guide-to-game-controllers-on-android.html

Its is an excellent read