How do I remove the wheels from my bike (not a typical nut)?
That's a pinhead security fastener. You should have got a key to match with the bike. Without the matching key number you can't get a replacement key either. 

Here's a picture of a pinhead key fom about 2010. Yours will look at bit different. The key number is under the red blob, and the pins to mate with the nut are in the silver bit on the right.  As you can see, mine is old and battered, so the original keyring attachment failed and I drilled a hole in it.

There's another format that looks like a hex nut on one end as well.