Random and brief fan spin up
First of all, we need to see if something is preventing your Mac from sleeping.


Open Terminal.app
Type in pmset -g assertions


What you are looking for is these lines:


  PreventUserIdleDisplaySleep
     PreventSystemSleep
     PreventUserIdleSystemSleep
     InternalPreventDisplaySleep 


and 


  Listed by owning process:
    pid xx(nameofdaemon): [0x000000090000012c] 01:45:46 BlahBlah named: "com.randomnam" 


Copy and paste all of that into your question.

Next, try resetting the SMC (System Management Controller)

Taken from Apple Support


Shut down the computer.
Unplug the computer's power cord.
Wait fifteen seconds.
Attach the computer's power cord.
Wait five seconds, then press the power button to turn on the computer.


Finally, check the Console Logs


Open Console.app
Click in the search box and type in sleep
Find the entries that match the time your Mac was sleeping
Copy and paste it in your answer


The first and third fixes weren't really fixes. I just hope that the data you paste will help find the cause.