Threadless stem keeps loosening
Possible causes:


Deformation of the steerer around the clamp area, possibly from overtorquing.
The steerer is cracking and hence lacks dimensional stability. I've seen this cause the exact issue you're talking about and it's scary, so I recommend looking very closely for it. Take off any spacers below the stem to get a better look when doing so.
Using a plug and topcap assembly that doesn't provide as much internal support as the fork is designed to need. Sometimes plug assemblies are treated as totally interchangeable and in practice this isn't really the case. If this is what's going on it can easily have already caused cracking. If it still looks good and the plug assembly isn't snug inside where the stem clamps, try getting one that is.
Plain old poor fit/tolerance between the stem and steerer. This is pretty unlikely. 
Using a single-bolt stem with the carbon steerer. This is a big no-no and could have caused the sort of deformation mentioned above.


Bear in mind that 5Nm, with or without carbon prep in most cases, really is plenty when things are working right (given common lightweight stems that call for 5, of course). It sounds like something else has gone wrong and I doubt that more torque is going to be the fix.