Reencoding H.264 to H.264 made it 40% larger
You are using a quality based variable bitrate compression factor on something that already has artifacts.  That means it is going to attempt to preserve the video exactly as it is, artifacts included and artifacts tend to not encode well.  This means you have a video input which is likely harder to encode than the original and you chose a quality based compression that is also likely higher quality than your original 12 Mbps could manage.  This means an overall higher data rate must be used to try to avoid additional loss from the second generation of encoding, even with the extra optimizations that a slower compression cycle can handle, you aren't able to overcome the added complexity of the first encoding and the quality setting being used.

If this was any kind of HD stream that's not really that surprising as 12Mbps is a very heavily compressed end consumer level of compression for any HD formats.  It's not really a big surprise that it doesn't re-encode well.