x265 encoding test: 12bit results a bigger file size than 8bit of the same config?
crf=18 rate-control produces bigger files in 12-bit mode.  If you're trying to find some encode settings that work well for some content, you shouldn't assume that the same CRF at different bit depths is going to be the same.  (Either the same SSIM or the same perceptual visual quality).

You haven't said anything about quality, or quality-per-filesize.  Presumably the bigger files also look a bit better, since 12-bit x265 has about the same quality per bitrate as 8-bit x265.  (I've tested with 10 vs. 8, but not with 12).

CRF is not an exact target-quality setting.  So if you want to compare settings, use 2pass with the same bitrate for both and look at quality.  Use either SSIM or preferably human visual inspection.  (Some people like to pause/zoom, but don't just do that.  Some kinds of artifacts / distortion are noticeable when the video is playing, but many are a lot less noticeable.  Pause/zoom helps when checking your visual impression of "sharpness" / "more detail", so figure out what was giving you that impression, and maybe also other things to look for to see if you can still notice them while playing the video.)

For actual encodes once you find settings that you like, CRF is great, but it's no good for comparing the quality-per-bitrate of different encode settings.



8-bit vs. 10/12-bit

There may still be some gains in compression efficiency for x265, but they're definitely smaller if they exist at all.  10 or 12-bit might even look a tiny bit worse.  See discussion on doom9, linked by Michael in a comment.  I haven't followed the latest discussions, so I'm not sure what the current consensus is on 10-bit x265 for 8-bit video.  Even if there is a small gain, it may not be worth the speed penalty.

It's definitely nowhere near a factor of 12/8 like some commenters are suggesting based on "simple math".  A lossy video codec like h.265 isn't very similar to simple lossless compression like ZIP.

x264 does benefit in general from running in 10-bit mode even when the original source is 8-bit, and so is the final display (but again, you shouldn't expect the same CRF to give the same bitrate or the same quality at different depths).  8-bit h.265 has higher-precision motion vectors than 8-bit h.264, so that part of the reasoning doesn't apply to x265.

Remember that both h.264 and h.265 store the information in the video as quantized frequency-domain coefficients.  With trellis / rdoq, it even tweaks the quantization to compress well with the final entropy-coder (e.g. CABAC in h.264), so the same number of bits can represent the same amount of information, whether the entropy-coder had 8-bit or 12-bit input.  8-bit is in some ways just a speed-hack.

More bits means that you can get closer (smaller error) while still having some error.  So the encoder has more choice when trading off distortion vs. bitrate.  This may be partly why 10-bit x264 suffers less from "banding" artifacts in gradients: it has more choice in representing the DC coefficient, or in having small values in the AC coefficients.



The higher the input bit-depth (for a given output bitrate), the more bits the encoder has to throw away.  A lot of those extra bits are zeros if your original video was 8-bit.  (Lossy encoding already throws away a lot of bits, like from 12 bpp per frame (for YUV 4:2:0 chroma-sub-sampling with 8-bit components) down to 0.15 bits/pixel/frame and still look pretty close to visually transparent, especially at high resolutions where typical video has its information content distributed over more pixels.)

If you're going to rescale, presumably going to 10 or 12-bit before rescaling will let the rescale blend pixels with less loss of information.  (e.g. a 10-bit value can exactly store the average of four 8-bit values.)