How could an 8-bit encoding take more memory than a 10-bit encoding?
Memory size

This is certainly not true for any encoding as reducing 10 bits to 8 bits is by definition reducing the size.

However, it is possible that some codecs adds a segment for alpha channel turning 24-bits (3 bytes (RGB) per pixel) into 32-bits (3 bytes for color RGB + 1 byte for alpha channel).

This way it may appear as if the video takes more memory, but it's the added alpha channel that adds to the data. In fact it would take the same space as the original 10-bits data (assuming there is only RGB) as it would use 32-bits to hold the 30 bits it represent. This is because in memory data needs to be aligned to a full 4/8 bytes segment to work efficiently so 30-bits data would take 32-bits (4x4 bytes) memory (this will become very technical and programming oriented so I'll leave it with that).

File size

The reduction of one number of bits to another can be performed in various ways such as shifting for example the 10 bits 2 bit places into 8 bits effectively cutting the two lower bits. This is the quick and dirty way of doing it leaving in general poor result but not affecting compression so much.

The other way is to do rounding and introducing noise to the data, what is called dithering. The quality is visually much better with this technique as a reduced representation, but it has one draw back: This noise will remove coherence in the signal and can increase the file size (but not memory size which would be the size of the raw image - the software will have to use uncompressed data for processing) as it's harder to compress as compression always bases it self on coherency. Noise is basically random numbers without much structure (I'm not gonna go into chaos theory and such :-) ).

For this reason the file size may become larger.