I need to train a CNN (YOLO to be precise) on RGB + depth information. I have the RGB input as a separate PNG file and the depth map as a separate PNG file. These PNG files are 8 bit depth, 3 channel files. I need to create a single PNG file of 16 bit depth, 3 channels which is essentially a combination of both the original files. What is the right way to combine / encode RGB + depth information? Data 1: R, G, B channels of 8 bits eachData 2: D1, D2, D3 channels of 8 bits eachRequired: C1, C2, C3 channels of 16 bits eachCurrent idea I'm currently using a bitwise shifting.C1 = R << 8 | D1C2 = G << 8 | D2C3 = B << 8 | D3(shift by 8 bits left and then OR)QuestionsNotice that the channels left shifted (R, G and B) obtain higher significant bits and dominate over D1, D2, and D3 while forming C1, C2, C3. The CNN loses the information present in the depth map. Is there a better encoding?A small note about the depth map. The depth information requires 16bits and is contained in D2 and D3. D1 channel is a pseudo channelwith only gray scale information of the image. The final encodingshould focus on D2 and D3 more.Another idea To make sure no channel dominates over the other, I was thinking of interleaving the two 8 bit sequences to form a 16 bit sequence. If channel R has b1, b2, ..., b8 bit sequence and channel D1 has bb1, bb2, ..., bb8 bit sequence then the combined 16 bit C1 channel will have b1, bb1, b2, bb2, ..., b8, bb8 bit sequence. Any thoughts on this?
RGB + Depth Encoding for CNNs
neural network;classification;deep learning;feature selection;image recognition