How do I set the bone length after setting a new bone matrix?
I am looking for a similar answer, but I may be able to help.

The "forward" (or "backward" if you see moving from head to tail as moving backward) vector direction is +Z (0,0,1), try setting head to (0,0,0), tail to (0,0,1) and then set length and matrix in that specific order.

The problem I am having is that the custom matrix I set, does not remain the same:

IOW: bone.matrix = custom.matrix, bone.matrix ~= custom.matrix BUT bone.matrix != custom.matrix

my current guess is that my custom matrix coming from a left hand coordinate system is something Blender "fixes" behind the scenes to an equivalent matrix, this however makes it hard to import a model and export to the exact same file that was imported on the first place.