Disable deforming with armature
I've found the solution! If you go into Weight paint for the disformed object, you can select individual bones. In my case, I selected the left upper arm bone and then painted the cube blue, so that a rotation of the arm bone doesn't affect the other cube