Weird Mesh Behavior Moving Bone
As weird as it looks, that is behaving exactly like it should. To get joints to deform cleanly relies on a couple of things.


Mesh Topology: A mesh meant to be deformed should be built so that it deforms cleanly. It's hard to tell from you image, but I'll assume the mesh is good. When I say the mesh is good, by that I mean that the faces flow in a way such that when the joint is bent, the faces still flow in a logical way.
Joint Placement: This is one of the things that I can tell is causing you trouble. The placement of the joints is important for a cleanly deforming mesh. The best way to find out where a joint should be placed is to look at your own joints if you are going for a realistically proportioned character. If you are going for a stylized character, or even a realistic one, the easiest way to figure out where the joint should be is to look at the joint when it is straight and imagine what it would look like when it's bent. Then you can infer where the joint should be placed based on the position that would result in the rotation you desire.
Vertex Weights: Even with great topology and properly placed joints, you will still need to make sure each bone is controlling the sections of the mesh that you want it to. Usually for joints, good deformation can be achieved bu simply making sure there is a bit of a smooth transition between areas with full weight and areas with none. The default weighting is usually OK.


I would highly recommend watching some tutorials on rigging to see where joints should be placed to get good deformation. Ultimately it just take some practice so that you can understand what causes a mesh to deform and how you can achieve the results you want.

As I mentioned before though, I would definitely try adjusting the joint position first.