I can remove crank arms without a crank puller, is this a problem? I'm worried the crankarms might be loose
If you are able to seat the crank arm onto the axle such that it hits the green area you marked (i.e., it will slide all the way down the taper), then the crank arm is "worn out." The taper on the crank has been enlarged by wear or over tightening to the point where it can no longer form a solid connection to the axle. The "right" fix would be a new crank arm, but it seems like you could try the ideas suggested in the comments above. Do check the condition described by @mikes, if the axle is sticking out beyond the crank arm then grinding will not work, but then you are also likely to have quite a bit of room for a shim :-).

If you're game to experiment, my biggest concern would be that there might be cracks starting to develop in the corners of the crank arm socket from the stresses of the arm trying to rotate on the axle. Eventually those could lead to the crank arm failing at the socket.