nodes in a kubernetes cluster in azure unable to communicate with one another
With help from a Microsoft Engineer, we were able to resolve our issue.

The trick to it was that the node that could communicate with the outside world was always the one that the kubernetes tunnel pod was running on.  This indicated a problem with communication between nodes.  The traceroute indicated that this, with no network security group traffic restrictions, was bouncing between the node1 ip, to the load balancer ip, back to the node2 ip.  

we had a security rule that restricted all traffic that was not on our specifically approved list.

One of the debug steps I had taken was to open all traffic between the load balancer ip to the load balancer ip.  However, this failed to consider the fact that the traffic was not from the load balancer to the load balancer, but from nodex to the load balancer then from the load balancer to nodey.  

We added an inbound and outbound traffic rule that allowed traffic from the subnet ips that nodes can possibly be on to the load balancer, as well as within the same subnet for good meansure.  This resolved the issue.

If that's confusing, I can provide images that show the new rules and explain it a little clearer, but I'm hoping that is clear enough for anyone who experiences a similar issue.