Why perceptron does not converge on data not linearly separable
The behavior appears to actually depend on the learning rate $\eta$; a smaller $\eta$ affects which points are misclassified in the next iteration, which affects the weight update more than just by the simple scaling you alluded to.
With appropriately small learning rates though, it seems you are guaranteed convergence to some local minimum, if you avoid certain degenerate situations that would give e.g. a constant loss surface.  Exactly what you describe happens at these minima: the loss to the misclassified points of either class equal each other.
I put together a short demonstration in this colab notebook (github link).  Below are some animations of the evolution of the decision line during the gradient descent, starting at the top with a large learning rate and decreasing it from there.  In this situation, there should be two local minima, the two &quot;tangents&quot; to the &quot;blobs&quot; (possibly pulled in a little, misclassifying a couple of extra points if it can decrease the loss on the central points a little more than the cost for the newly misclassified points).