Aggregated SUM from "Referencing entity" relationship, how to include nodes without relationship
I was debugging like crazy to figure this out and then after checking some values in views_handler_field_field I realized what was wrong, and for sure it weren't Views! I forgot, or rather though of it as irrelevant, that there was third content type - a comment, that had a field "field_contact". The problem occurred because Views when summing up, was also using existing "comment" nodes, but because they didn't have "field_payment_amount" at all - Views got confused. So the real answer is that we need to filter out any additional types that are using "field_contact" field in our relationship.

I guess Views could act a bit better on this, and not get confused but use only available fields from proper nodes to create a sum, but it isn't much problem as long as we can filter out useless (for SUM) types that are using our reference field.

This explains why I got random/wrong results without type filtering on relationship, but when I added filter for "payment" - I didn't get "0" results, because then view assumed that relationship is required. The proper solution is then EXCLUDING wrong types that are in our relationship.