How to debug ajax in form?
Eureka!

So, I still am not sure what the issue was 100% but I am guessing that there was a name collision with multiple different forms and since Drupal uses name "op" for every submit button since the dark ages of the internet I guess the ajax was trying to get itself applied to the other form that had submit button as well(named op as well) but that did not work of course so that resulted in this weird behavior.

So the way I have fixed this was by setting the #name AND #id attributes on the submit button in my form. I have set the #id which I normally don't do in ajax forms because I would get wrong drupalSettings.ajax entry.

I didn't do this in the first place because the form was working just fine for logged-in users + just fine for anonymous users on entity listings so it didn't made any sense but hey, the more you know.