Disable (or workaround) form rebuild with managed_file field
This was happening due to the Clientside Validation module being enabled.

For anyone having this issue, I had to override the form settings in Clientside Validation for this particular form, setting "Validate fields on submit" to "No".