Custom submit handler on form loaded via entity.form_builder
As per a comment from another user, the form is already processed after being returned by the form builder. That said, you can still alter values, but not add form elements or alter actions.

The solution for me was to do a form alter on the form before hand, and give it a submission handler, along with a hidden field. Then when I get the form in the code above, I now set the hidden field to contain a relevant value (Group ID). In my submission handler, I only proceed if that hidden value != NULL. This prevents me from doing anything on the core entity form which I do not want to affect.

Thanks for the hint 4k4!