Add execute button to Commerce POS modal?
Instead of a regular "page" callback, implement the page as a form with an action (submit) button. When the form is submitted and validates, the modal will be released courtesy of the way ctools_modal_form_wrapper() works.

POS_Command was not the right way to think about the problem in this case. POS_Button_Command is better suited to a simple, pre-defined operation as in setting the order user from a list of users. I was trying to do a more complex operation in the modal, gather some data from form inputs, then run some commerce commands on the order.

Short answer: The POS_Button_Modal_Page class works for loading a custom modal page. Implement the page as a form, and do the execution code in the form_submit handler instead of using POS_Command.