How to raise a form error in \CRM_Core_Form::postProcess()
You are correct that postProcess runs too late. Consider the conventional request lifecycle for HTML_QuickForm/CRM_Core_Form:



At the end of the Validate phase, it makes a major decision about whether to rerender the form with errors. However, the postProcess function is part of the Process phase which should only run on valid forms.

The conventional way to set a validation error in HTML_QuickForm is to register a validation function. For example, in CRM_Group_Form_Edit::buildQuickForm(), note how it calls addFormRule(...) (and how the static function formRule() conditionally returns an array of errors). This runs at the correct time.

For mitigations, you might consider:


Add some checks in the validation phase to predict whether the error condition will arise. (Ex: If the issue is a missing dependency or inadequate RAM, then validate the dependency or RAM beforehand.)
Report the error via CRM_Core_Error::fatal(), throw new Exception(), or CRM_Core_Error::statusBounce().
Don't rely on HTML_QuickForm as the main form processor. Implement client-side logic with Javascript and AJAX callbacks.
(Evil but possibly working) Do the heavy lifting during the validate phase. (This will, e.g., make it hard for third-parties to hook into the form... but it may work...)
(Evil but possibly working) Dig into the mechanics of the CRM_Core_Form lifecycle and figure out a way to short-circuit/cross-wire the phases for your controller.