How can I protect a disabled form from being submitted?
Found the issue! It seems to be a Drupal bug.

#access set in #after_build does not completely override the #access settings, set in hook_form_alter.

In theory, a form is altered by hook_form_alter and hook_form_alter is altered by #after-build. When I try setting #access in hook_form_alter, to alter the form, everything works. However, when I try setting #access permissions in #after_build, to alter hook_form_alter, it only hides the form, still allowing anyone to submit it, by altering the code on the client side.

As a temporary solution, I disabled URL alias permissions, by unchecking "Administer URL aliases" and "Create and edit URL aliases".