Remove big valley in phase plot
A good part of Engineering Design is trying not to reinvent the wheel and learn from other’s past mistakes and make improvements.  There are many non-linear factors not included in your simulation such as hysteresis from friction and slack in every moving part and a lack of design specs for inputs and output error.  A single step response to voltage is not the best test as motor BEMF changes the torque.

use Current control and feedback for smooth acceleration/braking rather than step voltage.

To accelerate an inverted arm, you must go backwards first to tilt the arm forward then accelerate enough at peak velocity to tilt the arm backwards in preparation for braking towards target position.

research the conclusions of a dozen similar yet different inverted broom or pendulum thesis papers and cite the flaws and solutions you propose with multiple loops for ideal acceleration, velocity, tilt angle and horizontal position with errors and tolerances allowed.

Here is one example that met some criteria yet failed in the end to be robust. Read the conclusions of each paper first and follow my advice. The contents will give you some insights to theory but be incomplete.