AC vs. DC in Low Speed, High Torque application
What you have is a permanent-magnet synchronous motor. That is the same thing as a brushless DC motor. The motor may be too large for your controller or the controller may not be adjusted properly for the motor. If the motor and controller is suitable and adjusted properly, the problem could be that the engine-generator set is being loaded too heavily. Check to see if the DC voltage drops when the load is high.

The motor-controller combination that you have should perform as well as a brushed DC motor and controller or an induction motor and controller. One technology is not necessarily better than the others, but some models and designs of each may be better than the others. With any of them, the weakest link is the one that will be the limiting factor. A good design, with properly matched motor, controller and power source is essential. When the limit is reached, the controller should limit the current, not fault out.

Re comments and further consideration:

With the optimum design, a permanent-magnet synchronous motor (PMSM) should be capable of providing the most torque per amp of any motor technology. However, all designs are compromises involving various measures of performance and manufacturing cost. Selecting the best motor is a mater of evaluating the specifications of motors under consideration.

The performance of the motor must be evaluated based on a careful analysis of the requirements of the application. Tractor pulling is a contest in which a load is pulled as far as possible. As the load is pulled, the required force is increased by shifting the load weight from a wheel and axle to a skid thus increasing the friction between the load carrier and the ground. It should be defined whether the load increase is based on time or distance and whether or not there is a time limit.

It would probably be advantageous to operate the engine at maximum power for the entire pull. Since power is torque multiplied by speed, that would mean reducing speed and increasing torque during the pull. The most efficient means of doing that would likely be a continuously variable transmission (CVT) such as a hydrostatic transmission. A manual gearshift would be just as efficient, but probably not quite as effective. However the gearshift or both the gearshift and the CVT may be prohibited by contest rules.

A series connected, brushed DC motor inherently slows down and increases torque as its load increases. That characteristic may make it preferable to more efficient brushless types of motors for this application.

An induction motor can be controlled to operate with constant torque over the low end of its speed range and constant power with declining torque at higher speeds. With a current limiting controller, that would provide performance somewhat similar to the performance of a series DC motor.

The torque capability of a PMDC motor will be pretty much constant over its entire speed range with no capability to trade speed for torque. The torque capability is limited by the current available. In this application, the motor must decelerate to a stop pretty quickly when the current limit is reached. There may be a way to trade voltage for current in the supply and control systems. That could have the effect of trading speed for torque if the motor can operate at a higher current for the required time.

Before all else, it important to make sure that the selection of the fixed ratio of the drivetrain gearing provides as much torque as possible by reducing the speed as low as the application will tolerate.