AC brushless motor as generator
You can not use an ESC. The best thing to use is a charge controller that rectifies the output of the generator and increases or decreases the DC output voltage to the level that is best for the battery. It might be possible to just connect a rectifier between the generator and battery, but generator voltage would need to be fairly close to the voltage required charge the battery. A transformer could be used to convert it to the proper range. The speed of the generator would then need to be controlled to some degree in order to avoid charging the battery with excessive current. If a three-phase motor is being used as a generator, a three-phase rectifier is required. If it is not a three-phase motor, some more research is required.
Note that I have assumed that by &quot;brushless AC motor&quot; you mean a permanent magnet motor that might be called a brushless DC motor or a permanent-magnet, synchronous motor. Induction motors are also brushless except for the rarely-seen, wound-rotor slipring-motor. It is best to state exactly what type of motor you intend to use.
With a permanent-magnet, synchronous motor, connected to an un-controlled rectifier, DC power is easily generated simply by turning the motor shaft. The generated voltage is unregulated and determined by the speed at which the generator is driven. However it should be relatively easy to get enough regulated power from that to control a voltage boosting charger.
With an ESC of any kind, considerable modification of the design would be required to convert it to a battery charge converter.
The only answer to a question of this type that is reasonable to provide here is to outline a general concept that is a reasonable approach to the problem. In order to proceed, you must first completely define the characteristics of the generator and the prime mover including the expected operating speed range and controllability of the prime mover. That dictates the requirements of the generator power processing system.
The second step is to define the power processing approach and determine the parameters of its basic elements.