Converting a 9.6V NiCad Drill to Li-on
Do you have the C rating and the capacity of the Li-Po battery this will give you how much the current the battery is able to deliver continuously. 

The buck regulator will cause the current draw to be less on the high side so if it is the over current protection that is stopping you this might be the reason.See this page for introduction

You might overcome this by limiting the inrush current by a NPC thermistor or connecting a load in series and then disconnect it when the coils in the motor has been charged.

Also remember that the motor is rated for a given current at a given voltage. If you connect it to the Li-Po at a higher voltage the current will be greater and the potential to burn out the motor is present it definitely will run faster and hotter. 

The best solution would probably be to run with a buck-converter. The protective circuit would hopefully cut out the power when your battery reaches 11.1 V but some circuitry to detect and cut off the battery should be implemented as well as to not damage the battery pack.