Library for driving P10 LED Display with ESP32
You can use Arduino libraries with ESP-IDF fairly easily by including arduino-esp32 as a component of ESP-IDF. After all, arduino-esp32 is based on ESP-IDF. 

If you really, really don’t want to touch anything remotely to do with Arduino then I would suggest you peruse the code of one of the libraries for P10 (such as this) and extract the Arduino independent parts.