Starting the VFD from a specified frequency instead of 0 Hz
By starting at a low frequency with a VFD, a motor can produce its full rated torque while drawing no more than its rated running current. It can supply up to about 150 percent of rated torque with proportional current. When a motor is started by switching on the power supply, the motor will supply little or no more starting torque while drawing several times more current. A VFD that is capable of supplying that much current would be much more expensive and provide no faster acceleration.

You should check the VFD to see if the current limit adjustment is set to the maximum setting. Also set the acceleration rate to the actual rate than you need rather than the fastest rate. That will allow the VFD to accelerate normally rather than under current limit control. That may or may not provide better performance depending on the VFD design.

The best acceleration performance can be obtained if the VFD is a sensorless vector model. It must be configured for vector operation rather than V/Hz operation. If it is a good design, that will allow it to accelerate as rapidly as possible under current limit control. Make sure that the VFD is properly tuned to match the motor. Make sure that the VFD is a model that is rated to supply at least 150% of rated current intermittently for acceleration.

If none of that is sufficient, you may need a larger motor and VFD.

Added re configuration settings

You should check the configuration settings carefully to make sure there is not something like initial DC braking or alternate acceleration settings that are causing a delay.