Getting 120v single phase & 240v three phase out of 240v single phase receptical
I do not see where you will have any issue with this arrangement - I am assuming your VFD has a means of control for motor speed ; whether that be a pot on the front, a PID loop control sensor or a PLC connected to it.
Not really something in Home Improvement as most homeowners don't have a VFD .

Since you are not going to use them at the same time - I think your setup will work with out issue. 

You might want to look at the VFD specific requirements on grounding and neutrals.. depending on the VFD you might need to do some things different regarding your ground and neutral connections, also if using a PID loop control you might have some specific instructions..

I assume you know this info because you are using a VFD - but maybe not ..if you don't what make and model VFD ?? Yaskawa , Rockwell other ??