So I'm working on a project that's turning out to be monstrously difficult. I'm trying to automate the rigging process inside of blender using the addon "Auto-rig pro". This addon has a feature called smart rig that automatically places a skeleton into a mesh. Using this I've been able to automate everything but one step.

In order for the rig to be created automatically you have to manually place markers on the neck, chin, shoulders, wrists, ankles, and groin. If you've ever used mixamo's autorigger it's basically just that. So in order to fully automate the rigging process I've resorted to using a neural net.

I've been doing a lot of research on machine learning, but unfortunately it appears that I'm the first to attempt something like this (as far as I know). After significant effort I've managed to install Tensorflow and Keras into blender, but I start running into problems when I try to figure out how to feed the data to the neural net.

Normally a neural net is fed raw images, but I need to feed it the 3d view. The marker placement is done in a straight on orthographic view, so I only have to worry about 2 dimensions here which is nice. I could render each model as an image and input it into the neural network, then have the neural net output the appropriate marker coordinates, but rendering takes time and this method seems somewhat unreliable.

Honestly I'm feeling pretty damn overwhelmed by this whole process, I'd never written anything in python until a few days ago, and I've never tried to make a neural net do anything more sophisticated than recognizing simple number patterns.

If you have any suggestions or ideas at all I'd love to hear them and I'm happy to answer any questions you have. Just feeling like anyone other than myself is considering this problem would mean a lot to me.

Also I apologize for any typos/grammatical errors I may have made. After working on this whole neural net dilemma for 8 hours straight my brain is pretty much completely fried.