Body part recognition inside of blender using a neural net So I'm working on a project that's turning out to be monstrously difficult. I'm trying to automate the rigging process inside of blender using the addon "Auto-rig pro". This addon has a feature called smart rig that automatically places a skeleton into a mesh. Using this I've been able to automate everything but one step.

In order for the rig to be created automatically you have to manually place markers on the neck, chin, shoulders, wrists, ankles, and groin. If you've ever used mixamo's autorigger it's basically just that. So in order to fully automate the rigging process I've resorted to using a neural net.

I've been doing a lot of research on machine learning, but unfortunately it appears that I'm the first to attempt something like this (as far as I know). After significant effort I've managed to install Tensorflow and Keras into blender, but I start running into problems when I try to figure out how to feed the data to the neural net.

Normally a neural net is fed raw images, but I need to feed it the 3d view. The marker placement is done in a straight on orthographic view, so I only have to worry about 2 dimensions here which is nice. I could render each model as an image and input it into the neural network, then have the neural net output the appropriate marker coordinates, but rendering takes time and this method seems somewhat unreliable.

Honestly I'm feeling pretty damn overwhelmed by this whole process, I'd never written anything in python until a few days ago, and I've never tried to make a neural net do anything more sophisticated than recognizing simple number patterns.

If you have any suggestions or ideas at all I'd love to hear them and I'm happy to answer any questions you have. Just feeling like anyone other than myself is considering this problem would mean a lot to me.

Also I apologize for any typos/grammatical errors I may have made. After working on this whole neural net dilemma for 8 hours straight my brain is pretty much completely fried.
Consider this to your approach:


feeding your net just a 2d image is limiting (and more work to set it up) when you could feed it whole 3d mesh. You could even build an octree and know what is where in space and feed this to the neural net, basically voxel data instead of pixels
probably the whole neural net thing is not even necessary. There is a popular method of extracting skeletons from meshes by Laplacian smoothing the mesh: Skeleton Extraction by Mesh Contraction:


  
  Our method extracts a 1D skeletal shape by performing geometry contraction using constrained Laplacian smoothing.


You will find this in open source C++ CGAL library here. This is probably enough to know where the neck and crotch is and where limbs end.
other method is: Mean Curvature Skeletons:


  


Code here.