Converting a Keras model to PyTorch
I'm sorry that this is not an exact solution to your problem but a work around I did, since I also tried to convert the models, but I didn't find any magic library to make it happen and I can't spend much time on this.

If you already trained the model you will also need to convert the weights.
Keras and TF weights are in hdf5 format, while pytorch weights are pickle, so you need to convert the weights and import the model.

Check out this post: How to convert my tensorflow model to pytorch model?. There you can see how you should add the weights one by one for each layer. It seems too much trouble.

Another option for the weights is this one (to be implemented while you train the model in keras): Pickling Keras Models ... but you will still need to convert the ANN architecture to Pytorch.

On the other hand, implementing models in keras is pretty easy and nice to get started with ANN architectures, so you could give it a go.

To sum up: I just found a keras library that implements the ANN backbone model that I needed, in order to train the model and make inference later, which is pretty straight forward. 

Please let me know if you find a better option.