Python package to assess text coherence
This task looks similar to what is called text segmentation, in particular topic segmentation. I don't know any python package to do it but apparently Google gives a good few results for "semantic text segmentation python" (I'm not sure that this is the best phrase, you might want to try variations).

Note: this is still an active NLP research topic as far as I know. I don't know how fast reliable python packages are written and maybe there are some for this, but I wouldn't be too surprised if there were only research prototypes available at this stage.