How to select 'cutoff' of classifier probability
What you're looking for is something along the line of an ROC curve:
Using the threshold as a decision parameter, you can observe the trade-off between FPR (False Positive Rate: how many of the articles not belonging to the author will be correctly classified) and TPR (True Positive Rate, aka recall: how many of the articles which are really by the author will be classified as such).
When the parameter is at one end, you'll classify all documents as belonging to the author (100% recall, but pretty bad precision), and at the other hand, you'll have 100% precision but pretty bad recall.
The plot will allow you to decide on a value that satisfies your requirements (i.e. how much will your precision suffer when you want 95% recall). You can select it based on your desired value in one metric (e.g. 95% recall), but really I'd just plot it and have a look. You can do it in SKLearn with plot_roc_curve.