A Proper Outlier Detection For the Attached Figure
One approach would be to fit a linear regression and look at Cooks Distance to detect outliers.
https://en.m.wikipedia.org/wiki/Cook%27s_distance

Some R background: http://r-statistics.co/Outlier-Treatment-With-R.html

Some Python discussion: https://stackoverflow.com/a/52322232/9524424