Python: advice on which machine learning algorithm to use for a problem which involves lots of randomness I'm new to machine learning so I'll summarize my problem with two examples without getting technical (because I can't).
The dog vs. cat classification example is solvable, in the sense that a human can tell you whether its a dog or a cat with certainty. Many machine learning algorithms are able to replicate the human performance and identify dog or cat with near certainty.
For my problem, there is no certainty, only a slight-better-than-random prediction. I am trying to predict whether a person who was recently released from incarceration will commit a crime within the next year. Lets assume the actual chances of re-offending are about 50/50. If I could use machine learning to make a modestly-better-than-random prediction, that would be a huge win for me. More specifically, if 50/50 is a random guess, then if I could achieve 55% to 60% success rate, that would be considered wildly successful.
I know this task is possible since I have used a dataset (with around 50 features and 100,000 observations) to make a &quot;man-made&quot; linear regression which achieves around 52% out of sample.
I have tried SKLearn's logistic regression and XGBoost but their performance has been lower than my &quot;man made&quot; attempt. I'm assuming that is because these algorithms aren't meant to deal with a prediction of an event that is mostly random.
Given that I am dealing with the prediction of an event which is mostly random and I am only looking to achieve slightly-better-than-random predictions, is there a machine learning algorithm / strategy you could recommend to best tackle this problem?
It sounds like you have good data - 50 columns and 100,00 rows!
I would do exploratory data analysis (EDA) and look for variables (columns) that are correlated with the response variable (re-offending) but NOT correlated with each other. If you can find a handful (~10) of these then you can build an excellent regression model.
Other techniques to try could include random forests and cluster analysis. Both can be done really quickly in Python so you can compare lots of different hyperparameter iterations.