How to train a model to predict if 2 samples refer to the same thing? I have 2 ddbb with around 60.000 samples each. Both have the same features (same column names) that represent particular things with text or categories (turned into numbers). Each sample in a ddbb is assumed to refer to a different particular thing. But there are some objects that are represented in both ddbb, yet with somewhat different values in the same-name column (like different open descriptions, or classified as another category).
The aim is to train a ML model that recognizes when the two “descriptions” refer to the same thing.
We have manually recognized a few thousand &quot;duplicated&quot; cases that we have labelled correspondingly. This label is the target variable to learn. Yet, the supervised classification examples I've seen, work with a single data frame and/or predict something about a single row of features.
How does it work when we are not trying to predict what a sample represents, but the relation between 2 samples (specifically now if they refer to the same object or not)?
I don't even know how to feed 2 dataframes to sk-learn or even auto-sklearn (or similars) so that it can handle the task of recognizing if a sample from one represents the same thing than another sample in the other one... Maybe I should concat them so that there's just 1 df, but then it would have to compare all with all the rests... Or doesn't make it a lot of difference?
Any idea or hint about how to proceed here or how to frame the problem better?
If I understand correctly, it sounds somewhat like an NLP problem. For each row, and each column, you will compute the similarity between entries across dataframes e.g.
Your two dataframes:
Row182, ColumnA, DataFrame1 = XYZ_ABC_123
Row182, ColumnA, DataFrame2 = XYZ_ABC_152

New data frame:
Row182, ColumnA = 0.8

Repeat for all rows and columns. You would then set some threshold at an acceptable similarity rate / accuracy of your similarity metric matrix. To highlight the most severe cross-overs.
You could have a look at https://www.nltk.org/, but also depending on the data entries you could just do a pure pythonic solution e.g. 80% match between string entries.
I'm guessing at what your data looks like, so some examples there could help