How to check rapidly if an element is present in a large set of data
If you knew that every article had a DOI, you could just store a hashtable of the DOIs of the papers that you've already harvested.  Of course, in practice, many papers don't have a DOI or don't list their DOI.

If you knew that the paper would be listed with the correct title and authors, you could use a hash of the titles and authors and store that in a hashtable.  Of course, in practice it is common for titles to be mis-spelled, or for there to be variations in how they are listed (e.g., changes in capitalization; do you list the full first name of each author or just their first initial; and so on and so on).

If you knew the list of all variations, you could try to canonicalize the title and authors (e.g., lowercase the title, use only the last names of the authors, and so on).  Of course, in practice we probably can't know all the variations.

If you figured that all variations would be at small edit distance, you could use fuzzy matching and some kind of database that allows approximate match lookups.  (e.g., Efficient map data structure supporting approximate lookup, How fast can we identifiy almost-duplicates in a list of strings?, How to compare/cluster millions of strings?)  However, I suspect that in practice edit distance might not be enough to find all matches.  Also, there is a risk that you might end up conflating two different papers (for instance, I have seen a series of papers with titles like "Catching the Bugblatter, Part I" and "Catching the Bugblatter, Part II" with the same authors).

Hopefully all of this is conveying that in practice I suspect the problem is messy and there is no single clean answer.  Perhaps a pragmatic solution is to use some of the above techniques to find obvious matches, with the idea that this will take care of most cases of matches.  For the remainder, just accept that you might store the article multiple times, but hopefully this won't be too common.  Ultimately, storage space is extremely cheap, so storing the same article a few times probably isn't so bad.