Street address clustering?
My instinct would say to start with a standard dataset and map all addresses ‒ both from your first and second datasets ‒ to that. In the standard dataset, each physical location would have a unique address.

The obvious source for the standard dataset is OpenStreetMap. You can query and dowload data from it using the Overpass API. Its query language, just like the whole OSM project, is well documented in my experience. Overpass Turbo is the user-friendly way with a graphical user interface to get started with the querying. Once you have developed a query, you can download data using e.g. wget, you just have to convert the query into the right URL at the Overpass API Convert Form by entering it into the window Overpass API Convert Form and selecting compact Overpass QL.


  wget --timeout=0 http://overpass-api.de/api/interpreter?data=[your query in compact Overpass QL form] -O filename


In OSM, some addresses are associated with buildings, which are ways (closed polygons), and some are nodes inside the buildings' outlines, so that's a bit of complication.

This all sounds doable. However, the OSM is not complete, so it very much depends on your area of interest whether it's worthwhile to use OpenStreetMap.