"""De-duplicate the vimorg_id column in the plugins table."""

import re

import rethinkdb as r

import db.plugins
import db.util

r_conn = db.util.r_conn


if __name__ == '__main__':
    grouped_plugins = r.table('plugins').group('vimorg_id').run(r_conn())

    for vimorg_id, plugins in grouped_plugins.iteritems():
        if not vimorg_id:
            continue

        # We only need to concern ourselves with duplicated vim.org IDs
        if len(plugins) == 1:
            continue

        print '\nPlugin with vim.org ID %s occurs %s times' % (vimorg_id,
                len(plugins))

        to_delete = []
        to_keep = []

        for plugin in plugins:
            # Plugins scraped from github.com/vim-scripts that don't have
            # a vimorg_url must have not been successfully matched to the
            # corresponding vim.org plugin. These can be removed.
            if (plugin['github_vim_scripts_repo_name'] and not
                    plugin['vimorg_url']):
                print ('Delete %s because it is an unmatched '
                        'github.com/vim-scripts plugin' % (plugin['slug']))
                to_delete.append(plugin)
                continue

            # If no GitHub info is available for this plugin, it's probably
            # a duplicate vim.org plugin scraped after the original
            if (not plugin['github_owner'] and not
                    plugin['github_vim_scripts_repo_name']):
                print ('Delete %s because it is an extra vim.org plugin' %
                        plugin['slug'])
                to_delete.append(plugin)
                continue

            # Otherwise, we have an original plugin that should be preserved
            print 'Keep plugin %s' % plugin['slug']
            to_keep.append(plugin)

        # We expect to keep at least two plugins because plugins with the same
        # vim.org ID can only accumulate when there is already a duplicate
        # (these original duplicates arose due to mistakenly associating
        # vim.org IDs with GitHub repos whose homepages were vim.org URLs).
        assert len(to_keep) >= 2

        # Delete the newly-scraped extra plugins that accumulated due to
        # existing duplicates.
        for plugin in to_delete:
            r.table('plugins').get(plugin['slug']).delete().run(r_conn())

        # Out of the ones to keep, only one should get the vim.org ID. Pick the
        # one that has the most users.
        most_used = max(to_keep, key=lambda p: max(p['github_bundles'],
                p['github_vim_scripts_bundles']))
        for plugin in to_keep:
            if plugin['slug'] != most_used['slug']:
                r.table('plugins').get(plugin['slug']).update(
                        {'vimorg_id': ''}).run(r_conn())

        print 'Plugin %s gets to keep its vim.org ID' % most_used['slug']

Add script to fix plugins with vim.org ID duplicates
Add script to fix plugins with vim.org ID duplicates

Summary:
If you look at vimawesome.com/?p=3 right now, you will see many duplicated
"taglist.vim" plugins.

A long time ago, when we scraped a github repo whose homepage was a vim.org
plugin, we would assume that was the definitive github repo associated with
that vim.org plugin and set that repo's vim.org ID. But it turns out, of
course, that there are many different github repos that all set their homepage
to the same vim.org plugin. This led to multiple plugins with the same vim.org
ID in our DB, which violates an invariant assumed throughout the codebase that
vim.org ID is unique in the plugins table.

This was fixed the night before launching, but I forgot/didn't have time to update the data,
and so the data was in an inconsistent state.

With the invariant violated, scraping the same set of sources no longer
guaranteed idempotence, and so each new scrape would accumulate new duplicates.
Basically, when the scraper tries to match a source that has a vim.org ID and
it sees multiple existing plugins with the same vim.org ID, it doesn't know
which one to match to, so it inserts a new row instead of attempting a match.
This led to the proliferation of duplicates you see now.

So this script removes the extra accumulated plugins, and also fixes duplicate
vim.org IDs by only allowing the plugin with the most users to keep its vim.org
ID.

Test Plan:
Locally. Before fix, ran
`PYTHONPATH=. python tools/scrape/scrape.py -s vim.org`, and saw that a bunch of new plugins were inserted when they should
have been associated. Ran
`PYTHONPATH=. python tool/scrape/scrape.py -s github-vim-scripts`,
and saw new plugins were inserted when they should've been associated.

After fix, ran the two commands above again, and ensured no new plugins were
inserted. Also ensured there were no duplicated vim.org IDs

Also did a dry run of the script and looked at script output. For the first
6 plugins or so, manually verified the script made the right decisions in which
plugins to delete, which to keep, and which to assign the vim.org ID.

Ran the script twice to ensure the second time it outputted nothing (meaning it successfully de-duped). Also ran the script after re-scraping to verify there were no new dupes introduced.

Reviewers: xymostech, spicyj

Reviewed By: spicyj

Differential Revision: http://phabricator.benalpert.com/D212
