Quickly pruning edges below average weight for a membership-based weighting
Yes, we can make your idea work.  I'll show how below, and prove that it is correct (it does not miss any edges).

The first step will be to sort the groups by decreasing size.  Say their sizes are $s_1 \ge s_2 \ge s_3 \ge \dots$.  Say $t$ is your threshold.  Find the smallest $k$ such that

$$1/s_1 + 1/s_2 + \dots + 1/s_k \ge t.$$

Lemma. No edge can have weight above the threshold $t$ based only upon membership in groups $1,2,\dots,k-1$.  In other words, every edge with above-threshold weight must be in at least one of the groups $k,k+1,k+2,\dots$.

This observation lets you apply your idea, to optimize the performance of the algorithm.

Algorithm. Sort the groups by decreasing size and find $k$, as above.  For each of groups $k,k+1,k+2,\dots$, look at all pairs of points within the same group, and compute the edge weight of each such edge.  Output the ones whose edge weight is above the threshold $t$.

This algorithm can be proven to output all above-threshold edges, thanks to the lemma above.