An Efficient Lookup Table in redis--implemented using redis sets?
Question: I want to use redis to store a large set of user_ids and with each of these
ids, a "group id" to which that user was previously assigned:

    
    
    User_ID | Group_ID
       1043 | 2 
       2403 | 1
    

The number of user_ids is fairly large (~ 10 million); the number of unique
group ids is about 3 - 5.

My purpose for this LuT is routine:

  * _find the group id for a given user_ ; and

  * _return a list of other users (of specified length) with the same group id as that given user_

There might be an idiomatic way to do this in redis or at least a way that's
most efficient. If so i would like to know what it is. Here's a simplified
version of my working implementation (using the python client):

    
    
    # assume a redis server is already running 
    # create some model data:
    import numpy as NP
    NUM_REG_USERS = 100
    user_id = NP.random.randint(1000, 9999, NUM_REG_USERS)
    cluster_id = NP.random.randint(1, 4, NUM_REG_USERS)
    D = zip(cluster_id, user_id)
    
    from redis import Redis
    # r = Redis()
    
    # populate the redis LuT:
    for t in D :
        r.sadd( t[0], t[1] )
    
    # the queries:
    # is user_id 1034 in Group 1?
    r.sismember("1", 1034)
    
    # return 10 users in the same Group 1 as user_id 1034:
    r.smembers("1")[:10]     # assume user_id 1034 is in group 1
    

So i have implemented this LuT using ordinary redis **_sets_** ; each set is
keyed to a Group ID (1, 2, or 3), so there are three sets in total.

Is this the most efficient way store this data given the type of queries i
want to run against it?


Answer: Using sets is a good basic approach, though there are a couple of things in
there you may want to change:

Unless you store the group ID for each a user somewhere you will need 5 round
trips to get the group for a particular user - the operation itself is O(1),
but you still need to consider latency. Usually it is fairly easy to do this
without too much effort - you have lots of other properties stored for each
user, so it is trivial to add one for group id.

You probably want SRANDMEMBER rather than SMEMBERS - I think SMEMBERS will
return the same 10 items from your million item set every time.

