Random List of millions of elements in Python Efficiently
Question: I have read this [answer](http://stackoverflow.com/questions/1022141/best-way-
to-randomize-a-list-of-strings-in-python) potentially as the best way to
randomize a list of strings in Python. I'm just wondering then if that's the
most efficient way to do it because I have a list of about 30 million elements
via the following code:

    
    
    import json
    from sets import Set
    from random import shuffle
    
    a = []
    
    for i in range(0,193):
        json_data = open("C:/Twitter/user/user_" + str(i) + ".json")
        data = json.load(json_data)
        for j in range(0,len(data)):
            a.append(data[j]['su'])
    new = list(Set(a))
    print "Cleaned length is: " + str(len(new))
    
    ## Take Cleaned List and Randomize it for Analysis
    shuffle(new)
    

If there is a more efficient way to do it, I'd greatly appreciate any advice
on how to do it.

Thanks,


Answer: A couple of possible suggestions:

    
    
    import json
    from random import shuffle
    
    a = set()
    for i in range(193):
        with open("C:/Twitter/user/user_{0}.json".format(i)) as json_data:
            data = json.load(json_data)
            a.update(d['su'] for d in data)
    
    print("Cleaned length is {0}".format(len(a)))
    
    # Take Cleaned List and Randomize it for Analysis
    new = list(a)
    shuffle(new)
    

.

  * the only way to know if this is faster is to profile it!
  * do you prefer sets.Set to the built-in set() for a reason?
  * I have introduced a with clause (preferred way of opening files, as it guarantees they get closed)
  * it did not appear that you were doing anything with 'a' as a list except converting it to a set; why not make it a set from the start?
  * rather than iterate on an index, then do a lookup on the index, I just iterate on the data items...
  * which makes it easily rewriteable as a generator expression

