Reading a gzipped file with JSON data for thousands of users I'm a newbie in Python.

I need to read a GZ file contains thousands of users. 
But I think my code is not optimized in this case. Could anyone can help me? What should I do to improve performance? Or is it standard?

if os.path.isfile("file.gz"):
    with gzip.GzipFile("file.gz", 'r') as fin:   
        for line in fin:
            if get_new_user is True:
                if datetime.strptime(json.loads(line).get('UpdatedAt'), '%Y-%m-%dT%H:%M:%S.%fZ').date() == (datetime.today()-timedelta(1)).date():
                    users.append(json.loads(line))
            else:
                users.append(json.loads(line))    
    os.remove("file.gz")
    return users
It looks like your code is somewhat non-optimal.


In the case get_new_user is True, you call json.loads(line) once for the "UpdatedAt" test, and if that passes, you call it again, returning effectively the same value, to append the users list.
In the case of get_new_user is True, you call (datetime.today()-timedelta(1)).date() once for each user, yet it is effectively constant.  You should move this out of the loop.


You could refactor your code like this:

threshold = (datetime.today()-timedelta(1)).date()
for line in fin:
    user = json.loads(line)
    if get_new_user is True:
        if datetime.strptime(user.get('UpdatedAt'), '%Y-%m-%dT%H:%M:%S.%fZ').date() == threshold:
            users.append(user)
    else:
        users.append(user)    


You are doing the get_new_user is True test for each user/line in the file.  That too is constant, and only needs to be done once.  It is a tiny gain, but you could put the for loop inside the if and else clauses.

if get_new_user is True:
    threshold = (datetime.today()-timedelta(1)).date()
    for line in fin:
        user = json.loads(line)
        if datetime.strptime(user.get('UpdatedAt'), '%Y-%m-%dT%H:%M:%S.%fZ').date() == threshold:
            users.append(user)
else:
    for line in fin:
        users.append(json.loads(line))


And maybe using some list comprehensions and a generator expression for fun ...

if get_new_user is True:
    threshold = (datetime.today()-timedelta(1)).date()
    users = [ user for user in (json.loads(line) for line in fin)
              if datetime.strptime(user.get('UpdatedAt'),
                                   '%Y-%m-%dT%H:%M:%S.%fZ'
                                  ).date() == threshold ]
else:
    users = [ json.loads(line) for line in fin ]


Or if you are truly brave ... almost a 1-liner:

threshold = (datetime.today()-timedelta(1)).date()
users = [ user for user in (json.loads(line) for line in fin)
          if get_new_user is not True or
             datetime.strptime(user.get('UpdatedAt'),
                               '%Y-%m-%dT%H:%M:%S.%fZ'
                              ).date() == threshold ]




I've left your if get_new_user is True: statement alone, but it is a little non-Pythonic.  Usually, you'd just use if get_new_user:, but those two statements are very different.  The first is true if and only if get_new_user is True whereas the second would be true when get_new_user contains any non-falsey value, such as: True, or "True" or 1, or 17 or "False", or [0], ... etc.