I have a code that grabs quickly the twitter followers for 12 different users. After appending to a Pandas DataFrame, the data is compared to the same file pulled the day before. It allows me to see which users have gained, lost, and returned followers. The code works fine. However, the for loop to compare changes between days is slow. Any ideas how to help me out with the for loop section?TRANSFORMATION FUNCTIONS# Transformation Functions# Gained Usersdef gained_users(account, new, old):    gained_followers = []    old_user_ids = old.Follower_ID[old.Handles == account].unique().tolist()    new_user_ids = new.Follower_ID[new.Handles == account].unique().tolist()    gained = list(set(new_user_ids) - set(old_user_ids))    gained_followers.extend(map(str, gained))    return gained_followers# Lost Usersdef lost_users(account, new, old):    old_user_ids = old.Follower_ID[old.Handles == account].unique().tolist()    new_user_ids = new.Follower_ID[new.Handles == account].unique().tolist()    lost = list(set(old_user_ids) - set(new_user_ids))    return lost# Returned Usersdef returned_users(account, new, old):    returned_followers = []    new_user_ids = new.Follower_ID[new.Handles == account].unique().tolist()    returned_user_ids = old.Follower_ID[(old.Handles == account) & (old.End_Date.notnull() == True)].unique().tolist()    returned =  list(set(returned_user_ids).intersection(new_user_ids))    returned_followers.extend(map(str, returned))    return returned_followersFOR LOOP SECTION# Add Returned Usersfor username in lookup_users:    returned_ids = returned_users(username, new_followers_df, historical_followers_df)    if returned_ids:        historical_followers_df.loc[(historical_followers_df[Handles] == username) & (historical_followers_df[Follower_ID] == ids), Start_Date] = today        historical_followers_df.loc[(historical_followers_df[Handles] == username) & (historical_followers_df[Follower_ID] == ids), Returned_After_Days] = pd.to_datetime(historical_followers_df.Start_Date) - pd.to_datetime(historical_followers_df.End_Date)         historical_followers_df.loc[(historical_followers_df[Handles] == username) & (historical_followers_df[Follower_ID] == ids), End_Date] = np.NaN# Add Lost Usersfor username in lookup_users:    lost_ids = lost_users(username, new_followers_df, historical_followers_df)    if lost_ids:        for ids in lost_ids:            historical_followers_df.loc[(historical_followers_df[Handles] == username) & (historical_followers_df[Follower_ID] == ids), End_Date] = today# Add Gained Usersfor username in lookup_users:    new_ids = gained_users(username, new_followers_df, historical_followers_df)    if new_ids:            gained_users_list = pd.DataFrame({            Handles: username,            Follower_ID: new_ids,            Start_Date: today})            historical_followers_df = gained_users_list.append(historical_followers_df, ignore_index=True)EDIT:Hi @Graipher thanks for your help. The reasoning makes sense and the structure is much neater! Hopefully if you don't mind, can you please answer these three questions:1. Could you explain the old_account = old.Handles == account passage, as I have never seen it before!2. If I run the code as it is it says that old_followers = user_followers & (historical_followers_df[Follower_ID] == ids) is this because ids in the filter is not specified? If so, Is it correct to say that:a. I have to amend the categorize_users function above. Adding:  old = map(str, old_user_ids)  and   return old, gained, lost, returnedb. In the for loop where I call a function I add a new variable ids so that it looks like: ids, new_ids, lost_ids, returned_ids = categorize_users(username, new_followers_df, historical_followers_df)3. Finally, I think that there's a problem with new_ids as it says that TypeError: object of type 'map' has no len()a. Is this a code problem? I can't really figure that part out.
Calculate Twitter Follower Changes Over Time
python;performance;pandas;twitter
You should try to avoid calculating the same things over and over. The first example are your transformation functions. While it is nice that you have separated the concerns, it probably costs you quite a bit here. It might be better to make this one function:def categorize_users(new, old):        Return the gained, lost and returned followers    from the two `pandas.DataFrame`s `old` and `new`.        old_user_ids = set(old.Follower_ID)    new_user_ids = set(new.Follower_ID)    returned_user_ids = set(old.Follower_ID[old.End_Date.notnull()])    gained = map(str, new_user_ids - old_user_ids)    lost = map(str, old_user_ids - new_user_ids)    returned = map(str, returned_user_ids.intersection(new_user_ids))    return gained, lost, returnedNote that I re-used everything that was used more than once. I also skipped the unique part, because the set already does it, but you might want to add it back in to see if it is any faster.Now, to your actual for loop:for username in lookup_users:    new_user_followers = new_followers_df[Handles] == username    old_user_followers = historical_followers_df[Handles] == username    new_ids, lost_ids, returned_ids = categorize_users(        username,        new_followers_df[new_user_followers],        historical_followers_df[old_user_followers])    # Add Returned Users    if returned_ids:        for ids in returned_ids:            old_followers = user_followers & (historical_followers_df[Follower_ID] == ids)            historical_followers_df.loc[old_followers, Start_Date] = today            historical_followers_df.loc[old_followers,Returned_After_Days] = pd.to_datetime(historical_followers_df.Start_Date) - pd.to_datetime(historical_followers_df.End_Date)            historical_followers_df.loc[old_followers, End_Date] = np.NaN    # Add Lost Users    if lost_ids:        for ids in lost_ids:            old_followers = user_followers & (historical_followers_df[Follower_ID] == ids)            historical_followers_df.loc[old_followers, End_Date] = today    # Add New Users    if new_ids:        gained_users_list = pd.DataFrame({            Handles: username,            Follower_ID: new_ids,            Start_Date: today})        historical_followers_df = gained_users_list.append(            historical_followers_df, ignore_index=True)I made your three for loops into one, because now we can actually get all the three different classes in one go. I also put the selection of the relevant old followers which need to be modified into its own variable as well, so it is not computed four times per loop.I also moved the filtering of the dataframes to one user already in the loop, so that categorize_users becomes more general.