Python pandas tagging entries in dataframe shortening code
Question: So I am using python pandas have an the following variables:

  * a dataframe `df` with a column 'TAG' I created to tag data into groups based on data from a column 'IDnumber'. 
  * regex patterns stored in arrays `pattern1`, `pattern2`, `pattern2-2`, ...etc
  * an array `group` which is filled with strings (ie: 'software', 'engineering', 'marketing'...etc).

The code is filling in the column df.TAG with strings from the array `group`
based on the regex patterns `pattern1`, `pattern2`, `pattern22`, ...etc

So far I have working code but there is redundancy in having multiple for
loops that look the same

    
    
    for i in range(len(pattern1)):
        df.loc[df.IDnumber.str.contains(pattern1[i]) & (df.TAG == ''),'TAG'] = group[1]
    
    for i in range(len(pattern2)):
        df.loc[df.IDnumber.str.contains(pattern2[i]) & (df.TAG == ''),'TAG'] = group[2]
    
    for i in range(len(pattern22)):
        df.loc[df.IDnumber.str.contains(pattern22[i]) & (df.TAG == ''),'TAG'] = group[2]
    
    for i in range(len(pattern33)):
        df.loc[df.IDnumber.str.contains(pattern33[i]) & (df.TAG == ''),'TAG'] = group[3]
    
    for i in range(len(pattern3)):
        df.loc[df.IDnumber.str.contains(pattern3[i]) & (df.TAG == ''),'TAG'] = group[3]
    

I am also getting a warning.

    
    
    SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame
      df.loc[df.IDnumber.str.contains(pattern1[i]),'TAG'] = group[1]
    

But the code works so I would like to know if there is a way to make the code
more efficient by reducing the number of for loops and remove the warning
without using `pd.options.mode.chained_assignment = None` to suppress the
warnings.


Answer: Your first for-loop:

    
    
    for i in range(len(pattern1)):
        df.loc[df.IDnumber.str.contains(pattern1[i]) & (df.TAG == ''),'TAG'] = group[1]
    

can be replaced with

    
    
    empty = (df.TAG == '')
    mask = df.IDnumber.str.contains('|'.join(pattern1)) & empty
    df.loc[mask, 'TAG'] = group[1]
    

This might be faster, since the entire loop is being replaced with one regex
pattern. A similar refactoring can be done for your second and last `for-
loops`.

But your third and fourth `for-loops` perplex me: `for i in
range(len(pattern2-2)):`. Python names can not contain hyphens. So what does
`pattern2-2` mean? If `pattern2-2` is just another array of strings (albeit
with an invalid variable name!?) then your third and fourth loops can be
handled the same as shown above.

* * *

If all the patterns are simply arrays of strings, then you could refactor all
the for-loops with something like

    
    
    import itertools as IT
    patterns = [pattern1, pattern2, pattern3, pattern4, pattern5]
    empty = (df.TAG == '')
    for pattern, grp in IT.izip(patterns, group):
        mask = df.IDnumber.str.contains('|'.join(pattern)) & empty
        df.loc[mask, 'TAG'] = grp
    

Note that whenever you have numbered variable names, such as `pattern1`,
`pattern2`, etc. it is usually a sign that theses variables should be replaced
by a single variable which is a list or tuple, such as `patterns` above. Then
instead of referencing `pattern1`, you'd simply use `patterns[0]`.

