Pandas updating/adding columns to rows incrementally using dictionary key values
Since there is an e missing in the input dataframe in col A provided by you, I have added it:

#input
A_list = ['a', 'b', 'c', 'd', 'f']
df = pd.DataFrame({'A': ['a', 'b', 'c', 'd','e','f'], 'B': [1,2,3,3,2,1]})




You can start by joining the list you have:

pat='({})'.format('|'.join(A_list))
#pat --&gt; '(a|b|c|d|f)'


Then using series.str.extract() I am extracting the matching keys from the series based on the pattern we created.

s=df.A.str.extract(pat,expand=False) #expand=False returns a series for further assignment
print(s)




0      a
1      b
2      c
3      d
4    NaN
5      f


Once you have this series, you can decide what you want to do with it. For,example if I take your function:

def gen_data(key):
    #THIS FUNCTION IS JUST AN EXAMPLE THE COLUMNS ARE NOT NECESSARILY RELATED OR USE THE KEY 
    data_dict = {'C':key*2, 'D':key, 'E':key*3}
    return data_dict


And do the below:

df.join(pd.DataFrame(s.apply(gen_data).values.tolist()))


We get the desired output:

   A  B    C    D    E
0  a  1   aa    a  aaa
1  b  2   bb    b  bbb
2  c  3   cc    c  ccc
3  d  3   dd    d  ddd
4  e  2  NaN  NaN  NaN
5  f  1   ff    f  fff


However I personally wouldn't use apply unless mandatory, so here is another way using df.assign() where you can pass a dictionary of the extracted series and assign it to the dataframe:

df=df.assign(**{'C':s*2,'D':s,'E':s*3})




   A  B    C    D    E
0  a  1   aa    a  aaa
1  b  2   bb    b  bbb
2  c  3   cc    c  ccc
3  d  3   dd    d  ddd
4  e  2  NaN  NaN  NaN
5  f  1   ff    f  fff