Pandas add calculated row for every row in a dataframe I have a dataframe like so:
id  variable  value
1      x        5
1      y        5
2      x        7
2      y        7

Now for every row, I want to add a calculated row. This is what I am doing as of now:
a = 2
b = 5
c = 1
d = 3
df2 = pd.DataFrame(columns =  [&quot;id&quot;, &quot;variable&quot;, &quot;value&quot;])
for index, row in df.iterrows():
    if row['variable'] == 'x':
        df2 = df2.append({'id':row['id'], 'variable':'x1', 'value':a*row['value']+b}, ignore_index=True)
    else:
        df2 = df2.append({'id':row['id'], 'variable':'y1', 'value':c*row['value']+d}, ignore_index=True)
df = pd.concat([df, df2])
df = df.sort_values(['id', 'variable'])

And so finally I get:
id  variable  value
1      x        5
1      x1       15
1      y        5
1      y1       8
2      x        7
2      x1       19
2      y        7
2      y1       10

But surely there must be a better way to do this. Perhaps where I could avoid for loop and sorting, as there are a lot of rows.
iterrows
Of all the ways to iterate over a pandas DataFrame, iterrows is the worst. This creates a new series for each row. this series also has a single dtype, so it gets upcast to the least general type needed. This can lead to unexpected loss of information (large ints converted to floats), or loss in performance (object dtype).
Slightly better is itertuples
append
Appending row per row can be very slow (link1link2)
Better would be to assembly them in a list, and make a new DataFrame in 1 go.
vectorise
One easy change you can make is not iterating over the database in 'Python' space, but using boolean indexing
x = df[&quot;variable&quot;] == &quot;x&quot;
y = df[&quot;variable&quot;] == &quot;y&quot;

df2 = df.copy()
df2.loc[x, &quot;value&quot;] = df.loc[x, &quot;value&quot;] * a + b
df2.loc[y, &quot;value&quot;] = df.loc[y, &quot;value&quot;] * c + d
df2.loc[x, &quot;variable&quot;] = &quot;x1&quot;
df2.loc[y, &quot;variable&quot;] = &quot;y1&quot;

wide vs long
This can be made a lot easier by reforming your dataframe by making it a bit wider:
df_reformed = (
    df.set_index([&quot;id&quot;, &quot;variable&quot;]).unstack(&quot;variable&quot;).droplevel(0, axis=1)
)


variable  x   y
id        
1     5   5
2     7   7


Then you can calculate x1 and y1 vectorised:
df_reformed.assign(
    x1=df_reformed[&quot;x&quot;] * a + b, 
    y1=df_reformed[&quot;y&quot;] * c + d
)

and then convert this back to the long format:
result = (
    df_reformed.assign(
        x1=df_reformed[&quot;x&quot;] * a + b, 
        y1=df_reformed[&quot;y&quot;] * c + d
    )
    .stack()
    .rename(&quot;value&quot;)
    .sort_index()
    .reset_index()
)