Python pandas: Take column of counts and create DataFrame with a row per count
You are correct in thinking that iterrows is a very bad sign for Pandas code. Even worse is building up a DataFrame one row at a time like this with pd.concat - the performance implications are dreadful.

Instead of reaching for loops, your first step should be to check if there is a vectorized DataFrame method you could use. In this case.. perhaps there isn't. 

Next step is dropping into NumPy. And lo and behold, there is numpy.repeat, which allows you to repeat an array along an axis with another array of counts. We can wrap that up in a function. 

def repeat_frame(df, counts):            
    rep_array = np.repeat(df.values, counts, axis=0)
    return pd.DataFrame(rep_array, columns=df.columns)

# would be called with 
repeat_frame(df[['a', 'b']], df['count'])


This runs in 450 µs on my machine compared to your current solution in 20.2 ms for your sample data. On a larger sample size with 1000 rows, it runs in 511 µs compared to 6.86 s for your current solution, roughly 13000 times faster. 



If you find yourself wanting to use pd.concat or similar in any situation like this to build up a DataFrame row-by-row, stop! There is going to be a faster way. 


Check for vectorized Pandas functions that can help you out
Ditto for NumPy and SciPy after that 


And if you are certain you have some wild iteration logic that cannot be vectorized (which is unlikely), create an empty array of the necessary shape and consider using Numba or Cython to speed up filling it up with a loop.