Is population growth irrelevant with respect to per capita GDP growth
Indeed, there is branch of literature about semi-endogenous models (Jones, 1995, Segerstrom 1999 etc.). They were basically arguing that there was a scale effect in endogenous growth models, which means that when population increases, by default, growth increases (it is due to the fact that there would be more researcher in society, this is the scale effect.) and this branch of literature shows that growth rate at the long run is equal to the growth rate of population, which is an exogenous term. It is why they called it as "semi-endogenous".

This was challenging endogenous models but the empirical literature states that that the semi-endogenous model does not fit to data.

Some empirical references are : 

Ha and Howitt (2007) 

http://onlinelibrary.wiley.com/doi/10.1111/j.1538-4616.2007.00045.x/full

and 

Madsen (2008) 

http://link.springer.com/article/10.1007/s10887-007-9024-0