What is the physical origin of conjugation?
I think you have posed an interesting question. Upon looking through some literature, I have chanced upon the answer to your question in Fleming (2010), p. 30-31. The diagram below was also taken from the discussion in the book. 

What you have proposed are two ways of constructing the $\pi$ molecular orbitals of butadiene from the $\pi$ molecular orbitals of two ethene molecules. What is normally discussed is the one shown on the left, whereby we consider the interaction between the two filled MOs of exactly the same energy and a similar interaction between the two unfilled MOs of equivalent energy. Considering the bonding and antibonding combinations allows us to derive the four $\pi$ Mos of butadiene. 

You also propose considering $\pi$ to $\pi^*$ conjugation as an alternative way of deriving the butadiene $\pi$ MOs. Diagramfmatically, what you are essentially referring to is shown in the MO diagram on the right. Firstly, this interaction cannot be the main interaction between the two ethene $\pi$ systems simply because it would result in the butadiene $\pi$ MOs comprising of two separate levels of two degenerate MOs each, which is quite different from the picture which we are aware of, which has 4 $\pi$ MOs of different energies. Secondly, note that this interaction is considerably weaker due to the large energy difference between the $\pi$ and $\pi^*$ MOs of an ethene molecule. Recall that for a strong interaction between any two orbitals, one of the important conditions is that the energy difference between them must not be too large. In fact, the strongest interaction is obtained when the energy difference between them is zero. This interaction would contribute to the energies of the $\pi$ MOs in butadiene but the contribution would surely be less than the one that is highlighted in the previous paragraph. 



To construct the MOs of the butadiene, it is incomplete to just consider the stronger interactions shown on the left. For completeness, we would have to then mix the resultant MOs obtained from these two interactions (i.e. both left and right diagrams), factoring in the extent of contribution of each interaction. Naturally, we would be curious as to what is the effect of the weaker interaction on the MO energies derived from the stronger interaction. Fleming (2010) describes the effects rather detailedly in the text:


  Mixing these two sets together, and allowing for the greater contribution from the stronger interactions, we get the set of orbitals shown in the figure below. Thus, to take just the filled orbitals, we see that $\psi_1$ is derived by the interaction of $\pi$ with $\pi$ in a bonding sense ($\psi_a$), lowering the energy of $\psi_1$ below that of the $\pi$ orbital, and by the interaction of $\pi$ with $\pi^*$ in a bonding sense ($\psi_w$), also lowering the energy below that of the $\pi$ orbital. Since the former is a strong interaction and the latter weak, the net effect is to lower the energy of $\psi_1$ below the $\pi$ level, but by a little more than the amount ($\beta$ in simple Huckel theory) that a $\pi$ orbital is lowered below the p level in making the $\pi$ bond of ethylene. However, $\psi_2$ is derived from the interaction of $\pi$ with $\pi$ in an antibonding sense ($\psi_b$), raising the energy above that of the $\pi$ orbital, and by the interaction of $\pi^*$ with $\pi$ in a bonding sense ($\psi_x$), lowering it again. Since the former is a strong interaction and the latter weak, the net effect is to raise the energy of $\psi_2$ above the $\pi$ level, but not by as much as a $\pi^*$ orbital is raised above the p level in making the p bond of ethylene. Yet another way of looking at this system is to say that the orbitals $\psi_1$ and $\psi_2$ and the orbitals $\psi_3^*$ and $\psi_4^*$ mutually repel each other.




Does it really matter if we factor in the weaker interaction into our considerations? Yes, it does. If we were to not do that, we would observe that the stabilisation predicted would not be as great as the one which we would predict from just looking at the MO diagram shown on the left. In fact, as Martin writes in the comments, the interaction between orbitals would produce an antibonding MO that is more antibonding than the bonding MO produced is bonding. If we were to not consider the weaker interaction, then we would predict that $\psi_2$ is raised more in energy from the original $\pi$ level than $\psi_1$ is lowered in energy from the original $\pi$ level (i.e. $\ce {E_2}$ is larger than $\ce {E_1}$), and this is clearly not the case. 

Of course, as you mentioned, we can also build the $\pi$ MOs of butadiene from first principles via the Huckel method. I am not very well-versed with the mathematics behind the method but I do believe that the method discussed above and the original Huckel method would converge in terms of the results obtained eventually. Hence, they are equally valid approaches to the problem.

Reference 

Fleming, I. Molecular Orbitals and Organic Chemical Reactions (Reference ed.). United Kingdom : Wiley, 2010.