During QE what prevents the banks from selling government debt to invest in the stock market instead of lending out the money?
During QE what prevents the banks from selling government debt to invest in the stock market instead of lending out the money?

Nothing. Depending on which country we are talking about there might be a regulation preventing/limiting some banks investing in a stock market, in order to make banking sector less prone to collapse. For example, Basel rules mandate that banks have to keep certain level of capital buffer for their investments in stock market in order to make them resilient (see discussion about that here). This is very context dependent, there is actually no such thing as just a bank, there are commercial banks, investment banks, retail banks etc - all are regulated slightly differently and it is beyond scope of an SE answer to provide detailed overview rules for all individual institutions.
Besides that the only thing that would restrict them from doing so is that at some point it would not be profitable, especially if there is sufficient demand for loans.  Both equity and credit prices are subject to supply and demand forces.

But what if the banks instead of lending out money put most of their money in the stock market?

Ceteris paribus, this will stimulate economic activity the same way as issuing a loan would. From a macro perspective there is a little difference between bank buying from a firm A a bond with face value 1000USD with 10% coupon rate, versus bank buying 1000USD worth of stock expected to pay 100$ yearly dividend. What stimulates economic activity is firms getting the money (no matter whether as a loan or in exchange for equity) which allows them to increase investment.
If anything buying equity could potentially stimulate economy more, since there is some evidence that debt overhangs make firms invest less (e.g. see this Fed St. Louis explainer). Consequently, in situation where firms are already overleveraged such action would likely stimulate economic activity even more than just increase in borrowing.