Can overshooting tops enter the stratosphere, or will they rather push the tropopause upward?
This is an interesting question and I feel the best answer is "a little bit of both".  Consider your picture without the overshooting top and just the anvil.  The tropopause is clearly defined as the anvil spreads out horizontally against the static stability of the stratosphere.  

Now let us consider a single parcel in the updraft, accelerating upward due to CAPE and hitting the tropopause with significant vertical momentum.  This parcel is full of tropospheric air but makes an excursion into the stratosphere until the negative buoyancy it experiences reduces its vertical velocity to 0 (the top of the overshooting top) and then continues to accelerate downward to eventually oscillate about its equilibrium level (level of neutral buoyancy).  For this single parcel I would argue that it is in the stratosphere.

Now consider all of the updraft parcels doing this.  They are all transporting tropospheric air upward into the stratosphere -- inside that cloud it is all tropospheric air and toward the edges it is a mix of tropospheric and stratospheric air due to entrainment.  The tropospheric air transported upward that mixes with the stratospheric air will reduce the static stability locally and may eventually modify the temperature gradient such that you consider the tropopause lifted.  

I may run a simulation later with many grid levels around the tropopause and see what a model has to say.  I'll edit with my results.