What characteristic(s) of inverse agonists allow for inhibitory effects?
The short answer is that an inverse agonist binds preferentially to the inactive state of the receptor, so it reduces the receptor's intrinsic, basal activity. The agonist was only raising the activity, not switching it on.

The most basic model of receptor pharmacology is that the receptor, $\ce{R}$, and the agonist, $\ce{A}$, are in a chemical equilibrium: $\ce{A}+\ce{R}\rightleftharpoons\ce{AR}$. In this model, the agonist bound receptor, $\ce{AR}$, generates the biological response. But in truth, this isn't a very accurate model since nothing's actually happened to the receptor. Why should $\ce{R}$ generate a biological response when it's bound but not when it's unbound? This model also wouldn't explain inverse agonists.

The del Castillo Katz model rectifies this problem by suggesting that there are two equilibria. The first is for agonist binding. And the second is for a conformational change in the receptor to an activated state, $\ce{R}^*$, that causes the biological response. Hence, $\ce{A}+\ce{R}\rightleftharpoons \ce{AR} \rightleftharpoons \ce{AR^*}$. This activated state is a different shape of the receptor, which allows a biological response. Depending on the receptor type, it could be an opened ion channel, a 7TM protein which destabilises the binding of a G-protein or more.

For many receptors (such as the GABAA receptor), the receptor can go into the active, $\ce{R^*}$, state without an agonist being bound. Hence we have the equilibrium $\ce{R}\rightleftharpoons \ce{R^*}$. This is called constitutive activity - the receptor generates some level of biological response by itself. So this shows us the chemical properties of inverse agonists. An inverse agonist binds with a higher affinity to the $\ce{R}$ form of the receptor than the $\ce{R^*}$ form. So the equilibrium shifts to the left and the receptor has reduced constitutive activity. So the drug reduces the biological effect by pulling it below the basal level. See the figure below: figure 3-1 from (Blumenthal and Garrison, 2011). NB: they opt for $\ce{R_i}$/$\ce{R_a}$ instead of $\ce{R}$/$\ce{R^*}$ and $\ce{L}$ for a generic ligand.



An agonist and inverse agonist definitely don't have the same structure otherwise they'd be the exact same molecule. They might have the same pharmacophore but they necessarily have some chemical difference that makes the inverse agonist prefer binding to $\ce{R^*}$ but not the agonist. To be honest, I'm not convinced that they even need the same pharmacophore. Although as far as I know in most cases, they do. It's not too difficult to imagine a situation where the conformational change from $\ce{R}$ to $\ce{R^*}$ moves residues in a binding pocket and prevents (sterically/electronically) the inverse agonist from binding as easily. I'm not so sure how the energy levels would work here so you'll have to get back to me on that or ask the chem/phys stackexchanges.

My crude figure below has a 7TM protein (Green) with a G protein (brown) attached. The receptor can undergo a conformational change to the $\ce{R^*}$ form (horizontally flipped) which causes the G protein to dissociate and cause some biological effect. The agonist (red star) can bind into the pocket but fits with a much higher affinity for the $\ce{R^*}$ form since the conformational change gives it a nice binding pocket. Since it fits better in the $\ce{R^*}$ state, the equilibrium shifts to the right so the G protein prefers to dissociate and the agonist introduces a biological effect.



On the other hand, the inverse agonist may fit better into the $\ce{R}$ binding pocket than the $\ce{R^*}$ binding pocket. Hence, it shifts the equilibrium to the $\ce{R}$ state, so the G protein prefers not dissociate and the biological effect is reduced.



Further reading: (Blumenthal, D. K. and Garrison, J. C. (2011). 'Pharmacodynamics: Molecular Mechanisms of Drug Action' in Goodman and Gilman. 12th ed. The Pharmacological Basis of Therapeutics) and ( Rang and Dale. (2015). 'How drugs act: general principles' in Pharmacology. 8th ed.)