Deciding which group twists out of the plane when ortho effect takes place
I am not sure the 3D conformation stills as reported on PubChem are all that accurate. Although crystal structures are sometimes referenced for specific molecules, some of the displays are in fact models, some of which do not make sense to me.
Here are some observations regarding the three molecules you noted.
2-Nitrobenzoic acid
The PubChem model shows both oxygen atoms of the nitro- group to be precisely coplanar with the phenyl ring. In the crystal structure, these oxygen atoms have a dihedral angle of $125.5^\circ$ and $129.1^\circ$ with respect to the aryl, i.e. away from coplanarity.
The PubChem model shows both oxygen atoms of the carboxylic acid to be precisely normal to the phenyl ring. In the crystal structure, these oxygen atoms have a dihedral angle of $152.9^\circ$ and $162.7^\circ$ with respect to the aryl, i.e. closer to coplanarity.




Pubchem
Crystal structure entry CCSJWHN










N,N-Dimethyl-2-nitroaniline
I could not find the crystal structure of this molecule. However, I’m not sure it makes sense that the N,N-dimethyl amine adopt an $\ce{sp^2}$ character at the nitrogen center, at least as far as the Pubchem model representation is concerned. In such a pose, there can’t be extended resonance from the phenyl to amine as obtained with unsubstituted aniline. Either the methyls are near coplanar with the aryl to allow the pi orbitals to enter resonance or the amine is in $\ce{sp^3}$.
2’-Aminoacetophenone
I could not find the crystal structure of this molecule. What I found was the 4,5-dimethyl variant as 1-(2-amino-4,5-dimethylphenyl)ethan-1-one.
The 2’-Aminoacetophenone PubChem model shows the acetone oxygen to be precisely $30.0^\circ$ out of the aniline plane. In contrast, the crystal structure of 1-(2-amino-4,5-dimethylphenyl)ethan-1-one indicates that oxygen to be precisely in plane with the aniline. This coplanarity may not be surprising in this case as there is an intramolecular hydrogen bond between the two substituents.




Pubchem
Crystal structure entry 1825318 (BEXTUW)








2'-Aminoacetophenone
1-(2-amino-4,5-dimethylphenyl)ethan-1-one




Original Post:

To my understanding, two things would effect this; one would be the extra stabilisation/destabilisation of a group due to not being in resonance any longer once it moves out of the plane of the benzene ring, and the second would be the &quot;bulkiness&quot; of the group itself, which would energetically be measured as the group's A-Value.

I think steric hindrance, you noted as “bulkiness” and estimated with an A-Values is the primary factor. There is little room for compromise on this one as two atoms can’t occupy the same space. The second factor is resonance as you noted. Another influencing factor is the presence of intramolecular interactions, such as a hydrogen bond. Solvation and pH are other influential factors to the ortho effect.