simulating gain vs parameter in LTSpice
According to the Datasheet of the AD8337 (Rev. D), page 21/28 under section Gain control considerations:

The offset voltage effect of the AD8337, as with all VGAs, can
appear as a complex waveform when observed across the range
of VGAIN voltage. Generated by multiple sources, each device has
a unique offset voltage (VOS) profile while the GAIN input is
swept through its voltage range.

At least from what I can understand, you're seeing the influence of \$V_{GAIN}\$ on the offset voltage, which directly affects the output voltage as well.