Variable bitrate for AAC - how to reach it?
According to the wiki, you are likely to get worse results with the default AAC encoder when using VBR. See: https://trac.ffmpeg.org/wiki/Encode/AAC

One suggestion would be to use libfdk_aac. The other would be to demux the audio, re-encode it using a different encoder, and then remux that file back in to the video using ffmpeg.