Full video+audio recording with QuickTime on Mac OS High Sierra?
There are three different auto levels involved with a recording. The first two can be set from System Preferences, as shown below.



The input volume is the record level from either the built-in mic or input jack. The output volume controls level for the internal speakers other device (such as headphone) you may plug into the output jack. The third auto level is set in QuickTime, as shown below.



This is the level used monitor what is being recorded. This is not the same level as the output volume shown in System Preferences.

If you are using the built-in mic or an external microphone and the internal or some other external speakers, then the System Preferences output volume and/or the Quick time level should set to zero. 

Although not directly asked in our question, I might as well add the following:

If you are trying to make a recording where the sound is being produced by the computer, then you need to plug a jumper into the input and output jacks. Say for example, you want to record video either from the camera or the screen and you want to sound to be music from iTunes, then you will to feed the sound from the output jack back into the input jack.