Apps store settings/cache (the Android directory) on SD card even though it's not where they reside? Phone: Galaxy Note 4, Android 5.1.1

I would expect apps to store their settings/cache in a place where they reside, i.e. if app is on MMC the settings should be on MMC and if it's SD they should be on SD (one would hope!).

However I observe that after a reboot - with an empty SD card plugged in (and to my horror) - a lot of applications (YouTube for example) create directories and files on the SD card (under Android directory), typically most of them being empty. What I want is to confine my entire apps storage to MMC and I'll keep my pictures and movies on SD.

Is there a reason why Android/Samsung is preventing me from doing that? Is there a setting I forgot to turn on/off?

Cheers in advance for any help, much appreciated.
Settings of the apps are stored in their data folder and not in where they reside. The data folders of apps can be found under /data/data, a directory that a non-root user can't access.

What you see are just some additional data folders some apps tend to create.