Shouldn't the asset "View source" privilege limit access to an asset source in the selection modal?
There is an issue filed for this here https://github.com/craftcms/cms/issues/887 that you might want to consider thumbs-upping or commenting on.