Does " would like to access files in your Downloads folder" give complete access to directory, or just certain files in it?
No: the permission is for unlimited access to that folder.
There's nothing to stop an app secretly uploading all your files to some server, except that people would notice and denounce the company, and there's no commercial benefit in doing it anyway.
The Documents folder is another location that has  privileged access, but the trouble is that every app that can open and save files is likely to need access to it. If you create another user folder outside Documents, then it will be open to all user processes.
You can install something like Little Snitch, which will regularly worry you with information about all outgoing internet connections, and let you allow/deny them.