Why can apps access the whole SD card? Is there a technical reason for why apps that only need their own data are not limited to one folder? I do realise that some apps use other apps' data, but I'd be much more likely to install an app that only asked for permission to store files in one, specific folder.
SD cards generally use the FAT file system, which does not allow for filesystem permissions.  This means that the Android OS would have to manage a custom list of files, their permissions, a mapping of apps to files/folders, etc. in order to prevent apps from accessing the whole card.  This would be a terrible mess.  Not only complicated, but it would require a lot of storage space.  And then one day you take out your SD card and remove some pictures, and the entire system breaks.  You get the idea.