Which files/folders on my SDCard are unsafe to delete manually?
I'm tempted to say everything is safe to delete.  It should be, with a storage device intended to be removed and used elsewhere.  But I know some apps will be unhappy.  So a good rule is not to delete anything that's clearly used by an app that you care about.

It's probably a good idea not to delete level 0 directories that start with a . such as /.android_secure/ (protected apps) either.  Those folders are generally used by the system and, while I don't believe Android ever stores anything important on the SD card (besides apps if you consider them important), there's probably no reason to delete them either.  /Android/ and /data/, /sd/ and /*.img, and /settings.db (if present) are also used by the system.  Those are app data, recovery images, and various settings respectively.

Also watch out for a /DCIM/ folder or similar, which may have any pictures you've taken.  Presumably you'd want to back them up.  Likewise with /download/ and /media/.

You'll want to watch out for /update.zip, /busybox, /clockworkmod/, and the like as well if you're rooted or have a custom recovery.

Note if, like me, you have "internal SD" as well as external, then most of the above is stored on the internal SD.  The external SD appears as /external_sd/ within the internal SD, or may be mounted at /emmc/ or similar.  You can apply the rules above to both storage areas.