Is there a freely downloadable LEGO set and part dataset?
I wasn't able to find anything like this, so I contacted the admins of the sites that I mentioned in the question. Rebrickable was kind enough to send me a dump of their set inventories under a Creative Commons 3.0 BY-SA license, which basically means that you can do whatever you want with the data as long as Rebrickable is acknowledged as the source and you share any updates that you make.
I've created an archive which contains a spreadsheet for each set that was in the dump that I received. Hopefully this is useful to someone else.
Lego Set Inventories as of March 2014
Update:
The raw data that I used to create these spreadsheets is now available directly from Rebrickable and is updated monthly there. I also maintain software to build an SQLite database from these dumps.