Folding basket - is there such a thing?
A subset of panniers that are designed for shopping bags are often called "grocery panniers," the distinguishing features being an open top and dimensions similar to the base of a paper grocery bag (not that you can get those very much anymore). 

They are typically easily removable (just hooks), and in my experience do not bounce around alot, as was suggested in another answer. There is an elastic cord to the lower hooks that keeps them secure. 

If you want to leave them on your bike, they do fold compactly against the rack and usually have velcro or snap closures to hold them folded. I usually just remove them when I'm not using them. 

Here is an example: