What are some inexpensive easy to use SPI devices I can experiment with?
74HC595 : 8 bit serial-in parallel-out shift register with holding register; this might not sound like SPI to you but it definitely is.

MCP23S17 : a bit above $1, but a really versatile 16-bin I/O extender. (MCP23017 is essentially the same chip but with an I2C interface).

RFM73 or NRF24L01 : more like $3, but nice little short-distance radio modules.