Can you still set a background color style in Textedit in Mavericks?
This might depend on whether your document is formatted as rich text (.rtf, .docx, etc.) or plain text (.txt).

If your document is plain text, I'm not sure why you would want to do this anyway, since your color changes won't be saved when you save the document.

When the document is formatted as rich text, you should see the rich text editor toolbar at the top of your window. The highlighting button is in the center, and looks like an "a" with a red slash through it (if you have no highlight color selected).  When there is a highlight color selected, it looks like an "a" with the highlight color as the background.

Hope that helps!