32kHz crystal does not oscillate
The oscillator of the ATTINY2313 has a minimum crystal frequency of about 400 kHz. Your 32 kHz crystal is far off and thus unlikely to work.

The datasheet is a bit misleading: You can operate the 2313 with 32 kHz, but you'd need an activly driven clock signal (external oscillator) and not a simple passive crystal.