Why does flash size depend on target hardware?
Because they're different subarchitectures (avr25 for the '13A, and avr5 for the '328P) and have different peripherals. As such, they have both slightly different instruction sets and different numbers of interrupt vectors. The interrupts alone (10 vs. 26, 1 code address vs. 2 per vector) result in a 26*4-10*2=84 byte difference.

The differing speed between the devices can matter as well, since _delay_*() with a slower device can mean less than 256/65536 loops whereas a faster device would need more than 256/65536 loops. This can mean a difference of 4 to 8 bytes each time one of those functions is called.