Does Attiny13 pull a lot more current in sleep mode than Attiny85 or did I do something wrong?
The ATtiny13 is spec'ed to draw ~0.2uA without WDT and ~4uA with WDT @3V @25C so if you are seeing 100uA then something else is going on here.

There is a lot happening in the above code. I'd scale it back to just the minimum code needed to put the chip into power down mode and then measure how much current that uses without anything connected to the chip besides power (no LEDs, inputs, not even programmer).  You should see something close to the above numbers. Then start adding in more code and hardware, checking the power usage after each iteration so you can find the issue.