Ver2.0 SDHC Card long response time after ACMD41 during initialization
Reading your question I have got an analogy: stopping vehicle's engine at the red traffic light for a minute or two. You think that: you save a little gas, you save a little on engine wearout, and you believe you are green because do not emit gases within that minute or two.
The issue in this analogy is that all assumptions may be wrong - unless engine is especially designed for regular start-top.
Ordinary engine when started: consumes more gas; emits more pollutiuon; and wears much more.
Now think about your case: how much power SD-card restart consumes versus sleep/inactivity mode?  I also advise you looking for special low power card, or the one which is designed for regular power off/power on operations (read: faster startup time and low power consumption during initialization). Not sure such exist, or they may exist for industrial market only (and will cost more than consumer cards).
Update: notice the following in the specification:

6.4.2 Power Down and Power Cycle
When the host shuts down the power, the card VDD shall be lowered to less than 0.5Volt for a minimum period of 1ms. During power down, DAT, CMD, and CLK should be disconnected or driven to logical 0 by the host to avoid a situation that the operating current is drawn through the signal lines.

Thus measuring consumption of SD-card at its power pin may not be enough - current may still flow through signal pins.
In my opinion, for card to consume minimal power in power-on state you must set CLK low (thus use SPI mode 0, or at least ensure it does not toggle) and CS high.
If you still want to stick to power cycle mode when cards may really show long startup times, then there will be no electrical engineering solution, you must perform handling in software. For example, divide data buffer into two parts, and fill one part while starting SD-card and flushing second part. It will assume you will be starting card twice more times, but will ensure no interruption with cards with startup time up to ~250 ms.

And I forgot to mention I do not use pull-up resistors in my circuit because the SPI worked without them. But can they improve the performance?

The will not affect performance, they will affect power consumption. As far as I know resistors may be used for designs critical to metastability, as you drive the only card and perform full initialization from scratch and without MCU output signal floating you do not need pull-ups.
And by the way, forgot to mention: SD-card may have internal pull-ups (most probably weak ones - at least on CS pin), which may be configurable. You may want to ensure disabling internal pull-ups.