Delayed VCC rise time with Parallax Smart Card Adapter
Rise time is determined by how much current flows into the capacitor. T = CV/I, so time increases when current is reduced. Anything that limits the charging current or bypasses it away from the capacitor will increase the rise time. To calculate the rise time you need to know the internal resistance and current limit of your power supply, and how much current any other circuitry might steal.     

In your circuit the pullup resistors should normally be open circuit, but they could be switched to either Gnd or Vcc. If for some reason they were connected to Gnd then they could steal current and increase rise time. However the current they would draw is rather small (0.6mA at 3V), so even if CLK and IO were both pulled low it shouldn't have a significant effect. 

A rise time of 450us without card inserted suggests that your power supply is current limited to about 67mA. According to ISO7816 the maximum current a card can draw under normal operating conditions is 200mA. Your power supply should be able to supply at least that much, and the rise time should 150uS or less.  

When a card it inserted it will also take some current from Vcc, which will further increase rise time if power supply current is limited. Since the card contains an MCU with complex circuitry, its current draw could vary non-linearly or even "randomly", so you should not rely on it being consistent. Using an output pin on your microcontroller to supply power directly is risky, even if it seems to work.