How to run second disconnect/load center from meter
The issue with 3/4 wires isn't about ampacity.  It's about grounding electrode systems and 

Where's your main panel?

Electrical service is 3-wire from the pole to the main panel, and it operates under service rules.  After the main panel, power must be carried 4-wire and it operates under the feeder rules.  

Right now it's arguable (and possibly grandfathered) that the disconnect and  your run from garage to house is all part of the service. To follow this logic through, the house has the one main panel and thus the one official neutral-ground bond. The cable run benefits from the 310.16b7 uprate, meaning the wire only needs to be rated for 83% of ampacity, or 83A not 100A.  

If you add the garage panel as you mean to, with both disconnects right next to each other, the house run can no longer be a service.  It moves the "main panel" point to the garage.   Now the house panel is a subpanel. It will need separation of neutral and ground in the panel and all the way back to the garage.  The connection is feeder not service, and ground must be present. That means the permissive 83% uprate goes away and the wire must be rated for full 100A, so it may be now too small. 

The upside is that ground can be retrofitted separately, it doesn't require total replacement of the cable.  Ground does not need to be bundled with the conductors because it does not flow current except in emergencies.  It can follow any practical route.