Botulisum and Food safety
I would chunk it, better to waste it than risk illness. Fresh garlic and olive oil left at room temperature for a significant amount of time (not like the 10 minutes it might sit during prep for addition to soup, pasta, etc.) makes a prime breeding ground for botulism.

Potentially dangerous things like this shouldn't sit out longer than 2 hours at room temperature. Three days in the fridge would be fine if it were safe when it went in, but in this case it wasn't.

For further information:

Brief write up on harvesting and storing garlic from UC Davis

FDA sheet on the temperature danger zone and two hour rule

Flavored vinegars and oils, article from Colorado State Extension