Google Now time estimation to work
How long have you used the feature? On my GNex, it took roughly 2 work weeks for the feature to adapt to my work hours, which are 8:00-16:00, so not that complicated. If you're working in shifts or by some other schedule, it might take longer for the feature to learn when to show which destination.