if potentially confusing.
you have been taught incorrectly - you can use "could" when speaking about a past possibility, you just need to know how to use it correctly.
he could have gone there yesterday, but he did not.
the problem with your version "he could go there yesterday..." is not the use of "could", but the tense of your verb to go. if you are speaking about the past even as a possibility then the verb should be in the past tense.
he wanted to go there yesterday, but he could not.
because the verb "wanted" is in the past tense it is correct to use "go" because that is what he wanted at 