When I log into App Store on my iMac I get a notification on my iPhone saying someone's trying to log in. But the location is wrong
As your question states, you received the notification on your iPhone instantly after logging into your account via your iMac. Since you know you just logged in, and then instantly got the notification, the chances this was just a coincidence and someone bypassed your login at the same time, is extremely unlikely. 

Much more likely is that the location data was off. There are many factors that can be used to determine someone's location and the accuracy of location data can be affected by all of these.

For example, if your iMac is connecting through a VPN then the location it displays may be totally different to where it actually is, even to the extent it appears to be in another country/continent/hemisphere. This is only one example, as I'm sure you would know if you are connecting via a VPN.

Other factors affecting location include WiFi networks (SSIDs in particular), the method of your internet connection, IP addresses, core location constants, and the list goes on. 

Sometimes, depending on the configuration of your router, the DNS servers being used can change and this too can affect accuracy. 

So, in a nutshell, you need not worry. However, it is good practice to regularly change your account password, so perhaps this is a good reminder to do just that! :)