Can I create a random access CSPRNG from a hash? Can I use a keyed hash of a counter as a secure CSPRNG?
E.g. 4th random block is Hash(4, seed)

If not, what are the possible attacks?
There is a name for "random access PRG" : it's called a pseudorandom function (PRF). Any block cipher is a pseudorandom function (that's more-or-less the definition of what it means to be a secure block cipher), so you can use, say, AES directly. If all you have is a hash function, then the HMAC construction also gives you a PRF.