What are alternatives to the random oracle model for modelling hash functions?
(I am interpreting this question in the following way due to the comments.)  In implementations of cryptographic systems, hash functions like the SHA-256 are defined by a deterministic algorithm mapping arbitrary-length bit-strings to some fixed-length bit-strings.  This situation is troublesome for formal analyses (reductions) of these systems because, when modeled in the straightforward way, there is no obvious way to "assume" some important security notions like collision-resistance for such functions.  To be explicit, we might take a function $H:\{0,1\}^* \rightarrow \{0,1\}^{256}$ and say that


  $H$ is collision-resistant if no probabilistic poly-time algorithm outputs distinct $x,y$ such that $H(x) = H(y)$.


Under this definition, no function $H$ is collision resistant:  By the pigeon-hole principle such an $x,y$ exists, and there also exists a constant-time algorithm that outputs them.  (We're leaving aside the additional problem that "probabilistic poly-time" is ill-defined here.)  Most modifications of the definition, like asking for long $x,y$ or something, are unsatisfying.  Let me mention also that some properties are (I believe) achievable, such as when the $x$ is chosen at random and the adversary has to find a $y$ that collides.

But if one models the hash as a keyed function that take an additional key input, then these security notions can be plausibly assumed to be achieved.  Thus we often give security reductions where a keyed hash function is used and random hash key is chosen in the model.  Though this is incongruous with reality, it seems fine.

To better address this disparity, Rogaway in suggested that we forget about the formal assumption per se and instead focusing on the existence of an explicit reduction to doing something with the hash function.  This is exploiting the fact that we apparently don't "know" of an algorithm finding collisions in SHA-256, even though there is a "constant-time" algorithm for doing exactly that.

By the way, the random oracle model generally addresses a different issue:  sometimes we don't have any way to reduce security to any reasonable property of our (keyed or keyless) hash function, so we instead analyze the system in a model where the evaluations of the hash are replaced by evaluations of a random oracle (which everyone can access).  Incidentally, doing this also makes the hash appear to be keyless and thus superficially "closer to reality," but this is in fact not the case:  Now the entire function has become the key, and indeed an exponentially long key.