Postgres caching and bytea
I think the first question is how much RAM the system has.  If you have enough that the database fits comfortably in RAM then this is not a major problem.  Basically the OS disk cache is very fast, and if the db is sitting there effectively, then you don't have anything to worry about.  If you are on a system with, say, 4GB RAM and not a whole lot else running, I wouldn't worry.

The problem happens when this is not true, and then the shared buffers try to hold the most frequently used information.  I would expect that caching would be ok there too. What you have to remember is that the PostgreSQL cache is slower than the OS cache because it provides features aimed at keeping frequently used data in the cache.

Of course all of the above depends highly on your query patterns.