Title: Using caching to optimize a timeline in Rails
Tags: ruby-on-rails;caching;memcached;redis
Question: I'm hoping to get advice on the proper use of caching to speed up a timeline query in Rails. Here's the background:

I'm developing an iPhone app with a Rails backend. It's a social app, and like other social apps, its primary view is a timeline (i.e., newsfeed) of messages. This works very much like Twitter, where the timeline is made up of messages of the user and of his/her followers. The main query in the API request to retrieve the timeline is the following:

```@messages = Message.where("user_id in (?) OR user_id = ?", current_user.followed_users.map(&amp;:id), current_user)
```

Now this query gets quite inefficient, particularly at scale, so I'm looking into caching. Here are the two things I'm planning to do:

1) Use Redis to cache timelines as lists of message ids

Part of what makes this query so expensive is figuring out which messages to display on-the-fly. My plan here is to keep create a Redis list of message ids for each user. Assuming I build this correctly when a Timeline API request comes in I can call Redis to get a pre-processed ordered list of the ids of the messages to display. For example, I might get something like this: "[21, 18, 15, 14, 8, 5]"

2) Use Memcached to cache individual message objects

While I believe the first point will help a great deal, there's still the potential problem of retrieving the individual message objects from the database. The message objects can get quite big. With them, I return related objects like comments, likes, the user, etc. Ideally, I would cache these individual message objects as well. This is where I'm confused.

Without caching, I would simply make a query call like this to retrieve the message objects:

```@messages = Message.where("id in (?)", ids_from_redis)
```

Then I would return the timeline:

```respond_with(:messages =&gt; @messages.as_json) # includes related likes, comments, user, etc.
```

Now given my desire to utilize Memcache to retrieve individual message objects, it seems like I need to retrieve the messages one at a time. Using psuedo-code I'm thinking something like this:

```ids_from_redis.each do |m|
  message = Rails.cache.fetch("message_#{m}") do
     Message.find(m).as_json
  end
  @messages << message
end
```

Here are my two specific questions (sorry for the lengthy build): 

1) Does this approach generally make sense (redis for lists, memcached for objects)? 

2) Specifically, on the pseudo-code below, is this the only way to do this? It feels inefficient grabbing the messages one-by-one but I'm not sure how else to do it given my intention to do object-level caching.

Appreciate any feedback as this is my first time attempting something like this.



Here is another answer: On the face of it, this seems reasonable. Redis is well suited to storing lists etc, can be made persistent etc, and memcached will be very fast for retrieving individual messages, even if you call it sequentially like that.

The issue here is that you're going to need to clear/supplement that redis cache each time a message is posted. It seems a bit of a waste just to clear the cache in this circumstance, because you'll already have gone to the trouble of identifying every recipient of the message.

So, without wishing to answer the wrong question, have you thought about 'rendering' the visibility of messages into the database (or redis, for that matter) when each message is posted? Something like this:

```class Message
  belongs_to :sender
  has_many   :visibilities

  before_create :render_visibility
    sender.followers.each do |follower|
      visibilities.build(:user =&gt; follower)
    end
  def 
end
```

You could then render the list of messages quite simply:

```class User
  has_many :visibilities
  has_many :messages, :through =&gt; :visibilities
end

# in your timeline view:
<%= current_user.messages.each { |message| render message } %&gt;
```

I would then add of individual messages like this:

```# In your message partial, caching individual rendered messages:
<%= cache(message) do %&gt;
  <!-- render your message here --&gt;
<% end %&gt;
```

I would also then add caching of entire timelines like this:

```# In your timeline view
<%= cache("timeline-for-#{current_user}-#{current_user.messages.last.cache_key}") do %&gt;
  <%= current_user.messages.each { |message| render message } %&gt;
<% end %&gt;
```

What this should achieve (I've not tested it) is that the entire timeline HTML will be cached until a new message is posted. When that happens, the timeline will be re-rendered, but all the individual messages will come from the cache rather than being rendered again (with the possible exception of any new ones that haven't been viewed by anyone else!)

Note that this assumes that the message rendering is the same for every user. If it isn't, you'll need to cache the messages per user too, which would be a bit of a shame, so try not to do this if you can!

FWIW, I believe this is vaguely (and I mean vaguely) what twitter do. They have a 'big data' approach to it though, where the tweets are exploded and inserted into follower timelines across a large cluster of machines. What I've described here will struggle to scale in a write-heavy environment with lots of followers, although you could improve this somewhat by using resque or similar.

P.S. I've been a bit lazy with the code here - you should look to refactor this to move e.g. the timeline cache key generation into a helper and/or the person model.

Comment for this answer: @pejmanjohn I don't think sorting would need to be done on the fly, but one implication that I definately should have highlighted is that this approach changes the semantics of your network. Users will only see messages from the point where they follow them. With your method, they'd see earlier ones also.
Comment for this answer: @pejmanjohn r.e. your latter point on your original approach - there's potential there, but you'd still need to identify all the followers for a given message in order to add it to the lists. That's what I meant when I said 'or redis for that matter' - you could do the above just using redis, and that might well be faster :)
Comment for this answer: Thanks for this. I follow your method, but am still wrapping my brain around its implications. From what I gather, one potential drawback here is that sorting would need to be done on-the-fly (given that following/unfollowing can happen at anytime). With regards to my original approach, if I were to ensure persistence and efficiently inserted into lists, rather than clearing/recreating, perhaps I avoid the issues you raise?
Comment for this answer: Also I should note, like you raise, I do plan to use a background worker for fanning out the message

{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 0, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2013:00:01/08/13', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<ruby-on-rails><caching><memcached><redis>', 'Title': 'Using caching to optimize a timeline in Rails', 'ViewCount': 701, 'id': 14206376, 'user_AboutMe': '', 'user_AccountId': 1404639.0, 'user_DisplayName': 'pejmanjohn', 'user_Reputation': 1027.0, 'user_Views': 74.0, 'user_id': 1332687.0}}