title: Conceptualizing a Redis graph?
body: <p>I'm fairly new to Redis and I love it so far. I had an idea about implementing a graph in Redis, looked it up, and sure enough, it's been done before, but I have a couple questions about it.</p>
  
  <p>Twitter used FlockDB with Redis <a href="http://blog.gopivotal.com/pivotal/case-studies-2/case-study-staple-yourself-to-a-tweet-to-understand-30-billion-redis-updates-per-day" rel="nofollow">(source)</a> to relate their keys and values with tweets, followers, etc. Pinterest uses Redis lists for each user(their followers, pins, etc) to load the requested data <a href="http://blog.gopivotal.com/pivotal/case-studies-2/using-redis-at-pinterest-for-billions-of-relationships" rel="nofollow">(source)</a> by going through lists and grabbing the requested keys.</p>
  
  <p>I'm very interested in using Redis the way Pinterest does. I found a code explanation <a href="http://www.rediscookbook.org/implement_a_social_graph.html" rel="nofollow">here</a>.</p>
  
  <p>Here's what I can't figure out:</p>
  
  <p>All of this data (follower lists, pin list, etc) needs to be stored in a second database that can persist the data. Wouldn't it take a lot of work/maintenance to constantly keep Redis' list data and the other database's data in sync? Is there an easy way to do this?</p>
  
  <p>Wouldn't using Redis lists for every user like Pinterest does, get very memory intensive compared to a solution like FlockDB?</p>
  
  <p>I apologize if I misconstrued any of the source information. It would be helpful if someone shared some insight on this concept. Thanks in advance.</p>
  
  <p>PS: My applications relationships and feeds are social-network-esque. Not totally unlike the aforementioned Twitter/Pinterest.</p>
  
  <p><strong>Update:</strong> </p>
  
  <p>I'm not sure this question will ever be answered, but I'll update it with information I've learned. I've configured a Cassandra cluster write all of the physical objects in tables and the Redis cluster to store highly trafficked lists (data relationships) that would otherwise be their own join table in SQL or CQL (many-many relationship tables like <code>conversation_users</code>). This works extremely well because <strong>A.</strong> Cassandra isn't very good with secondary indexes and cannot look up multiple secondary-indexed values in a single query, so with the Redis set, I can essentially hand Cassandra a list of the needed primary IDs for it to read, <strong>B.</strong> Cassandra is best used as a writing > reading tool, so Redis takes a huge load off of Cassandra by giving it the necessary primary IDs, nearly eliminating multiple queries for a single set of information, <strong>C.</strong> (Again with the secondary indexes) Think about the performance of scanning an entire table for values vs grabbing a simple key and its set values. <strong>D.</strong> (the maintenance) It's possible, and I'm in the process of building background scripts that serialize nearly-expired-from-low-usage Redis sets into Cassandra columns (again using Redis for it's strength: memory, and Cassandra for its strength: logging), then later writing those serialized sets back into Redis when they are requested (post-expiration).</p>
  
  <p>I still would like to know which information to cache in Redis and what to keep in Cassandra. I can't stick all data relationships in Redis (though I'd like to) because my potential social graph is already enormous. For instance, should I keep actual recently-read/wrote objects in Redis along with feeds with those object IDs? This would take up a lot of RAM, but it lowers reads from Cassandra for newer objects. Or should I keep mostly relationships in Redis (followers, follow feed, blocked_users, liked posts, etc)? I like the second route better, but it means more querying of Cassandra for the actual objects, which I'd like to eliminate without paying a million dollars a month in RAM. What objects/relationships should I elect to keep in Redis? I learn well from code examples, so if you happen to find anything similar, I'd be appreciative. Please give me your thoughts, if you have any ideas! Thank you.</p>