Seeking advice on database architecture -- given my problem, what tools should I learn?
I used R for several years but have since moved to Python and so I have a hard time understanding output.OLS's data hierarchy. Nonetheless, here are my thoughts.

In-Memory Databases: If you're struggling to fit your R object in memory, then I'm guessing the memory requirement is too great for your laptop (i.e., the problem isn't due a 32-bit installation of R). If that's the case, putting an in-memory db, like MongoDB, on your laptop will only transfer the memory problem to a new technology. In other words, 6 GB in R will be 6 GB in Mongo, so if your laptop simply can't handle 6 GB, then it doesn't matter how you put it there. You could set up a MongoDB on AWS, which could hold 100s of GB, but this can be expensive and also difficult if you don't have AWS experience. 

On the other hand, MongoDB is good at storing hierarchical data like lists and hash tables / dictionaries (I forgot what R's term for that is) and is fast if you have enough memory. 

Relational Databases (RDMS): I would recommend going this route and creating a wide and deep table. You can still get value out of an RDBMS even if your machine has limited memory because everything is efficiently stored on disk. If you create table indexes on the columns you expect to use/filter/select most, you can very quickly query the data that you want even if your data is multiple GB. If you don't create table indexes, your queries will be horribly slow, and you'll probably abandon the RDBMS effort entirely (so make sure to index). 

A long time ago my go-to package for accessing MySQL through R was RMySQL. MySQL has been around for many years, so you'll be able to find most answers to your problems on Goggle, but MySQL user interfaces like Sequel Pro (for Mac only) have sort of fallen behind on maintenance, so setting up tables and databases can be a little more challenging than they used to be. You'll now have to go the command-line route. 

Postgres has now replaced MySQL as the open-source DB of choice, but it lacks good user-interface support and so you'll probably have to set everything up via the command line. I started using Postgres after I stopped using R, so I never used a Postgres R package, but it looks like there is one here. 

Whether you go with MySQL or Postgres, you'll probably have to use the command-line outside of R to set up the database, but after that creating, dropping, joining, or querying tables can all be done through R. RDBMS systems are also great because you can guarantee data quality (no duplicates) by creating primary keys -- I found this feature very helpful for my R modeling. In any case, SQL is a very basic language that every data-oriented person should know so setting up an RDBMS and possibly learning SQL are skills worth learning.