Best open source DB technology to store and query large, sparse matrices I have a large vector space model, in which each point is a multi-variate observation. In other words, I have a large number of pretty large records, mostly of ints and floats. I started with in-memory matrices, but given a large number of points (1B+) and a large number of dimensions (1K+), I soon realized that that wasn't an option.

I need to

1) make the whole thing persistent and dynamic (I want to add/update/remove points/dimensions easily)

2) query the VSM efficiently, performing similarity queries (given a point, find top K similar points) and range queries (find all points within a subspace).

Based on some reading, it seems that there isn't an off-the-shelf package to do this. What would you use to implement this kind of scenario? Dumping each point as a row in a relational db didn't work because the large number of dimensions. Even after normalization, the relational db was clearly too slow to perform similarity queries.

Moreover, I need to compute the dot product on chunks of the matrix.
What technology would you recommend in this case? For example, I have read about array databases, and OLAP cubes, but I would like to have some advice from real DBAs.

Thanks, Mulone
You should take a look at SciDB - a database designed for precisely this use case. It has as as its major driver Michael Stonebraker, a man who knows a lot about databases - it would be an exaggeration to say that he's been involved with more database systems than most people have had hot dinners, but not by much :-)!


  "including Ingres, Illustra, Cohera, StreamBase Systems, Vertica,
  VoltDB, and Paradigm4. He was previously the Chief Technical Officer
  (CTO) of Informix. He is also an editor for the book Readings in
  Database Systems".


Google "michael stonebraker scidb sparse matrix" and subsets thereof and I think that it might be a good fit for what you want.