Implementing a basic graph database engine
Question: I need to implement a simple graph database engine, what are the things should
I consider? First, I am confused between which data structure to use, I mean
graph representation (like adjacency matrix or adjacency list) or the actual
graph itself? I need this to be scalable. Later how do I store the graph in
the hard disk as files? After I store the graph data in the form of files, I
would also need a way to selectively load only certain files into the graph,
since I can not load everything at once into the RAM. Sorry for being vague,
but I need someone to point me in the right direction. Also please suggest the
language I can use, can I use python for this project? Thank you.


Answer: Depending on your needs you will implement different interface to the database
ie. an adjacency matrix or the graph itself.

Instead of using a file based database, the important step forward you can
take is use a key/value store like
[bsddb](http://stackoverflow.com/questions/tagged/bsddb),
[leveldb](http://stackoverflow.com/questions/tagged/leveldb) or
[wiredtiger](http://stackoverflow.com/questions/tagged/wiredtiger) (prefered).
This will deal with caching often accessed files, provide ACID semantic, and
indices if you use wiredtiger.

The storage layer made upon the key/value store, can have several layout. It
depends on the final interface you need.

To get started with developing custom databases using key/value stores I
recommend you read questions answered about mostly leveldb and bsddb on SO.

Like the following:

  * [store list in key value database](http://stackoverflow.com/questions/18513419/store-list-in-key-value-database)
  * [How to give multiple values to a single key using a dictionary?](http://stackoverflow.com/questions/19873620/how-to-give-multiple-values-to-a-single-key-using-a-dictionary/32147466#32147466)
  * [Use integer keys in Berkeley DB with python (using bsddb3)](http://stackoverflow.com/questions/18664940/use-integer-keys-in-berkeley-db-with-python-using-bsddb3/32147596#32147596)
  * [Expressing multiple columns in berkeley db in python?](http://stackoverflow.com/questions/2399643/expressing-multiple-columns-in-berkeley-db-in-python)

