best way to parse a big (500mb) json file and insert into mongoDB
Question: > **Possible Duplicate:**  
>  [Importing large json file into
> mongodb](http://stackoverflow.com/questions/5126718/importing-large-json-
> file-into-mongodb)

I have a fairly big JSON file (500+MB) that'd I'd like to edit and insert into
a database in MongoDB. I tried using a python script but it crashes. what
would be the best way to do this?


Answer: First of all, it is important to note that Mongo has a maximum document size
of 16MB, explained here: <http://www.mongodb.org/display/DOCS/Documents> This
is why Mike Christensen was asking if the data inside your file was one giant
object, or if it is split into multiple documents.

If you are completely new to MongoDB, it will help to familiarize yourself
with the way information is stored in a Mongo Database. In a nutshell, a
Database is made of Collections, which are made up of JSON-like objects called
Documents, which contain keys and values.

This is explained in the "Mongo data model" section of the Introduction page:
<http://www.mongodb.org/display/DOCS/Introduction>

Here are some other good resources for getting started with MongoDB:

<http://www.mongodb.org/display/DOCS/Inserting> This gives an explanation of
Mongo Documents, what they look like, and how they are stored in a Collection.

<http://www.mongodb.org/display/DOCS/Tutorial> This is an introductory
document which contains notes on how to get up and running with Mongo, as well
as notes on schema design.

<http://try.mongodb.org/> This is an interactive tutorial, which runs right in
your web browser, and will give you a better sense of how documents are stored
in a Mongo Collection.

<http://mongly.com/> This is a website created by a very active member of the
MongoDB community named Karl Seguin. It includes many resources that are
perfect for getting started with Mongo, including "The Little MongoDB Book"
and a more in-depth interactive tutorial.

<http://www.mongodb.org/display/DOCS/Python+Language+Center> Because you
mentioned working with Python, here is the link to the Python Language center.
The official Python driver for MongoDB is called PyMongo.

<http://api.mongodb.org/python/current/> This is the official documentation
for the PyMongo Driver.

Forgive me if you have already discovered the above resources on your own.
This response is also for the benefit of any other readers who may be new to
MongoDB.

Now to address your actual question of how to import the information inside
your JSON file.

There is a utility called mongoimport that will take a single file that
contains one JSON string per line and insert it into a Mongo collection. The
documentation on this may be found here:

<http://www.mongodb.org/display/DOCS/Import+Export+Tools>

The caveat is that the code for mongoimport is not terribly complex, and you
may find that it does not create documents in exactly the structure that you
are looking for. When importing data into a new Mongo collection from another
source (such as a JSON file) the official recommendation is to write a custom
script, as you first attempted to do.

What, exactly was the error that you received from your Python script? Was it
an error that the document size was greater than 16MB and could not be
inserted into a collection, or did Python itself choke trying to open your
500MB file? If the answer is the former, you will have to modify your script
to split up your giant document(s) into smaller ones. If the answer is the
latter, we might have a more difficult issue to deal with. If the error was
something else entirely, please post it and we can work through it.

Hopefully the resources I have provided will give you a better understanding
of how documents are stored in a Mongo Collection, and what your original JSON
file should look like before it can be inserted into a Mongo Database.

Finally, thanks for trying Mongo! Welcome to the Community! If you have any
follow-up questions, we are here to help!

