Importing Archived TweetStream Twitter Ouput into Mongodb?
Question: I have around 1000 lines of twitter data captured using python tweetstream.
The data was collected using the simple tweetstream example of:

    
    
    >>> stream = tweetstream.SampleStream("username", "password")
    >>> for tweet in stream:
    ...     print tweet
    

which outputs like:

    
    
    {u'user': {u'follow_request_sent': None,
    u'profile_use_background_image': True,
    u'profile_background_image_url_https': u'https://si0.twimg.com/
    profile_background_images/
    181013334/25957_1367646636642_1395984493_31038644_61586_n.jpg',
    u'verified': False, u'profile_image_url_https': u'https://
    si0.twimg.com/profile_images/1820265868/rosajennifer_normal.jpg',
    u'profile_sidebar_fill_color': u'DDEEF6', u'id': 46478005,
    u'profile_text_color': u'333333', u'followers_count': 505,
    u'protected': False, u'location': u'', u'default_profile_image':
    False, u'listed_count': 4, u'utc_offset': -21600, u'statuses_count':
    35923, u'description': u'Take me as I am or watch me as I go. . .\n
    \n', u'friends_count': 315, u'profile_link_color': u'0084B4',
    u'profile_image_url': u'http://a1.twimg.com/profile_images/1820265868/
    rosajennifer_normal.jpg', u'notifications': None,
    u'show_all_inline_media': True, u'geo_enabled': False,
    u'profile_background_color': u'C0DEED', u'id_str': u'46478005',
    u'profile_background_image_url': u'http://a2.twimg.com/
    profile_background_images/
    181013334/25957_1367646636642_1395984493_31038644_61586_n.jpg',
    u'name': u'rosa jennifer', u'lang': u'en', u'following': None,
    u'profile_background_tile': True, u'favourites_count': 82,
    u'screen_name': u'rosajennifer', u'url': u'http://www.facebook.com/
    profile.php?ref=profile&id=1329240058', u'created_at': u'Thu Jun 11
    20:11:28 +0000 2009', u'contributors_enabled': False, u'time_zone':
    u'Central Time (US & Canada)', u'profile_sidebar_border_color':
    u'C0DEED', u'default_profile': False, u'is_translator': False},
    u'favorited': False, u'contributors': None, u'entities':
    {u'user_mentions': [{u'indices': [1, 14], u'id': 90939650, u'id_str':
    u'90939650', u'name': u'Dajuan(Dae-John)', u'screen_name':
    u'Juan_Ton5oup'}], u'hashtags': [], u'urls': []}, u'text':
    u'\u201c@Juan_Ton5oup: Spanish girls love jeans with animals outlined
    on the back pockets.\u201dfoh lmao', u'created_at': u'Tue Feb 14
    00:27:32 +0000 2012', u'truncated': False, u'retweeted': False,
    u'in_reply_to_status_id': None, u'coordinates': None, u'id':
    169216166617817088, u'source': u'<a href="http://twitter.com/#!/
    download/ipad" rel="nofollow">Twitter for iPad</a>',
    u'in_reply_to_status_id_str': None, u'in_reply_to_screen_name': None,
    u'id_str': u'169216166617817088', u'place': None, u'retweet_count': 0,
    u'geo': None, u'in_reply_to_user_id_str': None,
    u'in_reply_to_user_id': None}
    

I have a single file of ~1000 of these, each on a seperate line. I've tried
mongoimport as well as a dozen other methods but I can't seem to get this data
imported. Mongoimport passes back this error:

    
    
    Sat Mar 10 12:51:00 Assertion: 10340:Failure parsing JSON string near:
    u'user': {
    0x581762 0x528994 0xaa29f3 0xaa4ca3 0xa9b7dd 0xa9f772 0x34df82169d
    0x4fe679
     mongoimport(_ZN5mongo11msgassertedEiPKc+0x112) [0x581762]
     mongoimport(_ZN5mongo8fromjsonEPKcPi+0x444) [0x528994]
     mongoimport(_ZN6Import8parseRowEPSiRN5mongo7BSONObjERi+0x8b3)
    [0xaa29f3]
     mongoimport(_ZN6Import3runEv+0x16e3) [0xaa4ca3]
     mongoimport(_ZN5mongo4Tool4mainEiPPc+0x169d) [0xa9b7dd]
     mongoimport(main+0x32) [0xa9f772]
     /lib64/libc.so.6(__libc_start_main+0xed) [0x34df82169d]
     mongoimport(__gxx_personality_v0+0x3d1) [0x4fe679]
    exception:Failure parsing JSON string near: u'user': {
    

I assume this is because the string is not actual json, it's some sort of
(json like) format.

Can anyone help?


Answer: The first problem as you noted, the following is not valid JSON, it's a python
dictionary: `{u'indices':`.

Second problem, why are you trying to use `mongoimport`? In python you can
just save the dictionary to the database. This is basically the first example
of how to use MongoDB.

    
    
    >>> from pymongo import Connection
    >>> connection = Connection('localhost', 27017)
    >>> db = connection.test_database
    >>> posts = db.posts
    >>> stream = tweetstream.SampleStream("username", "password")
    >>> for tweet in stream:
    ...     posts.insert(post)
    

