text mining and text analytics posted new things in fact we can run the api request periodically and gather data over time while at any given time you're limited to thousand postsnothing stops you from growing your own database over the course of months it' worth noting the following script might take about an hour to complete if you don' feel like waitingfeel free to proceed and use the downloadable sqlite file alsoif you run it now you are not likely to get the exact same output as when it was first run to create the output shown in this let' look at our data retrieval functionas shown in the following listing listing reddit data retrieval and storage in sqlite specific fields of the topic are appended to the list we only use the title and text throughout the exercise but the topic id would be useful for building your own (biggerdatabase of topics insert all topics into sqlite database from subredditsget hottest , (in our casetopics def prawgetdata(limit,subredditname)topics get_subreddit(subredditnameget_hot(limit=limitcommentinsert [topicinsert [topicnbr this part is an informative for topic in topicsprint and not necessary if (float(topicnbr)/limit)* in xrange( , )for code to work it only print '**********topic:str(topic idinforms you about the *********completestr((float(topicnbr)/limit)* download progress ****topicnbr + trytopicinsert append((topic title,topic selftext,topic idsubredditname)exceptpass tryappend comments for comment in topic commentsto list these are commentinsert append((comment body,comment idnot used in the topic title,topic selftext,topic id,subredditname)exercise but now exceptyou have them for experimentation pass print '********************************print 'inserting data into sqlitec executemany('insert into topics values (?,?,?,?)'topicinsertprint 'inserted topicsc executemany('insert into comments values (?,?,?,?,?,?)'commentinsertprint 'inserted commentsconn commit(insert all for subject in subredditsprawgetdata(limit=limit,subredditname=subjectcommit changes (data insertionsto database without the commitno data will be inserted comments into sqlite database the function is executed for all subreddits we specified earlier