spidering twitter using database in this sectionwe will create simple spidering program that will go through twitter accounts and build database of them notebe very careful when running this program you do not want to pull too much data or run the program for too long and end up having your twitter access shut off one of the problems of any kind of spidering program is that it needs to be able to be stopped and restarted many times and you do not want to lose the data that you have retrieved so far you don' want to always restart your data retrieval at the very beginning so we want to store data as we retrieve it so our program can start back up and pick up where it left off we will start by retrieving one person' twitter friends and their statuseslooping through the list of friendsand adding each of the friends to database to be retrieved in the future after we process one person' twitter friendswe check in our database and retrieve one of the friends of the friend we do this over and overpicking an "unvisitedpersonretrieving their friend listand adding friends we have not seen to our list for future visit we also track how many times we have seen particular friend in the database to get some sense of their "popularityby storing our list of known accounts and whether we have retrieved the account or notand how popular the account is in database on the disk of the computerwe can stop and restart our program as many times as we like this program is bit complex it is based on the code from the exercise earlier in the book that uses the twitter api here is the source code for our twitter spidering applicationfrom urllib request import urlopen import urllib error import twurl import json import sqlite import ssl twitter_url 'conn sqlite connect('spider sqlite'cur conn cursor(cur execute(''create table if not exists twitter (name textretrieved integerfriends integer)'''ignore ssl certificate errors ctx ssl create_default_context(ctx check_hostname false ctx verify_mode ssl cert_none