handling large data on single computer listing creating customers in the database import mysqldb import pandas as pd user '****password '****database 'testmc mysqldb connect('localhost',user,password,databasecursor mc cursor(nr_customers colnames ["movie% % for in range( , )pd np random seed( generated_customers pd np random randint( , , nr_customersreshape(nr_customers, first we establish the connectionyou'll need to fill out your own usernamepasswordand schema-name (variable "database"next we simulate database with customers and create few observations data pd dataframe(generated_customerscolumns list(colnames)data to_sql('cust',mcflavor 'mysql'index trueif_exists 'replace'index_label 'cust_id'store the data inside pandas data frame and write the data frame in mysql table called "custif this table already existsreplace it we create customers and randomly assign whether they did or didn' see certain movieand we have movies in total the data is first created in pandas data frame but is then turned into sql code noteyou might run across warning when running this code the warning statesthe "mysqlflavor with dbapi connection is deprecated and will be removed in future versions mysql will be further supported with sqlalchemy engines feel free to already switch to sqlalchemy or another library we'll use sqlalchemy in other but used mysqldb here to broaden the examples to efficiently query our database later on we'll need additional data preparationincluding the following thingscreating bit strings the bit strings are compressed versions of the columnscontent ( and valuesfirst these binary values are concatenatedthen the resulting bit string is reinterpreted as number this might sound abstract now but will become clearer in the code defining hash functions the hash functions will in fact create the bit strings adding an index to the tableto quicken data retrieval creating bit strings now you make an intermediate table suited for queryingapply the hash functionsand represent the sequence of bits as decimal number finallyyou can place them in table firstyou need to create bit strings you need to convert the string " to binary or numeric value to make the hamming function work we opted for numeric representationas shown in the next listing