we begin by loading the dataset and looking at some statisticsfrom collections import defaultdict from itertools import chain file format is line per transaction of the form ' dataset [[int(tokfor tok in ,line strip(split()for line in open('retail dat')count how often each product was purchasedcounts defaultdict(intfor elem in chain(*dataset)counts[elem+ we can plot small histogram as followsof times bought of products just once twice or thrice four to seven times eight to times to times to times to times to times times or more there are many products that have only been bought few times for example percent of products were bought four or less times howeverthis represents only percent of purchases this phenomenon that many products are only purchased small number of times is sometimes labeled "the long tail"and has only become more prominent as the internet made it cheaper to stock and sell niche items in order to be able to provide recommendations for these productswe would need lot more data there are few open source implementations of basket analysis algorithms out therebut none that are well-integrated with scikit-learn or any of the other packages we have been using thereforewe are going to implement one classic algorithm ourselves this algorithm is called the apriori algorithmand it is bit old (it was published in by rakesh agrawal and ramakrishnan srikant)but it still works (algorithmsof coursenever stop workingthey just get superceded by better ideas