while computers keep getting faster and have more memorythe size of the data has grown as well in factdata has grown faster than computational speedand this means that it has grown faster than our ability to process it it is not easy to say what is big data and what is notso we will adopt an operational definitionwhen data is so large that it becomes too cumbersome to work withwe refer to it as big data in some areasthis might mean petabytes of data or trillions of transactionsdata that will not fit into single hard drive in other casesit may be one hundred times smallerbut just difficult to work with we will first build upon some of the experience of the previous and work with what we can call the medium data setting (not quite big databut not small eitherfor this we will use package called jugwhich allows us to do the followingbreak up your pipeline into tasks cache (memoizeintermediate results make use of multiple coresincluding multiple computers on grid the next step is to move to true "big data"and we will see how to use the cloud (in particularthe amazon web services infrastructurewe will now use another python packagestarclusterto manage clusters learning about big data the expression "big datadoes not mean specific amount of dataneither in the number of examples nor in the number of gigabytesterabytesor petabytes taken up by the data it means the followingwe have had data growing faster than the processing power some of the methods and techniques that worked well in the past now need to be redoneas they do not scale well