handling large data on single computer and incorporate best practices and state-of-the art technologies spend your time on getting things donenot on reinventing and repeating others people' effortsunless it' for the sake of understanding how things work then you must consider your hardware limitation get the most out of your hardware resources on computer can be idlewhereas other resources are over-utilized this slows down programs and can even make them fail sometimes it' possible (and necessaryto shift the workload from an overtaxed resource to an underutilized resource using the following techniquesfeed the cpu compressed data simple trick to avoid cpu starvation is to feed the cpu compressed data instead of the inflated (rawdata this will shift more work from the hard disk to the cpuwhich is exactly what you want to dobecause hard disk can' follow the cpu in most modern computer architectures make use of the gpu sometimes your cpu and not your memory is the bottleneck if your computations are parallelizableyou can benefit from switching to the gpu this has much higher throughput for computations than cpu the gpu is enormously efficient in parallelizable jobs but has less cache than the cpu but it' pointless to switch to the gpu when your hard disk is the problem several python packagessuch as theano and numbaprowill use the gpu without much programming effort if this doesn' sufficeyou can use cuda (compute unified device architecturepackage such as pycuda it' also well-known trick in bitcoin miningif you're interested in creating your own money use multiple threads it' still possible to parallelize computations on your cpu you can achieve this with normal python threads reduce your computing needs "working smart hard achievement this also applies to the programs you write the best way to avoid having large data problems is by removing as much of the work as possible up front and letting the computer work only on the part that can' be skipped the following list contains methods to help you achieve thisprofile your code and remediate slow pieces of code not every piece of your code needs to be optimizeduse profiler to detect slow parts inside your program and remediate these parts use compiled code whenever possiblecertainly when loops are involved whenever possible use functions from packages that are optimized for numerical computations instead of implementing everything yourself the code in these packages is often highly optimized and compiled otherwisecompile the code yourself if you can' use an existing packageuse either just-in-time compiler or implement the slowest parts of your code in