Why Numpy and Pandas arrays consuming more memory than source data?
Memory consumption depends very much on the way data is stored. For example 1 as string takes only one byte, as an int it takes two bytes and eight bytes as double. Then there is the overhead of creating it as in Object of DaataFrame and Series. All this is done for efficient processing.
As a general rule of thumb data respresentation in memory will have larger size than in storage.
BigData means data which is too large to fit in the memory (or process in a single machine). So it makes no sense to parse the whole data and load it in memory.  
All BigData processing engines depend splitting the data into chunks and processing the chunks individually (and parallely), then combining these intermediate results into one.
I am new to bigdata, I want to parse the whole data, so I cant split it when i try to use numpy array for processing 1 GB data it takes 4GB memory (In real time I am dealing with huge data). Is there any optimized way to use these array for this much data or any special function to handle huge data.