Machine Learning -Issues with big dataset
you can segment your data on few models which output will be input to the next model which will give you result.
Basically its RNN architecture.
Put such massive data in one network just not possible due to memory limitation.
I am trying to apply Machine Learning to a Kaggle.com dataset. The dimension of my dataset is 244768 x 34756. Now at this size none of the scikit algorithms work.
I thought i would apply PCA , but even that doesnt scale up to this dataset. 
Is there anyway i can reduce redundant data from my training dataset? I can reduce dimension by applying PCA , but if i could apply PCA.
Since i am doing document classification , i resampled my dataset to 244768*5672 , by reducing the word vector size. PCA cant be applied to even this dataset.
Can i apply PCA by doing this method. Suppose my matrix is A - 
X = A.T * A
pca(X) (X becomes 5672 x 5672 matrix)
Will this give me wrong answers?
Also when i apply Logistic regression , can i train the model incrementally , .ie
If A = 10000 x 500
Can i take 1000 x 500 , to logistic.fit(A) and then do the same for other rows? Is this kind of training wrong?