I have noticed very strange behaviours when using scikit learn to train a svm classifier on a dataset that has an input dimension 24700 x 11200. Once split into train/test set, the training data part after has >18500 samples.

My code looks like below:

```
def feature_scale(M):
    scaler = MinMaxScaler(feature_range=(0, 1))
    return scaler.fit_transform(M)

M = feature_extraction(....) # method to create the 24700x11200 matrix
X_train_data, X_test_data, y_train, y_test = \
            train_test_split(M, self.raw_data['class'],
                             test_size=0.25,
                             random_state=42)
X_train_data=feature_scale(X_train_data) 
X_test_data =feature_scale(X_test_data) 
y_train = y_train.astype(int)
y_test = y_test.astype(int)

classifier = svm.LinearSVC(class_weight='balanced', C=0.01, penalty='l2', loss='squared_hinge',multi_class='ovr')

print("### test sfm...")
fs=SelectFromModel(LogisticRegression(class_weight='balanced', penalty="l1", C=0.01))       
X_=fs.fit_transform(X_train, y_train)
print(np.count_nonzero(X_)) # LINE C0, prints: 23828534
print(X_.shape)             # LINE C0, prints: (18587, 1282), the product is exactly 23828534
print("### end \n\n\n")

#classifier.fit(X_, y_train_data) #LINE D

print("### test kb...")
fs=SelectKBest(k=1000, score_func=f_classif)
X_=fs.fit_transform(X_train, y_train)
print(np.count_nonzero(X_))  #LINE E0, prints (18587, 1000)
print(X_.shape)              #LINE E1, prints 18587000
print("### end \n\n\n")

```

M is a sparse feature matrix. Firstly I load a CSV data where each line is a sentence and there are 24700 lines (samples). Then I call a series of NLP processes to process these lines to extract features such as n-gram, pos tags, and transform the raw data in to a 24700 x m feature matrix, where m=11200 in this case. The matrix is sparse.

Observations: **(UPDATED in response to the reply - thanks for this)**

As you can see the code tests two kinds of feature selection, one using SelectFromModel, using a **LogisticRegression** algorithm; the other uses SelectKBest, using **f\_classif** scoring function. I have tested both, to print the transformed feature matrix.

Both methods creates a **dense matrix where every element is filled**, the only difference is that SelectFromModel produces a matrix with 1282 features, while SelectKBest with 1000 features.

**My first question is**: why are these feature matrices fully populated? I expect them to be dense but not completely filled.

Then continue to further test the code, if I put line D back, i.e., apply svm on the SelectFromModel transformed feature matrix, the problem hangs, while both CPU and memory are 0 usage. An error is generated as below:

```
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

```

However, if Line D is put back but moved to after SelectKBest, i.e., if the svm classifier uses the feature matrix created by SelectKBest, it works alright, no errors whatsoever.

**So my second question is** what difference between the two transformed feature matrices can cause that error to SVM, and why does it cause it to hang? I have **32G memory in total.**

**UPDATE**: I noticed that the server I ran experiments on had no swap. I allocated a swap of 8G, and re-run the experiment, now notice that the swap is all used: total= 8191, used=8182, free=9. I wonder if this indicates that previously there was no enough memory, and for some reason after a while the system just hangs, even system stats shows CPU and memory usage as 0?

Thanks