Is there a way to replace existing values with NaN I'm experimenting with the algorithms in iPython Notebooks and would like to know if I can replace the existing values in a dataset with Nan (about 50% or more) at random positions with each column having different proportions of Nan values.

I'm using the Iris dataset for this experimentation to see how the algorithms work and which one works the best. The link for the dataset is here.

Thanks in advance for the help.
Randomly replace values in a numpy array

# The dataset
data = pd.read_csv('iris.data')
mat = data.iloc[:,:4].as_matrix()


Set the number of values to replace. For example 20%:

# Edit: changed len(mat) for mat.size
prop = int(mat.size * 0.2)


Randomly choose indices of the numpy array:

i = [random.choice(range(mat.shape[0])) for _ in range(prop)]
j = [random.choice(range(mat.shape[1])) for _ in range(prop)]


Change values with NaN

mat[i,j] = np.NaN


Dropout for any array dimension

Another way to do that with an array of more than 2 dimensions would be to use the numpy.put() function:

import numpy as np
import random
from sklearn import datasets

data = datasets.load_iris()['data']

def dropout(a, percent):
    # create a copy
    mat = a.copy()
    # number of values to replace
    prop = int(mat.size * percent)
    # indices to mask
    mask = random.sample(range(mat.size), prop)
    # replace with NaN
    np.put(mat, mask, [np.NaN]*len(mask))
    return mat


This function returns a modified array:

modified = dropout(data, 0.2)


We can verify that the correct number of values have been modified:

np.sum(np.isnan(modified))/float(data.size)


[out]:

0.2