title: Python - Unintended replacement of original data
body: <p>I am currently trying to run a program that takes a 2D array and filters out Nan values and high-amplitude noise. </p>
  
  <p>The program does this by deconstructing the 2D array first into row slices and then by index of each row, replacing Nan values and values that are greater than the 1st standard deviation of the residuals of the data, with values from an array of a polynomial fit to the data of equal size. the following block of code is the main bulk of the program, designed to do this via nested for loops:</p>
  
  <pre><code># I have to remove the first and last columns of data and filter them on their own,
  # as per instruction. The next 24 lines of code do this.
  data_initial = data[:,0]
  data_final = data[:,-1]
  
  # Create Polynomial fit to data, excluding NaNs. 
  idx = np.isfinite(x) &amp; np.isfinite(data_initial)
  coeff_initial = p.polyfit(x[idx], data_initial[idx], 30, full=True)
  pfit_initial = p.polyval(x, coeff_initial[0])
  
  idy = np.isfinite(x) &amp; np.isfinite(data_final)
  coeff_final = p.polyfit(x[idy], data_final[idy], 30, full=True)
  pfit_final = p.polyval(x, coeff_final[0])
  
  # replace NaN values in first and last profiles with their corresponding 
  # polynomial fit values
  for i in range(0,len(data_initial)):
    if np.isnan(data_initial[i]) == True:
      data_initial[i] = pfit_initial[i]
  
  for i in range(0,len(data_final)):
    if np.isnan(data_final[i]) == True:
      data_final[i] = pfit_final[i]
  
  data_initial = data_initial.reshape(len(data_initial),1)
  data_final = data_final.reshape(len(data_final),1)
  
  data_smooth = np.array([])
  
  for i in range(0,len(x)):
    data_set = data[i,1:-1] # "data" is the name of the original 2D array
  
    # The next 3 lines are responsible for fitting a polynomial line to each row of the data
    ids = np.isfinite(y[1:-1]) &amp; np.isfinite(data_set)
    coeff = p.polyfit(y[1:-1][ids], data_set[ids], 20)
    pfit = p.polyval(y[1:-1], coeff)
  
    # The next nested for loop is designed to replace any Nan values with values
    # from the polynomial fit line "pfit"
    for k in range(0,len(data_set)):
      if np.isnan(data_set[k]) == True:
        data_set[k] = pfit[k]
  
    # The next 2 lines calculates the residual noise of the data and the 1st standard 
    # deviation of the data
    residuals = data_set - pfit
    standard_dev = 1*np.std(residuals)
  
    # The next nested for loop is designed to replace any values inside "residuals" 
    # with polynomial fit values if they exceed the 1st standard deviation
    for j in range(0,len(residuals)):
      if abs(residuals[j]) &gt;= abs(standard_dev):
        data_set[j] = pfit[j]
  
    # The final four lines inside of the overarching loop reshape the data so that 
    # each row can be stacked to create a 2D array
    if len(data_smooth) == 0:
      data_smooth = data_set[None,:]
    else:
      data_smooth = np.vstack((data_smooth,data_set))
  
  # These 2 lines add the previously sliced first and last columns back to the now 
  # filtered 2D array of data.
  data_smooth_1 = np.hstack((data_initial,data_smooth))
  Data_filtered = np.hstack((data_smooth_1,data_final))
  </code></pre>
  
  <p>However, when this program runs through completely (and it does so without error), it doesn't appear to do anything to the original data. both the original and "filtered" data sets are identical, bit by bit. Am I doing something wrong when I reassign values in the nested for loops? I am at a loss and have been trying to solve this problem for days. please help!</p>
  
  <p>UPDATE:
  I figured out that it is filtering the data like it should be, but instead the problem lies in that it is redefining every element of the previous array. Why is this happening?</p>