title: Numpy shape mismatch
body: <p>Good day,</p>
  <p>Sorry for the long detail!!!
  I have 258 input files which includes 15 years of data. The purpose of this script is extract a particular column assigned as a variable within a user entered start and end year. The script was originally structured to extract the whole column in that time period, but now I am further filtering the desired values to be extracted as I am only interested in values between 0-1 in that column. But, since the original script was designed to handle 731 samples (365 days + 365 days + 1), the script automatically looks for 731 sample values. And since I am not experienced with the Netcdf function in Scipy and Numpy modules that much, I don't know how to restructure this part of the script according to the filtered days having the desired values. As each year will have different number of values between 0-1, I am having trouble assigning the initial sample size according to the filtered days found by the script. I did found a way around it by giving dummy values of -1 to avoid broadcasting issues and to meet the requirement of 731 sample size, but when I open the Netcdf file in ArcGIS I see this, which is okay as the max and min values are 1 and -1 respectively, so I am not able to see my desired values in the raster properly as ArcGIS only shows most of the -1s values in the display. I hope I am able to explain my problem. Can anyone please help me with structuring the sample size according to the filtered values found by the script? Although, I did find NetCDF4Excel, an Excel add-in to just delete the dummy values stored in the .nc file in Excel, but it seems like this add-in doesn't work on 64-bit systems. I know the problem lies with days, but I don't know how to fix it. Oh just forgot to mention that it's a 3D array containing the X coordinate, Y coordinate and date/time.
  <a href="https://i.stack.imgur.com/PGP05.png" rel="nofollow noreferrer">enter image description here</a></p>
  <pre><code># for what date?
  start_year = input(&quot;Enter start year:&quot;)
  end_year = input(&quot;End year:&quot;)
  
  inidate = datetime.date(start_year,1,1)
  enddate = datetime.date(end_year,12,31)
  
  days = enddate.toordinal() - inidate.toordinal()+1 
  </code></pre>
  <p>This is the part of the script I use for filtering values in a particular column:</p>
  <pre><code>for l in lixo:
          if int(l.split(&quot;\t&quot;)[0]) in range(inidate.year, enddate.year+1):
              if var==3:
                  if previousValue==-10:                    
                      previousValue=float(l.split(&quot;\t&quot;)[var])
                      dado.append(-1)
                  else:
                      currentValue=float(l.split(&quot;\t&quot;)[var])
                      if currentValue==0 and previousValue&gt;0:
                          dado[-1]=previousValue
                          dado.append(currentValue)
                      else:
                          dado.append(-1)
                      previousValue=currentValue
              else:
                  dado.append(float(l.split(&quot;\t&quot;)[var]))
          # putting data inside array.
          # Since data has lat &amp; lon fixed uses dimension [:,lat_index,lon_index]
  
      print dado
  </code></pre>
  <p>The whole script is as follows:</p>
  <pre><code>import os
  import sys
  # handle dates...
  import datetime
  # SciPy netCDF and NumPy
  from scipy.io.netcdf import *
  from numpy import *
  
  skip_lines = 6
  
  # building file list and sorted lat lon list
  file_list = os.listdir(sys.argv[1])
  
  lat_t = []
  lon_t = []
  lat = []
  lon = []
  
  for f in file_list:
      lat_t.append(float(f.split(&quot;_&quot;)[1]))
      lon_t.append(float(f.split(&quot;_&quot;)[2]))
  
  for i in lat_t:
      if i not in lat:
          lat.append(i)
  
  for i in lon_t:
      if i not in lon:
          lon.append(i)
  # putting in order. Lat should be from top to bottom
  # lon from left to right 
  lon.sort()
  lat.sort()
  lat.reverse()
  
  del(lat_t)
  del(lon_t)
  
  #determining the parameter to use
  print &quot;Choose output parameter&quot;
  varini = input('Choose output (1 a 8)&gt;')
  
  #getting the column right
  if int (varini) &lt; 8:
      var = varini + 2
  #set name of out_file. Named after parameter choice
  if var == 3:
      var_txt = &quot;ABCD&quot;
      var_name = &quot;ABCD&quot;
  # for what date?
  start_year = input(&quot;Enter start year:&quot;)
  end_year = input(&quot;End year:&quot;)
  
  inidate = datetime.date(start_year,1,1)
  enddate = datetime.date(end_year,12,31)
  
  days = enddate.toordinal() - inidate.toordinal()+1
  
  print &quot;Go grab a coffee, this could take a while...&quot;
  
  #
  # create array containig all data
  # This is going to be huge. Create an array with -9999 (NoData)
  # Then populate the array by reading each input file
  #
  
  all_data = zeros([days,len(lat),len(lon)], float)-9999
  
  c = len(file_list)
  
  # for each file in list
  for f in file_list:
      # get lat &amp; lon and it's index
      latitude = float(f.split(&quot;_&quot;)[1])
      longitude = float(f.split(&quot;_&quot;)[2])
      lat_id = lat.index(latitude)
      lon_id = lon.index(longitude)
  
      print &quot;%i files to write.&quot; % c
      c = c -1
  
      infile = open(sys.argv[1]+f, &quot;r&quot;)
      # here we skip the number of header lines
      # variable set at the beginning of the code
      lixo = infile.readlines()[skip_lines:]
      infile.close()
      dado = []
      previousValue = -10
  
      for l in lixo:
          if int(l.split(&quot;\t&quot;)[0]) in range(inidate.year, enddate.year+1):
              if var==3:
                  if previousValue==-10:                    
                      previousValue=float(l.split(&quot;\t&quot;)[var])
                      dado.append(-1)
                  else:
                      currentValue=float(l.split(&quot;\t&quot;)[var])
                      if currentValue==0 and previousValue&gt;0:
                          dado[-1]=previousValue
                          dado.append(currentValue)
                      else:
                          dado.append(-1)
                      previousValue=currentValue
              else:
                  dado.append(float(l.split(&quot;\t&quot;)[var]))
  
          # putting data inside array.
          # Since data has lat &amp; lon fixed uses dimension [:,lat_index,lon_index]
  
      print dado
  all_data[:,lat_id,lon_id] = dado
  
  # writing NetCDF
  #
  
  ncfile = netcdf_file(var_txt+&quot;.nc&quot;, &quot;w&quot;)
  
  ncfile.Conventions = &quot;COARDS&quot;
  ncfile.history = &quot;Created using flux2cdf.py. &quot; + datetime.date.today().isoformat()
  ncfile.production = &quot;ABCD output&quot;
  
  ncfile.start_date = inidate.isoformat()
  ncfile.end_date = enddate.isoformat()
  
  #create dimensions
  ncfile.createDimension(&quot;X&quot;, len(lon))
  ncfile.createDimension(&quot;Y&quot;, len(lat))
  ncfile.createDimension(&quot;T&quot;, days)
  
  #create variables
  latvar = ncfile.createVariable(&quot;Y&quot;, &quot;f4&quot;, (&quot;Y&quot;,))
  latvar.long_name = &quot;Latitude&quot;
  latvar.units = &quot;degrees_north&quot;
  latvar[:] = lat
  
  lonvar = ncfile.createVariable(&quot;X&quot;, &quot;f4&quot;, (&quot;X&quot;,))
  lonvar.long_name = &quot;Longitude&quot;
  lonvar.units = &quot;degrees_east&quot;
  lonvar[:] = lon
  
  timevar = ncfile.createVariable(&quot;T&quot;, &quot;f4&quot;, (&quot;T&quot;,))
  timevar.long_name = &quot;Time&quot;
  timevar.units = &quot;days since &quot; + inidate.isoformat()
  timevar[:] = range(0, days)
  
  data_var = ncfile.createVariable(var_txt, &quot;f4&quot;, (&quot;T&quot;,&quot;Y&quot;,&quot;X&quot;))
  data_var.long_name = var_name+&quot; calculated by ABCD&quot;
  data_var.missing_value = -9999.0
  data_var.units = &quot;milimeters&quot;
  data_var[:] = all_data
  
  ncfile.close()
  </code></pre>
  <p>Sample data that's written in the .nc file with the required filtered values have been highlighted in yellow  shown below:</p>
  <p><a href="https://i.stack.imgur.com/elpMd.png" rel="nofollow noreferrer">enter image description here</a></p>
  <p>Sample data:</p>
  <pre><code>245 files to write.
  [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 0.0978, 0.0, -1, -1, -1, -1, -1, -1, -1, -1, 0.3112, 0.0, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 0.3112, 0.0, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 0.0978, 0.0, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 0.1334, 0.0, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 0.0978, 0.0, -1, -1, -1, -1, -1, 0.0978, 0.0, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]
  244 files to write.
  </code></pre>