import numpy as np

"""
Explanation: ASCAT data loading <br><br>
data reading support (research of Hans Lievens)
End of explanation
"""


!ls ./ASCAT/

"""
Explanation: Received information about dataset (email-conversation)
Provided files in a folder (ftp)
End of explanation
"""


filename = "1323"

"""
Explanation: First draft reading of the binary data
In the following section, I'll test the reading for the following dataset/file:
End of explanation
"""


struct1 = np.dtype([('gpi', np.int32)])

"""
Explanation: INDEX FILE
The data-type structure was provided (and apparently numpy-styling):
End of explanation
"""


f = open("".join(["./ASCAT/", filename, ".idx"]), "rb")  # reopen the file
f.seek(208, os.SEEK_SET)  # seek

idx_data = np.fromfile(f, dtype=struct1).astype('int32') # astype erbij om de omzetting te verzekeren

"""
Explanation: We have to skip the  first 208 bytes to avoid the reading of the header, so I checked on Stackoverflow how to do this: http://stackoverflow.com/questions/14245094/how-to-read-part-of-binary-file-with-numpy
End of explanation
"""


idx_data.shape # evaluate size before I print it...

idx_data[:50]

"""
Explanation: Checking the data:
End of explanation
"""


np.unique(idx_data)

np.unique(idx_data).size

"""
Explanation: I read a list of numbers, as described in the email. So the grid-cells are clustered together for a set of lines. Let's see how the unique values look like:
End of explanation
"""


struct2 = np.dtype([('jd', np.double),  ('sig', np.float32), ('sig_noise', np.float32), 
                   ('dir', np.dtype('S1')),    ('pdb',np.ubyte), ('azcorr_flag', 
                                                                  np.dtype([('f', np.ubyte),  
                                                                            ('m',np.ubyte),  
                                                                            ('a', np.ubyte)]))])

f = open(''.join(["./ASCAT/", filename, ".dat"]), "rb")  # reopen the file
f.seek(208, os.SEEK_SET)  # seek

data = np.fromfile(f, dtype=struct2)

data

"""
Explanation: This data sets contains 1379 different grid points that can be selected
DAT-FILE itself
Reading in, considering the provided data-type and the 208 bit header to skip
End of explanation
"""


data[1000]

"""
Explanation: A single element:
End of explanation
"""


# Decide about a grid point
grid_point = 2301829

# select rows/position for the given grid point
indxs_point = np.where(idx_data == grid_point)

# Get these rows out of the data-array
my_selection = data[indxs_point]

my_selection

"""
Explanation: As suggested by Hans, I should check the second number to be in between 0 and -30, since this is backscatter value. First part looks like a date sequence... OK!
Extract data from a gridpoint
So considering the email, following steps are undertaken:
* Decide about a number/grid point to select
* Check for which rows this number is in the file and save these indices
* Select the rows from the .dat file based on the saved indices
Let's try it out, based on the idx_data and the data arrays:
End of explanation
"""


import scipy.io as sio

"""
Explanation: Hans works in Matlab, so I'd like to provide the necessary data as a .mat-file. Exporting to .mat files is available as a scipy package extension:
End of explanation
"""


sio.savemat("data_hans", {''.join(['grid_point1_',str(grid_point)]):my_selection, ''.join(['grid_point2_',str(grid_point)]):my_selection})

"""
Explanation: Some testing on how to export it: different grid points in a single file is possible, but maybe not feasible for >1000 grid_points...
End of explanation
"""


from convert_ascat import convert_ascat_to_matlab

"""
Explanation: This works and creates a matlab file, that kan be loaded by Matlab as structure. OK!
Bring this together in a single functionality
This information is what I needed and provided and are the building blocks I need. However, I'd like to make this available as a functionality to make this reusable:
* within python, in case I would need it as a function within another workflow
* from command line; as such, I can quickly run it fromn cmd when new data enters
Learnt all the things above, I made a function to do the analysis above in a file called convert_ascat.py.
FUNCTION
End of explanation
"""


my_data = convert_ascat_to_matlab("./ASCAT/1323", grid_point_id=2301795, byte2skip=208)

"""
Explanation: So, let's test the functionality as a function...
Extracting a single grid point, by naming the ID
End of explanation
"""


all_data = convert_ascat_to_matlab("./ASCAT/1323", 
                                   grid_point_id='all', 
                                   byte2skip=208)

"""
Explanation: Extracting all grid points, all in separate files
End of explanation
"""


