from campaigns import campaigns
dbs = [c for c in campaigns if 'simz' in c]
print(dbs)

"""
Explanation: Update BIOM file with data from STOQS
Given a .biom file and multiple STOQS databases, explore Next Generation Sequence and associated STOQS data
Executing this Notebook requires a personal STOQS server. Follow the steps to build your own development system &mdash; this will take a few hours and depends on a good connection to the Internet.  Once your server is up log into it (after a cd ~/Vagrants/stoqsvm) and activate your virtual environment with the usual commands:
vagrant ssh -- -X
cd /vagrant/dev/stoqsgit
source venv-stoqs/bin/activate
pip install biom-format

Then load all of the SIMZ databases with the commands below. In order to have all the subsample analysis data (Sampled Parameters) loaded it's necessary to have SIMZ&lt;month&gt;&lt;year&gt; directories containing those .csv files. (See the subsample_csv_files attribute setting in the load script for the campaign.)
cd stoqs
ln -s mbari_campaigns.py campaigns.py
export DATABASE_URL=postgis://stoqsadm:CHANGEME@127.0.0.1:5432/stoqs
loaders/load.py --db stoqs_simz_aug2013 stoqs_simz_oct2013 \
stoqs_simz_spring2014 stoqs_simz_jul2014 stoqs_simz_oct2014 
loaders/load.py --db stoqs_simz_aug2013 stoqs_simz_oct2013 \
stoqs_simz_spring2014 stoqs_simz_jul2014 stoqs_simz_oct2014 --updateprovenance

Loading these database will take a few hours. Once it's finished you can interact with the data quite efficiently, as this Notebook demonstrates. Launch Jupyter Notebook with:
cd contrib/notebooks
../../manage.py shell_plus --notebook

navigate to this file and open it. You will then be able to execute the cells and experiment.
Make a Python list of all SIMZ database from the campaigns on our system.
End of explanation
"""


biom_file = '../../loaders/MolecularEcology/BIOM/otu_table_newsiernounclass_wmetadata.biom'
from biom import load_table
table = load_table(biom_file)
print(table.ids(axis='sample'))
print(table.ids(axis='observation')[:5])

"""
Explanation: Open a .biom file that contains sequence data from Net Tows conducted on these campaigns. (You will need to create the BIOM directory and copy the .biom file there as we generally do not keep data files in the STOQS git repository.)
End of explanation
"""


nettows = {}
for db in dbs:
    for s in Sample.objects.using(db).filter(sampletype__name='VerticalNetTow'
                ).order_by('instantpoint__activity__name'):
        print(s.instantpoint.activity.name, db)
        nettows[s.instantpoint.activity.name] = db

"""
Explanation: Find all the VerticalNetTow Sample identifiers for all our SIMZ campaigns. (For hints on names to filter on use the STOQS REST api to explore Sample data for a campaign, e.g.: http://localhost:8000/stoqs_simz_aug2013/api/sample.html.) These will be our links to the environmental and other sample data.
End of explanation
"""


stoqs_sample_data = {}
for s, b in [('simz2013c{:02d}_NetTow1'.format(int(n[4:])), n) for n in table.ids()]:
    sps = SampledParameter.objects.using(nettows[s]
                                        ).filter(sample__instantpoint__activity__name=s)
    # Values of BIOM metadata must be strings, even if they are numbers
    stoqs_sample_data[b] = {sp.parameter.name: str(float(sp.datavalue)) for sp in sps}

"""
Explanation: It looks as though the BIOM table ids (SIMZ1, SIMZ2, ...) correspond to the STOQS s.instantpoint.activity.names (simz2013c01_NetTow1, simz2013c02_NetTow1, ...). Let's loop through the STOQS sample names and BIOM file sample ids, extract relevant data from STOQS and populate a dictionary formatted for adding metadata back to the BIOM file.
End of explanation
"""


new_table = table.copy()
new_table.add_metadata(stoqs_sample_data)
with open(biom_file.replace('.biom', '_stoqs.biom'), 'w') as f:
    new_table.to_json('explore_BIOM_data_for_SIMZ.ipynb', f)

"""
Explanation: Create a copy of the original table, add the new metadata to the samples and save to a new file name.
End of explanation
"""


import pprint
pp = pprint.PrettyPrinter(indent=4)
print('Original: ' + biom_file)
print('-' * len('Original: ' + biom_file))
pp.pprint(table.metadata()[:2])
print()
print('New: ' + biom_file.replace('.biom', '_stoqs.biom'))
print('-' * len('New: ' + biom_file.replace('.biom', '_stoqs.biom')))
pp.pprint(new_table.metadata()[:2])

"""
Explanation: Compare the first two metadata records from the original table and the new table.
End of explanation
"""


from IPython.display import Image
Image('../../../doc/Screenshots/Screen_Shot_2015-10-24_at_10.37.52_PM.png')

"""
Explanation: To test the results, upload the new file to http://phinch.org/ and you should see something like this:
End of explanation
"""


