Can't access Bluemix object store from my Notebook
Question: I'm trying to read a couple of JSON files from my Bluemix object store into a
Jupyter notebook using Python. I've followed the examples I've found, but I'm
still getting a "No such file or directory" error.

Here is the code that should authenticate the object store and identify the
files:

    
    
    # Set up Spark
    from pyspark import SparkContext
    from pyspark import SparkConf
    
    if('config' not in globals()):
        config = SparkConf().setAppName('warehousing_sql').setMaster('local')
    if('sc' not in globals()):
        sc= SparkContext(conf=config)
    
    # Set the Hadoop configuration.
    def set_hadoop_config(name, credentials):
        prefix = "fs.swift.service." + name
        hconf = sc._jsc.hadoopConfiguration()
        hconf.set(prefix + ".auth.url", credentials['auth_url']+'/v3/auth/tokens')
        hconf.set(prefix + ".auth.endpoint.prefix", "endpoints")
        hconf.set(prefix + ".tenant", credentials['project_id'])
        hconf.set(prefix + ".username", credentials['user_id'])
        hconf.set(prefix + ".password", credentials['password'])
        hconf.setInt(prefix + ".http.port", 8080)
        hconf.set(prefix + ".region", credentials['region'])
        hconf.setBoolean(prefix + ".public", True)
    
    # Data Sources (generated by Insert to code)
        credentials = {
      'auth_url':'https://identity.open.softlayer.com',
      'project':'***',
      'project_id':'****',
      'region':'dallas',
      'user_id':'****',
      'domain_id':'****',
      'domain_name':'****',
      'username':'****',
      'password':"""****""",
      'filename':'Warehousing-data.json',
      'container':'notebooks',
      'tenantId':'****'
    }
    
    set_hadoop_config('spark', credentials)
    
    # The data files should now be accessible through URLs of the form
    # swift://notebooks.spark/filename.json
    

Here is the calling code:

    
    
    ...
    resource_path= "swift://notebooks.spark/"
    Warehousing_data_json = "Warehousing-data.json"
    Warehousing_sales_data_nominal_scenario_json = "Warehousing-sales_data-nominal_scenario.json"
    ...
    

Here is the error: IOError: [Errno 2] No such file or directory:
'swift://notebooks.spark/Warehousing-data.json'

I'm sorry if this seems like a novice question (which I admit I am), but I
think it's ridiculously complicated to set this up and really bad form to rely
on an undocumented method SparkContext._jsc.hadoopConfiguration().

* * *

Added in response to Hobert's and Sven's comments:

Thanks Hobert. I don’t understand your comment about the definition for
"swift://notebooks**.spark**/" Unless I misunderstand the logic of the sample
I followed (which is essentially identical to what Sven shows in his
response), this path results from the call to sc._jsc.hadoopConfiguration(),
but it’s hard to know what this call actually does, since the
HadoopConfiguation class is not documented.

I also do not understand the alternatives to “use/add that definition for the
Hadoop configuration” or “alternatively, … use swift client inside of Spark to
access the JSON.” I suppose I would prefer the latter since I make no other
use of Hadoop in my notebook. Please point me to a more detailed explanation
of these alternatives.

Thanks Sven. You are correct that I did not show the actual reading of the
JSON files. The reading actually occurs within a method that is part of the
API for
[DOcplexcloud](https://developer.ibm.com/docloud/documentation/docloud/python-
api/). Here is the relevant code in my notebook:

    
    
    resource_path= "swift://notebooks.spark/"
    Warehousing_data_json = "Warehousing-data.json"
    Warehousing_sales_data_nominal_scenario_json = "Warehousing-sales_data-nominal_scenario.json"
    
    resp = client.execute(input= [{'name': "warehousing.mod",
                                   'file': StringIO(warehousing_data_dotmod + warehousing_inputs + warehousing_dotmod + warehousing_outputs)},
                                  {'name': Warehousing_data_json,
                                   'filename': resource_path + Warehousing_data_json},
                                  {'name': Warehousing_sales_data_nominal_scenario_json,
                                   'filename': resource_path + Warehousing_sales_data_nominal_scenario_json}],
                          output= "results.json",
                          load_solution= True,
                          log= "solver.log",
                          gzip= True,
                          waittime= 300,
                          delete_on_completion= True)
    

Here is the stack trace:

    
    
    IOError                                   Traceback (most recent call last)
    <ipython-input-8-67cf709788b3> in <module>()
         29                       gzip= True,
         30                       waittime= 300,
    ---> 31                       delete_on_completion= True)
         32 
         33 result = WarehousingResult(json.loads(resp.solution.decode("utf-8")))
    
    /gpfs/fs01/user/sbf1-4c17d3407da8d0-a7ea98a5cc6d/.local/lib/python2.7/site-packages/docloud/job.pyc in execute(self, input, output, load_solution, log, delete_on_completion, timeout, waittime, gzip, parameters)
        496         # submit job
        497         jobid = self.submit(input=input, timeout=timeout, gzip=gzip,
    --> 498                             parameters=parameters)
        499         response = None
        500         completed = False
    
    /gpfs/fs01/user/sbf1-4c17d3407da8d0-a7ea98a5cc6d/.local/lib/python2.7/site-packages/docloud/job.pyc in submit(self, input, timeout, gzip, parameters)
        436                                 gzip=gzip,
        437                                 timeout=timeout,
    --> 438                                 parameters=parameters)
        439         # run model
        440         self.execute_job(jobid, timeout=timeout)
    
    /gpfs/fs01/user/sbf1-4c17d3407da8d0-a7ea98a5cc6d/.local/lib/python2.7/site-packages/docloud/job.pyc in create_job(self, **kwargs)
        620                 self.upload_job_attachment(job_id, 
        621                                            attid=inp.name,
    --> 622                                            data=inp.get_data(),
        623                                            gzip=gzip)
        624         return job_id
    
    /gpfs/fs01/user/sbf1-4c17d3407da8d0-a7ea98a5cc6d/.local/lib/python2.7/site-packages/docloud/job.pyc in get_data(self)
        110         data = self.data
        111         if self.filename is not None:
    --> 112             with open(self.filename, "rb") as f:
        113                 data = f.read()
        114         if self.file is not None:
    
    IOError: [Errno 2] No such file or directory: 'swift://notebooks.spark/Warehousing-data.json'
    

This notebook works just fine when I run it locally and resource_path is a
path on my own machine.

Sven, your code seems pretty much identical to what I have, and it follows
closely the sample I copied, so I do not understand why yours works and mine
doesn’t.

I have verified that the files are present on my Instance_objectstore.
Therefore it seems that swift://notebooks.spark/ does not point to this
objectstore. How that would happen has been a mystery to me from the start.
Again, the HadoopConfiguation class is not documented, so it is not possible
to know how it makes the association between the URL and the objectstore.


Answer: The error message you get `IOError: [Errno 2] No such file or directory:
'swift://notebooks.spark/Warehousing-data.json'` means that at that path there
is no such file. I think the setup of the Hadoop configuration was successful
otherwise you would get a different error message complaining about missing
credentials settings.

I have tested in a Python notebook on Bluemix the following code and it worked
for me. I took the sample code from the latest sample notebooks showing how to
load data from Bluemix Object Storage V3.

Method for setting the Hadoop configuration:

    
    
    def set_hadoop_config(credentials):
        """This function sets the Hadoop configuration with given credentials, 
        so it is possible to access data using SparkContext"""
    
        prefix = "fs.swift.service." + credentials['name']
        hconf = sc._jsc.hadoopConfiguration()
        hconf.set(prefix + ".auth.url", credentials['auth_url']+'/v3/auth/tokens')
        hconf.set(prefix + ".auth.endpoint.prefix", "endpoints")
        hconf.set(prefix + ".tenant", credentials['project_id'])
        hconf.set(prefix + ".username", credentials['user_id'])
        hconf.set(prefix + ".password", credentials['password'])
        hconf.setInt(prefix + ".http.port", 8080)
        hconf.set(prefix + ".region", credentials['region'])
        hconf.setBoolean(prefix + ".public", True)
    

Insert credentials for associated Bluemix Object Storave V3:

    
    
    credentials_1 = {
      'auth_url':'https://identity.open.softlayer.com',
      'project':'***',
      'project_id':'***',
      'region':'dallas',
      'user_id':'***',
      'domain_id':'***',
      'domain_name':'***',
      'username':'***',
      'password':"""***""",
      'filename':'people.json',
      'container':'notebooks',
      'tenantId':'***'
    }
    

Set Hadopp configuration with given credentials:

    
    
    credentials_1['name'] = 'spark'
    set_hadoop_config(credentials_1)
    

Read JSON file usind `sc.textFile()` into an `RDD` and print out first 5 rows:

    
    
    data_rdd = sc.textFile("swift://" + credentials_1['container'] + "." + credentials_1['name'] + "/" + credentials_1['filename'])
    data_rdd.take(3)
    

Output:

    
    
    [u'{"name":"Michael"}',
     u'{"name":"Andy", "age":30}',
     u'{"name":"Justin", "age":19}']
    

Read JSON file using `sqlContext.read.json()` into a `DataFrame`and output
first 5 rows:

    
    
    data_df = sqlContext.read.json("swift://" + credentials_1['container'] + "." + credentials_1['name'] + "/" + credentials_1['filename'])
    data_df.take(3)
    

Output:

    
    
    [Row(age=None, name=u'Michael'),
     Row(age=30, name=u'Andy'),
     Row(age=19, name=u'Justin')]
    

