How to Access a Text File from IBM DSX Scala-Spark Notebook I have recently faced problem in accessing text files that I have uploaded from my local computer to the IBM DSX Cloud Object Storage so that I can use it in my Spark-Scala notebooks. It appeared to me that the general configuration of the platform is focused on CSV format; so is most of the help material. In fact, I figured a way around when I changed the extension of my .txt file to .csv. Through this post and the answer followed, I am trying to share my experience with others.

This post is useful, if:


You have created a IBM DSX notebook with “Language=Scala” and “Spark version=2.1 OR 2.0″[did not try with other versions]
AND You are trying to access a text file [did not try other formats] that you have uploaded from your personal computer to the IBM Cloud Object Storage
In order to access files that are in Cloud Object Storage, you need to provide the path of the file in the form of a CloudObjectStorage instance. Add following lines, after the file credentials, to your code:

var soc = new CloudObjectStorage(sc, credentials, null, “bluemix_cos”) var myFile = sc.textFile(soc.url(“your_bucket”,”data.txt”))

In the second code line above, “your_bucket” is the value of “BUCKET” in the file credentials.

There is one last thing we need to do before we are good to go. The CloudObjectStorage constructor accepts credentials in a way that is for the CSV files. Luckily the main difference between the two formats is the HashMap key names; hence, we can easily manage this problem by changing the above credentials manually, as mentioned below:

“IBM_API_KEY_ID” as “apiKey”

“IAM_SERVICE_ID” as “serviceId”

“ENDPOINT” as “endPoint”

“IBM_AUTH_ENDPOINT” as “iamServiceEndpoint”

For further details, you can visit the link below:
http://rehmans.org/2018/01/31/how-to-access-a-text-file-from-ibm-dsx-scala-spark-notebook/