title: Spark using PySpark read images
body: <p>Hi there I have a lot of images (lower millions) that I need to do classification on. I am using Spark and managed to read in all the images in the format of <code>(filename1, content1), (filename2, content2) ...</code> into a big RDD. </p>
  
  <pre><code>images = sc.wholeTextFiles("hdfs:///user/myuser/images/image/00*")  
  </code></pre>
  
  <p>However, I got really confused what to do with the unicode representation of the image. </p>
  
  <p>Here is an example of one image/file: </p>
  
  <pre><code>(u'hdfs://NameService/user/myuser/images/image/00product.jpg', u'\ufffd\ufffd\ufffd\ufffd\x00\x10JFIF\x00\x01\x01\x01\x00`\x00`\x00\x00\ufffd\ufffd\x01\x1eExif\x00\x00II*\x00\x08\x00\x00\x00\x08\x00\x12\x01\x03\x00\x01\x00\x00\x00\x01\x00\x00\x00\x1a\x01\x05\x00\x01\x00\x00\x00n\x00\x00\x00\x1b\x01\x05\x00\x01\x00\x00\x00v\x00\x00\x00(\x01\x03\x00\x01\x00\x00\x00\x02\x00\x00\x001\x01\x02\x00\x0b\x00\x00\x00~\x00\x00\x002\x01\x02\x00\x14\x00\x00\x00\ufffd\x00\x00\x00\x13\x02\x03\x00\x01\x00\x00\x00\x01\x00\x00\x00i\ufffd\x04\x00\x01\x00\x00\x00\ufffd\x00\x00\x00\x00\x00\x00\x00`\x00\x00\x00\x01\x00\x00\x00`\x00\x00\x00\x01\x00\x00\x00GIMP 2.8.2\x00\x002013:07:29 10:41:35\x00\x07\x00\x00\ufffd\x07\x00\x04\x00\x00\x000220\ufffd\ufffd\x02\x00\x04\x00\x00\x00407\x00\x00\ufffd\x07\x00\x04\x00\x00\x000100\x01\ufffd\x03\x00\x01\x00\x00\x00\ufffd\ufffd\x00\x00\x02\ufffd\x04\x00\x01\x00\x00\x00\x04\x04\x00\x00\x03\ufffd\x04\x00\x01\x00\x00\x00X\x01\x00\x00\x05\ufffd\x04\x00\x01\x00\x00\x00\ufffd\x00\x00\x00\x00\x00\x00\x00\x02\x00\x01\x00\x02\x00\x04\x00\x00\x00R98\x00\x02\x00\x07\x00\x04\x00\x00\x000100\x00\x00\x00\x00\ufffd\ufffd\x04_http://ns.adobe.com/xap/1.0/\x00&lt;?xpacket begin=\'\ufeff\' id=\'W5M0MpCehiHzreSzNTczkc9d\'?&gt;\n&lt;x:xmpmeta xmlns:x=\'adobe:ns:meta/\'&gt;\n&lt;rdf:RDF xmlns:rdf=\'http://www.w3.org/1999/02/22-rdf-syntax-ns#\'&gt;\n\n &lt;rdf:Description xmlns:exif=\'http://ns.adobe.com/exif/1.0/\'&gt;\n  &lt;exif:Orientation&gt;Top-left&lt;/exif:Orientation&gt;\n  &lt;exif:XResolution&gt;96&lt;/exif:XResolution&gt;\n  &lt;exif:YResolution&gt;96&lt;/exif:YResolution&gt;\n  &lt;exif:ResolutionUnit&gt;Inch&lt;/exif:ResolutionUnit&gt;\n  &lt;exif:Software&gt;ACD Systems Digital Imaging&lt;/exif:Software&gt;\n  &lt;exif:DateTime&gt;2013:07:29 10:37:00&lt;/exif:DateTime&gt;\n  &lt;exif:YCbCrPositioning&gt;Centered&lt;/exif:YCbCrPositioning&gt;\n  &lt;exif:ExifVersion&gt;Exif Version 2.2&lt;/exif:ExifVersion&gt;\n  &lt;exif:SubsecTime&gt;407&lt;/exif:SubsecTime&gt;\n  &lt;exif:FlashPixVersion&gt;FlashPix Version 1.0&lt;/exif:FlashPixVersion&gt;\n  &lt;exif:ColorSpace&gt;Uncalibrated&lt;/exif:ColorSpace&gt;\n  
  </code></pre>
  
  <p>Looking closer, there are actually some characters look like the metadata like </p>
  
  <pre><code>...
  &lt;x:xmpmeta xmlns:x=\'adobe:ns:meta/\'&gt;\n&lt;rdf:RDF xmlns:rdf=\'http://www.w3.org/1999/02/22-rdf-syntax-ns#\'&gt;\n\n 
  &lt;rdf:Description xmlns:exif=\'http://ns.adobe.com/exif/1.0/\'&gt;\n  
  &lt;exif:Orientation&gt;Top-left&lt;/exif:Orientation&gt;\n  
  &lt;exif:XResolution&gt;96&lt;/exif:XResolution&gt;\n  
  &lt;exif:YResolution&gt;96&lt;/exif:YResolution&gt;\n  
  ...
  </code></pre>
  
  <p>My previous experience was using the package scipy and related functions like 'imread' ... and the input is usually a filename. Now I really got lost what does those unicode mean and what I can do to transform it into a format that I am familiar with. </p>
  
  <p>Can anyone share with me how can I read in those unicode into a scipy image (ndarray)?</p>