title: OpenCV read images from pyspark and pass to a Keras model
body: <p>This is a follow-up question to the answer posted <a href="https://stackoverflow.com/a/52274419/554481">here</a>. I'm using PySpark 2.4.4. I have a bunch of images (some .png some .jpeg) stored on Google Cloud Storage (GCS) that I need to pass to a Tensorflow model. I'm getting my images like this.</p>
  
  <pre><code>images = spark.read.format("image").option("dropInvalid", False).load("gs://my-bucket/my_image.jpg")
  images = images.collect()
  image = cv2.imdecode(np.frombuffer(images[0].image.data, np.uint8), cv2.IMREAD_COLOR)
  </code></pre>
  
  <p>Based on the OpenCV documentation I've read, it seems like OpenCV isn't able to understand my data format. I know this because <code>cv2.imdecode(...)</code> returns <code>None</code>. The official Spark <a href="https://spark.apache.org/docs/latest/ml-datasource#image-data-source" rel="nofollow noreferrer">documentation</a> explicitly mentions compatibility with OpenCV, so I know it's possible. </p>
  
  <p>Eventually I want to be able to do this.</p>
  
  <pre><code>prediction = model.predict(np.array([image]))[0]
  </code></pre>
  
  <p>Outside of Spark, if I get my image not from GCS but from an http endpoint, all I have to do is this, which works.</p>
  
  <pre><code>resp = urllib.request.urlopen(image_url)
  image = resp.read()
  prediction = model.predict(np.array([image]))[0]
  </code></pre>
  
  <p>To get a better sense of what the model is looking for, this is what the data should look like before it's passed into the <code>np.array([...])</code> part.</p>
  
  <pre><code>print(resp.read())
  &gt;&gt;&gt; b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\ ...'
  </code></pre>
  
  <p>I can confirm that the images aren't corrupted when they're on GCS. When I download the same image from GCS to my laptop, and then read it like this, I get a similarly looking format. The model is also able to consume the image this way. I've also visually inspected the downloaded GCS image, and it looks fine.</p>
  
  <pre><code>with open("./my_image.jpeg", "rb") as image:
      print(image.read())
  &gt;&gt;&gt; b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\ ...'
  
  </code></pre>