title: How to get a response for a streaming url on google app engine (python)
body: <p>I am trying to verify if a online radio url is delivering music and if the url was redirected or not (this happens if for some reason the request url is wrong or not active). I found some advices here <a href="https://stackoverflow.com/questions/23036285/fetching-url-in-python-with-google-app-engine">Fetching url in python with google app engine</a>. However, for an url that delivers Content-Type:audio/mpeg it doesn't seem to work.</p>
  
  <p>On my local machine using python 2.7.6 urllib2.urlopen everything is fine:</p>
  
  <pre><code>try:
      print "begin urlopen"
      url = urllib2.urlopen("http://streaming.radionomy.com/jamaican-roots-radio")
      print "end urlopen"
  
  except Exception, e:
      print e
  </code></pre>
  
  <p>gives</p>
  
  <blockquote>
    <p>begin urlopen</p>
    
    <p>end urlopen</p>
  </blockquote>
  
  <p>I can the read N bytes from the returned object (which is a socket._fileobject) and use the method <strong>geturl()</strong> to get the actual url from which the stream is coming (if there was no redirection the request url and the retrieved resource url are the same)</p>
  
  <p>The problems arise using dev_appserver.py for google appengine (I didn't deployed yet). The call never returns:</p>
  
  <blockquote>
    <p>begin urlopen</p>
    
    <p>WARNING  2015-06-12 14:31:43,599 urlfetch_stub.py:504] Stripped prohibited headers from URLFetch request: ['Host']</p>
  </blockquote>
  
  <p>and "end urlopen" is never printed.</p>
  
  <p>I understand the warning error, so I switched (as suggested in the link above) to urlfetch:</p>
  
  <pre><code>try:
      print "begin fetch"
      url = urlfetch.fetch("http://streaming.radionomy.com/jamaican-roots-radio")
      print "end fetch"
  
  except Exception, e:
      print e
  </code></pre>
  
  <p>gives</p>
  
  <blockquote>
    <p>begin</p>
  </blockquote>
  
  <p>The warnings is gone, but again the call doesn't return.  </p>
  
  <p>For a normal webpage url, everything is as expected. I guess that the problem is the response object that is never finished. Also using</p>
  
  <pre><code>urlfetch.set_default_fetch_deadline(5)
  </code></pre>
  
  <p>doesn't change the situation, probably because the data are continuously streamed from the server (and therefore no timeout is called??).
  I also tried the low level <strong>httplib.HTTPConnection</strong>, but after making the request the <strong>getresponse()</strong> function never returns.</p>
  
  <p>To my purpose, the response header would be enough. But on the server (which is not under my control) the HEAD method is not implemented (despite being listed in Access-Control-Allow-Methods, as it can bee seen from a browser)</p>
  
  <pre><code>curl -X HEAD -i http://streaming.radionomy.com/jamaican-roots-radio
  </code></pre>
  
  <blockquote>
    <p>HTTP/1.0 501 Not Implemented</p>
  </blockquote>
  
  <p>I didn't find any question on stackoverflow covering the case of a stream url except this one
  <a href="https://stackoverflow.com/questions/2543339/how-to-call-twitters-streaming-filter-feed-with-urllib2-httplib">How to call Twitter&#39;s Streaming/Filter Feed with urllib2/httplib?</a>. Unfortunately, the suggested response is not very helpful for me ("Using Twitter's 'standard' API").</p>
  
  <p><strong>Any idea I can solve this problem?</strong></p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>On google appengine (not on dev_appserver.py as above) the problems are similar:</p>
  
  <ul>
  <li>with a deadline of 5 sec</li>
  </ul>
  
  <blockquote>
    <p>Deadline exceeded while waiting for HTTP response from URL...</p>
  </blockquote>
  
  <ul>
  <li>with a deadline of 60 sec</li>
  </ul>
  
  <blockquote>
    <p>Traceback (most recent call last):</p>
    
    <p>File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 267, in Handle result = handler(dict(self._environ), self._StartResponse)</p>
    
    <p>File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in <strong>call</strong>
        rv = self.router.dispatch(request, response)</p>
    
    <p>File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
        return route.handler_adapter(request, response)</p>
    
    <p>File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in <strong>call</strong>
        return handler.dispatch()</p>
    
    <p>File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
        return method(*args, **kwargs)</p>
    
    <p>File "/base/data/home/apps/s~radiosnoozers/3.384985169499124712/controllers/checkurl.py", line 80, in get
        print e</p>
    
    <p>File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/request_environment.py", line 94, in write
        self._request.errors.write(data)</p>
    
    <p>File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/logservice/logservice.py", line 287, in write
        self._write(line)</p>
    
    <p>File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/logservice/logservice.py", line 307, in _write
        if self._request != logsutil.RequestID():</p>
    
    <p>DeadlineExceededError</p>
  </blockquote>
  
  <p>The timeout is respected and there are no difference by using using allow_truncated=True. In any case, no access to the response...</p>
  
  <p>I really don't know what is going on, but thanks for the given suggestions.</p>