How to stream upload recorder audio as multipart message to a http/2 endpoint?
I am implementing the Alexa Voice Service in a native iOS app (swift 2.3) and i'm struggling with the http/2 implementation.
"The request sent to AVS is a multipart message: the first part a JSON-formatted object, the second part binary audio captured by the microphone. We encourage streaming (chunking) captured audio to the Alexa Voice Service to reduce latency; the stream should contain 10ms of captured audio per chunk (320 bytes)."
To be a bit more clear: I want to upload (stream) the audio to AVS while I'm still recording the user's voice.
I already succeeded in uploading a full sound sample and sending that to AVS as a whole. However, the streaming implemtation is what I need and I have no idea how to accomplish this.
Can somebody give me some guidance ?
Re: How to stream upload recorder audio as multipart message to a http/2 endpoint?
I presume you’re asking about the networking side of this, not the audio recording side. I can help with the former but not the latter.
You can use URLSession to do a streamed upload by creating the upload task with uploadTask(withStreamedRequest:). The session will then call the urlSession(_:task:needNewBodyStream:) delegate method to get an input stream for the upload body. It will then open and read (asynchronously) data from that stream, uploading it as it becomes available.
IMPORTANT It’s possible for the session to call your delegate method more than once. For example, if the request fails with an authentication error, the session may automatically send the request again, and that will need a new body stream.
IMPORTANT Remember that the session reads the data from the stream asynchronously. If you choose to implement the second option, you must support asynchronous reads properly.
First of all, thank you so much for showing interest in the question! I posted this topic on several other forums but people just ignored me there..
Ì looked into the uploadTask(withStreamedRequest:) function you talked about in your reply. I grasp the concept of the delegate methods but one thing I could not understand was how the multipart message is going to work. I tried to make it work today but kept getting 400 responses from AVS.
let contentType = "multipart/form-data; boundary=\(boundry)"
You’re definitely getting mixed up here. When you use urlSession(_:task:needNewBodyStream:) that stream has to yield the entire HTTP body. You’re trying to use two streams, one that contains the multipart/form-data preamble, which you set up via the httpBodyStream property, and another that contains the bulk data for that form, which you set up via urlSession(_:task:needNewBodyStream:). That won’t work. You need one stream that yields the entire body.
Once you have this basic test working you can then go on to investigate using an indefinite stream.
btw Given that you’re using Swift 3 you should skip NSMutableData and use a Swift Data value. You can make this mutable by declaring it using var.
I modified the code to meet the requirements of your previous reply. I now get 200 responses from AVS!
However, AVS sends back data containing the binary audio to play in my app but I never receive the data in the urlSession:didReceive data function. Is this normal behaviour ?
let contentType = "multipart/form-data; boundary=\(boundary)"
However, AVS sends back data containing the binary audio to play in my app but I never receive the data in the urlSession:didReceive data function.
How did you determine that the server actually send you any data in that case?
Is this normal behaviour ?
I would expect you to get the ‘did receive data’ call with the body data of the response.
When this function threw a response, the audio of AVS was contained in the data parameter of the function. That's why I expect to receive data in the current implementation with the delegate methods but that ain't happening.
So, to confirm, you've implemented the urlSession(_:dataTask:didReceive:) delegate callback and it’s not getting called, right?
Weird. I tried this out here in my office and, while I saw some weird stuff, I definitely get the response data back from the server.
The server did not recognise my form data.
AFAICT I’ve done nothing wrong here. Looking at a packet trace of what I sent on the wire, the only difference between the two requests is the transfer encoding. The identity request uses no transfer encoding and the chunked request uses a transfer encoding of Chunked. This is both expected (NSURLSession doesn’t know the length of the body stream and thus has to use chunked encoding) and correct (HTTP 1.1 requires that the server support chunked encoding). However, it’s clear that httpbin.org is not handling this properly.
Coming back to your situation, I’m not sure what this means for you. To do a streamed upload you’re going to have to use chunked encoding. I presume that the server you’re talking to will accept that, but it’s hard to know for sure because I don’t control the server.
I discuss this idea in depth in my Debugging HTTP Server-Side Errors post.
I searched for the cause of the didReceivedata function not being called and found out that I just forgot to call the completion handler in the didreceiveresponse function.. To be clear: I receive the data now, so that's working!
Now there is one part missing to make all of this work.. We are now sending prerecorded sound samples but how do we get to stream the sound of the voice recorder directly to the request. You mentioned it could be possible with something called a stream pair ?
How could I implement this in my current request code/configuration ?
PS. I really appreciate the amount of effort and time you spend on this thread!
I receive the data now, so that's working!
… how do we get to stream the sound of the voice recorder directly to the request. You mentioned it could be possible with something called a stream pair ?
That data will show up on the input stream, whereupon it’ll be read by the upload task and sent to the server.
The upload task will then open that stream, read data from it, and send it to the server. The complication here is that the upload task will read the stream asynchronously, by scheduling it on a run loop, and correctly implementing that is a bit tricky (in fact, it wasn’t even supported prior to iOS 7).
Currently i am recroding the audio data into a temp file and then fetching data from that file in form of NSData (while setting up request to upload).
How can I write audio data directly to my request body also uploading it at the same time??
I am using AVAudioRecorder for recording audio into a temp file.
I can’t help you with the audio side of this. I’m going to assume that you have some way to get chunks of audio in a format that your server will accept. If not, you’ll need to ask questions about that over in Media > AVFoundation (Audio).
Create an upload that uses a streamed body by calling uploadTask(withStreamedRequest:).
Once you resume the request, URLSession will call urlSession(_:task:needNewBodyStream:) to get a body stream for the request.
You can create a bound stream pair.
I’ll discuss the details of each below.
Your implementation of the urlSession(_:task:needNewBodyStream:) method must call the completion handler with that input stream.
Subclassing InputStream is simple in the synchronous case but your subclass must support async reads. This should be possible on recent systems but it’s quite tricky. I generally recommend the bound stream pair approach.
Once you have a bound stream pair your can pass the input stream to the completion handler (step 4 above) and then you ‘own’ the output stream. You should run the stream asynchronously, writing data to it in chunks as that data gets recorded. Any data you write to the stream will be sent as a chunk of the HTTP body (that is, using the chunked Transfer Encoding).
The output stream here is no different from any other output stream, and there’s tonnes of examples of how to run an output stream asynchronously.
Thanks @eskimo for details you have mentioned here.
and providing reuest body via NSInputStream in the delegate "needNewBodyStream". As you can see the audio data is coming from a file (already recorded with correct audio format).
So do you mean that now i should get the chunks in real time from the recorder and pass it to same inputStream in that delegate method??
and when the delegate gets called then i should pass the audio part via inputstream (not sure if we can pass body in different streams i.e httpBody/HttpsBodyStream and rest in "needNewBodyStream" delegate via NSInputStream).
Also i had this one doubt, previously i was using "uploadTaskWithRequest" for same request and now this - "uploadTaskWithStreamedRequest" .
which is almost same (in both cases - uploadTaskWithRequest and StreamRequest), still same chunks of data are being sent. Is it expected behaviour, or should there be some difference between the dataBytes sent over the network?
Can you start a new thread for this? The indentation here is getting in the way of me reading your post.
Also, perhaps you can use this as an opportunity to describe the overall big picture of your project, because I’m not sure I’ve got a good handle on that right now.
Could you provide the link to the new thread? I'm trying to do something very similar, that is take microphone samples and send them via chunks. Thank you!