title: Upload a video file by chunks
body: <p><strong>Yes, it's a long question with a lot of detail...</strong> So, my question is: How can I stream an upload to Vimeo in segments?</p>
  
  <p><strong>For anyone wanting to copy and debug on their own machine:</strong> Here are the things you need:</p>
  
  <ul>
  <li>My code <a href="http://collabedit.com/8hwwy" rel="noreferrer">here</a>.</li>
  <li>Include the Scribe library found <a href="https://github.com/fernandezpablo85/scribe-java" rel="noreferrer">here</a></li>
  <li>Have a valid video file (mp4) which is at least greater than 10 MB and put it in the directory <code>C:\test.mp4</code> or change that code to point wherever yours is.</li>
  <li>That's it! Thanks for helping me out!</li>
  </ul>
  
  <p><strong>Big update:</strong> I've left a working API Key and Secret for Vimeo in the code <a href="http://collabedit.com/8hwwy" rel="noreferrer">here</a>. So as long as you have a Vimeo account, all the code should work just fine for you once you've allowed the application and entered your token. Just copy the code from that link into a project on your favorite IDE and see if you can fix this with me. I'll give the bounty to whoever gives me the working code. Thanks! Oh, and don't expect to use this Key and Secret for long. Once this problem's resolved I'll delete it. :)</p>
  
  <p><strong>Overview of the problem:</strong> The problem is when I send the last chunk of bytes to Vimeo and then verify the upload, the response returns that the length of all the content is the length of only the last chunk, not all the chunks combined as it should be.</p>
  
  <p><strong>SSCCE Note:</strong> I have my entire SSCCE <a href="http://collabedit.com/8hwwy" rel="noreferrer">here</a>. I put it somewhere else so it can be <strong>C</strong> ompilable. It is NOT very <strong>S</strong> hort (about 300 lines), but hopefully you find it to be <strong>S</strong> elf-contained, and it's certainly an <strong>E</strong> xample!). I am, however, posting the relevant portions of my code in this post.</p>
  
  <p><strong>This is how it works:</strong> When you upload a video to Vimeo via the streaming method (see Upload API documentation <a href="http://vimeo.com/api/docs/upload" rel="noreferrer">here</a> for setup to get to this point), you have to give a few headers: endpoint, content-length, and content-type. The documentation says it ignores any other headers. You also give it a payload of the byte information for the file you're uploading. And then sign and send it (I have a method which will do this using <a href="https://github.com/fernandezpablo85/scribe-java" rel="noreferrer">scribe</a>).</p>
  
  <p><strong>My problem:</strong> Everything works great when I just send the video in one request. My problem is in cases when I'm uploading several bigger files, the computer I'm using doesn't have enough memory to load all of that byte information and put it in the HTTP PUT request, so I have to split it up into 1 MB segments. This is where things get tricky. The documentation mentions that it's possible to "resume" uploads, so I'm trying to do that with my code, but it's not working quite right. Below, you'll see the code for sending the video. <strong>Remember</strong> my SSCCE is <a href="http://collabedit.com/8hwwy" rel="noreferrer">here</a>.</p>
  
  <p><strong>Things I've tried:</strong> I'm thinking it has something to do with the Content-Range header... So here are the things I've tried in changing what the Content-Range header says...</p>
  
  <ul>
  <li>Not adding content range header to the first chunk</li>
  <li><p>Adding a prefix to the content range header (each with a combination of the previous header):</p>
  
  <ul>
  <li>"bytes"</li>
  <li>"bytes " (throws connection error, see the very bottom for the error) --> It appears in the <a href="http://vimeo.com/api/docs/upload" rel="noreferrer">documentation</a> that this is what they're looking for, but I'm pretty sure there are typos in the documentation because they have the content-range header on their "resume" example as: <code>1001-339108/339108</code> when it should be <code>1001-339107/339108</code>. So... Yeah...</li>
  <li>"bytes%20"</li>
  <li>"bytes:"</li>
  <li>"bytes: "</li>
  <li>"bytes="</li>
  <li>"bytes= "</li>
  </ul></li>
  <li><p>Not adding anything as a prefix to the content range header</p></li>
  </ul>
  
  <p>Here's the code:</p>
  
  <pre><code>/**
  * Send the video data
  *
  * @return whether the video successfully sent
  */
  private static boolean sendVideo(String endpoint, File file) throws FileNotFoundException, IOException {
    // Setup File
    long contentLength = file.length();
    String contentLengthString = Long.toString(contentLength);
    FileInputStream is = new FileInputStream(file);
    int bufferSize = 10485760; // 10 MB = 10485760 bytes
    byte[] bytesPortion = new byte[bufferSize];
    int byteNumber = 0;
    int maxAttempts = 1;
    while (is.read(bytesPortion, 0, bufferSize) != -1) {
      String contentRange = Integer.toString(byteNumber);
      long bytesLeft = contentLength - byteNumber;
      System.out.println(newline + newline + "Bytes Left: " + bytesLeft);
      if (bytesLeft &lt; bufferSize) {
        //copy the bytesPortion array into a smaller array containing only the remaining bytes
        bytesPortion = Arrays.copyOf(bytesPortion, (int) bytesLeft);
        //This just makes it so it doesn't throw an IndexOutOfBounds exception on the next while iteration. It shouldn't get past another iteration
        bufferSize = (int) bytesLeft;
      }
      byteNumber += bytesPortion.length;
      contentRange += "-" + (byteNumber - 1) + "/" + contentLengthString;
      int attempts = 0;
      boolean success = false;
      while (attempts &lt; maxAttempts &amp;&amp; !success) {
        int bytesOnServer = sendVideoBytes("Test video", endpoint, contentLengthString, "video/mp4", contentRange, bytesPortion, first);
        if (bytesOnServer == byteNumber) {
          success = true;
        } else {
          System.out.println(bytesOnServer + " != " + byteNumber);
          System.out.println("Success is not true!");
        }
        attempts++;
      }
      first = true;
      if (!success) {
        return false;
      }
    }
    return true;
  }
  
  /**
  * Sends the given bytes to the given endpoint
  *
  * @return the last byte on the server (from verifyUpload(endpoint))
  */
  private static int sendVideoBytes(String videoTitle, String endpoint, String contentLength, String fileType, String contentRange, byte[] fileBytes, boolean addContentRange) throws FileNotFoundException, IOException {
    OAuthRequest request = new OAuthRequest(Verb.PUT, endpoint);
    request.addHeader("Content-Length", contentLength);
    request.addHeader("Content-Type", fileType);
    if (addContentRange) {
      request.addHeader("Content-Range", contentRangeHeaderPrefix + contentRange);
    }
    request.addPayload(fileBytes);
    Response response = signAndSendToVimeo(request, "sendVideo on " + videoTitle, false);
    if (response.getCode() != 200 &amp;&amp; !response.isSuccessful()) {
      return -1;
    }
    return verifyUpload(endpoint);
  }
  
  /**
  * Verifies the upload and returns whether it's successful
  *
  * @param endpoint to verify upload to
  * @return the last byte on the server
  */
  public static int verifyUpload(String endpoint) {
    // Verify the upload
    OAuthRequest request = new OAuthRequest(Verb.PUT, endpoint);
    request.addHeader("Content-Length", "0");
    request.addHeader("Content-Range", "bytes */*");
    Response response = signAndSendToVimeo(request, "verifyUpload to " + endpoint, true);
    if (response.getCode() != 308 || !response.isSuccessful()) {
      return -1;
    }
    String range = response.getHeader("Range");
    //range = "bytes=0-10485759"
    return Integer.parseInt(range.substring(range.lastIndexOf("-") + 1)) + 1;
    //The + 1 at the end is because Vimeo gives you 0-whatever byte where 0 = the first byte
  }
  </code></pre>
  
  <p>Here's the signAndSendToVimeo method:</p>
  
  <pre><code>/**
  * Signs the request and sends it. Returns the response.
  *
  * @param service
  * @param accessToken
  * @param request
  * @return response
  */
  public static Response signAndSendToVimeo(OAuthRequest request, String description, boolean printBody) throws org.scribe.exceptions.OAuthException {
    System.out.println(newline + newline
            + "Signing " + description + " request:"
            + ((printBody &amp;&amp; !request.getBodyContents().isEmpty()) ? newline + "\tBody Contents:" + request.getBodyContents() : "")
            + ((!request.getHeaders().isEmpty()) ? newline + "\tHeaders: " + request.getHeaders() : ""));
    service.signRequest(accessToken, request);
    printRequest(request, description);
    Response response = request.send();
    printResponse(response, description, printBody);
    return response;
  }
  </code></pre>
  
  <p>And here's <strong>some</strong> (an example... All of the output can be found <a href="http://collabedit.com/aqk8n" rel="noreferrer">here</a>) of the output from the printRequest and printResponse methods: <strong>NOTE</strong> This output changes depending on what the <code>contentRangeHeaderPrefix</code> is set to and the <code>first</code> boolean is set to (which specifies whether or not to include the Content-Range header on the first chunk).</p>
  
  <pre><code>We're sending the video for upload!
  
  
  Bytes Left: 15125120
  
  
  Signing sendVideo on Test video request:
      Headers: {Content-Length=15125120, Content-Type=video/mp4, Content-Range=bytes%200-10485759/15125120}
  
  sendVideo on Test video &gt;&gt;&gt; Request
  Headers: {Authorization=OAuth oauth_signature="zUdkaaoJyvz%2Bt6zoMvAFvX0DRkc%3D", oauth_version="1.0", oauth_nonce="340477132", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="5cb447d1fc4c3308e2c6531e45bcadf1", oauth_token="460633205c55d3f1806bcab04174ae09", oauth_timestamp="1334336004", Content-Length=15125120, Content-Type=video/mp4, Content-Range=bytes: 0-10485759/15125120}
  Verb: PUT
  Complete URL: http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d
  
  sendVideo on Test video &gt;&gt;&gt; Response
  Code: 200
  Headers: {null=HTTP/1.1 200 OK, Content-Length=0, Connection=close, Content-Type=text/plain, Server=Vimeo/1.0}
  
  
  Signing verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d request:
      Headers: {Content-Length=0, Content-Range=bytes */*}
  
  verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d &gt;&gt;&gt; Request
  Headers: {Authorization=OAuth oauth_signature="FQg8HJe84nrUTdyvMJGM37dpNpI%3D", oauth_version="1.0", oauth_nonce="298157825", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="5cb447d1fc4c3308e2c6531e45bcadf1", oauth_token="460633205c55d3f1806bcab04174ae09", oauth_timestamp="1334336015", Content-Length=0, Content-Range=bytes */*}
  Verb: PUT
  Complete URL: http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d
  
  verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d &gt;&gt;&gt; Response
  Code: 308
  Headers: {null=HTTP/1.1 308 Resume Incomplete, Range=bytes=0-10485759, Content-Length=0, Connection=close, Content-Type=text/plain, Server=Vimeo/1.0}
  Body: 
  
  
  Bytes Left: 4639360
  
  
  Signing sendVideo on Test video request:
      Headers: {Content-Length=15125120, Content-Type=video/mp4, Content-Range=bytes: 10485760-15125119/15125120}
  
  sendVideo on Test video &gt;&gt;&gt; Request
  Headers: {Authorization=OAuth oauth_signature="qspQBu42HVhQ7sDpzKGeu3%2Bn8tM%3D", oauth_version="1.0", oauth_nonce="183131870", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="5cb447d1fc4c3308e2c6531e45bcadf1", oauth_token="460633205c55d3f1806bcab04174ae09", oauth_timestamp="1334336015", Content-Length=15125120, Content-Type=video/mp4, Content-Range=bytes%2010485760-15125119/15125120}
  Verb: PUT
  Complete URL: http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d
  
  sendVideo on Test video &gt;&gt;&gt; Response
  Code: 200
  Headers: {null=HTTP/1.1 200 OK, Content-Length=0, Connection=close, Content-Type=text/plain, Server=Vimeo/1.0}
  
  
  Signing verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d request:
      Headers: {Content-Length=0, Content-Range=bytes */*}
  
  verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d &gt;&gt;&gt; Request
  Headers: {Authorization=OAuth oauth_signature="IdhhhBryzCa5eYqSPKAQfnVFpIg%3D", oauth_version="1.0", oauth_nonce="442087608", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="5cb447d1fc4c3308e2c6531e45bcadf1", oauth_token="460633205c55d3f1806bcab04174ae09", oauth_timestamp="1334336020", Content-Length=0, Content-Range=bytes */*}
  Verb: PUT
  Complete URL: http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d
  
  4639359 != 15125120
  verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d &gt;&gt;&gt; Response
  Success is not true!
  Code: 308
  Headers: {null=HTTP/1.1 308 Resume Incomplete, Range=bytes=0-4639359, Content-Length=0, Connection=close, Content-Type=text/plain, Server=Vimeo/1.0}
  Body: 
  </code></pre>
  
  <p>Then the code goes on to complete the upload and set video information (you can see that in <a href="http://collabedit.com/8hwwy" rel="noreferrer">my full code</a>).</p>
  
  <p><strong>Edit 2:</strong> Tried removing the "%20" from the content-range and received this error making connection. I must use either "bytes%20" or not add "bytes" at all...</p>
  
  <pre><code>Exception in thread "main" org.scribe.exceptions.OAuthException: Problems while creating connection.
      at org.scribe.model.Request.send(Request.java:70)
      at org.scribe.model.OAuthRequest.send(OAuthRequest.java:12)
      at autouploadermodel.VimeoTest.signAndSendToVimeo(VimeoTest.java:282)
      at autouploadermodel.VimeoTest.sendVideoBytes(VimeoTest.java:130)
      at autouploadermodel.VimeoTest.sendVideo(VimeoTest.java:105)
      at autouploadermodel.VimeoTest.main(VimeoTest.java:62)
  Caused by: java.io.IOException: Error writing to server
      at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:622)
      at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:634)
      at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1317)
      at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
      at org.scribe.model.Response.&lt;init&gt;(Response.java:28)
      at org.scribe.model.Request.doSend(Request.java:110)
      at org.scribe.model.Request.send(Request.java:62)
      ... 5 more
  Java Result: 1
  </code></pre>
  
  <p><strong>Edit 1:</strong> Updated the code and output. Still need help!</p>