eBay LMS uploadFile - Cannot specify file with valid format
Question: I am attempting to use the eBay Large Merchant Services API to upload calls in
bulk. This API and documentation are notoriously bad (I found a blog post that
explains it and rails against it [here](http://medium.com/on-coding/dont-
write-your-api-this-way-a1b745078b94)). The closest I can find to a working
Python solution is based on
[this](http://stackoverflow.com/questions/8482135/http-post-request-and-
headers-with-mime-attachments-multipart-related-and-xop) Question/Answer and
his related [Github](https://github.com/wrhansen/eBay-LMS-API) version which
is no longer maintained.

I have followed these examples very closely and as best I can tell they never
fully worked in the first place. I emailed the original author and he says it
was never put in production. However it should be very close to working.

All of my attempts result in a **error 11 Please specify a File with Valid
Format** message.

I have tried various ways to construct this call, using both the github
method, or a method via the email.mime package, and via Requests library, and
the http.client library. The output below is what I believe is the closest I
have to a what it should be. The file attached is using the Github's
[sample](https://github.com/wrhansen/eBay-LMS-
API/blob/master/examples/AddItem.xml) XML file, and it is read and gzipped
using the [same methods](https://github.com/wrhansen/eBay-LMS-
API/blob/master/lmslib.py#L326-L346) as in the Github repo.

Any help on this would be appreciated. I feel like I have exhausted my
possibilities.

    
    
        POST https://storage.sandbox.ebay.com/FileTransferService
        X-EBAY-SOA-SERVICE-VERSION: 1.1.0
        Content-Type: multipart/related; boundary=MIME_boundary; type="application/xop+xml"; start="<0.urn:uuid:86f4bbc4-cfde-4bf5-a884-cbdf3c230bf2>"; start-info="text/xml"
        User-Agent: python-requests/2.5.1 CPython/3.3.4 Darwin/14.1.0
        Accept: */*
        Accept-Encoding: gzip, deflate
        X-EBAY-SOA-SECURITY-TOKEN: **MYSANDBOXTOKEN**
        X-EBAY-SOA-SERVICE-NAME: FileTransferService
        Connection: keep-alive
        X-EBAY-SOA-OPERATION-NAME: uploadFile
        Content-Length: 4863
    
    
        --MIME_boundary
        Content-Type: application/xop+xml; charset=UTF-8; type="text/xml; charset=UTF-8"
        Content-Transfer-Encoding: binary
        Content-ID: <0.urn:uuid:86f4bbc4-cfde-4bf5-a884-cbdf3c230bf2>
    
        <uploadFileRequest xmlns:sct="http://www.ebay.com/soaframework/common/types" xmlns="http://www.ebay.com/marketplace/services">
        <taskReferenceId>50009042491</taskReferenceId>
        <fileReferenceId>50009194541</fileReferenceId>
        <fileFormat>gzip</fileFormat>
        <fileAttachment>
        <Size>1399</Size>
        <Data><xop:Include xmlns:xop="http://www.w3.org/2004/08/xop/include" href="cid:urn:uuid:1b449099-a434-4466-b8f6-3cc2d59797da"/></Data>
        </fileAttachment>
        </uploadFileRequest>
    
    
        --MIME_boundary
        Content-Type: application/octet-stream
        Content-Transfer-Encoding: binary
        Content-ID: <urn:uuid:1b449099-a434-4466-b8f6-3cc2d59797da>
    
        b'\x1f\x8b\x08\x08\x1a\x98\xdaT\x02\xffuploadcompression.xml\x00\xcdW[W\xdb8\x10~\x0e\xbfB\x9b}\x858\xd0\xb4\x14\x8e\xebnn\x94l\xe3\x96M\xccv\xfb\xd4#\xec!\xd1V\x96\\I\x06\xfc\xefw$\xdbI\x9cK\x0f\xec\xbe,\xe5P{f\xbe\xb9i\xe6\xb3\xed\xbf\x7fJ9y\x00\xa5\x99\x14\xef\xda\xa7\x9dn\x9b\x80\x88e\xc2\xc4\xe2]\xfb6\xba:y\xdb~\x1f\x1c\xf9\x83\x9c\x7f\x1fQC\xc7O\xf1\x92\x8a\x05\xcc\xe0G\x0e\xdah\x82p\xa1\xdf\xb5s%.\xe1\x8e\x16\x974c\xfa\x12\x06\xd3\x01\xd50\x94i&\x05\x08\xa3\xdb\xc1Q\xcb\xbf\x06\x9a\x80\xc2\xab\x96\xffg\x1908\xef\x9d\xfb^}c\x15sf`2\nN\xbb]\xdf\xab\xae\x11\xe9\xad\xa1-\xbf\x9f$\x13\x03i\x95\xc1\x0b\x12 \x04\xd1c\xa5\xa4\x9ab\t9]@\x00\xe2\xdb\xed\xdc\xf7\x9a\xc2\xca\xf2\x0bU\x02\xbb0\x85\x07\xe0\xc15[,}\xaf!\xaa\xcc\x0e\x95\xd2\xf2C\xd0\x1a\xfda\t\x11&\x8a8\xf2\t\x1e\xc9\xdc({y%UJ\x8d\xef\xad\x8d\x10\x83\x0e}[\x9b\xc3\xb7\xfc\x88\x19\x0e\xc1\xf5\xefC2\xffz\x12\xf6\xff\xc2\xff\xec\xdf\xc9\x84\x9c\x91\xeb\xf14\x1cG\xa4\xff)\xba\x9e\xf5\x87\x93hLB/\x1c\x8f&\xb7\xa1\xef\x95\xb8\xd2\xc7\x08t\xacXflV\xd1\x92i\x82\xbf\x94\xc4Rr\xb2\x04\x9e\x829&\xccX\xe1\x9d\xa2"!\x023\xbc+\x88Y\x02y\xa4\xc5/\xbe\xb7\x89/=\xde(\x96RU\x0c\xa9\x81\x85TE)m\xf9\xf5=V\xf2\xe6\xbcw\xe1{\x1b\x82\x12\xe8\xedE\xfasC\x95AU\x0c\xc1\xc5E\xe7\x02\x91\x1b\x92\xd2d(E\xc2l\n\xe5h\xe0llJ\x8e\x1a\xf1C\x9ae\xd8\xe0>\xe7\xf2\x11\x92 R9\xacs\xd9R\xd6\xdesa0\x1d;\t\xf5u\xa5\xc9\x95\xc2m\xb0\xaa\x11\xea\xea\xbb\xaa\xb3Lg\xd4\xc4\xcb\x88\xa5\x10\xd2\xa7\xe0\x156kKT\x1aN\x99;\xfdQ\xae\xa8k\xe3\x88\x16\xfa\xdb)\x16\xb1\xadh\x98GE\x06\xc1\x15{\x82\xc4u\xc2\x8e\xc5\n\xe1t\xd5i\xd0"\xc5\x01\x0f\xc1,e\xa2\x03\xbc\xbd\xa1\x1c[\xdd\x14\xaflQ9N)\xe3\xb8D\n\'/\xb0+\xf3\x9bb\xb8\\:a:\xb6\xd5wb\x99:\x07\xdb\xb6\x95\x13\x16\x9b\\\xc1\x08\x0c\xaat}\xfa\x95\xf4v6\r\x96\xc6d\x97\x9e\x07\x9d]\xb7\x1e\x9e\xff\x02\xb4\x87#\xedu\xcf\xce\xce\xbbo\xbd\xd7\xe7oN^\xbf9\xfd\xa6\xd3\xce\xdf\xd9\x02\xe3\xae\x1d\xd5S\xb3\'\xa0\x7f#\xb5\xa1|(\x13\x08z\x17\xbd\xb3\x1e\x9a\xad%\xa5\xc9\x1f9\x15\x86\x99"\xb0\xad^\xdd\x94\xba\x19\xa0Kq#9\x8bW\x03<\x83\xfb\\$\x9f\xcbQ\xafy\xce\xf7\x1a\xe2\x86\xe9\x8e\xd1Zm\xbd\xeb/\xcc,\x99\xa8\x90\xe5\xa1\xf7\xac\xe9\xaer\x1f.8\xed\x11\x0b\xdaBl\xd9\xf6\xe3\x182\x03u~[\xd2\x15v\xcbl\xbf\x8f\x1aM\x0e\xc2k\xe0\x0e)\xb4\xeaV \xb9( \x19\xa8\x94\x19\x04\x1c\x13x\xb2P"\x05\x01\x0e\xb1QR\xb0\x18\x19\x07R}L,\xe1\xa49r\xf8\x1d\x10&p\x9dqK\x13\xf2\xe8\xea$X~\x82\xe5\x13yO\x14\xc4\x80\xd1:$\x04e\xc3\xe0H\xc1\x06\xd0\x91V\\\x13\x82\xc3\x13\xca9\xc9h\xfc\x9d.p]\x8eIJENy\x152\xa3\x98\xdf\xa3T\xdf\x11khl\x9c0\x17\x94\x1bP\x90tH$m\xd6\xae\x9cc\x92q\xc0\x07\x89u\xefLc\x8c*SPD\x83z\xc0\xb5$F\x96\xe9=\x00\xd2\xea,\xec\xff\xea\xbc\xc9\\\xad|\x90{\xa4\xfa\x0e\x19O\xc7\xc3h\xf6\xf9\xd3dH\x90\xad\xc3\xf91Ir\x07G\xaee\xe8/\x83\x98QN\x04\xb5\xc3Nb*\x84t\xf5\xd5n\x12\xeb\x07\x9d\x17\x18\x8fj\xac\xd3\xc6\xb1\xcd\xd6\x92\x03/0\xc3\x07\x9b>I\x18\xe6c\xb8\xe5p%\xf3\xc5\xb2\xf2\x8f\x1b\x8c\x11\x8c\xcd\xd36\xe3\x9e\xba\xa5R\xbaS\x1d\xe9.\xd1#3/\x99\xa3\xcb!\xae\xd6\re\xc9\xa0\xa8\xe6g\x90\x17\xa0\x90\xa7\x0f\xe9\x0f\xe2\x0f#\xebm\xdf\xdd\xcc\x95\x9b\r\x06X\xc9\xe6\xe51\x94qKr\xd8\xd6\x05\xf5}I\x86\xf8p\x11\tU\xc9:\x89\xda\xae\x19\xad\x92\xda\x0c\x83n\xa7\xbbc\xee\x14{!\xc8\x97n\x12-\x1b\x1d\x00o\x99\xecu\x83\xb4/\x95\xe3\xaf\x1d\xf8JU\x02\xc7O\x19\xa0?H\xeaJ\xeeq\xd6\x91\x95v\xe4\xae=W\n\xa0\xf6\x17\x18\xf7xl\x88l{\xbd\xc3\xfd\xf5\'\x02\xf7D\xd02\xfd\xbdv\x07\x8e\xa1j|\x03\xbf\xff\x14\xf6\x1d\x02\xae^\xf9\xf8\x9d\x8c\xf0\xc5t>j\x07g\xf8\xb6\xf0\xf6\xf0\xb9\x1c\xec\xe7\xd9\xcf\xfb\xe9p\x91\x9c\xca\xb8|*\x0f\xfb\xa5\xfd\x86\xc8\xb5\xe8y}\xf8\xff\xb4\xeb\xd5\xbfl\xd7\xab\x97\xb5\xab\x8f\xec\xc8b\xaa\xf75m\xc7x\x9c+\x99\xc1\xb3L\xfb\x9a\xd1\xe7\x19\xde\xfe\xa7\xf3\xaa5\xe5\xfb\x17\xb7\xef\xe8\r\xd1\x91[\xb8\x98\xe7\tlE\x99~\xb4+\xb7Os\x183\x19\xbd\x1c\x13~}9\xe6\xc3\xe0\xe5\x98\xf9\x87\x9fa\xe6\xc09\xa8\xbdz}\xa3\xe0\x1e\xec\xf4AE0\xcf4>\xda`\x9e\xe6\xfb\x9e\xfd\x16\x0c`@\x8bP\x1a\xa9t\xf9qX\xeb>\xde\xba\xaf\x02\xfc9\xb1\xbb\x8d\xb7n.\xbc\xfaS\xca\xf7\x9a\xdf\x8cVv\xe4{\x87\xbeiQ]\xff\xfb\x07\xe0\xf2E>\x1f\x0f\x00\x00'
        --MIME_boundary--
    

**EDIT:** Showing the methods I have tried to actually generate the binary
data:

    
    
    with open('testpayload.xml', mode='rt') as myfile:
            full_xml = myfile.read()  # replacing my full_xml with testpayload.xml, from Github repo
    
        # METHOD 1: Simple gzip.compress()
    
        payload = gzip.compress(bytes(full_xml, 'UTF-8'))
    
        # METHOD 2: Implementing http://stackoverflow.com/a/8507012/1281743 which should produce a gzip-compatible string with header
    
        import io
        out = io.BytesIO()
        with gzip.GzipFile(fileobj=out, mode="w") as f:
            f.write(bytes(full_xml, 'UTF-8'))
        payload = out.getvalue()
    
        # METHOD 3: Simply reading in a pre-compressed version of same testpayload.xml file. Compressed on commandline with 'gzip testpayload.xml'
    
        with open('testpayload.xml.gz', mode='rb') as myfile:
            payload = myfile.read()
    
        # METHOD 4: Adoping the _generate_date() function from github eBay-LMS-API repo http://goo.gl/YgFyBi
    
        mybuffer = io.BytesIO()
        fp = open('testpayload.xml', 'rb')  # This file is from https://github.com/wrhansen/eBay-LMS-API/
        # Create a gzip object that reads the compression to StringIO buffer
        gzipbuffer = gzip.GzipFile('uploadcompression.xml.gz', 'wb', 9, mybuffer)
        gzipbuffer.writelines(fp)
        gzipbuffer.close()
        fp.close()
    
        mybuffer.seek(0)
        payload = mybuffer.read()
        mybuffer.close()
    
        # DONE: send payload to API call
    
        r = ebay.fileTransfer_upload_file(file_id='50009194541', task_id='50009042491', gzip_data=payload)
    

**EDIT2, THE QUASI-SOLUTION:** Python was not actually sending binary data,
but rather a string representation of binary, which did not work. I ended up
having to base64-encode the file first. I had tried that before, but have been
waived off of that by a suggestion on the eBay forums. I just changed one of
the last headers to `Content-Transfer-Encoding: base64` and I got a success
response.

**EDIT3, THE REAL SOLUTION:** Base 64 encoding only solved my problems with
the example XML file. I found that my own XML payload was NOT sending
successfully (same error as before), but that the script on Github could send
that same file just fine. The difference was that the Github script is Python2
and able to mix string and binary data without differentiation. To send binary
data in Python3 I just had to make a few simple changes to change all the
strings in the request_part and binary_part into bytes, so that the bytes from
the gzip payload would concatenate with it. Looks like this:

    
    
    binary_part = b'\r\n'
    binary_part += b'--MIME_boundary\r\n'
    binary_part += b'Content-Type: application/octet-stream\r\n'
    binary_part += b'Content-Transfer-Encoding: binary\r\n'
    binary_part += b'Content-ID: <' + URN_UUID_ATTACHMENT.encode('utf-8') + b'>\r\n\r\n'
    binary_part += gzip_data + b'\r\n'
    binary_part += b'--MIME_boundary--'
    

So, no base64 encoding, just needed to figure out how to make Python send real
binary data. And the original github repo **does** work as advertised, in
python2.


Answer: So after scanning all my code about this, it looks like it's setup right. So
if your error is a good error then it means your binary data is wrong. Can you
show how you're gzipping and then reading that data back in? Here's what I'm
doing

    
    
    $dir = '/.../ebayUpload.gz';
    if(is_file($dir)) unlink($dir);
    $gz = gzopen($dir,'w9');
    gzwrite($gz, $xmlFile);
    chmod($dir, 0777);
    gzclose($gz);
    
    // open that file as data;
    $handle = fopen($dir, 'r');
    $fileData = fread($handle, filesize($dir));
    fclose($handle);
    

And in your case the $xmlFile should be the string found in
<https://github.com/wrhansen/eBay-LMS-API/blob/master/examples/AddItem.xml>

I should add that this is how I'm using $fileData

    
    
     $binaryPart = '';
     $binaryPart .= "--" . $boundry . $CRLF;
     $binaryPart .= 'Content-Type: application/octet-stream' . $CRLF;
     $binaryPart .= 'Content-Transfer-Encoding: binary' . $CRLF;
     $binaryPart .= 'Content-ID: <urn:uuid:'.$uuid_attachment.'>' . $CRLF . $CRLF;
     $binaryPart .= $fileData . $CRLF;
     $binaryPart .= "--" . $boundry . "--";
    

