Read/Write Sequence Files Containing Thrift Records Using Hadoop Streaming with Python
Question: I would like to Read/Write sequence files containing Thrift records using
Hadoop Streaming with Python. I have looked at the following and its seems
this is possible after HADOOP-1722 but if someone has done this already and
can give an example, that would be great.

<http://mojodna.net/2013/12/27/binary-streaming-with-hadoop-and-nodejs.html>

[How to use "typedbytes" or "rawbytes" in Hadoop
Streaming?](http://stackoverflow.com/questions/15171514/how-to-use-typedbytes-
or-rawbytes-in-hadoop-streaming)

<http://static.last.fm/johan/huguk-20090414/klaas-hadoop-1722.pdf>

<https://issues.apache.org/jira/browse/HADOOP-1722>

The key is to be able to read thrift objects from stdin in Python.


Answer: I finally got this done with [Hadoopy](http://www.hadoopy.com/en/latest/).

This is my simple Thrift object.

    
    
    struct Test {
        1: required string foo;
        2: required string bar;
    }
    

I generated the python definitions using the command line tool, and zipped the
directory. Then I generated some data and pushed it into HDFS using loadtb.

Here is the code to de-serialize the data and simply write it out as a string.

    
    
    import hadoopy
    from thrift.protocol import TBinaryProtocol
    from thrift.TSerialization import deserialize
    import sys
    
    class Mapper(object):
        def __init__(self):
            sys.path.append('lib/test.zip/gen-py')
            from Test.ttypes import Test
            self.protocol_factory = TBinaryProtocol.TBinaryProtocolFactory
            self.test = Test()
        def map(self, key, value):
            deserialize(self.test, key, protocol_factory=self.protocol_factory())
            yield 1, self.test.foo + " "  + self.test.bar
    
    if __name__ == '__main__':
        hadoopy.run(Mapper)
    

This is the driver script.

    
    
    import hadoopy
    
        hadoopy.launch('/data/fuscala', '/data/fuscala-output', 'fuscala1.py', remove_output = True, files = ["test.zip"])
        data = hadoopy.readtb('/data/fuscala-output')
        for i, j in data:
            print i,j
    

