problem with deserialization xml to objects - unwanted split by special chars
Question: I try to deserialize xml to objects, and i met a problem with encoding of
various items in xml tree.

**XML example:**

    
    
    <?xml version="1.0" encoding="utf-8"?>
    <results>
      <FlightTravel>
        <QuantityOfPassengers>6</QuantityOfPassengers>
        <Id>N5GWXM</Id>
        <InsuranceId>330992</InsuranceId>
        <TotalTime>3h 00m</TotalTime>
        <TransactionPrice>540.00</TransactionPrice>
        <AdditionalPrice>0</AdditionalPrice>
        <InsurancePrice>226.56</InsurancePrice>
        <TotalPrice>9561.31</TotalPrice>
        <CompanyName>XXXXX</CompanyName>
        <TaxID>111-11-11-111</TaxID>
        <InvoiceStreet>Jagiellońska</InvoiceStreet>
        <InvoiceHouseNo>8</InvoiceHouseNo>
        <InvoiceZipCode>Jagiellońska</InvoiceZipCode>
        <InvoiceCityName>Warszawa</InvoiceCityName>
        <PayerStreet>Jagiellońska</PayerStreet>
        <PayerHouseNo>8</PayerHouseNo>
        <PayerZipCode>11-111</PayerZipCode>
        <PayerCityName>Warszawa</PayerCityName>
        <PayerEmail>no-reply@xxxx.pl</PayerEmail>
        <PayerPhone>123123123</PayerPhone>
        <Segments>
          <Segment0>
            <DepartureAirport>WAW</DepartureAirport>
            <DepartureDate>śr. 06 lip</DepartureDate>
            <DepartureTime>07:50</DepartureTime>
            <ArrivalAirport>VIE</ArrivalAirport>
            <ArrivalDate>śr. 06 lip</ArrivalDate>
            <ArrivalTime>09:15</ArrivalTime>
          </Segment0>
          <Segment1>
            <DepartureAirport>VIE</DepartureAirport>
            <DepartureDate>śr. 06 lip</DepartureDate>
            <DepartureTime>10:00</DepartureTime>
            <ArrivalAirport>SZG</ArrivalAirport>
            <ArrivalDate>śr. 06 lip</ArrivalDate>
            <ArrivalTime>10:50</ArrivalTime>
          </Segment1>
        </Segments>
      </FlightTravel>
    </results>
    

**XML Deserialization function in python:**

    
    
    # -*- coding: utf-8 -*-
    
    from lxml import etree
    import codecs
    
    class TitleTarget(object):
        def __init__(self):
            self.text = []
        def start(self, tag, attrib):
            self.is_title = True #if tag == 'Title' else False
        def end(self, tag):
            pass
        def data(self, data):
            if self.is_title:
                self.text.append(data)
        def close(self):
            return self.text
    
    parser = etree.XMLParser(target = TitleTarget())
    
    infile = 'Flights.xml'
    results = etree.parse(infile, parser)
    
    out = open('wynik.txt', 'w')
    out.write('\n'.join(results))
    out.close()
    

**Output:**

['6', 'N5GWXM', '330992', '3h 00m', '540.00 ', '0', '226.56', '9561.31',
'XXXXX', '111-11-11-111', 'Jagiello', 'ń', 'ska', '8', 'Jagiello', 'ń', 'ska',
'Warszawa', 'Jagiello', 'ń', 'ska', '8', '11-111', 'Warszawa', 'no-
reply@xxxx.pl', '123123123', 'WAW', 'ś', 'r. 06 lip', '07:50', 'VIE', 'ś', 'r.
06 lip', '09:15', 'VIE', 'ś', 'r. 06 lip', '10:00', 'SZG', 'ś', 'r. 06 lip',
'10:50']

In item '**Jagiellońska** ' is special char '**ń** '. When parser appending
data to array then char 'ń' is some king of split character and my question is
why this is happening? Rest of items are appending to array correctly. In item
'śr 06.lip' is exactly the same situation.


Answer: The problem is that the `data` method of your target class may be called more
than once per element. This may happen if the feeder crosses a block boundary,
for example. Looks like it can also happen when it hits a non-ASCII character.
This is ancient legend. I can't find where this is documented. However if you
change your target class to something like the following, it will work. I have
tested it on your data.

    
    
    class TitleTarget(object):
        def __init__(self):
            self.text = []
        def start(self, tag, attrib):
            self.is_title = True #if tag == 'Title' else False
            if self.is_title:
                self.text.append(u'')
        def end(self, tag):
            pass
        def data(self, data):
            if self.is_title:
                self.text[-1] += data
        def close(self):
            return self.text
    

To get a better grasp of what your output is like, do `print repr(results)`
after the parse call. You should now see such pieces of unsplit text as

    
    
    u'Jagiello\u0144ska\n    '
    u'\u015br. 06 lip\n        '
    

