Removing brackets from Scrapy json output
Question: Final part of my code is to load data from my scrapy pipeline to my pandas
dataframe.

A sample result is as below:

    
    
    {"Message": ["\r\n", " Profanity directed toward staff.  ", "\r\n Profanity directed toward warden ", "  \r\n  "], "Desc": "https://www.tdcj.state.tx.us/death_row/dr_info/nicholsjoseph.jpg"}
    

When loaded to the dataframe the [] brackets are stil in there with "\r\n". A
quick search shows me that this is because of encoding and it is quite common
for scrapping.

Can anybody give me an idea of a pythonic way to get a rather cleaner output?

I am expecting something like

    
    
    {"Message: "Profanity directed toward staff. Profanity directed toward warden", "Desc": "https://www.tdcj.state.tx.us/death_row/dr_info/nicholsjoseph.jpg"}
    

Edited to add item class and spider:

Item.py

    
    
    from scrapy.item import Item, Field
    from scrapy.loader import ItemLoader
    from scrapy.loader.processors import TakeFirst, MapCompose, Join
    
    
    class DeathItem(Item):
    
        firstName = Field()
        lastName = Field()
        Age = Field()
        Date = Field()
        Race = Field()
        County = Field()
        Message = Field(
            input_processor=MapCompose(unicode.strip),
            output_processor=Join())
        Desc = Field()
        Mid = Field()
    

spider.py

    
    
    from urlparse import urljoin
    import scrapy
    from texasdeath.items import DeathItem
    
    
    class DeathSpider(scrapy.Spider):
        name = "death"
        allowed_domains = ["tdcj.state.tx.us"]
        start_urls = [
            "https://www.tdcj.state.tx.us/death_row/dr_executed_offenders.html"
        ]
        def parse(self, response):
            sites = response.xpath('//table/tbody/tr')
            for site in sites:
                item = DeathItem()
                item['Mid'] = site.xpath('td[1]/text()').extract()
                item['firstName'] = site.xpath('td[5]/text()').extract()
                item['lastName'] = site.xpath('td[4]/text()').extract()
                item['Age'] = site.xpath('td[7]/text()').extract()
                item['Date'] = site.xpath('td[8]/text()').extract()
                item['Race'] = site.xpath('td[9]/text()').extract()
                item['County'] = site.xpath('td[10]/text()').extract()
    
                url = urljoin(response.url, site.xpath("td[2]/a/@href").extract_first())
                urlLast = urljoin(response.url, site.xpath("td[3]/a/@href").extract_first())
    
                if url.endswith(("jpg","no_info_available.html")):
                    item['Desc'] = url
                    if urlLast.endswith("no_last_statement.html"):
                        item['Message'] = "No last statement"
                        yield item
                    else:
                        request = scrapy.Request(urlLast, meta={"item" : item}, callback =self.parse_details2)
                        yield request
                else:        
                    request = scrapy.Request(url, meta={"item": item,"urlLast" : urlLast}, callback=self.parse_details)
                    yield request
    
        def parse_details(self, response):
            item = response.meta["item"]
            urlLast = response.meta["urlLast"]
            item['Desc'] = response.xpath("//*[@id='body']/p[3]/text()").extract()
            if urlLast.endswith("no_last_statement.html"):
                item["Message"] = "No last statement"
                return item
            else:
                request = scrapy.Request(urlLast, meta={"item": item}, callback=self.parse_details2)
                return request
    
        def parse_details2(self, response):
            item = response.meta["item"]
            item['Message'] = response.xpath("//div/p[contains(., 'Last Statement:')]/following-sibling::node()/descendant-or-self::text()").extract()
            return item
    

I basically want a output in clean text to be loaded to my pandas dataframe.
However all the unwanted characters such as: [],\r\n\t to be left out.

Basically for the data to appear such as in the web.


Answer: You need to tweak the way the extracted item field is post-processed. For that
`Scrapy` has the [Item
Loaders](http://doc.scrapy.org/en/latest/topics/loaders.html) with input and
output processors. In your case, you need the `Join()` and
`MapCompose(unicode.strip)`:

    
    
    from scrapy.loader import ItemLoader
    from scrapy.loader.processors import TakeFirst, MapCompose, Join
    
    class MyItemLoader(ItemLoader):
        default_output_processor = TakeFirst()
    
        message_in = MapCompose(unicode, unicode.strip)
        message_out = Join()
    

