Creating a csv file using scrapy I've created a script using Python in association with Scrapy to parse the movie names and its years spread across multiple pages from a torrent site. My goal here is to write the parsed data in a CSV file other than using the built-in command provided by Scrapy, because when I do this:

scrapy crawl  torrentdata -o outputfile.csv -t csv


I get a blank line in every alternate row in the CSV file.

However, I thought to go in a slightly different way to achieve the same thing. Now, I get a data-laden CSV file in the right format when I run the following script. Most importantly I made use of a with statement while creating a CSV file so that when the writing is done the file gets automatically closed. I used crawlerprocess to execute the script from within an IDE. 


  My question: Isn't it a better idea for me to follow the way I tried below?


This is the working script:

import scrapy
from scrapy.crawler import CrawlerProcess
import csv

class TorrentSpider(scrapy.Spider):
    name = "torrentdata"
    start_urls = ["https://yts.am/browse-movies?page={}".format(page) for page in range(2,20)] #get something within list
    itemlist = []

    def parse(self, response):
        for record in response.css('.browse-movie-bottom'):
            items = {}
            items["Name"] = record.css('.browse-movie-title::text').extract_first(default='')
            items["Year"] = record.css('.browse-movie-year::text').extract_first(default='')
            self.itemlist.append(items)

        with open("outputfile.csv","w", newline="") as f:
            writer = csv.DictWriter(f,['Name','Year'])
            writer.writeheader()
            for data in self.itemlist:
                writer.writerow(data)

c = CrawlerProcess({
    'USER_AGENT': 'Mozilla/5.0',   
})
c.crawl(TorrentSpider)
c.start()
By putting the CSV exporting logic into the spider itself, you are re-inventing the wheel and not using all the advantages of Scrapy and its components and, also, making the crawling slower as you are writing to disk in the crawling stage every time the callback is triggered. 

As you mentioned, the CSV exporter is built-in, you just need to yield/return items from the parse() callback:

import scrapy


class TorrentSpider(scrapy.Spider):
    name = "torrentdata"
    start_urls = ["https://yts.am/browse-movies?page={}".format(page) for page in range(2,20)] #get something within list

    def parse(self, response):
        for record in response.css('.browse-movie-bottom'):
            yield {
                "Name": record.css('.browse-movie-title::text').extract_first(default=''),
                "Year": record.css('.browse-movie-year::text').extract_first(default='')
            }


Then, by running:

scrapy runspider spider.py -o outputfile.csv -t csv


(or the crawl command)

you would have the following in the outputfile.csv:

Name,Year
"Faith, Love &amp; Chocolate",2018
Bennett's Song,2018
...
Tender Mercies,1983
You Might Be the Killer,2018