Title: Scrapy on AWS EC2 : where to write the items?
Tags: amazon-ec2;scrapy;scrapyd
Question: I have a working spider on my local machine, which writes items to a local postgres database.
I am now trying to run the same spider through scrapyd on an EC2 instance. This obviously won't work, because the code (models, pipelines, settings files) refers to a database on my local machine.
Which adaptations should I implement to make this work ?

Here is the accepted answer: Found it, answer was easier than i thought. In the settings.py file, delete the settings for ITEM_PIPELINES and DATABASE. After deletion, deploy the project through scrapyd on EC2.
By default, items will now be written as JSON-lines. This can be overridden with FEED_FORMAT and FEED_URI :
sudo curl http:/xxxxxxxxx.us-west-2.compute.amazonaws.com:6800/schedule.json -d project=xxxxxxxxxx -d spider=xxxxxxxxx -d setting=FEED_URI=/var/lib/scrapyd/items/xxxxxxxxxx.csv -d setting=FEED_FORMAT=csv
Comment for this answer: you can also you the RDS service if you still want to store the items to a DB.
Comment for this answer: Did exactly that now, using a Postgres instance on RDS. And for free !


{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 0, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2014:00:11/18/14', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<amazon-ec2><scrapy><scrapyd>', 'Title': 'Scrapy on AWS EC2 : where to write the items?', 'ViewCount': 1159, 'id': 27006123, 'user_AboutMe': '<p>Data sciencist</p>\n', 'user_AccountId': 5239720.0, 'user_DisplayName': 'S Leon', 'user_Reputation': 311.0, 'user_Views': 48.0, 'user_id': 4187181.0}}