We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
ItemLoaders can break if instantiated with pre-populated items
Before I start, I know item loaders have been a big source of discussion for a long time; I'm only opening this issue because the latest release breaks some of our spiders.

In one of our projects, our [Autounit](https://github.com/scrapinghub/scrapy-autounit) tests fail under 1.7.1 due to some item loaders which are created from partially populated items. I suspect the relevant change is #3819 (which BTW I think inadvertently closes #3046).
Personally I think a better approach here would be something _closer_ to the solution proposed in #3149, although not exactly the same.

Consider the following:

```python
In [1]: import scrapy

In [2]: scrapy.__version__
Out[2]: '1.6.0'

In [3]: from scrapy.loader import ItemLoader
   ...: lo = ItemLoader(item={'key': 'value'})
   ...: lo.add_value('key', 'other value')
   ...: print(lo.load_item())
{'key': ['other value']}
```

```python
In [1]: import scrapy

In [2]: scrapy.__version__
Out[2]: '1.7.1'

In [3]: from scrapy.loader import ItemLoader
   ...: lo = ItemLoader(item={'key': 'value'})
   ...: lo.add_value('key', 'other value')
   ...: print(lo.load_item())
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-6aa64a41edb1> in <module>
      1 from scrapy.loader import ItemLoader
      2 lo = ItemLoader(item={'key': 'value'})
----> 3 lo.add_value('key', 'other value')
      4 print(lo.load_item())

~/venv-temporal/lib/python3.6/site-packages/scrapy/loader/__init__.py in add_value(self, field_name, value, *processors, **kw)
     77                 self._add_value(k, v)
     78         else:
---> 79             self._add_value(field_name, value)
     80
     81     def replace_value(self, field_name, value, *processors, **kw):

~/venv-temporal/lib/python3.6/site-packages/scrapy/loader/__init__.py in _add_value(self, field_name, value)
     93         processed_value = self._process_input_value(field_name, value)
     94         if processed_value:
---> 95             self._values[field_name] += arg_to_iter(processed_value)
     96
     97     def _replace_value(self, field_name, value):

TypeError: must be str, not list
```
I'm not directly opening a PR because I think this needs discussion. What if we changed
```python
for field_name, value in item.items():
    self._values[field_name] = self._process_input_value(field_name, value)
```

to

```python
for field_name, value in item.items():
    self._add_value(field_name, value)
```

which calls `arg_to_iter` internally?

With that change, the following happens which is more reasonable IMHO:
```python
In [3]: from scrapy.loader import ItemLoader 
   ...: lo = ItemLoader(item={'key': 'value'}) 
   ...: lo.add_value('key', 'other value') 
   ...: print(lo.load_item())                                                                                                                                                                                                                 
{'key': ['value', 'other value']}
```

Looking forward to reading your thoughts on the matter

/cc @Gallaecio @kmike @andrewbaxter @fcanobrash @sortafreel
`ItemLoader` fields initialized from `item` are reprocessed
### Description

#3804 introduced a bug where `ItemLoader` fields are reprocessed.
Related #3897.

### Steps to Reproduce

```python
from pprint import pprint

from scrapy import Field, Item
from scrapy.loader import ItemLoader
from scrapy.loader.processors import TakeFirst


class X(Item):
    x = Field(output_processor=TakeFirst())


loader = ItemLoader(X())
loader.add_value("x", ["value1", "value2"])
x = loader.load_item()
pprint(x)
# {'x': 'value1'}

pprint(ItemLoader(x).load_item())
# {'x': 'v'}
```

**Expected behavior:** `ItemLoader` initialized from the `x` item does not reprocess its fields and  loads `{'x': 'value1'}`.

**Actual behavior:** `ItemLoader` initialized from the `x` item reprocesses its fields and  loads `{'x': 'v'}`.

### Versions

```
Scrapy       : 1.7.3
lxml         : 4.4.1.0
libxml2      : 2.9.9
cssselect    : 1.1.0
parsel       : 1.5.2
w3lib        : 1.21.0
Twisted      : 19.7.0
Python       : 3.6.5 (default, May  3 2018, 10:08:28) - [GCC 5.4.0 20160609]
pyOpenSSL    : 19.0.0 (OpenSSL 1.1.1c  28 May 2019)
cryptography : 2.7
Platform     : Linux-4.4.0-127-generic-x86_64-with-LinuxMint-18.1-serena
```

### Additional context

Here's the behavior of the previous version:

```
Scrapy       : 1.6.0
lxml         : 4.4.0.0
libxml2      : 2.9.9
cssselect    : 1.0.3
parsel       : 1.5.1
w3lib        : 1.20.0
Twisted      : 19.7.0
Python       : 3.6.5 (default, May  3 2018, 10:08:28) - [GCC 5.4.0 20160609]
pyOpenSSL    : 19.0.0 (OpenSSL 1.1.1c  28 May 2019)
cryptography : 2.7
Platform     : Linux-4.4.0-127-generic-x86_64-with-LinuxMint-18.1-serena
```

```python
# {'x': 'value1'}
# {'x': 'value1'}
```

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `scrapy/loader/__init__.py`
Content:
```
1 """Item Loader
2 
3 See documentation in docs/topics/loaders.rst
4 
5 """
6 from collections import defaultdict
7 import six
8 
9 from scrapy.item import Item
10 from scrapy.selector import Selector
11 from scrapy.utils.misc import arg_to_iter, extract_regex
12 from scrapy.utils.python import flatten
13 
14 from .common import wrap_loader_context
15 from .processors import Identity
16 
17 
18 class ItemLoader(object):
19 
20     default_item_class = Item
21     default_input_processor = Identity()
22     default_output_processor = Identity()
23     default_selector_class = Selector
24 
25     def __init__(self, item=None, selector=None, response=None, parent=None, **context):
26         if selector is None and response is not None:
27             selector = self.default_selector_class(response)
28         self.selector = selector
29         context.update(selector=selector, response=response)
30         if item is None:
31             item = self.default_item_class()
32         self.context = context
33         self.parent = parent
34         self._local_item = context['item'] = item
35         self._local_values = defaultdict(list)
36         # Preprocess values if item built from dict
37         # Values need to be added to item._values if added them from dict (not with add_values)
38         for field_name, value in item.items():
39             self._values[field_name] = self._process_input_value(field_name, value)
40 
41     @property
42     def _values(self):
43         if self.parent is not None:
44             return self.parent._values
45         else:
46             return self._local_values
47 
48     @property
49     def item(self):
50         if self.parent is not None:
51             return self.parent.item
52         else:
53             return self._local_item
54 
55     def nested_xpath(self, xpath, **context):
56         selector = self.selector.xpath(xpath)
57         context.update(selector=selector)
58         subloader = self.__class__(
59             item=self.item, parent=self, **context
60         )
61         return subloader
62 
63     def nested_css(self, css, **context):
64         selector = self.selector.css(css)
65         context.update(selector=selector)
66         subloader = self.__class__(
67             item=self.item, parent=self, **context
68         )
69         return subloader
70 
71     def add_value(self, field_name, value, *processors, **kw):
72         value = self.get_value(value, *processors, **kw)
73         if value is None:
74             return
75         if not field_name:
76             for k, v in six.iteritems(value):
77                 self._add_value(k, v)
78         else:
79             self._add_value(field_name, value)
80 
81     def replace_value(self, field_name, value, *processors, **kw):
82         value = self.get_value(value, *processors, **kw)
83         if value is None:
84             return
85         if not field_name:
86             for k, v in six.iteritems(value):
87                 self._replace_value(k, v)
88         else:
89             self._replace_value(field_name, value)
90 
91     def _add_value(self, field_name, value):
92         value = arg_to_iter(value)
93         processed_value = self._process_input_value(field_name, value)
94         if processed_value:
95             self._values[field_name] += arg_to_iter(processed_value)
96 
97     def _replace_value(self, field_name, value):
98         self._values.pop(field_name, None)
99         self._add_value(field_name, value)
100 
101     def get_value(self, value, *processors, **kw):
102         regex = kw.get('re', None)
103         if regex:
104             value = arg_to_iter(value)
105             value = flatten(extract_regex(regex, x) for x in value)
106 
107         for proc in processors:
108             if value is None:
109                 break
110             _proc = proc
111             proc = wrap_loader_context(proc, self.context)
112             try:
113                 value = proc(value)
114             except Exception as e:
115                 raise ValueError("Error with processor %s value=%r error='%s: %s'" %
116                                  (_proc.__class__.__name__, value,
117                                   type(e).__name__, str(e)))
118         return value
119 
120     def load_item(self):
121         item = self.item
122         for field_name in tuple(self._values):
123             value = self.get_output_value(field_name)
124             if value is not None:
125                 item[field_name] = value
126 
127         return item
128 
129     def get_output_value(self, field_name):
130         proc = self.get_output_processor(field_name)
131         proc = wrap_loader_context(proc, self.context)
132         try:
133             return proc(self._values[field_name])
134         except Exception as e:
135             raise ValueError("Error with output processor: field=%r value=%r error='%s: %s'" % \
136                 (field_name, self._values[field_name], type(e).__name__, str(e)))
137 
138     def get_collected_values(self, field_name):
139         return self._values[field_name]
140 
141     def get_input_processor(self, field_name):
142         proc = getattr(self, '%s_in' % field_name, None)
143         if not proc:
144             proc = self._get_item_field_attr(field_name, 'input_processor', \
145                 self.default_input_processor)
146         return proc
147 
148     def get_output_processor(self, field_name):
149         proc = getattr(self, '%s_out' % field_name, None)
150         if not proc:
151             proc = self._get_item_field_attr(field_name, 'output_processor', \
152                 self.default_output_processor)
153         return proc
154 
155     def _process_input_value(self, field_name, value):
156         proc = self.get_input_processor(field_name)
157         _proc = proc
158         proc = wrap_loader_context(proc, self.context)
159         try:
160             return proc(value)
161         except Exception as e:
162             raise ValueError(
163                 "Error with input processor %s: field=%r value=%r "
164                 "error='%s: %s'" % (_proc.__class__.__name__, field_name,
165                                     value, type(e).__name__, str(e)))
166 
167     def _get_item_field_attr(self, field_name, key, default=None):
168         if isinstance(self.item, Item):
169             value = self.item.fields[field_name].get(key, default)
170         else:
171             value = default
172         return value
173 
174     def _check_selector_method(self):
175         if self.selector is None:
176             raise RuntimeError("To use XPath or CSS selectors, "
177                 "%s must be instantiated with a selector "
178                 "or a response" % self.__class__.__name__)
179 
180     def add_xpath(self, field_name, xpath, *processors, **kw):
181         values = self._get_xpathvalues(xpath, **kw)
182         self.add_value(field_name, values, *processors, **kw)
183 
184     def replace_xpath(self, field_name, xpath, *processors, **kw):
185         values = self._get_xpathvalues(xpath, **kw)
186         self.replace_value(field_name, values, *processors, **kw)
187 
188     def get_xpath(self, xpath, *processors, **kw):
189         values = self._get_xpathvalues(xpath, **kw)
190         return self.get_value(values, *processors, **kw)
191 
192     def _get_xpathvalues(self, xpaths, **kw):
193         self._check_selector_method()
194         xpaths = arg_to_iter(xpaths)
195         return flatten(self.selector.xpath(xpath).getall() for xpath in xpaths)
196 
197     def add_css(self, field_name, css, *processors, **kw):
198         values = self._get_cssvalues(css, **kw)
199         self.add_value(field_name, values, *processors, **kw)
200 
201     def replace_css(self, field_name, css, *processors, **kw):
202         values = self._get_cssvalues(css, **kw)
203         self.replace_value(field_name, values, *processors, **kw)
204 
205     def get_css(self, css, *processors, **kw):
206         values = self._get_cssvalues(css, **kw)
207         return self.get_value(values, *processors, **kw)
208 
209     def _get_cssvalues(self, csss, **kw):
210         self._check_selector_method()
211         csss = arg_to_iter(csss)
212         return flatten(self.selector.css(css).getall() for css in csss)
213
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/scrapy/loader/__init__.py b/scrapy/loader/__init__.py
--- a/scrapy/loader/__init__.py
+++ b/scrapy/loader/__init__.py
@@ -1,19 +1,19 @@
-"""Item Loader
+"""
+Item Loader
 
 See documentation in docs/topics/loaders.rst
-
 """
 from collections import defaultdict
+
 import six
 
 from scrapy.item import Item
+from scrapy.loader.common import wrap_loader_context
+from scrapy.loader.processors import Identity
 from scrapy.selector import Selector
 from scrapy.utils.misc import arg_to_iter, extract_regex
 from scrapy.utils.python import flatten
 
-from .common import wrap_loader_context
-from .processors import Identity
-
 
 class ItemLoader(object):
 
@@ -33,10 +33,9 @@
         self.parent = parent
         self._local_item = context['item'] = item
         self._local_values = defaultdict(list)
-        # Preprocess values if item built from dict
-        # Values need to be added to item._values if added them from dict (not with add_values)
+        # values from initial item
         for field_name, value in item.items():
-            self._values[field_name] = self._process_input_value(field_name, value)
+            self._values[field_name] += arg_to_iter(value)
 
     @property
     def _values(self):
@@ -132,8 +131,8 @@
         try:
             return proc(self._values[field_name])
         except Exception as e:
-            raise ValueError("Error with output processor: field=%r value=%r error='%s: %s'" % \
-                (field_name, self._values[field_name], type(e).__name__, str(e)))
+            raise ValueError("Error with output processor: field=%r value=%r error='%s: %s'" %
+                             (field_name, self._values[field_name], type(e).__name__, str(e)))
 
     def get_collected_values(self, field_name):
         return self._values[field_name]
@@ -141,15 +140,15 @@
     def get_input_processor(self, field_name):
         proc = getattr(self, '%s_in' % field_name, None)
         if not proc:
-            proc = self._get_item_field_attr(field_name, 'input_processor', \
-                self.default_input_processor)
+            proc = self._get_item_field_attr(field_name, 'input_processor',
+                                             self.default_input_processor)
         return proc
 
     def get_output_processor(self, field_name):
         proc = getattr(self, '%s_out' % field_name, None)
         if not proc:
-            proc = self._get_item_field_attr(field_name, 'output_processor', \
-                self.default_output_processor)
+            proc = self._get_item_field_attr(field_name, 'output_processor',
+                                             self.default_output_processor)
         return proc
 
     def _process_input_value(self, field_name, value):
@@ -174,8 +173,8 @@
     def _check_selector_method(self):
         if self.selector is None:
             raise RuntimeError("To use XPath or CSS selectors, "
-                "%s must be instantiated with a selector "
-                "or a response" % self.__class__.__name__)
+                               "%s must be instantiated with a selector "
+                               "or a response" % self.__class__.__name__)
 
     def add_xpath(self, field_name, xpath, *processors, **kw):
         values = self._get_xpathvalues(xpath, **kw)

{"golden_diff": "diff --git a/scrapy/loader/__init__.py b/scrapy/loader/__init__.py\n--- a/scrapy/loader/__init__.py\n+++ b/scrapy/loader/__init__.py\n@@ -1,19 +1,19 @@\n-\"\"\"Item Loader\n+\"\"\"\n+Item Loader\n \n See documentation in docs/topics/loaders.rst\n-\n \"\"\"\n from collections import defaultdict\n+\n import six\n \n from scrapy.item import Item\n+from scrapy.loader.common import wrap_loader_context\n+from scrapy.loader.processors import Identity\n from scrapy.selector import Selector\n from scrapy.utils.misc import arg_to_iter, extract_regex\n from scrapy.utils.python import flatten\n \n-from .common import wrap_loader_context\n-from .processors import Identity\n-\n \n class ItemLoader(object):\n \n@@ -33,10 +33,9 @@\n         self.parent = parent\n         self._local_item = context['item'] = item\n         self._local_values = defaultdict(list)\n-        # Preprocess values if item built from dict\n-        # Values need to be added to item._values if added them from dict (not with add_values)\n+        # values from initial item\n         for field_name, value in item.items():\n-            self._values[field_name] = self._process_input_value(field_name, value)\n+            self._values[field_name] += arg_to_iter(value)\n \n     @property\n     def _values(self):\n@@ -132,8 +131,8 @@\n         try:\n             return proc(self._values[field_name])\n         except Exception as e:\n-            raise ValueError(\"Error with output processor: field=%r value=%r error='%s: %s'\" % \\\n-                (field_name, self._values[field_name], type(e).__name__, str(e)))\n+            raise ValueError(\"Error with output processor: field=%r value=%r error='%s: %s'\" %\n+                             (field_name, self._values[field_name], type(e).__name__, str(e)))\n \n     def get_collected_values(self, field_name):\n         return self._values[field_name]\n@@ -141,15 +140,15 @@\n     def get_input_processor(self, field_name):\n         proc = getattr(self, '%s_in' % field_name, None)\n         if not proc:\n-            proc = self._get_item_field_attr(field_name, 'input_processor', \\\n-                self.default_input_processor)\n+            proc = self._get_item_field_attr(field_name, 'input_processor',\n+                                             self.default_input_processor)\n         return proc\n \n     def get_output_processor(self, field_name):\n         proc = getattr(self, '%s_out' % field_name, None)\n         if not proc:\n-            proc = self._get_item_field_attr(field_name, 'output_processor', \\\n-                self.default_output_processor)\n+            proc = self._get_item_field_attr(field_name, 'output_processor',\n+                                             self.default_output_processor)\n         return proc\n \n     def _process_input_value(self, field_name, value):\n@@ -174,8 +173,8 @@\n     def _check_selector_method(self):\n         if self.selector is None:\n             raise RuntimeError(\"To use XPath or CSS selectors, \"\n-                \"%s must be instantiated with a selector \"\n-                \"or a response\" % self.__class__.__name__)\n+                               \"%s must be instantiated with a selector \"\n+                               \"or a response\" % self.__class__.__name__)\n \n     def add_xpath(self, field_name, xpath, *processors, **kw):\n         values = self._get_xpathvalues(xpath, **kw)\n", "issue": "ItemLoaders can break if instantiated with pre-populated items\nBefore I start, I know item loaders have been a big source of discussion for a long time; I'm only opening this issue because the latest release breaks some of our spiders.\r\n\r\nIn one of our projects, our [Autounit](https://github.com/scrapinghub/scrapy-autounit) tests fail under 1.7.1 due to some item loaders which are created from partially populated items. I suspect the relevant change is #3819 (which BTW I think inadvertently closes #3046).\r\nPersonally I think a better approach here would be something _closer_ to the solution proposed in #3149, although not exactly the same.\r\n\r\nConsider the following:\r\n\r\n```python\r\nIn [1]: import scrapy\r\n\r\nIn [2]: scrapy.__version__\r\nOut[2]: '1.6.0'\r\n\r\nIn [3]: from scrapy.loader import ItemLoader\r\n   ...: lo = ItemLoader(item={'key': 'value'})\r\n   ...: lo.add_value('key', 'other value')\r\n   ...: print(lo.load_item())\r\n{'key': ['other value']}\r\n```\r\n\r\n```python\r\nIn [1]: import scrapy\r\n\r\nIn [2]: scrapy.__version__\r\nOut[2]: '1.7.1'\r\n\r\nIn [3]: from scrapy.loader import ItemLoader\r\n   ...: lo = ItemLoader(item={'key': 'value'})\r\n   ...: lo.add_value('key', 'other value')\r\n   ...: print(lo.load_item())\r\n---------------------------------------------------------------------------\r\nTypeError                                 Traceback (most recent call last)\r\n<ipython-input-3-6aa64a41edb1> in <module>\r\n      1 from scrapy.loader import ItemLoader\r\n      2 lo = ItemLoader(item={'key': 'value'})\r\n----> 3 lo.add_value('key', 'other value')\r\n      4 print(lo.load_item())\r\n\r\n~/venv-temporal/lib/python3.6/site-packages/scrapy/loader/__init__.py in add_value(self, field_name, value, *processors, **kw)\r\n     77                 self._add_value(k, v)\r\n     78         else:\r\n---> 79             self._add_value(field_name, value)\r\n     80\r\n     81     def replace_value(self, field_name, value, *processors, **kw):\r\n\r\n~/venv-temporal/lib/python3.6/site-packages/scrapy/loader/__init__.py in _add_value(self, field_name, value)\r\n     93         processed_value = self._process_input_value(field_name, value)\r\n     94         if processed_value:\r\n---> 95             self._values[field_name] += arg_to_iter(processed_value)\r\n     96\r\n     97     def _replace_value(self, field_name, value):\r\n\r\nTypeError: must be str, not list\r\n```\r\nI'm not directly opening a PR because I think this needs discussion. What if we changed\r\n```python\r\nfor field_name, value in item.items():\r\n    self._values[field_name] = self._process_input_value(field_name, value)\r\n```\r\n\r\nto\r\n\r\n```python\r\nfor field_name, value in item.items():\r\n    self._add_value(field_name, value)\r\n```\r\n\r\nwhich calls `arg_to_iter` internally?\r\n\r\nWith that change, the following happens which is more reasonable IMHO:\r\n```python\r\nIn [3]: from scrapy.loader import ItemLoader \r\n   ...: lo = ItemLoader(item={'key': 'value'}) \r\n   ...: lo.add_value('key', 'other value') \r\n   ...: print(lo.load_item())                                                                                                                                                                                                                 \r\n{'key': ['value', 'other value']}\r\n```\r\n\r\nLooking forward to reading your thoughts on the matter\r\n\r\n/cc @Gallaecio @kmike @andrewbaxter @fcanobrash @sortafreel\n`ItemLoader` fields initialized from `item` are reprocessed\n### Description\r\n\r\n#3804 introduced a bug where `ItemLoader` fields are reprocessed.\r\nRelated #3897.\r\n\r\n### Steps to Reproduce\r\n\r\n```python\r\nfrom pprint import pprint\r\n\r\nfrom scrapy import Field, Item\r\nfrom scrapy.loader import ItemLoader\r\nfrom scrapy.loader.processors import TakeFirst\r\n\r\n\r\nclass X(Item):\r\n    x = Field(output_processor=TakeFirst())\r\n\r\n\r\nloader = ItemLoader(X())\r\nloader.add_value(\"x\", [\"value1\", \"value2\"])\r\nx = loader.load_item()\r\npprint(x)\r\n# {'x': 'value1'}\r\n\r\npprint(ItemLoader(x).load_item())\r\n# {'x': 'v'}\r\n```\r\n\r\n**Expected behavior:** `ItemLoader` initialized from the `x` item does not reprocess its fields and  loads `{'x': 'value1'}`.\r\n\r\n**Actual behavior:** `ItemLoader` initialized from the `x` item reprocesses its fields and  loads `{'x': 'v'}`.\r\n\r\n### Versions\r\n\r\n```\r\nScrapy       : 1.7.3\r\nlxml         : 4.4.1.0\r\nlibxml2      : 2.9.9\r\ncssselect    : 1.1.0\r\nparsel       : 1.5.2\r\nw3lib        : 1.21.0\r\nTwisted      : 19.7.0\r\nPython       : 3.6.5 (default, May  3 2018, 10:08:28) - [GCC 5.4.0 20160609]\r\npyOpenSSL    : 19.0.0 (OpenSSL 1.1.1c  28 May 2019)\r\ncryptography : 2.7\r\nPlatform     : Linux-4.4.0-127-generic-x86_64-with-LinuxMint-18.1-serena\r\n```\r\n\r\n### Additional context\r\n\r\nHere's the behavior of the previous version:\r\n\r\n```\r\nScrapy       : 1.6.0\r\nlxml         : 4.4.0.0\r\nlibxml2      : 2.9.9\r\ncssselect    : 1.0.3\r\nparsel       : 1.5.1\r\nw3lib        : 1.20.0\r\nTwisted      : 19.7.0\r\nPython       : 3.6.5 (default, May  3 2018, 10:08:28) - [GCC 5.4.0 20160609]\r\npyOpenSSL    : 19.0.0 (OpenSSL 1.1.1c  28 May 2019)\r\ncryptography : 2.7\r\nPlatform     : Linux-4.4.0-127-generic-x86_64-with-LinuxMint-18.1-serena\r\n```\r\n\r\n```python\r\n# {'x': 'value1'}\r\n# {'x': 'value1'}\r\n```\n", "before_files": [{"content": "\"\"\"Item Loader\n\nSee documentation in docs/topics/loaders.rst\n\n\"\"\"\nfrom collections import defaultdict\nimport six\n\nfrom scrapy.item import Item\nfrom scrapy.selector import Selector\nfrom scrapy.utils.misc import arg_to_iter, extract_regex\nfrom scrapy.utils.python import flatten\n\nfrom .common import wrap_loader_context\nfrom .processors import Identity\n\n\nclass ItemLoader(object):\n\n    default_item_class = Item\n    default_input_processor = Identity()\n    default_output_processor = Identity()\n    default_selector_class = Selector\n\n    def __init__(self, item=None, selector=None, response=None, parent=None, **context):\n        if selector is None and response is not None:\n            selector = self.default_selector_class(response)\n        self.selector = selector\n        context.update(selector=selector, response=response)\n        if item is None:\n            item = self.default_item_class()\n        self.context = context\n        self.parent = parent\n        self._local_item = context['item'] = item\n        self._local_values = defaultdict(list)\n        # Preprocess values if item built from dict\n        # Values need to be added to item._values if added them from dict (not with add_values)\n        for field_name, value in item.items():\n            self._values[field_name] = self._process_input_value(field_name, value)\n\n    @property\n    def _values(self):\n        if self.parent is not None:\n            return self.parent._values\n        else:\n            return self._local_values\n\n    @property\n    def item(self):\n        if self.parent is not None:\n            return self.parent.item\n        else:\n            return self._local_item\n\n    def nested_xpath(self, xpath, **context):\n        selector = self.selector.xpath(xpath)\n        context.update(selector=selector)\n        subloader = self.__class__(\n            item=self.item, parent=self, **context\n        )\n        return subloader\n\n    def nested_css(self, css, **context):\n        selector = self.selector.css(css)\n        context.update(selector=selector)\n        subloader = self.__class__(\n            item=self.item, parent=self, **context\n        )\n        return subloader\n\n    def add_value(self, field_name, value, *processors, **kw):\n        value = self.get_value(value, *processors, **kw)\n        if value is None:\n            return\n        if not field_name:\n            for k, v in six.iteritems(value):\n                self._add_value(k, v)\n        else:\n            self._add_value(field_name, value)\n\n    def replace_value(self, field_name, value, *processors, **kw):\n        value = self.get_value(value, *processors, **kw)\n        if value is None:\n            return\n        if not field_name:\n            for k, v in six.iteritems(value):\n                self._replace_value(k, v)\n        else:\n            self._replace_value(field_name, value)\n\n    def _add_value(self, field_name, value):\n        value = arg_to_iter(value)\n        processed_value = self._process_input_value(field_name, value)\n        if processed_value:\n            self._values[field_name] += arg_to_iter(processed_value)\n\n    def _replace_value(self, field_name, value):\n        self._values.pop(field_name, None)\n        self._add_value(field_name, value)\n\n    def get_value(self, value, *processors, **kw):\n        regex = kw.get('re', None)\n        if regex:\n            value = arg_to_iter(value)\n            value = flatten(extract_regex(regex, x) for x in value)\n\n        for proc in processors:\n            if value is None:\n                break\n            _proc = proc\n            proc = wrap_loader_context(proc, self.context)\n            try:\n                value = proc(value)\n            except Exception as e:\n                raise ValueError(\"Error with processor %s value=%r error='%s: %s'\" %\n                                 (_proc.__class__.__name__, value,\n                                  type(e).__name__, str(e)))\n        return value\n\n    def load_item(self):\n        item = self.item\n        for field_name in tuple(self._values):\n            value = self.get_output_value(field_name)\n            if value is not None:\n                item[field_name] = value\n\n        return item\n\n    def get_output_value(self, field_name):\n        proc = self.get_output_processor(field_name)\n        proc = wrap_loader_context(proc, self.context)\n        try:\n            return proc(self._values[field_name])\n        except Exception as e:\n            raise ValueError(\"Error with output processor: field=%r value=%r error='%s: %s'\" % \\\n                (field_name, self._values[field_name], type(e).__name__, str(e)))\n\n    def get_collected_values(self, field_name):\n        return self._values[field_name]\n\n    def get_input_processor(self, field_name):\n        proc = getattr(self, '%s_in' % field_name, None)\n        if not proc:\n            proc = self._get_item_field_attr(field_name, 'input_processor', \\\n                self.default_input_processor)\n        return proc\n\n    def get_output_processor(self, field_name):\n        proc = getattr(self, '%s_out' % field_name, None)\n        if not proc:\n            proc = self._get_item_field_attr(field_name, 'output_processor', \\\n                self.default_output_processor)\n        return proc\n\n    def _process_input_value(self, field_name, value):\n        proc = self.get_input_processor(field_name)\n        _proc = proc\n        proc = wrap_loader_context(proc, self.context)\n        try:\n            return proc(value)\n        except Exception as e:\n            raise ValueError(\n                \"Error with input processor %s: field=%r value=%r \"\n                \"error='%s: %s'\" % (_proc.__class__.__name__, field_name,\n                                    value, type(e).__name__, str(e)))\n\n    def _get_item_field_attr(self, field_name, key, default=None):\n        if isinstance(self.item, Item):\n            value = self.item.fields[field_name].get(key, default)\n        else:\n            value = default\n        return value\n\n    def _check_selector_method(self):\n        if self.selector is None:\n            raise RuntimeError(\"To use XPath or CSS selectors, \"\n                \"%s must be instantiated with a selector \"\n                \"or a response\" % self.__class__.__name__)\n\n    def add_xpath(self, field_name, xpath, *processors, **kw):\n        values = self._get_xpathvalues(xpath, **kw)\n        self.add_value(field_name, values, *processors, **kw)\n\n    def replace_xpath(self, field_name, xpath, *processors, **kw):\n        values = self._get_xpathvalues(xpath, **kw)\n        self.replace_value(field_name, values, *processors, **kw)\n\n    def get_xpath(self, xpath, *processors, **kw):\n        values = self._get_xpathvalues(xpath, **kw)\n        return self.get_value(values, *processors, **kw)\n\n    def _get_xpathvalues(self, xpaths, **kw):\n        self._check_selector_method()\n        xpaths = arg_to_iter(xpaths)\n        return flatten(self.selector.xpath(xpath).getall() for xpath in xpaths)\n\n    def add_css(self, field_name, css, *processors, **kw):\n        values = self._get_cssvalues(css, **kw)\n        self.add_value(field_name, values, *processors, **kw)\n\n    def replace_css(self, field_name, css, *processors, **kw):\n        values = self._get_cssvalues(css, **kw)\n        self.replace_value(field_name, values, *processors, **kw)\n\n    def get_css(self, css, *processors, **kw):\n        values = self._get_cssvalues(css, **kw)\n        return self.get_value(values, *processors, **kw)\n\n    def _get_cssvalues(self, csss, **kw):\n        self._check_selector_method()\n        csss = arg_to_iter(csss)\n        return flatten(self.selector.css(css).getall() for css in csss)\n", "path": "scrapy/loader/__init__.py"}], "after_files": [{"content": "\"\"\"\nItem Loader\n\nSee documentation in docs/topics/loaders.rst\n\"\"\"\nfrom collections import defaultdict\n\nimport six\n\nfrom scrapy.item import Item\nfrom scrapy.loader.common import wrap_loader_context\nfrom scrapy.loader.processors import Identity\nfrom scrapy.selector import Selector\nfrom scrapy.utils.misc import arg_to_iter, extract_regex\nfrom scrapy.utils.python import flatten\n\n\nclass ItemLoader(object):\n\n    default_item_class = Item\n    default_input_processor = Identity()\n    default_output_processor = Identity()\n    default_selector_class = Selector\n\n    def __init__(self, item=None, selector=None, response=None, parent=None, **context):\n        if selector is None and response is not None:\n            selector = self.default_selector_class(response)\n        self.selector = selector\n        context.update(selector=selector, response=response)\n        if item is None:\n            item = self.default_item_class()\n        self.context = context\n        self.parent = parent\n        self._local_item = context['item'] = item\n        self._local_values = defaultdict(list)\n        # values from initial item\n        for field_name, value in item.items():\n            self._values[field_name] += arg_to_iter(value)\n\n    @property\n    def _values(self):\n        if self.parent is not None:\n            return self.parent._values\n        else:\n            return self._local_values\n\n    @property\n    def item(self):\n        if self.parent is not None:\n            return self.parent.item\n        else:\n            return self._local_item\n\n    def nested_xpath(self, xpath, **context):\n        selector = self.selector.xpath(xpath)\n        context.update(selector=selector)\n        subloader = self.__class__(\n            item=self.item, parent=self, **context\n        )\n        return subloader\n\n    def nested_css(self, css, **context):\n        selector = self.selector.css(css)\n        context.update(selector=selector)\n        subloader = self.__class__(\n            item=self.item, parent=self, **context\n        )\n        return subloader\n\n    def add_value(self, field_name, value, *processors, **kw):\n        value = self.get_value(value, *processors, **kw)\n        if value is None:\n            return\n        if not field_name:\n            for k, v in six.iteritems(value):\n                self._add_value(k, v)\n        else:\n            self._add_value(field_name, value)\n\n    def replace_value(self, field_name, value, *processors, **kw):\n        value = self.get_value(value, *processors, **kw)\n        if value is None:\n            return\n        if not field_name:\n            for k, v in six.iteritems(value):\n                self._replace_value(k, v)\n        else:\n            self._replace_value(field_name, value)\n\n    def _add_value(self, field_name, value):\n        value = arg_to_iter(value)\n        processed_value = self._process_input_value(field_name, value)\n        if processed_value:\n            self._values[field_name] += arg_to_iter(processed_value)\n\n    def _replace_value(self, field_name, value):\n        self._values.pop(field_name, None)\n        self._add_value(field_name, value)\n\n    def get_value(self, value, *processors, **kw):\n        regex = kw.get('re', None)\n        if regex:\n            value = arg_to_iter(value)\n            value = flatten(extract_regex(regex, x) for x in value)\n\n        for proc in processors:\n            if value is None:\n                break\n            _proc = proc\n            proc = wrap_loader_context(proc, self.context)\n            try:\n                value = proc(value)\n            except Exception as e:\n                raise ValueError(\"Error with processor %s value=%r error='%s: %s'\" %\n                                 (_proc.__class__.__name__, value,\n                                  type(e).__name__, str(e)))\n        return value\n\n    def load_item(self):\n        item = self.item\n        for field_name in tuple(self._values):\n            value = self.get_output_value(field_name)\n            if value is not None:\n                item[field_name] = value\n\n        return item\n\n    def get_output_value(self, field_name):\n        proc = self.get_output_processor(field_name)\n        proc = wrap_loader_context(proc, self.context)\n        try:\n            return proc(self._values[field_name])\n        except Exception as e:\n            raise ValueError(\"Error with output processor: field=%r value=%r error='%s: %s'\" %\n                             (field_name, self._values[field_name], type(e).__name__, str(e)))\n\n    def get_collected_values(self, field_name):\n        return self._values[field_name]\n\n    def get_input_processor(self, field_name):\n        proc = getattr(self, '%s_in' % field_name, None)\n        if not proc:\n            proc = self._get_item_field_attr(field_name, 'input_processor',\n                                             self.default_input_processor)\n        return proc\n\n    def get_output_processor(self, field_name):\n        proc = getattr(self, '%s_out' % field_name, None)\n        if not proc:\n            proc = self._get_item_field_attr(field_name, 'output_processor',\n                                             self.default_output_processor)\n        return proc\n\n    def _process_input_value(self, field_name, value):\n        proc = self.get_input_processor(field_name)\n        _proc = proc\n        proc = wrap_loader_context(proc, self.context)\n        try:\n            return proc(value)\n        except Exception as e:\n            raise ValueError(\n                \"Error with input processor %s: field=%r value=%r \"\n                \"error='%s: %s'\" % (_proc.__class__.__name__, field_name,\n                                    value, type(e).__name__, str(e)))\n\n    def _get_item_field_attr(self, field_name, key, default=None):\n        if isinstance(self.item, Item):\n            value = self.item.fields[field_name].get(key, default)\n        else:\n            value = default\n        return value\n\n    def _check_selector_method(self):\n        if self.selector is None:\n            raise RuntimeError(\"To use XPath or CSS selectors, \"\n                               \"%s must be instantiated with a selector \"\n                               \"or a response\" % self.__class__.__name__)\n\n    def add_xpath(self, field_name, xpath, *processors, **kw):\n        values = self._get_xpathvalues(xpath, **kw)\n        self.add_value(field_name, values, *processors, **kw)\n\n    def replace_xpath(self, field_name, xpath, *processors, **kw):\n        values = self._get_xpathvalues(xpath, **kw)\n        self.replace_value(field_name, values, *processors, **kw)\n\n    def get_xpath(self, xpath, *processors, **kw):\n        values = self._get_xpathvalues(xpath, **kw)\n        return self.get_value(values, *processors, **kw)\n\n    def _get_xpathvalues(self, xpaths, **kw):\n        self._check_selector_method()\n        xpaths = arg_to_iter(xpaths)\n        return flatten(self.selector.xpath(xpath).getall() for xpath in xpaths)\n\n    def add_css(self, field_name, css, *processors, **kw):\n        values = self._get_cssvalues(css, **kw)\n        self.add_value(field_name, values, *processors, **kw)\n\n    def replace_css(self, field_name, css, *processors, **kw):\n        values = self._get_cssvalues(css, **kw)\n        self.replace_value(field_name, values, *processors, **kw)\n\n    def get_css(self, css, *processors, **kw):\n        values = self._get_cssvalues(css, **kw)\n        return self.get_value(values, *processors, **kw)\n\n    def _get_cssvalues(self, csss, **kw):\n        self._check_selector_method()\n        csss = arg_to_iter(csss)\n        return flatten(self.selector.css(css).getall() for css in csss)\n", "path": "scrapy/loader/__init__.py"}]}