We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
werkzeug.formparser is really slow with large binary uploads
When I perform a `multipart/form-data` upload of any large binary file in Flask, those uploads are very easily CPU bound (with Python consuming 100% CPU) instead of I/O bound on any reasonably fast network connection.

A little bit of CPU profiling reveals that almost all CPU time during these uploads is spent in `werkzeug.formparser.MultiPartParser.parse_parts()`. The reason this that the method `parse_lines()` yields _a lot_ of very small chunks, sometimes [even just single bytes](https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/formparser.py#L439):

``` python
# we have something in the buffer from the last iteration.
# this is usually a newline delimiter.
if buf:
    yield _cont, buf
    buf = b''
```

So `parse_parts()` goes through a lot of small iterations (more than 2 million for a 100 MB file) processing single "lines", always writing just very short chunks or even single bytes into the output stream. This adds a lot of overhead slowing down those whole process and making it CPU bound very quickly.

A quick test shows that a speed-up is very easily possible by first collecting the data in a `bytearray` in `parse_lines()` and only yielding that data back into `parse_parts()` when `self.buffer_size` is exceeded. Something like this:

``` python
buf = b''
collect = bytearray()
for line in iterator:
    if not line:
        self.fail('unexpected end of stream')

    if line[:2] == b'--':
        terminator = line.rstrip()
        if terminator in (next_part, last_part):
            # yield remaining collected data
            if collect:
                yield _cont, collect
            break

    if transfer_encoding is not None:
        if transfer_encoding == 'base64':
            transfer_encoding = 'base64_codec'
        try:
            line = codecs.decode(line, transfer_encoding)
        except Exception:
            self.fail('could not decode transfer encoded chunk')

    # we have something in the buffer from the last iteration.
    # this is usually a newline delimiter.
    if buf:
        collect += buf
        buf = b''

    # If the line ends with windows CRLF we write everything except
    # the last two bytes.  In all other cases however we write
    # everything except the last byte.  If it was a newline, that's
    # fine, otherwise it does not matter because we will write it
    # the next iteration.  this ensures we do not write the
    # final newline into the stream.  That way we do not have to
    # truncate the stream.  However we do have to make sure that
    # if something else than a newline is in there we write it
    # out.
    if line[-2:] == b'\r\n':
        buf = b'\r\n'
        cutoff = -2
    else:
        buf = line[-1:]
        cutoff = -1

    collect += line[:cutoff]

    if len(collect) >= self.buffer_size:
        yield _cont, collect
        collect.clear()
```

This change alone reduces the upload time for my 34 MB test file from 4200 ms to around 1100 ms over localhost on my machine, that's almost a 4X increase in performance. All tests are done on Windows (64-bit Python 3.4), I'm not sure if it's as much of a problem on Linux.

It's still mostly CPU bound, so I'm sure there is even more potential for optimization. I think I'll look into it when I find a bit more time.


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `src/werkzeug/sansio/multipart.py`
Content:
```
1 import re
2 from dataclasses import dataclass
3 from enum import auto
4 from enum import Enum
5 from typing import cast
6 from typing import List
7 from typing import Optional
8 from typing import Tuple
9 
10 from .._internal import _to_bytes
11 from .._internal import _to_str
12 from ..datastructures import Headers
13 from ..exceptions import RequestEntityTooLarge
14 from ..http import parse_options_header
15 
16 
17 class Event:
18     pass
19 
20 
21 @dataclass(frozen=True)
22 class Preamble(Event):
23     data: bytes
24 
25 
26 @dataclass(frozen=True)
27 class Field(Event):
28     name: str
29     headers: Headers
30 
31 
32 @dataclass(frozen=True)
33 class File(Event):
34     name: str
35     filename: str
36     headers: Headers
37 
38 
39 @dataclass(frozen=True)
40 class Data(Event):
41     data: bytes
42     more_data: bool
43 
44 
45 @dataclass(frozen=True)
46 class Epilogue(Event):
47     data: bytes
48 
49 
50 class NeedData(Event):
51     pass
52 
53 
54 NEED_DATA = NeedData()
55 
56 
57 class State(Enum):
58     PREAMBLE = auto()
59     PART = auto()
60     DATA = auto()
61     EPILOGUE = auto()
62     COMPLETE = auto()
63 
64 
65 # Multipart line breaks MUST be CRLF (\r\n) by RFC-7578, except that
66 # many implementations break this and either use CR or LF alone.
67 LINE_BREAK = b"(?:\r\n|\n|\r)"
68 BLANK_LINE_RE = re.compile(b"(?:\r\n\r\n|\r\r|\n\n)", re.MULTILINE)
69 LINE_BREAK_RE = re.compile(LINE_BREAK, re.MULTILINE)
70 # Header values can be continued via a space or tab after the linebreak, as
71 # per RFC2231
72 HEADER_CONTINUATION_RE = re.compile(b"%s[ \t]" % LINE_BREAK, re.MULTILINE)
73 
74 
75 class MultipartDecoder:
76     """Decodes a multipart message as bytes into Python events.
77 
78     The part data is returned as available to allow the caller to save
79     the data from memory to disk, if desired.
80     """
81 
82     def __init__(
83         self,
84         boundary: bytes,
85         max_form_memory_size: Optional[int] = None,
86     ) -> None:
87         self.buffer = bytearray()
88         self.complete = False
89         self.max_form_memory_size = max_form_memory_size
90         self.state = State.PREAMBLE
91 
92         # Note in the below \h i.e. horizontal whitespace is used
93         # as [^\S\n\r] as \h isn't supported in python.
94 
95         # The preamble must end with a boundary where the boundary is
96         # prefixed by a line break, RFC2046. Except that many
97         # implementations including Werkzeug's tests omit the line
98         # break prefix. In addition the first boundary could be the
99         # epilogue boundary (for empty form-data) hence the matching
100         # group to understand if it is an epilogue boundary.
101         self.preamble_re = re.compile(
102             br"%s?--%s(--[^\S\n\r]*%s?|[^\S\n\r]*%s)"
103             % (LINE_BREAK, boundary, LINE_BREAK, LINE_BREAK),
104             re.MULTILINE,
105         )
106         # A boundary must include a line break prefix and suffix, and
107         # may include trailing whitespace. In addition the boundary
108         # could be the epilogue boundary hence the matching group to
109         # understand if it is an epilogue boundary.
110         self.boundary_re = re.compile(
111             br"%s--%s(--[^\S\n\r]*%s?|[^\S\n\r]*%s)"
112             % (LINE_BREAK, boundary, LINE_BREAK, LINE_BREAK),
113             re.MULTILINE,
114         )
115 
116     def receive_data(self, data: Optional[bytes]) -> None:
117         if data is None:
118             self.complete = True
119         elif (
120             self.max_form_memory_size is not None
121             and len(self.buffer) + len(data) > self.max_form_memory_size
122         ):
123             raise RequestEntityTooLarge()
124         else:
125             self.buffer.extend(data)
126 
127     def next_event(self) -> Event:
128         event: Event = NEED_DATA
129 
130         if self.state == State.PREAMBLE:
131             match = self.preamble_re.search(self.buffer)
132             if match is not None:
133                 if match.group(1).startswith(b"--"):
134                     self.state = State.EPILOGUE
135                 else:
136                     self.state = State.PART
137                 data = bytes(self.buffer[: match.start()])
138                 del self.buffer[: match.end()]
139                 event = Preamble(data=data)
140 
141         elif self.state == State.PART:
142             match = BLANK_LINE_RE.search(self.buffer)
143             if match is not None:
144                 headers = self._parse_headers(self.buffer[: match.start()])
145                 del self.buffer[: match.end()]
146 
147                 if "content-disposition" not in headers:
148                     raise ValueError("Missing Content-Disposition header")
149 
150                 disposition, extra = parse_options_header(
151                     headers["content-disposition"]
152                 )
153                 name = cast(str, extra.get("name"))
154                 filename = extra.get("filename")
155                 if filename is not None:
156                     event = File(
157                         filename=filename,
158                         headers=headers,
159                         name=name,
160                     )
161                 else:
162                     event = Field(
163                         headers=headers,
164                         name=name,
165                     )
166                 self.state = State.DATA
167 
168         elif self.state == State.DATA:
169             # Return up to the last line break as data, anything past
170             # that line break could be a boundary - more data may be
171             # required to know for sure.
172             lines = list(LINE_BREAK_RE.finditer(self.buffer))
173             if len(lines):
174                 data_length = del_index = lines[-1].start()
175                 match = self.boundary_re.search(self.buffer)
176                 if match is not None:
177                     if match.group(1).startswith(b"--"):
178                         self.state = State.EPILOGUE
179                     else:
180                         self.state = State.PART
181                     data_length = match.start()
182                     del_index = match.end()
183 
184                 data = bytes(self.buffer[:data_length])
185                 del self.buffer[:del_index]
186                 more_data = match is None
187                 if data or not more_data:
188                     event = Data(data=data, more_data=more_data)
189 
190         elif self.state == State.EPILOGUE and self.complete:
191             event = Epilogue(data=bytes(self.buffer))
192             del self.buffer[:]
193             self.state = State.COMPLETE
194 
195         if self.complete and isinstance(event, NeedData):
196             raise ValueError(f"Invalid form-data cannot parse beyond {self.state}")
197 
198         return event
199 
200     def _parse_headers(self, data: bytes) -> Headers:
201         headers: List[Tuple[str, str]] = []
202         # Merge the continued headers into one line
203         data = HEADER_CONTINUATION_RE.sub(b" ", data)
204         # Now there is one header per line
205         for line in data.splitlines():
206             if line.strip() != b"":
207                 name, value = _to_str(line).strip().split(":", 1)
208                 headers.append((name.strip(), value.strip()))
209         return Headers(headers)
210 
211 
212 class MultipartEncoder:
213     def __init__(self, boundary: bytes) -> None:
214         self.boundary = boundary
215         self.state = State.PREAMBLE
216 
217     def send_event(self, event: Event) -> bytes:
218         if isinstance(event, Preamble) and self.state == State.PREAMBLE:
219             self.state = State.PART
220             return event.data
221         elif isinstance(event, (Field, File)) and self.state in {
222             State.PREAMBLE,
223             State.PART,
224             State.DATA,
225         }:
226             self.state = State.DATA
227             data = b"\r\n--" + self.boundary + b"\r\n"
228             data += b'Content-Disposition: form-data; name="%s"' % _to_bytes(event.name)
229             if isinstance(event, File):
230                 data += b'; filename="%s"' % _to_bytes(event.filename)
231             data += b"\r\n"
232             for name, value in cast(Field, event).headers:
233                 if name.lower() != "content-disposition":
234                     data += _to_bytes(f"{name}: {value}\r\n")
235             data += b"\r\n"
236             return data
237         elif isinstance(event, Data) and self.state == State.DATA:
238             return event.data
239         elif isinstance(event, Epilogue):
240             self.state = State.COMPLETE
241             return b"\r\n--" + self.boundary + b"--\r\n" + event.data
242         else:
243             raise ValueError(f"Cannot generate {event} in state: {self.state}")
244
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/src/werkzeug/sansio/multipart.py b/src/werkzeug/sansio/multipart.py
--- a/src/werkzeug/sansio/multipart.py
+++ b/src/werkzeug/sansio/multipart.py
@@ -88,6 +88,7 @@
         self.complete = False
         self.max_form_memory_size = max_form_memory_size
         self.state = State.PREAMBLE
+        self.boundary = boundary
 
         # Note in the below \h i.e. horizontal whitespace is used
         # as [^\S\n\r] as \h isn't supported in python.
@@ -113,6 +114,18 @@
             re.MULTILINE,
         )
 
+    def last_newline(self) -> int:
+        try:
+            last_nl = self.buffer.rindex(b"\n")
+        except ValueError:
+            last_nl = len(self.buffer)
+        try:
+            last_cr = self.buffer.rindex(b"\r")
+        except ValueError:
+            last_cr = len(self.buffer)
+
+        return min(last_nl, last_cr)
+
     def receive_data(self, data: Optional[bytes]) -> None:
         if data is None:
             self.complete = True
@@ -166,12 +179,14 @@
                 self.state = State.DATA
 
         elif self.state == State.DATA:
-            # Return up to the last line break as data, anything past
-            # that line break could be a boundary - more data may be
-            # required to know for sure.
-            lines = list(LINE_BREAK_RE.finditer(self.buffer))
-            if len(lines):
-                data_length = del_index = lines[-1].start()
+            if self.buffer.find(b"--" + self.boundary) == -1:
+                # No complete boundary in the buffer, but there may be
+                # a partial boundary at the end. As the boundary
+                # starts with either a nl or cr find the earliest and
+                # return up to that as data.
+                data_length = del_index = self.last_newline()
+                more_data = True
+            else:
                 match = self.boundary_re.search(self.buffer)
                 if match is not None:
                     if match.group(1).startswith(b"--"):
@@ -180,12 +195,14 @@
                         self.state = State.PART
                     data_length = match.start()
                     del_index = match.end()
-
-                data = bytes(self.buffer[:data_length])
-                del self.buffer[:del_index]
+                else:
+                    data_length = del_index = self.last_newline()
                 more_data = match is None
-                if data or not more_data:
-                    event = Data(data=data, more_data=more_data)
+
+            data = bytes(self.buffer[:data_length])
+            del self.buffer[:del_index]
+            if data or not more_data:
+                event = Data(data=data, more_data=more_data)
 
         elif self.state == State.EPILOGUE and self.complete:
             event = Epilogue(data=bytes(self.buffer))

{"golden_diff": "diff --git a/src/werkzeug/sansio/multipart.py b/src/werkzeug/sansio/multipart.py\n--- a/src/werkzeug/sansio/multipart.py\n+++ b/src/werkzeug/sansio/multipart.py\n@@ -88,6 +88,7 @@\n         self.complete = False\n         self.max_form_memory_size = max_form_memory_size\n         self.state = State.PREAMBLE\n+        self.boundary = boundary\n \n         # Note in the below \\h i.e. horizontal whitespace is used\n         # as [^\\S\\n\\r] as \\h isn't supported in python.\n@@ -113,6 +114,18 @@\n             re.MULTILINE,\n         )\n \n+    def last_newline(self) -> int:\n+        try:\n+            last_nl = self.buffer.rindex(b\"\\n\")\n+        except ValueError:\n+            last_nl = len(self.buffer)\n+        try:\n+            last_cr = self.buffer.rindex(b\"\\r\")\n+        except ValueError:\n+            last_cr = len(self.buffer)\n+\n+        return min(last_nl, last_cr)\n+\n     def receive_data(self, data: Optional[bytes]) -> None:\n         if data is None:\n             self.complete = True\n@@ -166,12 +179,14 @@\n                 self.state = State.DATA\n \n         elif self.state == State.DATA:\n-            # Return up to the last line break as data, anything past\n-            # that line break could be a boundary - more data may be\n-            # required to know for sure.\n-            lines = list(LINE_BREAK_RE.finditer(self.buffer))\n-            if len(lines):\n-                data_length = del_index = lines[-1].start()\n+            if self.buffer.find(b\"--\" + self.boundary) == -1:\n+                # No complete boundary in the buffer, but there may be\n+                # a partial boundary at the end. As the boundary\n+                # starts with either a nl or cr find the earliest and\n+                # return up to that as data.\n+                data_length = del_index = self.last_newline()\n+                more_data = True\n+            else:\n                 match = self.boundary_re.search(self.buffer)\n                 if match is not None:\n                     if match.group(1).startswith(b\"--\"):\n@@ -180,12 +195,14 @@\n                         self.state = State.PART\n                     data_length = match.start()\n                     del_index = match.end()\n-\n-                data = bytes(self.buffer[:data_length])\n-                del self.buffer[:del_index]\n+                else:\n+                    data_length = del_index = self.last_newline()\n                 more_data = match is None\n-                if data or not more_data:\n-                    event = Data(data=data, more_data=more_data)\n+\n+            data = bytes(self.buffer[:data_length])\n+            del self.buffer[:del_index]\n+            if data or not more_data:\n+                event = Data(data=data, more_data=more_data)\n \n         elif self.state == State.EPILOGUE and self.complete:\n             event = Epilogue(data=bytes(self.buffer))\n", "issue": "werkzeug.formparser is really slow with large binary uploads\nWhen I perform a `multipart/form-data` upload of any large binary file in Flask, those uploads are very easily CPU bound (with Python consuming 100% CPU) instead of I/O bound on any reasonably fast network connection.\n\nA little bit of CPU profiling reveals that almost all CPU time during these uploads is spent in `werkzeug.formparser.MultiPartParser.parse_parts()`. The reason this that the method `parse_lines()` yields _a lot_ of very small chunks, sometimes [even just single bytes](https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/formparser.py#L439):\n\n``` python\n# we have something in the buffer from the last iteration.\n# this is usually a newline delimiter.\nif buf:\n    yield _cont, buf\n    buf = b''\n```\n\nSo `parse_parts()` goes through a lot of small iterations (more than 2 million for a 100 MB file) processing single \"lines\", always writing just very short chunks or even single bytes into the output stream. This adds a lot of overhead slowing down those whole process and making it CPU bound very quickly.\n\nA quick test shows that a speed-up is very easily possible by first collecting the data in a `bytearray` in `parse_lines()` and only yielding that data back into `parse_parts()` when `self.buffer_size` is exceeded. Something like this:\n\n``` python\nbuf = b''\ncollect = bytearray()\nfor line in iterator:\n    if not line:\n        self.fail('unexpected end of stream')\n\n    if line[:2] == b'--':\n        terminator = line.rstrip()\n        if terminator in (next_part, last_part):\n            # yield remaining collected data\n            if collect:\n                yield _cont, collect\n            break\n\n    if transfer_encoding is not None:\n        if transfer_encoding == 'base64':\n            transfer_encoding = 'base64_codec'\n        try:\n            line = codecs.decode(line, transfer_encoding)\n        except Exception:\n            self.fail('could not decode transfer encoded chunk')\n\n    # we have something in the buffer from the last iteration.\n    # this is usually a newline delimiter.\n    if buf:\n        collect += buf\n        buf = b''\n\n    # If the line ends with windows CRLF we write everything except\n    # the last two bytes.  In all other cases however we write\n    # everything except the last byte.  If it was a newline, that's\n    # fine, otherwise it does not matter because we will write it\n    # the next iteration.  this ensures we do not write the\n    # final newline into the stream.  That way we do not have to\n    # truncate the stream.  However we do have to make sure that\n    # if something else than a newline is in there we write it\n    # out.\n    if line[-2:] == b'\\r\\n':\n        buf = b'\\r\\n'\n        cutoff = -2\n    else:\n        buf = line[-1:]\n        cutoff = -1\n\n    collect += line[:cutoff]\n\n    if len(collect) >= self.buffer_size:\n        yield _cont, collect\n        collect.clear()\n```\n\nThis change alone reduces the upload time for my 34 MB test file from 4200 ms to around 1100 ms over localhost on my machine, that's almost a 4X increase in performance. All tests are done on Windows (64-bit Python 3.4), I'm not sure if it's as much of a problem on Linux.\n\nIt's still mostly CPU bound, so I'm sure there is even more potential for optimization. I think I'll look into it when I find a bit more time.\n\n", "before_files": [{"content": "import re\nfrom dataclasses import dataclass\nfrom enum import auto\nfrom enum import Enum\nfrom typing import cast\nfrom typing import List\nfrom typing import Optional\nfrom typing import Tuple\n\nfrom .._internal import _to_bytes\nfrom .._internal import _to_str\nfrom ..datastructures import Headers\nfrom ..exceptions import RequestEntityTooLarge\nfrom ..http import parse_options_header\n\n\nclass Event:\n    pass\n\n\n@dataclass(frozen=True)\nclass Preamble(Event):\n    data: bytes\n\n\n@dataclass(frozen=True)\nclass Field(Event):\n    name: str\n    headers: Headers\n\n\n@dataclass(frozen=True)\nclass File(Event):\n    name: str\n    filename: str\n    headers: Headers\n\n\n@dataclass(frozen=True)\nclass Data(Event):\n    data: bytes\n    more_data: bool\n\n\n@dataclass(frozen=True)\nclass Epilogue(Event):\n    data: bytes\n\n\nclass NeedData(Event):\n    pass\n\n\nNEED_DATA = NeedData()\n\n\nclass State(Enum):\n    PREAMBLE = auto()\n    PART = auto()\n    DATA = auto()\n    EPILOGUE = auto()\n    COMPLETE = auto()\n\n\n# Multipart line breaks MUST be CRLF (\\r\\n) by RFC-7578, except that\n# many implementations break this and either use CR or LF alone.\nLINE_BREAK = b\"(?:\\r\\n|\\n|\\r)\"\nBLANK_LINE_RE = re.compile(b\"(?:\\r\\n\\r\\n|\\r\\r|\\n\\n)\", re.MULTILINE)\nLINE_BREAK_RE = re.compile(LINE_BREAK, re.MULTILINE)\n# Header values can be continued via a space or tab after the linebreak, as\n# per RFC2231\nHEADER_CONTINUATION_RE = re.compile(b\"%s[ \\t]\" % LINE_BREAK, re.MULTILINE)\n\n\nclass MultipartDecoder:\n    \"\"\"Decodes a multipart message as bytes into Python events.\n\n    The part data is returned as available to allow the caller to save\n    the data from memory to disk, if desired.\n    \"\"\"\n\n    def __init__(\n        self,\n        boundary: bytes,\n        max_form_memory_size: Optional[int] = None,\n    ) -> None:\n        self.buffer = bytearray()\n        self.complete = False\n        self.max_form_memory_size = max_form_memory_size\n        self.state = State.PREAMBLE\n\n        # Note in the below \\h i.e. horizontal whitespace is used\n        # as [^\\S\\n\\r] as \\h isn't supported in python.\n\n        # The preamble must end with a boundary where the boundary is\n        # prefixed by a line break, RFC2046. Except that many\n        # implementations including Werkzeug's tests omit the line\n        # break prefix. In addition the first boundary could be the\n        # epilogue boundary (for empty form-data) hence the matching\n        # group to understand if it is an epilogue boundary.\n        self.preamble_re = re.compile(\n            br\"%s?--%s(--[^\\S\\n\\r]*%s?|[^\\S\\n\\r]*%s)\"\n            % (LINE_BREAK, boundary, LINE_BREAK, LINE_BREAK),\n            re.MULTILINE,\n        )\n        # A boundary must include a line break prefix and suffix, and\n        # may include trailing whitespace. In addition the boundary\n        # could be the epilogue boundary hence the matching group to\n        # understand if it is an epilogue boundary.\n        self.boundary_re = re.compile(\n            br\"%s--%s(--[^\\S\\n\\r]*%s?|[^\\S\\n\\r]*%s)\"\n            % (LINE_BREAK, boundary, LINE_BREAK, LINE_BREAK),\n            re.MULTILINE,\n        )\n\n    def receive_data(self, data: Optional[bytes]) -> None:\n        if data is None:\n            self.complete = True\n        elif (\n            self.max_form_memory_size is not None\n            and len(self.buffer) + len(data) > self.max_form_memory_size\n        ):\n            raise RequestEntityTooLarge()\n        else:\n            self.buffer.extend(data)\n\n    def next_event(self) -> Event:\n        event: Event = NEED_DATA\n\n        if self.state == State.PREAMBLE:\n            match = self.preamble_re.search(self.buffer)\n            if match is not None:\n                if match.group(1).startswith(b\"--\"):\n                    self.state = State.EPILOGUE\n                else:\n                    self.state = State.PART\n                data = bytes(self.buffer[: match.start()])\n                del self.buffer[: match.end()]\n                event = Preamble(data=data)\n\n        elif self.state == State.PART:\n            match = BLANK_LINE_RE.search(self.buffer)\n            if match is not None:\n                headers = self._parse_headers(self.buffer[: match.start()])\n                del self.buffer[: match.end()]\n\n                if \"content-disposition\" not in headers:\n                    raise ValueError(\"Missing Content-Disposition header\")\n\n                disposition, extra = parse_options_header(\n                    headers[\"content-disposition\"]\n                )\n                name = cast(str, extra.get(\"name\"))\n                filename = extra.get(\"filename\")\n                if filename is not None:\n                    event = File(\n                        filename=filename,\n                        headers=headers,\n                        name=name,\n                    )\n                else:\n                    event = Field(\n                        headers=headers,\n                        name=name,\n                    )\n                self.state = State.DATA\n\n        elif self.state == State.DATA:\n            # Return up to the last line break as data, anything past\n            # that line break could be a boundary - more data may be\n            # required to know for sure.\n            lines = list(LINE_BREAK_RE.finditer(self.buffer))\n            if len(lines):\n                data_length = del_index = lines[-1].start()\n                match = self.boundary_re.search(self.buffer)\n                if match is not None:\n                    if match.group(1).startswith(b\"--\"):\n                        self.state = State.EPILOGUE\n                    else:\n                        self.state = State.PART\n                    data_length = match.start()\n                    del_index = match.end()\n\n                data = bytes(self.buffer[:data_length])\n                del self.buffer[:del_index]\n                more_data = match is None\n                if data or not more_data:\n                    event = Data(data=data, more_data=more_data)\n\n        elif self.state == State.EPILOGUE and self.complete:\n            event = Epilogue(data=bytes(self.buffer))\n            del self.buffer[:]\n            self.state = State.COMPLETE\n\n        if self.complete and isinstance(event, NeedData):\n            raise ValueError(f\"Invalid form-data cannot parse beyond {self.state}\")\n\n        return event\n\n    def _parse_headers(self, data: bytes) -> Headers:\n        headers: List[Tuple[str, str]] = []\n        # Merge the continued headers into one line\n        data = HEADER_CONTINUATION_RE.sub(b\" \", data)\n        # Now there is one header per line\n        for line in data.splitlines():\n            if line.strip() != b\"\":\n                name, value = _to_str(line).strip().split(\":\", 1)\n                headers.append((name.strip(), value.strip()))\n        return Headers(headers)\n\n\nclass MultipartEncoder:\n    def __init__(self, boundary: bytes) -> None:\n        self.boundary = boundary\n        self.state = State.PREAMBLE\n\n    def send_event(self, event: Event) -> bytes:\n        if isinstance(event, Preamble) and self.state == State.PREAMBLE:\n            self.state = State.PART\n            return event.data\n        elif isinstance(event, (Field, File)) and self.state in {\n            State.PREAMBLE,\n            State.PART,\n            State.DATA,\n        }:\n            self.state = State.DATA\n            data = b\"\\r\\n--\" + self.boundary + b\"\\r\\n\"\n            data += b'Content-Disposition: form-data; name=\"%s\"' % _to_bytes(event.name)\n            if isinstance(event, File):\n                data += b'; filename=\"%s\"' % _to_bytes(event.filename)\n            data += b\"\\r\\n\"\n            for name, value in cast(Field, event).headers:\n                if name.lower() != \"content-disposition\":\n                    data += _to_bytes(f\"{name}: {value}\\r\\n\")\n            data += b\"\\r\\n\"\n            return data\n        elif isinstance(event, Data) and self.state == State.DATA:\n            return event.data\n        elif isinstance(event, Epilogue):\n            self.state = State.COMPLETE\n            return b\"\\r\\n--\" + self.boundary + b\"--\\r\\n\" + event.data\n        else:\n            raise ValueError(f\"Cannot generate {event} in state: {self.state}\")\n", "path": "src/werkzeug/sansio/multipart.py"}], "after_files": [{"content": "import re\nfrom dataclasses import dataclass\nfrom enum import auto\nfrom enum import Enum\nfrom typing import cast\nfrom typing import List\nfrom typing import Optional\nfrom typing import Tuple\n\nfrom .._internal import _to_bytes\nfrom .._internal import _to_str\nfrom ..datastructures import Headers\nfrom ..exceptions import RequestEntityTooLarge\nfrom ..http import parse_options_header\n\n\nclass Event:\n    pass\n\n\n@dataclass(frozen=True)\nclass Preamble(Event):\n    data: bytes\n\n\n@dataclass(frozen=True)\nclass Field(Event):\n    name: str\n    headers: Headers\n\n\n@dataclass(frozen=True)\nclass File(Event):\n    name: str\n    filename: str\n    headers: Headers\n\n\n@dataclass(frozen=True)\nclass Data(Event):\n    data: bytes\n    more_data: bool\n\n\n@dataclass(frozen=True)\nclass Epilogue(Event):\n    data: bytes\n\n\nclass NeedData(Event):\n    pass\n\n\nNEED_DATA = NeedData()\n\n\nclass State(Enum):\n    PREAMBLE = auto()\n    PART = auto()\n    DATA = auto()\n    EPILOGUE = auto()\n    COMPLETE = auto()\n\n\n# Multipart line breaks MUST be CRLF (\\r\\n) by RFC-7578, except that\n# many implementations break this and either use CR or LF alone.\nLINE_BREAK = b\"(?:\\r\\n|\\n|\\r)\"\nBLANK_LINE_RE = re.compile(b\"(?:\\r\\n\\r\\n|\\r\\r|\\n\\n)\", re.MULTILINE)\nLINE_BREAK_RE = re.compile(LINE_BREAK, re.MULTILINE)\n# Header values can be continued via a space or tab after the linebreak, as\n# per RFC2231\nHEADER_CONTINUATION_RE = re.compile(b\"%s[ \\t]\" % LINE_BREAK, re.MULTILINE)\n\n\nclass MultipartDecoder:\n    \"\"\"Decodes a multipart message as bytes into Python events.\n\n    The part data is returned as available to allow the caller to save\n    the data from memory to disk, if desired.\n    \"\"\"\n\n    def __init__(\n        self,\n        boundary: bytes,\n        max_form_memory_size: Optional[int] = None,\n    ) -> None:\n        self.buffer = bytearray()\n        self.complete = False\n        self.max_form_memory_size = max_form_memory_size\n        self.state = State.PREAMBLE\n        self.boundary = boundary\n\n        # Note in the below \\h i.e. horizontal whitespace is used\n        # as [^\\S\\n\\r] as \\h isn't supported in python.\n\n        # The preamble must end with a boundary where the boundary is\n        # prefixed by a line break, RFC2046. Except that many\n        # implementations including Werkzeug's tests omit the line\n        # break prefix. In addition the first boundary could be the\n        # epilogue boundary (for empty form-data) hence the matching\n        # group to understand if it is an epilogue boundary.\n        self.preamble_re = re.compile(\n            br\"%s?--%s(--[^\\S\\n\\r]*%s?|[^\\S\\n\\r]*%s)\"\n            % (LINE_BREAK, boundary, LINE_BREAK, LINE_BREAK),\n            re.MULTILINE,\n        )\n        # A boundary must include a line break prefix and suffix, and\n        # may include trailing whitespace. In addition the boundary\n        # could be the epilogue boundary hence the matching group to\n        # understand if it is an epilogue boundary.\n        self.boundary_re = re.compile(\n            br\"%s--%s(--[^\\S\\n\\r]*%s?|[^\\S\\n\\r]*%s)\"\n            % (LINE_BREAK, boundary, LINE_BREAK, LINE_BREAK),\n            re.MULTILINE,\n        )\n\n    def last_newline(self) -> int:\n        try:\n            last_nl = self.buffer.rindex(b\"\\n\")\n        except ValueError:\n            last_nl = len(self.buffer)\n        try:\n            last_cr = self.buffer.rindex(b\"\\r\")\n        except ValueError:\n            last_cr = len(self.buffer)\n\n        return min(last_nl, last_cr)\n\n    def receive_data(self, data: Optional[bytes]) -> None:\n        if data is None:\n            self.complete = True\n        elif (\n            self.max_form_memory_size is not None\n            and len(self.buffer) + len(data) > self.max_form_memory_size\n        ):\n            raise RequestEntityTooLarge()\n        else:\n            self.buffer.extend(data)\n\n    def next_event(self) -> Event:\n        event: Event = NEED_DATA\n\n        if self.state == State.PREAMBLE:\n            match = self.preamble_re.search(self.buffer)\n            if match is not None:\n                if match.group(1).startswith(b\"--\"):\n                    self.state = State.EPILOGUE\n                else:\n                    self.state = State.PART\n                data = bytes(self.buffer[: match.start()])\n                del self.buffer[: match.end()]\n                event = Preamble(data=data)\n\n        elif self.state == State.PART:\n            match = BLANK_LINE_RE.search(self.buffer)\n            if match is not None:\n                headers = self._parse_headers(self.buffer[: match.start()])\n                del self.buffer[: match.end()]\n\n                if \"content-disposition\" not in headers:\n                    raise ValueError(\"Missing Content-Disposition header\")\n\n                disposition, extra = parse_options_header(\n                    headers[\"content-disposition\"]\n                )\n                name = cast(str, extra.get(\"name\"))\n                filename = extra.get(\"filename\")\n                if filename is not None:\n                    event = File(\n                        filename=filename,\n                        headers=headers,\n                        name=name,\n                    )\n                else:\n                    event = Field(\n                        headers=headers,\n                        name=name,\n                    )\n                self.state = State.DATA\n\n        elif self.state == State.DATA:\n            if self.buffer.find(b\"--\" + self.boundary) == -1:\n                # No complete boundary in the buffer, but there may be\n                # a partial boundary at the end. As the boundary\n                # starts with either a nl or cr find the earliest and\n                # return up to that as data.\n                data_length = del_index = self.last_newline()\n                more_data = True\n            else:\n                match = self.boundary_re.search(self.buffer)\n                if match is not None:\n                    if match.group(1).startswith(b\"--\"):\n                        self.state = State.EPILOGUE\n                    else:\n                        self.state = State.PART\n                    data_length = match.start()\n                    del_index = match.end()\n                else:\n                    data_length = del_index = self.last_newline()\n                more_data = match is None\n\n            data = bytes(self.buffer[:data_length])\n            del self.buffer[:del_index]\n            if data or not more_data:\n                event = Data(data=data, more_data=more_data)\n\n        elif self.state == State.EPILOGUE and self.complete:\n            event = Epilogue(data=bytes(self.buffer))\n            del self.buffer[:]\n            self.state = State.COMPLETE\n\n        if self.complete and isinstance(event, NeedData):\n            raise ValueError(f\"Invalid form-data cannot parse beyond {self.state}\")\n\n        return event\n\n    def _parse_headers(self, data: bytes) -> Headers:\n        headers: List[Tuple[str, str]] = []\n        # Merge the continued headers into one line\n        data = HEADER_CONTINUATION_RE.sub(b\" \", data)\n        # Now there is one header per line\n        for line in data.splitlines():\n            if line.strip() != b\"\":\n                name, value = _to_str(line).strip().split(\":\", 1)\n                headers.append((name.strip(), value.strip()))\n        return Headers(headers)\n\n\nclass MultipartEncoder:\n    def __init__(self, boundary: bytes) -> None:\n        self.boundary = boundary\n        self.state = State.PREAMBLE\n\n    def send_event(self, event: Event) -> bytes:\n        if isinstance(event, Preamble) and self.state == State.PREAMBLE:\n            self.state = State.PART\n            return event.data\n        elif isinstance(event, (Field, File)) and self.state in {\n            State.PREAMBLE,\n            State.PART,\n            State.DATA,\n        }:\n            self.state = State.DATA\n            data = b\"\\r\\n--\" + self.boundary + b\"\\r\\n\"\n            data += b'Content-Disposition: form-data; name=\"%s\"' % _to_bytes(event.name)\n            if isinstance(event, File):\n                data += b'; filename=\"%s\"' % _to_bytes(event.filename)\n            data += b\"\\r\\n\"\n            for name, value in cast(Field, event).headers:\n                if name.lower() != \"content-disposition\":\n                    data += _to_bytes(f\"{name}: {value}\\r\\n\")\n            data += b\"\\r\\n\"\n            return data\n        elif isinstance(event, Data) and self.state == State.DATA:\n            return event.data\n        elif isinstance(event, Epilogue):\n            self.state = State.COMPLETE\n            return b\"\\r\\n--\" + self.boundary + b\"--\\r\\n\" + event.data\n        else:\n            raise ValueError(f\"Cannot generate {event} in state: {self.state}\")\n", "path": "src/werkzeug/sansio/multipart.py"}]}