We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Synapse is overly accepting of content in the `unsigned` object in events received over federation
Synapse makes use of various properties within the `unsigned` object of events - either internally, or by passing them on to clients. One example is `replaces_state`, which is used to store the event id of the previous event with the same `type` and `state_key`, and is later used to populate the `prev_content` property for events served to clients.

The problem is that homeservers are free to populate `unsigned`, without it affecting the event hashes or signatures; a malicious or buggy homeserver could therefore populate the content with incorrect data.

Taking the example of `replaces_state`, Synapse overwrites this property when receiving an event, but only if there was previously an event with the same `type` and `state_key` in the room state; it is otherwise passed through unchanged. So, a malicious homeserver could confuse remote servers' clients by sending incorrect values of `replaces_state` over federation.

---

The specification is not clear on how unspecified properties within `unsigned` should be handled, but I think they should be stripped off by the receiving homeserver. This will ensure that if, in future, the C-S API spec is extended to specify new properties be added to `unsigned`, there will be no confusion about whether they were added by the local or remote homeserver.

As far as I am aware, the only properties that *should* be allowed in `unsigned` over federation are:
 * `invite_room_state`
 * `knock_room_state`
 * `age` - though see also https://github.com/matrix-org/synapse/issues/8429.

[Aside: in an ideal world, we might have different properties for "things added by the remote homeserver - treat with caution!" vs "things added by the local homeserver - can be trusted". However, that ship has probably sailed for now.] 

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `synapse/federation/federation_base.py`
Content:
```
1 # Copyright 2015, 2016 OpenMarket Ltd
2 # Copyright 2020 The Matrix.org Foundation C.I.C.
3 #
4 # Licensed under the Apache License, Version 2.0 (the "License");
5 # you may not use this file except in compliance with the License.
6 # You may obtain a copy of the License at
7 #
8 #     http://www.apache.org/licenses/LICENSE-2.0
9 #
10 # Unless required by applicable law or agreed to in writing, software
11 # distributed under the License is distributed on an "AS IS" BASIS,
12 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 # See the License for the specific language governing permissions and
14 # limitations under the License.
15 import logging
16 from typing import TYPE_CHECKING
17 
18 from synapse.api.constants import MAX_DEPTH, EventContentFields, EventTypes, Membership
19 from synapse.api.errors import Codes, SynapseError
20 from synapse.api.room_versions import EventFormatVersions, RoomVersion
21 from synapse.crypto.event_signing import check_event_content_hash
22 from synapse.crypto.keyring import Keyring
23 from synapse.events import EventBase, make_event_from_dict
24 from synapse.events.utils import prune_event, validate_canonicaljson
25 from synapse.http.servlet import assert_params_in_dict
26 from synapse.types import JsonDict, get_domain_from_id
27 
28 if TYPE_CHECKING:
29     from synapse.server import HomeServer
30 
31 
32 logger = logging.getLogger(__name__)
33 
34 
35 class FederationBase:
36     def __init__(self, hs: "HomeServer"):
37         self.hs = hs
38 
39         self.server_name = hs.hostname
40         self.keyring = hs.get_keyring()
41         self.spam_checker = hs.get_spam_checker()
42         self.store = hs.get_datastore()
43         self._clock = hs.get_clock()
44 
45     async def _check_sigs_and_hash(
46         self, room_version: RoomVersion, pdu: EventBase
47     ) -> EventBase:
48         """Checks that event is correctly signed by the sending server.
49 
50         Args:
51             room_version: The room version of the PDU
52             pdu: the event to be checked
53 
54         Returns:
55               * the original event if the checks pass
56               * a redacted version of the event (if the signature
57                 matched but the hash did not)
58               * throws a SynapseError if the signature check failed."""
59         try:
60             await _check_sigs_on_pdu(self.keyring, room_version, pdu)
61         except SynapseError as e:
62             logger.warning(
63                 "Signature check failed for %s: %s",
64                 pdu.event_id,
65                 e,
66             )
67             raise
68 
69         if not check_event_content_hash(pdu):
70             # let's try to distinguish between failures because the event was
71             # redacted (which are somewhat expected) vs actual ball-tampering
72             # incidents.
73             #
74             # This is just a heuristic, so we just assume that if the keys are
75             # about the same between the redacted and received events, then the
76             # received event was probably a redacted copy (but we then use our
77             # *actual* redacted copy to be on the safe side.)
78             redacted_event = prune_event(pdu)
79             if set(redacted_event.keys()) == set(pdu.keys()) and set(
80                 redacted_event.content.keys()
81             ) == set(pdu.content.keys()):
82                 logger.info(
83                     "Event %s seems to have been redacted; using our redacted copy",
84                     pdu.event_id,
85                 )
86             else:
87                 logger.warning(
88                     "Event %s content has been tampered, redacting",
89                     pdu.event_id,
90                 )
91             return redacted_event
92 
93         result = await self.spam_checker.check_event_for_spam(pdu)
94 
95         if result:
96             logger.warning("Event contains spam, soft-failing %s", pdu.event_id)
97             # we redact (to save disk space) as well as soft-failing (to stop
98             # using the event in prev_events).
99             redacted_event = prune_event(pdu)
100             redacted_event.internal_metadata.soft_failed = True
101             return redacted_event
102 
103         return pdu
104 
105 
106 async def _check_sigs_on_pdu(
107     keyring: Keyring, room_version: RoomVersion, pdu: EventBase
108 ) -> None:
109     """Check that the given events are correctly signed
110 
111     Raise a SynapseError if the event wasn't correctly signed.
112 
113     Args:
114         keyring: keyring object to do the checks
115         room_version: the room version of the PDUs
116         pdus: the events to be checked
117     """
118 
119     # we want to check that the event is signed by:
120     #
121     # (a) the sender's server
122     #
123     #     - except in the case of invites created from a 3pid invite, which are exempt
124     #     from this check, because the sender has to match that of the original 3pid
125     #     invite, but the event may come from a different HS, for reasons that I don't
126     #     entirely grok (why do the senders have to match? and if they do, why doesn't the
127     #     joining server ask the inviting server to do the switcheroo with
128     #     exchange_third_party_invite?).
129     #
130     #     That's pretty awful, since redacting such an invite will render it invalid
131     #     (because it will then look like a regular invite without a valid signature),
132     #     and signatures are *supposed* to be valid whether or not an event has been
133     #     redacted. But this isn't the worst of the ways that 3pid invites are broken.
134     #
135     # (b) for V1 and V2 rooms, the server which created the event_id
136     #
137     # let's start by getting the domain for each pdu, and flattening the event back
138     # to JSON.
139 
140     # First we check that the sender event is signed by the sender's domain
141     # (except if its a 3pid invite, in which case it may be sent by any server)
142     if not _is_invite_via_3pid(pdu):
143         try:
144             await keyring.verify_event_for_server(
145                 get_domain_from_id(pdu.sender),
146                 pdu,
147                 pdu.origin_server_ts if room_version.enforce_key_validity else 0,
148             )
149         except Exception as e:
150             errmsg = "event id %s: unable to verify signature for sender %s: %s" % (
151                 pdu.event_id,
152                 get_domain_from_id(pdu.sender),
153                 e,
154             )
155             raise SynapseError(403, errmsg, Codes.FORBIDDEN)
156 
157     # now let's look for events where the sender's domain is different to the
158     # event id's domain (normally only the case for joins/leaves), and add additional
159     # checks. Only do this if the room version has a concept of event ID domain
160     # (ie, the room version uses old-style non-hash event IDs).
161     if room_version.event_format == EventFormatVersions.V1 and get_domain_from_id(
162         pdu.event_id
163     ) != get_domain_from_id(pdu.sender):
164         try:
165             await keyring.verify_event_for_server(
166                 get_domain_from_id(pdu.event_id),
167                 pdu,
168                 pdu.origin_server_ts if room_version.enforce_key_validity else 0,
169             )
170         except Exception as e:
171             errmsg = (
172                 "event id %s: unable to verify signature for event id domain %s: %s"
173                 % (
174                     pdu.event_id,
175                     get_domain_from_id(pdu.event_id),
176                     e,
177                 )
178             )
179             raise SynapseError(403, errmsg, Codes.FORBIDDEN)
180 
181     # If this is a join event for a restricted room it may have been authorised
182     # via a different server from the sending server. Check those signatures.
183     if (
184         room_version.msc3083_join_rules
185         and pdu.type == EventTypes.Member
186         and pdu.membership == Membership.JOIN
187         and EventContentFields.AUTHORISING_USER in pdu.content
188     ):
189         authorising_server = get_domain_from_id(
190             pdu.content[EventContentFields.AUTHORISING_USER]
191         )
192         try:
193             await keyring.verify_event_for_server(
194                 authorising_server,
195                 pdu,
196                 pdu.origin_server_ts if room_version.enforce_key_validity else 0,
197             )
198         except Exception as e:
199             errmsg = (
200                 "event id %s: unable to verify signature for authorising server %s: %s"
201                 % (
202                     pdu.event_id,
203                     authorising_server,
204                     e,
205                 )
206             )
207             raise SynapseError(403, errmsg, Codes.FORBIDDEN)
208 
209 
210 def _is_invite_via_3pid(event: EventBase) -> bool:
211     return (
212         event.type == EventTypes.Member
213         and event.membership == Membership.INVITE
214         and "third_party_invite" in event.content
215     )
216 
217 
218 def event_from_pdu_json(pdu_json: JsonDict, room_version: RoomVersion) -> EventBase:
219     """Construct an EventBase from an event json received over federation
220 
221     Args:
222         pdu_json: pdu as received over federation
223         room_version: The version of the room this event belongs to
224 
225     Raises:
226         SynapseError: if the pdu is missing required fields or is otherwise
227             not a valid matrix event
228     """
229     # we could probably enforce a bunch of other fields here (room_id, sender,
230     # origin, etc etc)
231     assert_params_in_dict(pdu_json, ("type", "depth"))
232 
233     depth = pdu_json["depth"]
234     if not isinstance(depth, int):
235         raise SynapseError(400, "Depth %r not an intger" % (depth,), Codes.BAD_JSON)
236 
237     if depth < 0:
238         raise SynapseError(400, "Depth too small", Codes.BAD_JSON)
239     elif depth > MAX_DEPTH:
240         raise SynapseError(400, "Depth too large", Codes.BAD_JSON)
241 
242     # Validate that the JSON conforms to the specification.
243     if room_version.strict_canonicaljson:
244         validate_canonicaljson(pdu_json)
245 
246     event = make_event_from_dict(pdu_json, room_version)
247     return event
248
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/synapse/federation/federation_base.py b/synapse/federation/federation_base.py
--- a/synapse/federation/federation_base.py
+++ b/synapse/federation/federation_base.py
@@ -230,6 +230,10 @@
     # origin, etc etc)
     assert_params_in_dict(pdu_json, ("type", "depth"))
 
+    # Strip any unauthorized values from "unsigned" if they exist
+    if "unsigned" in pdu_json:
+        _strip_unsigned_values(pdu_json)
+
     depth = pdu_json["depth"]
     if not isinstance(depth, int):
         raise SynapseError(400, "Depth %r not an intger" % (depth,), Codes.BAD_JSON)
@@ -245,3 +249,24 @@
 
     event = make_event_from_dict(pdu_json, room_version)
     return event
+
+
+def _strip_unsigned_values(pdu_dict: JsonDict) -> None:
+    """
+    Strip any unsigned values unless specifically allowed, as defined by the whitelist.
+
+    pdu: the json dict to strip values from. Note that the dict is mutated by this
+    function
+    """
+    unsigned = pdu_dict["unsigned"]
+
+    if not isinstance(unsigned, dict):
+        pdu_dict["unsigned"] = {}
+
+    if pdu_dict["type"] == "m.room.member":
+        whitelist = ["knock_room_state", "invite_room_state", "age"]
+    else:
+        whitelist = ["age"]
+
+    filtered_unsigned = {k: v for k, v in unsigned.items() if k in whitelist}
+    pdu_dict["unsigned"] = filtered_unsigned

{"golden_diff": "diff --git a/synapse/federation/federation_base.py b/synapse/federation/federation_base.py\n--- a/synapse/federation/federation_base.py\n+++ b/synapse/federation/federation_base.py\n@@ -230,6 +230,10 @@\n     # origin, etc etc)\n     assert_params_in_dict(pdu_json, (\"type\", \"depth\"))\n \n+    # Strip any unauthorized values from \"unsigned\" if they exist\n+    if \"unsigned\" in pdu_json:\n+        _strip_unsigned_values(pdu_json)\n+\n     depth = pdu_json[\"depth\"]\n     if not isinstance(depth, int):\n         raise SynapseError(400, \"Depth %r not an intger\" % (depth,), Codes.BAD_JSON)\n@@ -245,3 +249,24 @@\n \n     event = make_event_from_dict(pdu_json, room_version)\n     return event\n+\n+\n+def _strip_unsigned_values(pdu_dict: JsonDict) -> None:\n+    \"\"\"\n+    Strip any unsigned values unless specifically allowed, as defined by the whitelist.\n+\n+    pdu: the json dict to strip values from. Note that the dict is mutated by this\n+    function\n+    \"\"\"\n+    unsigned = pdu_dict[\"unsigned\"]\n+\n+    if not isinstance(unsigned, dict):\n+        pdu_dict[\"unsigned\"] = {}\n+\n+    if pdu_dict[\"type\"] == \"m.room.member\":\n+        whitelist = [\"knock_room_state\", \"invite_room_state\", \"age\"]\n+    else:\n+        whitelist = [\"age\"]\n+\n+    filtered_unsigned = {k: v for k, v in unsigned.items() if k in whitelist}\n+    pdu_dict[\"unsigned\"] = filtered_unsigned\n", "issue": "Synapse is overly accepting of content in the `unsigned` object in events received over federation\nSynapse makes use of various properties within the `unsigned` object of events - either internally, or by passing them on to clients. One example is `replaces_state`, which is used to store the event id of the previous event with the same `type` and `state_key`, and is later used to populate the `prev_content` property for events served to clients.\r\n\r\nThe problem is that homeservers are free to populate `unsigned`, without it affecting the event hashes or signatures; a malicious or buggy homeserver could therefore populate the content with incorrect data.\r\n\r\nTaking the example of `replaces_state`, Synapse overwrites this property when receiving an event, but only if there was previously an event with the same `type` and `state_key` in the room state; it is otherwise passed through unchanged. So, a malicious homeserver could confuse remote servers' clients by sending incorrect values of `replaces_state` over federation.\r\n\r\n---\r\n\r\nThe specification is not clear on how unspecified properties within `unsigned` should be handled, but I think they should be stripped off by the receiving homeserver. This will ensure that if, in future, the C-S API spec is extended to specify new properties be added to `unsigned`, there will be no confusion about whether they were added by the local or remote homeserver.\r\n\r\nAs far as I am aware, the only properties that *should* be allowed in `unsigned` over federation are:\r\n * `invite_room_state`\r\n * `knock_room_state`\r\n * `age` - though see also https://github.com/matrix-org/synapse/issues/8429.\r\n\r\n[Aside: in an ideal world, we might have different properties for \"things added by the remote homeserver - treat with caution!\" vs \"things added by the local homeserver - can be trusted\". However, that ship has probably sailed for now.] \n", "before_files": [{"content": "# Copyright 2015, 2016 OpenMarket Ltd\n# Copyright 2020 The Matrix.org Foundation C.I.C.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport logging\nfrom typing import TYPE_CHECKING\n\nfrom synapse.api.constants import MAX_DEPTH, EventContentFields, EventTypes, Membership\nfrom synapse.api.errors import Codes, SynapseError\nfrom synapse.api.room_versions import EventFormatVersions, RoomVersion\nfrom synapse.crypto.event_signing import check_event_content_hash\nfrom synapse.crypto.keyring import Keyring\nfrom synapse.events import EventBase, make_event_from_dict\nfrom synapse.events.utils import prune_event, validate_canonicaljson\nfrom synapse.http.servlet import assert_params_in_dict\nfrom synapse.types import JsonDict, get_domain_from_id\n\nif TYPE_CHECKING:\n    from synapse.server import HomeServer\n\n\nlogger = logging.getLogger(__name__)\n\n\nclass FederationBase:\n    def __init__(self, hs: \"HomeServer\"):\n        self.hs = hs\n\n        self.server_name = hs.hostname\n        self.keyring = hs.get_keyring()\n        self.spam_checker = hs.get_spam_checker()\n        self.store = hs.get_datastore()\n        self._clock = hs.get_clock()\n\n    async def _check_sigs_and_hash(\n        self, room_version: RoomVersion, pdu: EventBase\n    ) -> EventBase:\n        \"\"\"Checks that event is correctly signed by the sending server.\n\n        Args:\n            room_version: The room version of the PDU\n            pdu: the event to be checked\n\n        Returns:\n              * the original event if the checks pass\n              * a redacted version of the event (if the signature\n                matched but the hash did not)\n              * throws a SynapseError if the signature check failed.\"\"\"\n        try:\n            await _check_sigs_on_pdu(self.keyring, room_version, pdu)\n        except SynapseError as e:\n            logger.warning(\n                \"Signature check failed for %s: %s\",\n                pdu.event_id,\n                e,\n            )\n            raise\n\n        if not check_event_content_hash(pdu):\n            # let's try to distinguish between failures because the event was\n            # redacted (which are somewhat expected) vs actual ball-tampering\n            # incidents.\n            #\n            # This is just a heuristic, so we just assume that if the keys are\n            # about the same between the redacted and received events, then the\n            # received event was probably a redacted copy (but we then use our\n            # *actual* redacted copy to be on the safe side.)\n            redacted_event = prune_event(pdu)\n            if set(redacted_event.keys()) == set(pdu.keys()) and set(\n                redacted_event.content.keys()\n            ) == set(pdu.content.keys()):\n                logger.info(\n                    \"Event %s seems to have been redacted; using our redacted copy\",\n                    pdu.event_id,\n                )\n            else:\n                logger.warning(\n                    \"Event %s content has been tampered, redacting\",\n                    pdu.event_id,\n                )\n            return redacted_event\n\n        result = await self.spam_checker.check_event_for_spam(pdu)\n\n        if result:\n            logger.warning(\"Event contains spam, soft-failing %s\", pdu.event_id)\n            # we redact (to save disk space) as well as soft-failing (to stop\n            # using the event in prev_events).\n            redacted_event = prune_event(pdu)\n            redacted_event.internal_metadata.soft_failed = True\n            return redacted_event\n\n        return pdu\n\n\nasync def _check_sigs_on_pdu(\n    keyring: Keyring, room_version: RoomVersion, pdu: EventBase\n) -> None:\n    \"\"\"Check that the given events are correctly signed\n\n    Raise a SynapseError if the event wasn't correctly signed.\n\n    Args:\n        keyring: keyring object to do the checks\n        room_version: the room version of the PDUs\n        pdus: the events to be checked\n    \"\"\"\n\n    # we want to check that the event is signed by:\n    #\n    # (a) the sender's server\n    #\n    #     - except in the case of invites created from a 3pid invite, which are exempt\n    #     from this check, because the sender has to match that of the original 3pid\n    #     invite, but the event may come from a different HS, for reasons that I don't\n    #     entirely grok (why do the senders have to match? and if they do, why doesn't the\n    #     joining server ask the inviting server to do the switcheroo with\n    #     exchange_third_party_invite?).\n    #\n    #     That's pretty awful, since redacting such an invite will render it invalid\n    #     (because it will then look like a regular invite without a valid signature),\n    #     and signatures are *supposed* to be valid whether or not an event has been\n    #     redacted. But this isn't the worst of the ways that 3pid invites are broken.\n    #\n    # (b) for V1 and V2 rooms, the server which created the event_id\n    #\n    # let's start by getting the domain for each pdu, and flattening the event back\n    # to JSON.\n\n    # First we check that the sender event is signed by the sender's domain\n    # (except if its a 3pid invite, in which case it may be sent by any server)\n    if not _is_invite_via_3pid(pdu):\n        try:\n            await keyring.verify_event_for_server(\n                get_domain_from_id(pdu.sender),\n                pdu,\n                pdu.origin_server_ts if room_version.enforce_key_validity else 0,\n            )\n        except Exception as e:\n            errmsg = \"event id %s: unable to verify signature for sender %s: %s\" % (\n                pdu.event_id,\n                get_domain_from_id(pdu.sender),\n                e,\n            )\n            raise SynapseError(403, errmsg, Codes.FORBIDDEN)\n\n    # now let's look for events where the sender's domain is different to the\n    # event id's domain (normally only the case for joins/leaves), and add additional\n    # checks. Only do this if the room version has a concept of event ID domain\n    # (ie, the room version uses old-style non-hash event IDs).\n    if room_version.event_format == EventFormatVersions.V1 and get_domain_from_id(\n        pdu.event_id\n    ) != get_domain_from_id(pdu.sender):\n        try:\n            await keyring.verify_event_for_server(\n                get_domain_from_id(pdu.event_id),\n                pdu,\n                pdu.origin_server_ts if room_version.enforce_key_validity else 0,\n            )\n        except Exception as e:\n            errmsg = (\n                \"event id %s: unable to verify signature for event id domain %s: %s\"\n                % (\n                    pdu.event_id,\n                    get_domain_from_id(pdu.event_id),\n                    e,\n                )\n            )\n            raise SynapseError(403, errmsg, Codes.FORBIDDEN)\n\n    # If this is a join event for a restricted room it may have been authorised\n    # via a different server from the sending server. Check those signatures.\n    if (\n        room_version.msc3083_join_rules\n        and pdu.type == EventTypes.Member\n        and pdu.membership == Membership.JOIN\n        and EventContentFields.AUTHORISING_USER in pdu.content\n    ):\n        authorising_server = get_domain_from_id(\n            pdu.content[EventContentFields.AUTHORISING_USER]\n        )\n        try:\n            await keyring.verify_event_for_server(\n                authorising_server,\n                pdu,\n                pdu.origin_server_ts if room_version.enforce_key_validity else 0,\n            )\n        except Exception as e:\n            errmsg = (\n                \"event id %s: unable to verify signature for authorising server %s: %s\"\n                % (\n                    pdu.event_id,\n                    authorising_server,\n                    e,\n                )\n            )\n            raise SynapseError(403, errmsg, Codes.FORBIDDEN)\n\n\ndef _is_invite_via_3pid(event: EventBase) -> bool:\n    return (\n        event.type == EventTypes.Member\n        and event.membership == Membership.INVITE\n        and \"third_party_invite\" in event.content\n    )\n\n\ndef event_from_pdu_json(pdu_json: JsonDict, room_version: RoomVersion) -> EventBase:\n    \"\"\"Construct an EventBase from an event json received over federation\n\n    Args:\n        pdu_json: pdu as received over federation\n        room_version: The version of the room this event belongs to\n\n    Raises:\n        SynapseError: if the pdu is missing required fields or is otherwise\n            not a valid matrix event\n    \"\"\"\n    # we could probably enforce a bunch of other fields here (room_id, sender,\n    # origin, etc etc)\n    assert_params_in_dict(pdu_json, (\"type\", \"depth\"))\n\n    depth = pdu_json[\"depth\"]\n    if not isinstance(depth, int):\n        raise SynapseError(400, \"Depth %r not an intger\" % (depth,), Codes.BAD_JSON)\n\n    if depth < 0:\n        raise SynapseError(400, \"Depth too small\", Codes.BAD_JSON)\n    elif depth > MAX_DEPTH:\n        raise SynapseError(400, \"Depth too large\", Codes.BAD_JSON)\n\n    # Validate that the JSON conforms to the specification.\n    if room_version.strict_canonicaljson:\n        validate_canonicaljson(pdu_json)\n\n    event = make_event_from_dict(pdu_json, room_version)\n    return event\n", "path": "synapse/federation/federation_base.py"}], "after_files": [{"content": "# Copyright 2015, 2016 OpenMarket Ltd\n# Copyright 2020 The Matrix.org Foundation C.I.C.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport logging\nfrom typing import TYPE_CHECKING\n\nfrom synapse.api.constants import MAX_DEPTH, EventContentFields, EventTypes, Membership\nfrom synapse.api.errors import Codes, SynapseError\nfrom synapse.api.room_versions import EventFormatVersions, RoomVersion\nfrom synapse.crypto.event_signing import check_event_content_hash\nfrom synapse.crypto.keyring import Keyring\nfrom synapse.events import EventBase, make_event_from_dict\nfrom synapse.events.utils import prune_event, validate_canonicaljson\nfrom synapse.http.servlet import assert_params_in_dict\nfrom synapse.types import JsonDict, get_domain_from_id\n\nif TYPE_CHECKING:\n    from synapse.server import HomeServer\n\n\nlogger = logging.getLogger(__name__)\n\n\nclass FederationBase:\n    def __init__(self, hs: \"HomeServer\"):\n        self.hs = hs\n\n        self.server_name = hs.hostname\n        self.keyring = hs.get_keyring()\n        self.spam_checker = hs.get_spam_checker()\n        self.store = hs.get_datastore()\n        self._clock = hs.get_clock()\n\n    async def _check_sigs_and_hash(\n        self, room_version: RoomVersion, pdu: EventBase\n    ) -> EventBase:\n        \"\"\"Checks that event is correctly signed by the sending server.\n\n        Args:\n            room_version: The room version of the PDU\n            pdu: the event to be checked\n\n        Returns:\n              * the original event if the checks pass\n              * a redacted version of the event (if the signature\n                matched but the hash did not)\n              * throws a SynapseError if the signature check failed.\"\"\"\n        try:\n            await _check_sigs_on_pdu(self.keyring, room_version, pdu)\n        except SynapseError as e:\n            logger.warning(\n                \"Signature check failed for %s: %s\",\n                pdu.event_id,\n                e,\n            )\n            raise\n\n        if not check_event_content_hash(pdu):\n            # let's try to distinguish between failures because the event was\n            # redacted (which are somewhat expected) vs actual ball-tampering\n            # incidents.\n            #\n            # This is just a heuristic, so we just assume that if the keys are\n            # about the same between the redacted and received events, then the\n            # received event was probably a redacted copy (but we then use our\n            # *actual* redacted copy to be on the safe side.)\n            redacted_event = prune_event(pdu)\n            if set(redacted_event.keys()) == set(pdu.keys()) and set(\n                redacted_event.content.keys()\n            ) == set(pdu.content.keys()):\n                logger.info(\n                    \"Event %s seems to have been redacted; using our redacted copy\",\n                    pdu.event_id,\n                )\n            else:\n                logger.warning(\n                    \"Event %s content has been tampered, redacting\",\n                    pdu.event_id,\n                )\n            return redacted_event\n\n        result = await self.spam_checker.check_event_for_spam(pdu)\n\n        if result:\n            logger.warning(\"Event contains spam, soft-failing %s\", pdu.event_id)\n            # we redact (to save disk space) as well as soft-failing (to stop\n            # using the event in prev_events).\n            redacted_event = prune_event(pdu)\n            redacted_event.internal_metadata.soft_failed = True\n            return redacted_event\n\n        return pdu\n\n\nasync def _check_sigs_on_pdu(\n    keyring: Keyring, room_version: RoomVersion, pdu: EventBase\n) -> None:\n    \"\"\"Check that the given events are correctly signed\n\n    Raise a SynapseError if the event wasn't correctly signed.\n\n    Args:\n        keyring: keyring object to do the checks\n        room_version: the room version of the PDUs\n        pdus: the events to be checked\n    \"\"\"\n\n    # we want to check that the event is signed by:\n    #\n    # (a) the sender's server\n    #\n    #     - except in the case of invites created from a 3pid invite, which are exempt\n    #     from this check, because the sender has to match that of the original 3pid\n    #     invite, but the event may come from a different HS, for reasons that I don't\n    #     entirely grok (why do the senders have to match? and if they do, why doesn't the\n    #     joining server ask the inviting server to do the switcheroo with\n    #     exchange_third_party_invite?).\n    #\n    #     That's pretty awful, since redacting such an invite will render it invalid\n    #     (because it will then look like a regular invite without a valid signature),\n    #     and signatures are *supposed* to be valid whether or not an event has been\n    #     redacted. But this isn't the worst of the ways that 3pid invites are broken.\n    #\n    # (b) for V1 and V2 rooms, the server which created the event_id\n    #\n    # let's start by getting the domain for each pdu, and flattening the event back\n    # to JSON.\n\n    # First we check that the sender event is signed by the sender's domain\n    # (except if its a 3pid invite, in which case it may be sent by any server)\n    if not _is_invite_via_3pid(pdu):\n        try:\n            await keyring.verify_event_for_server(\n                get_domain_from_id(pdu.sender),\n                pdu,\n                pdu.origin_server_ts if room_version.enforce_key_validity else 0,\n            )\n        except Exception as e:\n            errmsg = \"event id %s: unable to verify signature for sender %s: %s\" % (\n                pdu.event_id,\n                get_domain_from_id(pdu.sender),\n                e,\n            )\n            raise SynapseError(403, errmsg, Codes.FORBIDDEN)\n\n    # now let's look for events where the sender's domain is different to the\n    # event id's domain (normally only the case for joins/leaves), and add additional\n    # checks. Only do this if the room version has a concept of event ID domain\n    # (ie, the room version uses old-style non-hash event IDs).\n    if room_version.event_format == EventFormatVersions.V1 and get_domain_from_id(\n        pdu.event_id\n    ) != get_domain_from_id(pdu.sender):\n        try:\n            await keyring.verify_event_for_server(\n                get_domain_from_id(pdu.event_id),\n                pdu,\n                pdu.origin_server_ts if room_version.enforce_key_validity else 0,\n            )\n        except Exception as e:\n            errmsg = (\n                \"event id %s: unable to verify signature for event id domain %s: %s\"\n                % (\n                    pdu.event_id,\n                    get_domain_from_id(pdu.event_id),\n                    e,\n                )\n            )\n            raise SynapseError(403, errmsg, Codes.FORBIDDEN)\n\n    # If this is a join event for a restricted room it may have been authorised\n    # via a different server from the sending server. Check those signatures.\n    if (\n        room_version.msc3083_join_rules\n        and pdu.type == EventTypes.Member\n        and pdu.membership == Membership.JOIN\n        and EventContentFields.AUTHORISING_USER in pdu.content\n    ):\n        authorising_server = get_domain_from_id(\n            pdu.content[EventContentFields.AUTHORISING_USER]\n        )\n        try:\n            await keyring.verify_event_for_server(\n                authorising_server,\n                pdu,\n                pdu.origin_server_ts if room_version.enforce_key_validity else 0,\n            )\n        except Exception as e:\n            errmsg = (\n                \"event id %s: unable to verify signature for authorising server %s: %s\"\n                % (\n                    pdu.event_id,\n                    authorising_server,\n                    e,\n                )\n            )\n            raise SynapseError(403, errmsg, Codes.FORBIDDEN)\n\n\ndef _is_invite_via_3pid(event: EventBase) -> bool:\n    return (\n        event.type == EventTypes.Member\n        and event.membership == Membership.INVITE\n        and \"third_party_invite\" in event.content\n    )\n\n\ndef event_from_pdu_json(pdu_json: JsonDict, room_version: RoomVersion) -> EventBase:\n    \"\"\"Construct an EventBase from an event json received over federation\n\n    Args:\n        pdu_json: pdu as received over federation\n        room_version: The version of the room this event belongs to\n\n    Raises:\n        SynapseError: if the pdu is missing required fields or is otherwise\n            not a valid matrix event\n    \"\"\"\n    # we could probably enforce a bunch of other fields here (room_id, sender,\n    # origin, etc etc)\n    assert_params_in_dict(pdu_json, (\"type\", \"depth\"))\n\n    # Strip any unauthorized values from \"unsigned\" if they exist\n    if \"unsigned\" in pdu_json:\n        _strip_unsigned_values(pdu_json)\n\n    depth = pdu_json[\"depth\"]\n    if not isinstance(depth, int):\n        raise SynapseError(400, \"Depth %r not an intger\" % (depth,), Codes.BAD_JSON)\n\n    if depth < 0:\n        raise SynapseError(400, \"Depth too small\", Codes.BAD_JSON)\n    elif depth > MAX_DEPTH:\n        raise SynapseError(400, \"Depth too large\", Codes.BAD_JSON)\n\n    # Validate that the JSON conforms to the specification.\n    if room_version.strict_canonicaljson:\n        validate_canonicaljson(pdu_json)\n\n    event = make_event_from_dict(pdu_json, room_version)\n    return event\n\n\ndef _strip_unsigned_values(pdu_dict: JsonDict) -> None:\n    \"\"\"\n    Strip any unsigned values unless specifically allowed, as defined by the whitelist.\n\n    pdu: the json dict to strip values from. Note that the dict is mutated by this\n    function\n    \"\"\"\n    unsigned = pdu_dict[\"unsigned\"]\n\n    if not isinstance(unsigned, dict):\n        pdu_dict[\"unsigned\"] = {}\n\n    if pdu_dict[\"type\"] == \"m.room.member\":\n        whitelist = [\"knock_room_state\", \"invite_room_state\", \"age\"]\n    else:\n        whitelist = [\"age\"]\n\n    filtered_unsigned = {k: v for k, v in unsigned.items() if k in whitelist}\n    pdu_dict[\"unsigned\"] = filtered_unsigned\n", "path": "synapse/federation/federation_base.py"}]}