We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Unhandled JSON pointer errors
When setting a schema on resources, unresolvable references are not checked upon creation.

When creating a resource that is checked against that schema, a `RefResolutionError` is raised by `jsonschema` and not caught.

For example, creating the schema:

```json
{"data": {
    "schema": {
      "properties": {
        "address": { "$ref": "#/definitions/address" }
      }
    }
  }
}
```

POST data:

```json
{ "data": {
   "address": "seattle"
  }
}
```

This will crash the server with a 500 error and the following message:

```shell
"Unresolvable JSON pointer: %r" % fragment
jsonschema.exceptions.RefResolutionError: Unresolvable JSON pointer: 'definitions/address' 
```

I can work on this, I was just wondering what you all felt the approach should be.

It can either check when creating the schema or upon each invocation of it (i.e., POST data). The only slight trouble I can see with checking and not allowing unresolvable schema is that the schema can be a url, so if it is not resolving at the time of creation, it will fail.

That seems fair to me; does that seem like a reasonable fix?


Unhandled JSON pointer errors
When setting a schema on resources, unresolvable references are not checked upon creation.

When creating a resource that is checked against that schema, a `RefResolutionError` is raised by `jsonschema` and not caught.

For example, creating the schema:

```json
{"data": {
    "schema": {
      "properties": {
        "address": { "$ref": "#/definitions/address" }
      }
    }
  }
}
```

POST data:

```json
{ "data": {
   "address": "seattle"
  }
}
```

This will crash the server with a 500 error and the following message:

```shell
"Unresolvable JSON pointer: %r" % fragment
jsonschema.exceptions.RefResolutionError: Unresolvable JSON pointer: 'definitions/address' 
```

I can work on this, I was just wondering what you all felt the approach should be.

It can either check when creating the schema or upon each invocation of it (i.e., POST data). The only slight trouble I can see with checking and not allowing unresolvable schema is that the schema can be a url, so if it is not resolving at the time of creation, it will fail.

That seems fair to me; does that seem like a reasonable fix?



--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `kinto/schema_validation.py`
Content:
```
1 import colander
2 from jsonschema import Draft4Validator, ValidationError, SchemaError, validate
3 from pyramid.settings import asbool
4 
5 from kinto.core import utils
6 from kinto.core.errors import raise_invalid
7 from kinto.views import object_exists_or_404
8 
9 
10 class JSONSchemaMapping(colander.SchemaNode):
11     def schema_type(self, **kw):
12         return colander.Mapping(unknown='preserve')
13 
14     def deserialize(self, cstruct=colander.null):
15         # Start by deserializing a simple mapping.
16         validated = super().deserialize(cstruct)
17 
18         # In case it is optional in parent schema.
19         if not validated or validated in (colander.null, colander.drop):
20             return validated
21         try:
22             check_schema(validated)
23         except ValidationError as e:
24             self.raise_invalid(e.message)
25         return validated
26 
27 
28 def check_schema(data):
29     try:
30         Draft4Validator.check_schema(data)
31     except SchemaError as e:
32         message = e.path.pop() + e.message
33         raise ValidationError(message)
34 
35 
36 def validate_schema(data, schema, ignore_fields=[]):
37     required_fields = [f for f in schema.get('required', []) if f not in ignore_fields]
38     # jsonschema doesn't accept 'required': [] yet.
39     # See https://github.com/Julian/jsonschema/issues/337.
40     # In the meantime, strip out 'required' if no other fields are required.
41     if required_fields:
42         schema = {**schema, 'required': required_fields}
43     else:
44         schema = {f: v for f, v in schema.items() if f != 'required'}
45 
46     data = {f: v for f, v in data.items() if f not in ignore_fields}
47 
48     try:
49         validate(data, schema)
50     except ValidationError as e:
51         if e.path:
52             field = e.path[-1]
53         elif e.validator_value:
54             field = e.validator_value[-1]
55         else:
56             field = e.schema_path[-1]
57         e.field = field
58         raise e
59 
60 
61 def validate_from_bucket_schema_or_400(data, resource_name, request, ignore_fields=[]):
62     """Lookup in the parent objects if a schema was defined for this resource.
63 
64     If the schema validation feature is enabled, if a schema is/are defined, and if the
65     data does not validate it/them, then it raises a 400 exception.
66     """
67     settings = request.registry.settings
68     schema_validation = 'experimental_collection_schema_validation'
69     # If disabled from settings, do nothing.
70     if not asbool(settings.get(schema_validation)):
71         return
72 
73     bucket_id = request.matchdict["bucket_id"]
74     bucket_uri = utils.instance_uri(request, 'bucket', id=bucket_id)
75     buckets = request.bound_data.setdefault('buckets', {})
76     if bucket_uri not in buckets:
77         # Unknown yet, fetch from storage.
78         bucket = object_exists_or_404(request,
79                                       collection_id='bucket',
80                                       parent_id='',
81                                       object_id=bucket_id)
82         buckets[bucket_uri] = bucket
83 
84     # Let's see if the bucket defines a schema for this resource.
85     metadata_field = "{}:schema".format(resource_name)
86     bucket = buckets[bucket_uri]
87     if metadata_field not in bucket:
88         return
89 
90     # Validate or fail with 400.
91     schema = bucket[metadata_field]
92     try:
93         validate_schema(data, schema, ignore_fields=ignore_fields)
94     except ValidationError as e:
95         raise_invalid(request, name=e.field, description=e.message)
96
```

Path: `kinto/views/records.py`
Content:
```
1 from pyramid.security import Authenticated
2 from pyramid.settings import asbool
3 
4 from kinto.core import resource, utils
5 from kinto.core.errors import raise_invalid
6 from kinto.views import object_exists_or_404
7 from kinto.schema_validation import (validate_from_bucket_schema_or_400, validate_schema,
8                                      ValidationError)
9 
10 
11 _parent_path = '/buckets/{{bucket_id}}/collections/{{collection_id}}'
12 
13 
14 @resource.register(name='record',
15                    collection_path=_parent_path + '/records',
16                    record_path=_parent_path + '/records/{{id}}')
17 class Record(resource.ShareableResource):
18 
19     schema_field = 'schema'
20 
21     def __init__(self, request, **kwargs):
22         # Before all, first check that the parent collection exists.
23         # Check if already fetched before (in batch).
24         collections = request.bound_data.setdefault('collections', {})
25         collection_uri = self.get_parent_id(request)
26         if collection_uri not in collections:
27             # Unknown yet, fetch from storage.
28             bucket_uri = utils.instance_uri(request, 'bucket', id=self.bucket_id)
29             collection = object_exists_or_404(request,
30                                               collection_id='collection',
31                                               parent_id=bucket_uri,
32                                               object_id=self.collection_id)
33             collections[collection_uri] = collection
34         self._collection = collections[collection_uri]
35 
36         super().__init__(request, **kwargs)
37 
38     def get_parent_id(self, request):
39         self.bucket_id = request.matchdict['bucket_id']
40         self.collection_id = request.matchdict['collection_id']
41         return utils.instance_uri(request, 'collection',
42                                   bucket_id=self.bucket_id,
43                                   id=self.collection_id)
44 
45     def process_record(self, new, old=None):
46         """Validate records against collection or bucket schema, if any."""
47         new = super().process_record(new, old)
48 
49         # Is schema validation enabled?
50         settings = self.request.registry.settings
51         schema_validation = 'experimental_collection_schema_validation'
52         if not asbool(settings.get(schema_validation)):
53             return new
54 
55         # Remove internal and auto-assigned fields from schemas and record.
56         internal_fields = (self.model.id_field,
57                            self.model.modified_field,
58                            self.schema_field,
59                            self.model.permissions_field)
60 
61         # The schema defined on the collection will be validated first.
62         if 'schema' in self._collection:
63             schema = self._collection['schema']
64 
65             try:
66                 validate_schema(new, schema, ignore_fields=internal_fields)
67             except ValidationError as e:
68                 raise_invalid(self.request, name=e.field, description=e.message)
69 
70             # Assign the schema version to the record.
71             schema_timestamp = self._collection[self.model.modified_field]
72             new[self.schema_field] = schema_timestamp
73 
74         # Validate also from the record:schema field defined on the bucket.
75         validate_from_bucket_schema_or_400(new, resource_name="record", request=self.request,
76                                            ignore_fields=internal_fields)
77 
78         return new
79 
80     def collection_get(self):
81         result = super().collection_get()
82         self._handle_cache_expires(self.request.response)
83         return result
84 
85     def get(self):
86         result = super().get()
87         self._handle_cache_expires(self.request.response)
88         return result
89 
90     def _handle_cache_expires(self, response):
91         """If the parent collection defines a ``cache_expires`` attribute,
92         then cache-control response headers are sent.
93 
94         .. note::
95 
96             Those headers are also sent if the
97             ``kinto.record_cache_expires_seconds`` setting is defined.
98         """
99         is_anonymous = Authenticated not in self.request.effective_principals
100         if not is_anonymous:
101             return
102 
103         cache_expires = self._collection.get('cache_expires')
104         if cache_expires is None:
105             by_bucket = '{}_record_cache_expires_seconds'.format(self.bucket_id)
106             by_collection = '{}_{}_record_cache_expires_seconds'.format(
107                 self.bucket_id, self.collection_id)
108             settings = self.request.registry.settings
109             cache_expires = settings.get(by_collection,
110                                          settings.get(by_bucket))
111 
112         if cache_expires is not None:
113             response.cache_expires(seconds=int(cache_expires))
114
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/kinto/schema_validation.py b/kinto/schema_validation.py
--- a/kinto/schema_validation.py
+++ b/kinto/schema_validation.py
@@ -1,5 +1,5 @@
 import colander
-from jsonschema import Draft4Validator, ValidationError, SchemaError, validate
+from jsonschema import Draft4Validator, ValidationError, SchemaError, RefResolutionError, validate
 from pyramid.settings import asbool
 
 from kinto.core import utils
@@ -56,6 +56,13 @@
             field = e.schema_path[-1]
         e.field = field
         raise e
+    # Raise an error here if a reference in the schema doesn't resolve.
+    # jsonschema doesn't provide schema validation checking upon creation yet,
+    # it must be validated against data.
+    # See https://github.com/Julian/jsonschema/issues/399
+    # For future support https://github.com/Julian/jsonschema/issues/346.
+    except RefResolutionError as e:
+        raise e
 
 
 def validate_from_bucket_schema_or_400(data, resource_name, request, ignore_fields=[]):
@@ -93,3 +100,5 @@
         validate_schema(data, schema, ignore_fields=ignore_fields)
     except ValidationError as e:
         raise_invalid(request, name=e.field, description=e.message)
+    except RefResolutionError as e:
+        raise_invalid(request, name='schema', description=str(e))
diff --git a/kinto/views/records.py b/kinto/views/records.py
--- a/kinto/views/records.py
+++ b/kinto/views/records.py
@@ -5,7 +5,7 @@
 from kinto.core.errors import raise_invalid
 from kinto.views import object_exists_or_404
 from kinto.schema_validation import (validate_from_bucket_schema_or_400, validate_schema,
-                                     ValidationError)
+                                     ValidationError, RefResolutionError)
 
 
 _parent_path = '/buckets/{{bucket_id}}/collections/{{collection_id}}'
@@ -66,6 +66,8 @@
                 validate_schema(new, schema, ignore_fields=internal_fields)
             except ValidationError as e:
                 raise_invalid(self.request, name=e.field, description=e.message)
+            except RefResolutionError as e:
+                raise_invalid(self.request, name='schema', description=str(e))
 
             # Assign the schema version to the record.
             schema_timestamp = self._collection[self.model.modified_field]

{"golden_diff": "diff --git a/kinto/schema_validation.py b/kinto/schema_validation.py\n--- a/kinto/schema_validation.py\n+++ b/kinto/schema_validation.py\n@@ -1,5 +1,5 @@\n import colander\n-from jsonschema import Draft4Validator, ValidationError, SchemaError, validate\n+from jsonschema import Draft4Validator, ValidationError, SchemaError, RefResolutionError, validate\n from pyramid.settings import asbool\n \n from kinto.core import utils\n@@ -56,6 +56,13 @@\n             field = e.schema_path[-1]\n         e.field = field\n         raise e\n+    # Raise an error here if a reference in the schema doesn't resolve.\n+    # jsonschema doesn't provide schema validation checking upon creation yet,\n+    # it must be validated against data.\n+    # See https://github.com/Julian/jsonschema/issues/399\n+    # For future support https://github.com/Julian/jsonschema/issues/346.\n+    except RefResolutionError as e:\n+        raise e\n \n \n def validate_from_bucket_schema_or_400(data, resource_name, request, ignore_fields=[]):\n@@ -93,3 +100,5 @@\n         validate_schema(data, schema, ignore_fields=ignore_fields)\n     except ValidationError as e:\n         raise_invalid(request, name=e.field, description=e.message)\n+    except RefResolutionError as e:\n+        raise_invalid(request, name='schema', description=str(e))\ndiff --git a/kinto/views/records.py b/kinto/views/records.py\n--- a/kinto/views/records.py\n+++ b/kinto/views/records.py\n@@ -5,7 +5,7 @@\n from kinto.core.errors import raise_invalid\n from kinto.views import object_exists_or_404\n from kinto.schema_validation import (validate_from_bucket_schema_or_400, validate_schema,\n-                                     ValidationError)\n+                                     ValidationError, RefResolutionError)\n \n \n _parent_path = '/buckets/{{bucket_id}}/collections/{{collection_id}}'\n@@ -66,6 +66,8 @@\n                 validate_schema(new, schema, ignore_fields=internal_fields)\n             except ValidationError as e:\n                 raise_invalid(self.request, name=e.field, description=e.message)\n+            except RefResolutionError as e:\n+                raise_invalid(self.request, name='schema', description=str(e))\n \n             # Assign the schema version to the record.\n             schema_timestamp = self._collection[self.model.modified_field]\n", "issue": "Unhandled JSON pointer errors\nWhen setting a schema on resources, unresolvable references are not checked upon creation.\r\n\r\nWhen creating a resource that is checked against that schema, a `RefResolutionError` is raised by `jsonschema` and not caught.\r\n\r\nFor example, creating the schema:\r\n\r\n```json\r\n{\"data\": {\r\n    \"schema\": {\r\n      \"properties\": {\r\n        \"address\": { \"$ref\": \"#/definitions/address\" }\r\n      }\r\n    }\r\n  }\r\n}\r\n```\r\n\r\nPOST data:\r\n\r\n```json\r\n{ \"data\": {\r\n   \"address\": \"seattle\"\r\n  }\r\n}\r\n```\r\n\r\nThis will crash the server with a 500 error and the following message:\r\n\r\n```shell\r\n\"Unresolvable JSON pointer: %r\" % fragment\r\njsonschema.exceptions.RefResolutionError: Unresolvable JSON pointer: 'definitions/address' \r\n```\r\n\r\nI can work on this, I was just wondering what you all felt the approach should be.\r\n\r\nIt can either check when creating the schema or upon each invocation of it (i.e., POST data). The only slight trouble I can see with checking and not allowing unresolvable schema is that the schema can be a url, so if it is not resolving at the time of creation, it will fail.\r\n\r\nThat seems fair to me; does that seem like a reasonable fix?\r\n\r\n\nUnhandled JSON pointer errors\nWhen setting a schema on resources, unresolvable references are not checked upon creation.\r\n\r\nWhen creating a resource that is checked against that schema, a `RefResolutionError` is raised by `jsonschema` and not caught.\r\n\r\nFor example, creating the schema:\r\n\r\n```json\r\n{\"data\": {\r\n    \"schema\": {\r\n      \"properties\": {\r\n        \"address\": { \"$ref\": \"#/definitions/address\" }\r\n      }\r\n    }\r\n  }\r\n}\r\n```\r\n\r\nPOST data:\r\n\r\n```json\r\n{ \"data\": {\r\n   \"address\": \"seattle\"\r\n  }\r\n}\r\n```\r\n\r\nThis will crash the server with a 500 error and the following message:\r\n\r\n```shell\r\n\"Unresolvable JSON pointer: %r\" % fragment\r\njsonschema.exceptions.RefResolutionError: Unresolvable JSON pointer: 'definitions/address' \r\n```\r\n\r\nI can work on this, I was just wondering what you all felt the approach should be.\r\n\r\nIt can either check when creating the schema or upon each invocation of it (i.e., POST data). The only slight trouble I can see with checking and not allowing unresolvable schema is that the schema can be a url, so if it is not resolving at the time of creation, it will fail.\r\n\r\nThat seems fair to me; does that seem like a reasonable fix?\r\n\r\n\n", "before_files": [{"content": "import colander\nfrom jsonschema import Draft4Validator, ValidationError, SchemaError, validate\nfrom pyramid.settings import asbool\n\nfrom kinto.core import utils\nfrom kinto.core.errors import raise_invalid\nfrom kinto.views import object_exists_or_404\n\n\nclass JSONSchemaMapping(colander.SchemaNode):\n    def schema_type(self, **kw):\n        return colander.Mapping(unknown='preserve')\n\n    def deserialize(self, cstruct=colander.null):\n        # Start by deserializing a simple mapping.\n        validated = super().deserialize(cstruct)\n\n        # In case it is optional in parent schema.\n        if not validated or validated in (colander.null, colander.drop):\n            return validated\n        try:\n            check_schema(validated)\n        except ValidationError as e:\n            self.raise_invalid(e.message)\n        return validated\n\n\ndef check_schema(data):\n    try:\n        Draft4Validator.check_schema(data)\n    except SchemaError as e:\n        message = e.path.pop() + e.message\n        raise ValidationError(message)\n\n\ndef validate_schema(data, schema, ignore_fields=[]):\n    required_fields = [f for f in schema.get('required', []) if f not in ignore_fields]\n    # jsonschema doesn't accept 'required': [] yet.\n    # See https://github.com/Julian/jsonschema/issues/337.\n    # In the meantime, strip out 'required' if no other fields are required.\n    if required_fields:\n        schema = {**schema, 'required': required_fields}\n    else:\n        schema = {f: v for f, v in schema.items() if f != 'required'}\n\n    data = {f: v for f, v in data.items() if f not in ignore_fields}\n\n    try:\n        validate(data, schema)\n    except ValidationError as e:\n        if e.path:\n            field = e.path[-1]\n        elif e.validator_value:\n            field = e.validator_value[-1]\n        else:\n            field = e.schema_path[-1]\n        e.field = field\n        raise e\n\n\ndef validate_from_bucket_schema_or_400(data, resource_name, request, ignore_fields=[]):\n    \"\"\"Lookup in the parent objects if a schema was defined for this resource.\n\n    If the schema validation feature is enabled, if a schema is/are defined, and if the\n    data does not validate it/them, then it raises a 400 exception.\n    \"\"\"\n    settings = request.registry.settings\n    schema_validation = 'experimental_collection_schema_validation'\n    # If disabled from settings, do nothing.\n    if not asbool(settings.get(schema_validation)):\n        return\n\n    bucket_id = request.matchdict[\"bucket_id\"]\n    bucket_uri = utils.instance_uri(request, 'bucket', id=bucket_id)\n    buckets = request.bound_data.setdefault('buckets', {})\n    if bucket_uri not in buckets:\n        # Unknown yet, fetch from storage.\n        bucket = object_exists_or_404(request,\n                                      collection_id='bucket',\n                                      parent_id='',\n                                      object_id=bucket_id)\n        buckets[bucket_uri] = bucket\n\n    # Let's see if the bucket defines a schema for this resource.\n    metadata_field = \"{}:schema\".format(resource_name)\n    bucket = buckets[bucket_uri]\n    if metadata_field not in bucket:\n        return\n\n    # Validate or fail with 400.\n    schema = bucket[metadata_field]\n    try:\n        validate_schema(data, schema, ignore_fields=ignore_fields)\n    except ValidationError as e:\n        raise_invalid(request, name=e.field, description=e.message)\n", "path": "kinto/schema_validation.py"}, {"content": "from pyramid.security import Authenticated\nfrom pyramid.settings import asbool\n\nfrom kinto.core import resource, utils\nfrom kinto.core.errors import raise_invalid\nfrom kinto.views import object_exists_or_404\nfrom kinto.schema_validation import (validate_from_bucket_schema_or_400, validate_schema,\n                                     ValidationError)\n\n\n_parent_path = '/buckets/{{bucket_id}}/collections/{{collection_id}}'\n\n\n@resource.register(name='record',\n                   collection_path=_parent_path + '/records',\n                   record_path=_parent_path + '/records/{{id}}')\nclass Record(resource.ShareableResource):\n\n    schema_field = 'schema'\n\n    def __init__(self, request, **kwargs):\n        # Before all, first check that the parent collection exists.\n        # Check if already fetched before (in batch).\n        collections = request.bound_data.setdefault('collections', {})\n        collection_uri = self.get_parent_id(request)\n        if collection_uri not in collections:\n            # Unknown yet, fetch from storage.\n            bucket_uri = utils.instance_uri(request, 'bucket', id=self.bucket_id)\n            collection = object_exists_or_404(request,\n                                              collection_id='collection',\n                                              parent_id=bucket_uri,\n                                              object_id=self.collection_id)\n            collections[collection_uri] = collection\n        self._collection = collections[collection_uri]\n\n        super().__init__(request, **kwargs)\n\n    def get_parent_id(self, request):\n        self.bucket_id = request.matchdict['bucket_id']\n        self.collection_id = request.matchdict['collection_id']\n        return utils.instance_uri(request, 'collection',\n                                  bucket_id=self.bucket_id,\n                                  id=self.collection_id)\n\n    def process_record(self, new, old=None):\n        \"\"\"Validate records against collection or bucket schema, if any.\"\"\"\n        new = super().process_record(new, old)\n\n        # Is schema validation enabled?\n        settings = self.request.registry.settings\n        schema_validation = 'experimental_collection_schema_validation'\n        if not asbool(settings.get(schema_validation)):\n            return new\n\n        # Remove internal and auto-assigned fields from schemas and record.\n        internal_fields = (self.model.id_field,\n                           self.model.modified_field,\n                           self.schema_field,\n                           self.model.permissions_field)\n\n        # The schema defined on the collection will be validated first.\n        if 'schema' in self._collection:\n            schema = self._collection['schema']\n\n            try:\n                validate_schema(new, schema, ignore_fields=internal_fields)\n            except ValidationError as e:\n                raise_invalid(self.request, name=e.field, description=e.message)\n\n            # Assign the schema version to the record.\n            schema_timestamp = self._collection[self.model.modified_field]\n            new[self.schema_field] = schema_timestamp\n\n        # Validate also from the record:schema field defined on the bucket.\n        validate_from_bucket_schema_or_400(new, resource_name=\"record\", request=self.request,\n                                           ignore_fields=internal_fields)\n\n        return new\n\n    def collection_get(self):\n        result = super().collection_get()\n        self._handle_cache_expires(self.request.response)\n        return result\n\n    def get(self):\n        result = super().get()\n        self._handle_cache_expires(self.request.response)\n        return result\n\n    def _handle_cache_expires(self, response):\n        \"\"\"If the parent collection defines a ``cache_expires`` attribute,\n        then cache-control response headers are sent.\n\n        .. note::\n\n            Those headers are also sent if the\n            ``kinto.record_cache_expires_seconds`` setting is defined.\n        \"\"\"\n        is_anonymous = Authenticated not in self.request.effective_principals\n        if not is_anonymous:\n            return\n\n        cache_expires = self._collection.get('cache_expires')\n        if cache_expires is None:\n            by_bucket = '{}_record_cache_expires_seconds'.format(self.bucket_id)\n            by_collection = '{}_{}_record_cache_expires_seconds'.format(\n                self.bucket_id, self.collection_id)\n            settings = self.request.registry.settings\n            cache_expires = settings.get(by_collection,\n                                         settings.get(by_bucket))\n\n        if cache_expires is not None:\n            response.cache_expires(seconds=int(cache_expires))\n", "path": "kinto/views/records.py"}], "after_files": [{"content": "import colander\nfrom jsonschema import Draft4Validator, ValidationError, SchemaError, RefResolutionError, validate\nfrom pyramid.settings import asbool\n\nfrom kinto.core import utils\nfrom kinto.core.errors import raise_invalid\nfrom kinto.views import object_exists_or_404\n\n\nclass JSONSchemaMapping(colander.SchemaNode):\n    def schema_type(self, **kw):\n        return colander.Mapping(unknown='preserve')\n\n    def deserialize(self, cstruct=colander.null):\n        # Start by deserializing a simple mapping.\n        validated = super().deserialize(cstruct)\n\n        # In case it is optional in parent schema.\n        if not validated or validated in (colander.null, colander.drop):\n            return validated\n        try:\n            check_schema(validated)\n        except ValidationError as e:\n            self.raise_invalid(e.message)\n        return validated\n\n\ndef check_schema(data):\n    try:\n        Draft4Validator.check_schema(data)\n    except SchemaError as e:\n        message = e.path.pop() + e.message\n        raise ValidationError(message)\n\n\ndef validate_schema(data, schema, ignore_fields=[]):\n    required_fields = [f for f in schema.get('required', []) if f not in ignore_fields]\n    # jsonschema doesn't accept 'required': [] yet.\n    # See https://github.com/Julian/jsonschema/issues/337.\n    # In the meantime, strip out 'required' if no other fields are required.\n    if required_fields:\n        schema = {**schema, 'required': required_fields}\n    else:\n        schema = {f: v for f, v in schema.items() if f != 'required'}\n\n    data = {f: v for f, v in data.items() if f not in ignore_fields}\n\n    try:\n        validate(data, schema)\n    except ValidationError as e:\n        if e.path:\n            field = e.path[-1]\n        elif e.validator_value:\n            field = e.validator_value[-1]\n        else:\n            field = e.schema_path[-1]\n        e.field = field\n        raise e\n    # Raise an error here if a reference in the schema doesn't resolve.\n    # jsonschema doesn't provide schema validation checking upon creation yet,\n    # it must be validated against data.\n    # See https://github.com/Julian/jsonschema/issues/399\n    # For future support https://github.com/Julian/jsonschema/issues/346.\n    except RefResolutionError as e:\n        raise e\n\n\ndef validate_from_bucket_schema_or_400(data, resource_name, request, ignore_fields=[]):\n    \"\"\"Lookup in the parent objects if a schema was defined for this resource.\n\n    If the schema validation feature is enabled, if a schema is/are defined, and if the\n    data does not validate it/them, then it raises a 400 exception.\n    \"\"\"\n    settings = request.registry.settings\n    schema_validation = 'experimental_collection_schema_validation'\n    # If disabled from settings, do nothing.\n    if not asbool(settings.get(schema_validation)):\n        return\n\n    bucket_id = request.matchdict[\"bucket_id\"]\n    bucket_uri = utils.instance_uri(request, 'bucket', id=bucket_id)\n    buckets = request.bound_data.setdefault('buckets', {})\n    if bucket_uri not in buckets:\n        # Unknown yet, fetch from storage.\n        bucket = object_exists_or_404(request,\n                                      collection_id='bucket',\n                                      parent_id='',\n                                      object_id=bucket_id)\n        buckets[bucket_uri] = bucket\n\n    # Let's see if the bucket defines a schema for this resource.\n    metadata_field = \"{}:schema\".format(resource_name)\n    bucket = buckets[bucket_uri]\n    if metadata_field not in bucket:\n        return\n\n    # Validate or fail with 400.\n    schema = bucket[metadata_field]\n    try:\n        validate_schema(data, schema, ignore_fields=ignore_fields)\n    except ValidationError as e:\n        raise_invalid(request, name=e.field, description=e.message)\n    except RefResolutionError as e:\n        raise_invalid(request, name='schema', description=str(e))\n", "path": "kinto/schema_validation.py"}, {"content": "from pyramid.security import Authenticated\nfrom pyramid.settings import asbool\n\nfrom kinto.core import resource, utils\nfrom kinto.core.errors import raise_invalid\nfrom kinto.views import object_exists_or_404\nfrom kinto.schema_validation import (validate_from_bucket_schema_or_400, validate_schema,\n                                     ValidationError, RefResolutionError)\n\n\n_parent_path = '/buckets/{{bucket_id}}/collections/{{collection_id}}'\n\n\n@resource.register(name='record',\n                   collection_path=_parent_path + '/records',\n                   record_path=_parent_path + '/records/{{id}}')\nclass Record(resource.ShareableResource):\n\n    schema_field = 'schema'\n\n    def __init__(self, request, **kwargs):\n        # Before all, first check that the parent collection exists.\n        # Check if already fetched before (in batch).\n        collections = request.bound_data.setdefault('collections', {})\n        collection_uri = self.get_parent_id(request)\n        if collection_uri not in collections:\n            # Unknown yet, fetch from storage.\n            bucket_uri = utils.instance_uri(request, 'bucket', id=self.bucket_id)\n            collection = object_exists_or_404(request,\n                                              collection_id='collection',\n                                              parent_id=bucket_uri,\n                                              object_id=self.collection_id)\n            collections[collection_uri] = collection\n        self._collection = collections[collection_uri]\n\n        super().__init__(request, **kwargs)\n\n    def get_parent_id(self, request):\n        self.bucket_id = request.matchdict['bucket_id']\n        self.collection_id = request.matchdict['collection_id']\n        return utils.instance_uri(request, 'collection',\n                                  bucket_id=self.bucket_id,\n                                  id=self.collection_id)\n\n    def process_record(self, new, old=None):\n        \"\"\"Validate records against collection or bucket schema, if any.\"\"\"\n        new = super().process_record(new, old)\n\n        # Is schema validation enabled?\n        settings = self.request.registry.settings\n        schema_validation = 'experimental_collection_schema_validation'\n        if not asbool(settings.get(schema_validation)):\n            return new\n\n        # Remove internal and auto-assigned fields from schemas and record.\n        internal_fields = (self.model.id_field,\n                           self.model.modified_field,\n                           self.schema_field,\n                           self.model.permissions_field)\n\n        # The schema defined on the collection will be validated first.\n        if 'schema' in self._collection:\n            schema = self._collection['schema']\n\n            try:\n                validate_schema(new, schema, ignore_fields=internal_fields)\n            except ValidationError as e:\n                raise_invalid(self.request, name=e.field, description=e.message)\n            except RefResolutionError as e:\n                raise_invalid(self.request, name='schema', description=str(e))\n\n            # Assign the schema version to the record.\n            schema_timestamp = self._collection[self.model.modified_field]\n            new[self.schema_field] = schema_timestamp\n\n        # Validate also from the record:schema field defined on the bucket.\n        validate_from_bucket_schema_or_400(new, resource_name=\"record\", request=self.request,\n                                           ignore_fields=internal_fields)\n\n        return new\n\n    def collection_get(self):\n        result = super().collection_get()\n        self._handle_cache_expires(self.request.response)\n        return result\n\n    def get(self):\n        result = super().get()\n        self._handle_cache_expires(self.request.response)\n        return result\n\n    def _handle_cache_expires(self, response):\n        \"\"\"If the parent collection defines a ``cache_expires`` attribute,\n        then cache-control response headers are sent.\n\n        .. note::\n\n            Those headers are also sent if the\n            ``kinto.record_cache_expires_seconds`` setting is defined.\n        \"\"\"\n        is_anonymous = Authenticated not in self.request.effective_principals\n        if not is_anonymous:\n            return\n\n        cache_expires = self._collection.get('cache_expires')\n        if cache_expires is None:\n            by_bucket = '{}_record_cache_expires_seconds'.format(self.bucket_id)\n            by_collection = '{}_{}_record_cache_expires_seconds'.format(\n                self.bucket_id, self.collection_id)\n            settings = self.request.registry.settings\n            cache_expires = settings.get(by_collection,\n                                         settings.get(by_bucket))\n\n        if cache_expires is not None:\n            response.cache_expires(seconds=int(cache_expires))\n", "path": "kinto/views/records.py"}]}