Rethinking CRUD for REST API Designs
Rethinking CRUD For REST API Designs - Palantir Blog
When I hear someone say “we’re about to create new service X,” it’s often followed by: “it’ll use a simple CRUD interface for the V0.” While this design approach appealed to me once upon a time, increasingly I’ve started to feel like it’s not the best design fit. In this post, I talk through some real-world issues I’ve run into using CRUD and how they might change our approach to good API design.As a refresher, the term CRUD stands for Create, Read, Update, Delete, and it attempts to describe the “four basic operations of persistent storage.” The term originates from early work with relational databases. It looks a lot like the INSERT, SELECT, UPDATE, and DELETE operations of standard SQL, and since it maps very closely with the HTTP verbs POST, GET, PUT, and DELETE, its use is also widespread in web and library API design. CRUD feels very intuitive for any resource-centric workflow — you would first “create” it, after which could you can “read” its current value, “update” it to a new one, or “delete” it.The ProblemWhile CRUD at first glance seems pretty easy to understand, implement, and use, I find it far less appealing in practice. To explain what I mean, let’s take a look at CRUD’s individual functions:Create means create-unless-exists: If the resource already exists, fail this operation. If two valid but different create operations arrive at once, at least one must fail.Read is just read: Get the most recent value or null if it doesn’t exist.Update means update-if-exists: Overwrite the current value with the provided new value wholesale. If the resource doesn’t exist, fail this operation. If two valid but different update operations arrive at once, accept both, but pick a “latest” one to serve all future reads.Delete means delete-if-exists: If the resource doesn’t exist, fail this operation. If two valid delete operations arrive at once, delete the resource, and allow both to pass.So where’s the problem? I see a few.First, many clients don’t need to differentiate between Create, Update, and Delete — they want an omnibus “set” operation. This is especially true when APIs identify resources with a randomly generated ID with extremely low collision chance, such as a UUID or a hash. I’ve run into APIs where almost every client call includes a code block like the one below, which seems silly after a certain point:if (newValue != null) {    try {        backend.insertRecord(id, newValue);    } catch (ResourceAlreadyExistsException e) {        backend.updateRecord(id, newValue);    }} else {    backend.deleteRecord(id);}Second, while some resources are atomic and cannot be split up, many other resources can be broken down into multiple sub-components or fields. Most clients will only want to update one field or part of a resource at a time, but CRUD requires them to read and write the entire resource to update a single field properly. This complicates client logic, introduces unnecessary performance overhead, and drastically exacerbates the concurrency issue I describe in my next step. Imagine if writing a single word in a document editor required re-sending the entire document back to the server!Third, many clients’ editing workflows are three-step transactions. They will run a read on the resource to get its current value, run some computation based on that value and the user’s input, and then write the result back in an update. Services are inherently multiple access, but the update command is just an overwrite with no conflict or synchronization semantics built-in. At any point, multiple users could be reading, computing, updating a resource simultaneously, and clobbering each other’s edits. This is commonly handled with separate locking or transaction start/end commands, but these are often difficult to get correct and performant. Additionally, because they’re not part of the CRUD definition, they’re often forgotten until after the first few bugs.Finally, in practice, and particularly for us at Palantir, CRUD omits two other operations that are vitally important to any initial REST API design: Search and Security-Change/Hard Delete. Both operations have a history of forcing drastic changes to a service backend if they are not considered at a project’s outset. For example:A search operation takes in some search criteria (e.g., last-modified-today, or within-50-km-of-point) and returns a page of resources that fit the criteria.The security-change and hard-delete operations take a set of resources and proactively either change their security or wipes it from persistent storage, and does so in a way that existing clients have now lost access to those resources.The Solution(s)Considered together, this suggests that CRUD is not the one-size-fits-all API it is often billed to be. We should take a hard look at a service’s workflows when we first design an API and think carefully about what form it should take. I propose a small set of alternative patterns for API designers to consider:Get-and-SetThis is the simplest solution and one, I would argue, API designers should always consider. It consists of two operations:Get: Same as CRUD’s Read. Return the most recent value for a resource, null if it doesn’t exist.Set: The omnibus create/update/delete, also known as last-write-wins. Set the resource to the provided value. If the provided value is null, delete it. If two differing set operations arrive concurrently, pick a “latest” to win. Optionally, also return its previous value.Most file-service APIs natively follow this pattern. For example, indexing workflows, such as indexing database objects into ElasticSearch, follow this model.Get-and-PatchFor resources that are divisible into multiple fields that can be updated separately, get-and-set can be easily expanded into a slightly different pattern. Get-and-Patch consists of two operations:Get: Same as Get from get-and-set.Patch: An omnibus create/update/delete operation, but one that allows you only to set some of the fields on a resource. Missing fields mean “keep the existing value.” Deleting a resource is done via a synthetic “deleted” field.We use this pattern extensively for our configuration service APIs. JSON Patches are an example of this as well.Timestamp-CheckedFor applications that follow the three-step read-compute-update pattern or otherwise require transactions or protection from concurrent edits, use logical clocks to allow for optimistic concurrency.In addition to each resource’s current value, store a timestamp of the most recent write to that resource. Then expose two operations:Read-Return-Timestamp: Similar to Get, but also return the timestamp of the most recent write.Write-Check-Timestamp: Similar to Set, but in addition to providing the new value, also provide the timestamp that the client first read the resource. If the resource’s service-side timestamp matches the provided timestamp, execute the write and return the new timestamp. If the timestamps are different, reject the write and return the service’s value so that the client may rerun its computation and resubmit an updated value.You can also think about this as a variant of Check-and-Set (CAS). Firebase’s firestore API also includes this pattern as a preconditions check.Like most optimistic concurrency patterns, this approach has limitations; if expected concurrency is too high or if clients have expensive computations, this could result in too much lost work in retries. In these cases, it may be best to pick a different pattern and add transactional APIs around it.Append-Only-LogFor applications with clearly defined edits, or applications where clients really shouldn’t be touching values directly, it may be worth doing away with the traditional update-with-new-value operation entirely.Same as timestamp-checked: Store both the resource’s current value and a timestamp for the most recent write. Then expose two (or three) operations:Read-Return-Timestamp: Same as timestamp-checked, return the resource’s current value and timestamp.Append-Return-Updated: Instead of providing the value to write to the service, the service defines a set of edit operations that can be applied to its resources. The clients then provide one of those operations to the service to run on the resource. The updated value is then returned to the client.Load-Since-Timestamp: Optional operation. Suppose clients are capable of computing the results of edit operations themselves. In that case, clients can provide the timestamp of their most recent read of a resource and then receive all edit operations that have been applied to the resource since that timestamp.Operational Transforms and CRDT-based applications are perhaps the most advanced applications of this pattern, which is used extensively in our collaboration apps. Simpler ones exist, such as a banking API that allows you to withdraw, deposit, and transfer money but not manually set your account balance.ConsiderationsTwo things to note about the above patterns: 1) I did not include the search or re-ACL/hard-delete operations for any of these, since they’re common to all approaches and largely unaffected by which pattern you pick; and 2) these patterns are not mutually exclusive — it is entirely possible to have an API with a create-unless-exists and update-check-timestamp operation, for example.None of this is to say that CRUD has no place in service API design — only that it should not be the pattern of first-resort. In particular, CRUD is useful when a client considers the creation or destruction of a resource meaningfully significant and different from editing that resource, or when the service needs to do significant validation or computation that is not necessary on edit.A few examples include situations where the primary key of a resource consists of real-world information, such as a social security number where resource key collisions are possible and dangerous, whether accidental or malicious. Another could be where create operations accept substantially more information than updates, such as creating an online food order vs. changing the driver’s tip.ConclusionWhen creating an initial API for a resource-centric service, consider:If you need separate operations for create, update, and delete, or if client workflows treat them the same.How multiple clients viewing and editing a resource at the same time will want to handle concurrent edits.If clients should directly set the entire range of properties of a resource, parts of the resource, or should have allowable edits defined for them.How clients will search for and handle security-related changes to resources initially rather than an afterthoughtDoing so will serve you and your service much better in the long run and be much less likely to end up as crud.AuthorDavid Xiao, Software Engineering Lead at Palantir