Replace RESTful APIs with JSON-Pure
Replace RESTful APIs with JSON-Pure

    Using JSON and WebSockets



How did we get here?
RESTful APIs, the big lie challenged the widely-held belief
that RESTful APIs are a good idea for modern web applications.
The inspiration to write the post came from two sources. First, we
were writing an new API for an SPA, so the topic was certainly at the top of
my mind.  Second, we are recruiting development team members, and the
candidates placed a surprising amount of importance on RESTful APIs.

And now the big sequel
The goal of the big lie therefore was to provide an objective,
easy to read, and well-ordered list of issues with RESTful APIs.
In other words, the goal was to clearly define the problem before
suggesting a solution.

This “big sequel” continues where the big lie ended and looks at some
best API practices when freed from the constraints of REST. However, before we do
let’s clear up a big misconception many readers had about the original post.

REST is good
REST is going to be around for a long time. It works very well for
content delivery and we have an excellent and expensive infrastructure built around it.
I’ve never disputed this.  The original post, for example, contains this sentence:


  REST is a great mechanism for many things such as content delivery, and it
has served us well for two decades.


But I had made a mistake in the original post: I
had titled it  REST, the big lie instead of RESTful APIs, the big lie.
I recognized the original title was too broad, but by the time I had fixed it
a few hours later the damage had been done: search engines, links,
and the social networks kept the original name.  And, as they say, you
can’t uncook an egg. This title misled many people to think the article was
condeming all of REST.  So let me be clear: REST is essential and good.
And RESTful APIs are much less so.

RESTful APIs, not so much
The big lie about RESTful APIs is the commonly held belief that
they are almost always the best choice for web application APIs.  In truth, they
are particularly ill-suited for modern web applications like SPAs.
Most of issues are detailed in the the big lie, and we are not going
to rehash them here.

Now, let’s get to the good stuff.

JSON-Pure best practices
RESTful APIs, the big lie concluded with a list of best-practices
that evolved over 8 years of developing non-RESTful APIs for commercial
Single Page web Applications.
JSON-Pure APIs - a name I made up simply for brevity and convenience -
follow these conventions.  There isn’t an official standard or an Internet Engineering Task Force project for JSON-Pure APIs.  At least not yet.

Below are my recommendations JSON-Pure APIs.  The primary goal is to provide the best possible user experience for modern SPAs. Certainly the practices shown below are nothing new or revolutionary. You will see echoes of SOAP, JSON RPC, JSON API, JSend, JSON LD and Hydra in the recommendations - and probably a few other dozen standards I somehow missed.


  Use WebSockets and JSON
  Use an existing vocabulary
  Use four message types
  Use asynchronous transactions
  Use a reliable request
  Separate the request from the transport
  Use a reliable response
  Make it easy to debug
  Make it portable


This list is just slightly expanded from the summary of the big lie.
Let’s look at each in detail, starting from the top.

Use Websockets and JSON
We recommend using the WebSockets transport with the JSON data exchange format.

WebSockets
The WebSockets transport begins as an HTTP/S request which is then upgraded by
the client and server to a WebSocket connection.  This means it generally passes
through firewalls as well as HTTP/S, yet it provides truly asynchronous, full-duplex,
persistent messaging channel between client and server with
just a few bytes
of overhead per message.  Compare this to HTTP/S, where each message can require
multiple kilobytes of overhead thanks to verbose headers and cookies.
As a result, WebSockets not only provides great new capabilities, it can
also be 4-10x faster.

Prior to WebSockets, real-time messaging between a browser and a server was expensive
and error-prone.  I spent some quality time with XMPP, BOSH, and Strophe.  Trust me, it’s
extremely painful in comparison.

Are WebSockets always required? Of course not. But the transport is easy to implement
on modern browsers and its capabilities are extremely compelling for SPAs.
APIs transported over WebSockets are decidedly not RESTful as there is only a
single send() method for transport, and there are no visible response codes in
web server logs.  And that’s just fine.

JSON
JSON is a compelling data exchange format for web applications because JavaScript
provide very fast, safe methods to read and write the data.  That means that
at least on the browser client, we will be using JSON.parse() and
JSON.stringify() to parse and send the data respectively.  These very same methods are available on the server if we use Node.js; if not, there are robust JSON libraries for all other major web server platforms.

JSON is also the native format of many minimal-structure databases
such as MongoDB and CouchDB.  Even when it’s not the native format, Node.js
interfaces to other databases such as PostgreSQL return query responses in
JSON.  This means
marshalling
data from a database to a client can have substantially lower overhead compared
ORMs such as Active Record (Ruby) or Hibernate (Java).

JSON does have its weaknesses compared other formats.  Most notably a JSON
document does not typically include a Document Type Declaration (DTD)
which is useful to ensure the message contents are valid. We have found that
that JSON-Schema addresses much of this deficit,
and is discussed further below.

Now let’s look at using a clean, familiar vocabulary.

Use an existing vocabulary
The best APIs use a vocabulary that is widely understood
with little nuance. This greatly reduces the cognitive overhead required
to read or write messages.

Action verbs
The CRUD
vocabulary is very widely understood. We use a superset for our actions verbs.
Here they are organized by message type:


  
    
      Request
      Response
      Confirmation
    
  
  
    
      create
      CREATED or CREATE_FAIL
      abort, done, retry
    
    
      retrieve
      RETRIEVED or RETRIEVE_FAIL
      abort, done, retry
    
    
      update
      UPDATED or UPDATE_FAIL
      abort, done, retry
    
    
      delete
      DELETED or DELETE_FAIL
      abort, done, retry
    
    
      flush
      FLUSHED or FLUSH_FAIL
      abort, done, retry
    
  


Server response verbs are directly correlated with requests and are in
UPPERCASE to differentiat While at first this may appear almost trite, it
actually is quite important and useful, as we shall see when we discuss
indirect responses below.

We added the flush verb so the client can tell the server when it has
removed data from its storage.  Thus when the server distributes notification
of changed records, it doesn’t have to send messages to clients that have
flushed the containing record or data set.

Application status messages
Place all application status messages in a in a log_table (Update
2018-12-29: this used to be called log_list but I have since adopted the
convention of using table to indicate a list of objects or list of lists).
The JSON below shows all available log levels.  I have adjusted this to match
the syslog standard.

{ log_table: [
    { level_int : 0, level_str : "emerg", /* detail */ },
    { level_int : 1, level_str : "alert", ... },
    { level_int : 2, level_str : "crit",  ... },
    { level_int : 3, level_str : "error", ... },
    { level_int : 4, level_str : "warn",  ... },
    { level_int : 5, level_str : "notice",... },
    { level_int : 6, level_str : "info",  ... },
    { level_int : 7, level_str : "debug", ... },
  ],
  /* ... */
}

/* detail: log_id, code_key, code_str, user_msg */


The level_int value is useful to implement log filtering, either for the
recipient, or during a request.

Here is an example of a full error message:

{ log_table :[
    { code_key  : "400",
      code_str  : "Bad request",
      level_int : 3,
      level_str : "error",
      log_id    : "1096",
      user_msg  : "action_str required"
    }
  ],
  /* ... */
}


Notice how we mapped a REST code to the error.  We do that for convenience and
familiarity.  However, we generally use the REST error codes as a subset of
our full vocabulary, since we can extend it as the application requires.

Use four message types
SPA APIs generally require four message types:


  Requests
  Direct responses
  Indirect responses
  Confirmations


Requests
Requests are messages sent from clients requesting data manipulation or
delivery.  An example request to change a specific record would have the
update action verb.

Direct responses
Direct responses are messages from the server to attached clients directly in
response to a request. A direct response to the update request in the
example would use the UPDATED action verb.

Indirect responses
Indirect responses are sent from the server to clients without a direct
corresponding request.  This happens when the server needs to notify a client
of a data change, such as an updated, deleted, or new record. An indirect
response for the example above would have an UPDATED action verb.

Confirmations
Confirmations are sent from a client to a server to confirm a response has
been received and processed. The client may indicate success with done,
cancel the transaction with abort, or re-request the message with retry.
Generally it is a good idea to avoid the complexities of supporting the idea
of “partial success”.

Use a standard message structure
API messages should have a standard, minimal structure that is consistent and
easy to read.

Message formats
Here is a template of suggested JSON message formats for the request,
response, and confirm phases of a transaction:

// request
{ action_str    : "retrieve",  /* action verb */
  data_type     : "person",    /* record data */
  log_table      : [ /* application messages   */ ],
  request_map   : { /* request parameters     */ },
  trans_map     : { /* transaction meta-data  */ }
}

// direct and indirect responses
{ action_str   : "RETRIEVED", /* action verb */
  data_type    : "person",    /* record data */
  log_table    : [ /* application messages   */ ],
  response_map : { /* returned data          */ },
  trans_map    : { /* transaction meta-data  */ }
}

// confirm
{ action_str  : "done",      /* action verb */
  data_type   : "person",    /* record data */
  log_table   : [ /* application messages   */ ],
  confirm_map : { /* confirmation details   */ },
  trans_map   : { /* transaction meta-data  */ }
}


We have added a suffix to JSON keys to indicate type.  This greatly helps
the API consumer comprehend the meaning of fields at little cost.
Suggested suffixes are detailed in Appendix A
of Single Page Web Applications and a cheat
sheet is freely available on GitHub.

Field meanings

  The action verb, action_str, is required. It is discussed above.
  The data_type indicator is application dependent.
  The application messages list, log_table, is optional. It is discussed
above.
  The request_map, response_map, and confirm_map are required for each
message type.
  The trans_map is required and includes transaction specific parameters.
It is discussed below.


Message aggregation
Multiple messages may be aggregated by placing them in a list, for example:

[ { /* message 1 */ },
  { /* message 2 */ },
  ....
]


Messages are processed in the order received. An API can get rather chatty
at times, so it can be a good idea to employ a strategy to aggregate messages
on both the client and the server. One approach is to employ a message queue
that is sent with a limited frequency.

Schema validation
We generally use a JSON schema to validate
messages.  Validation isn’t hard to implement, and it helps avoid silly and
gnarly errors that can occur with invalid messages.

Use asynchronous transactions
We suggest using request-response-confirm for API transactions instead of
the REST request-response convention.  First, let’s consider when
request-response is more appropriate.

Use request-response when appropriate
We want the classic request-response approach when the web browser is used
to display published content like this blog page. It is simple, clean,
and well-understood.  Any benefit of using a more complex transaction is
almost certainly outweighed by the costs.

Use request-response-confirm for APIs
SPAs, however, are better served by request-response-confirm transactions.
For example, if a client requests a resource, the server issues a response.
Once the client has received and processed the response, it will send a
confirm to the server.

The benefits of the request-response-confirm transaction include:


  
    The confirmation goes beyond the transport status – was the message
received? – and reports to the server the application status of the message
– was the message properly processed and consumed?  This is very important
for an SPA so the server may keep track of client data sets.
  
  
    The transaction is asynchronous. The client, for example, may take 20
seconds to consume a large data set; or perhaps the client may wait for
user approval before processing or denying a provided data set.
During this time, the server can continue to process other request while
while awaiting the confirmation.  Implementing such a delay over a typical
RESTful request-response cycle using something like BOSH can be difficult 
and expensive.
  


The request-response-confirm cycle is therefore often the best option
if we need to sync records between a rich client and server.

Use a reliable request
Don’t embed any application meaning into the transport request.
Instead, use the single, most reliable request method available.

On the transport layer, we are simply concerned about whether an API message is
successfully transferred or not.  Think of the transport layer as the “delivery
driver” of web traffic: He will make sure our new sweater gets delivered
(transport issue), but he’s not going to sit around for hours while we decide
if we like it or not (application issue) and want to send it back.

WebSockets provides only a single send() method for requests.
If we use HTTP for our API, it is wise to use only POST and ignore the other verbs.

Separate the request from the transport
A typical RESTful API call splits the application-level meaning of a message
between the message body and the headers.  We wish to instead leave headers
alone and place all application-level details in the message.  We do this
through the trans_map and using a simple end point.

The transaction map
The transaction map, or trans_map was introduced earlier and contains
transaction specific fields.  Here is an example:

trans_map  : {
  auth_key : "c76aa3577f8b5a60206f9d041c76034a3c1f",
  trans_id : "eb99ec08-7e90-400d-9585-62a1385ec158"
}


The trans_id should be generated by the originator and should be immutable
throughout the transaction.  The auth_key will vary by implementation and
client.  A transaction may require additional details not shown here such as
a server time stamp,  but this is a good start.

The end point
Use an API endpoint that fully describes the API source and version. If using
HTTP, for example, and endpoint might look like the following:

https://api.mycompany.com/3-1-a/


Better yet, put the API version (api_version) in the trans_map as that
make the API transport independent.

Send all directives through the action_str property.  Certainly, this
precludes any sort of caching that RESTful API fans will undoubtedly miss.
But the purpose of an an API is to communicate dynamic messages, not static
content.  Transfer of bulk data and caching can be done at many levels, and we
can leverage the good parts of HTTP and REST when we need to.  For example,
instead of delivering image data over our API, we can instead provide a URL to
a cache-able PNG image.

Use a reliable response
Don’t embed any application meaning into the transport response.  Instead, use
the single, most reliable response method available.  As with the request, we
are simply concerned about whether an API response is successfully transferred
or not.

A WebSockets transport doesn’t allow us to impart meaning to a response.
Either the message is received or it fails.  If we use HTTP, then we should we
use a 200 OK response whenever the transport succeeds, and leave any
application-level communication to the message content.

Make it easy to debug
Debugging JSON-Pure APIs is quite straight-forward.  First, we can check
if the message transport has succeeded.  Then we can proceed to inspect the
JSON for any and all application-level meaning.  A command-line JSON-schema
utility is useful for checking message validity during this process.

Make it portable
An application API should be easily portable over any transport method such as
HTTP/S, WebSockets, XMPP, or SMTP.  A JSON-Pure API should be transport neutral,
and should require only minor adjustment to work over nearly any transport.

Closing thoughts
The move away from RESTful APIs has already begun. Both FaceBook’s
Relay
and Netflix’s
Falcor
dispense with any pretense of being RESTful (Update 2018-12-29: And the
trend continues with GraphQL).

I expect that REST will continue to be used for the majority of web
traffic, especially content distribution.  However, the more sophisticated
APIs used by SPAs will move to more appropriate, non-RESTful design
patterns.  And that’s a good thing.

I hope you found this useful, and as always, any constructive feedback is
welcome!

Cheers, Mike


  