text: servant-util: Extending servant with Database Integration
  For backend applications,
  servant is the most frequenly used framework in Serokell.
  However, from time to time we find ourselves writing the same boilerplate that, in theory, allows generalization and extraction.
  The logic where this boilerplate appears may be not solely about webservers, but also about databases, and so the fact that
  servant does not help us here is justified.
  But still, it is convenient to have a library that covers some basic use cases of webserver-database integration.
  In this post, we are going to discuss features that would be nice to have and how our new
  servant-util library handles them.
  Overview of
  servant
  servant
  servant is the most popular
  Haskell framework for implementing applications around server-client architecture. It can provide REST API, raw HTML, etc.
  When writing our own server, we first write down the type-level description of the API. As an example, let’s consider a bookstore API:
  type API = "book" :> Capture "id" Int :> Get '[JSON] Book :<|> "book" :> "all" :> Get '[JSON] Book :<|> "book" :> ReqBody '[JSON] Book :> Post '[JSON] Result
  This declares three endpoints.
  Routing and any complex structure of our API are handled right in the types.
  Types of accepted data and the response should be provided, as well as the serialization method (
  JSON in our case), and that all is handled by
  servant automatically.
  The server handler will be only concerned about working with the simple Haskell values like
  Book.
  The framework also provides a few other integrations:
  - We can generate a client from the
  APItype. That will also handle serialization inside. The generated client works perfectly with GHCJS.
  So if the frontend is written in Haskell too (by using Reflex or some other framework) we can share the type of API between the frontend code and backend code to query the API, and
  servant will handle all the rest for you.
  - OpenAPI 2 and OpenAPI 3 documentation can also be generated from the API. Here we will be sure that the documentation remains up-to-date just because it is produced automatically.
  Server handlers for the mentioned
  API may look like:
  getBookById :: Int -> Handler Book getBooks :: Handler [Book] createBook :: Book -> Handler Result server :: Server API server = getBookById :<|> getBooks :<|> createBook
  Here, we can see that argument types and the result type for all the handlers are just plain Haskell datatypes.
  By default, handlers operate in
  Handler monad, but with
  servant, this is also customizable: we can also use
  mtl or any other approach.
  Other built-in
  servant features include:
  - GET parameters, response and request headers;
  - JSON serialization for requests and responses;
  - multiple possible results for a single endpoint (to include, for instance, error responses);
  - customization of error responses (for example, for serialization errors);
  - streaming.
  However, as the topic of this post suggests, there is also a lot of stuff missing:
  - database integration;
  - metrics (e.g. Prometeus);
  - semantic tracing and logging;
  - sorting and filtering for bulk requests;
  - pagination;
  - and others.
  So in one of our projects, we needed to implement our own thing that other companies usually don’t share, but we decided to share our work.
  And so the
  servant-util package was born.
  The rest of the post is a quick tour over this library, featuring some examples of what can it do.
  Introducing servant-util
  To showcase this, we will be working on a simple example: a web application, in whose UI we will display a table of some items, which can be goods in the shop, books in a library, films in a database, etc. The UI is pretty standard – there is a way to sort on some columns (indicated by triangles on the picture). Users also usually have a search field to filter items by name and pagination controls to traverse the entire list of items.
  To display that data, the UI must get it from somewhere, and in our case, it is provided by a Haskell backend written with
  servant.
  But the backend does not itself contain this data, so it goes to the database to obtain the data.
  This is a well-known structure of any web application.
  And the actual filtering, sorting, and pagination must happen somewhere.
  The easiest solution would be to tell your frontend colleague: well, just do it on your side. We will provide you with all the items from the database, and you will do any necessary processing.
  .png)
  Indeed, this is the most straightforward approach and also the most wrong one because we will transfer too much data between the backend and UI.
  A more proper solution would be to filter in the backend and send only a subset of data to UI.
  .png)
  However, fetching all the items from the database is still too much of a burden for the network between the backend and the database and the backend code itself.
  The most proper solution in 99% of all cases would be to filter, sort, and paginate using the built-in database features. This way, both UI and backend operate only with a subset of the data.
  .png)
  However, remember that it is actually the user who controls filtering and sorting to perform, and we need to propagate the user’s criteria for filtering and sorting to the database. And we need to communicate this data twice – first from the UI to the backend, and then from the backend to the database.
  The second part is easy – communicating filtering and sorting parameters to the database usually can be done via query commands like
  WHERE and
  ORDER BY.
  However, there is no standardized protocol to perform this communication between the UI and the backend. And, unfortunately, every web developer out there who is faced with this problem has to invent their own combinators, their own language of queries understandable by their server.
  We’ve searched for the most commonly used patterns in the world and built our library to support them.
  Features
  Sorting
  Suppose that our API has this
  Book type with 3 fields:
  data Book = Book { isbn :: Isbn , bookName :: Text , author :: Text , year :: Int }
  And a simple route that fetches all the books from the service:
  type GetBooks = "books" :> Get '[JSON] [Book]
  This route provides a
  GET method that returns a list of books serialized as
  JSON.
  And we would like to extend this route with get parameters to specify what to filter and sort on.
  What are our requirements?
  - Sort by
  isbn(ascending) by default.
  - Allow the user to sort on any field.
  The default sorting is necessary for a deterministic result. We don’t want to get the list in some random order produced by the database.
  For such purpose,
  servant-util provides a combinator called
  SortingParams. So we can extend our API like this:
  type GetBooks = "books" :> SortingParams '["isbn" ?: Isbn, "name" ?: Text, "author" ?: Text, "year" ?: Int] '["isbn" ?: 'Asc Isbn] :> Get '[JSON] [Book]
  This added
  SortingParams accepts two type-level lists:
  - Fields the user can sort on.
  - Fields that participate in base sorting, which will be applied lexicographically last in either case.
  We add this parameter at type level because we want it to be documented, affect generated clients, etc.
  Which queries our
  GetBooks route can now accept?
  Some examples:
  GET /books– just returns everything, no sorting except for the default
  isbnone will be applied.
  GET /books?sortBy=asc(name)– specifies sorting by name is ascending order. In case there are two books with the same name, they will be sorted by
  isbn.
  GET /books?sortBy=asc(name),desc(author)– sorts first by name in ascending order, then by author in descending order.
  GET /books?sortBy=+name,-author– an alternative syntax for the query above. You can pick the syntax depending on your aesthetics or on which characters can appear in field names.
  In our opinion, this API is enough for most use cases. Now let’s take a look at how to implement a handler for this API.
  SortingParams combinator makes the handler accept a respective
  SortingSpec argument that keeps what sorting the user asks us to perform.
  Inside this
  SortingSpec looks roughly like
  newtype SortingParams provided base = SortingParams [SortingItem] data SortingItem = SortingItem { siName :: Text -- ^ field name , siOrder :: SortingOrder -- ^ order to sort by } data SortingOrder = Descendant | Ascendant
  Later this specification can be used for:
  - applying sorting manually at the backend level;
  - translating the specification to SQL;
  - generating data (e.g. for a mock server).
  In our case, we want to use SQL queries. If we use
  beam-postgresql to write our queries, our handler will look like this:
  import Data.Beam.Query (all_, select, runSelectReturningList) import Servant.Util.Beam.Postgres (fieldSort, sortBy_) getBooks :: SortingSpec [...] [...] -> Handler [Book] getBooks sortingSpec = do runSelectReturningList . select $ sortBy_ sortingSpec sortingApp $ -- (A) all_ (books booksDB) where sortingApp Book{..} = -- (B) fieldSort @"isbn" isbn .*. fieldSort @"name" bookName .*. fieldSort @"author" author .*. HNil
  Here in
  (A), we use
  sortBy_ function provided by the adjacent
  servant-util-beam-pg package to add the necessary sorting.
  This will translate the user-provided sorting specification to the necessary
  ORDER BY SQL query.
  In
  (B), we define
  sortingApp that declares how sorting provided by the user must be applied to our database model.
  Here we associate the fields as they appear in our API with fields in our database.
  sortingApp here will have
  SortingApp BeamSortingBackend ["isbn" ?: Isbn, ...] type and will check that the association is correct.
  If some field name is wrong or types in API and in the database model mismatch, we will get a compile-time error.
  With this code, if the user provides us with
  sortBy=+name,-author parameter, we will submit the following SQL query to the database:
  SELECT * FROM books ORDER BY name ASC, author DESC
  So to recap:
  servant-utildefines query format for the GET sorting parameter.
  - This format is parsed to Sorting Spec.
  - Handler may use the spec directly or translate to SQL.
  - We provide
  servant-util-beam-pgwith Beam integration.
  - Reasonable OpenAPI documentation and client handlers can be generated automatically.
  In case you don’t want to use Beam, you can always declare your own database backend and provide the necessary typeclass instances to implement sorting.
  Filtering
  Having just sorting is always not enough for the UI in question, and we also have to support filtering.
  The users may want to:
  - filter by exact ISBN;
  - filter by a range of years;
  - filter by name substring;
  - filter by pattern.
  To support this,
  servant-util provides the dedicated
  FilteringParams combinator, so we can write:
  type GetBooks = FilteringParams [ "isbn" ?: 'AutoFilter Isbn , "name" ?: 'AutoFilter Text , "year" ?: 'AutoFilter Int ] :> Get '[JSON] [Books]
  In
  FilteringParams, we list field names that we allow to filter by and the corresponding types.
  'AutoFilter designates that the user is allowed to provide multiple filtering operations for the given field, and they will be automatically supported.
  The exact list of allowed filtering operations is specified on a per-type basis.
  For instance, if we want to allow exact matches and comparison operations for
  Isbn, we will write:
  type instance SupportedFilters Isbn = '[FilterMatching, FilterComparing]
  Now let’s explore what kind of filters we can create.
  FilterMatching
  The most basic filter that is usually provided.
  GET /books?isbn=12345– gets a book with the given ISBN.
  GET /books?isbn[neq]=12345– gets all books except for the one with given ISBN.
  GET /books?isbn[in]=[123,456,789]– gets books with the provided ISBN values.
  FilterComparing
  Numerical fields usually support comparing operations.
  GET /books?year[gt]=1990– gets books after
  1990.
  GET /books?year[gte]=1994&year[lte]=2007– gets books from the
  [1994..2007]range.
  FilterLike
  Textual filters.
  GET /books?author[like]=Alexander*– does a simple regex-like match on the author.
  GET /books?author[contains]=sweet– does a simple substring match.
  ManualFilter
  As opposed to
  'AutoFilter,
  'ManualFilter provides no automatic logic. Instead, it allows the user to pass a value and the backend to supply an arbitrary matching logic for it.
  Of course, it is possible to define custom filters.
  Note that the list of supported filters for basic types like
  Text or
  Int is hardcoded by the library and cannot be extended to include custom filter operations.
  But in the case of
  servant, it is a good practice to define custom datatypes and newtype wrappers for all types (at least for the sake of neater generated documentation). In such cases, any necessary filters can be defined for the API types.
  Like with sorting, for filtering we also provide an integration with the
  beam package.
  We provide a combinator called
  matches_ (A) that is compatible with
  beam’s standard combinator
  guard_.
  import Database.Beam.Query (all_, guard_, select, runSelectReturningList) import Servant.Util.Beam.Postgres (matches_) getBooks :: FilteringSpec [...] -> Handler [Book] getBooks filterSpec = do runSelectReturningList . select $ guard_ (matches_ filterSpec filterApp) $ -- (A) all_ (books booksDB) where filterApp Book{..} = -- (B) filterOn @"isbn" isbn .*. filterOn @"name" bookName .*. HNil
  Here we supply the
  filterSpec that we get from the user and construct a
  filterApp (B) that describes how to apply our filter to the concrete datatype, which is stored in the database.
  Pagination
  The last important part is pagination.
  For it, we provide a simple combinator called
  PaginationParams.
  type GetBooks = PaginationParams (DefPageSize 20) :> Get '[JSON] [Book]
  It contains the default page size – if the page size is not specified by the user, we display only
  20 items at once.
  The handler will accept – as one can guess –
  PaginationSpec argument, that internally looks just like:
  data PaginationSpec = PaginationSpec { psOffset :: Natural , psLimit :: Maybe (Positive Natural) }
  So it represents well-known offset-limit pagination, which has its own drawbacks but in basic cases can be used safely.
  As expected, with
  PaginationParams the server API starts accepting
  offset and
  limit query parameters, for instance:
  GET /books?offset=40&limit=20.
  Client generation
  One can also generate client handlers to construct queries to a server with books API.
  For this you use the standard
  client function from
  servant-client:
  getBooks :: ClientM GetBooks getBooks = client (Proxy @GetBooks) -- getBooks -- :: SortingSpec -- ["isbn" ?: Isbn, "name" ?: Text, "author" ?: Text] -- '["isbn" ?: Isbn] -- -> FilteringSpec -- [ "isbn" ?: 'AutoFilter Isbn, "year" ?: 'AutoFilter Int, -- , "name" ?: 'AutoFilter Text, "author" ?: 'AutoFilter Text -- ] -- -> PaginationSpec -- -> ClientM [Book]
  So by the API type, we obtain a
  getBooks function to perform a query in the monad
  ClientM.
  This
  ClientM is standard servant client stuff. It basically carries the coordinates of the server and some querying options.
  SortingSpec and other specifications can be constructed with the dedicated smart constructors. e.g. we can write:
  foo :: ClientM Int foo = do books <- getBooks sortingSpec filterSpec paginationSpec return books where sortingSpec = mkSortingSpec [asc #name, desc #author] filterSpec = mkFilteringSpec [#year ?/>= 1994, #year ?/<= 2007] paginationSpec = skipping 30 $ itemsOnPage 10
  Here
  mkSortingSpec and
  mkFilteringSpec are just aliases for
  fromList.
  When writing many scenarios, the best experience is achieved with
  OverloadedLists enabled in the module.
  We use the
  asc and
  desc combinators to specify sorting order and
  ?/-like operators to construct filters.
  Hash sign notation, like
  #year, means that the names are lifted to type-level for further type-level checks.
  Any mistake in a field name, as well as in a filter operation, will cause a compilation error.
  Types of values supplied to filters are inferred automatically, so we don’t need to manually specify the types like
  1994 :: Int.
  Swagger
  The library also supports the generation of OpenAPI 2 documentation, aka Swagger, and the generation of OpenAPI 3 is in progress.
  The documentation contains the full description of how sorting works, so you won’t need to provide any extra code for that. You just supply the servant combinator, and your fellow frontend developer will get all the information they need from the OpenAPI specification.
  .png)
  Recap
  Let’s recap what we have been seen in this post.
  servant-utilprovides sorting, filtering and pagination combinators (and the library is not limited to them);
  - integration with PostgreSQL via Beam out of the box, and can be extended to support other backends via adding few typeclass instances;
  - extensibility with type-classes: new backends, new filters, and so on;
  servant-clientis supported;
  - generation of Swagger spec;
  - dummy backend for prototyping w/o DB.
  This post is made after a talk Maxim Koltsov made at ZuriHac 2021. You can find the original talk on our YouTube channel.
  You can find the library on GitHub.
  Questions asked at the talk
  Q: Are other DB libraries supported? If not, is it easy to add our own?
  A: Currently, we only support
  beam. It is our SQL library of choice.
  But of course,
  servant-util does not depend on
  beam; integration with it is a separate package.
  So if you want to use some other library –
  opaleye
  persistent, or even MongoDB (you don’t have to use SQL) – it’s quite easy to add your own support because we have a typeclass for sorting and filtering backends. You only need to write one instance to translate your query to some query language, and they also play nicely together.
  We would be glad to provide any guidance if you want to implement your own DB integration and get stuck.
  Q: Isn’t it mostly the same as what GraphQL achieves?
  A: Yeah, this is quite an expected one.
  Indeed, with GraphQL we could get filtering, sorting and pagination for free, but GraphQL is not an extension of REST API – it’s a completely different world. It would be a great commitment to switch to GraphQL, and such a decision must be taken by the whole team at the project architecture planning stage. It should be supported by your frontend, it should be supported by your backend, and it should be supported by the database (AFAIB recently PostgreSQL started doing some GraphQL natively). For those who can’t afford to switch to GraphQL at the moment but still want to get some well-defined stuff for this task, we provide this library to achieve some of the goals with REST API.
  Q: What is the right package to use for generating documentation?
  A: Servant OpenAPI 3 is available from the fall of 2020. It has the same API as
  servant-swagger and mostly it is just a drop-in replacement.
  You just have to rename some imports, and if you are using
  servant-swagger, we encourage you to use
  servant-openapi3 because OpenAPI 3 spec is more mature than Swagger 2: it doesn’t have such irregularities or oddities that Swagger 2 has.
  In OpenAPI 3, one can describe
  oneof types. Actually, OpenAPI 3 started supporting sum types which is a big step forward from what Swagger 2 was because you couldn’t possibly describe some cases of types in Swagger 2.
  .jpg)
  .jpg)
  .jpg)
  