De-Duping the Duplication in Services Featuring: OpenAPI and AJV
De-duping the Duplication in Services Featuring: Swagger/OpenAPI and AJV
Good design lends itself to being Easier To Change [ETC]. However, this principle of ETC tends to get ignored when it comes to API documentation and service validation. Here the tenants of Do Not Repeat Yourself [DRY] are often neglected leaving services with multiple files potentially spanning hundreds if not thousands of lines of code with copious amounts of duplication.Development in services with monolithic validation engines and swagger documents then become a form of tech-debt. As these engines and documents often live outside of the code surface area that is getting changed the probability of them becoming out of sync increases.So What’s The Fix?I propose a new design pattern for developing your swagger documentation, and then letting your OpenAPI specification drive your validation.With our mission statement above, let us make sure we are all on the same page with our tool-chain. The NodeJS and the JavaScript ecosystem being what it is, it is an important step to understand our end goals.Service Documentation: Swagger 3.0 -- OpenAPIService Validation Engine: AJVnode-modules: swagger-jsdoc, openapi-validator-middlewareNodeJS Framework: ExpressWhile I acknowledge other validations engines exist (JOI and express-validator to name a few) AJV lends itself to a simple JSON feed and one that people already have written OpenAPI wrappers for! As for NodeJS frameworks, I chose to use express because that is what I am more familiar with. There is no reason this wouldn’t work with koa as the package openapi-validator-middleware even supports koa!So How Exactly Are You Removing Duplication?Each of the above packages has a specific goal.With swagger-jsdoc we are going to adhere to the earlier statement of Easier To Change. We will co-locate our swagger definitions in the route files themselves. This will allow future developers to see specification living with the code, making it more obvious to them that when they change the code in the route, to change that specification.openapi-validator-middleware has the ability to consume a generated OpenAPI Swagger document and use that for the validation engine. This package is a wrapper around AJV that allows us to have minimal code-changes for a large duplication removal.So What's This Look Like?So let us start with the validation piece, and for that, we take a peek at the file app.js where we describe our express app.First things first then; let's import our moduleconst swaggerValidation = require(‘openapi-validator-middleware’);After it is imported, we simply need to point it at our Swagger doc to configure it.swaggerValidation.init('swagger.yml');With the validation engine configured with our swagger, we just need to enforce it in our route definitions as middleware.api.get('/simple', swaggerValidation.validate, getSimple)With those 3 lines of code, we have configured our validation engine, tweaked it to our swagger specification and it is now enforcing its rules against the /simple route. No longer do you have to maintain a separate file Joi/AJV file to maintain your service validations — cool huh?OK, but about the swagger file? Won’t that be monstrous now?The answer is yes; because your swagger file will now have to have all your validation logic in it, it will be huge — but it should have had that info already. So with that in mind, we will let our other package swagger-jsdoc worry about maintaining the swagger file. Our goal here is Easier To Change remember? So we will co-locate our swagger definitions with our route file logic. With the code and documentation living in a single place when developers make changes they will hopefully be more encouraged to keep everything in sync. Not to mention any requirement to change the validation requirements of parameters/request-bodies instantly get reflected in the swagger doc as well.So here’s our get-simple.js that we defined earlier/** * @openapi *  /v1/acme: *    get: *      description: a simple get route that returns the `foo` query param *      parameters: *        - in: query *          name: foo *          schema: *            type: string *            minimum: 3 *      responses: *        200: *          description: a object witth the echoed query param. *          content: *            type: object *            properties: *              foo: *                type: string *                minimum: 3 */const getSimple = (req, res) => {  const { foo } = req.query;return res.status(200).json({ foo });};module.exports = getSimple;Wait I Have Some Questions!Is that 20 lines of comments for a 4 line route file? And how come foo is duplicated I thought we were removing duplication?To answer those questions yes, you will have a pretty large chunk of documentation here. It is inevitable, as we need to have the shell of swagger here, but it should help serve new developers looking at that file to know what the expectations for both the requests and responses are.As for the duplication you saw, I’m getting to it! That was showing the duplication for ease. Using the features of YAML we can actually remove some of that duplication all the while compartmentalizing our definitions even more.OK — just get to it, how do you do it?Leveraging YAML anchors we can create variable-like atomic definitions of our fields. But first, let's scaffold out our service a bit more and make some files/directories.mkdir swaggertouch swagger/first-name.ymltouch swagger/last-name.ymltouch swagger/user-id.ymlThis swagger folder, as you can see, will contain all of our swagger component definitions. This will ensure our definitions remain consistent as they get used across the various routes while removing duplication as they can now all share a single-source of truth — this folder.The Files# swagger/first-name.ymlx-template:  firstName: &firstName    type: string    minimum: 1    maximum: 30    description: the first name of our acme user# swagger/last-name.ymlx-template:  lastName: &lastName    type: string    minimum: 1    maximum: 30    description: the last name of our acme user# swagger/user-id.ymlx-template:  userId: &userId    type: string    minimum: 4    maximum: 4    pattern: '[0-9]{4}'    description: the unique identifier of our acme userWith our swagger field-components created, let's spin up some new routes using our new fields!put-create.js/** * @openapi *  /v1/acme/create: *    put: *      description: creates a fake user of the acme service *      requestBody: *        content: *          application/json: *            schema: *              type: object *              required: *                - firstName *                - lastName *              properties: *                firstName: *firstName *                lastName: *lastName *      responses: *        200: *          description: a object with the echoed firstName, lastName and a random userId. *          content: *            type: object *            properties: *              firstName: *firstName *              lastName: *lastName *              userId: *userId */const putCreate = (req, res) => {  const { firstName, lastName } = req.body;  const userId = Math.floor(1000 + Math.random() * 9000);return res.status(200).json({ firstName, lastName, userId: `${userId}` });};module.exports = putCreate;Look at that, we’ve made a more complicated request/response object and our total line count for the comments has 3 more lines! On top of that, even if you had no experience in the file you could determine its use-case and request/response contract by simply reading the first comment. See the Easier To Change benefits yet? Hypothetically if you had the requirement to allow for 60 character last names, you can simply change the swagger file last-name.yml and you would get both the Swagger Document updated as well as a validation rule in place enforcing it!OK — I’m Sold, But How Do You Turn Those Comments Into A Swagger Doc?swagger-generator.mjsimport fs from 'fs';import swaggerJsdoc from 'swagger-jsdoc';import { dirname } from 'path';import { fileURLToPath } from 'url';import packageJson from './package.json';const __dirname = dirname(fileURLToPath(import.meta.url));const options = {  format: '.yml',  definition: {    openapi: '3.0.0',    info: {      title: packageJson.name,      version: packageJson.version,    },  },  apis: ['./src/routes/*.js', './swagger/**/**.yml'], // files containing annotations};const runtime = async () => {  try {    const openapiSpecification = await swaggerJsdoc(options);    fs.writeFileSync(`${__dirname}/swagger.yml`, openapiSpecification);  } catch (e) {    console.log('broke', e);  }};runtime();The above script is the magic that will generate the OpenAPI Specification and generate the swagger.yml that the validation engine will consume. To help enforce good practices, and because all developers (myself included) are bad at remembering things I personally leverage Husky to ensure this file is generated. This would done as a pre-commit hook that will run the above script followed by a git add swagger.yml command.But how could you enforce that?CI CI CI! Because we only have a pre-commit hook to generate our swagger.yml, there is a valid concern. After all, the only worse than no documentation is bad/out-of-date documentation.What if a developer commits with a -n or simply makes the commit from the web-uiWell let me start by saying that they are a monster (especially if they are committing with -n!). But to help enforce this, it should be a build step when creating/bundling your application. Right with the test cases, we can re-run the swaggerJsDoc command and compare its output directly against the swagger.yml output. Any differences and stop the execution.Examples/ReferencesRepo Showcasing this process:swagger-jsdocopenapi-validator-middleware