text:
Archive for category DevOps
Effective DevOps Technical Analysis
Posted by Gary A. Stafford in DevOps, Software Development, Technology Consulting on December 26, 2016
Building a successful DevOps team is a challenge. Building a successful Agile DevOps team is an enormous challenge. Many Software Developers have experience working on Agile teams, often starting in college. On the contrary, most Operations Engineers and System Administrators have not participated on an Agile team. They frequently cut their teeth in highly reactive, support ticket-driven environments. Agile methodologies can often seem foreign and uncomfortable to them.
Adding to the challenge, most DevOps teams are expected to execute on a broad range of requirements. DevOps user stories can span the entire software development, delivery, and support lifecycle. Stories often include logging, monitoring, alerting, infrastructure provisioning, networking, security, continuous integration and deployment, and support.
Worst of all, many teams don't apply the same rigor to developing high-quality user stories for DevOps-related requirements as they do for software requirements. On teams where DevOps Engineers are integrated with Software Developers, I find this is often due to a lack of understanding of DevOps requirements. On stand-alone DevOps teams, I find DevOps Engineers often believe their stories don't need the same attention to details as their software development peers.
Successful Teams
I've been lucky enough to be part of multiple DevOps teams, both integrated and stand-alone. Those teams were a mix of engineers from both software development, system administration, and operations backgrounds. One of the reasons these teams were effective, was the presence of a Technical Analyst. The Technical Analyst role falls somewhere in-between an Agile Business Analyst and a traditional Systems Analyst; probably closer to the latter, on a DevOps team. The Technical Analyst ensures that user stories are Ready for Development, they ensure the stories meet the Definition of Ready.
There is a common perception that the Technical Analyst on an Agile team must be an experienced technologist. DevOps is a very broad and quickly evolving discipline. While a good understanding of Agile practices and software development are beneficial, I've found organization and investigation skills to be most advantageous to the Technical Analyst.
In this post, we will examine one particular methodology for optimizing the effectiveness of the Technical Analyst, through the use of a consistent approach to functional requirement analysis and story development.
Inadequate Requirements
Let's start with a typical greenfield requirement frequently assigned to DevOps teams, centralized logging.
Centralized Application Logging
User Story
As a Software Developer,
I need to view application logs,
So that I can monitor and troubleshoot running applications.
Acceptance Criterion 1
I can view the latest logs from multiple applications.
Acceptance Criterion 2
I can view application logs without having to log directly into each host on which the applications are running.
Acceptance Criterion 3
I can view the log entries in exact chronological order, based on timestamps.
I know some DevOps engineers that would gladly pick up this user story, make a handful of assumptions based on previous experiences, and manage to hack out a usable centralized application logging system. The ability to meet the end user's expectations would be entirely based on the experience (and luck) of the engineer, certainly not the vague and incomplete user story and acceptance criteria, and lack of any technical background details.
Multi-Faceted Analysis
Consistently delivering well-defined user stories and comprehensive acceptance criteria, which provide maximum business value with minimal resource utilization, requires a systematic analytical approach.
Based on my experience, I have found certain facets of the software development, delivery, and support lifecycles, affect or are affected by DevOps-related functional requirements. Every requirement should be uniformly analyzed against those facets to improve the overall quality of user stories and acceptance criteria. In no particular order, these facets include:
- Logging
- Monitoring
- Alerting
- Ownership
- Support
- Maintenance
- Backup and Recovery
- Security and Compliance
- Continuous Integration and Delivery
- Service Level Agreements (SLAs)
- Configuration Management
- Environment Management
- High Availability and Fault Tolerance
- Release Management
- Life Expectancy
- Architectural Approval
- Technology Preferences
- Infrastructure
- Data Management
- Training and Documentation
- Computer Networking
- Licensing and Legal
- Cost
- Technical Debt
- Other
For simplicity, I prefer to organize these facets into five general categories: Technology, Infrastructure, Delivery, Support, and Organizational.
Let's take a closer look at each facet, and how they could impact DevOps requirements.
Do the requirements introduce changes, which would or should produce logs? How should those logs be handled? Do those changes impact existing logging processes and systems?
Do the requirements introduce changes, which would or should be monitored? Do those changes impact existing monitoring processes and systems?
Do the requirements introduce changes, which would or should produce alerts? Do the requirements introduce changes, which impact existing alerting processes and systems?
Do the requirements introduce changes, which require an owner? Do those changes impact existing ownership models? An example of this might be regarding who owns the support and maintenance of new infrastructure.
Do the requirements introduce changes, which would or should require internal, external, or vendor support? Do those changes impact existing support models? For example, adding new service categories to a support ticketing system, like ServiceNow.
Do the requirements introduce changes, which would or should require regular and on-going maintenance? Do those changes impact existing maintenance processes and schedules? An example of maintenance might be regarding how upgrading and patching would be completed for a new monitoring system.
Backup and Recovery
Do the requirements introduce changes, which would or should require backup and recovery? Do the requirements introduce changes, which impact existing backup and recovery processes and systems?
Security and Compliance
Do the requirements introduce changes, which have security and compliance implications? Do those changes impact existing security and compliance requirements, processes, and systems? Compliance in IT frequently includes Regulatory Compliance with HIPAA, Sarbanes-Oxley, and PCI-DSS. The Security and Compliance facet is closely associated with Licensing and Legal.
Continuous Integration and Delivery
Do the requirements introduce changes, which require continuous integration and delivery of application or infrastructure code? Do those changes impact existing continuous integration and delivery processes and systems?
Service Level Agreements (SLAs)
Do the requirements introduce changes, which are or should be covered by Service Level Agreements (SLA)? Do the requirements introduce changes, which impact existing SLAs Often, these requirements are system performance- or support-related, centered around Mean Time Between Failures (MTBF), Mean Time to Repair, and Mean Time to Recovery (MTTR).
Configuration Management
Do the requirements introduce changes, which requires the management of new configuration or multiple configurations? Do those changes affect existing configuration or configuration management processes and systems? Configuration management also includes feature management (aka feature switches). Will the changes turned on or turned off for testing and release? Configuration Management is closely associated with Environment Managment and Release Management.
Environment Management
Do the requirements introduce changes, which affect existing software environments or require new software environments? Do those changes affect existing environments? Software environments typically include Development, Test, Performance, Staging, and Production.
High Availability and Fault Tolerance
Do the requirements introduce changes, which require high availability and fault tolerant configurations? Do those changes affect existing highly available and fault-tolerant systems?
Release Management
Do the requirements introduce changes, which require releases to the Production or other client-facing environments? Do those changes impact existing release management processes and systems?
Life Expectancy
Do the requirements introduce changes, which have a defined lifespan? For example, will the changes be part a temporary fix, a POC, an MVP, pilot, or a permanent solution? Do those changes impact the lifecycles of existing processes and systems? For example, do the changes replace an existing system?
Architectural Approval
Do the requirements introduce changes, which require the approval of the Client, Architectural Review Board (ARB), Change Approval Board (CAB), or other governing bodies? Do the requirements introduce changes, which impact other existing or pending technology choices? Architectural Approval is closely related to Technology Preferences, below. Often an ARB or similar governing body will regulate technology choices.
Technology Preferences
Do the requirements introduce changes, which might be impacted by existing preferred vendor relationships or previously approved technologies? Are the requirements impacted by blocked technology, including countries or companies blacklisted by the Government, such as the Specially Designated Nationals List (SDN) or Consolidated Sanctions List (CSL)? Does the client approve of the use of Open Source technologies? What is the impact of those technology choices on other facets, such as Licensing and Legal, Cost, and Support? Do those changes impact other existing or pending technology choices?
Do the requirements introduce changes, which would or should impact infrastructure? Is the infrastructure new or existing, is it physical, is it virtualized, is it in the cloud? If the infrastructure is virtualized, are there coding requirements about how it is provisioned? If the infrastructure is physical, how will it impact other facets, such as Technology Preferences, Cost, Support, Monitoring, and so forth? Do the requirements introduce changes, which impact existing infrastructure?
Data Management
Do the requirements introduce changes, which produce data or require the preservation of state? Do those changes impact existing data management processes and systems? Is the data ephemeral or transient? How is the data persisted? Is the data stored in a SQL database, NoSQL database, distributed key-value store, or flat-file? Consider how other aspects, such as Security and Compliance, Technology Preferences, and Architectural Approval, impact or are impacted by the various aspects of Data?
Training and Documentation
Do the requirements introduce changes, which require new training and documentation? This also includes architectural diagrams and other visuals. Do the requirements introduce changes, which impact training and documentation processes and practices?
Computer Networking
Do the requirements introduce changes, which require networking additions or modifications? Do those changes impact existing networking processes and systems? I have separated out Computer Networking from Infrastructure since networking is itself, so broad, involving such things as DNS, SDN, VPN, SSL, certificates, load balancing, proxies, firewalls, and so forth. Again, how do networking choices impact other aspects, like Security and Compliance?
Licensing and Legal
Do the requirements introduce changes, which requires licensing? Should or would the requirements require additional legal review? Are there contractual agreements that need authorization? Do the requirements introduce changes, which impact licensing or other legal agreements or contracts?
Do the requirements introduce changes, which requires the purchase of equipment, contracts, outside resources, or other goods which incur a cost? How and where will the funding come from to purchase those items? Are there licensing or other legal agreements and contracts required as part of the purchase? Who approves these purchases? Are they within acceptable budgetary expectations? Do those changes impact existing costs or budgets?
Technical Debt
Do the requirements introduce changes, which incur technical debt? What is the cost of that debt? How will impact other requirements and timelines? Do the requirements eliminate existing technical debt? Do those changes impact existing or future technical debt caused by other processes and systems?
Depending on your organization, industry, and DevOps maturity, you may prefer different aspects critical to your requirements.
Closer Analysis
Let's take a second look at our logging story, applying what we've learned. For each facet, we will ask the following three questions.
Is there any facet of _______ that impacts the logging requirement?
Will the logging requirement impact any facet of _______?
If yes, will the impacts be captured as part of this user story, or elsewhere?
For the sake of brevity, we will only look at the half the facets.
Although the requirement is about centralized logging, there is still analysis that should occur around Logging. For example, we should determine if there is a requirement that the centralized logging application should consume its own logs for easier supportability. If so, we should add an acceptance criterion to the original user story around the logging application's logs being captured.
If we are building a centralized application logging system, we will assume there will be a centralized logging application, possibly a database, and infrastructure on which the application is running. Therefore, it is reasonable to also assume that the logging application, database, and infrastructure need to be monitored. This would require interfacing with an existing monitoring system or creating a new monitoring system. If and how the system will be monitored needs to be clarified.
Interfacing with a current monitoring system will minimally require additional acceptance criteria. Developing a new monitoring system would undoubtedly require additional user story or stories, with their own acceptance criteria.
If we are monitoring the system, then we could assume that there should be a requirement to alert someone if monitoring finds the centralized application logging system becomes impaired. Similar to monitoring, this would require interfacing with an existing alerting system or creating a new alerting system. A bigger question, who do the alerts go to? These details should be clarified first.
Interfacing with an existing alerting system will minimally require additional acceptance criteria. Developing a new alerting system would require additional user story or stories, with their own acceptance criteria.
Who will own the new centralized application logging system? Ownership impacts many facets, such as Support, Maintenance, and Alerting. Ownership should be clearly defined, before the requirement's stories are played. Don't build something that does not have an owner, it will fail as soon as it is launched.
No additional user stories or acceptance criteria should be required.
We will assume that there are no Support requirements, other than what is already covered in Monitoring, Ownership, and Training and Documentation.
Similarly, we will assume that there are no Maintenance requirements, other than what is already covered in Ownership and Training and Documentation. Requirements regarding automating system maintenance in any way should be an additional user story or stories, with their own acceptance criteria.
Backup and Recovery
Similar to both Support and Maintenance, Backup and Recovery should have a clear owner. Regardless of the owner, integrating backup into the centralized application logging system should be addressed as part of the requirement. To create a critical support system such as logging, with no means of backup or recovery of the system or the data, is unwise.
Backup and Recovery should be an additional user story or stories, with their own acceptance criteria.
Security and Compliance
Since we are logging application activity, we should assume there is the potential of personally identifiable information (PII) and other sensitive business data being captured. Due to the security and compliance risks associated with improper handling of PII, there should be clear details in the logging requirement regarding such things as system access.
At a minimum, there are usually requirements for a centralized application logging system to implement some form of user, role, application, and environment-level authorization for viewing logs. Security and Compliance should be an additional user story or stories, with their own acceptance criteria. Any requirements around the obfuscation of sensitive data in the log entries would also be an entirely separate set of business requirements.
Continuous Integration and Delivery
We will assume that there are no Continuous Integration and Delivery requirements around logging. Any requirements to push the logs of existing Continuous Integration and Delivery systems to the new centralized application logging system would be a separate use story.
Service Level Agreements (SLAs)
A critical aspect of most DevOps user stories, which is commonly overlooked, Service Level Agreements. Most requirements should at least a minimal set of performance-related metrics associated with them. Defining SLAs also help to refine the scope of a requirement. For example, what is the anticipated average volume of log entries to be aggregated and over what time period? From how many sources will logs be aggregated? What is the anticipated average size of each log entry, 2 KB or 2 MB? What is the anticipated average volume of concurrent of system users?
Given these base assumptions, how quickly are new log entries expected to be available in the new centralized application logging system? What is the expected overall response time of the system under average load?
Directly applicable SLAs should be documented as acceptance criteria in the original user story, and the remaining SLAs assigned to the other new user stories.
Configuration Management
We will also assume that there no Configuration Management requirements around the new centralized application logging system. However, if the requirements prescribed standing up separately logging implementations, possibly to support multiple environments or applications, then Configuration Management would certainly be in scope.
Environment Management
Based on further clarification, we will assume we are only creating a single centralized application logging system instance. However, we still need to learn which application environments need to be captured by that system. Is this application only intended to support the Development and Test environments? Are the higher environments of Staging and Production precluded due to PII?
We should add an explicit acceptance criterion to the original user story around which application environments should be included and which should not.
High Availability and Fault Tolerance
The logging requirement should state whether or not this system is considered business critical. Usually, if a system is considered critical to the business, then we should assume the system requires monitoring, alerting, support, and backup. Additionally, the system should be highly available and fault tolerant.
Making the system highly available would surely have an impact on facets like Infrastructure, Monitoring, and Alerting. High Availability and Fault Tolerance could be a separate user story or stories, with its own acceptance criteria, as long as the original requirement doesn't dictate those features.
Other Aspects
We skipped half the facets. They would certainly expand our story list and acceptance criteria. We didn't address how the end user would view the logs. Should we assume a web browser? Which web browsers? We didn't determine if the system should be accessible outside the internal network. We didn't determine how the system would be configured and deployed. We didn't investigate if there are any vendor or licensing restrictions, or if there are any required training and documentation needs. Possibly, we will need Architectural Approval before we start to develop.
Analysis Results
Based on our limited analysis, we uncovered some further needs:
Further Clarification
We need additional information and clarification regarding Logging, Monitoring, Alerting, and Security and Compliance. We should also determine ownership of the resulting centralized application logging system, before it is built, to ensure Alerting, Maintenance, and Support, are covered.
Additional Acceptance Criteria
We need to add additional acceptance criteria to the original user story around SLAs, Security and Compliance, Logging, Monitoring, and Environment Management.I prefer to write acceptance criteria in the
I prefer to write acceptance criteria in the Given-When-Then format. I find the 'Given-When-Then' format tends to drive a greater level of story refinement and detail than the 'I' format. For example, compare:
I expect to see the most current log entries with no more than a two-minute time delay.
Given the centralized application logging system is operating normally and under average load,
When I view log entries for running applications, from my web browser,
Then I should see the most current log entries with no more than a two-minute time delay.
New User Stories
Following Agile best practices, best illustrated by the INVEST mnemonic, we need to add new, quality user stories, with their own acceptance criteria, around Monitoring, Alerting, Backup and Recovery, Security and Compliance, and High Availability and Fault Tolerance.
Examining all possible facets of each DevOps requirement will improve the overall quality of user stories and acceptance criteria, maximizing business value and minimizing resource utilization.
All opinions in this post are my own and not necessarily the views of my current employer or their clients.
Infrastructure as Code Maturity Model
Posted by Gary A. Stafford in Build Automation, Continuous Delivery, DevOps, Enterprise Software Development on November 25, 2016
Systematically Evolving an Organization's Infrastructure
Infrastructure and software development teams are increasingly building and managing infrastructure using automated tools that have been described as "infrastructure as code." – Kief Morris (Infrastructure as Code)
The process of managing and provisioning computing infrastructure and their configuration through machine-processable, declarative, definition files, rather than physical hardware configuration or the use of interactive configuration tools. – Wikipedia (abridged)
Convergence of CD, Cloud, and IaC
In 2011, co-authors Jez Humble, formerly of ThoughtWorks, and David Farley, published their ground-breaking book, Continuous Delivery. Humble and Farley's book set out, in their words, to automate the 'painful, risky, and time-consuming process' of the software 'build, deployment, and testing process.'
Over the next five years, Humble and Farley's Continuous Delivery made a significant contribution to the modern phenomena of DevOps. According to Wikipedia, DevOps is the 'culture, movement or practice that emphasizes the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes.'
In parallel with the growth of DevOps, Cloud Computing continued to grow at an explosive rate. Amazon pioneered modern cloud computing in 2006 with the launch of its Elastic Compute Cloud. Two years later, in 2008, Microsoft launched its cloud platform, Azure. In 2010, Rackspace launched OpenStack.
Today, there is a flock of 'cloud' providers. Their services fall into three primary service models: Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Since we will be discussing infrastructure, we will focus on IaaS and PaaS. Leaders in this space include Google Cloud Platform, RedHat, Oracle Cloud, Pivotal Cloud Foundry, CenturyLink Cloud, Apprenda, IBM SmartCloud Enterprise, and Heroku, to mention just a few.
Finally, fast forward to June 2016, O'Reilly releases Infrastructure as Code
Managing Servers in the Cloud, by Kief Morris, ThoughtWorks. This crucial work bridges many of the concepts first introduced in Humble and Farley's Continuous Delivery, with the evolving processes and practices to support cloud computing.
This post examines how to apply the principles found in the Continuous Delivery Maturity Model, an analysis tool detailed in Humble and Farley's Continuous Delivery, and discussed herein, to the best practices found in Morris' Infrastructure as Code.
Infrastructure as Code
Before we continue, we need a shared understanding of infrastructure as code. Below are four examples of infrastructure as code, as Wikipedia defined them, 'machine-processable, declarative, definition files.' The code was written using four popular tools, including HashiCorp Packer, Docker, AWS CloudFormation, and HashiCorp Terraform. Executing the code provisions virtualized cloud infrastructure.
HashiCorp Packer
Packer definition of an AWS EBS-backed AMI, based on Ubuntu.
{ "variables": { "aws_access_key": "", "aws_secret_key": "" }, "builders": [{ "type": "amazon-ebs", "access_key": "{{user `aws_access_key`}}", "secret_key": "{{user `aws_secret_key`}}", "region": "us-east-1", "source_ami": "ami-fce3c696", "instance_type": "t2.micro", "ssh_username": "ubuntu", "ami_name": "packer-example {{timestamp}}" }] }
Dockerfile, used to create a Docker image, and subsequently a Docker container, running MongoDB.
FROM ubuntu:16.04 MAINTAINER Docker RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927 RUN echo "deb http://repo.mongodb.org/apt/ubuntu" \ $(cat /etc/lsb-release | grep DISTRIB_CODENAME | cut -d= -f2)/mongodb-org/3.2 multiverse" | \ tee /etc/apt/sources.list.d/mongodb-org-3.2.list RUN apt-get update && apt-get install -y mongodb-org RUN mkdir -p /data/db EXPOSE 27017 ENTRYPOINT ["/usr/bin/mongod"]
AWS CloudFormation
AWS CloudFormation declaration for three services enabled on a running instance.
services: sysvinit: nginx: enabled: "true" ensureRunning: "true" files: - "/etc/nginx/nginx.conf" sources: - "/var/www/html" php-fastcgi: enabled: "true" ensureRunning: "true" packages: yum: - "php" - "spawn-fcgi" sendmail: enabled: "false" ensureRunning: "false"
HashiCorp Terraform
Terraform definition of an AWS m1.small EC2 instance, running NGINX on Ubuntu.
resource "aws_instance" "web" { connection { user = "ubuntu" } instance_type = "m1.small" Ami = "${lookup(var.aws_amis, var.aws_region)}" Key_name = "${aws_key_pair.auth.id}" vpc_security_group_ids = ["${aws_security_group.default.id}"] Subnet_id = "${aws_subnet.default.id}" provisioner "remote-exec" { inline = [ "sudo apt-get -y update", "sudo apt-get -y install nginx", "sudo service nginx start", ] } }
Cloud-based Infrastructure as a Service
The previous examples provide but the narrowest of views into the potential breadth of infrastructure as code. Leading cloud providers, such as Amazon and Microsoft, offer hundreds of unique offerings, most of which may be defined and manipulated through code — infrastructure as code.
What Infrastructure as Code?
The question many ask is, what types of infrastructure can be defined as code? Although vendors and cloud providers have their unique names and descriptions, most infrastructure is divided into a few broad categories:
- Compute
- Databases, Caching, and Messaging
- Storage, Backup, and Content Delivery
- Networking
- Security and Identity
- Monitoring, Logging, and Analytics
- Management Tooling
Continuous Delivery Maturity Model
We also need a common understanding of the Continuous Delivery Maturity Model. According to Humble and Farley, the Continuous Delivery Maturity Model was distilled as a model that 'helps to identify where an organization stands in terms of the maturity of its processes and practices and defines a progression that an organization can work through to improve.'
The Continuous Delivery Maturity Model is a 5×6 matrix, consisting of six areas of practice and five levels of maturity. Each of the matrix's 30 elements defines a required discipline an organization needs to follow, to be considered at that level of maturity within that practice.
Areas of Practice
The CD Maturity Model examines six broad areas of practice found in most enterprise software organizations:
- Build Management and Continuous Integration
- Environments and Deployment
- Release Management and Compliance
- Testing
- Data Management
- Configuration Management
Levels of Maturity
The CD Maturity Model defines five level of increasing maturity, from a score of -1 to 3, from Regressive to Optimizing:
- Level 3: Optimizing – Focus on process improvement
- Level 2: Quantitatively Managed – Process measured and controlled
- Level 1: Consistent – Automated processes applied across whole application lifecycle
- Level 0: Repeatable – Process documented and partly automated
- Level -1: Regressive – Processes unrepeatable, poorly controlled, and reactive
Maturity Model Analysis
The CD Maturity Model is an analysis tool. In my experience, organizations use the maturity model in one of two ways. First, an organization completes an impartial evaluation of their existing levels of maturity across all areas of practice. Then, the organization focuses on improving the overall organization's maturity, attempting to achieve a consistent level of maturity across all areas of practice. Alternately, the organization concentrates on a subset of the practices, which have the greatest business value, or given their relative immaturity, are a detriment to the other practices.
* CD Maturity Model Analysis Tool available on GitHub.
Infrastructure as Code Maturity Levels
Although infrastructure as code is not explicitly called out as a practice in the CD Maturity Model, many of it's best practices can be found in the maturity model. For example, the model prescribes automated environment provisioning, orchestrated deployments, and the use of metrics for continuous improvement.
Instead of trying to retrofit infrastructure as code into the existing CD Maturity Model, I believe it is more effective to independently apply the model's five levels of maturity to infrastructure as code. To that end, I have selected many of the best practices from the book, Infrastructure as Code, as well as from my experiences. Those selected practices have been distributed across the model's five levels of maturity.
The result is the first pass at an evolving Infrastructure as Code Maturity Model. This model may be applied alongside the broader CD Maturity Model, or independently, to evaluate and further develop an organization's infrastructure practices.
IaC Level -1: Regressive
Processes unrepeatable, poorly controlled, and reactive
- Limited infrastructure is provisioned and managed as code
- Infrastructure provisioning still requires many manual processes
- Infrastructure code is not written using industry-standard tooling and patterns
- Infrastructure code not built, unit-tested, provisioned and managed, as part of a pipeline
- Infrastructure code, processes, and procedures are inconsistently documented, and not available to all required parties
IaC Level 0: Repeatable
Processes documented and partly automated
- All infrastructure code and configuration are stored in a centralized version control system
- Testing, provisioning, and management of infrastructure are done as part of automated pipeline
- Infrastructure is deployable as individual components
- Leverages programmatic interfaces into physical devices
- Automated security inspection of components and dependencies
- Self-service CLI or API, where internal customers provision their resources
- All code, processes, and procedures documented and available
- Immutable infrastructure and processes
IaC Level 1: Consistent
Automated processes applied across whole application lifecycle
- Fully automated provisioning and management of infrastructure
- Minimal use of unsupported, 'home-grown' infrastructure tooling
- Unit-tests meet code-coverage requirements
- Code is continuously tested upon every check-in to version control system
- Continuously available infrastructure using zero-downtime provisioning
- Uses configuration registries
- Templatized configuration files (no awk/sed magic)
- Secrets are securely management
- Auto-scaling based on user-defined load characteristics
IaC Level 2: Quantitatively Managed
Processes measured and controlled
- Uses infrastructure definition files
- Capable of automated rollbacks
- Infrastructure and supporting systems are highly available and fault tolerant
- Externalized configuration, no black box API to modify configuration
- Fully monitored infrastructure with configurable alerting
- Aggregated, auditable infrastructure logging
- All code, processes, and procedures are well documented in a Knowledge Management System
- Infrastructure code uses declarative versus imperative programming model, maybe…
IaC Level 3: Optimizing
Focus on process improvement
- Self-healing, self-configurable, self-optimizing, infrastructure
- Performance tested and monitored against business KPIs
- Maximal infrastructure utilization and workload density
- Adheres to Cloud Native and 12-Factor patterns
- Cloud-agnostic code that minimizes cloud vendor lock-in
All opinions in this post are my own and not necessarily the views of my current employer or their clients.
Operational Readiness Analysis
Posted by Gary A. Stafford in Continuous Delivery, DevOps, Software Development on October 12, 2016
"Analysis: a detailed examination of the elements or structure of something, typically as a basis for discussion or interpretation." — Google
Recently, I had the opportunity, along with a colleague, to perform an operational readiness analysis of a client's new application platform. The platform was in late-stage development, only a few weeks from going live. Being relatively new to the project team, our objective was to determine the amount of work remaining to make the platform operational, from a technical perspective. As well, we wanted to ensure there were no gaps in the project's scope, as it related to technical operational readiness.
There were various approaches we could have taken to establish the amount of unfinished work and any existing gaps. We could have reviewed the state of the project in the client's Agile project management system. We could have discussed the project's state with the Product Owner, Project Manager, Team Lead, and Business Analyst. Neither of these methods on their own would have provided us with a complete picture.
The approach we ultimately took was what we've coined, an Operational Readiness Analysis, or our trendier title, the 'State of the DevOps'. This approach is particularly effective and highly interactive. The analysis may be conducted at any stage of the Software Development Lifecycle (SDLC), to review a project's operational state of readiness. The analysis involves five simple steps:
- Categorize
- Itemize
- Organize
- Prioritize
- Document
Grab Your Post-It Notes
We began, where all good ThoughtWorkers tend to begin, with a meeting room, whiteboard, dry-erase markers, and lots of brightly colored Post-It notes. ThoughtWorkers do love their Post-It notes. We gathered a small subset of the project team, including the Product Owner, Project Manager, and the team's DevOps Engineer.
Step 1: Categorize
To start, we broadly categorized the operational requirements into buckets of work. In my experience, categorization typically takes one of two forms, either the use of role-based descriptors, like 'Security' and 'Monitoring', or the use of the 'ilities', such as 'Scalability' and 'Maintainability'.
The 'ilities' are often referenced in regards to software architecture and the functional and nonfunctional requirements (NFRs) of a project. Although there are many 'ilities', project requirements frequently include Usability, Scalability, Maintainability, Reliability, Testability, Availability, and Serviceability. Although I enjoy referring to the 'ilities' when drafting project requirements, I find a broad audience more readily understands the high-level category descriptions.
We collectively agreed on eight categories, in addition to a catch-all 'Other' category, and distributed them on the whiteboard.
Role-based categories tend to include the following:
- Security and Compliance
- Monitoring and Alerting
- Logging
- Continuous Integration and Delivery
- Release Management
- Configuration
- Environment Management
- Infrastructure
- Networking
- Data Management
- High Availability and Fault Tolerance
- Performance
- Backup and Restoration
- Support and Maintenance
- Documentation and Training
- Technical Debt
- Other
Step 2: Itemize
Next, each person placed as many Post-It notes as they wanted, below the categories on the whiteboard. Each Post-It represented an item someone felt needed to be addressed as part the analysis. Items might represent requirements currently in progress, or requirements still untouched in the project's backlog. A Post-It note might contain a new requirement. Maybe there is a question about an aspect of the project's relative importance, or there were concerns about how an aspect of the project had been implemented.
Participants placed an average of five to ten Post-It notes on the whiteboard. The end product was a collective mind-map of the project's operational state (the following is only an example and doesn't represent the actual results of any real client analysis).
Whereas categories tend to be generic in nature, the items are usually particular to the project, and it's technology choices. The 'Monitoring/Logging' category might include individual items such as 'Log Rotation Policy' or 'Splunk vs. ELK?'. The 'Security' category might include individual items such as 'Need Password Rotation' or 'SSL Certificate Management'.
Participants often use one or two words to summarize more complex thoughts. For example, 'Rollback' might be stating, 'we still don't have a good rollback strategy for the database'. 'Release Frequency' may be questioning, 'why can't we release more frequently?'. Make sure to discuss and capture the participant's full thoughts behind each Post-It, the during the Organize, Prioritize, and Document stages.
Don't let participants give up too soon placing Post-It notes on the board. Often, one participant's Post-It will spark additional thoughts other team members. Don't be afraid to suggest a few items, if the group needs some initial motivation. Lastly, fear not, it's never too late to go back and add missing items, uncovered in later stages of analysis.
Step 3: Organize
Quickly and non-judgmentally, review all items on the whiteboard. Group duplicates together. Ask for clarification on items where necessary. Move miscategorized items, if the author and team agree there is a more appropriate category.
Examine the 'Other' category. If there are items in the 'Other' category that are similar, consider adding a new category to capture them. Maybe the team missed identifying a key category, earlier. Conversely, there is nothing wrong with leaving a few stragglers in the 'Other' category; don't overthink the exercise.
Participants should leave this stage with a general understanding of each item on the whiteboard.
Step 4: Prioritize
With all items identified, organized, and understood, discuss each item's status. Does the item represent incomplete requirements? Does everyone agree an item's requirements are complete? Team members may not agree on the completeness of particular items. Inconsistencies may reflect unclear requirements. More often, I find, disagreements are a result of an inconsistent or incomplete implementation of requirements across environments — Development, QA, Performance, Staging, and Production.
Why are operational requirements frequently implemented inconsistently? Often, a story is played early in the development stage, prior to all environments being available. For example, a story is played to add monitoring to Development and QA. The story is completed, tested, and closed. Afterward, the Performance environment is created. Later yet, the Staging and Production environments are created. But, there are no new stories to address monitoring the new environments.
One way to avoid this common issue with operational requirements is an effective DevOps automation strategy. In this example, an effective strategy would mean all new environments would get monitoring, automatically. The story should broadly address monitoring, and not, short-sightedly, monitoring of specific environments.
Often, only one or two project team members are responsible for 'all the DevOps stuff'. They alone possess an accurate perspective on an item's current status.
Gain agreement from the Product Owner, Project Manager, and key team members on the priority of incomplete requirements. Are they 'Day One' must-haves, or 'Day Two' and can be completed after the application goes live? Flag all high-priority items.
If the status of an item is unknown, such as 'why is the QA environment always down?!', note it as an open question and move on. If the item simply requires a quick answer to resolve, like 'do we backup the database?', answer it ('the database is replicated and backed up daily') and move on.
Step 5: Document
Snap a picture of the whiteboard and document its contents. We chose the client's knowledge management system as a vehicle to share the whiteboard's results. We created a table with columns for Category, Item, Description, Priority, Owner, and Notes. Also, it's helpful to have a column for the project's corresponding Story ID or Defect ID. Along with the whiteboard, capture key discussion points, questions, and concerns, raised by the participants.
Step 5: Document
Snap a picture of the whiteboard and document its contents. We chose Confluence to share the whiteboard results, using a table with columns for Category, Item, Description, Priority, Owner, and Notes. We also bulleted all key discussion points, questions, and concerns, raised by the participants.
Make the analysis results available to all team members. Let the team ask questions and poke holes in the results. Adjust the results if necessary.
Next Steps
The actions you take, given the results of the analysis, are up to you. In our case, we ensured all high-priority items were called out on the team's Agile board. Any new items were captured in the client's Agile project management system. Finally, we ensured open questions and concerns were addressed in a timely fashion. We continue to track each item's status weekly, throughout the launch and post-launch periods.
We anticipate conducting a follow-up analysis, thirty days after launch. The goal will be to evaluate the effectiveness of the first analysis and identify additional operational needs as the application enters a business-as-usual (BAU) application lifecycle phase.
Postscript: the 'ilities'
The 'ilities', courtesy of codesqueeze.com and en.wikipedia.org
- Scalability
The capability of a system, network, or process to handle a growing amount of work or its potential to be enlarged to accommodate that growth.
- Testability
The practical feasibility of observing a reproducible series of such counterexamples if they do exist.
- Reliability (Resilience)
The ability of a system or component to perform its required functions under stated conditions for a specified time.
- Usability (Performance)
The degree to which software can be used by specified consumers to achieve quantified objectives with effectiveness, efficiency, and satisfaction in a quantified context of use.
- Serviceability (Supportability)
The ability of technical support personnel to install, configure, and monitor computer products, identify exceptions or faults, debug or isolate faults to root cause analysis, and provide hardware or software maintenance in pursuit of solving a problem and restoring the product into service.
- Availability
The proportion of time a system is in a functioning condition. Defined in the service-level agreement (SLA).
- Maintainability
The ease with which a product can be maintained. Isolate and correct defects or their cause, prevent unexpected breakdowns, maximize a product's useful life, maximize efficiency, reliability, and safety, meet new requirements, make future maintenance easier, and cope with a changed environment.
All opinions in this post are my own and not necessarily the views of my current employer or their clients.
Tips for Better of Software Releases
Posted by Gary A. Stafford in Continuous Delivery, Software Development on September 28, 2016
Although the process of releasing software varies considerably from application to application, we would probably all agree there are common reasons releases tend to fail. Lack of planning and preparation, defective code, infrastructure and release tooling failures, and poor communication, all transcend technology choices and industries.
Indeed, no one can guarantee a release's success each and every time. However, there are steps we can take to improve the odds of executing consistently successful software releases. Based on my experience with a diverse set of enterprise software organizations, I've collected following suggestions for improving release success.
Manage the Release, Don't Let the Release Manage You
Engage an experienced Release Manager, someone who schedules the release, coordinates the resources, runs the release call, leads the release team through the plan, and communicates status to stakeholders. The Release Manager serves as a single point of contact (aka SPOC) for the release. Having a Release manager allows other release team members to focus on their specific tasks without needless distraction.
A Goal Without a Plan is Just a Wish
Always have an implementation plan (aka IPlan). Whether a single page of paper, or a hundred-page Microsoft Word document, script the entire release in advance. What are the required tasks? What is their order of execution? What resources are required to complete each task? What is the test plan? What is the contingency plan, if or when something goes awry?
Don't Boil the Ocean
Releases should have specific goals, the fewer, the better. Release goals should be clearly stated at the beginning of the implementation plan. If you feel you need to compress several goals into a single release, you are probably not releasing frequently enough.
For simplicity, separate software-centric releases from infrastructure-centric changes. Don't tie the success of either, to the success or failure of the other.
Expect Success, Plan for Failure
Channel your inner Boy Scout and always be prepared. Have a contingency plan. Plan for multiple failure scenarios. Individual task failures only lead to release failures when we don't have a plan to correct for the individual failures quickly. Are additional resources required in the event of a failure? If so, they should be available, and aware of the release plan and the contingency plan, in advance.
Seek the Understanding and Approval of Others
This is one of those times when seeking the understanding and the approval of others is a good thing. Review the plan in advance with all required resources and stakeholders. Ensure there is a complete comprehension of the plan, goals, resource requirements, and timing. Discuss the plan's level of risk to the organization and their customers. Seek the support and approval of stakeholders when required.
The More I Practice the Luckier I Get
Do a dry run of the plan, even if it is just a verbal walkthrough with the release team. Better yet, do a live run in a production-like staging environment. Adjust the plan if necessary. Also, don't forget to practice your contingency plan.
Should I Pack a Lunch?
Part of the plan′s review should be a discussion of timing. Release resources should know the estimated total time for the release (aka release window) — best case and not-so-best case. Also, if individual stages are expected to take more than a few minutes to complete, those times should be understood in advance. There is nothing worse than staring endlessly at the third stage of a ten stage continuous integration pipeline for 15 minutes, with no idea of how much long the task is expected to take.
Don't Start the Game without the Whole Team on the Field
Never start a release without all the required resources present and prepared. A quick release can turn into a long and painful experience if you are waiting on resources. In my experience, the longer a release takes to complete, the greater the risk of failure.
Make a Plan and Stick to It
You wrote the plan, you reviewed the plan, you practiced the plan, and you received the approval of your stakeholders for the plan. So, follow the plan.
If you absolutely must deviate from the plan, take the time to consider the impact and potential risks. Document the deviation for the post-mortem and for planning the next release.
Test Early, Test Often
Testing should not be the final step to confirm the release's success. Testing should be done continuously and automatically, throughout the release process. Test after each significant stage of the plan. It's easier to find and correct issues the early they are discovered.
Mute is Your Friend, Always be on Mute
Emotions will flare, words will uncontrollably leap from your lips on occasion, background noise makes following the release call difficult to follow, and diagnosing release issues isn't best done in front of an audience. Keep the release call focused on the plan and free of emotion.
Are We There Yet?
Consistently communicate before, during, and after the release. Let stakeholders know in advance when the release is starting and how long it is expected to take. Keep stakeholders aware of significant deviations to the plan, especially with customer-facing impact. Let everyone know when the release finishes, and a final release status. Keep communications germane.
Ensure sure everyone understands how the release status will be communicated, be it email, IM, persistent chat, or a web-based status page.
Mistakes are Meant for Learning, not Repeating
Successful or otherwise, follow each release with a blameless post-mortem. A post-mortem might only require a quick five-minute chat. Or, the whole release team might need an hour with a therapist (that's a joke). Discuss what went right, and what did not go as well as expected. Be keen to focus on repeat problems and problems that were not caught during the release. Most importantly, consider how to continuously improve the release process.
Releasing is a Game, Keep Score
Know your release's Key Performance Indicators (KPIs) and your Service Level Agreements (SLAs). Understand what matters most to your stakeholders and to your customers. Always know how you are performing against your KPIs and SLAs, and what you need to do to improve them. Dashboards are great tools to display KPI and SLA performance transparently.
Is your failure rate increasing or decreasing? Is one type of task responsible for a majority of your release failures? Is your release taking longer or shorter to complete each time? Did you experience an unexpected outage during the release? For how long? What is the volume of post-release issues caused by, but not discovered during the release?
All opinions in this post are my own and not necessarily the views of my current employer or their clients.
Spring Music Revisited: Java-Spring-MongoDB Web App with Docker 1.12
Posted by Gary A. Stafford in Build Automation, Continuous Delivery, DevOps, Enterprise Software Development, Java Development, Software Development on August 7, 2016
Build, test, deploy, and monitor a multi-container, MongoDB-backed, Java Spring web application, using the new Docker 1.12.
This post and the post's example project represent an update to a previous post, Build and Deploy a Java-Spring-MongoDB Application using Docker. This new post incorporates many improvements made in Docker 1.12, including the use of the new Docker Compose v2 YAML format. The post's project was also updated to use Filebeat with ELK, as opposed to Logspout, which was used previously.
In this post, we will demonstrate how to build, test, deploy, and manage a Java Spring web application, hosted on Apache Tomcat, load-balanced by NGINX, monitored by ELK with Filebeat, and all containerized with Docker.
We will use a sample Java Spring application, Spring Music, available on GitHub from Cloud Foundry. The Spring Music sample record album collection application was originally designed to demonstrate the use of database services on Cloud Foundry, using the Spring Framework. Instead of Cloud Foundry, we will host the Spring Music application locally, using Docker on VirtualBox, and optionally on AWS.
All files necessary to build this project are stored on the
docker_v2 branch of the garystafford/spring-music-docker repository on GitHub. The Spring Music source code is stored on the
springmusic_v2 branch of the garystafford/spring-music repository, also on GitHub.
Application Architecture
The Java Spring Music application stack contains the following technologies: Java, Spring Framework, AngularJS, Bootstrap, jQuery, NGINX, Apache Tomcat, MongoDB, the ELK Stack, and Filebeat. Testing frameworks include the Spring MVC Test Framework, Mockito, Hamcrest, and JUnit.
A few changes were made to the original Spring Music application to make it work for this demonstration, including:
- Move from Java 1.7 to 1.8 (including newer Tomcat version)
- Add unit tests for Continuous Integration demonstration purposes
- Modify MongoDB configuration class to work with non-local, containerized MongoDB instances
- Add Gradle
warNoStatictask to build WAR without static assets
- Add Gradle
zipStatictask to ZIP up the application's static assets for deployment to NGINX
- Add Gradle
zipGetVersiontask with a versioning scheme for build artifacts
- Add
context.xmlfile and
MANIFEST.MFfile to the WAR file
- Add Log4j
RollingFileAppenderappender to send log entries to Filebeat
- Update versions of several dependencies, including Gradle, Spring, and Tomcat
We will use the following technologies to build, publish, deploy, and host the Java Spring Music application: Gradle, git, GitHub, Travis CI, Oracle VirtualBox, Docker, Docker Compose, Docker Machine, Docker Hub, and optionally, Amazon Web Services (AWS).
To increase performance, the Spring Music web application's static content will be hosted by NGINX. The application's WAR file will be hosted by Apache Tomcat 8.5.4. Requests for non-static content will be proxied through NGINX on the front-end, to a set of three load-balanced Tomcat instances on the back-end. To further increase application performance, NGINX will also be configured for browser caching of the static content. In many enterprise environments, the use of a Java EE application server, like Tomcat, is still not uncommon.
Reverse proxying and caching are configured thought NGINX's
default.conf file, in the
server configuration section:
The three Tomcat instances will be manually configured for load-balancing using NGINX's default round-robin load-balancing algorithm. This is configured through the
default.conf file, in the
upstream configuration section:
Client requests are received through port
80 on the NGINX server. NGINX redirects requests, which are not for non-static assets, to one of the three Tomcat instances on port
The Spring Music application was designed to work with a number of data stores, including MySQL, Postgres, Oracle, MongoDB, Redis, and H2, an in-memory Java SQL database. Given the choice of both SQL and NoSQL databases, we will select MongoDB.
The Spring Music application, hosted by Tomcat, will store and modify record album data in a single instance of MongoDB. MongoDB will be populated with a collection of album data from a JSON file, when the Spring Music application first creates the MongoDB database instance.
Lastly, the ELK Stack with Filebeat, will aggregate NGINX, Tomcat, and Java Log4j log entries, providing debugging and analytics to our demonstration. A similar method for aggregating logs, using Logspout instead of Filebeat, can be found in this previous post.
Continuous Integration
In this post's example, two build artifacts, a WAR file for the application and ZIP file for the static web content, are built automatically by Travis CI, whenever source code changes are pushed to the
springmusic_v2 branch of the garystafford/spring-music repository on GitHub.
Following a successful build and a small number of unit tests, Travis CI pushes the build artifacts to the
build-artifacts branch on the same GitHub project. The
build-artifacts branch acts as a pseudo binary repository for the project, much like JFrog's Artifactory. These artifacts are used later by Docker to build the project's immutable Docker images and containers.
Build Notifications
Travis CI pushes build notifications to a Slack channel, which eliminates the need to actively monitor Travis CI.
Automation Scripting
.travis.yaml file, custom
gradle.build Gradle tasks, and the
deploy_travisci.sh script handles the Travis CI automation described, above.
Travis CI
.travis.yaml file:
gradle.build tasks:
deploy.sh file:
You can easily replicate the project's continuous integration automation using your choice of toolchains. GitHub or BitBucket are good choices for distributed version control. For continuous integration and deployment, I recommend Travis CI, Semaphore, Codeship, or Jenkins. Couple those with a good persistent chat application, such as Glider Labs' Slack or Atlassian's HipChat.
Building the Docker Environment
Make sure VirtualBox, Docker, Docker Compose, and Docker Machine, are installed and running. At the time of this post, I have the following versions of software installed on my Mac:
- Mac OS X 10.11.6
- VirtualBox 5.0.26
- Docker 1.12.1
- Docker Compose 1.8.0
- Docker Machine 0.8.1
To build the project's VirtualBox VM, Docker images, and Docker containers, execute the build script, using the following command:
sh ./build_project.sh. A build script is useful when working with CI/CD automation tools, such as Jenkins CI or ThoughtWorks go. However, to understand the build process, I suggest first running the individual commands, locally.
Deploying to AWS
By simply changing the Docker Machine driver to AWS EC2 from VirtualBox, and providing your AWS credentials, the
springmusic environment may also be built on AWS.
Build Process
Docker Machine provisions a single VirtualBox
springmusic VM on which host the project's containers. VirtualBox provides a quick and easy solution that can be run locally for initial development and testing of the application.
Next, the script creates a Docker data volume and project-specific Docker bridge network.
Next, using the project's individual Dockerfiles, Docker Compose pulls base Docker images from Docker Hub for NGINX, Tomcat, ELK, and MongoDB. Project-specific immutable Docker images are then built for NGINX, Tomcat, and MongoDB. While constructing the project-specific Docker images for NGINX and Tomcat, the latest Spring Music build artifacts are pulled and installed into the corresponding Docker images.
Docker Compose builds and deploys (6) containers onto the VirtualBox VM: (1) NGINX, (3) Tomcat, (1) MongoDB, and (1) ELK.
The NGINX
The Tomcat
Docker Compose v2 YAML
This post was recently updated for Docker 1.12, and to use Docker Compose v2 YAML file format. The post's
docker-compose.yml takes advantage of improvements in Docker 1.12 and Docker Compose v2 YAML. Improvements to the YAML file include eliminating the need to link containers and expose ports, and the addition of named networks and volumes.
The Results
Below are the results of building the project.
Testing the Application
Below are partial results of the curl test, hitting the NGINX endpoint. Note the different IP addresses in the
Upstream-Address field between requests. This test proves NGINX's round-robin load-balancing is working across the three Tomcat application instances:
music_app_2, and
Also, note the sharp decrease in the
Request-Time between the first three requests and subsequent three requests. The
Upstream-Response-Time to the Tomcat instances doesn't change, yet the total
Request-Time is much shorter, due to caching of the application's static assets by NGINX.
Spring Music Application Links
Assuming the
springmusic VM is running at
192.168.99.100, the following links can be used to access various project endpoints. Note the (3) Tomcat instances each map to randomly exposed ports. These ports are not required by NGINX, which maps to port 8080 for each instance. The port is only required if you want access to the Tomcat Web Console. The port, shown below, 32771, is merely used as an example.
- Spring Music Application: 192.168.99.100
- NGINX Status: 192.168.99.100/nginx_status
- Tomcat Web Console – music_app_1*: 192.168.99.100:32771/manager
- Environment Variables – music_app_1: 192.168.99.100:32771/env
- Album List (RESTful endpoint) – music_app_1: 192.168.99.100:32771/albums
- Elasticsearch Info: 192.168.99.100:9200
- Elasticsearch Status: 192.168.99.100:9200/_status?pretty
- Kibana Web Console: 192.168.99.100:5601
* The Tomcat user name is
admin and the password is
Helpful Links
- Cloud Foundry's Spring Music Example
- Getting Started with Gradle for Java
- Introduction to Gradle
- Spring Framework
- Understanding Nginx HTTP Proxying, Load Balancing, Buffering, and Caching
- Common conversion patterns for log4j's PatternLayout
- Spring @PropertySource example
- Java log4j logging
- Automate the Docker image build and publish processes
- Automate the Docker container build and deploy processes
- Automate post-deployment verification testing of project infrastructure
- Add Docker Swarm multi-host capabilities with overlay networking
- Update Spring Music with latest CF project revisions
- Include scripting example to stand-up project on AWS
- Add Consul and Consul Template for NGINX configuration
Scaffold a RESTful API with Yeoman, Node, Restify, and MongoDB
Posted by Gary A. Stafford in Continuous Delivery, Enterprise Software Development, Software Development on June 22, 2016
Using Yeoman, scaffold a basic RESTful CRUD API service, based on Node, Restify, and MongoDB.
NOTE: Generator updated on 11-13-2016 to v0.2.1.
Yeoman generators reduce the repetitive coding of boilerplate functionality and ensure consistency between full-stack JavaScript projects. For several recent Node.js projects, I created the generator-node-restify-mongodb Yeoman generator. This Yeoman generator scaffolds a basic RESTful CRUD API service, a Node application, based on Node.js, Restify, and MongoDB.
According to their website, Restify, used most notably by Netflix, borrows heavily from Express. However, while Express is targeted at browser applications, with templating and rendering, Restify is keenly focused on building API services that are maintainable and observable.
Along with Node, Restify, and MongoDB, theNode application's scaffolded by the Node-Restify-MongoDB Generator, also implements Bunyan, which includes DTrace, Jasmine, using jasmine-node, Mongoose, and Grunt.
Portions of the scaffolded Node application's file structure and code are derived from what I consider the best parts of several different projects, including generator-express, generator-restify-mongo, and generator-restify.
To begin, install Yeoman and the generator-node-restify-mongodb using npm. The generator assumes you have pre-installed Node and MongoDB.
npm install -g yo npm install -g generator-node-restify-mongodb
Then, generate the new project.
mkdir node-restify-mongodb cd $_ yo node-restify-mongodb
Yeoman scaffolds the application, creating the directory structure, copying required files, and running 'npm install' to load the npm package dependencies.
Using the Generated Application
Next, import the supplied set of sample widget documents into the local development instance of MongoDB from the supplied 'data/widgets.json' file.
NODE_ENV=development grunt mongoimport --verbose
Similar to Yeoman's Express Generator, this application contains configuration for three typical environments: 'Development' (default), 'Test', and 'Production'. If you want to import the sample widget documents into your Test or Production instances of MongoDB, first change the 'NODE_ENV' environment variable value.
NODE_ENV=production grunt mongoimport --verbose
To start the application in a new terminal window, use the following command.
npm start
The output should be similar to the example, below.
To test the application, using jshint and the jasmine-node module, the sample documents must be imported into MongoDB and the application must be running (see above). To test the application, open a separate terminal window, and use the following command.
npm test
The project contains a set of jasmine-node tests, split between the '/widgets' and the '/utils' endpoints. If the application is running correctly, you should see the following output from the tests.
Similarly, the following command displays a code coverage report, using the grunt, mocha, istanbul, and grunt-mocha-istanbul node modules.
grunt coverage
Grunt uses the grunt-mocha-istanbul module to execute the same set of jasmine-node tests as shown above. Based on those tests, the application's code coverage (statement, line, function, and branch coverage) is displayed.
You may test the running application, directly, by cURLing the '/widgets' endpoints.
curl -X GET -H "Accept: application/json" "http://localhost:3000/widgets"
For more legible output, try prettyjson.
npm install -g prettyjson curl -X GET -H "Accept: application/json" "http://localhost:3000/widgets" --silent | prettyjson curl -X GET -H "Accept: application/json" "http://localhost:3000/widgets/SVHXPAWEOD" --silent | prettyjson
The JSON-formatted response body from the HTTP GET requests should look similar to the output, below.
A much better RESTful API testing solution is Postman. Postman provides the ability to individually configure each environment and abstract that environment-specific configuration, such as host and port, from the actual HTTP requests.
Continuous Integration
As part of being published to both the npmjs and Yeoman registries, the generator-node-restify-mongodb generator is continuously integrated on Travis CI. This should provide an addition level of confidence to the generator's end-users. Currently, Travis CI tests the generator against Node.js v4, v5, and v6, as well as IO.js. Older versions of Node.js may have compatibility issues with the application.
Additionally, Travis CI feeds test results to Coveralls, which displays the generator's code coverage. Note the code coverage, shown below, is reported for the yeoman generator, not the generator's scaffolded application. The scaffolded application's coverage is shown above.
Application Details
API Endpoints
The scaffolded application includes the following endpoints.
# widget resources var PATH = '/widgets'; server.get({path: PATH, version: VERSION}, findDocuments); server.get({path: PATH + '/:product_id', version: VERSION}, findOneDocument); server.post({path: PATH, version: VERSION}, createDocument); server.put({path: PATH, version: VERSION}, updateDocument); server.del({path: PATH + '/:product_id', version: VERSION}, deleteDocument); # utility resources var PATH = '/utils'; server.get({path: PATH + '/ping', version: VERSION}, ping); server.get({path: PATH + '/health', version: VERSION}, health); server.get({path: PATH + '/info', version: VERSION}, information); server.get({path: PATH + '/config', version: VERSION}, configuraton); server.get({path: PATH + '/env', version: VERSION}, environment);
The Widget
The Widget is the basic document object used throughout the application. It is used, primarily, to demonstrate Mongoose's Model and Schema. The Widget object contains the following fields, as shown in the sample widget, below.
{ "product_id": "4OZNPBMIDR", "name": "Fapster", "color": "Orange", "size": "Medium", "price": "29.99", "inventory": 5 }
Use the mongo shell to access the application's MongoDB instance and display the imported sample documents.
mongo > show dbs > use node-restify-mongodb-development > show tables > db.widgets.find()
The imported sample documents should be displayed, as shown below.
Environmental Variables
The scaffolded application relies on several environment variables to determine its environment-specific runtime configuration. If these environment variables are present, the application defaults to using the Development environment values, as shown below, in the application's 'config/config.js' file.
var NODE_ENV = process.env.NODE_ENV || 'development'; var NODE_HOST = process.env.NODE_HOST || '127.0.0.1'; var NODE_PORT = process.env.NODE_PORT || 3000; var MONGO_HOST = process.env.MONGO_HOST || '127.0.0.1'; var MONGO_PORT = process.env.MONGO_PORT || 27017; var LOG_LEVEL = process.env.LOG_LEVEL || 'info'; var APP_NAME = 'node-restify-mongodb-';
Future Project TODOs
Future project enhancements include the following:
- Add filtering, sorting, field selection and paging
- Add basic HATEOAS-based response features
- Add authentication and authorization to production MongoDB instance
- Convert from out-dated jasmine-node to Jasmine?
Diving Deeper into 'Getting Started with Spring Cloud'
Posted by Gary A. Stafford in DevOps, Enterprise Software Development, Java Development, Software Development on February 15, 2016
Explore the integration of Spring Cloud and Spring Cloud Netflix tooling, through a deep dive into Pivotal's 'Getting Started with Spring Cloud' presentation.
Keeping current with software development and DevOps trends can often make us feel we are, as the overused analogy describes, drinking from a firehose, often several hoses at once. Recently joining a large client engagement, I found it necessary to supplement my knowledge of cloud-native solutions, built with the support of Spring Cloud and Spring Cloud Netflix technologies. One of my favorite sources of information on these subjects is presentations by people like Josh Long, Dr. Dave Syer, and Cornelia Davis of Pivotal Labs, and Jon Schneider and Taylor Wicksell of Netflix.
One presentation, in particular, Getting Started with Spring Cloud, by Long and Syer, provides an excellent end-to-end technical overview of the latest Spring and Netflix technologies. Josh Long's fast-paced, eighty-minute presentation, available on YouTube, was given at SpringOne2GX 2015 with co-presenter, Dr. Dave Syer, founder of Spring Cloud, Spring Boot, and Spring Batch.
As the presenters of Getting Started with Spring Cloud admit, the purpose of the presentation was to get people excited about Spring Cloud and Netflix technologies, not to provide a deep dive into each technology. However, I believe the presentation's Reservation Service example provides an excellent learning opportunity. In the following post, we will examine the technologies, components, code, and configuration presented in Getting Started with Spring Cloud. The goal of the post is to provide a greater understanding of the Spring Cloud and Spring Cloud Netflix technologies.
System Overview
The presentation's example introduces a dizzying array of technologies, which include:
Spring Boot
Stand-alone, production-grade Spring-based applications
Spring Data REST / Spring HATEOAS
Spring-based applications following HATEOAS principles
Spring Cloud Config
Centralized external configuration management, backed by Git
Netflix Eureka
REST-based service discovery and registration for failover and load-balancing
Netflix Ribbon
IPC library with built-in client-side software load-balancers
Netflix Zuul
Dynamic routing, monitoring, resiliency, security, and more
Netflix Hystrix
Latency and fault tolerance for distributed system
Netflix Hystrix Dashboard
Web-based UI for monitoring Hystrix
Spring Cloud Stream
Messaging microservices, backed by Redis
Spring Data Redis
Configuration and access to Redis from a Spring app, using Jedis
Spring Cloud Sleuth
Distributed tracing solution for Spring Cloud, sends traces via Thrift to the Zipkin collector service
Twitter Zipkin
Distributed tracing system, backed by Apache Cassandra
In-memory Java SQL database, embedded and server modes
Package applications with dependencies into standardized Linux containers
System Components
Several components and component sub-systems comprise the presentation's overall Reservation Service example. Each component implements a combination of the technologies mentioned above. Below is a high-level architectural diagram of the presentation's example. It includes a few additional features, added as part of this post.
Individual system components include:
Spring Cloud Config Server
Stand-alone Spring Boot application provides centralized external configuration to multiple Reservation system components
Spring Cloud Config Git Repo
Git repository containing multiple Reservation system components configuration files, served by Spring Cloud Config Server
H2 Java SQL Database Server (New)
This post substitutes the original example's use of H2's embedded version with a TCP Server instance, shared by Reservation Service instances
Reservation Service
Multi load-balanced instances of stand-alone Spring Boot application, backed by H2 database
Reservation Client
Stand-alone Spring Boot application (aka edge service or client-side proxy), forwards client-side load-balanced requests to the Reservation Service, using Eureka, Zuul, and Ribbon
Reservation Data Seeder (New)
Stand-alone Spring Boot application, seeds H2 with initial data, instead of the Reservation Service
Eureka Service
Stand-alone Spring Boot application provides service discovery and registration for failover and load-balancing
Hystrix Dashboard
Stand-alone Spring Boot application provides web-based Hystrix UI for monitoring system performance and Hystrix circuit-breakers
Zipkin Collector, Query, and Web, and Cassandra database, receives, correlates, and displays traces from Spring Cloud Sleuth
In-memory data structure store, acting as message broker/transport for Spring Cloud Stream
All the code for this post is available on Github, split between two repositories. The first repository, spring-cloud-demo, contains the source code for all of the components listed above, except the Spring Cloud Config Git Repo. To function correctly, the configuration files, consumed by the Spring Cloud Config Server, needs to be placed into a separate repository, spring-cloud-demo-config-repo.
The first repository contains a git submodule ,
docker-zipkin. If you are not familiar with submodules, you may want to take a moment to read the git documentation. The submodule contains a dockerized version of Twitter's OpenZipkin, docker-zipkin. To clone the two repositories, use the following commands. The
--recursive option is required to include the
docker-zipkin submodule in the project.
To try out the post's Reservation system example, you need to configure at least one property. The Spring Cloud Config Server needs to know the location of the Spring Cloud Config Repository, which is the second GitHub repository you cloned,
spring-cloud-demo-config-repo. From the root of the
spring-cloud-demo repo, edit the Spring Cloud Config Server
application.properties file, located in
config-server/src/main/resources/application.properties. Change the following property's value to your local path to the
spring-cloud-demo-config-repo repository:
There are a few ways you could run the multiple components that make up the post's example. I suggest running one component per terminal window, in the foreground. In this way, you can monitor the output from the bootstrap and startup processes of the system's components. Furthermore, you can continue to monitor the system's components once they are up and running, and receiving traffic. Yes, that is twelve terminal windows…
There is a required startup order for the components. For example, Spring Cloud Config Server needs to start before the other components that rely on it for configuration. Netflix's Eureka needs to start before the Reservation Client and ReservationServices, so they can register with Eureka on startup. Similarly, Zipkin needs to be started in its Docker container before the Reservation Client and Services, so Spring Cloud Sleuth can start sending traces. Redis needs to be started in its Docker container before Spring Cloud Stream tries to create the message queue. All instances of the Reservation Service needs to start before the Reservation Client. Once every component is started, the Reservation Data Seeder needs to be run once to create initial data in H2. For best results, follow the instructions below. Let each component start completely, before starting the next component.
Both Zipkin and Redis run in Docker containers. Redis runs in a single container. Zipkin's four separate components run in four separate containers. Be advised, Zipkin seems to have trouble successfully starting all four of its components on a consistent basis. I believe it's a race condition caused by Docker Compose simultaneously starting the four Docker containers, ignoring a proper startup order. More than half of the time, I have to stop Zipkin and rerun the docker command to get Zipkin to start without any errors.
If you've followed the instructions above, you should see the following Docker images and Docker containers installed and running in your local environment.
Spring Cloud Config Server
At the center of the Reservation system is Spring Cloud Config. Configuration, typically found in the
application.properties file, for the Reservation Services, Reservation Client, Reservation Data Seeder, Eureka Service, and Hystix Dashboard, has been externalized with Spring Cloud Config.
Each component has a
bootstrap.properties file, which modifies its startup behavior during the bootstrap phase of an application context. Each
bootstrap.properties file contains the component's name and the address of the Spring Cloud Config Server. Components retrieve their configuration from the Spring Cloud Config Server at runtime. Below, is an example of the Reservation Client's
bootstrap.properties file.
Spring Cloud Config Git Repo
In the presentation, as in this post, the Spring Cloud Config Server is backed by a locally cloned Git repository, the Spring Cloud Config Git Repo. The Spring Cloud Config Server's
application.properties file contains the address of the Git repository. Each properties file within the Git repository corresponds to a system component. Below, is an example of the
reservation-client.properties file, from the Spring Cloud Config Git Repo.
As shown in the original presentation, the configuration files can be viewed using HTTP endpoints of the Spring Cloud Config Server. To view the Reservation Service's configuration stored in the Spring Cloud Config Git Repo, issue an HTTP GET request to
http://localhost:8888/reservation-service/master. The master URI refers to the Git repo branch in which the configuration resides. This will return the configuration, in the response body, as JSON:
In a real Production environment, the Spring Cloud Config Server would be backed by a highly-available Git Server or GitHub repository.
Reservation Service
The Reservation Service is the core component in the presentation's example. The Reservation Service is a stand-alone Spring Boot application. By implementing Spring Data REST and Spring HATEOAS, Spring automatically creates REST representations from the
Reservation JPA Entity class of the Reservation Service. There is no need to write a Spring Rest Controller and explicitly code each endpoint.
Spring HATEOAS allows us to interact with the Reservation Entity, using HTTP methods, such as GET and POST. These endpoints, along with all addressable endpoints, are displayed in the terminal output when a Spring Boot application starts. For example, we can use an HTTP GET request to call the
reservations/{id} endpoint, such as:
The Reservation Service also makes use of the Spring RepositoryRestResource annotation. By annotating the
RepositoryReservation Interface, which extends
JpaRepository, we can customize export mapping and relative paths of the Reservation JPA Entity class. As shown below, the
RepositoryReservation Interface contains the
findByReservationName method signature, annotated with
/by-name endpoint, which accepts the
rn input parameter.
Calling the
findByReservationName method, we can search for a particular reservation by using an HTTP GET request to call the
reservations/search/by-name?rn={reservationName} endpoint.
Reservation Client
Querying the Reservation Service directly is possible, however, is not the recommended. Instead, the presentation suggests using the Reservation Client as a proxy to the Reservation Service. The presentation offers three examples of using the Reservation Client as a proxy.
The first demonstration of the Reservation Client uses the
/message endpoint on the Reservation Client to return a string from the Reservation Service. The message example has been modified to include two new endpoints on the Reservation Client. The first endpoint,
/reservations/client-message, returns a message directly from the Reservation Client. The second endpoint,
/reservations/service-message, returns a message indirectly from the Reservation Service. To retrieve the message from the Reservation Service, the Reservation Client sends a request to the endpoint Reservation Service's
/message endpoint.
To retrieve both messages, send separate HTTP GET requests to each endpoint:
The second demonstration of the Reservation Client uses a Data Transfer Object (DTO). Calling the Reservation Client's
reservations/names endpoint, invokes the
getReservationNames method. This method, in turn, calls the Reservation Service's
/reservations endpoint. The response object returned from the Reservation Service, a JSON array of reservation records, is deserialized and mapped to the Reservation Client's Reservation DTO. Finally, the method returns a collection of strings, representing just the names from the reservations.
To retrieve the collection of reservation names, an HTTP GET request is sent to the
/reservations/names endpoint:
Spring Cloud Stream
One of the more interesting technologies in the presentation is Spring's Spring Cloud Stream. The Spring website describes Spring Cloud Stream as a project that allows users to develop and run messaging microservices using Spring Integration. In other words, it provides native Spring messaging capabilities, backed by a choice of message buses, including Redis, RabbitMQ, and Apache Kafka, to Spring Boot applications.
A detailed explanation of Spring Cloud Stream would take an entire post. The best technical demonstration I have found is the presentation, Message Driven Microservices in the Cloud, by speakers Dr. David Syer and Dr. Mark Pollack, given in January 2016, also at SpringOne2GX 2015.
In the presentation, a new reservation is submitted via an HTTP POST to the
acceptNewReservations method of the Reservation Client. The method, in turn, builds (aka produces) a message, containing the new reservation, and publishes that message to the
queue.reservation queue.
queue.reservation queue is located in Redis, which is running inside a Docker container. To view the messages being published to the queue in real-time, use the
redis-cli, with the
monitor command, from within the Redis Docker container. Below is an example of tests messages pushed (
LPUSH) to the reservations queue from the Reservation Client.
The published messages are consumed by subscribers to the
reservation queue. In this example, the consumer is the Reservation Service. The Reservation Service's
acceptNewReservation method processes the message and saves the new reservation to the H2 database. In Spring Cloud Stream terms, the Reservation Client is the
Netflix Eureka
Netflix's Eureka, in combination with Netflix's Zuul and Ribbon, provide the ability to scale the Reservation Service horizontally, and to load balance those instances. By using the
@EnableEurekaClient annotation on the Reservation Client and Reservation Services, each instance will automatically register with Eureka on startup, as shown in the Eureka Web UI, below.
The names of the registered instances are in three parts: the address of the host on which the instance is running, followed by the value of the
spring.application.name property of the instance's
bootstrap.properties file, and finally, the port number the instance is running on. Eureka displays each instance's status, along with additional AWS information, if you are running on AWS, as Netflix does.
According to Spring in their informative post, Spring Cloud, service discovery is one of the key tenets of a microservice based architecture. Trying to hand-configure each client, or to rely on convention over configuration, can be difficult to do and is brittle. Eureka is the Netflix Service Discovery Server and Client. A client (Spring Boot application), registers with Eureka, providing metadata about itself. Eureka then receives heartbeat messages from each instance. If the heartbeat fails over a configurable timetable, the instance is normally removed from the registry.
The Reservation Client application is also annotated with
@EnableZuulProxy. Adding this annotation pulls in Spring Cloud's embedded Zuul proxy. Again, according to Spring, the proxy is used by front-end applications to proxy calls to one or more back-end services, avoiding the need to manage CORS and authentication concerns independently for all the backends. In the presentation and this post, the front end is the Reservation Client and the back end is the Reservation Service.
In the code snippet below from the
ReservationApiGatewayRestController, note the URL of the endpoint requested in the
getReservationNames method. Instead of directly calling
http://localhost:8000/reservations, the method calls
http://reservation-service/reservations. The
reservation-service segment of the URL is the registered name of the service in Eureka and contained in the Reservation Service's
bootstrap.properties file.
In the following abridged output from the Reservation Client, you can clearly see the interaction of Zuul, Ribbon, Eureka, and Spring Cloud Config. Note the Client application has successfully registering itself with Eureka, along with the Reservation Client's status. Also, note Zuul mapping the Reservation Service's URL path.
Load Balancing
One shortcoming of the original presentation was true load balancing. With only a single instance of the Reservation Service in the original presentation, there is nothing to load balance; it's more of a reverse proxy example. To demonstrate load balancing, we need to spin up additional instances of the Reservation Service. Following the post's component start-up instructions, we should have three instances of the Reservation Service running, on ports
8001, and
8002, each in separate terminal windows.
To confirm the three instances of the Reservation Service were successfully registered with Eureka, review the output from the Eureka Server terminal window. The output should show three instances of the Reservation Service registering on startup, in addition to the Reservation Client.
Viewing Eureka's web console, we should observe three members in the pool of Reservation Services.
Lastly, looking at the terminal output of the Reservation Client, we should see three instances of the Reservation Service being returned by Ribbon (aka the DynamicServerListLoadBalancer).
http://localhost:8050/reservations/names, Ribbon forwards the request to one of the three Reservation Service instances registered with Eureka. By default, Ribbon uses a round-robin load-balancing strategy to select an instance from the pool of available Reservation Services.
H2 Server
The original presentation's Reservation Service used an embedded instance of H2. To scale out the Reservation Service, we need a common database for multiple instances to share. Otherwise, queries would return different results, specific to the particular instance of Reservation Service chosen by the load-balancer. To solve this, the original presentation's embedded version of H2 has been replaced with the TCP Server client/server version of H2.
Thanks to more Spring magic, the only change we need to make to the original presentation's code is a few additional properties added to the Reservation Service's
reservation-service.properties file. This changes H2 from the embedded version to the TCP Server version.
Reservation Data Seeder
In the original presentation, the Reservation Service created several sample reservation records in its embedded H2 database on startup. Since we now have multiple instances of the Reservation Service running, the sample data creation task has been moved from the Reservation Service to the new Reservation Data Seeder. The Reservation Service only now validates the H2 database schema on startup. The Reservation Data Seeder now updates the schema based on its entities. This also means the seed data will be persisted across restarts of the Reservation Service, unlike in the original configuration.
Running the Reservation Data Seeder once will create several reservation records into the H2 database. To confirm the H2 Server is running and the initial reservation records were created by the Reservation Data Seeder, point your web browser to the H2 login page at
http://192.168.99.1:6889. and log in using the credentials in the
reservation-service.properties file.
The H2 Console should contain the RESERVATION table, which holds the reservation sample records.
Spring Cloud Sleuth and Twitter's Zipkin
According to the project description, "Spring Cloud Sleuth implements a distributed tracing solution for Spring Cloud. All your interactions with external systems should be instrumented automatically. You can capture data simply in logs, or by sending it to a remote collector service." In our case, that remote collector service is Zipkin.
Zipkin describes itself as, "a distributed tracing system. It helps gather timing data needed to troubleshoot latency problems in microservice architectures. It manages both the collection and lookup of this data through a Collector and a Query service." Zipkin provides critical insights into how microservices perform in a distributed system.
In the presentation, as in this post, the Reservation Client's main
ReservationClientApplication class contains the
alwaysSampler bean, which returns a new instance of
org.springframework.cloud.sleuth.sampler.AlwaysSampler. As long as Spring Cloud Sleuth is on the classpath and you have added
alwaysSampler bean, the Reservation Client will automatically generate trace data.
Sending a request to the Reservation Client's
service/message endpoint (
http://localhost:8050/reservations/service-message,), will generate a trace, composed of spans. in this case, the spans are individual segments of the HTTP request/response lifecycle. Traces are sent by Sleuth to Zipkin, to be collected. According to Spring, if
spring-cloud-sleuth-zipkin is available, then the application will generate and collect Zipkin-compatible traces using Brave). By default, it sends them via Apache Thrift to a Zipkin collector service on port
Zipkin's web-browser interface, running on port
8080, allows us to view traces and drill down into individual spans.
Zipkin contains fine-grain details about each span within a trace, as shown below.
Correlation IDs
Note the
x-trace-id and
x-span-id in the request header, shown below. Sleuth injects the trace and span IDs to the SLF4J MDC (Simple Logging Facade for Java – Mapped Diagnostic Context). According to Spring, IDs provides the ability to extract all the logs from a given trace or span in a log aggregator. The use of correlation IDs and log aggregation are essential for monitoring and supporting a microservice architecture.
Hystix and Hystrix Dashboard
The last major technology highlighted in the presentation is Netflix's Hystrix. According to Netflix, "Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services, and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable." Hystrix is essential, it protects applications from cascading dependency failures, an issue common to complex distributed architectures, with multiple dependency chains. According to Netflix, Hystrix uses multiple isolation techniques, such as bulkhead, swimlane, and circuit breaker patterns, to limit the impact of any one dependency on the entire system.
The presentation demonstrates one of the simpler capabilities of Hystrix, fallback. The
getReservationNames method is decorated with the
@HystrixCommand annotation. This annotation contains the
fallbackMethod. According to Netflix, a graceful degradation of a method is provided by adding a fallback method. Hystrix will call to obtain a default value or values, in case the main command fails. In the presentation's example, the Reservation Service, a direct dependency of the Reservation Client, has failed. The Reservation Service failure causes the failure of the Reservation Client.
In the presentation's example, the Reservation Service, a direct dependency of the Reservation Client, has failed. The Reservation Service failure causes the failure of the Reservation Client's
getReservationNames method to return a collection of reservation names. Hystrix redirects the application to the
getReservationNameFallback method. Instead of returning a collection of reservation names, the
getReservationNameFallback returns an empty collection, as opposed to an error message to the client.
A more relevant example involves Netflix movie recommendation service. In the event a failure of the recommendation service's method to return a collection of personalized list of movie recommendations to a customer, Hystrix fallbacks to a method that returns a generic list of the most popular movies to the customer. Netflix has determined that, in the event of a failure of their recommendation service, falling back to a generic list of movies is better than returning no movies at all.
The Hystrix Dashboard is a tool, available with Hystrix, to visualize the current state of Hystrix instrumented methods. Although visually simplistic, the dashboard effectively presents the health of calls to external systems, which are wrapped in a
HystrixCommand or
The Hystrix dashboard is a visual representation of the Hystrix Stream. This stream is a live feed of data sent by the Hystrix instrumented application, in this case, the Reservation Client. For a single Hystrix application, such as the Reservation Client, the feed requested from the application's
hystrix.stream endpoint is
In the post, as in the presentation, hitting the Reservation Client with a volume of requests, we observe normal activity in Hystrix Dashboard. All three instances of the Reservation Service are running and returning the collection of reservations from H2, to the Reservation Client.
If all three instances of the Reservation Service fail or the maximum latency is exceeded, the Reservation Client falls back to returning an empty collection in the response body. In the example below, 15 requests, representing 100% of the current traffic, to the
getReservationNames method failed and subsequently fell back to return an empty collection. Hystrix succeeded in helping the application gracefully fall back to an alternate response.
It's easy to see how Spring Cloud and Netflix's technologies are easily combined to create a performant, horizontally scalable, reliable system. With the addition of a few missing components, such metrics monitoring and log aggregation, this example could easily be scaled up to support a production-grade microservices-based, enterprise software platform.