text: Not Intuitive
  As we’ve discussed, moving from a web interface (with specific forms, help buttons, and UI elements that guide you in setting up jobs) to creating scripts, is not intuitive. One key part of the original UI job pages was the separation into sections, such as post-build processing, that guided users through the various phases. When moving to scripts, the elements for the different phases are available, but it’s not clear how to structure or order them out of the box. Worse, some familiar processes don’t have corresponding constructs in the nondeclarative DSL.
  Docker and the tools Directive
  The tools directive does not work on Docker or Dockerfile agents. The recommended practice is to use an image with the tools already installed.
  In addition to the tools directive to allow us to access the globally defined tools, we also have the options directive to allow us to set project-level options.
  Administering Credentials
  Administration for credentials can be done through the Configure Credentials interface, accessible under the Manage Jenkins menu. The options on this screen allow a Jenkins user to:
  Select which credential providers will be available to Jenkins to resolve credentials.
  Select the types of credentials that can be resolved and configured.
  Specify the types of credentials that can be included or excluded for a specific provider.
  Using Third-Party Libraries
  Shared libraries can also make use of third-party libraries using the @Grab annotation. The @Grab annotation is provided through the Grape dependency manager that is built into Groovy. It allows you to pull in any dependency from a Maven repository, such as Maven Central. This can be done from trusted libraries, but does not work in the Groovy Sandbox.
  Here’s an example function using @Grab to pull in an Apache Commons dependency. In a similar vein to our other examples, we’re using a “stopwatch” function here to time how long execution of a command takes. The routine is written entirely with Groovy code (as noted earlier, libraries have access to all Groovy constructs):
  // vars/timedCommand5
  @Grab('org.apache.commons:commons-lang3:3.4+')
  import org.apache.commons.lang.time.StopWatch
  def call(String cmdToRun) {
  def sw = new StopWatch()
  def proc = "$cmdToRun".execute()
  sw.start()
  proc.waitFor()
  sw.stop()
  println("The process took ${(sw.getTime()/1000).toString()} seconds.\n")
  }
  Assuming this code has been pushed into the shared library area that is being implicitly loaded, the code could be invoked like this from a pipeline script:
  node {
  timedCommand5("sleep 10")
  }
  Other than the downloads for the libraries, the output would look something like this:
  [Pipeline] node
  Running on worker in /home/jenkins2/worker_node1/workspace/mypipe11
  [Pipeline] {
  [Pipeline] echo
  The process took 10.009 seconds.
  [Pipeline] }
  [Pipeline] // node
  [Pipeline] End of Pipeline
  Finished: SUCCESS
  ws Versus dir
  If you are not concerned with the locking aspect, you can just use the dir step to switch the directory you are executing in.
  EnvInject
  A popular plugin for use with Freestyle jobs is EnvInject. This plugin allows injecting a wide set of values as environment variables for use in a Freestyle project. Unfortunately, at the time of this writing, this plugin is not compatible with pipelines.
  Docker Pipeline Global Variable
  The third way to work with Docker through a pipeline is to use the methods associated with the Jenkins docker global variable. To have this available, you need to have the Docker Pipeline plugin installed. We have mentioned global variables elsewhere in the book before, but a little explanation is still in order.
  Global Configuration
  Figure 12-11 shows an example of the global configuration for a SonarQube server. (This is done on the Configure System page.)
  [image: ]
  Figure 12-11. SonarQube global configuration in Jenkins
  Note the option to “Enable injection of SonarQube server configuration as build environment variables.” This may be needed in cases where tools like Maven incorporated environment variables as part of their scanning command. In the job definition, there is a corresponding option to prepare the environment.
  We also need to have the SonarQube scanner installed and configured. Figure 12-12 shows an example of this. (Note that it is configured on the Global Tool Configuration screen.)
  [image: ]
  Figure 12-12. SonarQube scanner global configuration
  Once we have the application installed and running, the Jenkins plugin installed, and the global system and tool configuration done, we are ready to make use of SonarQube in our pipeline for the analysis. Consistent with our migration/conversion theme throughout the book, we’ll look first at how it would have been used in a Freestyle project.
  Foreword
  Preface
  How to Use This Book
  Conventions Used in This Book
  Using Code Examples
  O’Reilly Safari
  How to Contact Us
  Acknowledgments
  1. Introducing Jenkins 2
  What Is Jenkins 2?
  The Jenkinsfile
  Declarative Pipelines
  Blue Ocean Interface
  New Job Types in Jenkins 2
  Reasons for the Shift
  DevOps Movement
  Assembling Pipelines
  Resumability
  Configurability
  Sharing Workspaces
  Specialized Knowledge
  Access to Logic
  Pipeline Source Management
  Competition
  Meeting the Challenges
  Compatibility
  Pipeline Compatibility
  Plugin Compatibility
  Checking Compatibility
  Summary
  2. The Foundations
  Syntax: Scripted Pipelines Versus Declarative Pipelines
  Choosing Between Scripted and Declarative Syntax
  Systems: Masters, Nodes, Agents, and Executors
  Master
  Node
  Agent
  Executor
  Creating Nodes
  Structure: Working with the Jenkins DSL
  node
  stage
  steps
  Supporting Environment: Developing a Pipeline Script
  Starting a Pipeline Project
  The Editor
  Working with the Snippet Generator
  Running a Pipeline
  Replay
  Summary
  3. Pipeline Execution Flow
  Triggering Jobs
  Build After Other Projects Are Built
  Build Periodically
  GitHub Hook Trigger for GitSCM Polling
  Poll SCM
  Quiet Period
  Trigger Builds Remotely
  User Input
  input
  Parameters
  Return Values from Multiple Input Parameters
  Parameters and Declarative Pipelines
  Flow Control Options
  timeout
  retry
  sleep
  waitUntil
  Dealing with Concurrency
  Locking Resources with the lock Step
  Controlling Concurrent Builds with Milestones
  Restricting Concurrency in Multibranch Pipelines
  Running Tasks in Parallel
  Conditional Execution
  Post-Processing
  Scripted Pipelines Post-Processing
  Declarative Pipelines and Post-Processing
  Summary
  4. Notifications and Reports
  Notifications
  Collaboration Services
  Reports
  Publishing HTML Reports
  Summary
  5. Access and Security
  Securing Jenkins
  Enabling Security
  Other Global Security Settings
  Credentials in Jenkins
  Credential Scopes
  Credential Domains
  Credential Providers
  Credential Stores
  Administering Credentials
  Selecting Credential Providers
  Selecting Credential Types
  Specifying Credential Types by Provider
  Creating and Managing Credentials
  Context Links
  Adding a New Domain and Credential
  Using the New Domain and Credential
  Advanced Credentials: Role-Based Access
  Basic Use
  Manage Roles
  Assign Roles
  Role Strategy Macros
  Working with Credentials in the Pipeline
  Username and Password
  SSH Keys
  Token Credentials
  Controlling Script Security
  Script Checking
  Script Approval
  Groovy Sandboxing
  Using Jenkins Credentials with Vault
  Approach
  Setup
  Creating a Policy
  Authentication
  Using Vault in Jenkins
  Summary
  6. Extending Your Pipeline
  Trusted Versus Untrusted Libraries
  Internal Versus External Libraries
  Internal Libraries
  External Libraries
  Getting a Library from the Source Repository
  Modern SCM
  Legacy SCM
  Using Libraries in Your Pipeline Script
  Automatic Downloading of Libraries from Source Control
  Library Scope Within Jenkins Items
  Library Structure
  Sample Library Routine
  Structure of Shared Library Code
  Using Third-Party Libraries
  Replaying External Code and Libraries
  A Closer Look at Trusted Versus Untrusted Code
  Summary
  7. Declarative Pipelines
  Motivation
  Not Intuitive
  Getting Groovy
  Additional Assembly Required
  The Structure
  Block
  Section
  Directives
  Steps
  Conditionals
  The Building Blocks
  pipeline
  agent
  environment
  tools
  options
  triggers
  parameters
  libraries
  stages
  Dealing with Nondeclarative Code
  Check Your Plugins
  Create a Shared Library
  Place Code Outside of the Pipeline Block
  The script Statement
  Using parallel in a Stage
  Script Checking and Error Reporting
  Declarative Pipelines and the Blue Ocean Interface
  Summary
  8. Understanding Project Types
  Common Project Options
  General
  Source Code Management
  Build Triggers
  Build Environment
  Build
  Post-Build Actions
  Types of Projects
  Freestyle Projects
  The Maven Project Type
  The Pipeline Project Type
  The External Job Project Type
  The Multiconfiguration Project Type
  Ivy Projects
  Folders
  Multibranch Pipeline Projects
  GitHub Organization Projects
  Bitbucket Team/Project Projects
  Summary
  9. The Blue Ocean Interface
  Part 1: Managing Existing Pipelines
  The Dashboard
  The Project-Specific Page
  The Run Page
  Part 2: Working with the Blue Ocean Editor
  Creating a New Pipeline Without an Existing Jenkinsfile
  Working in the Editor
  Editing an Existing Pipeline
  Importing and Editing Existing Pipelines
  Working with Pipelines from Non-GitHub Repositories
  Summary
  10. Conversions
  Common Preparation
  Logic and Accuracy
  Project Type
  Systems
  Access
  Global Configuration
  Plugins
  Shared Libraries
  Converting a Freestyle Pipeline to a Scripted Pipeline
  Source
  Compile
  Unit Tests
  Integration Testing
  Migrating the Next Parts of the Pipeline
  Converting from a Jenkins Pipeline Project to a Jenkinsfile
  Approach
  Final Steps
  Converting from a Scripted Pipeline to a Declarative Pipeline
  Sample Pipeline
  The Conversion
  Completed Conversion
  General Guidance for Conversions
  Summary
  11. Integration with the OS (Shells, Workspaces, Environments, and Files)
  Using Shell Steps
  The sh Step
  The bat Step
  The powershell Step
  Working with Environment Variables
  The withEnv Step
  Working with Workspaces
  Creating a Custom Workspace
  Cleaning a Workspace
  File and Directory Steps
  Working with Files
  Working with Directories
  Doing More with Files and Directories
  Summary
  12. Integrating Analysis Tools
  SonarQube Survey
  Working with Individual Rules
  Quality Gates and Profiles
  The Scanner
  Using SonarQube with Jenkins
  Global Configuration
  Using SonarQube in a Freestyle Project
  Using SonarQube in a Pipeline Project
  Leveraging the Outcome of the SonarQube Analysis
  SonarQube Integration Output with Jenkins
  Code Coverage: Integration with JaCoCo
  About JaCoCo
  Integrating JaCoCo with the Pipeline
  JaCoCo Output Integration with Jenkins
  Summary
  13. Integrating Artifact Management
  Publishing and Retrieving Artifacts
  Setup and Global Configuration
  Using Artifactory in a Scripted Pipeline
  Performing Other Tasks
  Downloading Specific Files to Specific Locations
  Uploading Specific Files to Specific Locations
  Setting Build Retention Policies
  Build Promotion
  Integration with a Declarative Pipeline
  Artifactory Integration with Jenkins Output
  Archiving Artifacts and Fingerprinting
  Summary
  14. Integrating Containers
  Configured as a Cloud
  Global Configuration
  Using Docker Images as Agents
  Using Cloud Images in a Pipeline
  Agent Created on the Fly for a Declarative Pipeline
  Docker Pipeline Global Variable
  Global Variables
  Docker Application Global Variable Methods
  Docker Image Global Variable Methods
  Docker Container Global Variable Methods
  Running Docker via the Shell
  Summary
  15. Other Interfaces
  Using the Command-Line Interface
  Using the Direct SSH Interface
  Using the CLI Client
  Using the Jenkins REST API
  Filtering Results
  Initiating Builds
  Using the Script Console
  Summary
  16. Troubleshooting
  Diving into Pipeline Steps
  Dealing with Serialization Errors
  Continuous Passing Style
  Serializing Pipelines
  NotSerializableException
  Handling Nonserializable Errors
  Identifying the Line in Your Script that Caused an Error
  Handling Exceptions in a Pipeline
  Using Nondeclarative Code Within a Declarative Pipeline
  Unapproved Code (Script and Method Approval)
  Unsupported Operations
  System Logs
  Timestamps
  Pipeline Durability Settings
  Summary
  Index
  Unapproved Code (Script and Method Approval)
  Since pipelines make the ability to run any arbitrary script a key part of Jenkins, safeguards are in place to make sure only approved scripts and methods are used.
  At the highest access level, Jenkins administrators can create and run any scripts. For nonadministrators, Jenkins includes two methods for script approval: manual via administrator and automatic via the Groovy Sandbox environment for certain cases.
  The Groovy Sandbox contains a whitelist of methods that are approved for nonadministrators to use in scripts. If the script is run in the Sandbox environment, then as long as the methods it uses are whitelisted, it is allowed to proceed without manual approval.
  If the script is run by a nonadministrator and it is not run in the Sandbox environment, then it will need to be manually approved by an administrator before it can be executed.
  Even if the script is run in the Sandbox environment, if it makes calls to methods not currently on the whitelist, those calls will need to be manually approved by an administrator before the script can be executed.
  Here’s an example error message caused by trying to use an unapproved jsonSlurper method:
  org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: unclassified method groovy.json.JsonSlurper parseText java.io.File
  at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:113)
  Here’s another example error:
  org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use new java.io.File java.lang.String
  Jenkins will then post an automatic request for approval via the In-process Script Approval function. This function allows those with proper permissions to approve calling the function.
  “Controlling Script Security” contains more details, and describes how the approval process is implemented.
  Even if your code is approved and seems perfectly legal, you may still run into certain cases where code is unsupported. See the next section for an example.
  post Section of a Declarative Pipeline
  Note that, as described in Chapter 7, the post section supports separate processing for build statuses such as success, failure, etc. In this case, we are simply using the always clause for a generic demonstration.
  These pipelines will produce an email like the following if there is a failure:
  ---------------------------- Original Message ----------------------------
  Subject: Status of pipeline: pipeline2 #1
  From: jenkins-demo@gmail.com
  Date: Tue, August 15, 2017 9:33 pm
  To: bcl@nclasters.org
  -------------------------------------------------------------------------- has result FAILURE
  If there is a successful build, this will look the same except FAILURE will be replaced with SUCCESS.
  While the built-in functionality covers the basic email needs, there may be times when you need or want to further customize and control the emails that Jenkins sends. The Email Extension plugin provides many additional options for expanding how emails are handled, but it also comes with trade-offs when using it in a pipeline environment. We’ll dive into it next.
  Extended email notifications
  In addition to the basic email functionality, there is also an Extended Email (email-ext) plugin that adds numerous additional options and levels of control for sending email through Jenkins. It contains a similar general mail configuration section similar to the mail plugin, but also adds functionality in three areas:
  Content
  It’s possible to dynamically modify the contents of an email notification’s subject and body.
  Recipients
  You can define which user roles should receive an email notification.
  Triggers
  You can specify what conditions should initiate sending an email notification. (Note that these do not currently apply to pipelines.)
  We’ll look at each of these areas in more detail next and see how to incorporate them where applicable.
  Global configuration
  The email-ext plugin requires some global configuration before being used in Pipeline jobs. Most of this is the same as the configuration we did for the basic email functionality (see Figure 4-4).
  [image: ]
  Figure 4-4. General configuration for extended emails
  A few new fields deserve further explanation. They include:
  Use List-ID Email Header
  Selecting this option allows you to apply a list-id header to all emails. As the help suggests, this can be useful with filtering and also avoiding autoresponders. The help contains example formats.
  Add ‘Precedence: bulk’ Email Header
  This option adds a header to emails from Jenkins. Based on a standard used by mail systems, this option should eliminate or cut down on autoresponses being sent back to Jenkins.
  Reply To List
  This isn’t a new one, but note that we can provide a comma-separated list of users/addresses.
  Emergency reroute
  If this field is filled in, all Jenkins emails will be sent to just that or those recipients. This could be useful for temporarily not allowing Jenkins to send out wider emails if there is an issue that warrants that.
  Excluded Recipients
  As the name implies, this can exclude (filter out) any email addresses from the list generated by other functionality in this plugin.
  Next, we’ll look at the functionality in the plugin that allows us to set the default elements of emails.
  Content
  Still in the global configuration, we have a set of fields that are intended to allow us to dynamically generate/modify the contents of the email notices sent out from Jenkins. Figure 4-5 shows these fields.
  [image: ]
  Figure 4-5. Default global content settings for extended email
  Timestamps
  We mentioned the Timestamper plugin briefly in Chapter 7. Its job is very simple — to add timestamps to the console output for a job. Doing this can help you figure out where your pipelines are getting stuck, consuming large amounts of system resources, or skipping processing on events they should be handling.
  With the plugin installed, turning on timestamps in a pipeline is simple. For Scripted Pipelines, you can just place a timestamps block around code that you want to track, as in:
  timestamps {
  <code/steps to time>
  }
  For Declarative Pipelines, you just add it in the options section of your pipeline, as in:
  options { timestamps() }
  The system clock and elapsed time formats can be configured in the Timestamper section of the Configure System screen. The kind of time information you see in the console log can be updated dynamically (when you are in the console log) by selecting the appropriate box in the control near the top (Figure 16-13).
  [image: ]
  Figure 16-13. Timestamper options in console output
  One of the things that can lead to increased time used in parts of your pipeline is how often data is serialized to disk. With recent versions of Jenkins, there is now a setting to control that: the pipeline speed/durability setting.
  Progress Indicators
  You may have noticed that as you’re going through these steps, progress is tracked down the side on a line with the familiar halos (the circular icons) on it. As steps are in progress, the halos are blue and hollow. As you complete steps, the halos become green, filled in, and checked off. This is intentional, as it is the same representation that Blue Ocean uses for presenting pipeline stages.
  After selecting an organization, you are presented with a list of repositories within that organization (Figure 9-40). There is a search field to filter the list if desired. From the list, you can choose a repository and then click the Create Pipeline button to get started.
  [image: ]
  Figure 9-40. Selecting a repository from the GitHub organization
  In our example, we’re going to choose the “blue-ocean-demo” repository. If the repository you choose already has a Jenkinsfile in it, then Jenkins will automatically attempt to create an instance of the pipeline, defined in that Jenkinsfile, and run it for you. In this case, there is no initial Jenkinsfile in this project, so Jenkins just reports that and provides a button to create a new pipeline (Figure 9-41).
  [image: ]
  Figure 9-41. No Jenkinsfiles found
  Clicking the Create Pipeline button takes us into the Blue Ocean pipeline editor, which we’ll discuss next.
  Visual Editor
  With the advent of the new Blue Ocean interface and Declarative Pipelines, a visual pipeline editor is available in Jenkins. The Blue Ocean interface and editor are discussed in Chapter 9.
  Chapter 5. Access and Security
  Being able to create pipelines-as-code offers enormous potential and flexibility. In Scripted Pipelines, calls to any Groovy construct or Jenkins functionality or external method can be keyed into the pipeline script. However, that also significantly increases the ability to accidentally or intentionally do something within the code for a pipeline that shouldn’t be done. So, security has to be a first-class concern — and a first-class feature — for both pipelines and the Jenkins environment they are created and run in.
  In this chapter, we’ll survey the different ways that Jenkins has for controlling access and security. We’ll first look at the overall security options, then we’ll survey the traditional credentials mechanisms that Jenkins offers and how to use those in pipelines.
  After that, we’ll do a deeper dive into the advanced functionality available via the Role-Based Access Control (RBAC) plugin. We’ll then explore how Jenkins can integrate with Vault, a modern approach to storing credentials with a limited lifetime.
  Finally, we’ll see what new features Jenkins 2 provides for ensuring that the steps in a pipeline have only the appropriate access and are executed in an approved context.
  Let’s start off by looking at the most basic options for securing Jenkins once it’s installed.
  Setting Build Retention Policies
  Setting build retention policies is done with properties related to the buildInfo object. First, we have to define a buildInfo object, as described earlier:
  def buildInfo = Artifactory.newBuildInfo()
  Then we can set appropriate properties, either as separate statements or in a combined form like this:
  buildInfo.retention maxBuilds: 3, maxDays, 5
  The Multiconfiguration Project Type
  This type of project is designed to simplify running a set of project builds that only differ in terms of parameters. For example, suppose you needed to run a test build against a set of five different browsers (IE, Firefox, Safari, etc.) and across a set of five different operating systems (Debian, Centos, Windows, etc.).
  Without the Multiconfiguration project type, you would need 25 jobs (5 browsers tested against each of 5 operating systems) to accomplish this. With the Multiconfiguration type, you only need one job that does the work of executing the various possible combinations for you.
  The way this works is that you define your base job to do whatever you need to do based on parameters that represent each of the different “axes” you are using. For the example just mentioned, one axis would be the set of browsers and the second would be the set of operating systems.
  Like the other project types we’ve discussed, the Multiconfiguration project has the common setup, environment, build, post-processing, and other configuration. sections. But it also includes a separate new Configuration Matrix section. This is where you define the axes that you want to include in the job. There are three types of axes that you can create. Each one takes a name that will become an environment variable (which you can use in the build step) and a definition. The types of axes that can be added to the Configuration Matrix are:
  Slaves
  This type of axis definition allows you to specify either a node’s name or a label on a node to include in the set of nodes to iterate over. (As discussed elsewhere in the book, a label is simply a tag or identifying name that we can attach to one or more nodes. Then we can select one or multiple nodes by specifying a label that they have.)
  Label expression
  This type of axis definition allows you to use advanced syntax to choose which set of nodes to include. For example, you can combine node labels and operators, as in label1&&label2, to indicate that only a node having both labels is eligible to be included.
  User-defined axis
  This type allows you to specify a set of items as values to iterate over in building the set of jobs.
  Multiconfiguration example
  Let’s consider a use case for this type of project. We have some jobs to build to create web pages for a set of company job families in each of several different regions (where each region has a dedicated node).
  In our setup, we have three worker nodes available, with various labels, defined like as in Table 8-1.
  Table 8-1. Available worker nodes
  Name
  Labels
  worker_node1
  northwest open region1
  worker_node2
  northeast open region2
  worker_node3
  southwest restricted region3
  The regions (and thus the one-to-one mapping to the nodes) make up one of the axes our job will use. For the other, we will use a set of company job families defined as development, infrastructure, management, and testing. We can now define the two axes in our Multiconfiguration project’s configuration as shown in Figure 8-41.
  [image: ]
  Figure 8-41. Defining the axes in the project’s configuration
  Note that within the Slaves list, we can select systems based on Labels or Individual Nodes. The latter term just means selecting them by their name (e.g., worker_node1).
  With our axes configured, we can set up our build step to use them. The name supplied when configuring each axis becomes an environment variable we can reference in our build step. For example, if we wanted to print out a message for each combination when the build ran, we could use a simple echo statement like the one in Figure 8-42 in our build step.
  [image: ]
  Figure 8-42. Build step using names of axes via environment variables
  We can then run our build and Jenkins will automatically create the appropriate matrix of jobs to run based on the allowed combinations of our axes (Figure 8-43).
  [image: ]
  Figure 8-43. Job matrix based on axes
  We can drill into any of the combinations by clicking on the blue ball in the matrix for the appropriate row and column. Figure 8-44 shows one example.
  [image: ]
  Figure 8-44. Drilling into results for a particular job
  From there, we can also drill into the console output for a particular run of a job in the matrix, as shown in Figure 8-45.
  [image: ]
  Figure 8-45. Console output for a particular job in the multiconfiguration matrix
  Note that you can have more than two axes, though at some point, having more axes may get prohibitive in terms of trying to navigate through the output.
  The Matrix Configuration section also includes a few additional options that may be helpful:
  Combination filter
  The default for a Multiconfiguration project is for Jenkins to build all combinations of values in the defined axes. If this is too many, or you need to limit which ones it builds, you can use this area to define filters to limit which combinations are built. An example might be:
  !(job_family=="management" && region=="northwest")
  to prevent running the job for management in the Northwest region. Note the use of the double equals sign here to check for equality. See the help for the step for more examples.
  Run each configuration sequentially
  This option tells Jenkins to build each possible combination one at a time (not in parallel). This might be needed to limit multiple jobs stepping on each other if using a shared resource, for example.
  Execute touchstone builds first
  This option allows you to specify a set of builds to run first as a sort of “sanity check.” Enabling this option brings up two additional fields. The first is for a combination filter (as discussed previously) to define which builds to run first. The second is to select the condition that those builds must match in order for the rest of the processing to continue (see Figure 8-46). Your choices for the second field are Stable or Unstable.
  [image: ]
  Figure 8-46. Configuring touchstone builds
  With the touchstone builds in place, the overall console output for the job would look something like Figure 8-47.
  [image: ]
  Figure 8-47. Overall console output for a Multiconfiguration job (with touchstone builds)
  Notice the various links in the console output that allow you to drill down to the output for various combination builds.
  Pipeline compatibility
  There is no direct correlation to the Multiconfiguration project encapsulated into a single pipeline step. However, if you are working in a Scripted Pipeline, you can use Groovy looping constructs to iterate across defined “axes” and create tasks that can then be executed in parallel. Based on of a CloudBees example, here’s corresponding pipeline code for the example in the previous section:
  def axisRegions = ["northwest","northeast","southwest"]
  def axisJobFamilies = ["developers","infrastructure",
  "management","testing"]
  def myTasks = [:]
  for(int i=0; i< axisRegions.size(); i++) {
  def axisRegionSetting = axisRegions[i]
  for(int j=0; j< axisJobFamilies.size(); j++) {
  def axisJobFamilySetting = axisJobFamilies[j]
  myTasks["${axisRegionSetting}/${axisJobFamilySetting}"] = {
  node(axisRegionSetting) {
  println "Running task on job family ${axisJobFamilySetting}
  for region ${axisRegionSetting}"
  }
  }
  }
  }
  stage ("BuildMatrix") {
  parallel myTasks
  }
  For more detail on how the parallel step works, see Chapter 3.
  GitHub Hook Trigger for GitSCM Polling
  A GitHub project configured as the source location in a Jenkins project can have a push hook (on the GitHub side) to trigger a build for the Jenkins project. When this is in place, a push to the repository causes the hook to fire and trigger Jenkins, which then invokes the Jenkins SCM polling functionality. So the SCM polling functionality has to be configured for this to work as well.
  Most of the initial work for this is in the setup for the hook side and in the source setup in the Jenkins project. More information is available on the Jenkins wiki.
  The syntax for setting the property in a Scripted Pipeline is as follows:
  properties([pipelineTriggers([githubPush()])])
  There currently isn’t a specific syntax for Declarative Pipelines.
  Working Around the Script Scope Limitation
  If you absolutely must have access to some value set within the script block outside of the scope of the script block, one option is to set an environment variable to the value within that block. You can then access the environment variable anywhere in your script. (Of course, this is not suitable if the value should not be accessible everywhere.) Here’s an example:
  pipeline {
  agent any
  stages {
  stage ('Artifactory') {
  steps {
  script {
  def server = Artifactory.server 'my-server-id'
  server.username = "my-username"
  env.SERVER_USERNAME = server.username
  }
  echo "${SERVER_USERNAME}"
  Use a shared library. You can encapsulate the nondeclarative code into a function in a shared library, and then load the library and call the function from our Declarative Pipeline. This is the preferred approach, because it does not involve putting any nondeclarative code in your pipeline script.
  Creating and using shared libraries is covered in detail in Chapter 6.
  Using a Class with vars
  As was done with the code in src, you could create a class to encapsulate the vars commands. However, doing so is problematic and not particularly beneficial.
  Automatic documentation references for global variables
  As mentioned previously, one of the types of files that can be in the vars section is a .txt file with the same name as the .groovy file containing code. This .txt file can be used for documentation on the operation and can be written in either Markdown or HTML. Figure 6-10 shows an example timedCommand.txt file. This file has corresponding user-facing documentation about the functions defined in our timedCommand.groovy file. The file is optional, but if created, it should be committed and pushed in the vars directory of the same shared library structure as the corresponding .groovy file.
  [image: ]
  Figure 6-10. Creating a timedCommand.txt file to correspond to our implementation file
  After the variable code has been loaded and executed in a successful run of a pipeline script, an entry for the variable will be added to the list of global variables in the pipeline syntax section (accessible via the Pipeline Syntax screen). Figure 6-11 shows how to get to that area through the interface.
  [image: ]
  Figure 6-11. Accessing the Global Variable Reference
  If you’re not familiar with the Global Variable Reference page, its purpose is to provide documentation on variables and their associated methods (see Figure 6-12).
  [image: ]
  Figure 6-12. The Global Variable Reference
  After we have a successful run of a job with our timedCommand variable, the contents of our timedCommand.txt file will be included in this page (Figure 6-13). This provides a convenient way of documenting any variables we add in the application.
  [image: ]
  Figure 6-13. Our timedCommand variable is included now in the Global Variable Reference
  Using global variables like steps
  You can create global variable definitions that act like steps in pipeline scripts. That is, they can be called like regular pipeline steps. The trick to this is to define the call method in the global variable’s definition. Let’s see what it would look like to do this for our timedCommand code. Since this is a slightly different version, we’ll refer to it as timedCommand2:
  // vars/timedCommand2
  def call (String cmd) {
  timestamps {
  cmdOutput = echo sh (script:"${cmd}", returnStdout:true).trim()
  }
  echo cmdOutput
  writeFile
  }
  We can use any valid pipeline DSL code in the body of the call. Let’s suppose that we decide we want to add code to write our output to a log file as well as printing it out. For this, we will need the writeFile sDSL statement. If we’re not clear on the syntax, we can use the pipeline syntax generator (aka Snippet Generator) to help us with that. Figure 6-14 shows using the Snippet Generator to determine the right format of the command for our purposes. (Notice that we just supply the variable names we intend to use in the individual fields.)
  [image: ]
  Figure 6-14. Using the Snippet Generator to get the correct syntax for the writeFile DSL call
  Now we can add the writeFile command to our function and pass in a value for the location:
  // vars/timedCommand2
  def call (String cmd, String logFilePath) {
  timestamps {
  cmdOutput = sh (script:"${cmd}", returnStdout:true).trim()
  }
  echo cmdOutput
  writeFile file: "${logFilePath}", text: "${cmdOutput}"
  }
  Here’s an example of using the command this way. Notice the invocation resembles a pipeline step:
  timedCommand2 'ls -la', 'listing.log'
  Suppose that we want to pass a block of code to a library “step”. When that happens, our library step will receive a Groovy closure. To handle this case, we define our step to accept the closure and then execute it:
  // vars/timedCommand3
  def call(Closure commands) {
  timestamps {
  commands()
  }
  }
  Or suppose we want to time how long it takes to read a file, do some kind of transformation on it, and then write the transformed data back out to another location. An example of how this could be called from a script is:
  timedCommand3 {
  def content = readFile file: '<path to huge datafile>'
  sh "<some processing on content>"
  writeFile file: '<path to transformed file>', text: content
  echo "Done"
  }
  This becomes even more useful if we want to do something like restrict the routine to a particular environment, such as a particular node. For example, we could set up two nodes, one running Windows and one running Linux, and then define two separate routines in vars, one to run our timing of commands on Windows and one to run the timing on Linux. Our code in vars might look like this:
  // vars/timedCommandWindows.groovy
  def call(Closure commands) {
  node('windows') {
  timestamps {
  commands()
  }
  }
  }
  // vars/timedCommandLinux.groovy
  def call(Closure commands) {
  node('linux') {
  timestamps {
  commands()
  }
  }
  }
  Finally, in this category, we can extend the call mechanism to create a simple framework that makes using the “step” in our scripts very simple and more like standard DSL calls with multiple values.
  This is done by delegating the values passed in to a mapping and then using the mapping in additional processing in the step. This is easiest to understand with an example:
  // vars/timedCommand4.groovy
  def call(body) {
  // collect assignments passed in into our mapping
  def settings = [:]
  body.resolveStrategy = Closure.DELEGATE_FIRST
  body.delegate = settings
  body()
  // now, time the commands
  timestamps {
  cmdOutput = echo sh (script:"${settings.cmd}", returnStdout:true).trim()
  }
  echo cmdOutput
  writeFile file: '${settings.logFilePath}', text: '${cmdOutput}'
  }
  In this form, we declare a Groovy map via the def settings = [:] syntax. Then the values we pass in get mapped and we can execute whatever other steps we need to. The references to delegate here have to do with Groovy functionality. A complete discussion of delegation behavior in Groovy is beyond the scope of this section, but you can essentially think of it as telling Groovy to allow us to reference any values passed in utilizing the mapping we’re doing in this function.
  Note that here, as in other vars steps, you should only use valid pipeline steps. Non-step Groovy code may not work or may have uncertain behavior.
  With this form, we can invoke the code from our pipeline script very simply, as shown here:
  node {
  timedCommand4 {
  cmd = 'sleep 5'
  logfilePath = 'log.out'
  }
  }
  What this really buys us is the ability to invoke our function with named parameters, passed in whatever order we choose. This can make our code in the pipeline script simpler and easier to understand and maintain.
  resources
  Non-Groovy files can be stored in this directory. They can be loaded via the libraryResource step in an external library.
  This is intended for allowing your external libraries to load up any additional non-Groovy files they may need. An example could be a datafile of some kind, such as an XML or JSON file, or any other file that the library needs to use. The file is loaded as a string.
  The syntax is straightforward. In your library code, you would have something like the following:
  def datafile = libraryResource 'org/conf/data/lib/datafile.ext'
  Config Files and Credentials
  Using some types of config files, such as ones associated with Maven, may require additional credential setup. See the Config File Provider plugin page for more details.
  Managing configuration files in a pipeline
  There is a configFileProvider step that you can use in your pipeline code. This is another block step, meaning that you invoke the step with some context, and it provides a closure in which you execute other code. So, for example, if you were using this functionality to access a properties file or custom XML file, you would first configure the file globally (as described previously). Then, in your pipeline code, you would invoke the step, passing in the file ID and related information. Then, within the step block, you could invoke other pipeline commands that use the properties file, the XML file, or their data.
  For example, suppose we have a config file set up as shown in Figure 8-24.
  [image: ]
  Figure 8-24. Example Groovy config file
  We could then add a step in our pipeline like the following to use it:
  configFileProvider(
  [configFile(fileId: 'my-groovy-script',
  variable: 'MY_GROOVY_SCRIPT',
  replaceTokens:true)]) {
  sh "cat ${MY_GROOVY_SCRIPT}"
  }
  Notice the syntax here. We first have the configFile parameter, which takes arguments that correspond to our Freestyle ones:
  fileId
  This is the file ID that was set up when you globally configured the file.
  Summary
  In this chapter, we’ve looked at pipeline constructs and steps that affect the overall execution flow of your pipeline.
  We started with seeing how to specify the kinds of events that you want to trigger your pipeline. And once triggered, we saw how to accept different kinds of input to direct the behavior of the pipeline.
  We looked at ways to have the pipeline try again when there’s a failure or move on after a certain time period. And we saw ways to deal with concurrency — both to prevent it for multiple runs of the same pipeline, and to leverage it for running tasks in parallel. And we noted how to provide conditional build execution.
  Finally, we looked at ways to accomplish post-build processing in pipelines similar to the functionality provided in Freestyle projects.
  All of this should give you a good start on controlling the flow of execution through your Scripted or Declarative Pipeline. In the next chapter, we’ll look at ways that Jenkins can send messages and notifications through some of the more popular communication tools.
  Directives
  A directive can be thought of as a statement or block of code that does any of the following in a pipeline:
  Defines values
  An example of this is the agent directive, which allows us to specify a node or container to run an entire pipeline or a stage in. If we wanted to run our pipeline on a node named worker, we could use agent ('worker').
  Configures behavior
  An example of this is the triggers directive that lets us configure how often Jenkins checks for source updates or triggers our pipeline. If we wanted it to retrigger our pipeline at 7 a.m. every weekday, we could use triggers { cron ('0 7 0 0 1-5') }.
  Specifies actions to be done
  An example of this is the stage directive, which is expected to have a steps section containing DSL steps to be executed.
  Deprecated Permissions
  In previous versions of the role-based access/matrix plugin, there were additional permissions that could be set:
  Overall/Run Scripts
  Overall/Upload Plugins
  Overall/Configure Update Center
  This was deemed a security risk because these permissions were as powerful in some cases as the Overall/Administer permissions, so now you need to have the Overall/Administer permission to automatically be able to run scripts without approval.
  If you do need to go back to the old insecure permissions for some reason, the org.jenkinsci.plugins.rolestrategy.permissions.DangerousPermissionHandlingMode.enableDangerousPermissions system property can be set to true.
  Folders and Vault
  It’s worth noting that Vault settings can also be configured at the folder level. Look for the Vault Plugin section in the folder configuration.
  With the basic Jenkins configuration out of the way, we can move on to using Vault in our pipeline.
  Using Vault in a Pipeline
  To use Vault in a pipeline, there are a couple of steps.
  First, we want to define an object that can identify which secret(s) and value(s) we want to access, as well as the environment variables we want to put them in for using in the pipeline. An example in Scripted Pipeline syntax is shown here (at the time of this writing, the Vault integration doesn’t seem to be supported for Declarative Pipelines):
  // define the secrets we want to access and the env variables
  // we want to put the retrieved values in
  def secrets = [
  [$class: 'VaultSecret', path: 'secret/example', secretValues:
  [[$class: 'VaultSecretValue', envVar: 'msg', vaultKey: 'value']]
  ]]
  Here, secret/example is the path, 'value' is the key to the keypair stored in Vault, and 'msg' is the environment variable we will access in our script.
  Once we have that set up, we can use the pipeline DSL step/construct "wrap" to access the credentials, as shown here:
  // inside this block our credentials will be accessible as env variables
  wrap([$class: 'VaultBuildWrapper', vaultSecrets: secrets]) {
  def myMsg = "The message is $msg"
  ...
  }
  Sample Pipeline
  Our simplistic pipeline is shown here in the form of a Jenkinsfile (converting pipelines to Jenkinsfiles was discussed in the previous section of this chapter):
  #!groovy
  @Library('Utilities@1.5')_
  node ('worker_node1') {
  try {
  stage('Source') {
  // always run with a new workspace
  cleanWs()
  stash name: 'test-sources', includes: 'build.gradle,src/test/'
  }
  stage('Build') {
  // run the gradle build
  gbuild2 'clean build -x test'
  }
  stage ('Test') {
  // execute required unit tests in parallel
  parallel (
  worker2: { node ('worker_node2'){
  // always run with a new workspace
  cleanWs()
  unstash 'test-sources'
  gbuild2 '-D test.single=TestExample1 test'
  }},
  worker3: { node ('worker_node3'){
  // always run with a new workspace
  cleanWs()
  unstash 'test-sources'
  gbuild2 '-D test.single=TestExample2 test'
  }},
  )
  }
  }
  catch (err) {
  echo "Caught: ${err}"
  }
  stage ('Notify') {
  mailUser('<email address>', "Finished")
  }
  }
  Let’s briefly discuss what this simple pipeline is doing.
  First, you see the Groovy designator at the top, preceding the loading of a shared library named Utilities. This library contains a build routine named gbuild2 that encapsulates our Gradle build call.
  The Source stage first cleans out the workspace and then pulls down the source code. Note the use of the checkout scm step again here. (As explained in “Approach”, this is sufficient since the Jenkinsfile is stored with the source code for the project, and so can interpret the SCM location to check out based on where it is.) Finally, this stage creates a stash of testware to share later with the Test stage.
  Next, the Build stage simply calls the routine from the shared library to build a set of Gradle targets, omitting the test target since that is handled in the next stage.
  The Test stage leverages the pipeline DSL parallel step to create two branches to run in parallel, each on a distinct node. Within the code for each branch, the workspace is cleaned, the contents of the stash are unstashed on the node so that the testware is present, and then a call is made to the shared library routine to build and run the particular set of tests.
  Finally, at the end of the pipeline, we include a Notify stage that simply calls a shared library routine to notify a user that the pipeline has completed. We include this at the very end, outside of the try-catch block, so that it is always executed, even if an exception happens within one of the other stages. This simulates the “post-build actions” processing of the traditional Jenkins Freestyle jobs, which always execute at the end of the build regardless of what completed (or not) during the build.
  Parallel Test Executor Plugin
  A separate plugin is available that can help with parallelizing sets of tests if they are taking up significant time in your pipeline. The Parallel Test Executor plugin looks at a run of your tests with the execution times and attempts to split the tests up into groups of roughly equal size. (This is done with a splitTests DSL step that is added by the plugin.) The groups are put into lists that you can then map into the parallel step in your pipeline. Optimally, each group would be mapped to run on a separate node.
  Using this plugin requires that your test environment/setup:
  Create JUnit-compatible XML files.
  Use a tool that can accept a test-exclusion list in a file.
  Alternative parallel syntax for Declarative Pipelines
  With the release of Declarative Pipeline 1.2 in September 2017, a new, alternative syntax was introduced for use in Declarative Pipelines. The new syntax more closely matches the structured form of Declarative Pipelines. It also doesn’t require the setup of a map or use of node, and produces output separated by each branch of the parallel operation.
  The new syntax elevates the parallel step to a separate construct within a stage. It can have stages defined within itself for each branch to run in parallel. Within each branch you can define an agent to run on and steps to execute just as you can for other Declarative Pipeline sections.
  An excerpt of a stage from a Declarative Pipeline that uses this syntax is shown here:
  stage('Unit Test') {
  parallel{
  stage ('Util unit tests') {
  agent { label 'worker_node2' }
  steps {
  cleanWs()
  unstash 'ws-src'
  gbuild4 ':util:test'
  }
  }
  stage ('API unit tests set 1') {
  agent { label 'worker_node3'}
  steps {
  // always run with a new workspace
  cleanWs()
  unstash 'ws-src'
  gbuild4 '-D test.single=TestExample1* :api:test'
  }
  }
  stage ('API unit tests set 2') {
  agent { label 'worker_node2' }
  steps {
  // always run with a new workspace
  cleanWs()
  unstash 'ws-src'
  gbuild4 '-D test.single=TestExample2* :api:test'
  }
  }
  }
  }
  As you can see, this syntax is somewhat “cleaner” than the map syntax and more consistent with the declarative syntax. When run, due to the individual stage definitions, it will also produce stage output for each “substage” (Figure 3-16) as opposed to the single set of output of the traditional parallel syntax (Figure 3-17).
  [image: ]
  Figure 3-16. Stage output for new parallel syntax
  [image: ]
  Figure 3-17. Stage output for traditional parallel syntax
  parallel and failFast
  Sometimes when doing multiple processing steps in a parallel block, you may want to quit processing all steps if one branch fails. For example, if you are doing deployment in one parallel section and testing in another section, you may want to abort the deployment if testing fails and abort the testing if deployment fails. To facilitate this, Jenkins pipelines can use the failFast option when invoking the parallel step.
  To use this option, you add failFast:true to the parallel step options. When this option is present and one of the branches in the parallel step fails, Jenkins will terminate all running branches.
  As an example, consider the following code. This is a simple Declarative Pipeline with one stage to demonstrate the parallel failFast usage. In the parallel step, we have the group1 branch that simply sleeps for 10 seconds and then echos out a message. The group2 branch sleeps for 5 seconds before throwing an error (via the error step) that will cause the failFast (the last argument to the parallel step) to fire. We have wrapped the group1 branch in catchError and timestamps steps so that we can detect when the branch is interrupted/terminated by the failFast operation:
  pipeline {
  agent any
  stages {
  stage ('Parallel') {
  steps {
  parallel (
  'group1': {
  timestamps {
  catchError {
  sleep 10
  echo 'Completed group1 processing'
  }
  }
  },
  'group2': {
  sleep 5
  error 'Error in group2 processing'
  },
  failFast: true
  )
  }
  }
  }
  }
  When we run this pipeline, we will get output like that shown in Figure 3-18.
  [image: ]
  Figure 3-18. Running with the failFast option enabled
  Looking at this output, you can see that 5 seconds into the group1 branch processing, the branch was terminated (note the “Exception” in the log). This was because after the 5-second sleep, the group2 branch threw the error. Then the failFast option terminated group1.
  If we were to take the failFast option out or set it to false, then we would still see the group2 branch terminate with the error, but the group1 branch would run to completion after the 10-second sleep, as shown in the alternate output in Figure 3-19.
  Figure 3-19. Running without the failFast option
  We move on now from dealing with running multiple operations in parallel to executing operations based on (potentially multiple) conditions.
  Dealing with Concurrency
  For the most part, having concurrency in your pipeline builds is a good thing. Typically, concurrency refers to parallelism — being able to run similar parts of your jobs concurrently on different nodes. This can be especially useful in cases such as running tests, as long as you limit duplicate access to resources appropriately.
  Another form of concurrency in Jenkins is when multiple builds of the same job try to run at the same time or use the same resources. In the case of very active repositories, branches, or pull requests, this may be an expected, common situation.
  But there may also be cases where this is not expected and not desirable. Let’s look briefly at two mechanisms that Jenkins pipelines have to address that situation.
  Nondeclarative Code and Blue Ocean
  If you plan to use your pipeline with Blue Ocean, be aware that the built-in editor is designed to work primarily with declarative syntax. Any nondeclarative syntax may be ignored or may not work as expected in the Blue Ocean editor.
  One of the challenges with using an input statement is what happens if you don’t get input in an expected amount of time. While waiting for input, the node is effectively stopped, waiting on a response. To prevent this from going on too long, you should consider wrapping the input call with another type of flow control construct: the timeout statement. We’ll discuss that in the next section.
  Docker Image Global Variable Methods
  After the methods focused on using the Docker application itself are the ones for working with Docker images. Most of these are fairly simple, and some are self-explanatory. Here’s the current text from the information page about this set of methods:
  image(id)
  Creates an Image object with a specified name or ID. See below.
  build(image[, args])
  Runs docker build to create and tag the specified image from a
  Dockerfile in the current directory. Additional args may be added,
  such as '-f Dockerfile.other --pull --build-arg
  http_proxy= .'. Like docker build,
  args must end with the build context. Returns the resulting Image
  object. Records a FROM fingerprint in the build.
  Image.id
  The image name with optional tag (mycorp/myapp,
  mycorp/myapp:latest) or ID (hexadecimal hash).
  Image.run([args, command])
  Uses docker run to run the image, and returns a Container which
  you could stop later. Additional args may be added, such as
  '-p 8080:8080 --memory-swap=-1'. Optional command is equivalent
  to Docker command specified after the image. Records a run
  fingerprint in the build.
  Image.withRun[(args[, command])] {…}
  Like run but stops the container as soon as its body exits,
  so you do not need a try-finally block.
  Image.inside[(args)] {…}
  Like withRun this starts a container for the duration of the body,
  but all external commands (sh) launched by the body run inside
  the container rather than on the host. These commands run in
  the same working directory (normally a Jenkins agent workspace),
  which means that the Docker server must be on localhost.
  Image.tag([tagname])
  Runs docker tag to record a tag of this image (defaulting to the
  tag it already has). Will rewrite an existing tag if one exists.
  Image.push([tagname])
  Pushes an image to the registry after tagging it as with the tag
  method. For example, you can use image.push 'latest' to publish
  it as the latest version in its repository.
  Image.pull()
  Runs docker pull. Not necessary before run, withRun, or inside.
  Image.imageName()
  The id prefixed as needed with registry information, such as
  docker.mycorp.com/mycorp/myapp. May be used if running your own
  Docker commands using sh.
  While the text explains the intent and general aspects of the operations, there is more background needed to put these into practice.
  First, note that Image (with a capital “I”) implies a reference to an instantiated image. Two of the methods do not expect that to be passed in — image(id) and build. That is because these two methods are called from the docker global variable and return an image. We’ve seen an example of that in the following lines from our earlier listing:
  myImage = docker.image("bclaster/jenkins-node:1.0")
  myImage.pull()
  In this case, we instantiate a variable to point to the returned image. Then we use that variable to invoke the pull()+ command for the specified image.
  Alternatively, we could skip the variable and instantiate an instance in the call:
  docker.image("bclaster/jenkins-node:1.0").pull()
  For the build method, you need to supply at least an image name. By default, it will use a Dockerfile in the current directory. If you need to pass additional arguments, you can pass them in the args area. You can pass the same string here as you would use if you were invoking Docker build directly on the command line. Just like with the arguments to the actual build command, you need to end with the build context (usually just ". will suffice unless you have a specific directory you need that has files to include):
  def myImage=docker.build("<registry/image:tag>","--build-arg
  ARG=value ./tmp-context-area")
  Here’s the script with the docker.build method:
  node() {
  def myImg
  stage ("Build image") {
  // download the dockerfile to build from
  git 'git@diyvb:repos/dockerResources.git'
  // build our docker image
  myImg = docker.build 'my-image:snapshot'
  }
  stage ("Get Source") {
  Figure 14-15 shows the console output running through constructing the Docker image from the Dockerfile.
  [image: ]
  Figure 14-15. Output from the previous script
  The other image-related methods are invoked from an instance of an image (thus the Image indicator). Many of these are self-explanatory because they mirror the basic image commands already found in Docker. These include tag, push, pull, and run. Others are slight variations, such as withRun, which stops the container for you after the body exits (as opposed to having to use some kind of explicit “post-build” teardown).
  However, one of these methods does quite a bit more. We’ll take a look at this method in more detail next.
  The inside method
  With the inside method, you choose the image you want to use and use this method to execute the build steps in the Docker image.
  When executed, the inside method will:
  Get an agent and a workspace (no node is required since the Docker container is effectively functioning as a node).
  If the image is not already present, pull it down.
  Start the container with that image.
  Mount the workspace from Jenkins. There are a few points to note here:
  This will appear as a volume inside the container.
  This will appear as the same file path.
  This must be on the same filesystem.
  Execute the build steps.
  Note that any sh (pipeline shell) commands are wrapped with docker exec to allow them to run in the container.
  Once completed, stop the container and get rid of the storage.
  Create a record that this image was used for this build. This facilitates image traceability, updates, etc.
  Additionally, options to pass to Docker can be specified. As an example, you could invoke <image name>.inside('-v ...').
  Here’s an example of a pipeline script using the Docker inside method to execute code:
  stage ("Get Source") {
  // run a command to get the source code down
  myImg.inside('-v /home/git/repos:/home/git/repos') {
  sh "rm -rf gradle-greetings"
  sh "git clone --branch test /home/git/repos/gradle-greetings.git"
  }
  }
  stage ("Run Build") {
  myImg.inside() {
  sh "cd gradle-greetings && gradle -g /tmp clean build -x test"
  }
  }
  Figure 14-16 shows the resulting Docker commands being processed in the console output.
  [image: ]
  Figure 14-16. Output from the previous script
  Node
  Node is the generic term that is used in Jenkins 2 to mean any system that can run Jenkins jobs. This covers both masters and agents, and is sometimes used in place of those terms. Furthermore, a node might be a container, such as one for Docker.
  A master node is always present in any Jenkins installation, but for the reasons already cited, it is not recommended to run jobs on the master node. We’ll talk more about how to define nodes in an upcoming section of this chapter.
  Enabling Security
  The top option on the global security configuration page is also the most high-level one — meaning that it encompasses the most related functionality. Without the “Enable security” option checked, security-checking operations are not enabled. With this option turned on, security can be configured along two dimensions — authentication and authorization.
  Authentication here refers to how users can identify themselves to the system, such as by user ID and password. This is now called “Security Realm” in Jenkins. Authorization refers to what permissions authorized users have. These two orthogonal dimensions can, together, implement nearly any desired security policy.
  Additional Assembly Required
  Building on a point raised earlier, additional code can be required to get the familiar Jenkins constructs we had in the web forms version. For example, the simple task of sending email after a failed build has to be handled with something like a try-catch-finally construct, instead of the familiar built-in post-build functionality.
  The following code highlights the contrast between sending emails after a failure in a Scripted Pipeline versus the way this was typically handled in traditional Jenkins, as shown in Figure 7-1.
  node {
  try {
  sendEmailStarted()
  stage('Source') {...}
  stage('Build') {...}
  ...
  sendEmailSuccess()
  } catch (err) {
  currentBuild.result = "FAILED"
  sendEmailFail()
  throw err
  }
  }
  [image: ]
  Figure 7-1. Post-build action in a Freestyle project in Jenkins
  For these and other reasons, the CloudBees staff, as part of the Jenkins community, created an expanded DSL and simpler environment for programming pipelines. Note that Declarative Pipelines are still pipelines-as-code. We are still using the same environment to code our pipelines; we enter the Declarative Pipeline syntax in the Pipeline tab script window or in Jenkinsfiles, just as we would for any other pipeline code. However, as we’ve noted, the Declarative Pipeline syntax is more structured and the environment provides improved DSL-specific validation and error checking. We’ll explore that structure next and discuss script checking and error reporting later in this chapter.
  A Closer Look at Trusted Versus Untrusted Code
  We previously discussed the distinction between trusted and untrusted code. We can demonstrate the difference here simply by trying to reference a restricted Jenkins object. In keeping with our time theme, we’ll attempt to use a Jenkins internal object that allows us to get the elapsed time since the build started — the getTimestampString method of the rawBuild object of the currentBuild. Putting this into a println, it might look like this:
  println "ELAPSED TIME: ${currentBuild.rawBuild.getTimestampString()}"
  First, we’ll try adding the line to our current pipeline script, as seen in Figure 6-20.
  [image: ]
  Figure 6-20. Adding the getTimestampString call to the main script
  Notice that we’re running this in the Groovy Sandbox. If we try to execute this script we’ll get a RejectedAccessException error, as seen in Figure 6-21. Since untrusted libraries run in the Sandbox, they get the same exception as if they were specifically limited to an untrusted type, such as a Folder, GitHub Organization, Bitbucket Team/Project, or Multibranch Pipeline project.
  [image: ]
  Figure 6-21. Access failure trying to use internal method in script
  However, if we add the method to a trusted library, such as our global shared library, and remove it from our pipeline script, it should work. Figure 6-22 shows the edit to add this to the appropriate runCommand library routine.
  [image: ]
  Figure 6-22. Adding the getTimestampString command to our trusted pipeline library
  And, in fact, when we run the original script now (without the call in it), we can see that the call gets executed successfully in the trusted library code (Figure 6-23) even though our main script is still running in the Sandbox.
  [image: ]
  Figure 6-23. Successfully executing the command through the trusted library instead of the restricted pipeline script
  environment
  This is an optional directive for your Declarative Pipeline. As the name implies, this directive allows you to specify names and values for environment variables that are then accessible within the scope of your pipeline. Like agent, you can have an instance of environment in the main pipeline definition and/or in individual stages. An environment definition in the top-level pipeline block will make the variable accessible to all steps in the pipeline. An environment definition within a stage will make the variable accessible to only the scope of the stage.
  Here is an example of defining an environment variable in this way:
  environment {
  TIMEZONE = "eastern"
  }
  Environment variable definitions can also incorporate variables that are already defined. The syntax for this is just to include the existing variable in the definition string in ${<variable>}:
  environment {
  TIMEZONE = "eastern"
  TIMEZONE_DS = "${TIMEZONE}_daylight_savings"
  }
  Credentials and environment variables
  We talked in Chapter 5 about the different kinds of credentials that can be used with pipelines. Each of those methods required the identifier of a set of credentials that had been defined in Jenkins. In the environment block, you can assign a global variable to a particular credentials ID. Then you can use that variable throughout your pipeline in place of the ID. This can simplify things if you need to specify the ID in multiple places. The syntax is to assign the variable name to the string credentials('<credentials-id>'). For example:
  environment {
  ADMIN_USER = credentials('admin-user')
  }
  In this case, we would have admin-user previously defined as the specified ID for some set of credentials. If you hadn’t explicitly specified a named string (admin-user) as the ID, you would use the identifying string that Jenkins automatically generates during the creation of the credentials.
  While we can define environment variables for anything we want, Jenkins provides a specific directive to access globally defined tools: the tools directive.
  Block
  A block here is really just any set of code that has a beginning and end. In Groovy, this translates to a closure (a section of code where the beginning and end are bracketed with { and }).
  While many parts of the pipeline are technically blocks, that term is used primarily to describe the overall pipeline block, which contains all of the code associated with a Declarative Pipeline.
  It looks like this:
  pipeline {
  // code in declarative syntax
  }
  Agent Created on the Fly for a Declarative Pipeline
  The Declarative Pipeline syntax includes special functionality for creating agents dynamically at the time they are needed. This is done by pointing the agent directive to a Dockerfile, from which it can run a container that uses a Docker image, set up to function as an agent. Most of these are just variations of the syntax for declaring an agent, as described here:
  agent { docker '<image>' }
  This short syntax tells Jenkins to pull the given image from Docker Hub and run the pipeline or stage in a container based on the image, on a dynamically provisioned node.
  agent docker { <elements> }
  This longer syntax allows for defining more specifics about the Docker agent. Three additional elements can be in the declaration (within {} block):
  image '<image>'
  Tells Jenkins to pull the given image and use it to run the pipeline code.
  label '<label>'
  Tells Jenkins to instantiate the container and “host” it on a node matching <label> (optional).
  args '<string>'
  Tells Jenkins to pass these arguments to the Docker container; uses the same Docker syntax as you would normally use (optional).
  Here’s an example usage:
  agent {
  docker {
  image "image-name"
  label "worker-node"
  args "-v /dir:dir"
  }
  }
  agent { dockerfile true }
  This short syntax is intended to be used when the source code repository that you retrieve has a Dockerfile in its root (note that dockerfile here is a literal). It tells Jenkins to build a Docker image using that Dockerfile, instantiate a container, and then run the pipeline or code from the stage in that container.
  agent dockerfile { <elements> }
  This longer syntax allows for defining more specifics about the Docker agent you are trying to create from a Dockerfile. Three additional elements can be added in the declaration (within the {} block):
  filename '<path to dockerfile>'
  Allows you to specify an alternate path to a Dockerfile, including a different name. Jenkins will try to build an image from the Dockerfile, instantiate a container, and use it to run the pipeline code.
  label '<label>'
  Tells Jenkins to instantiate the container and “host” it on a node matching <label> (optional).
  args '<string>'
  Tells Jenkins to pass these arguments to the Docker container; this should be the same syntax as normally used for Docker (optional).
  Here’s an example usage:
  agent {
  docker {
  filename "<subdir/dockerfile-name>"
  label "<agent label>"
  args "-v /dir:dir"
  }
  }
  reuseNode
  This tells Jenkins to reuse the same node and workspace that were defined for the original pipeline agent to “host” the resulting Docker container.
  This last one requires a bit of explanation. Remember that even though we are running a Docker container for our agent in these cases, we still have to have a system where Docker is actually hosted and running. That’s what the label argument in these calls is specifying: which system is hosting Docker.
  If we start our pipeline running on a particular node, then it may do operations that leave code or other input on the node (such as cloning source out of source control). If we later want to use a Docker container to do something in the pipeline (such as build the source), then it makes things simpler if we can just run/host the Docker container on the same underlying node. Since the code is already there and the Docker commands can mount the workspace as a path inside them, that simplifies this kind of setup. That’s what the reuseNode option is for — running an upcoming Docker container on the same node that we started with.
  Here’s an example:
  pipeline {
  agent label 'linux'
  ...
  stage('abc') {
  agent {
  docker {
  image 'ubuntu:16.6'
  reuseNode true
  ...
  Pipeline Model Definition
  Since we’re talking about which nodes can host Docker instances here, it’s a good place to mention one of the configuration items in Jenkins around Docker — the Pipeline Model Definition section (Figure 14-10).
  [image: ]
  Figure 14-10. Pipeline model definition configuration
  By default, Jenkins pipelines make the assumption that all agents are able to run Docker pipelines. However, in some cases, where you can’t run the Docker daemon directly, this assumption can be incorrect. In these cases, if you don’t explicitly specify an agent that can run Docker in your pipeline and you get one of the agents that can’t, your pipeline won’t work.
  Assuming you have a label on one or more of your agents that uniquely identifies them as being capable of running Docker, you can specify that label here. This tells Jenkins to use one of these agents that for any folder items that need Docker, but don’t directly specify an agent that can run it.
  Likewise, you can specify a Docker registry to use here that is scoped to just the items in the folder.
  Specifying Credential Types by Provider
  The last part of the Configure Credentials screen is the Restrictions section. This allows you to specify the types of credentials Jenkins will allow or exclude from a specific provider (see Figure 5-13). This is a way to fine-tune what Jenkins can use from a provider. Note that doing this is optional and not required.
  [image: ]
  Figure 5-13. Refining/restricting the credential types allowed from a credential provider
  The Add button in this section has two options (“include” and “exclude”). Selecting either will create a new page element allowing you to select a provider and then a type. If you have selected “include,” this type of credential will be included for that provider, and vice versa for “exclude.”
  If you need to set up restrictions for multiple types and/or multiple providers, adding all of the elements can take some time. However, as noted earlier, using this is optional, not required.
  Limiting Access for Build Jobs
  As discussed in “Credential Scopes”, most internal operations and connections between systems in Jenkins run at the “system” level — meaning they have full permissions to the system.
  For build jobs, this high level of access is not always advisable or desirable. To allow build jobs to run at lower levels of authorization, the Authorize Project plugin has been developed.
  With this plugin installed, new controls are added to the global security configuration page and to each individual build job that allow users to specify the “authorizations” (types of credentials) to use for the job. See “Access Control for Builds” for more on how this works.
  The Structure
  A declarative is made up of an outer block that contains directives and sections. Each section in turn can contain other sections, directives, and steps, and in some cases conditionals. The distinction between blocks, sections, and directives is somewhat arbitrary, but since they’re used in the formal documentation, we’ll define those and the other terms more clearly.
  Context Links
  On any of these credentials pages, the links in the tables are “context links” — meaning that if you hover over them, a small downward-pointing arrowhead will appear to the right of the link. Clicking on that will then display a small pop-up menu that will provide shortcuts to certain actions or quick navigation links.
  The basic rules for what shows up in the pop-ups when you click on one of these context links follows from the hierarchy of store, domain, credentials.
  If you click on the link itself:
  For a store link, it will take you to a page that shows information about the domains in the store.
  For a domain link, it will take you to a page that shows information about the credentials defined in that domain.
  For a credentials link, it will take you to a page that shows usage information for that credential (whether or not that credential has been recorded as used).
  If you click on the drop-down arrow next to the link:
  For a store link, it will give you a menu option to create a new domain.
  For a built-in domain, it will give you a menu option to create a new credential.
  For a custom domain (one that has been created by a user), it will give you a menu with options to create a new credential, configure the domain, or delete the domain.
  For a credential, you will get an option to update, move, or delete a credential.
  Chapter 16. Troubleshooting
  There can be a steep learning curve associated with migrating to Jenkins 2. In this chapter, I’ll attempt to explain some of the common or more complex issues you may run into, or point you to other sections of the book where they are explained.
  This is more a varied collection of tips and processes than a consistent flow of information, but this is by design, since the best method to troubleshoot a situation can vary widely depending on the circumstances.
  Let’s start out by looking at how we can drill in to get more details about the steps in our pipeline.
  The withEnv Step
  Jenkins includes withEnv, a special step for working with environment variables. Actually, this is a block step, meaning that it sets some context when it is invoked that is valid for any code put within its block.
  Here’s an example that we can look at and discuss:
  withEnv(["PATH+GRADLE=${tool 'gradle3'}/bin", 'USER=Jenkins2']) {
  sh 'echo PATH = $PATH'
  }
  In the step’s invocation, we are setting two environment variables: PATH and USER. Notice the PATH+ syntax here — this is a special syntax that is allowed for the withEnv step to prepend things to the path. In this case, we’re leveraging the tool pipeline step to add the path associated with “gradle3” in our Global Tool Configuration. Then we also set the USER environment variable.
  The PATH+... string is enclosed in double quotes, because we are using Groovy interpolation to resolve the value of ${tool 'gradle3'} as part of the string. But also notice that in the shell call (sh) within the body of the withEnv step, we are using single quotes, even though we use the $PATH value. The reason for this is that we want this $PATH to be interpreted by the shell itself, not Groovy. The use of single quotes means that Groovy will not try to interpret it and it will be passed on to the shell call as intended.
  So, this step will have output like the following (assuming gradle3 resolves to /usr/share/gradle):
  PATH = /usr/share/gradle/bin:/usr/local/sbin:/usr/local/bin:
  /usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:
  As you can see, the item we set in the context was prepended to the path (added at the start). This allows you to make sure the desired item is seen first.
  Chapter 2. The Foundations
  Now that you understand the big ideas around which Jenkins 2 is built, we can move on to how Jenkins 2 supports pipelines-as-code. A key first step is understanding the development environment that Jenkins provides specifically for working with pipelines. This includes the systems we run our pipelines on as well as the interfaces for creating, executing, and monitoring pipelines. Additionally, you need to know about some of the basic structures that make up a pipeline, and how they fit together. Together, these elements will provide a solid foundation to build on for the rest of the book.
  We’ll approach this task by concentrating on four basic areas:
  The two styles of syntax that can be used for creating pipelines
  The systems used to run the pipeline processes
  The basic structure of a pipeline
  The support environment (and tooling) that Jenkins provides for pipeline development and execution
  We’ll start by defining and disambiguating some key concepts and terminology used with pipelines. Then we’ll survey the required DSL structures. Along the way, we’ll look at how to use the built-in editor and how to use a new tool in Jenkins to help figure out pipeline syntax.
  Once you know how to input your pipeline code, we’ll move on to executing a pipeline and understanding the new views that Jenkins provides. We’ll also look at how to access logs from a run. Finally, we’ll explore new functionality in Jenkins that allows us to try out changes to pipelines, without overwriting our existing versions.
  Let’s get started by learning more about the different pipeline syntax styles supported in Jenkins 2.
  Developing Jenkinsfiles
  A typical approach that can work well for creating a pipeline as a Jenkinsfile is to first develop the pipeline code within the Jenkins application itself as a Pipeline project. This gives the benefit of quick turnaround and feedback while developing the code. When the pipeline is working to your satisfaction in the Pipeline project, then you can follow the process in this section to convert it into a Jenkinsfile.
  Typically, it only requires a few simple steps to migrate pipeline to a Jenkinsfile. The approach is outlined in the next section.
  Grayed-out Credentials
  If you happen to have a credential in this table that has the same ID as a credential in the parent context, it will be grayed out in this table to indicate that.
  The next table (in the middle) lists the credential stores available in the current context. The columns here are:
  Provider
  Store
  Domains
  The bottom block lists the credential stores available in the parent context. It has the same three columns as the preceding table.
  tools Without an Agent
  If an agent is not specified, such as when only agent none is used at the top of the pipeline, the tools directive does not have any effect. This is because there is no node/agent to make the tool available on.
  For example, suppose we had the configuration shown in Figure 7-4.
  [image: ]
  Figure 7-4. Global configuration for a Gradle version
  Then, in our tools block, we could refer to Gradle via:
  tools {
  gradle "gradle3.2"
  }
  The lefthand part of this declaration is a specific string defined in the pipeline model. As of this writing, the valid tool types you can specify in declarative syntax are:
  ant
  git
  gradle
  jdk
  jgit
  jgitapache (JGit with Apache HTTP client)
  maven
  Attempts to use other types that are not yet valid will result in an “Invalid tool type” error when running your pipeline.
  Extended Tool Types for Scripted Pipelines
  The tool DSL step (not the declarative section) can take an additional parameter of type. Some of the supported types correspond to the types you can specify in the declarative tools section, but some are only specifiable as class names currently and don’t fit in the tools section.
  Jenkins currently lists the full range of supported types as:
  ant, hudson.tasks.Ant$AntInstallation
  org.jenkinsci.plugins.docker.commons.tools.DockerTool
  git
  hudson.plugins.git.GitTool
  gradle
  hudson.plugins.gradle.GradleInstallation
  hudson.plugins.groovy.GroovyInstallation
  jdk
  hudson.model.JDK
  jgit
  org.jenkinsci.plugins.gitclient.JGitTool
  jgitapache
  org.jenkinsci.plugins.gitclient.JGitApacheTool
  maven
  hudson.tasks.Maven$MavenInstallation
  hudson.plugins.mercurial.MercurialInstallation
  hudson.plugins.sonar.SonarRunnerInstallation
  hudson.plugins.sonar.MsBuildSQRunnerInstallation
  If you use the Snippet Generator, you’ll see the more user-friendly versions of the names listed in the drop-down for the Tool Type parameter. Then, if needed, the class name is inserted when you select the type. For example, if you select SonarQube Scanner for the Tool Type and you have a scanner named sq-scanner configured in the global configuration, the generated step is:
  tool name: 'sq-scanner',
  type: 'hudson.plugins.sonar.SonarRunnerInstallation'
  In most cases, you don’t need to specify the type when using the tool step. The current exception would be if you had two different types of tools configured with the same name in the global configuration. Then the type value could be used as a differentiator.
  The righthand part should map to the Name field in the Global Tool Configuration.
  Once this is set up, the tool is autoinstalled and put on the path. We can then simply use the string gradle in place of the GRADLE_HOME path in our pipeline steps and Jenkins will map it back to this Gradle installation on our system. For example:
  steps {
  sh 'gradle clean compile'
  }
  Also, it’s worth noting that the tools directive can use the value of a parameter if you need to input a particular version to use. Here’s an example:
  pipeline {
  agent any
  parameters {
  string(name: 'gradleTool', defaultValue: 'gradle3',
  description: 'Gradle Version')
  }
  tools {
  gradle "${params.gradleTool}"
  }
  Just keep in mind that there is currently a limitation with the declarative syntax such that Jenkins doesn’t recognize that a build requires a parameter the first time the pipeline is run.
  tools is another directive that can be used either in the pipeline block or separately in a stage.
  The Editor
  As you begin to work with the editor, there a couple of features that are helpful to be aware of:
  Syntax checking
  Where possible, the editor will make an attempt to check for valid Groovy syntax and references. As Figure 2-8 shows, it will mark any problems it finds with a red square with an “X” in it beside the offending line.
  Figure 2-8. Error indications in the pipeline script window
  However, it is possible that not all the errors flagged are actual errors — in some cases the script may not have been able to yet resolve a dependency or an import that was recently created, although this is the exception rather than the rule.
  Extended error information
  While the “X” indicator provides a quick visual way to identify lines with problems, it is not very informative beyond that. You can see more information by hovering over the “X.” When you do this, a pop-up is displayed with the full text of the error (Figure 2-9).
  Figure 2-9. Hovering displays the full text of the error message
  Autocomplete
  The editor also includes autocomplete functionality for items like brackets. That is, if you type an opening bracket, {, the editor will automatically insert (after a space) a corresponding closing bracket, } (Figure 2-10). This is a convenient feature, but can also trip you up until you get accustomed to it. The reason is that if you’re in the habit of always typing a closing bracket and one is inserted for you, you’ll end up with an extra bracket in your program that won’t compile.
  Figure 2-10. Autocompletion of brackets
  Outside of the editor, we have an additional tool to help us get the syntax correct. It’s called the Snippet Generator.
  Setting Permissions Too Loosely
  It is possible to implement policies using authentication and authorization that actually make your Jenkins instance very insecure. For example, Jenkins provides an option to allow users to sign up for access — less secure authentication. If you also use very open authorization policies, such as allowing logged-in users to do most/all operations, then you are effectively leaving your system wide open to anyone who wants to use it in whatever way they want.
  The username/password login info is required for any operation unless anonymous users are specifically allowed to do the operation. In Jenkins 2, by default, logged-in users have full control and anonymous users have no access.
  Under the checkbox to enable security, we have the “Disable remember me” checkbox. Checking this option to enable it removes the “Remember me on this computer” option from the login screen.
  Next, we have the Access Control section. This section provides the options to configure the two dimensions we talked about earlier (authentication and authorization).
  Access Control — Security Realm
  This section allows us to specify which entity will be responsible for authenticating users to Jenkins. There are several choices.
  Delegate to servlet container
  The servlet container being referenced here is the one running the Jenkins instance. These days, this is usually Jetty, but it might also be Tomcat or some other servlet if the installation has been customized. With this option, you are allowing authentication via whatever mechanism the servlet container uses.
  The specifics of how to set this up depend on how authentication is configured for the particular servlet container being used. The best approach is to consult the documentation for the servlet container. Up until v1.163 this was the default security realm. It is not as likely to be used these days given the other options, but can still be worthwhile for backward compatibility, or if you have invested significant setup for authentication in the servlet container’s configuration.
  Jenkins’ own user database
  This option delegates authentication to the list of people maintained by/known to Jenkins. This is not a typical use case, but can be suitable for smaller, basic setups. Note that this includes not only all the users that Jenkins specifically knows about, but also users mentioned in commit messages.
  A suboption allows enabling users to “sign up” — meaning they can create their own accounts at the time they first need to log in to Jenkins. This suboption is disabled by default to more tightly control access.
  LDAP
  The Lightweight Directory Access Protocol (LDAP) is a software protocol for locating people, organizations, devices, and other resources on a network. If your company uses LDAP, this is where you can configure it for Jenkins. You can add more than one LDAP server (each having a different configuration if needed).
  Unix user/group database
  This option delegates authentication to the host Unix system’s user database. If this is used, users can log into Jenkins using their Unix username and password. Unix groups can also be used for authentication. If a user and a group have the same name, prepending an “@” onto the name differentiates it as a group. Note that there may be extra configuration required to make this all work, such as making Jenkins a member of the shadow group for access on operating systems that use that.
  Access Control — Authorization
  Once authenticated, Jenkins needs to know what kind of operations users should be allowed to do. Like in the Security Realm section, there are several options here.
  Anyone can do anything
  No real authentication is done with this option. Basically, everyone is considered as “trusted” — including anonymous users (even if they haven’t logged in yet). This is not recommended, but can be suitable in rare cases for completely trusted environments to allow unrestricted access for simplicity and efficiency.
  Legacy mode
  This mode emulates Jenkins behavior prior to v1.164: anyone who has the “admin” role has full control, and everyone else has read-only access.
  Logged-in users can do anything
  As the name implies, users must first log in, but then have full access. This is useful if you don’t mind allowing everyone full access, but want to keep track of who is doing what (via them being logged in).
  A suboption here enables anonymous users to have read-only access.
  Matrix-based security
  This option allows you to specify very specific permissions for individual users or groups via checkboxes in a matrix arrangement. The columns in the matrix are divided into categories (groupings) such as “Overall,” “Job,” “Run,” etc. Then underneath each of those items are further specific permissions related to that category.
  The rows of the matrix each represent a user or group. There are two default groups that are automatically added: “Anonymous Users” (users who have not logged in) and “Authenticated Users” (users who have logged in). A text box under the matrix allows you to add new users.
  Granting a particular permission to a user or group is just a matter of clicking in the box that corresponds to the appropriate row for the user/group and the column for the specific permission. Removing a permission involves just clicking again to clear the checkbox.
  At the end of each row are boxes you can click on to grant all permissions or remove all permissions for that user/group.
  Figure 5-2 shows an example matrix.
  [image: ]
  Figure 5-2. Example of matrix-based authentication
  Project-based matrix authorization strategy
  This option is an extension to the “Matrix-based security” model described in the preceding section. When selected, this adds a similar matrix to each project’s configuration page. This allows for configuration by user/group per project, so you can restrict access to some projects while allowing it for others.
  More specifically, when this option is set in the global security page, each project’s configuration page will have an “Enable project-based security” option in the General configuration section. Selecting this option will then present an authorization matrix for that project that can be configured like the global matrix to provide project-specific access. An additional option allows you to select whether to inherit permissions from a parent access control list, the globally defined permissions, or not at all.
  Figure 5-3 shows an example of one such matrix in a project.
  [image: ]
  Figure 5-3. Per-project authorization matrix
  Existing Issues
  While Blue Ocean provides an immense amount of functionality, it’s important to note that it is still relatively early in terms of formal releases as of the time of this writing. A number of issues may be encountered with using it beyond the basic, core functionality. In particular, when working with the pipeline editor, certain syntax that is valid for Declarative Pipelines cannot be entered or handled through the editor interface.
  Where these issues currently occur, we will note them and, where possible, suggest workarounds.
  You’re encouraged to always check the latest version of Blue Ocean and Jenkins to see whether a particular issue noted here has been fixed.
  Diving into Pipeline Steps
  While the Stage View provides a level of separation and detail on pieces of the pipeline, there may be times when it is beneficial to examine processing at an even lower level to troubleshoot an issue. The Pipeline Steps view provides this capability.
  To get to the Pipeline Steps view, you first need to go into the output screen for a single run of a build. You can use a URL of the form:
  http://<jenkins-location>/job/<job-name>/<build-number>
  or simply click on the build number in the Build History section of the Stage View page. This will take you to the specific output page for that build. On that page, in the menu on the left, will be a Pipeline Steps item (Figure 16-1).
  [image: ]
  Figure 16-1. The menu item to get to the Pipeline Steps screen
  After clicking on that, you’ll be taken to a screen that shows the breakdown of the pipeline by steps (Figure 16-2). Each row here represents a step. The first field in each row lists the step, along with the time it took to execute. This text is also a link to a more general, but very sparse, page about the step.
  [image: ]
  Figure 16-2. The Pipeline Steps screen
  On the righthand side of the row are any arguments that the step received, a screen icon that links to the console output (if that makes sense for the step), and a status indicator of whether the step was successful or not.
  With these data points, you can verify that steps got the expected arguments, see which steps used the most/least time, and view only the portion of the console output that pertain to a particular step. Figure 16-3 shows the result of clicking on the console output icon for a failed step shown in Figure 16-2.
  [image: ]
  Figure 16-3. Console output limited to the selected step
  The Pipeline Steps screen is also the way to get to the workspace from the web interface. Prior to Jenkins 2, there was a Workspace link on the output page for a build. This link is not surfaced any longer on that page. Instead, you have to dig deeper through this area to find it.
  Since a workspace is associated with a node, you first click on the console output icon for a pipeline step associated with allocating a node (Figure 16-4).
  [image: ]
  Figure 16-4. Selecting the console log for the “allocate node” step
  This takes you to the main screen for the step, where you can see the link for the workspace in the menu on the left (Figure 16-5).
  [image: ]
  Figure 16-5. Main screen for “allocate node” step
  Clicking that link takes you to the top level of the workspace. From here, you can drill down using the links provided, or you can type in a relative path in the text entry box (Figure 16-6).
  [image: ]
  Figure 16-6. Entering a relative path (to go to) on the Workspace screen
  Clicking the arrow at the end of the text box will then take you directly to that location (Figure 16-7).
  [image: ]
  Figure 16-7. The Workspace screen
  Browsing the workspace in this way can be another way to discover what differs from your expectations, and to help uncover the causes of problems.
  Some problems are not caused by the way the steps are organized or used in the pipeline, but by trying to use steps, methods, or libraries that aren’t serializable — that is, not able to save their state. This violates a requirement of Jenkins 2. Dealing with errors and problems around serialization is the topic of our next section.
  Favorites
  Note that wherever the star icon occurs (whether in a row or in a header), it has the same purpose — to toggle the “favorite” status of a project. However, if an entry is a container for other entries (such as a Multibranch Pipeline project that has other jobs in it), the favoriting functionality will work only if it can determine what the “default” object is. For example, if you select a Multibranch Pipeline project to favorite, it will favorite the master branch if there is one. If there isn’t a good default available, you’ll see an error like this:
  Favoriting Error
  No default branch (e.g. "master") to favorite.
  Finally, near the bottom of the dashboard screen will be some text that identifies the particular version of Blue Ocean that is running, and the version of Jenkins that it is being run on.
  Clicking on any of the projects listed on the dashboard will take you to the specific page for that project. We’ll discuss the contents of that page next.