The simplest way of dealing with multiple bash scripts
GitHub - countertenor/mozart: A language-agnostic helm-like script orchestrator
Mozart
If you struggle with maintaining a lot of scripts:
╰$ tree scripts
scripts
└── my-service
    ├── install
    │   ├── install-component1.sh
    │   ├── install-component2.sh
    │   └── install-component3.sh
    └── test
        ├── test-component1.sh
        ├── test-component2.sh
        └── test-component3.sh

then think of Mozart! Mozart is a script orchestrator. It can help package all scripts to a single binary, along with a CLI and and UI to manage them, with ZERO lines of code!

CLI:
Available CLI commands:

mozart execute my-service
mozart execute my-service install
mozart execute my-service install install-component1
mozart execute my-service install install-component2
mozart execute my-service install install-component3
mozart execute my-service test
mozart execute my-service test test-component1
mozart execute my-service test test-component2
mozart execute my-service test test-component3

What is mozart?
Mozart is a simple drop-in (no go-coding required) utility to orchestrate and attach a CLI and a UI to your scripts, making your independent, messy bunch of scripts into a well defined, orchestrated program complete with a CLI and UI!
Within minutes, instead of having hundreds of different scripts, you can have a single binary, complete with a CLI and a UI, which includes all those scripts. All you need to do is to drop the scripts into a folder structure (explained below). That's it!
Note: No code changes required, it is a simple drop-in type utility for your scripts.
What is an orchestrator?

What Docker compose is to Docker containers, Mozart is to scripts.


Manages packaging and distribution of scripts
Manages modularization – breaking down of big scripts into smaller scripts
Manages templating across scripts – runtime variable substitutions
Manages DRY policy across scripts
Manages all logs generated by scripts
Manages state of execution of all scripts – success/failure/last-exec time
Manages execution of all scripts – rerun/dry-run
Provides a CLI, a UI, a REST interface to interact with the scripts

How mozart differs from Ansible



Mozart
Ansible




Easy learning curve
Tough learning curve


No new software needed (only clone this repo)
Initial setup required


Very less changes to existing scripts (simple drag and drop capability)
More effort in making ansible modules


Working orchestrator within minutes
Takes time to create playbooks


Provides single file packaging for all scripts
Doesn't provide packaging (AFAIK?)


Useful for quick POC, testing, etc.
Suitable for heavy duty production deployments


Simpler to use for single host (multiple hosts requires some changes)
Suitable for multiple hosts from start



Table of Contents



What is mozart?
What is an orchestrator?
How mozart differs from Ansible
What exactly does an orchestrator do?

Life without an orchestrator
Benefits of using Mozart


Using your scripts with Mozart

Steps

1. Initial setup
2. Modularize
3. (Optional) Use templating
4. Build the binary




Mozart yaml file

Templating
Optional configuration parameters

Log sub-directory
Exec source
Delims
Arguments




Using common snippets across scripts

More info
Gotchas


Adding custom resources
CLI

Mozart commands
Executing modules
Checking the state

State of all modules
State of specific module




UI
Helpful resources

Good links for templating
Bash scripts through go
Versioning with go
Web sockets
Command execution
Error stack
Embedding static content
Go build tools




What exactly does an orchestrator do?
Simply speaking, an orchestrator manages execution of all of your scripts.
Life without an orchestrator
Suppose you had 2 bash scripts which you use to test 2 different components. Without an orchestrator:

You would have to ship both files to anyone who wants to use them. (imagine if you had 10 scripts, you would have to tar them up and send all).
If running on a shared system, you can never know if another person already ran the scripts (unless actually going through the effort of seeing if the tests ran).
Tendency to use lesser number of scripts for easy script execution management, but that makes each script huge. Breaking them down into multiple smaller scripts makes it easy to manage the code but it becomes harder to manage execution of all scripts.
No way of using common variables across the scripts (like version of the component being tested).
No way for one developer to look at the logs of a script executed by a second developer (unless the second developer explicitly routes the logs to an external file).
No way of accidentally preventing execution of a script more than once (like prevent first script from running again if it already ran once).
Manually execute each script through bash or python (no CLI or UI)

Benefits of using Mozart

Simple migration to Mozart - no Go code change necessary. Just create a directory and dump your scripts in that - it's that easy. (Discussed in detail below)
Single binary file which contains all your scripts AND brings with it the CLI along with the UI - so no need to send in bunch of scripts to anyone anymore.
Lets you modularize the scripts, which means you can have more number of smaller scripts which do smaller tasks. No need to maintain huge bash files anymore. The smaller the scripts - the easier it is for you to manage and maintain them.
Ability to use templating capabilities - similar to helm. Values in a yaml file are accessible by all scripts.
Public visibility of the state of execution of scripts, so everyone has a clear idea of whether scripts were executed or not.
Central logs for everyone to see.
Make sure scripts are executed just once - the orchestrator will not allow you to run the same script again without explicitly mentioning a Re-Run flag, thereby preventing accidental execution of the same script.
Ready to use CLI + UI to manage the execution.

Using your scripts with Mozart
Let us walk through how to actually add your scripts. There is one term that is of prime importance to Mozart - Modules.
Modules
Mozart works with the concept of modules and not the scripts directly. Modules are nothing but directories, created with the intention of performing 1 simple task. Each module (aka directory) can consist of either scripts or more nested modules (nested directories).
So in short, all your scripts need to be in a module, and Mozart will help you control the execution of those modules (instead of the scripts themselves).
Sample module
There is already a sample module present called test-module under resources/templates, which you can use to reference.
Steps
1. Initial setup

Install go - https://golang.org/doc/install
Clone repo - https://github.com/countertenor/mozart
Run the command export PATH=$PATH:$(go env GOPATH)/bin) (Please note - to make it persistent, you will have to add the command to your .bashrc)
Create a new directory inside resources/templates. This will be the base module under which all your modules will exist.
(Optional) Delete the existing test-module inside resources/templates if you want a clean slate. That folder is only for reference. Leaving that folder as it is will not do any harm.

2. Modularize


Look through your scripts, and identify the most basic steps that the scripts are supposed to be performing. Suppose I want to install a component called Symphony. I have one huge bash script for that. Some basic steps inside that bash script could be:

Pre-requisite check.
Installation of component.
Validation of install.
Uninstallation of component.



For each identified step, create a module(directory) within the base directory and add that part of the script within that directory. For example, continuing with the above example, the directory structure should look like this:
resources/templates
├── symphony                    (this is the base module)
│   ├── 00-pre-req              (first sub-module)
│   │   └── pre-req.sh
│   ├── 10-install              (second sub-module)
│   │   ├── 00-install-step1.sh
│   │   └── 10-install-step2.sh
│   ├── 20-validate             (third sub-module)
│   │   └── validate.sh
│   └── 30-uninstall            (fourth sub-module)
│       └── uninstall.sh

The huge bash script is now broken down into smaller scripts, each in its own module. This makes the script easy to manage, while giving the option to add more scripts in the future as needed.
Note: the xx-prefix before a module or script name is an optional prefix, through this you can control the order of execution of scripts/modules within the module.


Note: In case you don't want to break down your script into smaller scripts, you can create only the base module and drop your script in that directory.
3. (Optional) Use templating
You can do something like this within any script:
echo "{{.values.value1}} {{.values.value2}}"

These values are going to be fetched from a yaml file that you supply while invoking the CLI or the UI. (discussed later)
The yaml file should have something like this for the example above:
values:
value1: hello
value2: world

When you execute the corresponding module that contains the above script, you will see

This is discussed in detail below.
4. Build the binary
Run make - builds binaries for linux, mac and centOS environments, inside bin directory.
Voila! You have a single orchestrator binary with all your scripts in it.
Mozart yaml file
Providing an optional yaml file at runtime lets you enable certain templating features and configuration changes. If you do not need any changes, you can skip this section.
A sample blank configuration file can be generated from the binary itself, using the init command.
./bin/mozart init

Generated sample file :  mozart-sample.yaml

Templating
You can use the above yaml file to help with templating. If you refer to the test-module under resources/templates, you can see some examples of templating.
This idea is similar to helm.
Example:
In the file step1.sh, you see this:
#!/bin/bash

echo "{{.values.value1}} {{.values.value2}}"

The values in the brackets are the values that will be fetched from the yaml at runtime. So if you want to substitute some values at runtime, you replace the values with the {{ }} notation as you see above, and in the yaml file, add:
values:
  value1: hello
  value2: world

Mozart will substitute these values at runtime.
Optional configuration parameters
There are certain configuration parameters also that you can change using the same yaml file as above.
Log sub-directory
By default, log files are stored in /var/log/mozart directory (For linux and centOS), but if for some reason you want to add a sub-directory, you can do so by adding one line to the yaml file:
Example:

Then all the logs will go to:
var/log/mozart/my-log-dir

Exec source
This lets you choose the execution environment of any type of script that you include.
The format is file_ext: source
Example:
exec_source:
  py: /usr/bin/python
  sh: /bin/bash

This lets Mozart know that if you place any file with the extension of .sh, then run it using /bin/bash. If you place any file with the extension .py, then run it using /usr/bin/python.
Note: These are the only 2 extensions added by default in Mozart. If you add any other type of script apart from python or bash, you will need to add the execution source in the yaml.
Delims
This lets you change the default delimiters (default - {{, }})
Example 1:

Adding this line in the yaml file changes the delimiters to [[ ]]. So after this, you can use templating like:
echo "[[.values.value1]] [[.values.value2]]"

Example 2:

Adding this line in the yaml file changes the delimiters to << >> . So after this, you can use templating like:
echo "<<.values.value1>> <<.values.value2>>"

Arguments
This lets you pass arguments to the scripts themselves when they are being executed.
Example:
args:
  00-step1.sh: ["-s", "hello"]

This makes sure that whenever the 00-step1.sh file is being executed by Mozart, the -s and hello arguments are passed to the file at runtime.
Using common snippets across scripts
There might be a scenario in which some scripts have a lot of common code. It is never a good idea to duplicate logic across scripts (DRY principle).
To tackle this, you can make use of the common folder that is present under the resources folder. This folder has one purpose and one purpose only - to hold common snippets of information that will be needed by more than one script.
You can take a look at the static/resources/templates/example-module/10-python-module/00-common-example/python-1.py file for an example.
Example:
Suppose if you have a function that you want in more than one script, say
def my_func(str):
print('inside funct - ' + str)

Instead of having this function be duplicated across scripts, you add this function as its own file under the common folder:
╰$ cat static/resources/common/python/my_func

def my_func(str):
print('inside funct - ' + str)

Once the templating engine parses the files, it creates a mapping:
"key" - the filename ("my_func" in this case)
"value" - the content of the file itself

You can then substitute this function in any script using the key:

This will substitute the contents of the file.
More info


The files added under the common folder are also passes through the templating engine, so you can use templating in the files added to the common folder as well, something like this:
echo "{{.values.value1}} {{.values.value2}}"

These values will be parsed through the yaml file provided at runtime.


Sometimes you might want to add indentation to the above substituted lines of code (It is essential in python scripts). You can do so by using nindent (courtesy of sprig functions)



Gotchas


If you add a file under the common folder with an extension, for example, my_func.py, the key still remains my_func. That is because the templating engine gets confused when you try to access a key with a . So in cases where the file has an extension, the engine strips off the extension for the "key" value.


Names with - are not permitted to be used in go templating, therefore filenames with - in them are not permitted.


Adding custom resources
Sometimes you might want to add files under certain modules which you don't want mozart to execute along with the module, instead these files will be used by your scripts.
In such cases, you can prefix ! to the file names (or prefix ! to an entire directory, in which case all files inside that directory will not be executed)
Example:
ls static/resources/templates/example-module/00-bash-module/02-external-example

!external_file.sh
step1.sh

As you see here, there's a file present with the name !external_file.sh. Since this file is prefixed with a !, this won't be executed by mozart when you run the module3 module.
If your script wants to execute this file, you can do so by adding these lines (example code in static/resources/templates/example-module/00-bash-module/02-external-example/step1.sh):
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd -P )"
bash $DIR/!external_file.sh

OR
You can write this file to a custom location and execute it:
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd -P )"
cat $DIR/!external_file.sh > /your/location/file.sh
bash /your/location/file.sh

Note: The DIR command is needed since the execution directory for the script is not where the script resides. So unfortunately you cannot run a script that's present in the same folder using something like ./script_name, since . assumes current execution directory.
CLI
Once you build your binary, Mozart gives you a CLI:
Mozart commands
- init          Generate a blank sample config yaml file for the orchestrator
- execute       (executes all scripts in specified directory)
- state         (displays execution state of all modules, accepts optional args [module name])
- server        Starts the REST server
- version       (displays version info for the application)
--json                (gives output in JSON)

Global flags
-c                  (optional) (configuration file, defaults to 'mozart-sample.yaml')
-v                  (optional) (prints verbosely, useful for debugging)
-d, --dry-run       (optional) shows what scripts will run, but does not run the scripts
-n, --no-generate   (optional) do not generate bash scripts as part of execution, instead use the ones in generated folder. Useful for running local change to the scripts
-p, --parallel      (optional) Run all scripts in parallel
-r, --re-run        (optional) re-run script from initial state, ignoring previously saved state

Executing modules
Running the binary built in the earlier step, you will see something like this:
$ ./bin/mozart execute -h

Execute scripts inside any folder.
*****************************************
Available commands:

mozart execute symphony-module
mozart execute symphony-module pre-req
mozart execute symphony-module install
mozart execute symphony-module validate
mozart execute symphony-module uninstall
*****************************************

If you select a module that contains other modules, something like mozart execute symphony-module, that's where the ordering of the sub-modules comes into play, which you control by adding the prefix. Or you can choose to execute a sub-module directly.
Note: Mozart automatically removes any prefix of the form xx-before the module name.
Checking the state
Once you start the execution, the state command shows you the current state of execution of the various modules within Mozart, along with other information.
State of all modules
$ ./bin/mozart state

State: {
  "generated/symphony-module/00-pre-req": {
    "pre-req.sh": {
      "startTime": "2020-11-04T15:08:12.5981-08:00",
      "timeTaken": "8.218745ms",
      "lastSuccessTime": "2020-11-04 15:08:12.606306 -0800 PST m=+0.016747847",
      "lastErrorTime": "",
      "state": "success",
      "logFilePath": "logs/2020-11-04--15-08-12.597-pre-req.log"
    }
  },
  "generated/symphony-module/10-install": {
    "00-install-step1.sh": {
      "startTime": "2020-11-04T15:08:12.607053-08:00",
      "timeTaken": "9.723926ms",
      "lastSuccessTime": "2020-11-04 15:08:12.616754 -0800 PST m=+0.027195761",
      "lastErrorTime": "",
      "state": "success",
      "logFilePath": "logs/2020-11-04--15-08-12.606-00-install-step1.log"
    },
    "10-install-step2.sh": {
      "startTime": "2020-11-04T15:08:12.617443-08:00",
      "timeTaken": "7.333338ms",
      "lastSuccessTime": "2020-11-04 15:08:12.624767 -0800 PST m=+0.035208922",
      "lastErrorTime": "",
      "state": "success",
      "logFilePath": "logs/2020-11-04--15-08-12.617-10-install-step2.log"
    }
  },
  "generated/symphony-module/20-validate": {
    "validate.sh": {
      "startTime": "2020-11-04T15:08:12.625411-08:00",
      "timeTaken": "7.542653ms",
      "lastSuccessTime": "2020-11-04 15:08:12.632945 -0800 PST m=+0.043386468",
      "lastErrorTime": "",
      "state": "success",
      "logFilePath": "logs/2020-11-04--15-08-12.625-validate.log"
    }
  },
  "generated/symphony-module/30-uninstall": {
    "uninstall.sh": {
      "startTime": "2020-11-04T15:08:12.633649-08:00",
      "timeTaken": "8.040003ms",
      "lastSuccessTime": "2020-11-04 15:08:12.641679 -0800 PST m=+0.052120673",
      "lastErrorTime": "",
      "state": "success",
      "logFilePath": "logs/2020-11-04--15-08-12.633-uninstall.log"
    }
  }
}

State of specific module
To get the state of a particular module:
./mozart state validate

State: {
  "generated/symphony-module/20-validate": {
    "validate.sh": {
      "startTime": "2020-11-04T15:08:12.625411-08:00",
      "timeTaken": "7.542653ms",
      "lastSuccessTime": "2020-11-04 15:08:12.632945 -0800 PST m=+0.043386468",
      "lastErrorTime": "",
      "state": "success",
      "logFilePath": "logs/2020-11-04--15-08-12.625-validate.log"
    }
  }

UI
Developed by @toshakamath


Helpful resources
Good links for templating

https://golang.org/pkg/text/template/
https://forum.golangbridge.org/t/referencing-map-values-in-text-template/6253/5
https://medium.com/@IndianGuru/understanding-go-s-template-package-c5307758fab0
https://goinbigdata.com/example-of-using-templates-in-golang/
https://stackoverflow.com/questions/25689829/arithmetic-in-go-templates
https://stackoverflow.com/questions/21305865/golang-separating-items-with-comma-in-template
https://helm.sh/docs/chart_template_guide/functions_and_pipelines/

Bash scripts through go

https://blog.kowalczyk.info/article/wOYk/advanced-command-execution-in-go-with-osexec.html
https://stackoverflow.com/questions/18986943/in-golang-how-can-i-write-the-stdout-of-an-exec-cmd-to-a-file
https://stackoverflow.com/questions/19965795/how-to-write-log-to-file
https://yourbasic.org/golang/format-parse-string-time-date-example/

Versioning with go

https://www.atatus.com/blog/golang-auto-build-versioning/

Web sockets

https://www.zupzup.org/io-pipe-go/
https://gist.github.com/ifels/10392762
https://godoc.org/github.com/gorilla/websocket

Command execution

https://medium.com/@vCabbage/go-timeout-commands-with-os-exec-commandcontext-ba0c861ed738
https://stackoverflow.com/a/58572436
https://medium.com/@matryer/make-ctrl-c-cancel-the-context-context-bd006a8ad6ff

Error stack

https://github.com/gruntwork-io/terratest/blob/master/modules/helm/template.go#L83
https://godoc.org/github.com/go-errors/errors
https://github.com/gruntwork-io/gruntwork-cli/blob/master/errors/errors.go

Embedding static content

https://github.com/rakyll/statik/pull/101/files
golang/go#41191
https://blog.carlmjohnson.net/post/2021/how-to-use-go-embed/
https://github.com/akmittal/go-embed (example for react app)

Go build tools

https://blog.carlmjohnson.net/post/2021/how-to-use-go-embed/
https://dave.cheney.net/2013/10/12/how-to-use-conditional-compilation-with-the-go-build-tool
https://www.digitalocean.com/community/tutorials/customizing-go-binaries-with-build-tags
List all go files: go list -tags ui -f '{{.GoFiles}}' ./...

