Add script to help CI builds

Add script to help CI builds

The script implements steps for (1) setup, (2) build, and (3) gathering
results, including logic for special cases.  It also enables (4) more
extensive tweaking and parameterization of the build pipelines in
combination with Go.CD parameters and environment variables.

(1) Examples of setup steps are to add DL_DIR and SSTATE_DIR settings to
configuration, and to copy binary drivers into the project when that is
required.

(2) Examples of build steps are to run init.sh and bitbake with the
required parameters that could differ depending on build type.

(3) Examples of gathering results are to copy or move all artifacts from
diverse places in the build tree into a staging directory which can be
defined as a single place for the CI system to collect artifacts, in a
way that is robust so that it does not cause the pipeline to fail if
something is (allowed to be) missing in a particular configuration.  In
the future I expect also storing more build-statistics and information
could be added to this category.

(4) Examples of parameterization are that the chosen material can be
overridden by environment variables, forcing a particular fork, branch,
tag, or commit hash to be built, and that reuse of cached material can
be enabled/disabled similarly.  The script also stores sensible defaults
for such options in a single place.

We should expect frequent updates and improvements to this, to move more
complexity and special needs to a single place.

The reason for this is that previously much of this logic was encoded in
Tasks inside Jobs in Go.CD CI.  Initially the pipelines were simple and
usage of Go.CD Templates reduced the work somewhat, but over time
several special cases and other details have created complexity that
causes frequent breakage.  The script might also be useful locally *if*
the user carefully understands what it does.  For that reason it has a
catch that warns the user and makes it almost impossible to run "by
mistake".

Signed-off-by: Gunnar Andersson <df6c1dc70ee3404c16d7de0a5a8d978b44fa18d4@genivi.org>

#!/bin/bash

# (C) 2017 Gunnar Andersson
# LICENSE: MPLv2

# User warning!

# A casual user will need to read the script to understand what the first
# parameter shall be to make it run:
#
if [[ "$1" != CI_FLAG ]]; then

  cat <<EOT

READ THIS: This script was not designed for personal use.

If you still use it, you hereby agree to any applicable End-User License
Agreement (EULA) of the corresponding BSPs that are used.

It is still a useful script but a few things could be destructive if you use
this in your development environment instead of a "throw-away" environment like
CI.

Please make sure you read the script to understand what it does!

After that I hope you enjoy using it.

EOT

   exit 2
fi

# ---- General settings ----

# Fixed cache location on all agents that support this.
AGENT_STANDARD_DL_DIR="/var/cache/yocto/downloads"
AGENT_STANDARD_SSTATE_DIR="/var/cache/yocto/sstate"
AGENT_STANDARD_SGX_LOCATION="/var/go/sgx_bin"
AGENT_STANDARD_SGX_GEN3_LOCATION="/var/go/sgx_bin_gen3"

# ---- Helper functions ----

# This is some trickery, but quite useful
deref() { eval echo \$$1 ; }

ensure_var_is_defined() {
  var_name=$1
  current_value=$(deref $var_name)
  if [[ -z "$current_value" ]]; then
    echo "Please define environment var_name \$$var_name (found empty value)"
    fail=true  # We want all the errors reported, so don't halt yet.
  fi
}

define_with_default() {
  var_name=$1
  default="$2"
  current_value=$(deref $var_name)
  if [[ -z "$current_value" ]]; then
    eval $var_name="$default"
  fi
}

stop_if_failure() { [[ "$fail" == "true" ]] && exit 1 ; }

stop_immediately() {
  echo "Fatal error occurred - stopping ci-build script"
  exit 2
}

append_local_conf() {
  LOCAL_CONF="$BASEDIR/gdp-src-build/conf/local.conf"
  if [[ -f "$LOCAL_CONF" ]]; then
    echo -n "Appending to local.conf: "
    cat <<EOT | tee -a "$LOCAL_CONF"
$1
EOT
  else
    echo "Fatal: Did not find local.conf where expected"
    stop_immediately
  fi
}

stage_artifact() {
  mkdir -p staging
  cmd=$1
  shift

  if [[ -z "$1" ]] ; then
      echo "Skipped an artifact pattern that matched nothing"
      return
  fi

  for f in $@ ; do  # <- could be empty glob list
    if [[ -e "$f" ]] ; then
      if [[ $cmd == cp && -d "$f" ]] ; then
        flag=-a
      elif [[ $cmd == cp ]] ; then
        flag=--preserve=all
      fi
      $cmd $flag "$f" staging/
    else
      echo "Skipping non-existing (or already moved) artifact: $f"
    fi
  done
}

# ---- Main program ----

D=$(dirname "$0")
cd "$D"
BASEDIR=$(readlink -f "$PWD/..")

# The user must define this
ensure_var_is_defined TARGET

stop_if_failure

# The values can be overridden by defining environment variables
# If no value given, use this default:
define_with_default BUILD_SDK false
define_with_default RM_WORK false
define_with_default REUSE_STANDARD_DL_DIR true
define_with_default REUSE_STANDARD_SSTATE_DIR true
define_with_default SGX_DRIVERS $AGENT_STANDARD_SGX_LOCATION
define_with_default SGX_GEN_3_DRIVERS $AGENT_STANDARD_SGX_GEN3_LOCATION

stop_if_failure

git_gdp="https://github.com/GENIVI/genivi-dev-platform"
branch="master"

# For copying graphics
if [[ "$TARGET" == "koelsch" ]]; then
  GFX_MACHINE=porter
else
  GFX_MACHINE=$TARGET
fi

# cd workingdir
MACHINE="$TARGET" # For most boards - exceptions handled below

if [[ "$TARGET" == "r-car-m3-starter-kit" ]]; then
  MACHINE="m3ulcb"
fi

ensure_var_is_defined MACHINE
export MACHINE

if [[ "$REUSE_STANDARD_DL_DIR" == "true" ]]; then
  DL_DIR="$AGENT_STANDARD_DL_DIR"
fi

if [[ "$REUSE_STANDARD_SSTATE_DIR" == "true" ]]; then
  SSTATE_DIR="$AGENT_STANDARD_SSTATE_DIR"
fi

echo Configuration:
echo
echo "TARGET = $TARGET"
echo "BRANCH = $BRANCH"
echo "COMMIT = $COMMIT"
echo "RELEASE = $RELEASE"
echo "BUILD_SDK = $BUILD_SDK"
echo "SGX_DRIVERS = $SGX_DRIVERS"
echo "SGX_GEN_3_DRIVERS = $SGX_GEN_3_DRIVERS"
echo "REUSE_STANDARD_DL_DIR = $REUSE_STANDARD_DL_DIR"
echo "REUSE_STANDARD_SSTATE_DIR = $REUSE_STANDARD_SSTATE_DIR"
echo "DL_DIR = $DL_DIR"
echo "SSTATE_DIR = $DL_DIR"
echo "RM_WORK = $RM_WORK"

# build steps
cd "$BASEDIR/gdp-src-build"

# If DL/SSTATE are to be reused it is only through. the use of
# REUSE_STANDARD_DL_DIR and REUSE_STANDARD_SSTATE_DIR -  I.e. if any are left
# in the build directory we assume they shall be wiped out and not reused.

rm -rf tmp downloads cache sstate-cache

cd "$BASEDIR"

# Need to set an identity for some git patching done by recipes
git config user.name "CI build -- ignore"
git config user.email no_email@genivigo.com

# Normally the material (source code) is defined in the pipeline itself in the
# CIAT system but there are multiple ways to override it provided here.

if [[ -n "$FORK" ]]; then
  echo "***** NOTE: OVERRIDING REPOSITORY WITH \$FORK DEFINITION *****"
  git remote set-url origin "$FORK"
  git fetch
  # V *danger* V.  One reason why you should not use this script if it's not in CI
  git reset origin/master --hard
  # After this, submodule sync/update will be done by init.sh
  # this may need some further testing!
  git checkout $BRANCH  # <- note this should be ok even if $BRANCH is an empty value
fi

if [[ -n "$BRANCH" ]]; then
  echo "***** NOTE: OVERRIDING CHOSEN COMMIT USING \$BRANCH DEFINITION *****"
  git checkout $BRANCH
fi

if [[ -n "$TAG" ]]; then
  echo "***** NOTE: OVERRIDING CHOSEN COMMIT USING \$TAG DEFINITION *****"
  git checkout $TAG
fi

if [[ -n "$COMMIT" ]]; then
  echo "***** NOTE: OVERRIDING CHOSEN COMMIT USING \$COMMIT DEFINITION *****"
  git checkout $COMMIT
fi

# Remind us exactly what submodules hashes are used (this is already stated by
# go.cd when fetching materials, but materials can be overriden by FORK /
# BRANCH / TAG / COMMIT
echo "Submodules:"
git submodule

# Deal with special setup, copy binary drivers etc.
set -x
if [[ "$TARGET" == "r-car-m3-starter-kit" ]];  then
  cd renesas-rcar-gen3
  meta-rcar-gen3/docs/sample/copyscript/copy_evaproprietary_softwares.sh /var/go/sgx_bin_gen3/
fi

if [[ "$GFX_MACHINE" == "porter" || "$GFX_MACHINE" == silk ]]; then
  cd meta-renesas/meta-rcar-gen2
  ./copy_gfx_software_$GFX_MACHINE.sh /var/go/sgx_bin
  ./copy_mm_software_lcb.sh /var/go/sgx_bin/
fi
set +x

# INIT
echo "Running init.sh"
if [[ "$TARGET" == "dragonboard-410c" ]]; then
  source ./init.sh $TARGET accept-eula -f
else
  source ./init.sh $TARGET -f
fi

# LOCAL CONF MODIFICATIONS
if [[ "$RM_WORK" == "true" ]]; then
  append_local_conf 'INHERIT += "rm_work"'
fi

if [[ -n "$DL_DIR" ]]; then
  append_local_conf "DL_DIR = \"$DL_DIR\""
fi

if [[ -n "$SSTATE_DIR" ]]; then
  append_local_conf "SSTATE_DIR = \"$SSTATE_DIR\""
fi

if [[ -n "$BB_NUMBER_THREADS" ]]; then
  append_local_conf "BB_NUMBER_THREADS = \"$BB_NUMBER_THREADS\""
fi

if [[ -n "$PARALLEL_MAKE" ]]; then
  echo $PARALLEL_MAKE | egrep -q '^-j' || PARALLEL_MAKE="-j$PARALLEL_MAKE"
  append_local_conf "PARALLEL_MAKE = \"$PARALLEL_MAKE\""
fi

if [[ "$BUILD_SDK" != "true" ]]; then
  bitbake genivi-dev-platform
fi

if [[ "$BUILD_SDK" == "true" ]]; then
  bitbake genivi-dev-platform-sdk
fi

if [[ "$BUILD_SDK" == "true" ]]; then
  bitbake meta-ide-support
fi

rm -f logs.tar logs.tar.gz
find gdp-src-build/tmp/work \( -name "*.log" -o -name "log.*" -o -name "run.*" \) -print0 | xargs -0 tar uf logs.tar || true
gzip logs.tar || true

cd "$BASEDIR"
rm -rf staging
shopt -s nullglob
stage_artifact mv gdp-src-build/tmp/deploy/licenses
stage_artifact mv gdp-src-build/tmp/deploy/licenses/genivi-dev-platform*/license.manifest
stage_artifact mv gdp-src-build/tmp/deploy/sdk*
stage_artifact cp gdp-src-build/tmp/deploy/images/*/*
stage_artifact cp gdp-src-build/tmp/deploy/images/*
stage_artifact cp gdp-src-build/conf/*.conf
stage_artifact cp logs.tar.gz

mkdir -p staging/images
mv staging/{*201*ext*,*201*rootfs*,*sdimg*,*hddimg*,bzImage*201*} staging/images/ 2>/dev/null || true

echo Artifacts in staging/
ls -R staging/

error: pathspec 'scripts/ci-build.sh' did not match any file(s) known to git


Deploy updates

Deploy updates

#!/bin/bash
directory=build
branch=gh-pages
build_command() {
  npm run build
}

echo -e "\033[0;32mDeleting old content...\033[0m"
rm -rf $directory

echo -e "\033[0;32mChecking out $branch....\033[0m"
git worktree add $directory $branch

echo -e "\033[0;32mGenerating site...\033[0m"
build_command

echo -e "\033[0;32mDeploying $branch branch...\033[0m"
cd $directory &&
  git add --all &&
  git commit -m "Deploy updates" &&
  git push origin $branch

echo -e "\033[0;32mCleaning up...\033[0m"
git worktree remove $directory

error: pathspec 'scripts/gh-pages.sh' did not match any file(s) known to git


added ORACLE_HOME support

added ORACLE_HOME support

# Set up ORACLE_HOME
export ORACLE_HOME="${HOME}/opt/oracle/instantclient_11_2"

# Extend DYLD_LIBRARY_PATH
export DYLD_LIBRARY_PATH="${ORACLE_HOME}${DYLD_LIBRARY_PATH+:}${DYLD_LIBRARY_PATH}"

error: pathspec 'oracle/env.zsh' did not match any file(s) known to git


Keep swift build script invocation for lldb around.

#!/usr/bin/env bash

swift/utils/build-script \
  --lldb \
  --release \
  --test \
  -- \
  --skip-build-benchmarks \
  --no-swift-stdlib-assertions \
  --lldb-use-system-debugserver \
  --lldb-build-with-cmake \
  --skip-test-swift \
  --skip-test-cmark

error: pathspec 'scripts/swift-lldb.sh' did not match any file(s) known to git


Added unix script to launch pi application

git-svn-id: 9146c88ff6d39b48099bf954d15d68f687b3fa69@4150 28e8926c-6b08-0410-baaa-805c5e19b8d6

#!/bin/sh

workingDir=.

PROACTIVE=$workingDir/../../../.
. $PROACTIVE/scripts/unix/env.sh

$JAVACMD -Xmx256000000 -classpath $JAVA_HOME/lib/tools.jar:$PROACTIVE/compile/ant.jar:$PROACTIVE/compile/ant-launcher.jar:$PROACTIVE/compile/xml-apis.jar:$PROACTIVE/compile/xercesImpl.jar org.apache.tools.ant.Main -buildfile $PROACTIVE/src/org/objectweb/proactive/examples/pi/scripts/build.xml "$@"

error: pathspec 'scripts/unix/pi/pi.sh' did not match any file(s) known to git


ElasticSearch Korean 관련 설정

ElasticSearch Korean 관련 설정

#!/bin/bash
curl -XPUT http://localhost:9200/korean/ -d '{
  "settings": {
    "analysis": {
      "analyzer": {
        "korean" : {
          "type" : "standard",
          "tokenizer": "korean_tokenizer",
          "filter" : ["synonym", "my_keyword", "stop", "lowercase", "trim"]
        },
        "ngram_analyzer" : {
            "type" : "custom",
            "tokenizer" : "ngram_tokenizer",
            "filter" : ["synonym", "my_keyword", "lowercase", "trim"]
        },
        "edge_ngram_analyzer" : {
            "type" : "custom",
            "tokenizer" : "edge_ngram_tokenizer",
            "filter" : ["synonym", "my_keyword", "lowercase", "trim"]
        }
      },
      "tokenizer": {
        "korean_tokenizer": {
          "type": "mecab_ko_standard_tokenizer",
          "compound_noun_min_length": 100
        },
        "ngram_tokenizer" : {
            "type" : "nGram",
            "min_gram" : "1",
            "max_gram" : "5",
            "token_chars": [ "letter", "digit", "punctuation", "symbol" ]
        },
        "edge_ngram_tokenizer" : {
            "type" : "edgeNGram",
            "min_gram" : "1",
            "max_gram" : "5",
            "token_chars": [ "letter", "digit", "punctuation", "symbol" ]
        }
      },
      "filter": {
        "my_keyword" : {
            "type" : "keyword_marker",
            "keywords" : ["ttt"]
        },
        "synonym" : {
            "type" : "synonym",
            "synonyms" : [
                "11번가, 11st",
                "g마켓, 지마켓"
            ]
        }
      }
    }
  },
  "mappings": {
    "movie" : {
      "analyzer" : "korean",
      "index_analyzer" : "edge_ngram_analyzer",
      "search_analyzer" : "edge_ngram_analyzer",
      "properties": {
        "title" : {
          "type": "string",
          "fields" : {
            "h_pf" : {"type" : "string", "store" : "no", "index" : "not_analyzed", "omit_norms" : true, "index_options" : "offsets", "term_vector" : "with_positions_offsets", "include_in_all" : false, "boost": 20},
            "h_ko" : {"type" : "string", "index_analyzer" : "korean", "boost": 3},
            "h_ng" : {"type" : "string", "index_analyzer" : "ngram_analyzer", "boost": 1},
            "h_ed" : {"type" : "string", "index_analyzer" : "edge_ngram_analyzer", "boost": 2}
          }
        }
      }
    }
  }
}'

curl -XPOST 'http://localhost:9200/korean/text' -d '{"text": "아버지가 방에 들어간다"}'
curl -XGET http://localhost:9200/korean16/_analyze\?analyzer\=korean\&pretty\=true -d '아버지가 방에 들어간다.'

error: pathspec 'elasticsearch_korean/setup.sh' did not match any file(s) known to git


setup-root: remove call to ldconfig

setup-root: remove call to ldconfig

Recently added in commit 82b6d9ee, in addition to initializing
ld.so.conf that commit initialized ld.so.cache on each boot.
Unfortunately this may clobber references to libraries in /usr/share/oem
which is not mounted by the initrd. systemd provides ldconfig.service to
perform this task and selectively runs it on boots after upgrades after
/usr/share/oem is mounted. In production images all libraries
required to boot as far as sysinit.target are in the default library
path so we can get away without ld.so.cache until ldconfig.service runs.

#!/bin/sh
# -*- mode: shell-script; indent-tabs-mode: nil; sh-basic-offset: 4; -*-
# ex: ts=8 sw=4 sts=4 et filetype=sh

# /etc/machine-id after a new image is created:
COREOS_BLANK_MACHINE_ID="42000000000000000000000000000042"
MACHINE_ID_FILE="/sysroot/etc/machine-id"

# Run and log a command
bootengine_cmd() {
    ret=0
    "$@" >/tmp/bootengine.out 2>&1 || ret=$?
    vinfo < /tmp/bootengine.out
    if [ $ret -ne 0 ]; then
        warn "bootengine: command failed: $*"
        warn "bootengine: command returned $ret"
    fi
    return $ret
}

do_setup_root() {
    if ! bootengine_cmd mount -o remount,rw /sysroot; then
        warn "bootengine setup root: Can't remount root rw"
        return 0
    fi

    # Initialize base filesystem
    bootengine_cmd systemd-tmpfiles --root=/sysroot --create \
        baselayout.conf baselayout-etc.conf baselayout-usr.conf

    # Not all images provide this file so check before using it.
    if [ -e "/sysroot/usr/lib/tmpfiles.d/baselayout-ldso.conf" ]; then
        bootengine_cmd systemd-tmpfiles --root=/sysroot --create \
            baselayout-ldso.conf
    fi

    # Remove our phony id. systemd will initialize this during boot.
    if grep -qs "${COREOS_BLANK_MACHINE_ID}" "${MACHINE_ID_FILE}"; then
        bootengine_cmd rm "${MACHINE_ID_FILE}"
    fi
}

# Skip if root and root/usr are not mount points
if ismounted /sysroot && ismounted /sysroot/usr; then
    do_setup_root
fi

# PXE initrds may provide OEM
if [ -d /usr/share/oem ] && ismounted /sysroot/usr/share/oem; then
    cp -Ra /usr/share/oem/. /sysroot/usr/share/oem
fi

#!/bin/sh
# -*- mode: shell-script; indent-tabs-mode: nil; sh-basic-offset: 4; -*-
# ex: ts=8 sw=4 sts=4 et filetype=sh

# /etc/machine-id after a new image is created:
COREOS_BLANK_MACHINE_ID="42000000000000000000000000000042"
MACHINE_ID_FILE="/sysroot/etc/machine-id"

# Run and log a command
bootengine_cmd() {
    ret=0
    "$@" >/tmp/bootengine.out 2>&1 || ret=$?
    vinfo < /tmp/bootengine.out
    if [ $ret -ne 0 ]; then
        warn "bootengine: command failed: $*"
        warn "bootengine: command returned $ret"
    fi
    return $ret
}

do_setup_root() {
    if ! bootengine_cmd mount -o remount,rw /sysroot; then
        warn "bootengine setup root: Can't remount root rw"
        return 0
    fi

    # Initialize base filesystem
    bootengine_cmd systemd-tmpfiles --root=/sysroot --create \
        baselayout.conf baselayout-etc.conf baselayout-usr.conf

    # Not all images provide this file so check before using it.
    if [ -e "/sysroot/usr/lib/tmpfiles.d/baselayout-ldso.conf" ]; then
        bootengine_cmd systemd-tmpfiles --root=/sysroot --create \
            baselayout-ldso.conf
        bootengine_cmd ldconfig -X -r /sysroot
    fi

    # Remove our phony id. systemd will initialize this during boot.
    if grep -qs "${COREOS_BLANK_MACHINE_ID}" "${MACHINE_ID_FILE}"; then
        bootengine_cmd rm "${MACHINE_ID_FILE}"
    fi
}

# Skip if root and root/usr are not mount points
if ismounted /sysroot && ismounted /sysroot/usr; then
    do_setup_root
fi

# PXE initrds may provide OEM
if [ -d /usr/share/oem ] && ismounted /sysroot/usr/share/oem; then
    cp -Ra /usr/share/oem/. /sysroot/usr/share/oem
fi


Tested and removed the premature exits

Tested and removed the premature exits

error: pathspec 'Disable' did not match any file(s) known to git
error: pathspec 'MDM' did not match any file(s) known to git
error: pathspec 'Debugging.sh' did not match any file(s) known to git


Bash wrapper

Bash wrapper

configure it once; use forever

#!/usr/bin/bash

# edit part of the line below to reflect where you git cloned rd_filters
export FILTERS_RULES_DATA=~/src/rd_filters/rd_filters/data

rd_filters "$@"

error: pathspec 'rd_filters.sh' did not match any file(s) known to git


emacs: add doom's binaries to PATH

emacs: add doom's binaries to PATH

#!/usr/bin/env bash

# doom's binaries
export PATH=$HOME/.emacs.d/bin:$PATH

error: pathspec 'emacs/.shell/login.d/emacs.sh' did not match any file(s) known to git


jetty debug script

jetty debug script

#!/bin/bash
mvn -DMAVEN_OPTS="-Xmx1024m -Xms512m -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,address=4000,server=y,suspend=y" -Djetty.port=8080 jetty:run

error: pathspec 'encuestame-war/jetty-debug.sh' did not match any file(s) known to git


Added dashes to rvm commands specifying version number

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/jhsu802701/packer-debian-jessie-old.git/'


Bash script for tagging a release

Bash script for tagging a release

#!/bin/bash

# Fail the script on error
set -e

read -p "Semantic version (without the 'v'): " VERSION

if [[ $VERSION == v* ]]
then
  echo ERROR: Please specify the raw semantic version without a 'v' prefix eg. x.y.z
  exit 1
fi

REVISION=`git rev-parse --short HEAD`
echo Starting to tag $REVISION as v$VERSION...

git tag -a v$VERSION -m "Release v$VERSION"
if [ $? -eq 1 ]; then
  echo "Failed to tag; exiting..."
  exit 1
fi

echo Finished!
echo "NOTE: Remember to execute (when satisfied): git push origin v$VERSION"
echo

error: pathspec 'release-it.sh' did not match any file(s) known to git


Create vim_cmds.sh

Create vim_cmds.sh

# Where grep came from (RE being Regular Expression):
:g/RE/p

# Delete lines 10 to 20 inclusive:
:10,20d
# or with marks a and b:
:'a,'bd

# Delete lines that contain pattern:
:g/pattern/d

# Delete all empty lines:
:g/^$/d

# Delete lines in range that contain pattern:
:20,30/pattern/d
# or with marks a and b:
:'a,'b/pattern/d

# Substitute all lines for first occurance of pattern:
:%s/pattern/new/
:1,$s/pattern/new/

# Substitute all lines for pattern globally (more than once on the line):
:%s/pattern/new/g
:1,$s/pattern/new/g

# Find all lines containing pattern and then append -new to the end of each line:
:%s/\(.*pattern.*\)/\1-new/g

# Substitute range:
:20,30s/pattern/new/g
# with marks a and b:
:'a,'bs/pattern/new/g

# Swap two patterns on a line:
:s/\(pattern1\)\(pattern2\)/\2\1/

# Capitalize the first lowercase character on a line:
:s/\([a-z]\)/\u\1/
# more concisely:
:s/[a-z]/\u&/

# Capitalize all lowercase characters on a line:
:s/\([a-z]\)/\u\1/g
# more concisely:
:s/[a-z]/\u&/g

# Capitalize all characters on a line:
:s/\(.*\)/\U\1\E/

# Capitalize the first character of all words on a line:
:s/\<[a-z]/\u&/g

# Uncapitalize the first character of all words on a line:
:s/\<[A-Z]/\l&/g

# Change case of character under cursor:
~

# Change case of all characters on line:
g~~

# Change case of remaining word from cursor:
g~w

# Increment the number under the cursor:
<Ctrl-A>

# Decrement the number under the cursor:
<Ctrl-X>

# redraw:
<Ctrl-L>

# Turn on line numbering:
:set nu
# Turn it off:
:set nonu

# Number lines (filter the file through a unix command and replace with output):
:%!cat -n

# Sort lines:
:%!sort

# Sort on column #69:
:sort /.*\%69v/

# Sort and uniq:
:%!sort -u

# Read output of command into buffer:
:r !ls -l

# Refresh file from version on disk:
:e!

# Open a new window:
<Ctrl-W>n

# Open a new window with the same file (split):
<Ctrl-W>s

# Split window vertically:
<Ctrl-W>v

# Close current window:
<Ctrl-W>c
:q

# Make current window the only window:
<Ctrl-W>o

# Cycle to next window:
<Ctrl-W>w

# Move to window below current window:
<Ctrl-W>j

# Move to window above current window:
<Ctrl-W>k

# Move to window left of current window:
<Ctrl-W>h

# Move to window right of current window:
<Ctrl-W>l

# Set textwidth for automatic line-wrapping as you type:
:set textwidth=80

# Turn on syntax highlighting
:syn on
# Turn it off:
:syn off

# Force the filetype for syntax highlighting:
:set filetype=python
:set filetype=c
:set filetype=php

# Use lighter coloring scheme for a dark background:
:set background=dark

# Htmlize a file using the current syntax highlighting:
:so $VIMRUNTIME/syntax/2html.vim

# Or, htmlize from a command prompt:
in 2html.sh put:

#!/bin/sh
vim -n -c ':so $VIMRUNTIME/syntax/2html.vim' -c ':wqa' $1 > /dev/null 2> /dev/null

#Now just run:
shell> 2html.sh foo.py

error: pathspec 'linux/vim_cmds.sh' did not match any file(s) known to git


macos/aliases: rm git pulls from updates

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/cameronmalek/dotfiles.git/'


Fixed bug with ctags not being applied as part of main install process.

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/bkuhlmann/ruby_setup.git/'


Script to benchmark compilation time

Script to benchmark compilation time

#!/bin/bash

make clean > /dev/null

echo "Compile 1 RBM"
time make release_debug/bin/dll_compile_rbm_one > /dev/null

echo "Compile 1 Dynamic RBM"
time make release_debug/bin/dll_compile_dyn_rbm_one > /dev/null

echo "Compile 5 RBMs"
time make release_debug/bin/dll_compile_rbm > /dev/null

echo "Compile 5 Dynamic RBMs"
time make release_debug/bin/dll_compile_dyn_rbm > /dev/null

error: pathspec 'tools/compile_dyn.sh' did not match any file(s) known to git


update count_max.bash for osx

update count_max.bash for osx

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/j4ve1in/dotfiles.git/'


Add Installed Pkgs script

Add Installed Pkgs script

#!/bin/bash
#pkginstalls.sh
#creates text file with a list of all packages installed by date
#ref: https://help.ubuntu.com/community/ListInstalledPackagesByDate

#first append all info from archived logs

i=2
mycount=$(ls -l /var/log/dpkg.log.*.gz | wc -l)
nlogs=$(( $mycount + 1 ))

while [ $i -le $nlogs ]
do
if [ -e /var/log/dpkg.log.$i.gz ]; then
zcat /var/log/dpkg.log.$i.gz | grep "\ install\ " >> $HOME/pkgtmp.txt
fi
i=$(( $i+1 ))

done

#next append all info from unarchived logs

i=1
nulogs=$(ls -l /var/log/dpkg.log.* | wc -l)
nulogs=$(( $nulogs - $nlogs + 1 ))
while [ $i -le $nulogs ]
do
if [ -e /var/log/dpkg.log.$i ]; then
cat /var/log/dpkg.log.$i | grep "\ install\ " >> $HOME/pkgtmp.txt
fi
i=$(( $i+1 ))

done

#next append current log

cat /var/log/dpkg.log | grep "\ install\ " >> $HOME/pkgtmp.txt

#sort text file by date

sort -n $HOME/pkgtmp.txt > $HOME/pkginstalls.txt

rm $HOME/pkgtmp.txt

exit 0

error: pathspec 'pkginstalls.sh' did not match any file(s) known to git


Added DPI change to set-settings

Added DPI change to set-settings

#!/bin/bash

echo "xrandr --dpi 138/eDP1" >> /etc/X11/xinit/xinitrc

error: pathspec 'setup/set-settings.sh' did not match any file(s) known to git


Add pastel recipe

Add pastel recipe

GIT=https://github.com/robbycerantola/pastel.git

function recipe_stage {
    cp -rv res "$1/ui/pastel"
}

error: pathspec 'recipes/pastel/recipe.sh' did not match any file(s) known to git


ES population script

ES population script

#!/usr/bin/env bash

data_file="input.json"
ids=()
idx="es_idx_name"
estype="es_type_name"
es_host="localhost:9200"
id_key="_id"
confirm=false

usage() {
    local ex_code
    ex_code="${1:-0}"
    cat <<-__EOF_
${0##*/} [-i idx] [-t type] [-f file] [-H host ] [-k id key] [-c] [-h]
Add documents in file to ES
flags:

         -i index: ES index name
         -t type : ES document type
         -f file : json input file (defaults to input.json)
         -H host : ES host (defaults to localhost:9200)
         -k key  : document id key in json file (defaults to _id)
         -c      : Ask for confirmation (default false)
         -h      : display this message

Example JSON format:
[
    {
        "_id": "123",
        "field1": "foobar",
        "nested": {
            "inner": "string field",
            "another": 123,
        }
    },
    {
        "_id": "456",
        "field1": "second document",
        "nested" {
            "inner": "...",
            "another": 0
        }
    },
    {}
]

Note: The trailing empty object is necessairy, as is quoting the id (although we're working on this)
__EOF_

    exit "${ex_code}"
}

jq_installed() {
    command -v jq >/dev/null 2>&1 && return
    echo >&2 "jq not found, ${0##*/} depends on jq to parse the json file"
    exit 2
}

get_ids() {
    for id in $(jq -r ".[] | .${id_key}" "${data_file}"); do
        ids+=("${id}")
    done
}

get_payload_for_id() {
    local id
    local payload
    id="${1}"
    [ -z "${id}" ] && echo "No id given" && exit 1
    payload=$(sed -n "/\"${id_key}\": \"${id}\"/,/},/p" "${data_file}")
    [ ! -z "${payload}" ] && echo "{${payload%,}"
}

prompt_curl() {
    local id
    local payload
    local resp
    id="${1}"
    payload="${2}"
    read -p "Insert document ID ${id}? [Y/n/s] " -r -n 1 resp
    resp="${resp:-y}"
    if [[ "${resp}" =~ ^[sS]$ ]]; then
        echo "Showing document: "
        echo "${payload}"
        local ret
        ret=$(prompt_curl "${id}" "${payload}" )
        return "${ret}"
    fi
    if [[ "${resp}" =~ ^[nN]$ ]]; then
        return 1
    fi
    return 0
}

do_curl() {
    local payload
    for id in "${ids[@]}"; do
        payload=$(get_payload_for_id "${id}")
        if $confirm && ! prompt_curl "${id}" "${payload}"; then
            echo "skipping ${id}"
            continue
        fi
        curl -XPUT "${es_host}/${idx}/${estype}/${id}?pretty" \
            -H 'Content-Type: application/json' \
            -d "${payload}"
    done
}

## Check prerequisites first
jq_installed

if [ "$#" -gt 0 ]; then
    while getopts :i:t:f:H:k:ch opt; do
        case $opt in
            i)
                idx="${OPTARG}"
                ;;
            t)
                estype="${OPTARG}"
                ;;
            f)
                data_file="${OPTARG}"
                ;;
            H)
                es_host="${OPTARG}"
                ;;
            k)
                id_key="${OPTARG}"
                ;;
            c)
                confirm=true
                ;;
            h)
                usage 0
                ;;
            *)
                echo >&2 "Unknown option ${OPTARG}"
                usage 1
                ;;
        esac
    done
fi

echo "Extracting ID's from file"
get_ids
echo "Found ID's:"
for id in "${ids[@]}"; do
    echo "${id}"
done
echo "Adding documents to ES"
do_curl
echo "Done"

error: pathspec 'populate-es.sh' did not match any file(s) known to git


remove brew_install Safari Technology Preview

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/zlot/dotfiles.git/'


note security vulnerability

note security vulnerability

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/Daplie/walnut.git/'


Add PR summary generator script

Add PR summary generator script

#!/usr/bin/env sh

if [ -z $1 ]
then
    echo "pr2relnotes.sh: prints list of pull requests merged since <tag>"
    echo "  usage: $0 <tag> [pull-request-url (default: https://github.com/lrascao/rebar3_appup_plugin/pull/)]"
    exit 0
fi
export url=${2:-"https://github.com/lrascao/rebar3_appup_utils/pull/"}

git log --merges --pretty=medium $1..HEAD | \
awk -v url=$url '
    # first line of a merge commit entry
    /^commit / {mode="new"}

    # merge commit default message
    / +Merge pull request/ {
        page_id=substr($4, 2, length($4)-1);
        mode="started";
        next;
    }

    # line of content including title
    mode=="started" && /    [^ ]+/ {
        print "- [" substr($0, 5) "](" url page_id ")"; mode="done"
    }'

error: pathspec 'pr2relnotes.sh' did not match any file(s) known to git


`prepare-db.zsh` helper

`prepare-db.zsh` helper

#!/usr/bin/env zsh

echo "chrome push key please"
read KEY

echo "\
set g:berlin '{\"k\":\"4cca53e7-e9ef-4c16-bdfb-934ed4d20c6d\",\"l\":false}'\n
set u:berlin:abcdefg '{\"s\":\"chrome\",\"t\":\"$KEY\"}'\n
set m:berlin:abcdefgh '{\"d\":1443118824661,\"b\":\"test\"}'\n
exit\n" | redis-cli

error: pathspec 'prepare-db.zsh' did not match any file(s) known to git


add TPU-enabled Jupyter notebook creation on GCP's VertexAI Workbench

#!/usr/bin/env bash

# please configure your login, project region and zone first by running the "gcloud init" command
# region us-central1 zone us-central1-b usually have TPUs available

DEFAULT_TF_VERSION="2.9.2" # WARNING: for TF 2.3, the DLVM release is tf2-2-3-cpu but contains TF 2.3.1
IMAGE_FAMILY_PATTERN="tf-2-x-cpu"

usage()
{
    echo "Usage: create-tpu-deep-learning-vm.sh vm-name [ --machine-type | --tpu-type | --help | --nightly | --version ]"
    echo "The default machine type is n1-standard-8."
    echo "The default TPU type is v2-8."
    echo "The default Tensorflow version is $DEFAULT_TF_VERSION."
    echo "Supported Tensorflow versions are 2.1, 2.1.4, 2.2, 2.2.3, 2.3, 2.3.4, 2.4.0 to 2.4.4, 2.5.0 to 2.5.2, 2.6.0, 2.7.0, 2.7.1, 2.7.3, 2.8.0, 2.8.2, 2.9.1  and nightly."
    echo "You can use \"--version nightly\" or \"--nightly\" to obtain a nightly version of Tensorflow on the VM and TPU."
    echo "Please run \"gcloud init\" befor this script to set your default zone."
    echo "Example:"
    echo "./create-tpu-deep-learning-vm.sh my-machine --tpu-type v3-8 --version 2.3"
}

maj_min_version() # params: version. Return: sets global variable "maj_min_version"
if [[ "$1" =~ ([0-9]*\.[0-9]*)\.[0-9]* ]]; # if there is sub-version like 2.3.1, cut the version to 2.3. DLVM images are always on the latest minor version.
    then
        maj_min_version=${BASH_REMATCH[1]}; # this keeps
    else
        maj_min_version="$1"
    fi

create_vm() # params: machine_name, machine_type, tfnightly, version
{
    extra_install=""
    if [ "$3" != 0 ]; # if tf-nighty requested
    then
        # since DLVM move to conda, system pip does not work for installing tf-nightly anymore
        extra_install="./opt/conda/bin/pip install tf-nightly-cpu; ./opt/conda/bin/pip install keras-nightly; ./opt/conda/bin/pip install behave";
        maj_min_version $DEFAULT_TF_VERSION # result in variable "maj_min_version". No sub-minor TF versions in DLVM images.
        vm_version=$maj_min_version
        version_msg="tf-nightly (2.x)";
    else
        extra_install="./opt/conda/bin/pip install behave";
        maj_min_version "$4" # result in variable "maj_min_version". No sub-minor TF versions in DLVM images.
        vm_version=$maj_min_version
        version_msg="$4";s
    fi
    image_family=${IMAGE_FAMILY_PATTERN/2-x/${vm_version//./-}}
    echo "Creating VM named $1 of type $2 with Tensorflow $version_msg and image family $image_family. Check for it with \"gcloud compute instances list\""
    gcloud compute instances create $1 \
        --machine-type $2 \
        --image-project deeplearning-platform-release \
        --image-family $image_family \
        --scopes cloud-platform \
        --metadata proxy-mode=project_editors,startup-script="echo \"export TPU_NAME=$1\" > /etc/profile.d/tpu-env.sh; $extra_install" \
        --async

    # To list all possible deep learning VM images:
    # gcloud compute images list --project deeplearning-platform-release
}

create_tpu() # params: machine_name, tpu_type, tfnightly, version
{
    if [ "$3" != 0 ];  # if tf-nighty requested
    then
        tpu_version="nightly";
    else
        tpu_version=$4
    fi
    echo "Creating TPU named $1 with Tensorflow $tpu_version. Check for it with \"gcloud compute tpus list\""
    gcloud compute tpus create $1 \
        --version $tpu_version \
        --accelerator-type $2
}

# standard parameter processing bash code with defaults
tfnightly=0
version=$DEFAULT_TF_VERSION
tpu_type="v2-8"
machine_type="n1-standard-8"
machine_name=$1 # machine name always in first position
shift
while [ "$1" != "" ]; do
    case $1 in
        --machine-type )        shift
                                machine_type=$1
                                ;;
        --tpu-type )            shift
                                tpu_type=$1
                                ;;
        --version )             shift
                                version=$1
                                ;;
        -h | --help )           usage
                                exit
                                ;;
        --nightly )             tfnightly=1
                                ;;
        * )                     usage
                                exit 1
    esac
    shift
done

# parameter checks
if [ "$machine_name" = "" ] || [[ "$machine_name" = -* ]]; then
    usage
    exit 1
fi
if [ "$version" = "2.0" ] || [[ "$version" = 1*  ]]; then
    usage
    exit 1
fi
if [ "$version" = "nightly" ]; then
    tfnightly=1
fi

create_vm "$machine_name" "$machine_type" "$tfnightly" "$version"
create_tpu "$machine_name" "$tpu_type" "$tfnightly" "$version"

error: pathspec 'create-tpu-deep-learning-vm.sh' did not match any file(s) known to git


First commit of gvm installer

First commit of gvm installer

#!/usr/bin/env bash

display_error() {
	tput sgr0
	tput setaf 1
	echo "ERROR: $1"
	tput sgr0
	exit 1
}

update_profile() {
	[ -f "$1" ] || return 1

	grep "$source_line" "$1" > /dev/null 2>&1
	if [ $? -ne 0 ]; then
		echo "$source_line" >> "$1"
	fi
}

check_existing_go() {
	if [ "$GOROOT" == "" ]; then
		echo "No existing Go versions detected"
		return
	fi
	echo "Created profile for existing install of Go at $GOROOT"
	mkdir -p "$GVM_DEST/$GVM_NAME/environments" &> /dev/null || display_error "Failed to create environment directory"
	mkdir -p "$GVM_DEST/$GVM_NAME/pkgsets/system/global" &> /dev/null || display_error "Failed to create new package set"
	mkdir -p "$GVM_DEST/$GVM_NAME/gos/system" &> /dev/null || display_error "Failed to create new Go folder"
	cat << EOF > $GVM_DEST/$GVM_NAME/environments/system
# Automatically generated file. DO NOT EDIT!
export GVM_ROOT; GVM_ROOT="$GVM_DEST/$GVM_NAME"
export gvm_go_name; gvm_go_name="system"
export gvm_pkgset_name; gvm_pkgset_name="global"
export GOROOT; GOROOT="$GOROOT"
export GOPATH; GOPATH="$GVM_DEST/$GVM_NAME/pkgsets/system/global"
export PATH; PATH="$GVM_DEST/$GVM_NAME/pkgsets/system/global:$GOROOT/bin:$GVM_ROOT/bin:\$PATH"
EOF
}

BRANCH=${1:-master}
GVM_DEST=${2:-$HOME}
GVM_NAME="gvm"
SRC_REPO=${SRC_REPO:-https://github.com/moovweb/gvm.git}

[ "$GVM_DEST" = "$HOME" ] && GVM_NAME=".gvm"

[ -d "$GVM_DEST/$GVM_NAME" ] && display_error "Already installed!"
[ -d "$GVM_DEST" ] || mkdir -p "$GVM_DEST" > /dev/null 2>&1 || display_error "Failed to create $GVM_DEST"
[ -z "$(which git)" ] && display_error "Could not find git

  debian/ubuntu: apt-get install git
  redhat/centos: yum install git
  mac:   brew install git
"

# Is gvm-installer being called from the origin repo?
# If so, skip the clone and source locally!
# This prevents, the CI to break on non-merge commits

GIT_ROOT=$(git rev-parse --show-toplevel 2>/dev/null)

if [[ -z "$GIT_ROOT" || "$(basename "$GIT_ROOT")" != "gvm" ]]
then
  echo "Cloning from $SRC_REPO to $GVM_DEST/$GVM_NAME"

  git clone --quiet "$SRC_REPO" "$GVM_DEST/$GVM_NAME" 2> /dev/null ||
	  display_error "Failed to clone from $SRC_REPO into $GVM_DEST/$GVM_NAME"
else
  if [[ $GVM_DEST == *"$GIT_ROOT"* ]]
  then
    ln -s "$GIT_ROOT" "$GVM_DEST"
  else
    cp -r "$GIT_ROOT" "$GVM_DEST/$GVM_NAME"
  fi
fi

# GVM_DEST may be a non-relative path
# i.e: gvm-installer master foo
pushd . > /dev/null

cd "$GVM_DEST/$GVM_NAME" && git checkout --quiet "$BRANCH" 2> /dev/null ||	display_error "Failed to checkout $BRANCH branch"

popd > /dev/null

mv "$GVM_DEST/$GVM_NAME/.git" "$GVM_DEST/$GVM_NAME/git.bak"

source_line="[[ -s \"${GVM_DEST}/$GVM_NAME/scripts/gvm\" ]] && source \"${GVM_DEST}/$GVM_NAME/scripts/gvm\""
source_file="${GVM_DEST}/$GVM_NAME/scripts/gvm"

if [ -n "$ZSH_NAME" ]; then
	update_profile "$HOME/.zshrc"
elif [ "$(uname)" == "Linux" ]; then
	update_profile "$HOME/.bashrc" || update_profile "$HOME/.bash_profile"
elif [ "$(uname)" == "Darwin" ]; then
	update_profile "$HOME/.profile" || update_profile "$HOME/.bash_profile"
fi

if [ "$?" != "0" ]; then
	echo "Unable to locate profile settings file(Something like $HOME/.bashrc or $HOME/.bash_profile)"
	echo
	echo " You will have to manually add the following line:"
	echo
	echo "  $source_line"
	echo
fi

echo "export GVM_ROOT=$GVM_DEST/$GVM_NAME" > "$GVM_DEST/$GVM_NAME/scripts/gvm"
echo ". \$GVM_ROOT/scripts/gvm-default" >> "$GVM_DEST/$GVM_NAME/scripts/gvm"
check_existing_go
[[ -s "$GVM_DEST/$GVM_NAME/scripts/gvm" ]] && source "$GVM_DEST/$GVM_NAME/scripts/gvm"
echo "Installed GVM v${GVM_VERSION}"
echo
echo "Please restart your terminal session or to get started right away run"
echo " \`source ${source_file}\`"
echo

error: pathspec 'gvm/gvm-installer.sh' did not match any file(s) known to git


Add hack/vm-provision.sh

Add hack/vm-provision.sh

#!/bin/bash
set -euo pipefail
IFS=$'\n\t'

sed -i s/^Defaults.*requiretty/\#Defaults\ requiretty/g /etc/sudoers

if [ -f /usr/bin/generate_openshift_service ]
then
  sudo /usr/bin/generate_openshift_service
fi

error: pathspec 'hack/vm-provision.sh' did not match any file(s) known to git


Restore dump_kubernetes.sh function on OSX (#12159)

* Fixes for Bash 3.x and detecting non-running pods

* Address shellcheck warnings

#!/bin/bash
#
# Uses kubectl to collect cluster information.
# Dumps:
# - Logs of every container of every pod of every namespace.
# - Resource configurations for ingress, endpoints, custom resource
#   definitions, configmaps, secrets (names only) and "all" as defined by
#   kubectl.

COREDUMP_DIR="/var/lib/istio"

# Limit total log size to 100MB by default
DEFAULT_MAX_LOG_BYTES=100000000

error() {
  echo "$*" >&2
}

usage() {
  error 'Collect all possible data from a Kubernetes cluster using kubectl.'
  error ''
  error 'Usage:'
  error '  dump_kubernetes.sh [options]'
  error ''
  error 'Options:'
  error '  -d, --output-directory   directory to output files; defaults to'
  error '                               "istio-dump"'
  error '  -z, --archive            if present, archives and removes the output'
  error '                               directory'
  error '  -q, --quiet              if present, do not log'
  error '  -m, --max-bytes          max total bytes, 0=no limit, default='${DEFAULT_MAX_LOG_BYTES}
  error '  -l, --label              if set, dump logs only for pods with given labels e.g. "-l app=pilot -l istio=galley"'
  error '  -n, --namespace          if set, dump logs only for pods in the given namespaces e.g. "-n default -n istio-system"'
  error '  --error-if-nasty-logs    if present, exit with 255 if any logs'
  error '                               contain errors'
  exit 1
}

log() {
  local msg="${1}"
  if [ "${QUIET}" = false ]; then
    printf '%s\n' "${msg}"
  fi
}

parse_args() {
  local max_bytes="${DEFAULT_MAX_LOG_BYTES}"
  while [ "$#" -gt 0 ]; do
    case "${1}" in
      -d|--output-directory)
        local out_dir="${2}"
        shift 2 # Shift past option and value.
        ;;
      -z|--archive)
        local should_archive=true
        shift # Shift past flag.
        ;;
      -q|--quiet)
        local quiet=true
        shift # Shift past flag.
        ;;
      --error-if-nasty-logs)
        local should_check_logs_for_errors=true
        shift # Shift past flag.
        ;;
      -m|--max-bytes)
        max_bytes="${2}"
        shift 2
        ;;
      -l|--label)
        pod_labels+="${2} "
        shift 2
        ;;
      -n|--namespace)
        namespaces+="${2} "
        shift 2
        ;;
      *)
        usage
        ;;
    esac
  done

  readonly OUT_DIR="${out_dir:-istio-dump}"
  readonly SHOULD_ARCHIVE="${should_archive:-false}"
  readonly QUIET="${quiet:-false}"
  readonly SHOULD_CHECK_LOGS_FOR_ERRORS="${should_check_logs_for_errors:-false}"
  readonly LOG_DIR="${OUT_DIR}/logs"
  readonly RESOURCES_FILE="${OUT_DIR}/resources.yaml"
  readonly ISTIO_RESOURCES_FILE="${OUT_DIR}/istio-resources.yaml"
  readonly MAX_LOG_BYTES="${max_bytes}"
}

check_prerequisites() {
  local prerequisites=$*
  for prerequisite in ${prerequisites}; do
    if ! command -v "${prerequisite}" > /dev/null; then
      error "\"${prerequisite}\" is required. Please install it."
      return 1
    fi
  done
}

dump_time() {
  mkdir -p "${OUT_DIR}"
  date -u > "${OUT_DIR}/DUMP_TIME"
}

# mv_unless_max_exceeded src_file dest_file performs mv src_file dest_file unless the global variable stored_log_bytes
# would exceed max_log_bytes.
# If total not exceeded, file is moved and max_log_bytes updated, otherwise an error is logged.
# src_file is always deleted when calling this function.
mv_unless_max_exceeded() {
  local src_file="${1}"
  local dst_file="${2}"

  file_size=$(wc -c "${src_file}" | awk '{print $1}')
  local nsb=$((stored_log_bytes + file_size))

  if (("${nsb}" > "${MAX_LOG_BYTES}")); then
    log "Not storing ${log_file} because appending its ${file_size} bytes would exceed max logged bytes ${MAX_LOG_BYTES}"
    rm "${src_file}"
  else
    dirn=$(dirname "${dst_file}")
    mkdir -p "${dirn}"
    mv "${src_file}" "${dst_file}"
    stored_log_bytes="${nsb}"
  fi
}

dump_logs_for_container() {
  local namespace="${1}"
  local pod="${2}"
  local container="${3}"

  log "Retrieving logs for ${namespace}/${pod}/${container}"

  mkdir -p "${LOG_DIR}/${namespace}/${pod}"
  local log_file_head="${LOG_DIR}/${namespace}/${pod}/${container}"
  local temp_log_file="${LOG_DIR}/temp_log_file.log"

  local log_file="${log_file_head}.log"
  kubectl logs --namespace="${namespace}" "${pod}" "${container}" \
      > "${temp_log_file}"
  mv_unless_max_exceeded "${temp_log_file}" "${log_file}"

  local filter="?(@.name == \"${container}\")"
  local json_path='{.status.containerStatuses['${filter}'].restartCount}'
  local restart_count
  restart_count=$(kubectl get --namespace="${namespace}" \
      pod "${pod}" -o=jsonpath="${json_path}")
  # (There will be no restart_count if the pod status is for example "Pending")
  if [ -n "${restart_count}" ] && [ "${restart_count}" -gt 0 ]; then
    log "Retrieving previous logs for ${namespace}/${pod}/${container}"

    local log_previous_file
    log_previous_file="${log_file_head}_previous.log"
    kubectl logs --namespace="${namespace}" \
        --previous "${pod}" "${container}" \
        > "${temp_log_file}"
    mv_unless_max_exceeded "${temp_log_file}" "${log_previous_file}"
  fi
}

copy_core_dumps_if_istio_proxy() {
  local namespace="${1}"
  local pod="${2}"
  local container="${3}"
  local got_core_dump=false

  if [ "istio-proxy" = "${container}" ]; then
    local out_dir="${LOG_DIR}/${namespace}/${pod}"
    mkdir -p "${out_dir}"
    local core_dumps
    core_dumps=$(kubectl exec -n "${namespace}" "${pod}" -c "${container}" -- \
        find ${COREDUMP_DIR} -name 'core.*')
    for f in ${core_dumps}; do
      local out_file
      out_file="${out_dir}/$(basename "${f}")"

      kubectl exec -n "${namespace}" "${pod}" -c "${container}" -- \
          cat "${f}" > "${out_file}"

      log "Copied ${namespace}/${pod}/${container}:${f} to ${out_file}"
      got_core_dump=true
    done
  fi
  if [ "${got_core_dump}" = true ]; then
    return 254
  fi
}

# Run functions on each container. Each argument should be a function which
# takes 3 args: ${namespace} ${pod} ${container}.
# If any of the called functions returns error, tap_containers returns
# immediately with that error.
tap_containers() {
  local functions=("$@")
  if [ -z "${namespaces}" ]; then
    namespaces=$(kubectl get \
        namespaces -o=jsonpath="{.items[*].metadata.name}")
  fi
  for namespace in ${namespaces}; do
    local pods=""
    if [ -n "${pod_labels}" ]; then
      for label in $pod_labels; do
        pods+=$(kubectl get --namespace="${namespace}" -l"${label}" \
            pods -o=jsonpath='{.items[*].metadata.name}')" "
      done
    else
        pods=$(kubectl get --namespace="${namespace}" \
            pods -o=jsonpath='{.items[*].metadata.name}')
    fi
    for pod in ${pods}; do
      local containers
      containers=$(kubectl get --namespace="${namespace}" \
          pod "${pod}" -o=jsonpath='{.spec.containers[*].name}')
      for container in ${containers}; do
        for f in "${functions[@]}"; do
          "${f}" "${namespace}" "${pod}" "${container}" || return $?
        done
      done
    done
  done

  return 0
}

dump_kubernetes_resources() {
  log "Retrieving Kubernetes resource configurations"

  mkdir -p "${OUT_DIR}"
  # Only works in Kubernetes 1.8.0 and above.
  kubectl get --all-namespaces --export \
      all,jobs,ingresses,endpoints,customresourcedefinitions,configmaps,secrets,events \
      -o yaml > "${RESOURCES_FILE}"
}

dump_istio_custom_resource_definitions() {
  log "Retrieving Istio resource configurations"

  local istio_resources
  # Trim to only first field; join by comma; remove last comma.
  istio_resources=$(kubectl get customresourcedefinitions \
      --no-headers 2> /dev/null \
      | cut -d ' ' -f 1 \
      | tr '\n' ',' \
      | sed 's/,$//')

  if [ -n "${istio_resources}" ]; then
    kubectl get "${istio_resources}" --all-namespaces -o yaml \
        > "${ISTIO_RESOURCES_FILE}"
  fi
}

dump_resources() {
  dump_kubernetes_resources
  dump_istio_custom_resource_definitions

  mkdir -p "${OUT_DIR}"
  kubectl cluster-info dump > "${OUT_DIR}/cluster-info.dump.txt"
  kubectl describe pods -n istio-system > "${OUT_DIR}/istio-system-pods.txt"
  kubectl get events --all-namespaces -o wide > "${OUT_DIR}/events.txt"
}

dump_pilot_url(){
  local pilot_pod=$1
  local url=$2
  local dname=$3
  local outfile

  outfile="${dname}/$(basename "${url}")"

  log "Fetching ${url} from pilot"
  kubectl -n istio-system exec -i -t "${pilot_pod}" -c istio-proxy -- \
      curl "http://localhost:8080/${url}" > "${outfile}"
}

dump_pilot() {
  local pilot_pod
  pilot_pod=$(kubectl -n istio-system get pods -l istio=pilot \
      -o jsonpath='{.items[*].metadata.name}')

  if [ -n "${pilot_pod}" ]; then
    local pilot_dir="${OUT_DIR}/pilot"
    mkdir -p "${pilot_dir}"

    dump_pilot_url "${pilot_pod}" debug/configz "${pilot_dir}"
    dump_pilot_url "${pilot_pod}" debug/endpointz "${pilot_dir}"
    dump_pilot_url "${pilot_pod}" debug/adsz "${pilot_dir}"
    dump_pilot_url "${pilot_pod}" debug/authenticationz "${pilot_dir}"
    dump_pilot_url "${pilot_pod}" metrics "${pilot_dir}"
  fi
}

archive() {
  local parent_dir
  parent_dir=$(dirname "${OUT_DIR}")
  local dir
  dir=$(basename "${OUT_DIR}")

  pushd "${parent_dir}" > /dev/null || exit
  tar -czf "${dir}.tar.gz" "${dir}"
  popd > /dev/null || exit

  log "Wrote ${parent_dir}/${dir}.tar.gz"
}

check_logs_for_errors() {
  log "Searching logs for errors."
  grep -R --include "${LOG_DIR}/*.log" --ignore-case -e 'segmentation fault'
}

main() {
  local exit_code=0
  parse_args "$@"
  check_prerequisites kubectl
  dump_time
  dump_pilot
  dump_resources
  tap_containers "dump_logs_for_container" "copy_core_dumps_if_istio_proxy"
  exit_code=$?

  if [ "${SHOULD_CHECK_LOGS_FOR_ERRORS}" = true ]; then
    if ! check_logs_for_errors; then
      exit_code=255
    fi
  fi

  if [ "${SHOULD_ARCHIVE}" = true ] ; then
    archive
    rm -r "${OUT_DIR}"
  fi
  log "Wrote to ${OUT_DIR}"

  return ${exit_code}
}

stored_log_bytes=0

main "$@"

#!/bin/bash
#
# Uses kubectl to collect cluster information.
# Dumps:
# - Logs of every container of every pod of every namespace.
# - Resource configurations for ingress, endpoints, custom resource
#   definitions, configmaps, secrets (names only) and "all" as defined by
#   kubectl.

COREDUMP_DIR="/var/lib/istio"

# Limit total log size to 100MB by default
DEFAULT_MAX_LOG_BYTES=100000000

error() {
  echo "$*" >&2
}

usage() {
  error 'Collect all possible data from a Kubernetes cluster using kubectl.'
  error ''
  error 'Usage:'
  error '  dump_kubernetes.sh [options]'
  error ''
  error 'Options:'
  error '  -d, --output-directory   directory to output files; defaults to'
  error '                               "istio-dump"'
  error '  -z, --archive            if present, archives and removes the output'
  error '                               directory'
  error '  -q, --quiet              if present, do not log'
  error '  -m, --max-bytes          max total bytes, 0=no limit, default='${DEFAULT_MAX_LOG_BYTES}
  error '  -l, --label              if set, dump logs only for pods with given labels e.g. "-l app=pilot -l istio=galley"'
  error '  -n, --namespace          if set, dump logs only for pods in the given namespaces e.g. "-n default -n istio-system"'
  error '  --error-if-nasty-logs    if present, exit with 255 if any logs'
  error '                               contain errors'
  exit 1
}

log() {
  local msg="${1}"
  if [ "${QUIET}" = false ]; then
    printf '%s\n' "${msg}"
  fi
}

parse_args() {
  local max_bytes="${DEFAULT_MAX_LOG_BYTES}"
  while [ "$#" -gt 0 ]; do
    case "${1}" in
      -d|--output-directory)
        local out_dir="${2}"
        shift 2 # Shift past option and value.
        ;;
      -z|--archive)
        local should_archive=true
        shift # Shift past flag.
        ;;
      -q|--quiet)
        local quiet=true
        shift # Shift past flag.
        ;;
      --error-if-nasty-logs)
        local should_check_logs_for_errors=true
        shift # Shift past flag.
        ;;
      -m|--max-bytes)
        max_bytes="${2}"
        shift 2
        ;;
      -l|--label)
        pod_labels+="${2} "
        shift 2
        ;;
      -n|--namespace)
        namespaces+="${2} "
        shift 2
        ;;
      *)
        usage
        ;;
    esac
  done

  readonly OUT_DIR="${out_dir:-istio-dump}"
  readonly SHOULD_ARCHIVE="${should_archive:-false}"
  readonly QUIET="${quiet:-false}"
  readonly SHOULD_CHECK_LOGS_FOR_ERRORS="${should_check_logs_for_errors:-false}"
  readonly LOG_DIR="${OUT_DIR}/logs"
  readonly RESOURCES_FILE="${OUT_DIR}/resources.yaml"
  readonly ISTIO_RESOURCES_FILE="${OUT_DIR}/istio-resources.yaml"
  readonly MAX_LOG_BYTES="${max_bytes}"
}

check_prerequisites() {
  local prerequisites=$*
  for prerequisite in ${prerequisites}; do
    if ! command -v "${prerequisite}" > /dev/null; then
      error "\"${prerequisite}\" is required. Please install it."
      return 1
    fi
  done
}

dump_time() {
  mkdir -p "${OUT_DIR}"
  date -u > "${OUT_DIR}/DUMP_TIME"
}

# mv_unless_max_exceeded src_file dest_file performs mv src_file dest_file unless the global variable stored_log_bytes
# would exceed max_log_bytes.
# If total not exceeded, file is moved and max_log_bytes updated, otherwise an error is logged.
# src_file is always deleted when calling this function.
mv_unless_max_exceeded() {
  local src_file="${1}"
  local dst_file="${2}"

  file_size=$(wc -c "${src_file}" | awk '{print $1}')
  local nsb=$(("${stored_log_bytes}" + "${file_size}"))

  if (("${nsb}" > "${MAX_LOG_BYTES}")); then
    log "Not storing ${log_file} because appending its ${file_size} bytes would exceed max logged bytes ${MAX_LOG_BYTES}"
    rm "${src_file}"
  else
    dirn=$(dirname "${dst_file}")
    mkdir -p "${dirn}"
    mv "${src_file}" "${dst_file}"
    stored_log_bytes="${nsb}"
  fi
}

dump_logs_for_container() {
  local namespace="${1}"
  local pod="${2}"
  local container="${3}"

  log "Retrieving logs for ${namespace}/${pod}/${container}"

  mkdir -p "${LOG_DIR}/${namespace}/${pod}"
  local log_file_head="${LOG_DIR}/${namespace}/${pod}/${container}"
  local temp_log_file="${LOG_DIR}/temp_log_file.log"

  local log_file="${log_file_head}.log"
  kubectl logs --namespace="${namespace}" "${pod}" "${container}" \
      > "${temp_log_file}"
  mv_unless_max_exceeded "${temp_log_file}" "${log_file}"

  local filter="?(@.name == \"${container}\")"
  local json_path='{.status.containerStatuses['${filter}'].restartCount}'
  local restart_count
  restart_count=$(kubectl get --namespace="${namespace}" \
      pod "${pod}" -o=jsonpath="${json_path}")
  if [ "${restart_count}" -gt 0 ]; then
    log "Retrieving previous logs for ${namespace}/${pod}/${container}"

    local log_previous_file
    log_previous_file="${log_file_head}_previous.log"
    kubectl logs --namespace="${namespace}" \
        --previous "${pod}" "${container}" \
        > "${temp_log_file}"
    mv_unless_max_exceeded "${temp_log_file}" "${log_previous_file}"
  fi
}

copy_core_dumps_if_istio_proxy() {
  local namespace="${1}"
  local pod="${2}"
  local container="${3}"
  local got_core_dump=false

  if [ "istio-proxy" = "${container}" ]; then
    local out_dir="${LOG_DIR}/${namespace}/${pod}"
    mkdir -p "${out_dir}"
    local core_dumps
    core_dumps=$(kubectl exec -n "${namespace}" "${pod}" -c "${container}" -- \
        find ${COREDUMP_DIR} -name 'core.*')
    for f in ${core_dumps}; do
      local out_file
      out_file="${out_dir}/$(basename "${f}")"

      kubectl exec -n "${namespace}" "${pod}" -c "${container}" -- \
          cat "${f}" > "${out_file}"

      log "Copied ${namespace}/${pod}/${container}:${f} to ${out_file}"
      got_core_dump=true
    done
  fi
  if [ "${got_core_dump}" = true ]; then
    return 254
  fi
}

# Run functions on each container. Each argument should be a function which
# takes 3 args: ${namespace} ${pod} ${container}.
# If any of the called functions returns error, tap_containers returns
# immediately with that error.
tap_containers() {
  local functions=("$@")
  if [ -z "${namespaces}" ]; then
    namespaces=$(kubectl get \
        namespaces -o=jsonpath="{.items[*].metadata.name}")
  fi
  for namespace in ${namespaces}; do
    local pods=""
    if [ -n "${pod_labels}" ]; then
      for label in $pod_labels; do
        pods+=$(kubectl get --namespace="${namespace}" -l"${label}" \
            pods -o=jsonpath='{.items[*].metadata.name}')" "
      done
    else
        pods=$(kubectl get --namespace="${namespace}" \
            pods -o=jsonpath='{.items[*].metadata.name}')
    fi
    for pod in ${pods}; do
      local containers
      containers=$(kubectl get --namespace="${namespace}" \
          pod "${pod}" -o=jsonpath='{.spec.containers[*].name}')
      for container in ${containers}; do
        for f in "${functions[@]}"; do
          "${f}" "${namespace}" "${pod}" "${container}" || return $?
        done
      done
    done
  done

  return 0
}

dump_kubernetes_resources() {
  log "Retrieving kubernetes resource configurations"

  mkdir -p "${OUT_DIR}"
  # Only works in Kubernetes 1.8.0 and above.
  kubectl get --all-namespaces --export \
      all,jobs,ingresses,endpoints,customresourcedefinitions,configmaps,secrets,events \
      -o yaml > "${RESOURCES_FILE}"
}

dump_istio_custom_resource_definitions() {
  log "Retrieving istio resource configurations"

  local istio_resources
  # Trim to only first field; join by comma; remove last comma.
  istio_resources=$(kubectl get customresourcedefinitions \
      --no-headers 2> /dev/null \
      | cut -d ' ' -f 1 \
      | tr '\n' ',' \
      | sed 's/,$//')

  if [ -n "${istio_resources}" ]; then
    kubectl get "${istio_resources}" --all-namespaces -o yaml \
        > "${ISTIO_RESOURCES_FILE}"
  fi
}

dump_resources() {
  dump_kubernetes_resources
  dump_istio_custom_resource_definitions

  mkdir -p "${OUT_DIR}"
  kubectl cluster-info dump > "${OUT_DIR}/cluster-info.dump.txt"
  kubectl describe pods -n istio-system > "${OUT_DIR}/istio-system-pods.txt"
  kubectl get events --all-namespaces -o wide > "${OUT_DIR}/events.txt"
}

dump_pilot_url(){
  local pilot_pod=$1
  local url=$2
  local dname=$3
  local outfile

  outfile="${dname}/$(basename "${url}")"

  log "Fetching ${url} from pilot"
  kubectl -n istio-system exec -i -t "${pilot_pod}" -c istio-proxy -- \
      curl "http://localhost:8080/${url}" > "${outfile}"
}

dump_pilot() {
  local pilot_pod
  pilot_pod=$(kubectl -n istio-system get pods -l istio=pilot \
      -o jsonpath='{.items[*].metadata.name}')

  if [ -n "${pilot_pod}" ]; then
    local pilot_dir="${OUT_DIR}/pilot"
    mkdir -p "${pilot_dir}"

    dump_pilot_url "${pilot_pod}" debug/configz "${pilot_dir}"
    dump_pilot_url "${pilot_pod}" debug/endpointz "${pilot_dir}"
    dump_pilot_url "${pilot_pod}" debug/adsz "${pilot_dir}"
    dump_pilot_url "${pilot_pod}" debug/authenticationz "${pilot_dir}"
    dump_pilot_url "${pilot_pod}" metrics "${pilot_dir}"
  fi
}

archive() {
  local parent_dir
  parent_dir=$(dirname "${OUT_DIR}")
  local dir
  dir=$(basename "${OUT_DIR}")

  pushd "${parent_dir}" > /dev/null || exit
  tar -czf "${dir}.tar.gz" "${dir}"
  popd > /dev/null || exit

  log "Wrote ${parent_dir}/${dir}.tar.gz"
}

check_logs_for_errors() {
  log "Searching logs for errors."
  grep -R --include "${LOG_DIR}/*.log" --ignore-case -e 'segmentation fault'
}

main() {
  local exit_code=0
  parse_args "$@"
  check_prerequisites kubectl
  dump_time
  dump_pilot
  dump_resources
  tap_containers "dump_logs_for_container" "copy_core_dumps_if_istio_proxy"
  exit_code=$?

  if [ "${SHOULD_CHECK_LOGS_FOR_ERRORS}" = true ]; then
    if ! check_logs_for_errors; then
      exit_code=255
    fi
  fi

  if [ "${SHOULD_ARCHIVE}" = true ] ; then
    archive
    rm -r "${OUT_DIR}"
  fi
  log "Wrote to ${OUT_DIR}"

  return ${exit_code}
}

stored_log_bytes=0

main "$@"


Checkpoint

Checkpoint

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/katherinealbany/incubator.git/'


Add flac2alac

Add flac2alac

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/milamd/sh.git/'


Change travis script

Change travis script

#!/usr/bin/env bash

cd jtravis
mvn clean install

if [[ $? != 0 ]]
then
    echo "Error while installing jTravis"
    exit 1
fi

cd ..

cd repairnator
mvn clean install
mvn jacoco:report coveralls:report --fail-never

if [[ $? != 0 ]]
then
    echo "Error while installing repairnator"
    exit 1
fi

cd ..

cd travisFilter
mvn clean install

if [[ $? != 0 ]]
then
    echo "Error while installing travisFilter"
    exit 1
fi

#!/usr/bin/env bash

cd jtravis
mvn clean install

if [[ $? != 0 ]]
then
    echo "Error while installing jTravis"
    exit 1
fi

cd ..

cd repairnator
mvn clean install jacoco:report coveralls:report

if [[ $? != 0 ]]
then
    echo "Error while installing repairnator"
    exit 1
fi

cd ..

cd travisFilter
mvn clean install

if [[ $? != 0 ]]
then
    echo "Error while installing travisFilter"
    exit 1
fi


Make a symlink to the old fundamentals script

This way, older versions of the course materials will continue to work.

error: pathspec 'scripts/classroom/verify_fundamentals_settings.sh' did not match any file(s) known to git


Fix typo

Fix typo

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/Zonama/ZonamaDev.git/'


Created Ubuntu init script.

Created Ubuntu init script.

 On branch master

	new file:   ubuntu_setup/init.sh

#!/bin/bash
nosync=$HOME/nosync
setup=$nosync/setup
HOME_LOCAL=$HOME/local
HOME_LOCAL_BIN=$HOME_LOCAL/bin
BAK=$nosync/bak
bak_etc=$BAK/install-conf-bak.tar.bz2

# Create the local dir
if [ -d $HOME_LOCAL ]; then
    echo $HOME_LOCAL' dir already created'
else
    echo 'creating '$HOME_LOCAL' dir'
    mkdir $HOME_LOCAL
fi

# Create the local bin dir
if [ -d $HOME_LOCAL_BIN ]; then
    echo $HOME_LOCAL_BIN' dir already created'
else
    echo 'creating '$HOME_LOCAL_BIN' dir'
    mkdir $HOME_LOCAL_BIN
fi

# Create the nosync dir
if [ -d $nosync ]; then
    echo $nosync' dir already created'
else
    echo 'creating '$nosync' dir'
    mkdir $nosync
fi

# Create the bak sub dir in nosync
if [ -d $BAK ]; then
    echo $BAK' dir already created'
else
    echo 'creating '$BAK' dir'
    mkdir $BACK
fi

# Create the setup sub dir in nosync
if [ -d $setup ]; then
    echo $setup' dir already created'
else
    echo 'creating '$setup' dir'
    mkdir $setup
fi

# Move this script
if [ -f $HOME/init.sh ]; then
    mv $HOME/init.sh $setup/
fi

# Backup the /etc config dir while it's still fresh.
if [ -f $bak_etc ]; then
    echo '/etc has already been backed up to '$bak_etc
else
    echo 'making backup of /etc'
    tar -cjf $bak_etc /etc
fi

# Download the rsync script
if ! [ -f $HOME_LOCAL_BIN/syncpull ]; then
    echo 'getting syncpull.sh from GitHub'
    wget http://github.com/kixxauth/scripts/raw/master/syncpull.sh $HOME_LOCAL_BIN/syncpull
    if [ -f $HOME_LOCAL_BIN/syncpull ]; then
        chmod 764 $HOME_LOCAL_BIN/syncpull
    else
        echo 'unable to download syncpull.sh from GitHub'
    fi
else
    echo 'syncpull.sh already downloaded'
fi

# Download the applications installation script
if ! [ -f $setup/ubuntu-packages.sh ]; then
    echo 'getting ubuntu-packages.sh from GitHub'
    wget http://github.com/kixxauth/scripts/raw/master/ubuntu-packages.sh $setup/ubuntu-packages.sh
    if [ -f $setup/ubuntu-packages.sh ]; then
        chmod 764 $setup/ubuntu-packages.sh
    else
        echo 'unable to download ubuntu-packages.sh from GitHub'
        # We can't go on.
        exit
    fi
else
    echo 'ubuntu-packages.sh already downloaded'
fi

error: pathspec 'ubuntu_setup/init.sh' did not match any file(s) known to git


Use bundle name based off project name (if available) instead  of static name app.app

#!/bin/bash
# Copyright (c) 2012 Jonathan Penn (http://cocoamanifest.net/)

# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:

# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.

# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

# Tell bash that we want the whole script to fail if any part fails.
set -e

# We require a parameter for where to put the results and the test script
destination="$1"
ui_script="$2"

function main {
  # Load configuration
  # Not in a separate function because you can't exort arrays
  # https://stackoverflow.com/questions/5564418/exporting-an-array-in-bash-script
  # Will export languages and simulators bash variables
  export UISS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
  if [ -f "$UISS_DIR"/config-screenshots.sh ]; then
    source "$UISS_DIR"/config-screenshots.sh
  else
    if [ -f "$UISS_DIR"/config-screenshots.example.sh ]; then
      source "$UISS_DIR"/config-screenshots.example.sh
      echo "WARNING: Using example config-screenshots file, you should create your own"
    else
      echo "Configuration \"config-screenshots.sh\" does not exist! Aborting."
      exit 1
    fi
  fi

  _check_destination
  _check_ui_script
  _xcode clean build

  for simulator in "${simulators[@]}"; do
    for language in $languages; do
      _clean_trace_results_dir
      _run_automation "$ui_script" "$language" "$simulator"
      _copy_screenshots "$language"
    done
  done

  _close_sim

  echo
  echo "Screenshots complete!"
}

# Global variables to keep track of where everything goes
tmp_dir="/tmp"
build_dir="$tmp_dir/screen_shooter"
bundle_dir="$build_dir/app.app"
trace_results_dir="$build_dir/traces"

function _check_destination {
  # Abort if the destination directory already exists. Better safe than sorry.

  if [ -z "$destination" ]; then
    destination="$HOME/Desktop/screenshots"
  fi
  if [ -d "$destination" ]; then
    echo "Destination directory \"$destination\" already exists! Aborting."
    exit 1
  fi
}

function _check_ui_script {
  # Abort if the UI script does not exist.

  if [ -z "$ui_script" ]; then
    ui_script="./config-automation.js"
  fi
  if [ ! -f "$ui_script" ]; then
    if [ -f "./config-automation.example.js" ]; then
      ui_script="./config-automation.js"
      echo "WARNING: Using example config-automation, please create your own"
    else
      echo "Config-automation does not exist! Aborting."
      exit 1
    fi
  fi
}

function _xcode {
  # A wrapper around `xcodebuild` that tells it to build the app in the temp
  # directory. If your app uses workspaces or special schemes, you'll need to
  # specify them here.
  #
  # Use `man xcodebuild` for more information on how to build your project.
  if test -n "$(find . -maxdepth 1 -name '*.xcworkspace' -print -quit)"
  then
    base=$(basename *.xcworkspace .xcworkspace)
    # First build omits PRODUCT_NAME
    # Do NOT ask me why you need to build this twice for it to work
    # or how I became to know this fact
    xcodebuild -sdk "iphonesimulator$ios_version" \
      CONFIGURATION_BUILD_DIR="$build_dir/build" \
      -workspace "$base.xcworkspace" -scheme "$base" -configuration Debug \
      DSTROOT=$build_dir \
      OBJROOT=$build_dir \
      SYMROOT=$build_dir \
      ONLY_ACTIVE_ARCH=NO \
    "$@"
    xcodebuild -sdk "iphonesimulator$ios_version" \
      CONFIGURATION_BUILD_DIR="$build_dir/build" \
      -workspace "$base.xcworkspace" -scheme "$base" -configuration Debug \
      DSTROOT=$build_dir \
      OBJROOT=$build_dir \
      SYMROOT=$build_dir \
      ONLY_ACTIVE_ARCH=NO \
    "$@"
    cp -r "$build_dir/build/$base.app" "$build_dir"
    bundle_dir="$build_dir/$base.app"
  else
    xcodebuild -sdk "iphonesimulator$ios_version" \
      CONFIGURATION_BUILD_DIR=$build_dir \
      PRODUCT_NAME=app \
    "$@"
  fi
}

function _clean_trace_results_dir {
  # Removes the trace results directory. We need to do this because Instruments
  # keeps appending new trace runs and it's simpler for us to always assume
  # there's just one run recorded where we look for screenshots.

  rm -rf "$trace_results_dir"
  mkdir -p "$trace_results_dir"
}

function _run_automation {
  # Runs the UI Automation JavaScript file that actually takes the screenshots.

  automation_script="$1"
  language="$2"
  simulator="$3"

  echo "Running automation script \"$automation_script\"
          for \"$simulator\"
          in language \"${language}\"..."

  dev_tools_dir=`xcode-select -print-path`
  tracetemplate="Automation"

  # Check out the `unix_instruments.sh` script to see why we need this wrapper.
  DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
  "$DIR"/unix_instruments.sh \
    -w "$simulator" \
    -D "$trace_results_dir/trace" \
    -t "$tracetemplate" \
    "$bundle_dir" \
    -e UIARESULTSPATH "$trace_results_dir" \
    -e UIASCRIPT "$automation_script" \
    -AppleLanguages "($language)" \
    -AppleLocale "$language"

  find $trace_results_dir/Run\ 1/ -name *landscape*png -type f -exec sips -r -90 \{\} \;
}

function _copy_screenshots {
  # Since we're always clearing out the trace results before every run, we can
  # assume that any screenshots were saved in the "Run 1" directory. Copy them
  # to the destination's language folder!

  language="$1"

  mkdir -p "$destination/$language"
  cp $trace_results_dir/Run\ 1/*.png "$destination/$language"
}

function _close_sim {
  # I know, I know. It says "iPhone Simulator". For some reason,
  # that's the only way Applescript can identify it.
  osascript -e "tell application \"iPhone Simulator\" to quit"
}

main

#!/bin/bash
# Copyright (c) 2012 Jonathan Penn (http://cocoamanifest.net/)

# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:

# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.

# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

# Tell bash that we want the whole script to fail if any part fails.
set -e

# We require a parameter for where to put the results and the test script
destination="$1"
ui_script="$2"

function main {
  # Load configuration
  # Not in a separate function because you can't exort arrays
  # https://stackoverflow.com/questions/5564418/exporting-an-array-in-bash-script
  # Will export languages and simulators bash variables
  export UISS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
  if [ -f "$UISS_DIR"/config-screenshots.sh ]; then
    source "$UISS_DIR"/config-screenshots.sh
  else
    if [ -f "$UISS_DIR"/config-screenshots.example.sh ]; then
      source "$UISS_DIR"/config-screenshots.example.sh
      echo "WARNING: Using example config-screenshots file, you should create your own"
    else
      echo "Configuration \"config-screenshots.sh\" does not exist! Aborting."
      exit 1
    fi
  fi

  _check_destination
  _check_ui_script
  _xcode clean build

  for simulator in "${simulators[@]}"; do
    for language in $languages; do
      _clean_trace_results_dir
      _run_automation "$ui_script" "$language" "$simulator"
      _copy_screenshots "$language"
    done
  done

  _close_sim

  echo
  echo "Screenshots complete!"
}

# Global variables to keep track of where everything goes
tmp_dir="/tmp"
build_dir="$tmp_dir/screen_shooter"
bundle_dir="$build_dir/app.app"
trace_results_dir="$build_dir/traces"

function _check_destination {
  # Abort if the destination directory already exists. Better safe than sorry.

  if [ -z "$destination" ]; then
    destination="$HOME/Desktop/screenshots"
  fi
  if [ -d "$destination" ]; then
    echo "Destination directory \"$destination\" already exists! Aborting."
    exit 1
  fi
}

function _check_ui_script {
  # Abort if the UI script does not exist.

  if [ -z "$ui_script" ]; then
    ui_script="./config-automation.js"
  fi
  if [ ! -f "$ui_script" ]; then
    if [ -f "./config-automation.example.js" ]; then
      ui_script="./config-automation.js"
      echo "WARNING: Using example config-automation, please create your own"
    else
      echo "Config-automation does not exist! Aborting."
      exit 1
    fi
  fi
}

function _xcode {
  # A wrapper around `xcodebuild` that tells it to build the app in the temp
  # directory. If your app uses workspaces or special schemes, you'll need to
  # specify them here.
  #
  # Use `man xcodebuild` for more information on how to build your project.
  if test -n "$(find . -maxdepth 1 -name '*.xcworkspace' -print -quit)"
  then
    base=$(basename *.xcworkspace .xcworkspace)
    # First build omits PRODUCT_NAME
    # Do NOT ask me why you need to build this twice for it to work
    # or how I became to know this fact
    xcodebuild -sdk "iphonesimulator$ios_version" \
      CONFIGURATION_BUILD_DIR="$build_dir/build" \
      -workspace "$base.xcworkspace" -scheme "$base" -configuration Debug \
      DSTROOT=$build_dir \
      OBJROOT=$build_dir \
      SYMROOT=$build_dir \
      ONLY_ACTIVE_ARCH=NO \
    "$@"
    xcodebuild -sdk "iphonesimulator$ios_version" \
      CONFIGURATION_BUILD_DIR="$build_dir/build" \
      PRODUCT_NAME=app \
      -workspace "$base.xcworkspace" -scheme "$base" -configuration Debug \
      DSTROOT=$build_dir \
      OBJROOT=$build_dir \
      SYMROOT=$build_dir \
      ONLY_ACTIVE_ARCH=NO \
    "$@"
    cp -r "$build_dir/build/app.app" "$build_dir"
  else
    xcodebuild -sdk "iphonesimulator$ios_version" \
      CONFIGURATION_BUILD_DIR=$build_dir \
      PRODUCT_NAME=app \
    "$@"
  fi
}

function _clean_trace_results_dir {
  # Removes the trace results directory. We need to do this because Instruments
  # keeps appending new trace runs and it's simpler for us to always assume
  # there's just one run recorded where we look for screenshots.

  rm -rf "$trace_results_dir"
  mkdir -p "$trace_results_dir"
}

function _run_automation {
  # Runs the UI Automation JavaScript file that actually takes the screenshots.

  automation_script="$1"
  language="$2"
  simulator="$3"

  echo "Running automation script \"$automation_script\"
          for \"$simulator\"
          in language \"${language}\"..."

  dev_tools_dir=`xcode-select -print-path`
  tracetemplate="Automation"

  # Check out the `unix_instruments.sh` script to see why we need this wrapper.
  DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
  "$DIR"/unix_instruments.sh \
    -w "$simulator" \
    -D "$trace_results_dir/trace" \
    -t "$tracetemplate" \
    $bundle_dir \
    -e UIARESULTSPATH "$trace_results_dir" \
    -e UIASCRIPT "$automation_script" \
    -AppleLanguages "($language)" \
    -AppleLocale "$language"

  find $trace_results_dir/Run\ 1/ -name *landscape*png -type f -exec sips -r -90 \{\} \;
}

function _copy_screenshots {
  # Since we're always clearing out the trace results before every run, we can
  # assume that any screenshots were saved in the "Run 1" directory. Copy them
  # to the destination's language folder!

  language="$1"

  mkdir -p "$destination/$language"
  cp $trace_results_dir/Run\ 1/*.png "$destination/$language"
}

function _close_sim {
  # I know, I know. It says "iPhone Simulator". For some reason,
  # that's the only way Applescript can identify it.
  osascript -e "tell application \"iPhone Simulator\" to quit"
}

main


Build Debug configuration by default.

Build Debug configuration by default.

#!/bin/bash
# Copyright (c) 2012 Jonathan Penn (http://cocoamanifest.net/)

# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:

# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.

# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

# Tell bash that we want the whole script to fail if any part fails.
set -e

# We require a parameter for where to put the results and the test script
destination="$1"
ui_script="$2"

# The locale identifiers for the languages you want to shoot
# Use the format like en-US zh-Hans for filenames compatible with iTunes
# connect upload tool
# FYI: get the locale names for you existing app with iTMSTransporter and:
# grep locale ~/Desktop/*.itmsp/metadata.xml  | grep name | sort -u
languages="en-US fr ja"

# The simulators we want to run the script against, declared as a Bash array.
# Run `instruments -s devices` to get a list of all the possible string values.
declare -a simulators=(
"Resizable iPad"
)

function main {
  _check_destination
  _check_ui_script
  _xcode clean build

  for simulator in "${simulators[@]}"; do
    for language in $languages; do
      _clean_trace_results_dir
      _run_automation "$ui_script" "$language" "$simulator"
      _copy_screenshots "$language"
    done
  done

  _close_sim

  echo
  echo "Screenshots complete!"
}

# Global variables to keep track of where everything goes
tmp_dir="/tmp"
build_dir="$tmp_dir/screen_shooter"
bundle_dir="$build_dir/app.app"
trace_results_dir="$build_dir/traces"

function _check_destination {
  # Abort if the destination directory already exists. Better safe than sorry.

  if [ -z "$destination" ]; then
    destination="$HOME/Desktop/screenshots"
  fi
  if [ -d "$destination" ]; then
    echo "Destination directory \"$destination\" already exists! Aborting."
    exit 1
  fi
}

function _check_ui_script {
  # Abort if the UI script does not exist.

  if [ -z "$ui_script" ]; then
    ui_script="./shoot_the_screens.js"
  fi
  if [ ! -f "$ui_script" ]; then
    echo "UI script \"$ui_script\" does not exist! Aborting."
    exit 1
  fi
}

function _xcode {
  # A wrapper around `xcodebuild` that tells it to build the app in the temp
  # directory. If your app uses workspaces or special schemes, you'll need to
  # specify them here.
  #
  # Use `man xcodebuild` for more information on how to build your project.
  if test -n "$(find . -maxdepth 1 -name '*.xcworkspace' -print -quit)"
  then
    base=$(basename *.xcworkspace .xcworkspace)
    # First build omits PRODUCT_NAME
    # Do NOT ask me why you need to build this twice for it to work
    # or how I became to know this fact
    xcodebuild -sdk "iphonesimulator$ios_version" \
      CONFIGURATION_BUILD_DIR="$build_dir/build" \
      -workspace $base.xcworkspace -scheme $base -configuration Debug \
      DSTROOT=$build_dir \
      OBJROOT=$build_dir \
      SYMROOT=$build_dir \
      ONLY_ACTIVE_ARCH=NO \
    "$@"
    xcodebuild -sdk "iphonesimulator$ios_version" \
      CONFIGURATION_BUILD_DIR="$build_dir/build" \
      PRODUCT_NAME=app \
      -workspace $base.xcworkspace -scheme $base -configuration Debug \
      DSTROOT=$build_dir \
      OBJROOT=$build_dir \
      SYMROOT=$build_dir \
      ONLY_ACTIVE_ARCH=NO \
    "$@"
    cp -r "$build_dir/build/app.app" "$build_dir"
  else
    xcodebuild -sdk "iphonesimulator$ios_version" \
      CONFIGURATION_BUILD_DIR=$build_dir \
      PRODUCT_NAME=app \
    "$@"
  fi
}

function _clean_trace_results_dir {
  # Removes the trace results directory. We need to do this because Instruments
  # keeps appending new trace runs and it's simpler for us to always assume
  # there's just one run recorded where we look for screenshots.

  rm -rf "$trace_results_dir"
  mkdir -p "$trace_results_dir"
}

function _run_automation {
  # Runs the UI Automation JavaScript file that actually takes the screenshots.

  automation_script="$1"
  language="$2"
  simulator="$3"

  echo "Running automation script \"$automation_script\"
          for \"$simulator\"
          in language \"${language}\"..."

  dev_tools_dir=`xcode-select -print-path`
  tracetemplate="Automation"

  # Check out the `unix_instruments.sh` script to see why we need this wrapper.
  DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
  "$DIR"/unix_instruments.sh \
    -w "$simulator" \
    -D "$trace_results_dir/trace" \
    -t "$tracetemplate" \
    $bundle_dir \
    -e UIARESULTSPATH "$trace_results_dir" \
    -e UIASCRIPT "$automation_script" \
    -AppleLanguages "($language)" \
    -AppleLocale "$language" \
    "$@"

  find $trace_results_dir/Run\ 1/ -name *landscape*png -type f -exec sips -r -90 \{\} \;
}

function _copy_screenshots {
  # Since we're always clearing out the trace results before every run, we can
  # assume that any screenshots were saved in the "Run 1" directory. Copy them
  # to the destination's language folder!

  language="$1"

  mkdir -p "$destination/$language"
  cp $trace_results_dir/Run\ 1/*.png "$destination/$language"
}

function _close_sim {
  # I know, I know. It says "iPhone Simulator". For some reason,
  # that's the only way Applescript can identify it.
  osascript -e "tell application \"iPhone Simulator\" to quit"
}

main

#!/bin/bash
# Copyright (c) 2012 Jonathan Penn (http://cocoamanifest.net/)

# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:

# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.

# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

# Tell bash that we want the whole script to fail if any part fails.
set -e

# We require a parameter for where to put the results and the test script
destination="$1"
ui_script="$2"

# The locale identifiers for the languages you want to shoot
# Use the format like en-US zh-Hans for filenames compatible with iTunes
# connect upload tool
# FYI: get the locale names for you existing app with iTMSTransporter and:
# grep locale ~/Desktop/*.itmsp/metadata.xml  | grep name | sort -u
languages="en-US fr ja"

# The simulators we want to run the script against, declared as a Bash array.
# Run `instruments -s devices` to get a list of all the possible string values.
declare -a simulators=(
"Resizable iPad"
)

function main {
  _check_destination
  _check_ui_script
  _xcode clean build

  for simulator in "${simulators[@]}"; do
    for language in $languages; do
      _clean_trace_results_dir
      _run_automation "$ui_script" "$language" "$simulator"
      _copy_screenshots "$language"
    done
  done

  _close_sim

  echo
  echo "Screenshots complete!"
}

# Global variables to keep track of where everything goes
tmp_dir="/tmp"
build_dir="$tmp_dir/screen_shooter"
bundle_dir="$build_dir/app.app"
trace_results_dir="$build_dir/traces"

function _check_destination {
  # Abort if the destination directory already exists. Better safe than sorry.

  if [ -z "$destination" ]; then
    destination="$HOME/Desktop/screenshots"
  fi
  if [ -d "$destination" ]; then
    echo "Destination directory \"$destination\" already exists! Aborting."
    exit 1
  fi
}

function _check_ui_script {
  # Abort if the UI script does not exist.

  if [ -z "$ui_script" ]; then
    ui_script="./shoot_the_screens.js"
  fi
  if [ ! -f "$ui_script" ]; then
    echo "UI script \"$ui_script\" does not exist! Aborting."
    exit 1
  fi
}

function _xcode {
  # A wrapper around `xcodebuild` that tells it to build the app in the temp
  # directory. If your app uses workspaces or special schemes, you'll need to
  # specify them here.
  #
  # Use `man xcodebuild` for more information on how to build your project.
  if test -n "$(find . -maxdepth 1 -name '*.xcworkspace' -print -quit)"
  then
    base=$(basename *.xcworkspace .xcworkspace)
    # First build omits PRODUCT_NAME
    # Do NOT ask me why you need to build this twice for it to work
    # or how I became to know this fact
    xcodebuild -sdk "iphonesimulator$ios_version" \
      CONFIGURATION_BUILD_DIR="$build_dir/build" \
      -workspace $base.xcworkspace -scheme $base -configuration AdHoc \
      DSTROOT=$build_dir \
      OBJROOT=$build_dir \
      SYMROOT=$build_dir \
      ONLY_ACTIVE_ARCH=NO \
    "$@"
    xcodebuild -sdk "iphonesimulator$ios_version" \
      CONFIGURATION_BUILD_DIR="$build_dir/build" \
      PRODUCT_NAME=app \
      -workspace $base.xcworkspace -scheme $base -configuration AdHoc \
      DSTROOT=$build_dir \
      OBJROOT=$build_dir \
      SYMROOT=$build_dir \
      ONLY_ACTIVE_ARCH=NO \
    "$@"
    cp -r "$build_dir/build/app.app" "$build_dir"
  else
    xcodebuild -sdk "iphonesimulator$ios_version" \
      CONFIGURATION_BUILD_DIR=$build_dir \
      PRODUCT_NAME=app \
    "$@"
  fi
}

function _clean_trace_results_dir {
  # Removes the trace results directory. We need to do this because Instruments
  # keeps appending new trace runs and it's simpler for us to always assume
  # there's just one run recorded where we look for screenshots.

  rm -rf "$trace_results_dir"
  mkdir -p "$trace_results_dir"
}

function _run_automation {
  # Runs the UI Automation JavaScript file that actually takes the screenshots.

  automation_script="$1"
  language="$2"
  simulator="$3"

  echo "Running automation script \"$automation_script\"
          for \"$simulator\"
          in language \"${language}\"..."

  dev_tools_dir=`xcode-select -print-path`
  tracetemplate="Automation"

  # Check out the `unix_instruments.sh` script to see why we need this wrapper.
  DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
  "$DIR"/unix_instruments.sh \
    -w "$simulator" \
    -D "$trace_results_dir/trace" \
    -t "$tracetemplate" \
    $bundle_dir \
    -e UIARESULTSPATH "$trace_results_dir" \
    -e UIASCRIPT "$automation_script" \
    -AppleLanguages "($language)" \
    -AppleLocale "$language" \
    "$@"

  find $trace_results_dir/Run\ 1/ -name *landscape*png -type f -exec sips -r -90 \{\} \;
}

function _copy_screenshots {
  # Since we're always clearing out the trace results before every run, we can
  # assume that any screenshots were saved in the "Run 1" directory. Copy them
  # to the destination's language folder!

  language="$1"

  mkdir -p "$destination/$language"
  cp $trace_results_dir/Run\ 1/*.png "$destination/$language"
}

function _close_sim {
  # I know, I know. It says "iPhone Simulator". For some reason,
  # that's the only way Applescript can identify it.
  osascript -e "tell application \"iPhone Simulator\" to quit"
}

main


added build.sh to easily release official and beta builds

Signed-off-by: CHEF-KOCH <9f4a4111e9aa05e281acccd5195d7c51f900a2d7@gmail.com>

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/CHEF-KOCH/user.js-tweaker.git/'


simple daemon to start/stop cluster

simple daemon to start/stop cluster

#!/bin/bash
#
# Polls for the file radia-mpi.sh for a specific owner.
# If the file is removed, the cluster stops. If the file
# is newer, the cluster (re)starts. If the uptime on the
# current host is below a threshold, the cluster is stopped.
#
set -e
load_avg_threshold=2
poll_secs=15

container_id() {
    docker inspect -f '{{ .Id }}' radia-mpi 2>/dev/null || true
}

container_running() {
    if [[ $(docker inspect --format='{{ .State.Running }}' radia-mpi 2>/dev/null) == true ]]; then
        log radia-mpi=running
        return 0
    fi
    log radia-mpi=stopped
    return 1
}

integer_load_avg() {
    local up=( $(uptime) )
    local res="${up[-1]%.*}"
    log "load=$res"
    echo $res
}

log() {
    echo "$(date +%H%M%S) $@" 1>&2
}

mpi_stop() {
    container_id=
    log mpi_stop
    salt-call -l quiet state.apply jupyter-mpi-stop pillar="{'username': '$owner', force: true}"
}

mpi_start() {
    script_mtime=$(script_mtime)
    log mpi_start "owner=$owner script_mtime=$script_mtime script=$script"
    salt-call -l quiet state.apply jupyter-mpi-start pillar="{'username': '$owner'}"
    container_id=$(container_id)
    log mpi_start "container_id=$container_id"
}

new_job() {
    (( $(script_mtime) > $script_mtime ))
    local ret=$?
    if (( $ret == 0 )); then
        log "new_job script_mtime=$(script_mtime)"
    fi
    return $ret
}

no_job() {
    if (( $(script_mtime) == 0 )); then
        log "no_job script gone"
        return 0
    fi
    return 1
}

processes_ok() {
    if ! container_running; then
        return 1
    fi
    local np=$(ps axww | grep -c ':[0-9][0-9] bash /home/vagrant/jupyter/radia-mpi.sh$')
    local x
    for x in $(seq 10); do
        # Must consume most of CPU. If a single MPI process gets a
        # SEGV, the whole cluster will block for that process at
        # the next "barrier". The cluster has to stop.
        #
        # This particular check assumes that there is a
        # radia-mpi-host (worker) container running on this host,
        # too. Really we should poll all nodes to see if all
        # are ok. This might be a config param, but seems
        # like either the job is keeping all nodes busy
        # or it's not doing what it's supposed to be doing.
        if (( $(integer_load_avg) > $np - $load_avg_threshold )); then
            return 0
        fi
        # HACK: need to poll this so we don't get locked in a loop
        if new_job || no_job; then
            return 0
        fi
        sleep 60
    done
    return 1
}

script_mtime() {
    stat -c %Y "$script" 2>/dev/null || echo 0
}

main() {
    owner=$1
    if [[ -z $owner ]]; then
        echo "usage: bash $0 <owner>" 1>&2
        exit 1
    fi
    owner_home=/var/db/jupyterhub/home/$owner
    script=$owner_home/radia-mpi.sh
    container_id=$(container_id)
    script_mtime=$(script_mtime)
    log "script_mtime=$script_mtime container_id=$container_id"
    while true; do
        sleep $poll_secs
        if [[ -n $container_id ]]; then
            if ! processes_ok || new_job || no_job; then
                mpi_stop
            else
                : log 'main: job running'
            fi
            continue
        fi
        if new_job; then
            mpi_stop
            mpi_start
        else
            : log 'main: no job'
        fi
    done
    # this way the rest of the script doesn't get read in the
    # case of editing it while it is running
    exit
}

main "$@"

error: pathspec 'srv/salt/cluster/daemon.sh' did not match any file(s) known to git


chore(build): trigger npm postinstall when npm install is skipped

we temporarily need this because some of our dependencies are being installed via npm postinstall

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/HenriqueLimas/angular.git/'


format-code.sh

format-code.sh

#!/bin/sh
set -e
autoflake  -i -r --remove-all-unused-imports .
autoflake  -i -r --remove-unused-variables .
isort --recursive .
black .

error: pathspec 'format-code.sh' did not match any file(s) known to git


add feature-complete virtualenvwrapper replacement

#!/bin/bash

export VIRTUALENV_DIR=${VIRTUALENV_DIR:-$HOME/Environments}
mkdir -p "$VIRTUALENV_DIR"

function deactivate_venv() {
    if type -t deactivate > /dev/null; then
        deactivate;
    fi;
}

function workon() {
    deactivate_venv
    source "$VIRTUALENV_DIR/$1/bin/activate";
}

function mkvirtualenv() {
    if [ ! -d "$VIRTUALENV_DIR/$1" ]; then
        virtualenv "$VIRTUALENV_DIR/$1";
    fi;
    workon "$1";
}

function rmvirtualenv() {
    deactivate_venv
    rm -rf "$VIRTUALENV_DIR/$1";
}

function lsvirtualenv() {
    ls -1 "$VIRTUALENV_DIR";
}

error: pathspec 'virtualenvwrapperlite.sh' did not match any file(s) known to git


Create wakeonlan.sh

Create wakeonlan.sh

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/jason-ennis/smart-home.git/'


this is needed at nci

this is needed at nci

#!/bin/sh

PATH=/usr/bin:/bin
export PATH

find /local/content/web/mcl/mcl.nci.nih.gov/var -type d -print0 | xargs -0 chmod 755
find /local/content/web/mcl/mcl.nci.nih.gov/var -type f -print0 | xargs -0 chmod 644

error: pathspec 'support/fix.sh' did not match any file(s) known to git


Now activate all software in a single config script.

Enabling QIIME, usearch, additional tools and tutorial specific software.

export PYTHONPATH=/opt/exp_soft/qiime/lib/python2.7/site-packages/:$PYTHONPATH
export PATH=/opt/exp_soft/qiime/bin/:$PATH
export PATH=/opt/exp_soft/qiime/python-2.7.3/bin:$PATH
export RDP_JAR_PATH=/opt/exp_soft/qiime/app/rdp_classifier_2.2/rdp_classifier-2.2.jar
export PATH=/opt/exp_soft/qiime/R-3.0.2/bin:$PATH
export PATH=$PATH:/opt/exp_soft/qiime/app/ChimeraSlayer
export PYRO_LOOKUP_FILE=/opt/exp_soft/qiime/packages/extras/AmpliconNoiseV1.27/Data/LookUp_E123.dat
export SEQ_LOOKUP_FILE=/opt/exp_soft/qiime/packages/extras/AmpliconNoiseV1.27/Data/Tran.dat
export SOURCETRACKER_PATH=/opt/exp_soft/qiime/packages/extras/sourcetracker-0.9.5
export BLASTMAT=/opt/exp_soft/qiime/packages/extras/blast-2.2.22/data
export DONT_USE_MPI=1

error: pathspec 'config/activate_soft.sh' did not match any file(s) known to git