Added gcam alias

Added gcam alias

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/Victoor/My-Oh-ZSH-Config.git/'


new wuxiaworld downloader

new wuxiaworld downloader

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/slundi/scripts.git/'


Delete package.sh

Delete package.sh

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/mookins/j2mod.git/'


Tarball generation script.

Tarball generation script.

#!/bin/bash
. ubiquitous_bash.sh

mustBeRoot

#Execute instructions in ChRoot environment.
executeChRoot() {
        env -i HOME="/root" SHELL="/bin/bash" TERM="xterm" PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" $(which chroot) ./ChRoot /bin/bash -c "$@"
}

#Automatically merge proposed etc filesystem changes.
autoEtcUpdate() {
        executeChRoot "echo -e y\\ny\\ny\\ny\\ny\\ny\\ny\\ny\\ny | etc-update --automode -3"
}

#####-Pack Up OS-#####

./umountChRoot.sh

executeChRoot "tar -cpf mirage335OS-buildAssets.tar -C / ./usr/portage/distfiles ./usr/portage/packages"

executeChRoot "tar --exclude usr/portage/packages --exclude usr/portage/distfiles --exclude mirage335OS-buildAssets.tar -cpf mirage335OS-lite.tar -C / ."

error: pathspec 'package.sh' did not match any file(s) known to git


:walking: hit it and quit it repo haha BAM!

#!/bin/bash
# sample imagemagick lines to reformat screen captures to have ios like framing and drop shadow
# a rappleshot component
# designed for use with nautilus to give context menu click method of highlight and swiftly (1 click) processing
#   multiple image files of various extensions into cool Apple IOS Style Screen shots. Turn on drop shadow and effects
#   the checkboxes basically for your gnome-screenshots utlity then reprocess with this tool. same as in my other repo
#   if you use nautlus/Unity/Gnome then drop this in $HOME/.gnome2/nautilus scripts/... without the extension and
#   permission it root but executable
# by ray anthony
# 5/1/2015
while [ "$#" -gt 0 ]; do
        picture="$1"
		filename=$(basename "$1")
		extension="${filename##*.}"
		filename="${filename%.*}"
		convert "$picture" \( +clone -background black -shadow 80x20+0+15 \) +swap -background transparent -layers merge +repage "${filename}_rappleshot.png"
        shift
done

#!/bin/bash
while [ $# -gt 0 ]; do
        picture=$1
		filename=$(basename "$1")
		extension="${filename##*.}"
		filename="${filename%.*}"
		convert "$picture" \( +clone -background black -shadow 80x20+0+15 \) +swap -background transparent -layers merge +repage "${filename}_rappleshot${i}.png"
        shift
done

error: pathspec 'rappleshots.sh' did not match any file(s) known to git


não remove locale.alias

não remove locale.alias

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/zanardo/dockerfiles.git/'


contrib: revert changes to github-merge.sh in #5038

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/wastholm/bitcoin.git/'


Added a script to create a branch.

Added a script to create a branch.

git-svn-id: 9146c88ff6d39b48099bf954d15d68f687b3fa69@8692 28e8926c-6b08-0410-baaa-805c5e19b8d6

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/lpellegr/scheduling.git/'


updates.

updates.

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/pepperstack/nebula.git/'


Adiciona script gen-gh-pages

Adiciona script gen-gh-pages

BUILD_DIR="$TRAVIS_BUILD_DIR/gh-pages/"
mkdir -p $BUILD_DIR

cat > $BUILD_DIR/chromium.xml <<EOL
<?xml version='1.0' encoding='UTF-8'?>
<gupdate xmlns='http://www.google.com/update2/response' protocol='2.0'>
  <app appid='magjmaimhngoidhmklnpihkdjkggbpnp'>
    <updatecheck codebase='https://github.com/burlesco/burlesco/releases/download/$TRAVIS_TAG/burlesco-chromium.crx' version='${TRAVIS_TAG:1}' />
  </app>
</gupdate>
EOL

cat > $BUILD_DIR/firefox.json <<EOL
{
  "addons": {
    "burlesco@burles.co": {
      "updates": [
        {
          "version": "${TRAVIS_TAG:1}",
          "update_link": "https://github.com/burlesco/burlesco/releases/download/$TRAVIS_TAG/burlesco-firefox.xpi"
        }
      ]
    }
  }
}
EOL

error: pathspec 'scripts/gen-gh-pages.sh' did not match any file(s) known to git


Don't publish non-master docs to master location.

The current run-docs script only moves the publishing location on
tags or stable branches.  This makes it so that master docs stay
where they are, stable branches and tags have the current behavior,
and anything else is placed in the branch name.  This should affect
milestone-proposed (which will be published to .../milestone-proposed/)
and feature branches (.../feature/foo).  Currently, these two cases
are in a race with master -- last change wins.

Change-Id: If1e09d31a7001255746ba1a91b641b95f42fcfbb
Reviewed-on: https://review.openstack.org/31300
Reviewed-by: Clark Boylan <635fc31d01fa593720982fa0568e87170cedbe2c@gmail.com>
Approved: James E. Blair <e81a02a9d6405977f63a769d9f6ec8ed9d924325@inaugust.com>
Tested-by: Jenkins

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/open-switch/infra_puppet-jenkins.git/'


Fix extra repo regression

Fix extra repo regression

REPOLIST is replaced on heat side, it'sactually not shell variable,
using ${} causes that this placeholder is not replaced and the
script fails.

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/markllama/openshift-on-openstack.git/'


First commit of raw script

First commit of raw script

#!/bin/bash

# George miller
# 6-10-2022
# If you do not have permission to run, try: chmod u+x publicscript.sh
#
# The purpose of this script is to update the file locations of certain files in an xcode project
# by their status as public or private.
# a Ruby script exists for the facebook project.  We want a more robust script, for SDL_IOS

# "Ensure that all files marked public in the project.pbxproj file are in
# the public folder and the same for non-public files (in the private folder).
# Public files should also be referenced in SmartDeviceLink.h"

project_directory="../sdl_ios/"
project_file="SmartDeviceLink-iOS.xcodeproj/project.pbxproj"
path_pre="SmartDeviceLink/"
#public_dir=$path_pre"/public"
#private_dir=$path_pre"/private"

# Cd into the correct project directory
cd $project_directory

# Pick out the lines that have "Public" in them, and filters everything down to get the fileref
public_fileref_list=$(sed -n '/Public/{s/^.*fileRef//;s/^[[:space:]]*//;s/\/.*//;s/=//;p;}' $project_file)

# Loop through list of fileref and use sed on the project file to find the file path
public_filepaths=()
#other_filepaths=()

for public_fileref in $public_fileref_list
do
    # Pick out the lines with the file reference
    publicref_lines="$(sed -n "/$public_fileref/{p;}" $project_file)"
    # Trim everything before "path", then eliminate "=", " ", and quotes
    publicref_path=$(sed -n '/path/{s/^.*path//;s/=//;s/\"//g;s/^[[:space:]]*//g;p;}' <<< "$publicref_lines")
    # Eliminate everything after the ";"
    publicref_path=$(sed -n 's/\;.*//p;' <<< "$publicref_path")
    # All of these have the attribute "public"
    public_filepaths+=$publicref_path

    # Identify lines that do not have "public" in the path
    #if [[ "$publicref_path" != *"public"* ]]; then
    #    echo $publicref_path
    #    other_filepaths+=$publicref_path
    #fi
    #echo $publicref_path

    # Test to see if the file is at that path
    # Some of the paths already have the pre on them
    test_path=$publicref_path
    if [[ "$publicref_path" != *$path_pre* ]]; then
        test_path=$path_pre$publicref_path
    fi
    if [ ! -f $test_path ]; then
        echo $publicref_path
        echo $test_path
        echo "ALERT"
    fi
done

error: pathspec 'scripts/publicscript.sh' did not match any file(s) known to git


Further clean pip_new.sh. Only install auditwheel once.

PiperOrigin-RevId: 278966344
Change-Id: I91d91eb86b5a326b5253a026d2c6757ac70a1366

#!/usr/bin/env bash
# Copyright 2016 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
# Build the Python PIP installation package for TensorFlow and install
# the package.
#
# Usage:
#   pip_new.sh
#
# Required step(s):
#   Run configure.py prior to running this script.
#
# Required environment variable(s):
#   CONTAINER_TYPE:      (CPU | GPU)
#   OS_TYPE:             (UBUNTU | MACOS)
#   TF_PYTHON_VERSION:   (python2 | python2.7 | python3.5 | python3.7)
#   TF_BUILD_FLAGS:      Bazel build flags.
#                          e.g. TF_BUILD_FLAGS="--config=opt"
#   TF_TEST_FLAGS:       Bazel test flags.
#                          e.g. TF_TEST_FLAGS="--verbose_failures=true \
#                               --build_tests_only --test_output=errors"
#   TF_TEST_FILTER_TAGS: Filtering tags for bazel tests. More specifically,
#                        input tags for `--test_filter_tags` flag.
#                          e.g. TF_TEST_FILTER_TAGS="no_pip,-nomac,no_oss"
#   TF_TEST_TARGETS:     Bazel test targets.
#                          e.g. TF_TEST_TARGETS="//tensorflow/... \
#                               -//tensorflow/contrib/... \
#                               -//tensorflow/python/..."
#   IS_NIGHTLY:          Nightly run flag.
#                          e.g. IS_NIGHTLY=1  # nightly runs
#                          e.g. IS_NIGHTLY=0  # non-nightly runs
#
# Optional environment variables. If provided, overwrites any default values.
#   TF_PIP_TESTS:        PIP tests to run. If NOT specified, skips all tests.
#                          e.g. TF_PIP_TESTS="test_pip_virtualenv_clean \
#                               test_pip_virtualenv_clean \
#                               test_pip_virtualenv_oss_serial"
#   TF_PROJECT_NAME:     Name of the project. This string will be pass onto
#                        the wheel file name. For nightly builds, it will be
#                        overwritten to 'tf_nightly'. For gpu builds, '_gpu'
#                        will be appended.
#                          e.g. TF_PROJECT_NAME="tensorflow"
#                          e.g. TF_PROJECT_NAME="tf_nightly_gpu"
#   TF_PIP_TEST_ROOT:    Root directory for building and testing pip pkgs.
#                          e.g. TF_PIP_TEST_ROOT="pip_test"
#   TF_BUILD_BOTH_GPU_PACKAGES:    (1 | 0)
#                                  1 will build both tensorflow (w/ gpu support)
#                                  and tensorflow-gpu pip package. Will
#                                  automatically handle adding/removing of _gpu
#                                  suffix depending on what project name was
#                                  passed.
#
# To-be-deprecated variable(s).
#   GIT_TAG_OVERRIDE:    Values for `--git_tag_override`. This flag gets passed
#                        in as `--action_env` for bazel build and tests.
#   TF_BUILD_INSTALL_EXTRA_PIP_PACKAGES:
#                        Additonal pip packages to be installed.
#                        Caveat: pip version needs to be checked prior.
#
# ==============================================================================

# set bash options
set -e
set -x

###########################################################################
# General helper function(s)
###########################################################################

# Strip leading and trailing whitespaces
str_strip () {
  echo -e "$1" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'
}

# Convert string to all lower case
lowercase() {
  if [[ -z "${1}" ]]; then
    die "Nothing to convert to lowercase. No argument given."
  fi
  echo "${1}" | tr '[:upper:]' '[:lower:]'
}

check_global_vars() {
  # Check container type
  if ! [[ ${CONTAINER_TYPE} == "cpu" ]] && \
     ! [[ ${CONTAINER_TYPE} == "rocm" ]] && \
     ! [[ ${CONTAINER_TYPE} == "gpu" ]]; then
    die "Error: Provided CONTAINER_TYPE \"${CONTAINER_TYPE}\" "\
        "is not supported."
  fi
  # Check OS type
  if ! [[ ${OS_TYPE} == "ubuntu" ]] && \
     ! [[ ${OS_TYPE} == "macos" ]]; then
    die "Error: Provided OS_TYPE \"${OS_TYPE}\" is not supported."
  fi
  # Check build flags
  if [[ -z ${TF_BUILD_FLAGS} ]]; then
    die "Error: TF_BUILD_FLAGS is not specified."
  fi
  # Check test flags
  if [[ -z ${TF_TEST_FLAGS} ]]; then
    die "Error: TF_TEST_FLAGS is not specified."
  fi
  # Check test filter tags
  if [[ -z ${TF_TEST_FILTER_TAGS} ]]; then
    die "Error: TF_TEST_FILTER_TAGS is not specified."
  fi
  # Check test targets
  if [[ -z ${TF_TEST_TARGETS} ]]; then
    die "Error: TF_TEST_TARGETS is not specified."
  fi
  # Check nightly status
  if [[ -z ${IS_NIGHTLY} ]]; then
    die "Error: IS_NIGHTLY is not specified."
  fi
}

add_test_filter_tag() {
  EMPTY=""
  while true; do
    FILTER="${1:$EMPTY}"
    if ! [[ $TF_TEST_FILTER_TAGS == *"${FILTER}"* ]]; then
      TF_TEST_FILTER_TAGS="${FILTER},${TF_TEST_FILTER_TAGS}"
    fi
    shift
    if [[ -z "${1}" ]]; then
      break
    fi
  done
}

remove_test_filter_tag() {
  EMPTY=""
  while true; do
    FILTER="${1:$EMPTY}"
    TF_TEST_FILTER_TAGS="$(echo ${TF_TEST_FILTER_TAGS} | sed -e 's/^'${FILTER}',//g' -e 's/,'${FILTER}'//g')"
    shift
    if [[ -z "${1}" ]]; then
      break
    fi
  done
}

# Clean up bazel build & test flags with proper configuration.
update_bazel_flags() {
  # Add git tag override flag if necessary.
  GIT_TAG_STR=" --action_env=GIT_TAG_OVERRIDE"
  if [[ -z "${GIT_TAG_OVERRIDE}" ]] && \
    ! [[ ${TF_BUILD_FLAGS} = *${GIT_TAG_STR}* ]]; then
    TF_BUILD_FLAGS+="${GIT_TAG_STR}"
  fi
  # Clean up whitespaces
  TF_BUILD_FLAGS=$(str_strip "${TF_BUILD_FLAGS}")
  TF_TEST_FLAGS=$(str_strip "${TF_TEST_FLAGS}")
  # Cleaned bazel flags
  echo "Bazel build flags (cleaned):\n" "${TF_BUILD_FLAGS}"
  echo "Bazel test flags (cleaned):\n" "${TF_TEST_FLAGS}"
}

update_test_filter_tags() {
  # Add test filter tags
  # This script is for validating built PIP packages. Add pip tags.
  add_test_filter_tag -no_pip -nopip
  # MacOS filter tags
  if [[ ${OS_TYPE} == "macos" ]]; then
    remove_test_filter_tag nomac no_mac
    add_test_filter_tag -nomac -no_mac
  fi
  echo "Final test filter tags: ${TF_TEST_FILTER_TAGS}"
}

# Check currently running python and pip version
check_python_pip_version() {
  # Check if only the major version of python is provided by the user.
  MAJOR_VER_ONLY=0
  if [[ ${#PYTHON_VER} -lt 9 ]]; then
    # User only provided major version (e.g. 'python2' instead of 'python2.7')
    MAJOR_VER_ONLY=1
  fi

  # Retrieve only the version number of the user requested python.
  PYTHON_VER_REQUESTED=${PYTHON_VER:6:3}
  echo "PYTHON_VER_REQUESTED: ${PYTHON_VER_REQUESTED}"

  # Retrieve only the version numbers of the python & pip in use currently.
  PYTHON_VER_IN_USE=$(python --version 2>&1)
  PYTHON_VER_IN_USE=${PYTHON_VER_IN_USE:7:3}
  PIP_VER_IN_USE=$(${PIP_BIN_PATH} --version)
  PIP_VER_IN_USE=${PIP_VER_IN_USE:${#PIP_VER_IN_USE}-4:3}

  # If only major versions are applied, drop minor versions.
  if [[ $MAJOR_VER_ONLY == 1 ]]; then
    PYTHON_VER_IN_USE=${PYTHON_VER_IN_USE:0:1}
    PIP_VER_IN_USE=${PIP_VER_IN_USE:0:1}
  fi

  # Check if all versions match.
  echo -e "User requested python version: '${PYTHON_VER_REQUESTED}'\n" \
    "Detected python version in use: '${PYTHON_VER_IN_USE}'\n"\
    "Detected pip version in use: '${PIP_VER_IN_USE}'"
  if ! [[ $PYTHON_VER_REQUESTED == $PYTHON_VER_IN_USE ]]; then
    die "Error: Mismatch in python versions detected."
  else:
    echo "Python and PIP versions in use match the requested."
  fi
}

###########################################################################
# Setup: directories, local/global variables
###########################################################################

# Script directory and source necessary files.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "${SCRIPT_DIR}/builds_common.sh"

# Required global variables
# Checks on values for these vars are done in "Build TF PIP Package" section.
CONTAINER_TYPE=$(lowercase "${CONTAINER_TYPE}")
OS_TYPE=$(lowercase "${OS_TYPE}")
PYTHON_VER=$(lowercase "${TF_PYTHON_VERSION}")

# Python bin path
if [[ -z "$PYTHON_BIN_PATH" ]]; then
  die "Error: PYTHON_BIN_PATH was not provided. Did you run configure?"
fi

# Set optional environment variables; set to default in case not user defined.
DEFAULT_PIP_TESTS="" # Do not run any tests by default
DEFAULT_PROJECT_NAME="tensorflow"
DEFAULT_PIP_TEST_ROOT="pip_test"
DEFAULT_BUILD_BOTH_GPU_PACKAGES=0
# Take in optional global variables
PIP_TESTS=${TF_PIP_TESTS:-$DEFAULT_PIP_TESTS}
PROJECT_NAME=${TF_PROJECT_NAME:-$DEFAULT_PROJECT_NAME}
PIP_TEST_ROOT=${TF_PIP_TEST_ROOT:-$DEFAULT_PIP_TEST_ROOT}
BUILD_BOTH_GPU_PACKAGES=${TF_BUILD_BOTH_GPU_PACKAGES:-$DEFAULT_BUILD_BOTH_GPU_PACKAGES}

# Local variables
PIP_WHL_DIR="${KOKORO_ARTIFACTS_DIR}/tensorflow/${PIP_TEST_ROOT}/whl"
mkdir -p "${PIP_WHL_DIR}"
PIP_WHL_DIR=$(realpath "${PIP_WHL_DIR}") # Get absolute path
WHL_PATH=""
# Determine the major.minor versions of python being used (e.g., 2.7).
# Useful for determining the directory of the local pip installation.
PY_MAJOR_MINOR_VER=$(${PYTHON_BIN_PATH} -c "print(__import__('sys').version)" 2>&1 | awk '{ print $1 }' | head -n 1 | cut -c1-3)

if [[ -z "${PY_MAJOR_MINOR_VER}" ]]; then
  die "ERROR: Unable to determine the major.minor version of Python."
fi
echo "Python binary path to be used in PIP install: ${PYTHON_BIN_PATH} "\
"(Major.Minor version: ${PY_MAJOR_MINOR_VER})"
PYTHON_BIN_PATH_INIT=${PYTHON_BIN_PATH}
PIP_BIN_PATH="$(which pip${PY_MAJOR_MINOR_VER})"

# PIP packages
INSTALL_EXTRA_PIP_PACKAGES="portpicker scipy scikit-learn ${TF_BUILD_INSTALL_EXTRA_PIP_PACKAGES}"

###########################################################################
# Build TF PIP Package
###########################################################################

# First remove any already existing binaries for a clean start and test.
if [[ -d ${PIP_TEST_ROOT} ]]; then
  echo "Test root directory ${PIP_TEST_ROOT} already exists. Deleting it."
  sudo rm -rf ${PIP_TEST_ROOT}
fi

# Check that global variables are properly set.
check_global_vars

# Check if in a virtualenv and exit if yes.
IN_VENV=$(python -c 'import sys; print("1" if hasattr(sys, "real_prefix") else "0")')
if [[ "$IN_VENV" == "1" ]]; then
  echo "It appears that we are already in a virtualenv. Deactivating..."
  deactivate || source deactivate || die "FAILED: Unable to deactivate from existing virtualenv."
fi

# Configure python. Obtain the path to python binary.
source tools/python_bin_path.sh
# Assume PYTHON_BIN_PATH is exported by the script above.
if [[ -z "$PYTHON_BIN_PATH" ]]; then
  die "PYTHON_BIN_PATH was not provided. Did you run configure?"
fi

# Bazel build the file.
PIP_BUILD_TARGET="//tensorflow/tools/pip_package:build_pip_package"
# Clean bazel cache.
bazel clean
# Clean up and update bazel flags
update_bazel_flags
# Build. This outputs the file `build_pip_package`.
bazel build ${TF_BUILD_FLAGS} ${PIP_BUILD_TARGET} || \
  die "Error: Bazel build failed for target: '${PIP_BUILD_TARGET}'"

###########################################################################
# Test function(s)
###########################################################################

test_pip_virtualenv() {
  # Get args
  VENV_DIR_NAME=$1
  shift
  TEST_TYPE_FLAG=$1

  # Check test type args
  if ! [[ ${TEST_TYPE_FLAG} == "--oss_serial" ]] && \
     ! [[ ${TEST_TYPE_FLAG} == "--clean" ]] && \
     ! [[ ${TEST_TYPE_FLAG} == "" ]]; then
     die "Error: Wrong test type given. TEST_TYPE_FLAG=${TEST_TYPE_FLAG}"
  fi

  # Create virtualenv directory for test
  VENV_DIR="${PIP_TEST_ROOT}/${VENV_DIR_NAME}"

  # Activate virtualenv
  create_activate_virtualenv ${TEST_TYPE_FLAG} ${VENV_DIR}
  # Install TF with pip
  TIME_START=$SECONDS
  install_tensorflow_pip "${WHL_PATH}"
  TIME_ELAPSED=$(($SECONDS - $TIME_START))
  echo "Time elapsed installing tensorflow = ${TIME_ELAPSED} seconds"

  # cd to a temporary directory to avoid picking up Python files in the source
  # tree.
  TMP_DIR=$(mktemp -d)
  pushd "${TMP_DIR}"

  # Run a quick check on tensorflow installation.
  RET_VAL=$(python -c "import tensorflow as tf; t1=tf.constant([1,2,3,4]); t2=tf.constant([5,6,7,8]); print(tf.add(t1,t2).shape)")

  # Return to original directory. Remove temp dirs.
  popd
  sudo rm -rf "${TMP_DIR}"

  # Check result to see if tensorflow is properly installed.
  if ! [[ ${RET_VAL} == *'(4,)'* ]]; then
    echo "PIP test on virtualenv (non-clean) FAILED"
    return 1
  fi

  # Install extra pip packages, if specified.
  for PACKAGE in ${INSTALL_EXTRA_PIP_PACKAGES}; do
    echo "Installing extra pip package required by test-on-install: ${PACKAGE}"

    ${PIP_BIN_PATH} install ${PACKAGE}
    if [[ $? != 0 ]]; then
      echo "${PIP_BIN_PATH} install ${PACKAGE} FAILED."
      deactivate || source deactivate || die "FAILED: Unable to deactivate from existing virtualenv."
      return 1
    fi
  done

  # Run bazel test.
  run_test_with_bazel ${TEST_TYPE_FLAG}
  RESULT=$?

  # Deactivate from virtualenv.
  deactivate || source deactivate || die "FAILED: Unable to deactivate from existing virtualenv."
  sudo rm -rf "${VENV_DIR}"

  return $RESULT
}

###########################################################################
# Test helper function(s)
###########################################################################

create_activate_virtualenv() {
  VIRTUALENV_FLAGS="--system-site-packages"
  if [[ "${1}" == "--clean" ]]; then
    VIRTUALENV_FLAGS=""
    shift
  elif [[ "${1}" == "--oss_serial" ]]; then
    shift
  fi

  VIRTUALENV_DIR="${1}"
  if [[ -d "${VIRTUALENV_DIR}" ]]; then
    if sudo rm -rf "${VIRTUALENV_DIR}"
    then
      echo "Removed existing virtualenv directory: ${VIRTUALENV_DIR}"
    else
      die "Failed to remove existing virtualenv directory: ${VIRTUALENV_DIR}"
    fi
  fi

  if mkdir -p "${VIRTUALENV_DIR}"
  then
    echo "Created virtualenv directory: ${VIRTUALENV_DIR}"
  else
    die "FAILED to create virtualenv directory: ${VIRTUALENV_DIR}"
  fi

  # Use the virtualenv from the default python version (i.e., python-virtualenv)
  # to create the virtualenv directory for testing. Use the -p flag to specify
  # the python version inside the to-be-created virtualenv directory.
  ${PYTHON_BIN_PATH_INIT} -m virtualenv -p ${PYTHON_BIN_PATH_INIT} ${VIRTUALENV_FLAGS} ${VIRTUALENV_DIR} || \
    die "FAILED: Unable to create virtualenv"

  source "${VIRTUALENV_DIR}/bin/activate" || \
    die "FAILED: Unable to activate virtualenv in ${VIRTUALENV_DIR}"

  # Update .tf_configure.bazelrc with venv python path for bazel test.
  PYTHON_BIN_PATH="$(which python)"
  yes "" | ./configure
}

install_tensorflow_pip() {
  if [[ -z "${1}" ]]; then
    die "Please provide a proper wheel file path."
  fi

  # Set path to pip.
  PIP_BIN_PATH="$(which pip${PY_MAJOR_MINOR_VER})"

  # Print python and pip bin paths
  echo "PYTHON_BIN_PATH to be used to install the .whl: ${PYTHON_BIN_PATH}"
  echo "PIP_BIN_PATH to be used to install the .whl: ${PIP_BIN_PATH}"

  # Upgrade pip so it supports tags such as cp27mu, manylinux2010 etc.
  echo "Upgrade pip in virtualenv"

  # NOTE: pip install --upgrade pip leads to a documented TLS issue for
  # some versions in python
  curl https://bootstrap.pypa.io/get-pip.py | ${PYTHON_BIN_PATH} || \
    die "Error: pip install (get-pip.py) FAILED"

  # Check that requested python version matches configured one.
  check_python_pip_version

  # Force upgrade of setuptools. This must happen before the pip install of the
  # WHL_PATH, which pulls in absl-py, which uses install_requires notation
  # introduced in setuptools >=20.5. The default version of setuptools is 5.5.1,
  # which is too old for absl-py.
  ${PIP_BIN_PATH} install --upgrade setuptools==39.1.0 || \
    die "Error: setuptools install, upgrade FAILED"

  # Force tensorflow reinstallation. Otherwise it may not get installed from
  # last build if it had the same version number as previous build.
  PIP_FLAGS="--upgrade --force-reinstall"
  ${PIP_BIN_PATH} install ${PIP_FLAGS} ${WHL_PATH} || \
    die "pip install (forcing to reinstall tensorflow) FAILED"
  echo "Successfully installed pip package ${WHL_PATH}"

  # Force downgrade of setuptools. This must happen after the pip install of the
  # WHL_PATH, which ends up upgrading to the latest version of setuptools.
  # Versions of setuptools >= 39.1.0 will cause tests to fail like this:
  #   ImportError: cannot import name py31compat
  ${PIP_BIN_PATH} install --upgrade setuptools==39.1.0 || \
    die "Error: setuptools install, upgrade FAILED"

  # Install the future package in the virtualenv. Installing it in user system
  # packages does not appear to port it over when creating a virtualenv.
  #   ImportError: No module named builtins
  ${PIP_BIN_PATH} install --upgrade "future>=0.17.1" || \
    die "Error: future install, upgrade FAILED"

  # Install the gast package in the virtualenv. Installing it in user system
  # packages does not appear to port it over when creating a virtualenv.
  ${PIP_BIN_PATH} install --upgrade "gast==0.2.2" || \
    die "Error: gast install, upgrade FAILED"

}

run_test_with_bazel() {
  IS_OSS_SERIAL=0
  if [[ "${1}" == "--oss_serial" ]]; then
    IS_OSS_SERIAL=1
  fi
  TF_GPU_COUNT=${TF_GPU_COUNT:-4}

  # PIP tests should have a "different" path. Different than the one we place
  # virtualenv, because we are deleting and recreating it here.
  PIP_TEST_PREFIX=bazel_pip
  TEST_ROOT=$(pwd)/${PIP_TEST_PREFIX}
  sudo rm -rf $TEST_ROOT
  mkdir -p $TEST_ROOT
  ln -s $(pwd)/tensorflow $TEST_ROOT/tensorflow

  if [[ "${IS_OSS_SERIAL}" == "1" ]]; then
    remove_test_filter_tag -no_oss
    add_test_filter_tag oss_serial
  else
    add_test_filter_tag -oss_serial
  fi

  # Clean the bazel cache
  bazel clean
  # Clean up flags before running bazel commands
  update_bazel_flags
  # Clean up and update test filter tags
  update_test_filter_tags

  # Figure out how many concurrent tests we can run and do run the tests.
  BAZEL_PARALLEL_TEST_FLAGS=""
  if [[ $CONTAINER_TYPE == "gpu" ]]; then
    # Number of test threads is the number of GPU cards available.
    if [[ $OS_TYPE == "macos" ]]; then
      BAZEL_PARALLEL_TEST_FLAGS="--local_test_jobs=1"
    else
      BAZEL_PARALLEL_TEST_FLAGS="--local_test_jobs=${TF_GPU_COUNT} \
        --run_under=//tensorflow/tools/ci_build/gpu_build:parallel_gpu_execute"
    fi
  else
    # Number of test threads is the number of physical CPUs.
    if [[ $OS_TYPE == "macos" ]]; then
      BAZEL_PARALLEL_TEST_FLAGS="--local_test_jobs=$(sysctl -n hw.ncpu)"
    else
      BAZEL_PARALLEL_TEST_FLAGS="--local_test_jobs=$(grep -c ^processor /proc/cpuinfo)"
    fi
  fi

  if [[ ${IS_OSS_SERIAL} == 1 ]]; then
    BAZEL_PARALLEL_TEST_FLAGS="--local_test_jobs=1"
  fi

  TEST_TARGETS_SYMLINK=""
  for TARGET in ${TF_TEST_TARGETS[@]}; do
    TARGET_NEW=$(echo ${TARGET} | sed -e "s/\/\//\/\/${PIP_TEST_PREFIX}\//g")
    TEST_TARGETS_SYMLINK+="${TARGET_NEW} "
  done
  echo "Test targets (symlink): ${TEST_TARGETS_SYMLINK}"

  # Run the test.
  bazel test --build_tests_only ${TF_TEST_FLAGS} ${BAZEL_PARALLEL_TEST_FLAGS} --test_tag_filters=${TF_TEST_FILTER_TAGS} -k -- ${TEST_TARGETS_SYMLINK}

  unlink ${TEST_ROOT}/tensorflow
}

run_all_tests() {
  if [[ -z "${PIP_TESTS}" ]]; then
    echo "No test was specified to run. Skipping all tests."
    return 0
  fi
  FAIL_COUNTER=0
  PASS_COUNTER=0
  for TEST in ${PIP_TESTS[@]}; do

    # Run tests.
    case "${TEST}" in
    "test_pip_virtualenv_clean")
      test_pip_virtualenv venv_clean --clean
      ;;
    "test_pip_virtualenv_non_clean")
      test_pip_virtualenv venv
      ;;
    "test_pip_virtualenv_oss_serial")
      test_pip_virtualenv venv_oss --oss_serial
      ;;
    *)
      die "No matching test ${TEST} was found. Stopping test."
      ;;
    esac

    # Check and update the results.
    RETVAL=$?

    # Update results counter
    if [ ${RETVAL} -eq 0 ]; then
      echo "Test (${TEST}) PASSED. (PASS COUNTER: ${PASS_COUNTER})"
      PASS_COUNTER=$(($PASS_COUNTER+1))
    else
      echo "Test (${TEST}) FAILED. (FAIL COUNTER: ${FAIL_COUNTER})"
      FAIL_COUNTER=$(($FAIL_COUNTER+1))
    fi
  done
  printf "${PASS_COUNTER} PASSED | ${FAIL_COUNTER} FAILED"
  if [[ "${FAIL_COUNTER}" == "0" ]]; then
    printf "PIP tests ${COLOR_GREEN}PASSED${COLOR_NC}\n"
    return 0
  else:
    printf "PIP tests ${COLOR_RED}FAILED${COLOR_NC}\n"
    return 1
  fi
}

###########################################################################
# Build TF PIP Wheel file
###########################################################################

# Update the build flags for building whl.
# Flags: GPU, OS, tf_nightly, project name
GPU_FLAG=""
NIGHTLY_FLAG=""

# TF Nightly flag
if [[ "$IS_NIGHTLY" == 1 ]]; then
  # If 'nightly' is not specified in the project name already, then add.
  if ! [[ $PROJECT_NAME == *"nightly"* ]]; then
    echo "WARNING: IS_NIGHTLY=${IS_NIGHTLY} but requested project name \
    (PROJECT_NAME=${PROJECT_NAME}) does not include 'nightly' string. \
    Renaming it to 'tf_nightly'."
    PROJECT_NAME="tf_nightly"
  fi
  NIGHTLY_FLAG="--nightly_flag"
fi

# CPU / GPU flag
if [[ ${CONTAINER_TYPE} == "gpu" ]]; then
  GPU_FLAG="--gpu"
  if ! [[ $PROJECT_NAME == *"gpu"* ]]; then
    # Only update PROJECT_NAME if TF_PROJECT_NAME is not set
    if [[ -z "${TF_PROJECT_NAME}" ]]; then
      echo "WARNING: GPU is specified but requested project name (PROJECT_NAME=${PROJECT_NAME}) \
      does not include 'gpu'. Appending '_gpu' to the project name."
      PROJECT_NAME="${PROJECT_NAME}_gpu"
    fi
  fi
fi

./bazel-bin/tensorflow/tools/pip_package/build_pip_package ${PIP_WHL_DIR} ${GPU_FLAG} ${NIGHTLY_FLAG} "--project_name" ${PROJECT_NAME} || die "build_pip_package FAILED"

PY_DOTLESS_MAJOR_MINOR_VER=$(echo $PY_MAJOR_MINOR_VER | tr -d '.')
if [[ $PY_DOTLESS_MAJOR_MINOR_VER == "2" ]]; then
  PY_DOTLESS_MAJOR_MINOR_VER="27"
fi

# Set wheel path and verify that there is only one .whl file in the path.
WHL_PATH=$(ls "${PIP_WHL_DIR}"/"${PROJECT_NAME}"-*"${PY_DOTLESS_MAJOR_MINOR_VER}"*"${PY_DOTLESS_MAJOR_MINOR_VER}"*.whl)
if [[ $(echo "${WHL_PATH}" | wc -w) -ne 1 ]]; then
  echo "ERROR: Failed to find exactly one built TensorFlow .whl file in "\
  "directory: ${PIP_WHL_DIR}"
fi

WHL_DIR=$(dirname "${WHL_PATH}")

# Print the size of the wheel file.
echo "Size of the PIP wheel file built: $(ls -l ${WHL_PATH} | awk '{print $5}')"

# Build the other GPU package.
if [ "$BUILD_BOTH_GPU_PACKAGES" -eq "1" ]; then
   echo "====================================="\
   "Building the other GPU pip package."
  # Check container type
  if ! [[ ${CONTAINER_TYPE} == "gpu" ]]; then
    die "Error: CONTAINER_TYPE needs to be `GPU` to build GPU packages. Got "\
        "\"${CONTAINER_TYPE}\" instead."
  fi
  if [[ "$PROJECT_NAME" == *_gpu ]]; then
    NEW_PROJECT_NAME=${PROJECT_NAME%"_gpu"}
  else
    NEW_PROJECT_NAME="${PROJECT_NAME}_gpu"
  fi
  echo "The given gpu \$PROJECT_NAME is ${PROJECT_NAME}. The additional GPU "\
  "pip package will have project name ${NEW_PROJECT_NAME}."

  ./bazel-bin/tensorflow/tools/pip_package/build_pip_package ${PIP_WHL_DIR} ${GPU_FLAG} ${NIGHTLY_FLAG} "--project_name" ${NEW_PROJECT_NAME} || die "build_pip_package FAILED"
fi

# Run tests (if any is specified).
run_all_tests

if [[ ${OS_TYPE} == "ubuntu" ]]; then
  # Avoid Python3.6 abnormality by installing auditwheel here.
  set +e
  pip3 show auditwheel || "pip${PY_MAJOR_MINOR_VER}" show auditwheel
  pip3 install auditwheel==2.0.0 || "pip${PY_MAJOR_MINOR_VER}" install auditwheel==2.0.0
  sudo pip3 install auditwheel==2.0.0 || \
    sudo "pip${PY_MAJOR_MINOR_VER}" install auditwheel==2.0.0
  set -e
  auditwheel --version

  for WHL_PATH in $(ls ${PIP_WHL_DIR}/*.whl); do
    # Repair the wheels for cpu manylinux2010
    echo "auditwheel repairing ${WHL_PATH}"
    auditwheel repair --plat manylinux2010_x86_64 -w "${WHL_DIR}" "${WHL_PATH}"

    WHL_BASE_NAME=$(basename "${WHL_PATH}")
    AUDITED_WHL_NAME="${WHL_DIR}"/$(echo "${WHL_BASE_NAME//linux/manylinux2010}")
    if [[ -f ${AUDITED_WHL_NAME} ]]; then
      WHL_PATH=${AUDITED_WHL_NAME}
      echo "Repaired manylinux2010 wheel file at: ${WHL_PATH}"
    else
      die "WARNING: Cannot find repaired wheel."
    fi
  done
fi
echo "EOF: Successfully ran pip_new.sh"

#!/usr/bin/env bash
# Copyright 2016 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
# Build the Python PIP installation package for TensorFlow and install
# the package.
#
# Usage:
#   pip_new.sh
#
# Required step(s):
#   Run configure.py prior to running this script.
#
# Required environment variable(s):
#   CONTAINER_TYPE:      (CPU | GPU)
#   OS_TYPE:             (UBUNTU | MACOS)
#   TF_PYTHON_VERSION:   (python2 | python2.7 | python3.5 | python3.7)
#   TF_BUILD_FLAGS:      Bazel build flags.
#                          e.g. TF_BUILD_FLAGS="--config=opt"
#   TF_TEST_FLAGS:       Bazel test flags.
#                          e.g. TF_TEST_FLAGS="--verbose_failures=true \
#                               --build_tests_only --test_output=errors"
#   TF_TEST_FILTER_TAGS: Filtering tags for bazel tests. More specifically,
#                        input tags for `--test_filter_tags` flag.
#                          e.g. TF_TEST_FILTER_TAGS="no_pip,-nomac,no_oss"
#   TF_TEST_TARGETS:     Bazel test targets.
#                          e.g. TF_TEST_TARGETS="//tensorflow/... \
#                               -//tensorflow/contrib/... \
#                               -//tensorflow/python/..."
#   IS_NIGHTLY:          Nightly run flag.
#                          e.g. IS_NIGHTLY=1  # nightly runs
#                          e.g. IS_NIGHTLY=0  # non-nightly runs
#
# Optional environment variables. If provided, overwrites any default values.
#   TF_PIP_TESTS:        PIP tests to run. If NOT specified, skips all tests.
#                          e.g. TF_PIP_TESTS="test_pip_virtualenv_clean \
#                               test_pip_virtualenv_clean \
#                               test_pip_virtualenv_oss_serial"
#   TF_PROJECT_NAME:     Name of the project. This string will be pass onto
#                        the wheel file name. For nightly builds, it will be
#                        overwritten to 'tf_nightly'. For gpu builds, '_gpu'
#                        will be appended.
#                          e.g. TF_PROJECT_NAME="tensorflow"
#                          e.g. TF_PROJECT_NAME="tf_nightly_gpu"
#   TF_PIP_TEST_ROOT:    Root directory for building and testing pip pkgs.
#                          e.g. TF_PIP_TEST_ROOT="pip_test"
#   TF_BUILD_BOTH_GPU_PACKAGES:    (1 | 0)
#                                  1 will build both tensorflow (w/ gpu support)
#                                  and tensorflow-gpu pip package. Will
#                                  automatically handle adding/removing of _gpu
#                                  suffix depending on what project name was
#                                  passed.
#
# To-be-deprecated variable(s).
#   GIT_TAG_OVERRIDE:    Values for `--git_tag_override`. This flag gets passed
#                        in as `--action_env` for bazel build and tests.
#   TF_BUILD_INSTALL_EXTRA_PIP_PACKAGES:
#                        Additonal pip packages to be installed.
#                        Caveat: pip version needs to be checked prior.
#
# ==============================================================================

# set bash options
set -e
set -x

###########################################################################
# General helper function(s)
###########################################################################

# Strip leading and trailing whitespaces
str_strip () {
  echo -e "$1" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'
}

# Convert string to all lower case
lowercase() {
  if [[ -z "${1}" ]]; then
    die "Nothing to convert to lowercase. No argument given."
  fi
  echo "${1}" | tr '[:upper:]' '[:lower:]'
}

check_global_vars() {
  # Check container type
  if ! [[ ${CONTAINER_TYPE} == "cpu" ]] && \
     ! [[ ${CONTAINER_TYPE} == "rocm" ]] && \
     ! [[ ${CONTAINER_TYPE} == "gpu" ]]; then
    die "Error: Provided CONTAINER_TYPE \"${CONTAINER_TYPE}\" "\
        "is not supported."
  fi
  # Check OS type
  if ! [[ ${OS_TYPE} == "ubuntu" ]] && \
     ! [[ ${OS_TYPE} == "macos" ]]; then
    die "Error: Provided OS_TYPE \"${OS_TYPE}\" is not supported."
  fi
  # Check build flags
  if [[ -z ${TF_BUILD_FLAGS} ]]; then
    die "Error: TF_BUILD_FLAGS is not specified."
  fi
  # Check test flags
  if [[ -z ${TF_TEST_FLAGS} ]]; then
    die "Error: TF_TEST_FLAGS is not specified."
  fi
  # Check test filter tags
  if [[ -z ${TF_TEST_FILTER_TAGS} ]]; then
    die "Error: TF_TEST_FILTER_TAGS is not specified."
  fi
  # Check test targets
  if [[ -z ${TF_TEST_TARGETS} ]]; then
    die "Error: TF_TEST_TARGETS is not specified."
  fi
  # Check nightly status
  if [[ -z ${IS_NIGHTLY} ]]; then
    die "Error: IS_NIGHTLY is not specified."
  fi
}

add_test_filter_tag() {
  EMPTY=""
  while true; do
    FILTER="${1:$EMPTY}"
    if ! [[ $TF_TEST_FILTER_TAGS == *"${FILTER}"* ]]; then
      TF_TEST_FILTER_TAGS="${FILTER},${TF_TEST_FILTER_TAGS}"
    fi
    shift
    if [[ -z "${1}" ]]; then
      break
    fi
  done
}

remove_test_filter_tag() {
  EMPTY=""
  while true; do
    FILTER="${1:$EMPTY}"
    TF_TEST_FILTER_TAGS="$(echo ${TF_TEST_FILTER_TAGS} | sed -e 's/^'${FILTER}',//g' -e 's/,'${FILTER}'//g')"
    shift
    if [[ -z "${1}" ]]; then
      break
    fi
  done
}

# Clean up bazel build & test flags with proper configuration.
update_bazel_flags() {
  # Add git tag override flag if necessary.
  GIT_TAG_STR=" --action_env=GIT_TAG_OVERRIDE"
  if [[ -z "${GIT_TAG_OVERRIDE}" ]] && \
    ! [[ ${TF_BUILD_FLAGS} = *${GIT_TAG_STR}* ]]; then
    TF_BUILD_FLAGS+="${GIT_TAG_STR}"
  fi
  # Clean up whitespaces
  TF_BUILD_FLAGS=$(str_strip "${TF_BUILD_FLAGS}")
  TF_TEST_FLAGS=$(str_strip "${TF_TEST_FLAGS}")
  # Cleaned bazel flags
  echo "Bazel build flags (cleaned):\n" "${TF_BUILD_FLAGS}"
  echo "Bazel test flags (cleaned):\n" "${TF_TEST_FLAGS}"
}

update_test_filter_tags() {
  # Add test filter tags
  # This script is for validating built PIP packages. Add pip tags.
  add_test_filter_tag -no_pip -nopip
  # MacOS filter tags
  if [[ ${OS_TYPE} == "macos" ]]; then
    remove_test_filter_tag nomac no_mac
    add_test_filter_tag -nomac -no_mac
  fi
  echo "Final test filter tags: ${TF_TEST_FILTER_TAGS}"
}

# Check currently running python and pip version
check_python_pip_version() {
  # Check if only the major version of python is provided by the user.
  MAJOR_VER_ONLY=0
  if [[ ${#PYTHON_VER} -lt 9 ]]; then
    # User only provided major version (e.g. 'python2' instead of 'python2.7')
    MAJOR_VER_ONLY=1
  fi

  # Retrieve only the version number of the user requested python.
  PYTHON_VER_REQUESTED=${PYTHON_VER:6:3}
  echo "PYTHON_VER_REQUESTED: ${PYTHON_VER_REQUESTED}"

  # Retrieve only the version numbers of the python & pip in use currently.
  PYTHON_VER_IN_USE=$(python --version 2>&1)
  PYTHON_VER_IN_USE=${PYTHON_VER_IN_USE:7:3}
  PIP_VER_IN_USE=$(${PIP_BIN_PATH} --version)
  PIP_VER_IN_USE=${PIP_VER_IN_USE:${#PIP_VER_IN_USE}-4:3}

  # If only major versions are applied, drop minor versions.
  if [[ $MAJOR_VER_ONLY == 1 ]]; then
    PYTHON_VER_IN_USE=${PYTHON_VER_IN_USE:0:1}
    PIP_VER_IN_USE=${PIP_VER_IN_USE:0:1}
  fi

  # Check if all versions match.
  echo -e "User requested python version: '${PYTHON_VER_REQUESTED}'\n" \
    "Detected python version in use: '${PYTHON_VER_IN_USE}'\n"\
    "Detected pip version in use: '${PIP_VER_IN_USE}'"
  if ! [[ $PYTHON_VER_REQUESTED == $PYTHON_VER_IN_USE ]]; then
    die "Error: Mismatch in python versions detected."
  else:
    echo "Python and PIP versions in use match the requested."
  fi
}

###########################################################################
# Setup: directories, local/global variables
###########################################################################

# Script directory and source necessary files.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "${SCRIPT_DIR}/builds_common.sh"

# Required global variables
# Checks on values for these vars are done in "Build TF PIP Package" section.
CONTAINER_TYPE=$(lowercase "${CONTAINER_TYPE}")
OS_TYPE=$(lowercase "${OS_TYPE}")
PYTHON_VER=$(lowercase "${TF_PYTHON_VERSION}")

# Python bin path
if [[ -z "$PYTHON_BIN_PATH" ]]; then
  die "Error: PYTHON_BIN_PATH was not provided. Did you run configure?"
fi

# Set optional environment variables; set to default in case not user defined.
DEFAULT_PIP_TESTS="" # Do not run any tests by default
DEFAULT_PROJECT_NAME="tensorflow"
DEFAULT_PIP_TEST_ROOT="pip_test"
DEFAULT_BUILD_BOTH_GPU_PACKAGES=0
# Take in optional global variables
PIP_TESTS=${TF_PIP_TESTS:-$DEFAULT_PIP_TESTS}
PROJECT_NAME=${TF_PROJECT_NAME:-$DEFAULT_PROJECT_NAME}
PIP_TEST_ROOT=${TF_PIP_TEST_ROOT:-$DEFAULT_PIP_TEST_ROOT}
BUILD_BOTH_GPU_PACKAGES=${TF_BUILD_BOTH_GPU_PACKAGES:-$DEFAULT_BUILD_BOTH_GPU_PACKAGES}

# Local variables
PIP_WHL_DIR="${KOKORO_ARTIFACTS_DIR}/tensorflow/${PIP_TEST_ROOT}/whl"
mkdir -p "${PIP_WHL_DIR}"
PIP_WHL_DIR=$(realpath "${PIP_WHL_DIR}") # Get absolute path
WHL_PATH=""
# Determine the major.minor versions of python being used (e.g., 2.7).
# Useful for determining the directory of the local pip installation.
PY_MAJOR_MINOR_VER=$(${PYTHON_BIN_PATH} -c "print(__import__('sys').version)" 2>&1 | awk '{ print $1 }' | head -n 1 | cut -c1-3)

if [[ -z "${PY_MAJOR_MINOR_VER}" ]]; then
  die "ERROR: Unable to determine the major.minor version of Python."
fi
echo "Python binary path to be used in PIP install: ${PYTHON_BIN_PATH} "\
"(Major.Minor version: ${PY_MAJOR_MINOR_VER})"
PYTHON_BIN_PATH_INIT=${PYTHON_BIN_PATH}
PIP_BIN_PATH="$(which pip${PY_MAJOR_MINOR_VER})"

# PIP packages
INSTALL_EXTRA_PIP_PACKAGES="portpicker scipy scikit-learn ${TF_BUILD_INSTALL_EXTRA_PIP_PACKAGES}"

###########################################################################
# Build TF PIP Package
###########################################################################

# First remove any already existing binaries for a clean start and test.
if [[ -d ${PIP_TEST_ROOT} ]]; then
  echo "Test root directory ${PIP_TEST_ROOT} already exists. Deleting it."
  sudo rm -rf ${PIP_TEST_ROOT}
fi

# Check that global variables are properly set.
check_global_vars

# Check if in a virtualenv and exit if yes.
IN_VENV=$(python -c 'import sys; print("1" if hasattr(sys, "real_prefix") else "0")')
if [[ "$IN_VENV" == "1" ]]; then
  echo "It appears that we are already in a virtualenv. Deactivating..."
  deactivate || source deactivate || die "FAILED: Unable to deactivate from existing virtualenv."
fi

# Configure python. Obtain the path to python binary.
source tools/python_bin_path.sh
# Assume PYTHON_BIN_PATH is exported by the script above.
if [[ -z "$PYTHON_BIN_PATH" ]]; then
  die "PYTHON_BIN_PATH was not provided. Did you run configure?"
fi

# Bazel build the file.
PIP_BUILD_TARGET="//tensorflow/tools/pip_package:build_pip_package"
# Clean bazel cache.
bazel clean
# Clean up and update bazel flags
update_bazel_flags
# Build. This outputs the file `build_pip_package`.
bazel build ${TF_BUILD_FLAGS} ${PIP_BUILD_TARGET} || \
  die "Error: Bazel build failed for target: '${PIP_BUILD_TARGET}'"

###########################################################################
# Test function(s)
###########################################################################

test_pip_virtualenv() {
  # Get args
  VENV_DIR_NAME=$1
  shift
  TEST_TYPE_FLAG=$1

  # Check test type args
  if ! [[ ${TEST_TYPE_FLAG} == "--oss_serial" ]] && \
     ! [[ ${TEST_TYPE_FLAG} == "--clean" ]] && \
     ! [[ ${TEST_TYPE_FLAG} == "" ]]; then
     die "Error: Wrong test type given. TEST_TYPE_FLAG=${TEST_TYPE_FLAG}"
  fi

  # Create virtualenv directory for test
  VENV_DIR="${PIP_TEST_ROOT}/${VENV_DIR_NAME}"

  # Activate virtualenv
  create_activate_virtualenv ${TEST_TYPE_FLAG} ${VENV_DIR}
  # Install TF with pip
  TIME_START=$SECONDS
  install_tensorflow_pip "${WHL_PATH}"
  TIME_ELAPSED=$(($SECONDS - $TIME_START))
  echo "Time elapsed installing tensorflow = ${TIME_ELAPSED} seconds"

  # cd to a temporary directory to avoid picking up Python files in the source
  # tree.
  TMP_DIR=$(mktemp -d)
  pushd "${TMP_DIR}"

  # Run a quick check on tensorflow installation.
  RET_VAL=$(python -c "import tensorflow as tf; t1=tf.constant([1,2,3,4]); t2=tf.constant([5,6,7,8]); print(tf.add(t1,t2).shape)")

  # Return to original directory. Remove temp dirs.
  popd
  sudo rm -rf "${TMP_DIR}"

  # Check result to see if tensorflow is properly installed.
  if ! [[ ${RET_VAL} == *'(4,)'* ]]; then
    echo "PIP test on virtualenv (non-clean) FAILED"
    return 1
  fi

  # Install extra pip packages, if specified.
  for PACKAGE in ${INSTALL_EXTRA_PIP_PACKAGES}; do
    echo "Installing extra pip package required by test-on-install: ${PACKAGE}"

    ${PIP_BIN_PATH} install ${PACKAGE}
    if [[ $? != 0 ]]; then
      echo "${PIP_BIN_PATH} install ${PACKAGE} FAILED."
      deactivate || source deactivate || die "FAILED: Unable to deactivate from existing virtualenv."
      return 1
    fi
  done

  # Run bazel test.
  run_test_with_bazel ${TEST_TYPE_FLAG}
  RESULT=$?

  # Deactivate from virtualenv.
  deactivate || source deactivate || die "FAILED: Unable to deactivate from existing virtualenv."
  sudo rm -rf "${VENV_DIR}"

  return $RESULT
}

###########################################################################
# Test helper function(s)
###########################################################################

create_activate_virtualenv() {
  VIRTUALENV_FLAGS="--system-site-packages"
  if [[ "${1}" == "--clean" ]]; then
    VIRTUALENV_FLAGS=""
    shift
  elif [[ "${1}" == "--oss_serial" ]]; then
    shift
  fi

  VIRTUALENV_DIR="${1}"
  if [[ -d "${VIRTUALENV_DIR}" ]]; then
    if sudo rm -rf "${VIRTUALENV_DIR}"
    then
      echo "Removed existing virtualenv directory: ${VIRTUALENV_DIR}"
    else
      die "Failed to remove existing virtualenv directory: ${VIRTUALENV_DIR}"
    fi
  fi

  if mkdir -p "${VIRTUALENV_DIR}"
  then
    echo "Created virtualenv directory: ${VIRTUALENV_DIR}"
  else
    die "FAILED to create virtualenv directory: ${VIRTUALENV_DIR}"
  fi

  # Use the virtualenv from the default python version (i.e., python-virtualenv)
  # to create the virtualenv directory for testing. Use the -p flag to specify
  # the python version inside the to-be-created virtualenv directory.
  ${PYTHON_BIN_PATH_INIT} -m virtualenv -p ${PYTHON_BIN_PATH_INIT} ${VIRTUALENV_FLAGS} ${VIRTUALENV_DIR} || \
    die "FAILED: Unable to create virtualenv"

  source "${VIRTUALENV_DIR}/bin/activate" || \
    die "FAILED: Unable to activate virtualenv in ${VIRTUALENV_DIR}"

  # Update .tf_configure.bazelrc with venv python path for bazel test.
  PYTHON_BIN_PATH="$(which python)"
  yes "" | ./configure
}

install_tensorflow_pip() {
  if [[ -z "${1}" ]]; then
    die "Please provide a proper wheel file path."
  fi

  # Set path to pip.
  PIP_BIN_PATH="$(which pip${PY_MAJOR_MINOR_VER})"

  # Print python and pip bin paths
  echo "PYTHON_BIN_PATH to be used to install the .whl: ${PYTHON_BIN_PATH}"
  echo "PIP_BIN_PATH to be used to install the .whl: ${PIP_BIN_PATH}"

  # Upgrade pip so it supports tags such as cp27mu, manylinux2010 etc.
  echo "Upgrade pip in virtualenv"

  # NOTE: pip install --upgrade pip leads to a documented TLS issue for
  # some versions in python
  curl https://bootstrap.pypa.io/get-pip.py | ${PYTHON_BIN_PATH} || \
    die "Error: pip install (get-pip.py) FAILED"

  # Check that requested python version matches configured one.
  check_python_pip_version

  # Force upgrade of setuptools. This must happen before the pip install of the
  # WHL_PATH, which pulls in absl-py, which uses install_requires notation
  # introduced in setuptools >=20.5. The default version of setuptools is 5.5.1,
  # which is too old for absl-py.
  ${PIP_BIN_PATH} install --upgrade setuptools==39.1.0 || \
    die "Error: setuptools install, upgrade FAILED"

  # Force tensorflow reinstallation. Otherwise it may not get installed from
  # last build if it had the same version number as previous build.
  PIP_FLAGS="--upgrade --force-reinstall"
  ${PIP_BIN_PATH} install ${PIP_FLAGS} ${WHL_PATH} || \
    die "pip install (forcing to reinstall tensorflow) FAILED"
  echo "Successfully installed pip package ${WHL_PATH}"

  # Force downgrade of setuptools. This must happen after the pip install of the
  # WHL_PATH, which ends up upgrading to the latest version of setuptools.
  # Versions of setuptools >= 39.1.0 will cause tests to fail like this:
  #   ImportError: cannot import name py31compat
  ${PIP_BIN_PATH} install --upgrade setuptools==39.1.0 || \
    die "Error: setuptools install, upgrade FAILED"

  # Install the future package in the virtualenv. Installing it in user system
  # packages does not appear to port it over when creating a virtualenv.
  #   ImportError: No module named builtins
  ${PIP_BIN_PATH} install --upgrade "future>=0.17.1" || \
    die "Error: future install, upgrade FAILED"

  # Install the gast package in the virtualenv. Installing it in user system
  # packages does not appear to port it over when creating a virtualenv.
  ${PIP_BIN_PATH} install --upgrade "gast==0.2.2" || \
    die "Error: gast install, upgrade FAILED"

}

run_test_with_bazel() {
  IS_OSS_SERIAL=0
  if [[ "${1}" == "--oss_serial" ]]; then
    IS_OSS_SERIAL=1
  fi
  TF_GPU_COUNT=${TF_GPU_COUNT:-4}

  # PIP tests should have a "different" path. Different than the one we place
  # virtualenv, because we are deleting and recreating it here.
  PIP_TEST_PREFIX=bazel_pip
  TEST_ROOT=$(pwd)/${PIP_TEST_PREFIX}
  sudo rm -rf $TEST_ROOT
  mkdir -p $TEST_ROOT
  ln -s $(pwd)/tensorflow $TEST_ROOT/tensorflow

  if [[ "${IS_OSS_SERIAL}" == "1" ]]; then
    remove_test_filter_tag -no_oss
    add_test_filter_tag oss_serial
  else
    add_test_filter_tag -oss_serial
  fi

  # Clean the bazel cache
  bazel clean
  # Clean up flags before running bazel commands
  update_bazel_flags
  # Clean up and update test filter tags
  update_test_filter_tags

  # Figure out how many concurrent tests we can run and do run the tests.
  BAZEL_PARALLEL_TEST_FLAGS=""
  if [[ $CONTAINER_TYPE == "gpu" ]]; then
    # Number of test threads is the number of GPU cards available.
    if [[ $OS_TYPE == "macos" ]]; then
      BAZEL_PARALLEL_TEST_FLAGS="--local_test_jobs=1"
    else
      BAZEL_PARALLEL_TEST_FLAGS="--local_test_jobs=${TF_GPU_COUNT} \
        --run_under=//tensorflow/tools/ci_build/gpu_build:parallel_gpu_execute"
    fi
  else
    # Number of test threads is the number of physical CPUs.
    if [[ $OS_TYPE == "macos" ]]; then
      BAZEL_PARALLEL_TEST_FLAGS="--local_test_jobs=$(sysctl -n hw.ncpu)"
    else
      BAZEL_PARALLEL_TEST_FLAGS="--local_test_jobs=$(grep -c ^processor /proc/cpuinfo)"
    fi
  fi

  if [[ ${IS_OSS_SERIAL} == 1 ]]; then
    BAZEL_PARALLEL_TEST_FLAGS="--local_test_jobs=1"
  fi

  TEST_TARGETS_SYMLINK=""
  for TARGET in ${TF_TEST_TARGETS[@]}; do
    TARGET_NEW=$(echo ${TARGET} | sed -e "s/\/\//\/\/${PIP_TEST_PREFIX}\//g")
    TEST_TARGETS_SYMLINK+="${TARGET_NEW} "
  done
  echo "Test targets (symlink): ${TEST_TARGETS_SYMLINK}"

  # Run the test.
  bazel test --build_tests_only ${TF_TEST_FLAGS} ${BAZEL_PARALLEL_TEST_FLAGS} --test_tag_filters=${TF_TEST_FILTER_TAGS} -k -- ${TEST_TARGETS_SYMLINK}

  unlink ${TEST_ROOT}/tensorflow
}

run_all_tests() {
  if [[ -z "${PIP_TESTS}" ]]; then
    echo "No test was specified to run. Skipping all tests."
    return 0
  fi
  FAIL_COUNTER=0
  PASS_COUNTER=0
  for TEST in ${PIP_TESTS[@]}; do

    # Run tests.
    case "${TEST}" in
    "test_pip_virtualenv_clean")
      test_pip_virtualenv venv_clean --clean
      ;;
    "test_pip_virtualenv_non_clean")
      test_pip_virtualenv venv
      ;;
    "test_pip_virtualenv_oss_serial")
      test_pip_virtualenv venv_oss --oss_serial
      ;;
    *)
      die "No matching test ${TEST} was found. Stopping test."
      ;;
    esac

    # Check and update the results.
    RETVAL=$?

    # Update results counter
    if [ ${RETVAL} -eq 0 ]; then
      echo "Test (${TEST}) PASSED. (PASS COUNTER: ${PASS_COUNTER})"
      PASS_COUNTER=$(($PASS_COUNTER+1))
    else
      echo "Test (${TEST}) FAILED. (FAIL COUNTER: ${FAIL_COUNTER})"
      FAIL_COUNTER=$(($FAIL_COUNTER+1))
    fi
  done
  printf "${PASS_COUNTER} PASSED | ${FAIL_COUNTER} FAILED"
  if [[ "${FAIL_COUNTER}" == "0" ]]; then
    printf "PIP tests ${COLOR_GREEN}PASSED${COLOR_NC}\n"
    return 0
  else:
    printf "PIP tests ${COLOR_RED}FAILED${COLOR_NC}\n"
    return 1
  fi
}

###########################################################################
# Build TF PIP Wheel file
###########################################################################

# Update the build flags for building whl.
# Flags: GPU, OS, tf_nightly, project name
GPU_FLAG=""
NIGHTLY_FLAG=""

# TF Nightly flag
if [[ "$IS_NIGHTLY" == 1 ]]; then
  # If 'nightly' is not specified in the project name already, then add.
  if ! [[ $PROJECT_NAME == *"nightly"* ]]; then
    echo "WARNING: IS_NIGHTLY=${IS_NIGHTLY} but requested project name \
    (PROJECT_NAME=${PROJECT_NAME}) does not include 'nightly' string. \
    Renaming it to 'tf_nightly'."
    PROJECT_NAME="tf_nightly"
  fi
  NIGHTLY_FLAG="--nightly_flag"
fi

# CPU / GPU flag
if [[ ${CONTAINER_TYPE} == "gpu" ]]; then
  GPU_FLAG="--gpu"
  if ! [[ $PROJECT_NAME == *"gpu"* ]]; then
    # Only update PROJECT_NAME if TF_PROJECT_NAME is not set
    if [[ -z "${TF_PROJECT_NAME}" ]]; then
      echo "WARNING: GPU is specified but requested project name (PROJECT_NAME=${PROJECT_NAME}) \
      does not include 'gpu'. Appending '_gpu' to the project name."
      PROJECT_NAME="${PROJECT_NAME}_gpu"
    fi
  fi
fi

./bazel-bin/tensorflow/tools/pip_package/build_pip_package ${PIP_WHL_DIR} ${GPU_FLAG} ${NIGHTLY_FLAG} "--project_name" ${PROJECT_NAME} || die "build_pip_package FAILED"

PY_DOTLESS_MAJOR_MINOR_VER=$(echo $PY_MAJOR_MINOR_VER | tr -d '.')
if [[ $PY_DOTLESS_MAJOR_MINOR_VER == "2" ]]; then
  PY_DOTLESS_MAJOR_MINOR_VER="27"
fi

# Set wheel path and verify that there is only one .whl file in the path.
WHL_PATH=$(ls "${PIP_WHL_DIR}"/"${PROJECT_NAME}"-*"${PY_DOTLESS_MAJOR_MINOR_VER}"*"${PY_DOTLESS_MAJOR_MINOR_VER}"*.whl)
if [[ $(echo "${WHL_PATH}" | wc -w) -ne 1 ]]; then
  echo "ERROR: Failed to find exactly one built TensorFlow .whl file in "\
  "directory: ${PIP_WHL_DIR}"
fi

WHL_DIR=$(dirname "${WHL_PATH}")

# Print the size of the wheel file.
echo "Size of the PIP wheel file built: $(ls -l ${WHL_PATH} | awk '{print $5}')"

# Build the other GPU package.
if [ "$BUILD_BOTH_GPU_PACKAGES" -eq "1" ]; then
   echo "====================================="\
   "Building the other GPU pip package."
  # Check container type
  if ! [[ ${CONTAINER_TYPE} == "gpu" ]]; then
    die "Error: CONTAINER_TYPE needs to be `GPU` to build GPU packages. Got "\
        "\"${CONTAINER_TYPE}\" instead."
  fi
  if [[ "$PROJECT_NAME" == *_gpu ]]; then
    NEW_PROJECT_NAME=${PROJECT_NAME%"_gpu"}
  else
    NEW_PROJECT_NAME="${PROJECT_NAME}_gpu"
  fi
  echo "The given gpu \$PROJECT_NAME is ${PROJECT_NAME}. The additional GPU "\
  "pip package will have project name ${NEW_PROJECT_NAME}."

  ./bazel-bin/tensorflow/tools/pip_package/build_pip_package ${PIP_WHL_DIR} ${GPU_FLAG} ${NIGHTLY_FLAG} "--project_name" ${NEW_PROJECT_NAME} || die "build_pip_package FAILED"
fi

# Run tests (if any is specified).
run_all_tests

for WHL_PATH in $(ls ${PIP_WHL_DIR}/*.whl); do
  if [[ ${OS_TYPE} == "ubuntu" ]]; then
    # Avoid Python3.6 abnormality by installing auditwheel here.
    set +e
    pip3 show auditwheel || "pip${PY_MAJOR_MINOR_VER}" show auditwheel
    pip3 install auditwheel==2.0.0 || "pip${PY_MAJOR_MINOR_VER}" install auditwheel==2.0.0
    sudo pip3 install auditwheel==2.0.0 || \
      sudo "pip${PY_MAJOR_MINOR_VER}" install auditwheel==2.0.0
    set -e
    auditwheel --version

    # Repair the wheels for cpu manylinux2010
    echo "auditwheel repairing ${WHL_PATH}"
    auditwheel repair --plat manylinux2010_x86_64 -w "${WHL_DIR}" "${WHL_PATH}"

    WHL_BASE_NAME=$(basename "${WHL_PATH}")
    AUDITED_WHL_NAME="${WHL_DIR}"/$(echo "${WHL_BASE_NAME//linux/manylinux2010}")
    if [[ -f ${AUDITED_WHL_NAME} ]]; then
      WHL_PATH=${AUDITED_WHL_NAME}
      echo "Repaired manylinux2010 wheel file at: ${WHL_PATH}"
    else
      die "WARNING: Cannot find repaired wheel."
    fi
  fi
done

echo "EOF: Successfully ran pip_new.sh"


script: update first20h for better saving

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/2vek/dotfiles.git/'


tests: integration: add skeleton to test resilient routing

#!/bin/bash

# Multi-namespace test for link-state-lfa (resilient routing).

cleanup() {
    local ret=0
    pkill rinaperf
    for cont in a b c d; do
        ip netns exec ${cont} rlite-ctl reset || ret=1
        ip netns exec ${cont} rlite-ctl terminate || ret=1
        ip netns delete ${cont} || ret=1
    done
    [ "$ret" != 0 ] && return 1 || return 0
}

abort() {
    cleanup
    exit 1
}

# Create four namespaces, connected on the same LAN through a software
# bridge and veth pairs.
#
#     A----B
#     | \  |
#     |  \ |
#     |   \|
#     C----D
#
for cont in a b c d; do
    ip netns add ${cont} || abort
    ip netns exec ${cont} ip link set lo up || abort
    ip netns exec ${cont} rlite-uipcps -d || abort
    ip netns exec ${cont} rlite-ctl ipcp-create ${cont}.n normal normdif || abort
done
for li in ab ac ad bd cd; do
    ip link add veth.${li}l type veth peer name veth.${li}r || abort
    ip link set veth.${li}l up || abort
    ip link set veth.${li}r up || abort
    left=${li:0:1}
    right=${li:1:1}
    ip link set veth.${li}l netns $left || abort
    ip link set veth.${li}r netns $right || abort
    ip netns exec $left ip link set veth.${li}l up || abort
    ip netns exec $right ip link set veth.${li}r up || abort
    # Normal over shim setup in all the namespaces
    ip netns exec $left rlite-ctl ipcp-create ${li}l.eth shim-eth ${li}dif || abort
    ip netns exec $right rlite-ctl ipcp-create ${li}r.eth shim-eth ${li}dif || abort
    ip netns exec ${left} rlite-ctl ipcp-config ${li}l.eth netdev veth.${li}l || abort
    ip netns exec ${right} rlite-ctl ipcp-config ${li}r.eth netdev veth.${li}r || abort
    ip netns exec ${left} rlite-ctl ipcp-register ${left}.n ${li}dif || abort
    ip netns exec ${right} rlite-ctl ipcp-register ${right}.n ${li}dif || abort
done

# Carry out the enrollments, with A being the enrollment master.
ip netns exec a rlite-ctl dif-policy-mod normdif routing link-state-lfa || abort
ip netns exec a rlite-ctl dif-policy-param-mod normdif addralloc nack-wait 500ms
ip netns exec a rlite-ctl ipcp-enroller-enable a.n || abort
for li in ab ac ad; do
    left=${li:0:1}
    right=${li:1:1}
    ip netns exec ${right} rlite-ctl ipcp-enroll ${right}.n normdif ${li}dif ${left}.n || abort
done

ip netns exec a rlite-ctl dif-routing-show normdif || abort

for li in bd cd; do
    left=${li:0:1}
    right=${li:1:1}
    ip netns exec ${right} rlite-ctl ipcp-lower-flow-alloc ${right}.n normdif ${li}dif ${left}.n || abort
done

ip netns exec a rlite-ctl dif-routing-show normdif || abort

cleanup

error: pathspec 'tests/integration/0028-resilient-routing.sh' did not match any file(s) known to git


Fix missing directory errors

Fix missing directory errors

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/chizou/geowave.git/'


es_reindex.sh: change default values

es_reindex.sh: change default values

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/TOTVS/mdmpublic.git/'


Create batch_reproject.sh

Create batch_reproject.sh

mkdir out
for i in *.shp
do
	# print info:
	echo "----------------------------"
	echo "$i"
	# gdalinfo "$i"
	echo "----------------------------"
	# copy filename without extension:
	f="${i%.*}"
	echo "$f"
	# assign new projection and put in output folder
	ogr2ogr out/"$f".shp -t_srs "EPSG:4269" "$i"
done

error: pathspec 'batch_reproject.sh' did not match any file(s) known to git


file to prepare the production environment

#!/bin/sh

rake assets:precompile
cp -r app/assets/stylesheets/redmond/images/ public/assets/

error: pathspec 'prepare.sh' did not match any file(s) known to git


Add rust initializer hyperfine

Add rust initializer hyperfine

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/ernestas-poskus/dot13.git/'


Added promote script

Added promote script

#!/bin/bash
case $# in
    0 )
        echo "Usage: promote.sh stagech/nm"
        exit;;
    * )
        :;;
esac
ENV=$1
git tag -m "Promoting to $ENV" release/${ENV}-$(date +%Y%m%d-%H%M)
git push gitcorp --tags

error: pathspec 'promote.sh' did not match any file(s) known to git


Create a script for the package publishing

#!/usr/bin/env bash
npm login
npm publish .

error: pathspec 'publish.sh' did not match any file(s) known to git


Add a simple publish script

Add a simple publish script

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/mikejarrett/pipcheck.git/'


publish script

publish script

#!/bin/bash

bundle exec jekyll build && s3_website push


add prow presubmit (#294)

add prow presubmit (#294)

#!/bin/bash

# Copyright 2017 Istio Authors

#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at

#       http://www.apache.org/licenses/LICENSE-2.0

#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

WD=$(dirname $0)
WD=$(cd $WD; pwd)
ROOT=$(dirname $WD)

#######################################
# Presubmit script triggered by Prow. #
#######################################

# Exit immediately for non zero status
set -e
# Check unset variables
set -u
# Print commands
set -x

if [ "${CI:-}" == 'bootstrap' ]; then
  # Test harness will checkout code to directory $GOPATH/src/github.com/istio/istio
  # but we depend on being at path $GOPATH/src/istio.io/istio for imports
  ln -sf ${GOPATH}/src/github.com/istio ${GOPATH}/src/istio.io
  ROOT=${GOPATH}/src/istio.io/mixerclient
  cd ${GOPATH}/src/istio.io/mixerclient

  # Use the provided pull head sha, from prow.
  GIT_SHA="${PULL_PULL_SHA}"
else
  # Use the current commit.
  GIT_SHA="$(git rev-parse --verify HEAD)"
fi

cd $ROOT

echo 'Code Check'
script/check-license-headers
script/check-style

echo 'Running Unit Tests'
time bazel test --test_output=all //...

echo 'Running Asan Tests'
time bazel test --config=asan --test_output=all //...

error: pathspec 'mixerclient/prow/mixerclient-presubmit.sh' did not match any file(s) known to git


[Fix #426] Correct syntax error in variable assignment

#
# Integrates history-substring-search into Prezto.
#
# Authors:
#   Suraj N. Kurapati <sunaku@gmail.com>
#   Sorin Ionescu <sorin.ionescu@gmail.com>
#

# Load dependencies.
pmodload 'editor'

# Source module files.
source "${0:h}/external/zsh-history-substring-search.zsh"

#
# Search
#

zstyle -s ':prezto:module:history-substring-search:color' found \
  'HISTORY_SUBSTRING_SEARCH_HIGHLIGHT_FOUND' \
    || HISTORY_SUBSTRING_SEARCH_HIGHLIGHT_FOUND='bg=magenta,fg=white,bold'

zstyle -s ':prezto:module:history-substring-search:color' not-found \
  'HISTORY_SUBSTRING_SEARCH_HIGHLIGHT_NOT_FOUND' \
    || HISTORY_SUBSTRING_SEARCH_HIGHLIGHT_NOT_FOUND='bg=red,fg=white,bold'

zstyle -s ':prezto:module:history-substring-search' globbing-flags \
  'HISTORY_SUBSTRING_SEARCH_GLOBBING_FLAGS' \
    || HISTORY_SUBSTRING_SEARCH_GLOBBING_FLAGS='i'

if zstyle -t ':prezto:module:history-substring-search' case-sensitive; then
  HISTORY_SUBSTRING_SEARCH_GLOBBING_FLAGS="${HISTORY_SUBSTRING_SEARCH_GLOBBING_FLAGS//i}"
fi

if ! zstyle -t ':prezto:module:history-substring-search' color; then
  unset HISTORY_SUBSTRING_SEARCH_HIGHLIGHT_{FOUND,NOT_FOUND}
fi

#
# Key Bindings
#

if [[ -n "$key_info" ]]; then
  # Emacs
  bindkey -M emacs "$key_info[Control]P" history-substring-search-up
  bindkey -M emacs "$key_info[Control]N" history-substring-search-down

  # Vi
  bindkey -M vicmd "k" history-substring-search-up
  bindkey -M vicmd "j" history-substring-search-down

  # Emacs and Vi
  for keymap in 'emacs' 'viins'; do
    bindkey -M "$keymap" "$key_info[Up]" history-substring-search-up
    bindkey -M "$keymap" "$key_info[Down]" history-substring-search-down
  done
fi

#
# Integrates history-substring-search into Prezto.
#
# Authors:
#   Suraj N. Kurapati <sunaku@gmail.com>
#   Sorin Ionescu <sorin.ionescu@gmail.com>
#

# Load dependencies.
pmodload 'editor'

# Source module files.
source "${0:h}/external/zsh-history-substring-search.zsh"

#
# Search
#

zstyle -s ':prezto:module:history-substring-search:color' found \
  'HISTORY_SUBSTRING_SEARCH_HIGHLIGHT_FOUND' \
    || HISTORY_SUBSTRING_SEARCH_HIGHLIGHT_FOUND='bg=magenta,fg=white,bold'

zstyle -s ':prezto:module:history-substring-search:color' not-found \
  'HISTORY_SUBSTRING_SEARCH_HIGHLIGHT_NOT_FOUND' \
    || HISTORY_SUBSTRING_SEARCH_HIGHLIGHT_NOT_FOUND='bg=red,fg=white,bold'

zstyle -s ':prezto:module:history-substring-search' globbing-flags \
  'HISTORY_SUBSTRING_SEARCH_GLOBBING_FLAGS' \
    || HISTORY_SUBSTRING_SEARCH_GLOBBING_FLAGS='i'

if zstyle -t ':prezto:module:history-substring-search' case-sensitive; then
  HISTORY_SUBSTRING_SEARCH_GLOBBING_FLAGS=\
    "${HISTORY_SUBSTRING_SEARCH_GLOBBING_FLAGS//i}"
fi

if ! zstyle -t ':prezto:module:history-substring-search' color; then
  unset HISTORY_SUBSTRING_SEARCH_HIGHLIGHT_{FOUND,NOT_FOUND}
fi

#
# Key Bindings
#

if [[ -n "$key_info" ]]; then
  # Emacs
  bindkey -M emacs "$key_info[Control]P" history-substring-search-up
  bindkey -M emacs "$key_info[Control]N" history-substring-search-down

  # Vi
  bindkey -M vicmd "k" history-substring-search-up
  bindkey -M vicmd "j" history-substring-search-down

  # Emacs and Vi
  for keymap in 'emacs' 'viins'; do
    bindkey -M "$keymap" "$key_info[Up]" history-substring-search-up
    bindkey -M "$keymap" "$key_info[Down]" history-substring-search-down
  done
fi


Pacstrap is no more installed by default

Install it with pacman

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/jpigree/manarch.git/'


database runner that accepts additional arguments e.g. -config

#!/bin/sh

# Usage note: If you use -config FILE you must fully specify the path to FILE otherwise the program won't find the file.
# e.g. ~/dev/ensj-healthcheck/run-healthcheck.sh  -config `pwd`/db.properties -d my_database SOME_TEST

home=`dirname $0`
cp=$home:$home/build/classes
for jar in $home/lib/*.jar; do
    cp=$jar:$cp
done

$JAVA_HOME/bin/java -Duser.dir=$home -cp $cp -Xmx1500m org.ensembl.healthcheck.DatabaseTestRunner $*

error: pathspec 'run-healthcheck-db.sh' did not match any file(s) known to git


wrong slashes

wrong slashes

git-svn-id: 8699849808240b02471d01f5aaa03ec0c1cb68ee@245368 13f79535-47bb-0310-9956-ffa450edef68

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/apache/axis1-java.git/'


Don't install npm deps in tests again

Don't install npm deps in tests again

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/cargomedia/socket-redis.git/'


Added environment variable setup

Added environment variable setup

#!/bin/bash

# Copyright (c) 2016, Codrin-Victor Poienaru <cvpoienaru@gmail.com>.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice, this
# list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright notice,
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# This software is provided by the copyright holders and contributors "as is"
# and any express or implied warranties, including, but not limited to, the
# implied warranties of merchantability and fitness for a particular purpose are
# disclaimed. In no event shall the copyright holder or contributors be liable
# for any direct, indirect, incidental, special, exemplary, or consequential
# damages (including, but not limited to, procurement of substitute goods or
# services; loss of use, data, or profits; or business interruption) however
# caused and on any theory of liability, whether in contract, strict liability,
# or tort (including negligence or otherwise) arising in any way out of the use
# of this software, even if advised of the possibility of such damage.

# Directory structure
export ES_ROOT=~/Work/Projects/EntropySource
export ES_BIN=$ES_ROOT/bin
export ES_ETC=$ES_ROOT/etc
export ES_INCLUDE=$ES_ROOT/include
export ES_LIB=$ES_ROOT/lib
export ES_SCRIPT=$ES_ROOT/script
export ES_SRC=$ES_ROOT/src

# Third party paths
export GLIB_INCLUDE=/usr/include/glib-2.0

# Environment paths
export C_INCLUDE_PATH=$ES_INCLUDE:$GLIB_INCLUDE
export LIBRARY_PATH=$ES_LIB
export LD_LIBRARY_PATH=$ES_LIB
export PATH=$PATH:$ES_BIN

# File system permission mask
export ES_MASK=0022

# Build options
export ES_DEBUG=true
export ES_BUILD_THREADS=16

unset ES_FLAGS
if $ES_DEBUG; then
	export ES_FLAGS=-g
fi

error: pathspec 'script/vars.sh' did not match any file(s) known to git


Adding simple release script

Adding simple release script

#!/usr/bin/env bash

source env/bin/activate
python -m unittest discover

python setup.py sdist upload -r pypi

error: pathspec 'release.sh' did not match any file(s) known to git


fix tag name

fix tag name

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/lithiumtech/mineraloil-waiters.git/'


release script

release script

#!/bin/sh

set -e

VERSION=$1
[ ! -n "$VERSION" ] && echo "Enter release version: " && read VERSION

# post-release
PR_VERSION=$2
[ ! -n "$PR_VERSION" ] && echo "Enter post-release version: " && read PR_VERSION

echo "Releasing $VERSION - are you sure? (y/n):" && read CONFIRM && [ "$CONFIRM" != "y" ] && exit 0

[ ! -n "$MVN" ] && MVN=mvn

$MVN versions:set -DnewVersion=$VERSION -DgenerateBackupPoms=false && \
git add -u . && git commit -m "$VERSION" && \
$MVN -Prelease deploy && git tag protostuff-$VERSION && \
$MVN versions:set -DnewVersion=$PR_VERSION -DgenerateBackupPoms=false && \
git add -u . && git commit -m "$PR_VERSION" && \
git push -u origin master && git push origin --tags


Release script that invokes a gpg-agent

Release script that invokes a gpg-agent

#!/bin/sh
echo "Starting new gpg-agent"
gpg --armor --detach-sign --use-agent --sign pom.xml
if [ $? -ne 0 ] ; then
	echo "ERROR: Unable to run gpg properly"
	exit 1
fi

gpg --verify pom.xml.asc
if [ $? -ne 0 ]; then
	rm pom.xml.asc
    echo "It appears that you fat-fingered your GPG passphrase"
	exit 1
fi
rm pom.xml.asc

mvn release:prepare --batch-mode


Make get-version.sh executable.

Make get-version.sh executable.

#!/bin/sh

# Needed to remove the new line when echoing the version
nl='
'

VERSION=

usage="\
Usage: $0 version_file
Print a version string.
"

version_file="$1"
if test -z "$version_file"; then
  echo "$usage"
  exit 1
fi
if test -f $version_file; then
  . ./$version_file
else
  echo "Version file $version_file not found"
  exit 1
fi

if test -z $PKG_VERSION; then
  echo "No version found in $version_file"
  exit 1
fi

# Omit the trailing newline
echo -n "$PKG_VERSION"


rawautomake.sh: pushversion added

rawautomake.sh: pushversion added

#!/bin/bash
#
# updates to the next version and adds the changes to the index

# get the current version from the branch name
branch_name="$(git symbolic-ref HEAD 2>/dev/null)" ||
branch_name="(unnamed branch)"     # detached HEAD
branch_name=${branch_name##refs/heads/}
version=${branch_name##release-}
longversion=`sed -e 's/^\([0-9]*\)\.\?\([0-9]*\)\?\.\?\([0-9]*\)\?\([a-z]*[0-9]*\)\?$/\1.\2.\3\4/' -e 's/^\./0./' -e 's/\.\([^0-9]\)/.0\1/g' -e 's/\.\([^0-9]\)/.0\1/g' -e 's/\.$/.0/' <<< $version`
libversion=`sed -e 's/[^0-9.].*//' -e 's/\./:/g' <<< $longversion`

echo "updating to version $version ($longversion | $libversion)"

if ! grep '[0-9]\+:[0-9]\+:[0-9]\+' &>/dev/null <<< $libversion; then
    echo "error: libversion has wrong format" >&2
    exit 1
fi

# update the version information header entry
sed -i -e "/const char version\[\]/ s/\"[^\"]\+\"/\"$version\"/" include/config.hpp

# update autoconf information
sed -i -e "/AC_INIT(/ s/\[[^]]*\]/[$version]/2" configure.ac

# update libversion
sed -i -e "s/\(-version-info\)\s\+[0-9:]*/\1 $libversion/" Makefile.am

# confirm the changes
required="include/config.hpp configure.ac Makefile.am"
for file in $required; do
    if ! git ls-files -m 2>/dev/null | grep $file &>/dev/null; then
        echo "warning: file was not modified: $file" >&2
    fi
done

changed="$required"

git add $changed

git rm --cached $0

echo "done"

rm $0

error: pathspec 'rawautomake/pushversion.sh' did not match any file(s) known to git


Playing around with breaking apart an array

#!/usr/bin/env bash

screen -list
arr=($(screen -list | sed '1d;$d' | sed -e '$d' -e 's/\t//') )
for X in "${arr[@]}"; do
      if [[ "$X" = "(Detached)" ]]; then
	   echo "Screen $Previous is $X"
      elif [[ "$X" = "(Attached)" ]]; then
	   echo "Screen $Previous is $X"
      fi
      Previous=$X
done

error: pathspec 'screen-array-thing.sh' did not match any file(s) known to git


added scp-cluster.sh to copy files across cluster

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/austinbyers/parameter_server.git/'


remove byword shortcuts

remove byword shortcuts

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/vitorgalvao/dotfiles.git/'


ElasticBeanstalk deployment script

ElasticBeanstalk deployment script

#!/bin/sh
# Deploy to AWS ElasticBeanstalk
#
# You can either add those here, or configure them on the environment tab of your
# project settings. The version must be unique for each deployment and should
# probably include at least the commit hash
AWS_ACCESS_KEY_ID=""
AWS_SECRET_ACCESS_KEY=""
AWS_DEFAULT_REGION=""
AWS_S3_BUCKET=""
AWS_APPLICATION_NAME=""
AWS_APPLICATION_ENVIRONMENT=""
AWS_APPLICATION_VERSION=""

set -e

# install the AWS CLI, which isn't installed by default
pip install awscli

# for rails projects only, reset the database configuration, which
# gets adapted for use on Codeship automatically
git checkout -- config/database.yml

# create the archive and upload it to S3
zip -x \*/.git\* -x .git\* -x \*.hg\* -r "${HOME}/aws_deployment_${AWS_APPLICATION_VERSION}.zip"
aws s3 cp "${HOME}/aws_deployment.zip" "s3://${AWS_S3_BUCKET}/subfolder/aws_deployment_${AWS_APPLICATION_VERSION}.zip"

# create the new version on ElasticBeanstalk
aws elasticbeanstalk create-application-version --application-name "${AWS_APPLICATION_NAME}" --version-label "${AWS_APPLICATION_VERSION}" --source-bundle S3Bucket="${AWS_S3_BUCKET}",S3Key="subfolder/aws_deployment_${AWS_APPLICATION_VERSION}.zip"
aws elasticbeanstalk update-environment --environment-name "${AWS_APPLICATION_ENVIRONMENT}" --version-label "${AWS_APPLICATION_VERSION}"

# check if the deployment was successful
eb_deployment_check "${AWS_APPLICATION_NAME}" "${AWS_APPLICATION_ENVIRONMENT}"

error: pathspec 'deployments/elastic_beanstalk.sh' did not match any file(s) known to git


first shell script added

first shell script added

#!/bin/bash

clear

echo "The script starts now!"

echo "Hi, $USER"

echo

echo "These are other folks on this system:"
who

echo "Exiting"

error: pathspec 'script1.sh' did not match any file(s) known to git


added install_bx.sh

added install_bx.sh

#!/bin/bash

echo "Download Bluemix CLI"
wget --quiet --output-document=/tmp/Bluemix_CLI_amd64.tar.gz  http://public.dhe.ibm.com/cloud/bluemix/cli/bluemix-cli/latest/Bluemix_CLI_amd64.tar.gz
tar -xf /tmp/Bluemix_CLI_amd64.tar.gz --directory=/tmp

# Create bx alias
echo "#!/bin/sh" >/tmp/Bluemix_CLI/bin/bx
echo "/tmp/Bluemix_CLI/bin/bluemix \"\$@\" " >>/tmp/Bluemix_CLI/bin/bx
chmod +x /tmp/Bluemix_CLI/bin/*

export PATH="/tmp/Bluemix_CLI/bin:$PATH"

# Install Armada CS plugin
echo "Install the Bluemix container-service plugin"
bx plugin install container-service -r Bluemix

echo "Install kubectl"
wget --quiet --output-document=/tmp/Bluemix_CLI/bin/kubectl  https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x /tmp/Bluemix_CLI/bin/kubectl

if [ -n "$DEBUG" ]; then
  bx --version
  bx plugin list
fi

error: pathspec 'scripts/install_bx.sh' did not match any file(s) known to git


Test for * docker backend * filesystem mounts on ZFS * Pool config and state

#!/usr/bin/env bash

set -eu

echo -n "Checking for docker backend... "
driver=$(docker info | grep '^Storage Driver: ' | sed -r 's/^Storage Driver: (.*)/\1/')
if [[ "${driver}" != "zfs" ]]; then
    echo "ERROR wrong backend ${driver}"
    exit 23
else
    echo "OKAY."
fi

for filesystem in /home /data /var/lib/docker; do
    echo -n "Checking if ${filesystem} is on ZFS... "
    if ! (mount | grep -P "on ${filesystem}.*type zfs" > /dev/null); then
        echo "ERROR: ${filesystem} is fucked"
        exit 23
    else
        echo "OKAY."
    fi
done

echo -n "Checking number of zpools... "
NUM_ZPOOLS=$(sudo zpool list -H | wc -l)
if [[ "${NUM_ZPOOLS}" -eq "1" ]]; then
    echo "OKAY"
else
    echo "ERROR: one expected but got ${NUM_ZPOOLS}"
    exit 23
fi

echo -n "Checking zpool health... "
ZPOOL_HEALTH=$(sudo zpool status -x protonet_storage)
if [[ "${ZPOOL_HEALTH}" == "pool 'protonet_storage' is healthy" ]]; then
    echo "OKAY"
else
    echo "ERROR: ${ZPOOL_HEALTH}"
    exit 23
fi

error: pathspec 'scripts/test-disks.sh' did not match any file(s) known to git


re #49: vahecommit

re #49: vahecommit

git-svn-id: 5c313c7b604ce6bf72d7224b35e7462ee01c5e2b@91 7a690eda-67ec-4556-b061-4c86d960b817

#!/bin/bash

# This function returns the Astro-Julian Date.  The Julian
# date (JD) is a continuous count of days from 1 January 4713 BC.
# The following algorithm is good from years 1801 to 2099.
# See URL:
# http://aa.usno.navy.mil/faq/docs/JD_Formula.html
# for more information
# arguments: $1 = year in format YYYY, $2 = month, $3 = day
get_astro_JD()
{
typeset -i JDD

JDD=$(($3-32075+1461*($1+4800+($2-14)/12)/4+367*($2-2-($2-14)/12*12)/12-3*(($1+4900+($2-14)/12)/100)/4))
echo $JDD
}

# This function computes the gregorian date from the julian date - $1.
# Returns a date string in the form: YEAR MONTH DAY
# See URL:
# http://aa.usno.navy.mil/faq/docs/JD_Formula.html
# for more information
get_greg_from_JD()
{
typeset -i L
typeset -i N
typeset -i I
typeset -i J
typeset -i DAY
typeset -i MON
typeset -i YR

L=$(($1+68569)) # $1 is the julian date
N=$((4*L/146097))
L=$((L-(146097*N+3)/4))
I=$((4000*(L+1)/1461001))
L=$((L-1461*I/4+31))
J=$((80*L/2447))
DAY=$((L-2447*J/80))
L=$((J/11))
MON=$((J+2-12*L))
YR=$((100*(N-49)+I+L))

printf '%02i %02i %04i\n' $YR $MON $DAY
}

# pass needle as $1 and array should follow
function in_array()
{
	local i
	needle=$1
	shift 1
	[ -z "$1" ] && return 1	# array() undefined
	for i in $*
	do
		[ "$i" == "$needle" ] && return 0
	done
	return 1
}

# pass needle as $1 and array should follow
function is_lt()
{
	local i
	needle=$1
	shift 1
	for i in $*
	do
		[ "$needle" -le "$i" ] && return 1
	done
	return 0
}

# pass needle as $1 and array should follow
function get_lt()
{
	local i
	needle=$1
	shift 1
	for i in $*
	do
		[[ "$needle" -le "$i" ]] && ret_val=$needle
	done
	echo $ret_val
}

# pass needle as $1 and array should follow
function is_ge()
{
	[ "$1" -ge "$2" ] && return 1
	return 0
}

# pass needle as $1 and array should follow
function get_ge()
{
	local i
	needle=$1
	shift 1
	for i in $*
	do
		[[ "$needle" -ge "$i" ]] && ret_val=$needle
	done
	echo $ret_val
}

function max()
{
   echo ${1[${1[@]}]}
}

function last_event()
{
   I=0
#   [[ -z $LAST_EVENT ]] && LAST_EVENT=( $NOW_JD $NOW_TIME_H $NOW_TIME_M )
   [[ -z $LAST_EVENT ]] && LAST_EVENT=( 0 0 0 )

   MAX_TIME_H=${NOW_TIME_H}
   MAX_TIME_M=${NOW_TIME_M}

   while [ "$I" -le "366" ]
   do
      I_JD=$((NOW_JD-I))
      [[ "$I_JD" -lt "${cronline[8]}" ]] && return 0
      [[ "$I_JD" -lt "${LAST_EVENT[0]}" ]] && return 0
      I=$((I+1))

      I_DATE=`get_greg_from_JD $I_JD`
      I_DAY=`echo $I_DATE | cut -d' ' -f3`
      I_MONTH=`echo $I_DATE | cut -d' ' -f2`
      I_WEEKDAY=$((I_JD-I_JD/7*7+1));

      if [ ! ${cronline[3]} == "*" ]; then
         in_array ${I_DAY} ${cronline[3]}
         [[ $? -eq 0 ]] && continue
      fi
      if [ ! ${cronline[4]} == "*" ]; then
         in_array ${I_MONTH} ${cronline[4]}
         [[ $? -eq 0 ]] && continue
      fi
      if [ ! "${cronline[5]}" == "*" ]; then
         in_array ${I_WEEKDAY} ${cronline[5]}
         [[ $? -eq 0 ]] && continue
      fi

      echo "= Candidate JDay: ${I_JD}"
      #  At this point we have found a candidate day for latest event
      if [ "${I_JD}" -eq "${NOW_JD}" ];then
         if [ ${cronline[2]} = "*" ]; then
            echo "= EVENT_H=${NOW_TIME_H}"
            EVENT_H=${NOW_TIME_H}
         else
            is_ge ${NOW_TIME_H} ${cronline[2]}
            [[ $? -eq 0 ]] && continue
            EVENT_H=`get_ge ${NOW_TIME_H} ${cronline[2]}`
         fi
      else
         if [ ${cronline[2]} = "*" ]; then
            EVENT_H=23
         else
            EVENT_H=max ${cronline[2]}
         fi
      fi

      if [ "${I_JD}" -eq "${NOW_JD}" -a "${EVENT_H}" -eq "${NOW_TIME_H}" ]; then
         if [ "${cronline[1]}" = "*" ]; then
            EVENT_M=${NOW_TIME_M}
         else
            is_ge ${NOW_TIME_M} ${cronline[1]}
            [[ $? -eq 0 ]] && continue
            EVENT_M=`get_ge ${NOW_TIME_M} ${cronline[1]}`
         fi
      else
         if [ ${cronline[1]} = "*" ]; then
            EVENT_M=59
         else
            EVENT_M=max ${cronline[1]}
         fi
      fi

      LAST_EVENT=( ${I_JD} ${EVENT_H} ${EVENT_M} )

      break
      #echo $I_DATE

   done
}

# ZERO_JD=2454850 # Sunday, January 18, 2009
SCHEDULE_FILE=$1
NOW_DATE=`date +'%Y %m %d'`  # 2009 01 18
NOW_DATE_M=`date +'%m'`  # 01
NOW_DATE_D=`date +'%d'`  # 18
NOW_DATE_W=`date +'%u'`  # 1
NOW_TIME=`date +'%H %M %S'`  # 23 11 07
NOW_TIME_H=`date +'%H'`  # 23
NOW_TIME_M=`date +'%M'`  # 11
NOW_TIME_S=`date +'%S'`  # 07
NOW_JD=`get_astro_JD $NOW_DATE`
NOW_WD=$((NOW_JD-NOW_JD/7*7+1)) # 1 - MON, 2 - TUE, ..., 7 - SUN

set -f  # disable globbing. wildcards would be expanded otherwise
OIFS=$IFS; IFS=';'

firstline=1

cat ${SCHEDULE_FILE} | \
while read l
do
   if [ $firstline -eq 1 ]
   then
      firstline=0
      echo \|$l\|
      continue
   fi
   echo ===
   echo $l
   echo ===

   cronline=( $l ) # split $l by semicolon

   cronline[1]=`echo ${cronline[1]} | tr ',' ' '`
   cronline[2]=`echo ${cronline[2]} | tr ',' ' '`
   cronline[3]=`echo ${cronline[3]} | tr ',' ' '`
   cronline[4]=`echo ${cronline[4]} | tr ',' ' '`
   cronline[5]=`echo ${cronline[5]} | tr ',' ' '`
   SIFS=$IFS; IFS='-'
      cronline[8]=`get_astro_JD ${cronline[6]}`
      cronline[9]=`get_astro_JD ${cronline[7]}`
   IFS=$SIFS;

   PIFS=$IFS; IFS=" ,"
      last_event
   IFS=$PIFS

   echo '0 id           :'${cronline[0]}  # id
   echo '1 minutes      :'${cronline[1]}  # minutes
   echo '2 hours        :'${cronline[2]}  # hours
   echo '3 days         :'${cronline[3]}  # days
   echo '4 months       :'${cronline[4]}  # months
   echo '5 weekdays     :'${cronline[5]}  # weekdays
   echo '6 valid from   :'${cronline[6]}  # valid from
   echo '7 valid to     :'${cronline[7]}  # valid to
   echo '8 JD valid from:'${cronline[8]}  # JD valid from
   echo '9 JD valid to  :'${cronline[9]}  # JD valid to

   echo '=='${LAST_EVENT[*]}'=='

done

IFS=$OIFS

error: pathspec 'exe/screen/schedule.player.sh' did not match any file(s) known to git


SYNSETS contains the full path and does not need pwd prepended.

Tested on GCE with ubuntu 16.04. Mirrors change 157536266.

#!/bin/bash
# Copyright 2016 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================

# Script to download ImageNet Challenge 2012 training and validation data set.
#
# Downloads and decompresses raw images and bounding boxes.
#
# **IMPORTANT**
# To download the raw images, the user must create an account with image-net.org
# and generate a username and access_key. The latter two are required for
# downloading the raw images.
#
# usage:
#  ./download_imagenet.sh [dir name] [synsets file]
set -e

if [ "x$IMAGENET_ACCESS_KEY" == x -o "x$IMAGENET_USERNAME" == x ]; then
  cat <<END
In order to download the imagenet data, you have to create an account with
image-net.org. This will get you a username and an access key. You can set the
IMAGENET_USERNAME and IMAGENET_ACCESS_KEY environment variables, or you can
enter the credentials here.
END
  read -p "Username: " IMAGENET_USERNAME
  read -p "Access key: " IMAGENET_ACCESS_KEY
fi

OUTDIR="${1:-./imagenet-data}"
SYNSETS_FILE="${2:-./synsets.txt}"

echo "Saving downloaded files to $OUTDIR"
mkdir -p "${OUTDIR}"
CURRENT_DIR=$(pwd)
BBOX_DIR="${OUTDIR}bounding_boxes"
mkdir -p "${BBOX_DIR}"
cd "${OUTDIR}"

# Download and process all of the ImageNet bounding boxes.
BASE_URL="http://www.image-net.org/challenges/LSVRC/2012/nonpub"

# See here for details: http://www.image-net.org/download-bboxes
BOUNDING_BOX_ANNOTATIONS="${BASE_URL}/ILSVRC2012_bbox_train_v2.tar.gz"
BBOX_TAR_BALL="${BBOX_DIR}/annotations.tar.gz"
echo "Downloading bounding box annotations."
wget "${BOUNDING_BOX_ANNOTATIONS}" -O "${BBOX_TAR_BALL}" || BASE_URL_CHANGE=1
if [ $BASE_URL_CHANGE ]; then
  BASE_URL="http://www.image-net.org/challenges/LSVRC/2012/nnoupb"
  BOUNDING_BOX_ANNOTATIONS="${BASE_URL}/ILSVRC2012_bbox_train_v2.tar.gz"
  BBOX_TAR_BALL="${BBOX_DIR}/annotations.tar.gz"
fi
wget "${BOUNDING_BOX_ANNOTATIONS}" -O "${BBOX_TAR_BALL}"
echo "Uncompressing bounding box annotations ..."
tar xzf "${BBOX_TAR_BALL}" -C "${BBOX_DIR}"

LABELS_ANNOTATED="${BBOX_DIR}/*"
NUM_XML=$(ls -1 ${LABELS_ANNOTATED} | wc -l)
echo "Identified ${NUM_XML} bounding box annotations."

# Download and uncompress all images from the ImageNet 2012 validation dataset.
VALIDATION_TARBALL="ILSVRC2012_img_val.tar"
OUTPUT_PATH="${OUTDIR}validation/"
mkdir -p "${OUTPUT_PATH}"
cd "${OUTDIR}/.."
echo "Downloading ${VALIDATION_TARBALL} to ${OUTPUT_PATH}."
wget -nd -c "${BASE_URL}/${VALIDATION_TARBALL}"
tar xf "${VALIDATION_TARBALL}" -C "${OUTPUT_PATH}"

# Download all images from the ImageNet 2012 train dataset.
TRAIN_TARBALL="ILSVRC2012_img_train.tar"
OUTPUT_PATH="${OUTDIR}train/"
mkdir -p "${OUTPUT_PATH}"
cd "${OUTDIR}/.."
echo "Downloading ${TRAIN_TARBALL} to ${OUTPUT_PATH}."
wget -nd -c "${BASE_URL}/${TRAIN_TARBALL}"

# Un-compress the individual tar-files within the train tar-file.
echo "Uncompressing individual train tar-balls in the training data."

while read SYNSET; do
  echo "Processing: ${SYNSET}"

  # Create a directory and delete anything there.
  mkdir -p "${OUTPUT_PATH}/${SYNSET}"
  rm -rf "${OUTPUT_PATH}/${SYNSET}/*"

  # Uncompress into the directory.
  tar xf "${TRAIN_TARBALL}" "${SYNSET}.tar"
  tar xf "${SYNSET}.tar" -C "${OUTPUT_PATH}/${SYNSET}/"
  rm -f "${SYNSET}.tar"

  echo "Finished processing: ${SYNSET}"
done < "${SYNSETS_FILE}"

#!/bin/bash
# Copyright 2016 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================

# Script to download ImageNet Challenge 2012 training and validation data set.
#
# Downloads and decompresses raw images and bounding boxes.
#
# **IMPORTANT**
# To download the raw images, the user must create an account with image-net.org
# and generate a username and access_key. The latter two are required for
# downloading the raw images.
#
# usage:
#  ./download_imagenet.sh [dir name] [synsets file]
set -e

if [ "x$IMAGENET_ACCESS_KEY" == x -o "x$IMAGENET_USERNAME" == x ]; then
  cat <<END
In order to download the imagenet data, you have to create an account with
image-net.org. This will get you a username and an access key. You can set the
IMAGENET_USERNAME and IMAGENET_ACCESS_KEY environment variables, or you can
enter the credentials here.
END
  read -p "Username: " IMAGENET_USERNAME
  read -p "Access key: " IMAGENET_ACCESS_KEY
fi

OUTDIR="${1:-./imagenet-data}"
SYNSETS_FILE="${2:-./synsets.txt}"
SYNSETS_FILE="${PWD}/${SYNSETS_FILE}"

echo "Saving downloaded files to $OUTDIR"
mkdir -p "${OUTDIR}"
CURRENT_DIR=$(pwd)
BBOX_DIR="${OUTDIR}bounding_boxes"
mkdir -p "${BBOX_DIR}"
cd "${OUTDIR}"

# Download and process all of the ImageNet bounding boxes.
BASE_URL="http://www.image-net.org/challenges/LSVRC/2012/nonpub"

# See here for details: http://www.image-net.org/download-bboxes
BOUNDING_BOX_ANNOTATIONS="${BASE_URL}/ILSVRC2012_bbox_train_v2.tar.gz"
BBOX_TAR_BALL="${BBOX_DIR}/annotations.tar.gz"
echo "Downloading bounding box annotations."
wget "${BOUNDING_BOX_ANNOTATIONS}" -O "${BBOX_TAR_BALL}" || BASE_URL_CHANGE=1
if [ $BASE_URL_CHANGE ]; then
  BASE_URL="http://www.image-net.org/challenges/LSVRC/2012/nnoupb"
  BOUNDING_BOX_ANNOTATIONS="${BASE_URL}/ILSVRC2012_bbox_train_v2.tar.gz"
  BBOX_TAR_BALL="${BBOX_DIR}/annotations.tar.gz"
fi
wget "${BOUNDING_BOX_ANNOTATIONS}" -O "${BBOX_TAR_BALL}"
echo "Uncompressing bounding box annotations ..."
tar xzf "${BBOX_TAR_BALL}" -C "${BBOX_DIR}"

LABELS_ANNOTATED="${BBOX_DIR}/*"
NUM_XML=$(ls -1 ${LABELS_ANNOTATED} | wc -l)
echo "Identified ${NUM_XML} bounding box annotations."

# Download and uncompress all images from the ImageNet 2012 validation dataset.
VALIDATION_TARBALL="ILSVRC2012_img_val.tar"
OUTPUT_PATH="${OUTDIR}validation/"
mkdir -p "${OUTPUT_PATH}"
cd "${OUTDIR}/.."
echo "Downloading ${VALIDATION_TARBALL} to ${OUTPUT_PATH}."
wget -nd -c "${BASE_URL}/${VALIDATION_TARBALL}"
tar xf "${VALIDATION_TARBALL}" -C "${OUTPUT_PATH}"

# Download all images from the ImageNet 2012 train dataset.
TRAIN_TARBALL="ILSVRC2012_img_train.tar"
OUTPUT_PATH="${OUTDIR}train/"
mkdir -p "${OUTPUT_PATH}"
cd "${OUTDIR}/.."
echo "Downloading ${TRAIN_TARBALL} to ${OUTPUT_PATH}."
wget -nd -c "${BASE_URL}/${TRAIN_TARBALL}"

# Un-compress the individual tar-files within the train tar-file.
echo "Uncompressing individual train tar-balls in the training data."

while read SYNSET; do
  echo "Processing: ${SYNSET}"

  # Create a directory and delete anything there.
  mkdir -p "${OUTPUT_PATH}/${SYNSET}"
  rm -rf "${OUTPUT_PATH}/${SYNSET}/*"

  # Uncompress into the directory.
  tar xf "${TRAIN_TARBALL}" "${SYNSET}.tar"
  tar xf "${SYNSET}.tar" -C "${OUTPUT_PATH}/${SYNSET}/"
  rm -f "${SYNSET}.tar"

  echo "Finished processing: ${SYNSET}"
done < "${SYNSETS_FILE}"


add replace_pubsub_service_account_secrets() to install-clf.sh

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/simonsdave/cloudfeaster-services.git/'