问:
How to install trax, jax, jaxlib on M1 Mac on macOS 12?

trax

New to trax, I'm trying to run it locally (macOS 12.1, Apple Silicon ARM M1 processor, 8GB RAM, Anaconda), but I'm running into some issues.

In an environment with python 3.8.5, I installed trax running pip3 install trax==1.3.9 inside an (Anaconda) conda environment. Later, I ran into issues when trying to import trax layers in my code with from trax import layers as tl

```<code>RuntimeError: This version of jaxlib was built using AVX instructions, which your CPU and/or operating system do not support. You may be able work around this issue by building jaxlib from source.
</code>```

I rushed to start a new conda environment with python 3.10. Then ran into issue just trying to install trax:

```<code>macos ERROR: Could not find a version that satisfies the requirement tensorflow-text (from trax) (from versions: none)
ERROR: No matching distribution found for tensorflow-text
</code>```

I then created a new environment with python 3.9. Installation went fine, but then ran into the same error importing layers:

```<code>RuntimeError: This version of jaxlib was built using AVX instructions, which your CPU and/or operating system do not support. You may be able work around this issue by building jaxlib from source.
</code>```

installing jaxlib, jax before trax

I then tried building jaxlib from source following these instructions and got this error:

```<code>Building XLA and installing it in the jaxlib source tree...
./bazel-4.2.1-darwin-x86_64 run --verbose_failures=true --config=avx_posix --config=mkl_open_source_only :build_wheel -- --output_path=/my path/jax/dist --cpu=x86_64
ERROR: bazel does not currently work properly from paths containing spaces (/my path/jax).
b''
Traceback (most recent call last):
  File "/my path/jax/build/build.py", line 524, in <module>
    main()
  File "/my path/jax/build/build.py", line 519, in main
    shell(command)
  File "/my path/jax/build/build.py", line 53, in shell
    output = subprocess.check_output(cmd)
  File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./bazel-4.2.1-darwin-x86_64', 'run', '--verbose_failures=true', '--config=avx_posix', '--config=mkl_open_source_only', ':build_wheel', '--', '--output_path=/my path/jax/dist', '--cpu=x86_64']' returned non-zero exit status 36.
</code>```

Emphasis on the part that I initially missed that says: bazel does not currently work properly from paths containing spaces (/my path/jax).

I moved my /my path/ directory to a path without spaces /mypath/. Deleted and redownloaded jax directory. Still, the build (for CPU) with python build/build.py failed:

```<code>ERROR: /private/var/tmp/_bazel_a/2caf512c3c5e3f3f654bc58b48b8333a/external/llvm-project/llvm/BUILD.bazel:610:11: Generating code from table: include/llvm/IR/Intrinsics.td @llvm-project//llvm:intrinsic_XCore_gen__gen_intrinsic_enums__intrinsic_prefix_xcore_genrule failed: (Illegal instruction): bash failed: error executing command
  (cd /private/var/tmp/_bazel_a/2caf512c3c5e3f3f654bc58b48b8333a/execroot/__main__ && \
  exec env - \
    PATH=/myuserpath/opt/anaconda3/envs/mytraxenv/bin:/myuserpath/opt/anaconda3/condabin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/darwin-opt/bin/external/llvm-project/llvm/llvm-tblgen -I external/llvm-project/llvm/include -I external/llvm-project/clang/include -I $(dirname external/llvm-project/llvm/include/llvm/IR/Intrinsics.td) -gen-intrinsic-enums -intrinsic-prefix=xcore external/llvm-project/llvm/include/llvm/IR/Intrinsics.td  -o bazel-out/darwin-opt/bin/external/llvm-project/llvm/include/llvm/IR/IntrinsicsXCore.h')
Execution platform: @local_execution_config_platform//:platform
/bin/bash: line 1: 11140 Illegal instruction: 4  bazel-out/darwin-opt/bin/external/llvm-project/llvm/llvm-tblgen -I external/llvm-project/llvm/include -I external/llvm-project/clang/include -I $(dirname external/llvm-project/llvm/include/llvm/IR/Intrinsics.td) -gen-intrinsic-enums -intrinsic-prefix=xcore external/llvm-project/llvm/include/llvm/IR/Intrinsics.td -o bazel-out/darwin-opt/bin/external/llvm-project/llvm/include/llvm/IR/IntrinsicsXCore.h
Target //build:build_wheel failed to build
INFO: Elapsed time: 620.950s, Critical Path: 45.35s
INFO: 589 processes: 132 internal, 457 local.
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully
b''
Traceback (most recent call last):
  File "/mypath/jax/build/build.py", line 524, in <module>
    main()
  File "/mypath/jax/build/build.py", line 519, in main
    shell(command)
  File "/mypath/jax/build/build.py", line 53, in shell
    output = subprocess.check_output(cmd)
  File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./bazel-4.2.1-darwin-x86_64', 'run', '--verbose_failures=true', '--config=avx_posix', '--config=mkl_open_source_only', ':build_wheel', '--', '--output_path=/mypath/jax/dist', '--cpu=x86_64']' returned non-zero exit status 1.
</code>```

Tried again a couple of times (deleted and redownloaded jax directory) and the same line 528 in .../lib/python3.9/subprocess.py seemed to cause the issue but the output, while mostly the above, sometimes was slightly different. Making me suspect an issue with memory, given that I (admittedly) had not restarted my machine in weeks and it was starting to unresponsively slow.

I updated my XCode command line tools to version 12.2 (pretty sure).

I restarted my (8GB) machine. Deleted and redownloaded jax directory. I installed bazel version 5.0.0 with homebrew in case that would help. I was a bit concerned that it kept downloading an x86 version for my ARM processor. bazel installation went fine.

Started again from these instructions. The jaxlib build, though, made it clear that it wanted an earlier (4.2.1) version of bazel and downloaded it, as before:

```<code>b'\x1b[31mERROR: The project you\'re trying to build requires Bazel 4.2.1 (specified in /mypath/jax/.bazelversion), but it wasn\'t found in /opt/homebrew/Cellar/bazel/5.0.0/libexec/bin.\x1b[0m\n\nBazel binaries for all official releases can be downloaded from here:\n  https://github.com/bazelbuild/bazel/releases\n\nYou can download the required version directly using this command:\n  (cd "/opt/homebrew/Cellar/bazel/5.0.0/libexec/bin" && curl -fLO https://releases.bazel.build/4.2.1/release/bazel-4.2.1-darwin-x86_64 && chmod +x bazel-4.2.1-darwin-x86_64)\n'
</code>```

Once again, a different error in the same line 528. Showing full run now:

```<code>b'\x1b[31mERROR: The project you\'re trying to build requires Bazel 4.2.1 (specified in /mypath/jax/.bazelversion), but it wasn\'t found in /opt/homebrew/Cellar/bazel/5.0.0/libexec/bin.\x1b[0m\n\nBazel binaries for all official releases can be downloaded from here:\n  https://github.com/bazelbuild/bazel/releases\n\nYou can download the required version directly using this command:\n  (cd "/opt/homebrew/Cellar/bazel/5.0.0/libexec/bin" && curl -fLO https://releases.bazel.build/4.2.1/release/bazel-4.2.1-darwin-x86_64 && chmod +x bazel-4.2.1-darwin-x86_64)\n'
Downloading bazel from: https://github.com/bazelbuild/bazel/releases/download/4.2.1/bazel-4.2.1-darwin-x86_64
bazel-4.2.1-darwin-x86_64 [########################################] 100%
Bazel binary path: ./bazel-4.2.1-darwin-x86_64
Bazel version: 4.2.1
Python binary path: /myuserpath/opt/anaconda3/envs/mytraxenv/bin/python
Python version: 3.9
NumPy version: 1.21.2
MKL-DNN enabled: yes
Target CPU: x86_64
Target CPU features: release
CUDA enabled: no
TPU enabled: no
ROCm enabled: no

Building XLA and installing it in the jaxlib source tree...
./bazel-4.2.1-darwin-x86_64 run --verbose_failures=true --config=avx_posix --config=mkl_open_source_only :build_wheel -- --output_path=/mypath/jax/dist --cpu=x86_64
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'run' from /mypath/jax/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'run' from /mypath/jax/.bazelrc:
  Inherited 'build' options: --apple_platform_type=macos --macos_minimum_os=10.9 --announce_rc --define open_source_build=true --spawn_strategy=standalone --enable_platform_specific_config --define=no_aws_support=true --define=no_gcp_support=true --define=no_hdfs_support=true --define=no_kafka_support=true --define=no_ignite_support=true --define=grpc_no_ares=true -c opt --config=short_logs --copt=-DMLIR_PYTHON_PACKAGE_PREFIX=jaxlib.mlir.
INFO: Reading rc options for 'run' from /mypath/jax/.jax_configure.bazelrc:
  Inherited 'build' options: --strategy=Genrule=standalone --repo_env PYTHON_BIN_PATH=/myuserpath/opt/anaconda3/envs/mytraxenv/bin/python --action_env=PYENV_ROOT --python_path=/myuserpath/opt/anaconda3/envs/mytraxenv/bin/python --distinct_host_configuration=false
INFO: Found applicable config definition build:short_logs in file /mypath/jax/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:avx_posix in file /mypath/jax/.bazelrc: --copt=-mavx --host_copt=-mavx
INFO: Found applicable config definition build:mkl_open_source_only in file /mypath/jax/.bazelrc: --define=tensorflow_mkldnn_contraction_kernel=1
INFO: Found applicable config definition build:macos in file /mypath/jax/.bazelrc: --config=posix
INFO: Found applicable config definition build:posix in file /mypath/jax/.bazelrc: --copt=-fvisibility=hidden --copt=-Wno-sign-compare --cxxopt=-std=c++14 --host_cxxopt=-std=c++14
Loading:
Loading: 0 packages loaded
Analyzing: target //build:build_wheel (1 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (8 packages loaded, 286 targets configured)
Analyzing: target //build:build_wheel (10 packages loaded, 4271 targets configured)
Analyzing: target //build:build_wheel (10 packages loaded, 4271 targets configured)
Analyzing: target //build:build_wheel (11 packages loaded, 4494 targets configured)
INFO: Analyzed target //build:build_wheel (11 packages loaded, 7417 targets configured).

INFO: Found 1 target...
[0 / 3] [Prepa] BazelWorkspaceStatusAction stable-status.txt
[21 / 233] Compiling src/google/protobuf/generated_enum_util.cc; 2s local ... (6 actions, 3 running)
[28 / 233] Compiling src/google/protobuf/extension_set.cc; 4s local ... (8 actions, 7 running)
[114 / 523] Compiling src/google/protobuf/generated_message_util.cc; 2s local ... (8 actions running)
[140 / 594] Compiling platform/c++11/src/nsync_semaphore_mutex.cc; 0s local ... (8 actions, 7 running)
[237 / 597] Compiling src/google/protobuf/util/message_differencer.cc; 0s local ... (8 actions running)
[252 / 597] Compiling src/google/protobuf/util/message_differencer.cc; 4s local ... (8 actions running)
[272 / 597] Compiling src/google/protobuf/descriptor.pb.cc; 8s local ... (8 actions running)
[296 / 597] Compiling src/google/protobuf/descriptor.cc; 12s local ... (8 actions running)
[315 / 597] Compiling src/google/protobuf/descriptor_database.cc; 2s local ... (5 actions running)
[487 / 1,963] Compiling src/compiler/python_generator.cc; 2s local ... (8 actions running)
[579 / 3,117] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv3d.cc; 0s local ... (8 actions, 7 running)
[619 / 3,173] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv3d.cc; 8s local ... (8 actions running)
[687 / 3,270] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv3d.cc; 17s local ... (8 actions running)
[774 / 3,464] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv2d.cc; 27s local ... (8 actions running)
[1,204 / 4,860] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv2d.cc; 39s local ... (8 actions running)
[1,255 / 4,916] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 52s local ... (8 actions running)
[1,340 / 5,042] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 68s local ... (8 actions running)
[1,456 / 5,156] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 86s local ... (8 actions, 7 running)
[1,661 / 5,704] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 107s local ... (8 actions, 7 running)
[1,688 / 5,704] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 132s local ... (8 actions, 7 running)
[1,721 / 5,704] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 160s local ... (8 actions, 7 running)
[2,106 / 6,584] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 192s local ... (8 actions, 7 running)
[2,342 / 7,067] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 231s local ... (8 actions, 7 running)
[2,378 / 7,067] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 274s local ... (8 actions, 7 running)
[2,422 / 7,067] Compiling src/cpu/rnn/ref_rnn.cpp; 75s local ... (8 actions, 7 running)
[2,452 / 7,067] Compiling src/cpu/x64/gemm/f32/jit_avx512_core_f32_copy_at_kern_part1_autogen.cpp; 54s local ... (8 actions, 7 running)
[2,500 / 7,067] Compiling src/cpu/x64/gemm/f32/jit_avx512_core_f32_copy_at_kern_part1_autogen.cpp; 119s local ... (8 actions, 7 running)
[2,577 / 7,067] Compiling src/common/memory_zero_pad.cpp; 74s local ... (8 actions, 7 running)
ERROR: /private/var/tmp/_bazel_a/f5e9a3325f07a1f02c52d821857db47c/external/org_tensorflow/tensorflow/compiler/xla/BUILD:69:17: ProtoCompile external/org_tensorflow/tensorflow/compiler/xla/xla.pb.h failed: (Illegal instruction): protoc failed: error executing command
  (cd /private/var/tmp/_bazel_a/f5e9a3325f07a1f02c52d821857db47c/execroot/__main__ && \
  exec env - \
    PATH=/myuserpath/opt/anaconda3/envs/mytraxenv/bin:/myuserpath/opt/anaconda3/condabin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin \
  bazel-out/darwin-opt/bin/external/com_google_protobuf/protoc '--cpp_out=bazel-out/darwin-opt/bin/external/org_tensorflow' -Iexternal/org_tensorflow -Ibazel-out/darwin-opt/bin/external/org_tensorflow -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/any_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/source_context_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/type_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/api_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/descriptor_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/compiler_plugin_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/duration_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/empty_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/field_mask_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/struct_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/wrappers_proto external/org_tensorflow/tensorflow/compiler/xla/xla.proto)
Execution platform: @local_execution_config_platform//:platform
Target //build:build_wheel failed to build
INFO: Elapsed time: 631.189s, Critical Path: 283.35s
INFO: 2563 processes: 935 internal, 1628 local.
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully
b''
Traceback (most recent call last):
  File "/mypath/jax/build/build.py", line 524, in <module>
    main()
  File "/mypath/jax/build/build.py", line 519, in main
    shell(command)
  File "/mypath/jax/build/build.py", line 53, in shell
    output = subprocess.check_output(cmd)
  File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./bazel-4.2.1-darwin-x86_64', 'run', '--verbose_failures=true', '--config=avx_posix', '--config=mkl_open_source_only', ':build_wheel', '--', '--output_path=/mypath/jax/dist', '--cpu=x86_64']' returned non-zero exit status 1.
</code>```

I switched away from trying to build jaxlib. In late October 2021, a M1-compatible jaxlib wheel was released, so I tried:

```<code>pip install -U pip
pip install -U https://storage.googleapis.com/jax-releases/mac/jaxlib-0.1.74-cp39-none-macosx_11_0_arm64.whl
</code>```

but got

```<code>ERROR: jaxlib-0.1.74-cp39-none-macosx_11_0_arm64.whl is not a supported wheel on this platform.
</code>```

Tried upgrading python from 3.9 to 3.10, but got the same message.

Debugged by, in python, running

```<code>import platform
print(platform.machine())
</code>```

which showed that my python is still running on x86 architechture, since "Anaconda doesn't yet provide packages for M1/ARM".

[see successful (so far) jaxlib installation in own answer below]

back to trax

After successfully installing jaxlib and jax, when trying to install trax with (miniforge’s)conda install trax I get:

```<code>macos ERROR: Could not find a version that satisfies the requirement tensorflow-text (from trax) (from versions: none)
ERROR: No matching distribution found for tensorflow-text
</code>```0

I tried installing as trax itself suggests but ran into an error building hp5y:

```<code>macos ERROR: Could not find a version that satisfies the requirement tensorflow-text (from trax) (from versions: none)
ERROR: No matching distribution found for tensorflow-text
</code>```1

Looking now into this error and also looking into doing it with miniforge again but using Anaconda's channel to install trax. I'll update when I get to it.

Any pointers or anyone who managed to install trax on an M1 Mac it by any different means?

答:
Anaconda Source build

As of Mon, 7th of Feb 2022, installing using an Anaconda installation of python fails as Anaconda does not provide arm wheels for python.

When they do provide them, building from source should work automatically. If it does not, clone the repository, and use the following:

python build/build.py --target_cpu darwin_arm64
```
<p>If bazel does not build and you are using homebrew consider installing it, and removing the downloaded bazel files.</p>
<pre class="lang-sh prettyprint-override"><code>brew install bazel
rm -rf bazel-*
</code>```

Arm64 Python wheels

I recommend downloading arm64 wheels from the official CPython website.
The executable will be in /usr/local/bin/python3, and so will pip3.

This should allow you to install jax using

```<code>/usr/local/bin/pip3 install --upgrade 'jax[cpu]'
</code>```

To have access to the proper python with the command line/path, add the folder to the path.

export PATH=/usr/share/bin:$PATH
```
<h2>Virtual Env</h2>
<p>I recommend against the above method, and instead to use to use <code>virtual-env</code> from here https://virtualenv.pypa.io, to create virtual environments as with anaconda, and install jax in a virtual environment as above.</p>
<h2>Poetry</h2>
<p>Poetry is an even better option since it uses virtual-env, and allows you to build your projects and list dependencies just as anaconda.</p>
<p>There is a minor caveat as I am writing this, installing <code>scipy</code> when defining dependencies with poetry results in an odd error, installing <code>numpy</code> and <code>scipy</code> manually using</p>
```poetry shell # to go to virtual environment for the project
pip install --upgrade 'jax[cpu]' jaxlib scipy numpy
```
<h1>Trax</h1>
<p>Since anaconda is not option, installing hdf5 is a bit of a hassle, but it can be done with <code>brew</code>/<code>homebrew</code>. Assuming you have activated the correct venv:</p>
```brew install hdf5
export HDF5_DIR=/opt/homebrew/Cellar/hdf5/1.12.1
pip install trax
```
<p>Unfortunately this results in the following error:</p>
```import trax
/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/jax/_src/lib/__init__.py:32: UserWarning: JAX on Mac ARM machines is experimental and minimally tested. Please see https://github.com/google/jax/issues/5501 in the event of problems.
  warnings.warn("JAX on Mac ARM machines is experimental and minimally tested. "

***************************************************************
Failed to import TensorFlow. Please note that TensorFlow is not installed by default when you install TFDS. This allow you to choose to install either `tf-nightly` or `tensorflow`. Please install the most recent version of TensorFlow, by following instructions at https://tensorflow.org/install.
***************************************************************

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/trax/__init__.py", line 18, in <module>
    from trax import layers
  File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/trax/layers/__init__.py", line 23, in <module>
    from trax.layers.activation_fns import *
  File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/trax/layers/activation_fns.py", line 26, in <module>
    from trax import math
  File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/trax/math/__init__.py", line 18, in <module>
    from trax.math import jax as jax_math
  File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/trax/math/jax.py", line 28, in <module>
    import tensorflow_datasets as tfds
  File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/tensorflow_datasets/__init__.py", line 43, in <module>
    from tensorflow_datasets.core import tf_compat
  File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/tensorflow_datasets/core/__init__.py", line 22, in <module>
    tf_compat.ensure_tf_install()
  File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/tensorflow_datasets/core/tf_compat.py", line 48, in ensure_tf_install
    import tensorflow as tf  # pylint: disable=import-outside-toplevel
ModuleNotFoundError: No module named 'tensorflow'
```

Body:
trax

New to trax, I'm trying to run it locally (macOS 12.1, Apple Silicon ARM M1 processor, 8GB RAM, Anaconda), but I'm running into some issues.

In an environment with python 3.8.5, I installed trax running pip3 install trax==1.3.9 inside an (Anaconda) conda environment. Later, I ran into issues when trying to import trax layers in my code with from trax import layers as tl

RuntimeError: This version of jaxlib was built using AVX instructions, which your CPU and/or operating system do not support. You may be able work around this issue by building jaxlib from source.

I rushed to start a new conda environment with python 3.10. Then ran into issue just trying to install trax:

macos ERROR: Could not find a version that satisfies the requirement tensorflow-text (from trax) (from versions: none)
ERROR: No matching distribution found for tensorflow-text

I then created a new environment with python 3.9. Installation went fine, but then ran into the same error importing layers:

RuntimeError: This version of jaxlib was built using AVX instructions, which your CPU and/or operating system do not support. You may be able work around this issue by building jaxlib from source.

installing jaxlib, jax before trax

I then tried building jaxlib from source following these instructions and got this error:

Building XLA and installing it in the jaxlib source tree...
./bazel-4.2.1-darwin-x86_64 run --verbose_failures=true --config=avx_posix --config=mkl_open_source_only :build_wheel -- --output_path=/my path/jax/dist --cpu=x86_64
ERROR: bazel does not currently work properly from paths containing spaces (/my path/jax).
b''
Traceback (most recent call last):
  File "/my path/jax/build/build.py", line 524, in <module>
    main()
  File "/my path/jax/build/build.py", line 519, in main
    shell(command)
  File "/my path/jax/build/build.py", line 53, in shell
    output = subprocess.check_output(cmd)
  File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./bazel-4.2.1-darwin-x86_64', 'run', '--verbose_failures=true', '--config=avx_posix', '--config=mkl_open_source_only', ':build_wheel', '--', '--output_path=/my path/jax/dist', '--cpu=x86_64']' returned non-zero exit status 36.

Emphasis on the part that I initially missed that says: bazel does not currently work properly from paths containing spaces (/my path/jax).

I moved my /my path/ directory to a path without spaces /mypath/. Deleted and redownloaded jax directory. Still, the build (for CPU) with python build/build.py failed:

ERROR: /private/var/tmp/_bazel_a/2caf512c3c5e3f3f654bc58b48b8333a/external/llvm-project/llvm/BUILD.bazel:610:11: Generating code from table: include/llvm/IR/Intrinsics.td @llvm-project//llvm:intrinsic_XCore_gen__gen_intrinsic_enums__intrinsic_prefix_xcore_genrule failed: (Illegal instruction): bash failed: error executing command
  (cd /private/var/tmp/_bazel_a/2caf512c3c5e3f3f654bc58b48b8333a/execroot/__main__ && \
  exec env - \
    PATH=/myuserpath/opt/anaconda3/envs/mytraxenv/bin:/myuserpath/opt/anaconda3/condabin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/darwin-opt/bin/external/llvm-project/llvm/llvm-tblgen -I external/llvm-project/llvm/include -I external/llvm-project/clang/include -I $(dirname external/llvm-project/llvm/include/llvm/IR/Intrinsics.td) -gen-intrinsic-enums -intrinsic-prefix=xcore external/llvm-project/llvm/include/llvm/IR/Intrinsics.td  -o bazel-out/darwin-opt/bin/external/llvm-project/llvm/include/llvm/IR/IntrinsicsXCore.h')
Execution platform: @local_execution_config_platform//:platform
/bin/bash: line 1: 11140 Illegal instruction: 4  bazel-out/darwin-opt/bin/external/llvm-project/llvm/llvm-tblgen -I external/llvm-project/llvm/include -I external/llvm-project/clang/include -I $(dirname external/llvm-project/llvm/include/llvm/IR/Intrinsics.td) -gen-intrinsic-enums -intrinsic-prefix=xcore external/llvm-project/llvm/include/llvm/IR/Intrinsics.td -o bazel-out/darwin-opt/bin/external/llvm-project/llvm/include/llvm/IR/IntrinsicsXCore.h
Target //build:build_wheel failed to build
INFO: Elapsed time: 620.950s, Critical Path: 45.35s
INFO: 589 processes: 132 internal, 457 local.
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully
b''
Traceback (most recent call last):
  File "/mypath/jax/build/build.py", line 524, in <module>
    main()
  File "/mypath/jax/build/build.py", line 519, in main
    shell(command)
  File "/mypath/jax/build/build.py", line 53, in shell
    output = subprocess.check_output(cmd)
  File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./bazel-4.2.1-darwin-x86_64', 'run', '--verbose_failures=true', '--config=avx_posix', '--config=mkl_open_source_only', ':build_wheel', '--', '--output_path=/mypath/jax/dist', '--cpu=x86_64']' returned non-zero exit status 1.

Tried again a couple of times (deleted and redownloaded jax directory) and the same line 528 in .../lib/python3.9/subprocess.py seemed to cause the issue but the output, while mostly the above, sometimes was slightly different. Making me suspect an issue with memory, given that I (admittedly) had not restarted my machine in weeks and it was starting to unresponsively slow.

I updated my XCode command line tools to version 12.2 (pretty sure).

I restarted my (8GB) machine. Deleted and redownloaded jax directory. I installed bazel version 5.0.0 with homebrew in case that would help. I was a bit concerned that it kept downloading an x86 version for my ARM processor. bazel installation went fine.

Started again from these instructions. The jaxlib build, though, made it clear that it wanted an earlier (4.2.1) version of bazel and downloaded it, as before:

b'\x1b[31mERROR: The project you\'re trying to build requires Bazel 4.2.1 (specified in /mypath/jax/.bazelversion), but it wasn\'t found in /opt/homebrew/Cellar/bazel/5.0.0/libexec/bin.\x1b[0m\n\nBazel binaries for all official releases can be downloaded from here:\n  https://github.com/bazelbuild/bazel/releases\n\nYou can download the required version directly using this command:\n  (cd "/opt/homebrew/Cellar/bazel/5.0.0/libexec/bin" && curl -fLO https://releases.bazel.build/4.2.1/release/bazel-4.2.1-darwin-x86_64 && chmod +x bazel-4.2.1-darwin-x86_64)\n'

Once again, a different error in the same line 528. Showing full run now:

b'\x1b[31mERROR: The project you\'re trying to build requires Bazel 4.2.1 (specified in /mypath/jax/.bazelversion), but it wasn\'t found in /opt/homebrew/Cellar/bazel/5.0.0/libexec/bin.\x1b[0m\n\nBazel binaries for all official releases can be downloaded from here:\n  https://github.com/bazelbuild/bazel/releases\n\nYou can download the required version directly using this command:\n  (cd "/opt/homebrew/Cellar/bazel/5.0.0/libexec/bin" && curl -fLO https://releases.bazel.build/4.2.1/release/bazel-4.2.1-darwin-x86_64 && chmod +x bazel-4.2.1-darwin-x86_64)\n'
Downloading bazel from: https://github.com/bazelbuild/bazel/releases/download/4.2.1/bazel-4.2.1-darwin-x86_64
bazel-4.2.1-darwin-x86_64 [########################################] 100%
Bazel binary path: ./bazel-4.2.1-darwin-x86_64
Bazel version: 4.2.1
Python binary path: /myuserpath/opt/anaconda3/envs/mytraxenv/bin/python
Python version: 3.9
NumPy version: 1.21.2
MKL-DNN enabled: yes
Target CPU: x86_64
Target CPU features: release
CUDA enabled: no
TPU enabled: no
ROCm enabled: no

Building XLA and installing it in the jaxlib source tree...
./bazel-4.2.1-darwin-x86_64 run --verbose_failures=true --config=avx_posix --config=mkl_open_source_only :build_wheel -- --output_path=/mypath/jax/dist --cpu=x86_64
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'run' from /mypath/jax/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'run' from /mypath/jax/.bazelrc:
  Inherited 'build' options: --apple_platform_type=macos --macos_minimum_os=10.9 --announce_rc --define open_source_build=true --spawn_strategy=standalone --enable_platform_specific_config --define=no_aws_support=true --define=no_gcp_support=true --define=no_hdfs_support=true --define=no_kafka_support=true --define=no_ignite_support=true --define=grpc_no_ares=true -c opt --config=short_logs --copt=-DMLIR_PYTHON_PACKAGE_PREFIX=jaxlib.mlir.
INFO: Reading rc options for 'run' from /mypath/jax/.jax_configure.bazelrc:
  Inherited 'build' options: --strategy=Genrule=standalone --repo_env PYTHON_BIN_PATH=/myuserpath/opt/anaconda3/envs/mytraxenv/bin/python --action_env=PYENV_ROOT --python_path=/myuserpath/opt/anaconda3/envs/mytraxenv/bin/python --distinct_host_configuration=false
INFO: Found applicable config definition build:short_logs in file /mypath/jax/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:avx_posix in file /mypath/jax/.bazelrc: --copt=-mavx --host_copt=-mavx
INFO: Found applicable config definition build:mkl_open_source_only in file /mypath/jax/.bazelrc: --define=tensorflow_mkldnn_contraction_kernel=1
INFO: Found applicable config definition build:macos in file /mypath/jax/.bazelrc: --config=posix
INFO: Found applicable config definition build:posix in file /mypath/jax/.bazelrc: --copt=-fvisibility=hidden --copt=-Wno-sign-compare --cxxopt=-std=c++14 --host_cxxopt=-std=c++14
Loading:
Loading: 0 packages loaded
Analyzing: target //build:build_wheel (1 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (8 packages loaded, 286 targets configured)
Analyzing: target //build:build_wheel (10 packages loaded, 4271 targets configured)
Analyzing: target //build:build_wheel (10 packages loaded, 4271 targets configured)
Analyzing: target //build:build_wheel (11 packages loaded, 4494 targets configured)
INFO: Analyzed target //build:build_wheel (11 packages loaded, 7417 targets configured).

INFO: Found 1 target...
[0 / 3] [Prepa] BazelWorkspaceStatusAction stable-status.txt
[21 / 233] Compiling src/google/protobuf/generated_enum_util.cc; 2s local ... (6 actions, 3 running)
[28 / 233] Compiling src/google/protobuf/extension_set.cc; 4s local ... (8 actions, 7 running)
[114 / 523] Compiling src/google/protobuf/generated_message_util.cc; 2s local ... (8 actions running)
[140 / 594] Compiling platform/c++11/src/nsync_semaphore_mutex.cc; 0s local ... (8 actions, 7 running)
[237 / 597] Compiling src/google/protobuf/util/message_differencer.cc; 0s local ... (8 actions running)
[252 / 597] Compiling src/google/protobuf/util/message_differencer.cc; 4s local ... (8 actions running)
[272 / 597] Compiling src/google/protobuf/descriptor.pb.cc; 8s local ... (8 actions running)
[296 / 597] Compiling src/google/protobuf/descriptor.cc; 12s local ... (8 actions running)
[315 / 597] Compiling src/google/protobuf/descriptor_database.cc; 2s local ... (5 actions running)
[487 / 1,963] Compiling src/compiler/python_generator.cc; 2s local ... (8 actions running)
[579 / 3,117] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv3d.cc; 0s local ... (8 actions, 7 running)
[619 / 3,173] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv3d.cc; 8s local ... (8 actions running)
[687 / 3,270] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv3d.cc; 17s local ... (8 actions running)
[774 / 3,464] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv2d.cc; 27s local ... (8 actions running)
[1,204 / 4,860] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv2d.cc; 39s local ... (8 actions running)
[1,255 / 4,916] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 52s local ... (8 actions running)
[1,340 / 5,042] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 68s local ... (8 actions running)
[1,456 / 5,156] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 86s local ... (8 actions, 7 running)
[1,661 / 5,704] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 107s local ... (8 actions, 7 running)
[1,688 / 5,704] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 132s local ... (8 actions, 7 running)
[1,721 / 5,704] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 160s local ... (8 actions, 7 running)
[2,106 / 6,584] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 192s local ... (8 actions, 7 running)
[2,342 / 7,067] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 231s local ... (8 actions, 7 running)
[2,378 / 7,067] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 274s local ... (8 actions, 7 running)
[2,422 / 7,067] Compiling src/cpu/rnn/ref_rnn.cpp; 75s local ... (8 actions, 7 running)
[2,452 / 7,067] Compiling src/cpu/x64/gemm/f32/jit_avx512_core_f32_copy_at_kern_part1_autogen.cpp; 54s local ... (8 actions, 7 running)
[2,500 / 7,067] Compiling src/cpu/x64/gemm/f32/jit_avx512_core_f32_copy_at_kern_part1_autogen.cpp; 119s local ... (8 actions, 7 running)
[2,577 / 7,067] Compiling src/common/memory_zero_pad.cpp; 74s local ... (8 actions, 7 running)
ERROR: /private/var/tmp/_bazel_a/f5e9a3325f07a1f02c52d821857db47c/external/org_tensorflow/tensorflow/compiler/xla/BUILD:69:17: ProtoCompile external/org_tensorflow/tensorflow/compiler/xla/xla.pb.h failed: (Illegal instruction): protoc failed: error executing command
  (cd /private/var/tmp/_bazel_a/f5e9a3325f07a1f02c52d821857db47c/execroot/__main__ && \
  exec env - \
    PATH=/myuserpath/opt/anaconda3/envs/mytraxenv/bin:/myuserpath/opt/anaconda3/condabin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin \
  bazel-out/darwin-opt/bin/external/com_google_protobuf/protoc '--cpp_out=bazel-out/darwin-opt/bin/external/org_tensorflow' -Iexternal/org_tensorflow -Ibazel-out/darwin-opt/bin/external/org_tensorflow -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/any_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/source_context_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/type_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/api_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/descriptor_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/compiler_plugin_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/duration_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/empty_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/field_mask_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/struct_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/wrappers_proto external/org_tensorflow/tensorflow/compiler/xla/xla.proto)
Execution platform: @local_execution_config_platform//:platform
Target //build:build_wheel failed to build
INFO: Elapsed time: 631.189s, Critical Path: 283.35s
INFO: 2563 processes: 935 internal, 1628 local.
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully
b''
Traceback (most recent call last):
  File "/mypath/jax/build/build.py", line 524, in <module>
    main()
  File "/mypath/jax/build/build.py", line 519, in main
    shell(command)
  File "/mypath/jax/build/build.py", line 53, in shell
    output = subprocess.check_output(cmd)
  File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./bazel-4.2.1-darwin-x86_64', 'run', '--verbose_failures=true', '--config=avx_posix', '--config=mkl_open_source_only', ':build_wheel', '--', '--output_path=/mypath/jax/dist', '--cpu=x86_64']' returned non-zero exit status 1.

I switched away from trying to build jaxlib. In late October 2021, a M1-compatible jaxlib wheel was released, so I tried:

pip install -U pip
pip install -U https://storage.googleapis.com/jax-releases/mac/jaxlib-0.1.74-cp39-none-macosx_11_0_arm64.whl

but got

ERROR: jaxlib-0.1.74-cp39-none-macosx_11_0_arm64.whl is not a supported wheel on this platform.

Tried upgrading python from 3.9 to 3.10, but got the same message.

Debugged by, in python, running

import platform
print(platform.machine())

which showed that my python is still running on x86 architechture, since "Anaconda doesn't yet provide packages for M1/ARM".

[see successful (so far) jaxlib installation in own answer below]

back to trax

After successfully installing jaxlib and jax, when trying to install trax with (miniforge’s)conda install trax I get:

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - trax

Current channels:

  - https://conda.anaconda.org/conda-forge/osx-arm64
  - https://conda.anaconda.org/conda-forge/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

I tried installing as trax itself suggests but ran into an error building hp5y:

ERROR: Failed building wheel for h5py
ERROR: Could not build wheels for h5py, which is required to install pyproject.toml-based projects

Looking now into this error and also looking into doing it with miniforge again but using Anaconda's channel to install trax. I'll update when I get to it.

Any pointers or anyone who managed to install trax on an M1 Mac it by any different means?

Body:
Anaconda Source build

As of Mon, 7th of Feb 2022, installing using an Anaconda installation of python fails as Anaconda does not provide arm wheels for python.

When they do provide them, building from source should work automatically. If it does not, clone the repository, and use the following:

python build/build.py --target_cpu darwin_arm64

If bazel does not build and you are using homebrew consider installing it, and removing the downloaded bazel files.

brew install bazel
rm -rf bazel-*

Arm64 Python wheels

I recommend downloading arm64 wheels from the official CPython website.
The executable will be in /usr/local/bin/python3, and so will pip3.

This should allow you to install jax using

/usr/local/bin/pip3 install --upgrade 'jax[cpu]'

To have access to the proper python with the command line/path, add the folder to the path.

export PATH=/usr/share/bin:$PATH

Virtual Env

I recommend against the above method, and instead to use to use virtual-env from here https://virtualenv.pypa.io, to create virtual environments as with anaconda, and install jax in a virtual environment as above.

Poetry

Poetry is an even better option since it uses virtual-env, and allows you to build your projects and list dependencies just as anaconda.

There is a minor caveat as I am writing this, installing scipy when defining dependencies with poetry results in an odd error, installing numpy and scipy manually using

poetry shell # to go to virtual environment for the project
pip install --upgrade 'jax[cpu]' jaxlib scipy numpy

Trax

Since anaconda is not option, installing hdf5 is a bit of a hassle, but it can be done with brew/homebrew. Assuming you have activated the correct venv:

brew install hdf5
export HDF5_DIR=/opt/homebrew/Cellar/hdf5/1.12.1
pip install trax

Unfortunately this results in the following error:

import trax
/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/jax/_src/lib/__init__.py:32: UserWarning: JAX on Mac ARM machines is experimental and minimally tested. Please see https://github.com/google/jax/issues/5501 in the event of problems.
  warnings.warn("JAX on Mac ARM machines is experimental and minimally tested. "

***************************************************************
Failed to import TensorFlow. Please note that TensorFlow is not installed by default when you install TFDS. This allow you to choose to install either `tf-nightly` or `tensorflow`. Please install the most recent version of TensorFlow, by following instructions at https://tensorflow.org/install.
***************************************************************

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/trax/__init__.py", line 18, in <module>
    from trax import layers
  File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/trax/layers/__init__.py", line 23, in <module>
    from trax.layers.activation_fns import *
  File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/trax/layers/activation_fns.py", line 26, in <module>
    from trax import math
  File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/trax/math/__init__.py", line 18, in <module>
    from trax.math import jax as jax_math
  File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/trax/math/jax.py", line 28, in <module>
    import tensorflow_datasets as tfds
  File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/tensorflow_datasets/__init__.py", line 43, in <module>
    from tensorflow_datasets.core import tf_compat
  File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/tensorflow_datasets/core/__init__.py", line 22, in <module>
    tf_compat.ensure_tf_install()
  File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/tensorflow_datasets/core/tf_compat.py", line 48, in ensure_tf_install
    import tensorflow as tf  # pylint: disable=import-outside-toplevel
ModuleNotFoundError: No module named 'tensorflow'


来源: stackexchange

问:
Can I process a DataFrame using Polars without constructing the entire output in memory?
To load a large dataset into Polars efficiently one can use the lazy API and the scan_* functions. This works well when we are performing an aggregation (so we have a big input dataset but a small result). However, if I want to process a big dataset in it's entirety (for example, change a value in each row of a column), it seems that there is no way around using collect and loading the whole (result) dataset into memory.

Is it instead possible to write a LazyFrame to disk directly, and have the processing operate on chunks of the dataset sequentially, in order to limit memory usage?

答:
Edit (2023-01-08)

Polars' has growing support for streaming/out of core processing.

To run a query streaming collect your LazyFrame with collect(streaming=True).

If the result does not fit into memory, try to sink it to disk with sink_parquet.

Old answer (not true anymore).

Polars' algorithms are not streaming, so they need all data in memory for the operations like join, groupby, aggregations etc. So writing to disk directly would still have those intermediate DataFrames in memory.

There are of course things you can do. Depending on the type of query you do, it may lend itself to embarrassingly parallellizaton. A sum could for instance easily be computed in chunks.

You could also process columns in smaller chunks. This allows you to still compute harder aggregations/ computations.

Use lazy

If you have many filters in your query and polars is able to do them at the scan, your memory pressure is reduced to the selectivity ratio.

ContentLicense: CC BY-SA 4.0


问:
Is there a new solution for downloading package and dependencies for a given R version
I have seen the question asked here from 2018.  I'm wondering if there is a better answer today.

Our work computers are bedeviled by an entire IT security department that seems to exist to make them useless.  We are allowed to run R 3.6.3 (4.x hasn't been approved yet).  We cannot connect to CRAN from behind the corporate firewall.  In the past that meant we took our laptops home to install packages.  But now we have a download monitor that blocks CRAN downloads even when we're on our own wi-fi.

I was attempting to get around this by downloading the package .zip files on a personal machine, transferring them via CD, and then installing with repos=NULL.  I used this code

```<code>getPackages <- function(packs){
  packages <- unlist(
    tools::package_dependencies(packs, available.packages(),
                         which=c("Depends", "Imports"), recursive=TRUE)
  )
  packages <- union(packs, packages)
  packages
}
</code>```

to get the dependencies for the package I wanted and downloaded them all with download.packages().  I was trying to install tidyquant so it turned out there were 113 dependencies.

But of course this downloads the latest version of all 113 packages, several of which it turned out were incompatible with R 3.6.3.

The solution I referenced above suggested finding a compatible version and then using

```<code>install_version("<package name>",
                version = "<version number>",
                repos = "http://cran.us.r-project.org")
</code>```

But that would involve manually searching all 113 dependencies...plus I'm looking for a general purpose solution that I can share with the ~10 other people in my office that all have this issue now.

I'm hoping that since 2018 a better solution may have presented itself?

Update:
Based on answers below I tried

```<code>tmp <- tempfile()
dir.create(tmp)

checkpoint::checkpoint(snapshot_date = "2020-04-01",
                       r_version = "3.6.3",
                       checkpoint_location = tmp,
                       scan_now = FALSE)

packages <- getPackages(c("tidyquant"))

download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
</code>```

I get

```<code>Running create_checkpoint in the home directory may result
in checkpointing very many packages. Continue? (Y/n) Y

Creating checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
Using checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
Warning messages:
1: In create_checkpoint(snapshot_date, r_version, checkpoint_location,  :
  Specified R version not the same as current R version
2: In use_checkpoint(snapshot_date, r_version, checkpoint_location,  :
  Specified R version not the same as current R version
>
> packages <- getPackages(c("tidyquant"))
>
> download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
  cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1/PACKAGES'
Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
  no package ‘tidyquant’ at the repositories
Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
  no package ‘lubridate’ at the repositories
Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
  no package ‘PerformanceAnalytics’ at the repositories
</code>```

And the "no package" warnings go on for all 113 dependencies.

I think the key here is

```<code>Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
</code>```

Obviously there wouldn't be a 4.1 repository in April 2020.  I assume it's trying that since the machine is running 4.1.  So it seems we're tantalizingly close....

If it helps

```<code>> sessionInfo()
R version 4.1.2 (2021-11-01)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 22000)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252
system code page: 65001

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] compiler_4.1.2   tools_4.1.2      checkpoint_1.0.2
</code>```

答:
I'm not sure if it completely addresses your needs, but package checkpoint seems appropriate here. It allows you to download source packages from a snapshot of CRAN taken at a specified date, going back to 2014-09-17. R 4.0.0 was released around 2020-04-24, so the snapshot from 2020-04-01 should work for your purposes.

Here is a reproducible example:

tmp1 <- tempfile()
dir.create(tmp1)
cwd <- setwd(tmp1)

getOption("repos")
##                           CRAN
## "https://cloud.r-project.org/"

available.packages()[c("lattice", "Matrix", "nlme"), c("Version", "Repository")]
##         Version   Repository
## lattice "0.20-45" "https://cloud.r-project.org/src/contrib"
## Matrix  "1.4-0"   "https://cloud.r-project.org/src/contrib"
## nlme    "3.1-155" "https://cloud.r-project.org/src/contrib"

download.packages("Matrix", ".", type = "source")
## trying URL 'https://cloud.r-project.org/src/contrib/Matrix_1.4-0.tar.gz'
## Content type 'application/x-gzip' length 2849865 bytes (2.7 MB)
## ==================================================
## downloaded 2.7 MB
##
##      [,1]     [,2]
## [1,] "Matrix" "./Matrix_1.4-0.tar.gz"

tmp2 <- tempfile()
dir.create(tmp2)
checkpoint::checkpoint(snapshot_date = "2020-04-01",
                       r_version = "3.6.3",
                       checkpoint_location = tmp2,
                       scan_now = FALSE)
## Creating checkpoint directory /var/folders/n7/v9s56rmd5hn17d3f1qj13l7m0000gn/T//RtmpbrT5Br/filee2045e35c290/.checkpoint/2020-04-01/lib/aarch64-apple-darwin20/3.6.3
## Using checkpoint directory /private/var/folders/n7/v9s56rmd5hn17d3f1qj13l7m0000gn/T/RtmpbrT5Br/filee2045e35c290/.checkpoint/2020-04-01/lib/aarch64-apple-darwin20/3.6.3
## Warning messages:
## 1: In create_checkpoint(snapshot_date, r_version, checkpoint_location,  :
##   Specified R version not the same as current R version
## 2: In use_checkpoint(snapshot_date, r_version, checkpoint_location,  :
##   Specified R version not the same as current R version

getOption("repos")
##                                             CRAN
## "https://mran.microsoft.com/snapshot/2020-04-01"

available.packages()[c("lattice", "Matrix", "nlme"), c("Version", "Repository")]
##         Version   Repository
## lattice "0.20-40" "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
## Matrix  "1.2-18"  "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
## nlme    "3.1-145" "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"

download.packages("Matrix", ".", type = "source")
## trying URL 'https://mran.microsoft.com/snapshot/2020-04-01/src/contrib/Matrix_1.2-18.tar.gz'
## Content type 'application/octet-stream' length 1871705 bytes (1.8 MB)
## ==================================================
## downloaded 1.8 MB
##
##      [,1]     [,2]
## [1,] "Matrix" "./Matrix_1.2-18.tar.gz"

setwd(cwd)
unlink(c(tmp1, tmp2), recursive = TRUE)
```
<p>The warnings about version mismatch occur if you are not actually running R 3.6.3. They can be ignored if you are only downloading source packages, with the intention of installing them on another machine actually running 3.6.3.</p>
<p>You can take a look at the package <code>README</code> and <code>?checkpoint</code> for more details.</p>
<h4>Update</h4>
<p>If you are trying to download <em>binary</em> packages (<code>.zip</code> for Windows, <code>.tgz</code> for macOS) rather than <em>source</em> packages (<code>.tar.gz</code>), then <code>checkpoint</code> can get you into trouble. By default, <code>download.packages</code> and friends use <code>contrib.url(repos, type)</code> to construct a URL to search for package binaries.</p>
<pre class="lang-r prettyprint-override"><code>contrib.url("https://mran.microsoft.com/snapshot/2020-04-01/src/contrib", "win.binary")
## [1] "https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1"

contrib.url("https://mran.microsoft.com/snapshot/2020-04-01/src/contrib", "mac.binary")
## [1] "https://mran.microsoft.com/snapshot/2020-04-01/bin/macosx/contrib/4.1"
</code>```

But there is nothing at either URL. That is (in part) because contrib.url appends the R version that you are currently running, which might not have existed on your snapshot date. Hence:

download.packages("Matrix", ".", type = "win.binary")
## Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
##   cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1/PACKAGES'
## Warning in download.packages("Matrix", ".", type = "win.binary") :
##   no package 'Matrix' at the repositories
##      [,1] [,2]

download.packages("Matrix", ".", type = "mac.binary")
## Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/macosx/contrib/4.1:
##   cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/macosx/contrib/4.1/PACKAGES'
## Warning in download.packages("Matrix", ".", type = "mac.binary") :
##   no package 'Matrix' at the repositories
##      [,1] [,2]
```
<p>The URLs that you actually need are:</p>
<pre class="lang-r prettyprint-override"><code>## Windows
"https://cran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/3.6"

## macOS, OS X, whatever
"https://cran.microsoft.com/snapshot/2020-04-01/bin/macosx/el-capitan/contrib/3.6"
</code>```

The best way forward, in this case, may be to avoid checkpoint altogether and directly pass the valid URL to available.packages and download.packages:

pkg <- "tidyquant"
contriburl <- "https://cran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/3.6"

db <- available.packages(contriburl)
deps <- tools::package_dependencies(pkg, db, recursive = TRUE)[[pkg]]
download.packages(c(pkg, deps), ".", contriburl = contriburl, type = "win.binary")
```

Body:
I have seen the question asked here from 2018.  I'm wondering if there is a better answer today.

Our work computers are bedeviled by an entire IT security department that seems to exist to make them useless.  We are allowed to run R 3.6.3 (4.x hasn't been approved yet).  We cannot connect to CRAN from behind the corporate firewall.  In the past that meant we took our laptops home to install packages.  But now we have a download monitor that blocks CRAN downloads even when we're on our own wi-fi.

I was attempting to get around this by downloading the package .zip files on a personal machine, transferring them via CD, and then installing with repos=NULL.  I used this code

getPackages <- function(packs){
  packages <- unlist(
    tools::package_dependencies(packs, available.packages(),
                         which=c("Depends", "Imports"), recursive=TRUE)
  )
  packages <- union(packs, packages)
  packages
}

to get the dependencies for the package I wanted and downloaded them all with download.packages().  I was trying to install tidyquant so it turned out there were 113 dependencies.

But of course this downloads the latest version of all 113 packages, several of which it turned out were incompatible with R 3.6.3.

The solution I referenced above suggested finding a compatible version and then using

install_version("<package name>",
                version = "<version number>",
                repos = "http://cran.us.r-project.org")

But that would involve manually searching all 113 dependencies...plus I'm looking for a general purpose solution that I can share with the ~10 other people in my office that all have this issue now.

I'm hoping that since 2018 a better solution may have presented itself?

Update:
Based on answers below I tried

tmp <- tempfile()
dir.create(tmp)

checkpoint::checkpoint(snapshot_date = "2020-04-01",
                       r_version = "3.6.3",
                       checkpoint_location = tmp,
                       scan_now = FALSE)

packages <- getPackages(c("tidyquant"))

download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")

I get

Running create_checkpoint in the home directory may result
in checkpointing very many packages. Continue? (Y/n) Y

Creating checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
Using checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
Warning messages:
1: In create_checkpoint(snapshot_date, r_version, checkpoint_location,  :
  Specified R version not the same as current R version
2: In use_checkpoint(snapshot_date, r_version, checkpoint_location,  :
  Specified R version not the same as current R version
>
> packages <- getPackages(c("tidyquant"))
>
> download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
  cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1/PACKAGES'
Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
  no package ‘tidyquant’ at the repositories
Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
  no package ‘lubridate’ at the repositories
Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
  no package ‘PerformanceAnalytics’ at the repositories

And the "no package" warnings go on for all 113 dependencies.

I think the key here is

Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:

Obviously there wouldn't be a 4.1 repository in April 2020.  I assume it's trying that since the machine is running 4.1.  So it seems we're tantalizingly close....

If it helps

> sessionInfo()
R version 4.1.2 (2021-11-01)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 22000)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252
system code page: 65001

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] compiler_4.1.2   tools_4.1.2      checkpoint_1.0.2

Body:
I'm not sure if it completely addresses your needs, but package checkpoint seems appropriate here. It allows you to download source packages from a snapshot of CRAN taken at a specified date, going back to 2014-09-17. R 4.0.0 was released around 2020-04-24, so the snapshot from 2020-04-01 should work for your purposes.

Here is a reproducible example:

tmp1 <- tempfile()
dir.create(tmp1)
cwd <- setwd(tmp1)

getOption("repos")
##                           CRAN
## "https://cloud.r-project.org/"

available.packages()[c("lattice", "Matrix", "nlme"), c("Version", "Repository")]
##         Version   Repository
## lattice "0.20-45" "https://cloud.r-project.org/src/contrib"
## Matrix  "1.4-0"   "https://cloud.r-project.org/src/contrib"
## nlme    "3.1-155" "https://cloud.r-project.org/src/contrib"

download.packages("Matrix", ".", type = "source")
## trying URL 'https://cloud.r-project.org/src/contrib/Matrix_1.4-0.tar.gz'
## Content type 'application/x-gzip' length 2849865 bytes (2.7 MB)
## ==================================================
## downloaded 2.7 MB
##
##      [,1]     [,2]
## [1,] "Matrix" "./Matrix_1.4-0.tar.gz"

tmp2 <- tempfile()
dir.create(tmp2)
checkpoint::checkpoint(snapshot_date = "2020-04-01",
                       r_version = "3.6.3",
                       checkpoint_location = tmp2,
                       scan_now = FALSE)
## Creating checkpoint directory /var/folders/n7/v9s56rmd5hn17d3f1qj13l7m0000gn/T//RtmpbrT5Br/filee2045e35c290/.checkpoint/2020-04-01/lib/aarch64-apple-darwin20/3.6.3
## Using checkpoint directory /private/var/folders/n7/v9s56rmd5hn17d3f1qj13l7m0000gn/T/RtmpbrT5Br/filee2045e35c290/.checkpoint/2020-04-01/lib/aarch64-apple-darwin20/3.6.3
## Warning messages:
## 1: In create_checkpoint(snapshot_date, r_version, checkpoint_location,  :
##   Specified R version not the same as current R version
## 2: In use_checkpoint(snapshot_date, r_version, checkpoint_location,  :
##   Specified R version not the same as current R version

getOption("repos")
##                                             CRAN
## "https://mran.microsoft.com/snapshot/2020-04-01"

available.packages()[c("lattice", "Matrix", "nlme"), c("Version", "Repository")]
##         Version   Repository
## lattice "0.20-40" "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
## Matrix  "1.2-18"  "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
## nlme    "3.1-145" "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"

download.packages("Matrix", ".", type = "source")
## trying URL 'https://mran.microsoft.com/snapshot/2020-04-01/src/contrib/Matrix_1.2-18.tar.gz'
## Content type 'application/octet-stream' length 1871705 bytes (1.8 MB)
## ==================================================
## downloaded 1.8 MB
##
##      [,1]     [,2]
## [1,] "Matrix" "./Matrix_1.2-18.tar.gz"

setwd(cwd)
unlink(c(tmp1, tmp2), recursive = TRUE)

The warnings about version mismatch occur if you are not actually running R 3.6.3. They can be ignored if you are only downloading source packages, with the intention of installing them on another machine actually running 3.6.3.

You can take a look at the package README and ?checkpoint for more details.

Update

If you are trying to download binary packages (.zip for Windows, .tgz for macOS) rather than source packages (.tar.gz), then checkpoint can get you into trouble. By default, download.packages and friends use contrib.url(repos, type) to construct a URL to search for package binaries.

contrib.url("https://mran.microsoft.com/snapshot/2020-04-01/src/contrib", "win.binary")
## [1] "https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1"

contrib.url("https://mran.microsoft.com/snapshot/2020-04-01/src/contrib", "mac.binary")
## [1] "https://mran.microsoft.com/snapshot/2020-04-01/bin/macosx/contrib/4.1"

But there is nothing at either URL. That is (in part) because contrib.url appends the R version that you are currently running, which might not have existed on your snapshot date. Hence:

download.packages("Matrix", ".", type = "win.binary")
## Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
##   cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1/PACKAGES'
## Warning in download.packages("Matrix", ".", type = "win.binary") :
##   no package 'Matrix' at the repositories
##      [,1] [,2]

download.packages("Matrix", ".", type = "mac.binary")
## Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/macosx/contrib/4.1:
##   cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/macosx/contrib/4.1/PACKAGES'
## Warning in download.packages("Matrix", ".", type = "mac.binary") :
##   no package 'Matrix' at the repositories
##      [,1] [,2]

The URLs that you actually need are:

## Windows
"https://cran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/3.6"

## macOS, OS X, whatever
"https://cran.microsoft.com/snapshot/2020-04-01/bin/macosx/el-capitan/contrib/3.6"

The best way forward, in this case, may be to avoid checkpoint altogether and directly pass the valid URL to available.packages and download.packages:

pkg <- "tidyquant"
contriburl <- "https://cran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/3.6"

db <- available.packages(contriburl)
deps <- tools::package_dependencies(pkg, db, recursive = TRUE)[[pkg]]
download.packages(c(pkg, deps), ".", contriburl = contriburl, type = "win.binary")


问:
Summation of float in for loop gives numeric errors
when summing up values in a simple for loop, Python gives a numeric error. Any ideas what causes this behaviour?

x = 0.0
for _ in range(1000):
    x += 0.1
print(x)
>>> 99.9999999999986
```

答:
This is just an issue with floating point numbers in general as there are certain decimal numbers that cannot be accurately represented in the float variable type. For example in JavaScript 0.1 + 0.2  = 0.30000000000000004. Python will yield a similar result as will Java with Double types.

Do note that the result is extremely close and therefore "close enough" for financial applications and almost every other application type.


问:
Put everything to zero in a np.array except n highest entries
I'm looking for the quickest way to do the following:

```<code>arr = np.array([2.,0.5,1.3,4.5,7.4])
</code>```

I want to build a new array (or overwrite current) where I put to zero everything except the n highest entries.

```<code>[2.0,0,0,4.5,7.4]
</code>```

答:
I am not sure whether it is the fastest or not but it gives the expected output</p>
```<code>import numpy as np
array = np.array([2.,0.5,1.3,4.5,7.4])
l = len(array)
order = array.argsort()
ranks = order.argsort()

n = 3
array = [array[i] if ranks[i] >= l - n else 0 for i in range(len(ranks))]
print(array)

</code>```

Body:
I'm looking for the quickest way to do the following:

arr = np.array([2.,0.5,1.3,4.5,7.4])

I want to build a new array (or overwrite current) where I put to zero everything except the n highest entries.

[2.0,0,0,4.5,7.4]

Body:
I am not sure whether it is the fastest or not but it gives the expected output

import numpy as np
array = np.array([2.,0.5,1.3,4.5,7.4])
l = len(array)
order = array.argsort()
ranks = order.argsort()

n = 3
array = [array[i] if ranks[i] >= l - n else 0 for i in range(len(ranks))]
print(array)


问:
jq: filter nested array objects
Here my documents:

[
  {
    "id": "3e67b455-8cdb-4bc0-a5e1-f90253870fc9",
    "identifier": [
      {
        "system": {
          "value": "urn:oid:2.16.724.4.9.20.91-INVENTAT"
        },
        "value": {
          "value": "04374"
        }
      },
      {
        "system": {
          "value": "urn:oid:2.16.724.4.9.20.2-INVENTAT"
        },
        "value": {
          "value": "INFP3"
        }
      },
      {
        "system": {
          "value": "urn:oid:INVENTAT"
        },
        "value": {
          "value": "CBOU035"
        }
      }
    ]
  },
  {
    "id": "0f22e5ff-70bc-457f-bdaf-7afe86d478de",
    "identifier": [
      {
        "system": {
          "value": "urn:oid:2.16.724.4.9.20.91-INVENTAT"
        },
        "value": {
          "value": "04376"
        }
      },
      {
        "system": {
          "value": "urn:oid:2.16.724.4.9.20.2-INVENTAT"
        },
        "value": {
          "value": "INF07"
        }
      },
      {
        "system": {
          "value": "urn:oid:INVENTAT"
        },
        "value": {
          "value": "S527918"
        }
      }
    ]
  },
  {
    "id": "a1ea574c-438b-443c-ad87-d31d09d581f0",
    "identifier": [
      {
        "system": {
          "value": "urn:oid:2.16.724.4.9.20.91-INVENTAT"
        },
        "value": {
          "value": "08096"
        }
      },
      {
        "system": {
          "value": "urn:oid:2.16.724.4.9.20.2-INVENTAT"
        },
        "value": {
          "value": "INF04"
        }
      },
      {
        "system": {
          "value": "urn:oid:INVENTAT"
        },
        "value": {
          "value": "5635132"
        }
      }
    ]
  }
]
```
<p>I need to filter <code>.identifier</code> where <code>system.value="urn:oid:2.16.724.4.9.20.91-INVENTAT"</code>  or <code>system.value="urn:oid:2.16.724.4.9.20.2-INVENTAT"</code> and pick <code>.value.value</code>.</p>
<p>Desired output:</p>
<pre class="lang-json prettyprint-override"><code>[
  {
    "id": "3e67b455-8cdb-4bc0-a5e1-f90253870fc9",
    "oid1": "04374",
    "oid2": "INFP3"
  },
  {
    "id": "0f22e5ff-70bc-457f-bdaf-7afe86d478de",
    "oid1": "04376",
    "oid2": "INF07"
  },
  {
    "id": "a1ea574c-438b-443c-ad87-d31d09d581f0",
    "oid1": "08096",
    "oid2": "INF04"
  }
]
</code>```

I've tried:

```<code>map(
    {
        id,
        oid1: select(.identifier?[]?.system.value == "urn:oid:2.16.724.4.9.20.91-INVENTAT") | .identifier[].value.value,
        oid2: select(.identifier?[]?.system.value == "urn:oid:2.16.724.4.9.20.2-INVENTAT") | .identifier[].value.value
    }
)
</code>```

But output is not what I need: you can find it on this jqplay.

Any ideas?

Title: jq: filter nested array objects

Body:
Here my documents:

[
  {
    "id": "3e67b455-8cdb-4bc0-a5e1-f90253870fc9",
    "identifier": [
      {
        "system": {
          "value": "urn:oid:2.16.724.4.9.20.91-INVENTAT"
        },
        "value": {
          "value": "04374"
        }
      },
      {
        "system": {
          "value": "urn:oid:2.16.724.4.9.20.2-INVENTAT"
        },
        "value": {
          "value": "INFP3"
        }
      },
      {
        "system": {
          "value": "urn:oid:INVENTAT"
        },
        "value": {
          "value": "CBOU035"
        }
      }
    ]
  },
  {
    "id": "0f22e5ff-70bc-457f-bdaf-7afe86d478de",
    "identifier": [
      {
        "system": {
          "value": "urn:oid:2.16.724.4.9.20.91-INVENTAT"
        },
        "value": {
          "value": "04376"
        }
      },
      {
        "system": {
          "value": "urn:oid:2.16.724.4.9.20.2-INVENTAT"
        },
        "value": {
          "value": "INF07"
        }
      },
      {
        "system": {
          "value": "urn:oid:INVENTAT"
        },
        "value": {
          "value": "S527918"
        }
      }
    ]
  },
  {
    "id": "a1ea574c-438b-443c-ad87-d31d09d581f0",
    "identifier": [
      {
        "system": {
          "value": "urn:oid:2.16.724.4.9.20.91-INVENTAT"
        },
        "value": {
          "value": "08096"
        }
      },
      {
        "system": {
          "value": "urn:oid:2.16.724.4.9.20.2-INVENTAT"
        },
        "value": {
          "value": "INF04"
        }
      },
      {
        "system": {
          "value": "urn:oid:INVENTAT"
        },
        "value": {
          "value": "5635132"
        }
      }
    ]
  }
]

I need to filter .identifier where system.value="urn:oid:2.16.724.4.9.20.91-INVENTAT"  or system.value="urn:oid:2.16.724.4.9.20.2-INVENTAT" and pick .value.value.

Desired output:

[
  {
    "id": "3e67b455-8cdb-4bc0-a5e1-f90253870fc9",
    "oid1": "04374",
    "oid2": "INFP3"
  },
  {
    "id": "0f22e5ff-70bc-457f-bdaf-7afe86d478de",
    "oid1": "04376",
    "oid2": "INF07"
  },
  {
    "id": "a1ea574c-438b-443c-ad87-d31d09d581f0",
    "oid1": "08096",
    "oid2": "INF04"
  }
]

I've tried:

map(
    {
        id,
        oid1: select(.identifier?[]?.system.value == "urn:oid:2.16.724.4.9.20.91-INVENTAT") | .identifier[].value.value,
        oid2: select(.identifier?[]?.system.value == "urn:oid:2.16.724.4.9.20.2-INVENTAT") | .identifier[].value.value
    }
)

But output is not what I need: you can find it on this jqplay.

Any ideas?

Body:
Here is a ruby to do that:

ruby -r json -e '
def walk(x, filt)
    rtr=[]
    rep=["uab", "ub"]
    x.each{|e|
        rd={"id"=>e["id"]}.merge(
        e["identifier"].
            filter{|ea| filt.include?(ea["system"]["value"])}.
            map.with_index(1){|di, i| ["#{rep[i%2]}", "#{di["value"]["value"]}"]}.to_h)
        rtr << rd
        }
    rtr
end

data=JSON.parse($<.read)
puts walk(data, ["urn:oid:2.16.724.4.9.20.91-INVENTAT", "urn:oid:2.16.724.4.9.20.2-INVENTAT"]).to_json
' file

Prints:

[{"id":"3e67b455-8cdb-4bc0-a5e1-f90253870fc9","ub":"04374","uab":"INFP3"},{"id":"0f22e5ff-70bc-457f-bdaf-7afe86d478de","ub":"04376","uab":"INF07"},{"id":"a1ea574c-438b-443c-ad87-d31d09d581f0","ub":"08096","uab":"INF04"}]


问:
What causes the SVN Error "Not a working copy"?
Recently our Subversion (SVN) server was changed and we did a svn switch.

Since the working copy had a huge amount of unversioned resources, the working copy got locked and we started switching folder by folder for all folders under SVN, which works perfectly fine.

But at the topmost level of the repository, when I try to update files, I get an error that says svn: Working copy '.' locked. Trying to do cleanup does not help either; it causes errors like svn: 'content' is not a working copy directory.

A fresh checkout is not an option at all. How else can I clean up and release the locks and complete the svn switch process?

答:
I saw this error when I was performing mvn release:prepare on my code. In my case, I had migrated the project from SVN to GitHub, and had cloned the source code from GitHub. But in my pom.xml file, I missed to update the SCM location from SVN URL to GitHub URL, which caused the mismatch between the working directories.

Correcting URL to the GitHub location helped solve it for me.

OwnerDisplayName: Vijay Dev

LastEditorDisplayName: Vijay Dev


答:
You can use argsort and indexing. This solution is in place:

```<code>arr = np.array([2.,0.5,1.3,4.5,7.4])

N = 3

# get the sorted order
idx = np.argsort(arr)[:-N]

# replace all but the N highest with 0
arr[idx]= 0
</code>```

output: array([2. , 0. , 0. , 4.5, 7.4])

Title: Put everything to zero in a np.array except n highest entries

Body:
You can use argsort and indexing. This solution is in place:

arr = np.array([2.,0.5,1.3,4.5,7.4])

N = 3

# get the sorted order
idx = np.argsort(arr)[:-N]

# replace all but the N highest with 0
arr[idx]= 0

output: array([2. , 0. , 0. , 4.5, 7.4])


问:
Continue page execution even JavaScript found an error
When found an issue, the javascript is stopped the page loading. What is the way to prevent this? I am working on a buggy web application and I want to load the page fully before doing the testing. In the console, it shows an issue but the site is not loading. how to overcome this?</p>

答: JavaScript executes line by line, and when an error is reached JavaScript stops execution. It's a natural behavior. You just should to fix these errors. Or you can comment all the lines that cause an error, if it's not breaking the app. JavaScript can continue execution when an error is caught and properly handled by try...catch block in your code


问:
How can I change the accessible name of the "OK" hamburger button in an app built from the Xamarin AppShell template?
I've built a Xamarin Android app based on the Xamarin AppShell template in VS. The hamburger button which invokes the flyout gets announced by the TalkBack screen reader as having a name of "OK". This doesn't seem a good match for the purpose of the button, perhaps a name of "Menu" would be more helpful. I've not found a way to change the AutomationProperties.Name of the hamburger button.

How can I change the hamburger button such that TalkBack announces it as the "Menu" button?

答:
I'm not familiar with Xamarin but with basic Android coding, you'd use android:contentDescription to specify the text you want the screen reader (Talkback) to announce.

contentDescription is also mentioned in https://learn.microsoft.com/en-us/xamarin/android/app-fundamentals/accessibility


问:
ParserError in kivy
I am trying to learn kivy and create application in it. during the design language in .kv file, I found:

```<code>   File "D:\Python divane\pythonProject1\kivy\k#1.py", line 20, in <module>
     Care4Me().run()
   File "D:\Python divane\pythonProject1\lib\site-packages\kivy\app.py", line 949, in run
     self._run_prepare()
   File "D:\Python divane\pythonProject1\lib\site-packages\kivy\app.py", line 918, in _run_prepare
     self.load_kv(filename=self.kv_file)
   File "D:\Python divane\pythonProject1\lib\site-packages\kivy\app.py", line 691, in load_kv
     root = Builder.load_file(rfilename)
   File "D:\Python divane\pythonProject1\lib\site-packages\kivy\lang\builder.py", line 306, in load_file
     return self.load_string(data, **kwargs)
   File "D:\Python divane\pythonProject1\lib\site-packages\kivy\lang\builder.py", line 373, in load_string
     parser = Parser(content=string, filename=fn)
   File "D:\Python divane\pythonProject1\lib\site-packages\kivy\lang\parser.py", line 402, in __init__
     self.parse(content)
   File "D:\Python divane\pythonProject1\lib\site-packages\kivy\lang\parser.py", line 511, in parse
     objects, remaining_lines = self.parse_level(0, lines)
   File "D:\Python divane\pythonProject1\lib\site-packages\kivy\lang\parser.py", line 614, in parse_level
     _objects, _lines = self.parse_level(
   File "D:\Python divane\pythonProject1\lib\site-packages\kivy\lang\parser.py", line 624, in parse_level
     raise ParserException(self, ln,
 kivy.lang.parser.ParserException: Parser: File "D:\Python divane\pythonProject1\kivy\care4me.kv", line 3:
 ...
       1:<grid>
       2:    Label:
 >>    3:        text = 'OOOOoooo'
 ...
 Invalid property name
</code>```

This ParserError was throwmn by kivy.

The.kv file:

```<code><grid>
    Label:
        text = 'OOOOoooo'
</code>```

I am new to kivy and and trying to make an application that I have made in tkinter

答:
You must change to

text: "OOOOoooo"

(Use :, not =).

Title: ParserError in kivy

Body:
I am trying to learn kivy and create application in it. during the design language in .kv file, I found:

   File "D:\Python divane\pythonProject1\kivy\k#1.py", line 20, in <module>
     Care4Me().run()
   File "D:\Python divane\pythonProject1\lib\site-packages\kivy\app.py", line 949, in run
     self._run_prepare()
   File "D:\Python divane\pythonProject1\lib\site-packages\kivy\app.py", line 918, in _run_prepare
     self.load_kv(filename=self.kv_file)
   File "D:\Python divane\pythonProject1\lib\site-packages\kivy\app.py", line 691, in load_kv
     root = Builder.load_file(rfilename)
   File "D:\Python divane\pythonProject1\lib\site-packages\kivy\lang\builder.py", line 306, in load_file
     return self.load_string(data, **kwargs)
   File "D:\Python divane\pythonProject1\lib\site-packages\kivy\lang\builder.py", line 373, in load_string
     parser = Parser(content=string, filename=fn)
   File "D:\Python divane\pythonProject1\lib\site-packages\kivy\lang\parser.py", line 402, in __init__
     self.parse(content)
   File "D:\Python divane\pythonProject1\lib\site-packages\kivy\lang\parser.py", line 511, in parse
     objects, remaining_lines = self.parse_level(0, lines)
   File "D:\Python divane\pythonProject1\lib\site-packages\kivy\lang\parser.py", line 614, in parse_level
     _objects, _lines = self.parse_level(
   File "D:\Python divane\pythonProject1\lib\site-packages\kivy\lang\parser.py", line 624, in parse_level
     raise ParserException(self, ln,
 kivy.lang.parser.ParserException: Parser: File "D:\Python divane\pythonProject1\kivy\care4me.kv", line 3:
 ...
       1:<grid>
       2:    Label:
 >>    3:        text = 'OOOOoooo'
 ...
 Invalid property name

This ParserError was throwmn by kivy.

The.kv file:

<grid>
    Label:
        text = 'OOOOoooo'

I am new to kivy and and trying to make an application that I have made in tkinter


问:
How can I get Key press in MFC App when the app is inactive
I create an app using MFC Framework that auto hide on startup, and if I press SHIFT+W then it shows the Windows.

I inherited the function PreTranslateMessage() like that:

```<code>BOOL CTestAppDlg::PreTranslateMessage(MSG* pMsg){
    if (pMsg->message == WM_KEYDOWN){
        if (pMsg->wParam == 0x57){
            if (GetKeyState(VK_SHIFT) & 0x8000) {
                ShowWindow(SW_SHOW);
            }
        }
    }

    return CDialog::PreTranslateMessage(pMsg);
}
</code>```

But this function only catches the keypresses if MFC App is active. So if this App is hide in OnPaint() function with ShowWindow(SW_HIDE) then it cannot catch the SHIFT+W to show windows normally. How can I do for it? Thank all

答:
As explained under Keyboard Focus and Activation:

The system posts keyboard messages to the message queue of the foreground thread that created the window with the keyboard focus.

As a window gets hidden the system transfers keyboard focus to the next eligible window, causing your window to no longer receive keyboard input as you observed.

There are several ways to observe input globally. In this case the most appropriate solution is to just call RegisterHotKey and provide a CWnd::OnHotKey implementation for the respective receiver of the WM_HOTKEY message.

Body:
I create an app using MFC Framework that auto hide on startup, and if I press SHIFT+W then it shows the Windows.

I inherited the function PreTranslateMessage() like that:

BOOL CTestAppDlg::PreTranslateMessage(MSG* pMsg){
    if (pMsg->message == WM_KEYDOWN){
        if (pMsg->wParam == 0x57){
            if (GetKeyState(VK_SHIFT) & 0x8000) {
                ShowWindow(SW_SHOW);
            }
        }
    }

    return CDialog::PreTranslateMessage(pMsg);
}

But this function only catches the keypresses if MFC App is active. So if this App is hide in OnPaint() function with ShowWindow(SW_HIDE) then it cannot catch the SHIFT+W to show windows normally. How can I do for it? Thank all


问:
TSQL : Count distinct values overall and based on criteria by group
I have

Groupvar
Subvar
Val

G1
A
x

G1
A
x

G1
B
x

G1
B
y

G1
C
z

G1
C
z

G2
A
x

G2
A
x

G2
B
y

G2
B
z

G2
B
w

G2
C
z

and I want

Groupvar
All
count (distinct Val)
A
count(distinct Val
where Subvar=A)
B
count(distinct Val
where Subvar=B)
C
count(distinct Val
where Subvar=C)

G1
3
1
2
1

G2
4
1
3
1

Pseudo-code for this could be

```<code>    Select
      Groupvar,
      count(distinct x) as All,
      count( distinct x where Subvar='A') as A,
      count( distinct x where Subvar='B') as B,
      count( distinct x where Subvar='C') as C
    Group by Groupvar

</code>```

I can do this with

```<code>
    CREATE TABLE #have (
    Groupvar VARCHAR(2),
    Subvar   VARCHAR(1),
    Val VARCHAR(1));

    INSERT INTO #have (Groupvar, Subvar, Val )
    VALUES
        ('G1', 'A', 'x'),   ('G1', 'A', 'x'),   ('G1', 'B', 'x'),   ('G1', 'B', 'y'),   ('G1', 'C', 'z'),   ('G1', 'C', 'z'),
        ('G2', 'A', 'x'),   ('G2', 'A', 'x'),   ('G2', 'B', 'y'),   ('G2', 'B', 'z'),   ('G2', 'B', 'w'),   ('G2', 'C', 'z');

    WITH t1 AS (
      SELECT Groupvar, 'All' AS Subvar, COUNT(DISTINCT Val ) AS N_Val FROM #have T1
      GROUP BY Groupvar
      UNION
      SELECT Groupvar, Subvar, COUNT(DISTINCT Val ) AS N_Val FROM #have T1
      GROUP by Groupvar, Subvar
      )
    SELECT * FROM  t1
    PIVOT (SUM(N_Val) FOR Subvar IN([All],[A],[B],[C])) AS pt

</code>```

but I wondered if there was a way of doing it in a single select statement that looked more like my pseudo-code exampe?

答:
You need CASE expressions inside the COUNT() aggregate function to apply conditional aggregation:

```<code>SELECT Groupvar,
       COUNT(DISTINCT Val) [All],
       COUNT(DISTINCT CASE WHEN Subvar = 'A' THEN Val END) A,
       COUNT(DISTINCT CASE WHEN Subvar = 'B' THEN Val END) B,
       COUNT(DISTINCT CASE WHEN Subvar = 'C' THEN Val END) C
FROM tablename
GROUP BY Groupvar;
</code>```

See the demo.

Body:
I have

Groupvar
Subvar
Val

G1
A
x

G1
A
x

G1
B
x

G1
B
y

G1
C
z

G1
C
z

G2
A
x

G2
A
x

G2
B
y

G2
B
z

G2
B
w

G2
C
z

and I want

Groupvar
All
count (distinct Val)
A
count(distinct Val
where Subvar=A)
B
count(distinct Val
where Subvar=B)
C
count(distinct Val
where Subvar=C)

G1
3
1
2
1

G2
4
1
3
1

Pseudo-code for this could be

    Select
      Groupvar,
      count(distinct x) as All,
      count( distinct x where Subvar='A') as A,
      count( distinct x where Subvar='B') as B,
      count( distinct x where Subvar='C') as C
    Group by Groupvar

I can do this with

    CREATE TABLE #have (
    Groupvar VARCHAR(2),
    Subvar   VARCHAR(1),
    Val VARCHAR(1));

    INSERT INTO #have (Groupvar, Subvar, Val )
    VALUES
        ('G1', 'A', 'x'),   ('G1', 'A', 'x'),   ('G1', 'B', 'x'),   ('G1', 'B', 'y'),   ('G1', 'C', 'z'),   ('G1', 'C', 'z'),
        ('G2', 'A', 'x'),   ('G2', 'A', 'x'),   ('G2', 'B', 'y'),   ('G2', 'B', 'z'),   ('G2', 'B', 'w'),   ('G2', 'C', 'z');

    WITH t1 AS (
      SELECT Groupvar, 'All' AS Subvar, COUNT(DISTINCT Val ) AS N_Val FROM #have T1
      GROUP BY Groupvar
      UNION
      SELECT Groupvar, Subvar, COUNT(DISTINCT Val ) AS N_Val FROM #have T1
      GROUP by Groupvar, Subvar
      )
    SELECT * FROM  t1
    PIVOT (SUM(N_Val) FOR Subvar IN([All],[A],[B],[C])) AS pt

but I wondered if there was a way of doing it in a single select statement that looked more like my pseudo-code exampe?

Body:
You need CASE expressions inside the COUNT() aggregate function to apply conditional aggregation:

SELECT Groupvar,
       COUNT(DISTINCT Val) [All],
       COUNT(DISTINCT CASE WHEN Subvar = 'A' THEN Val END) A,
       COUNT(DISTINCT CASE WHEN Subvar = 'B' THEN Val END) B,
       COUNT(DISTINCT CASE WHEN Subvar = 'C' THEN Val END) C
FROM tablename
GROUP BY Groupvar;

See the demo.


问:
export a variable inside a function to be used in other modules
I'm quite new to programming. I have spend last 3 days trying to figure this out. So any help really appreciate it.

I have access to clientID inside a for loop. (Please look at the last line in the code below). There is only one ClientID which I'm successfully able to print.

I want this client ID to be exported from this module so that I can use it in other modules.

```<code>import { Configurations } from './models';

type MethodNames = 'init' | 'event';

export const DEFAULT_NAME = '_hw';

interface LoaderObject {
    q: Array<[MethodNames, {}]>;
}

export default (
    win: Window,
    defaultConfig: Configurations,
    scriptElement: Element | null,
    render: (element: HTMLElement, config: Configurations) => void
) => {

    const instanceName =
        scriptElement?.attributes.getNamedItem('id')?.value ?? DEFAULT_NAME;
    const loaderObject: LoaderObject = win[instanceName];
    if (!loaderObject || !loaderObject.q) {
        throw new Error(
            `Widget didn't find LoaderObject`);
    }

    if (win[`loaded-${instanceName}`]) {
        throw new Error(
            `Widget already loaded`)
        );
    }

    let targetElement: HTMLElement;

    for (let i = 0; i < loaderObject.q.length; i++) {
        const item = loaderObject.q[i];
        const methodName = item[0];
        console.log(methodName);
        if (i === 0 && methodName !== 'init') {
            throw new Error(
                `Failed to start Widget`);
        } else if (i !== 0 && methodName === 'init') {
            continue;
        }

        const valueObject = Object.assign(defaultConfig, item[1]);
        const clientID = valueObject.clientID
        console.log("ClientID", clientID)
    }

//rest of the code....

</code>```

I have also tried this. defined a variable clientID outside the for loop and then storing value from inside the for loop. But on printing, I'm getting undefined

```<code>var clientID;
console.log("....last Client ID", clientID)
</code>```
```<code>   const valueObject = Object.assign(defaultConfig, item[1]);
    clientID = valueObject.clientID

</code>```

答:
I think your problem is about your variable's scope. const is block-scoped so everything about clientID variable happens inside the for loop and does not affect outside. I guess you can use var or let up to your purpose. This is a quite explanatory article about difference between var, let, and const: https://www.freecodecamp.org/news/var-let-and-const-whats-the-difference/

MDN Document on const: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/const

Body:
I'm quite new to programming. I have spend last 3 days trying to figure this out. So any help really appreciate it.

I have access to clientID inside a for loop. (Please look at the last line in the code below). There is only one ClientID which I'm successfully able to print.

I want this client ID to be exported from this module so that I can use it in other modules.

import { Configurations } from './models';

type MethodNames = 'init' | 'event';

export const DEFAULT_NAME = '_hw';

interface LoaderObject {
    q: Array<[MethodNames, {}]>;
}

export default (
    win: Window,
    defaultConfig: Configurations,
    scriptElement: Element | null,
    render: (element: HTMLElement, config: Configurations) => void
) => {

    const instanceName =
        scriptElement?.attributes.getNamedItem('id')?.value ?? DEFAULT_NAME;
    const loaderObject: LoaderObject = win[instanceName];
    if (!loaderObject || !loaderObject.q) {
        throw new Error(
            `Widget didn't find LoaderObject`);
    }

    if (win[`loaded-${instanceName}`]) {
        throw new Error(
            `Widget already loaded`)
        );
    }

    let targetElement: HTMLElement;

    for (let i = 0; i < loaderObject.q.length; i++) {
        const item = loaderObject.q[i];
        const methodName = item[0];
        console.log(methodName);
        if (i === 0 && methodName !== 'init') {
            throw new Error(
                `Failed to start Widget`);
        } else if (i !== 0 && methodName === 'init') {
            continue;
        }

        const valueObject = Object.assign(defaultConfig, item[1]);
        const clientID = valueObject.clientID
        console.log("ClientID", clientID)
    }

//rest of the code....

I have also tried this. defined a variable clientID outside the for loop and then storing value from inside the for loop. But on printing, I'm getting undefined

var clientID;
console.log("....last Client ID", clientID)

   const valueObject = Object.assign(defaultConfig, item[1]);
    clientID = valueObject.clientID