[MSVC]Tensorflow build process takes 11 hours with 2589785 commit
Hi @Eva-An, I see you are using --jobs 8, --> This specifies a limit on the number of jobs that should be executed concurrently during the execution phase of the build. You can increase the number of jobs to improve the time taken. However, it also depends upon your CPU specification - no. of cores etc Hi @mraunak , I made four modifications and below are the settings and results.
1. set --jobs 16 --local_ram_resources=2048, it took 11 hours
2. set --jobs 8 --local_ram_resources=4096, it took 9 and a half hours
3. set --jobs 8 --local_ram_resources=12288, it took 9 and a half hours
4. set --jobs 16 --local_ram_reources=4069, it took 9 hours and 15 minutes Hi @Eva-An, thank you for the update. I ran the build command on my end, it took 30-35 minutes, my CPU specification is Intel Xeon with 24 cores and 56 Logical Processors. command used: 
bazel --output_user_root F:\bazelTemp build --config=opt --define=no_tensorflow_py_deps=true //tensorflow/tools/pip_package:build_pip_package

I removed --jobs and --local_ram_resources arguments and gave the machine the flexibility to allocate resources on its own. Could you please try just running the command:
bazel --output_user_root F:\bazelTemp build --config=opt --define=no_tensorflow_py_deps=true //tensorflow/tools/pip_package:build_pip_package Hi @Eva-An ,

The build time depends upon the no of CPU specifications like no of cores, Memory resources etc. Also its hard to compare the build time with different flags `--jobs` and `--local_ram_reources`. If jobs are more it is scalable (if enough RAM) and also more the memory allotted more will be the performance likely.

We need comparison without any difference in jobs or local_ram_sources etc. Like mentioned in above [comment](https://github.com/tensorflow/tensorflow/issues/60392#issuecomment-1524275096) please try a simple command for both commits you want to test and let us know the results. Request you to use `bazel clean --expunge` command before second build to clear the cache maintained by bazel.

Thanks!


 Hi All,

 I removed --jobs and --local_ram_resources arguments of my command, the test took 8 hours and 15 minutes. Command: bazel --output_user_root F:\bazelTemp build --config=opt --subcommands //tensorflow/tools/pip_package:build_pip_package

@SuryanarayanaY We will only execute the bazel command once on a machine.

Thanks,
Eva Hi @Eva-An, if the problem still persists, please check CPU utilization, and also make sure the CPU is not overloaded with other tasks while running the build command. Please let us know if you are still facing the issue Hi @mraunak, when I start building this project, I will kill other tasks. My machine is a virtual machine with Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz 2.69GHz and Windows specifications are Windows Server 2022 Datacenter.
So far, it still takes about 9 hours to build it on my machine. Hi @Eva-An, thank you for the information. Please run the command:
bazel --output_user_root F:\bazelTemp build --config=opt --jobs=50 -- subcommands //tensorflow/tools/pip_package:build_pip_package > run.log 2>&1

I believe Intel(R) Xeon(R) Platinum 8168 processors will have 48 logical processors. --jobs flag will force the system to run 48 jobs in parallel. Please share the log file where we can find the number of processors being utilized. e.g. 64 processors are utilized in the screenshot shared below.
![image](https://github.com/tensorflow/tensorflow/assets/83710963/e4e0cc68-6f37-426d-935e-bc68f5ff4bb2)


 Hi @mraunak, I tried to build the project with the command your provided, but it comes up with this issue: fatal error C1002:  compiler is out of heap space in pass 2. And I set the --local_ram_resources to 28G, this issue still appears. Hi, @Eva-An please try the Bazel cache clean command below and then run the build command shared above. pls check if  enough space is left on the disk (F:\), 

bazel --output_user_root=F:\bazelTemp clean --expunge

If the issue still persists, please share the log
<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Performance

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

master branch, commit: 2589785

### Custom Code

No

### OS Platform and Distribution

Windows Server 2022

### Mobile device

_No response_

### Python version

3.9

### Bazel version

5.3.0

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

It took 11 hours to build this project with 2589785 commits, however with 160f2bd, it took 6 hours.

**2589785 Commit Build log:**
INFO: Build completed successfully, 34131 total actions
INFO: Build completed successfully, 34131 total actions
##[debug] Command #6 exited with code [0].
 2023-04-12T07:34:32.2365460-07:00  Command ran in 11 hours, 17 minutes, 46 seconds

**160f2bd Commit Build log:**
INFO: Build completed successfully, 16950 total actions
INFO: Build completed successfully, 16950 total actions
##[debug] Command #6 exited with code [0].
 2023-04-14T23:52:20.2671170-07:00  Command ran in 5 hours, 27 minutes, 51 seconds

### Standalone code to reproduce the issue

```shell
git clone https://github.com/tensorflow/tensorflow.git F:\gitP\tensorflow\tensorflow
cd F:\gitP\tensorflow\tensorflow
git reset --hard 2589785
pip3 install -r tensorflow/tools/ci_build/release/requirements_common.txt 2>&1
set PATH=F:\gitP\tensorflow\tensorflow\..\tools;%path%
set PATH=F:\gitP\tensorflow\tensorflow\..\tools\msys64\usr\bin;%path%
yes "" 2>nul | python ./configure.py 2>&1
set BAZEL_VC=C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC
set BAZEL_VC_FULL_VERSION=14.35.32215
set PATH=F:\gitP\tensorflow\tensorflow\..\tools;%path%
set PATH=F:\gitP\tensorflow\tensorflow\..\tools\msys64\usr\bin;%path%
bazel --output_user_root F:\bazelTemp build --jobs 8 --config=opt --local_ram_resources=2048 --subcommands //tensorflow/tools/pip_package:build_pip_package 2>&1
```


### Relevant log output

_No response_</details>