TensorFlow Profiler did not work correctly.
@tilakrayal 
Do you think this issue is a bug in the TensorFlow Profiler?
Please let me know your opinion. @y-vectorfield,
I tried to execute the given tutorial and it was executed without any issue/error. Kindly find the gist of it [here](https://colab.research.google.com/gist/tilakrayal/c1d0879b01c06230fc2bdf0926cdfe47/tensorboard_profiling_keras.ipynb?hl=ja) and the screenshot for the reference.

![image](https://github.com/tensorflow/tensorflow/assets/81610181/674803e2-cedc-4d5f-82e0-ff166fd3a7e6)


Thank you!

 Same problem for me. My local setup with TF2.14 produces the same problem "No profile data was found."


If I execute the official notebook https://colab.research.google.com/github/tensorflow/tensorboard/blob/master/docs/tensorboard_profiling_keras.ipynb
then I encounter the same problem.

@tilakrayal What you show in the screenshot is not the profiler ! 



 Hello, @tilakrayal 
@TillBeemelmanns and I could implement TensorBoard.
However, we could not watch the result of the TensorFlow Profiler using TensorBoard.
I think this is a problem of the TensorFlow Profiler.
@zzzaries , @qiuminxu , what do you think about this problem?
Please let us know your opinions.

We could not confirm the following content.
>The **Profile** tab opens the Overview page which shows you a high-level summary of your model performance. Looking at the Step-time Graph on the right, you can see that the model is highly input bound (i.e., it spends a lot of time >in the data input piepline). The Overview page also gives you recommendations on potential next steps you can follow to optimize your model performance. 
 Hello @tilakrayal , 
     I wanted to use tensorboard for inference, 
tf.profiler.experimental.start('logs')
output = model.predict(encoded_input, verbose=False)[0]
tf.profiler.experimental.stop()
I have tried this way . But When the profiler tab opened its saying no profile data. May I know how we can get profile data for model inference

 @yisitu , @ronshapiro ,
What do you think about the above problem?
The TensorFlow Profiler(v2) did not work correctly and did not watch the measurement results. So I could resolve the problem. The solution for me was to manually copy and paste this file https://github.com/tensorflow/profiler/blob/master/demo/events.out.tfevents.1583461681.localhost.profile-empty

into the Tensorboard log directory of my experiment, then suddenly Tensorboard and the profiler-plugin load the results of the Tensorflow profiler.  
 @TillBeemelmanns ,
       Did you try for inference as well?
 @TillBeemelmanns 
Thank you very much for your report.

> The solution for me was to manually copy and paste this file

I think the profiler should create this event file automatically, not manually. Hello, @sachinprasadhs ,
What do you think about this issue??
I think this profiler should create an event file(event.out.tfevents.xxx.xxxx.profile-empty) automatically.
However, the latest version of this does not create it. Hi,

Thanks for reporting the issue, I have reported the issue to the concerned team, I was able to find similar issues which is reported here https://github.com/tensorflow/tensorflow/issues/61212 Hello, @sachinprasadhs 
I confirmed #61212.
This is a similar issue of this. Hi y-vectorfield, for the "No data Found" issue, could you please share 
- the screenshot of your log directory after collecting the profile so we can check on the profile data.
- How you collect the profile (if possible, the code structure)
Thanks!

And the profile inactive issue should be fixed soon with a new change.
 Hello, @zzzaries, thank you very much for your message.
@akote123 and I tried to measure the inference performance using this.
The example of measurement code is as follows.

```python
log_dir = "./logs"
with tf.profiler.experimental.Profile(log_dir, options):
    some inference codes are implemented
```

If we implement the above code, we get the following directory structure.

```bash
logs/
└── 20231116-004551
    └── plugins
        └── profile
            └── 2023_11_16_00_46_09
                └── intel-gpu-tensorflow-stable-diffusion-v2140-yvectorfield.xplane.pb
```

We can not get some files of measurement results. Hence, we can not watch the measurement results using TensorBoard.  Hi @zzzaries ,
    We also tried with 
tf.profiler.experimental.start('logs')
output = model.predict(encoded_input, verbose=False)[0]

when I do tensorboard --logdir=logs
I get profile page as attached .
<img width="926" alt="tensorboard" src="https://github.com/tensorflow/tensorflow/assets/133775732/ff25c873-e096-4f5f-be6e-34291a592917">
 @akote123 , thank you very much for your info sharing. Thanks @y-vectorfield @akote123 for the detailed description!

And @akote123 could you also share the directory structure like what y-vectorfield mentioned above? This could be very helpful for us to debug the issue as the directory structure and files inside will potentially determine if tensorboard can identify valid profiler runs data there, Thank you! @zzzaries, thank you for your message.
Can I help you to fix this issue?
I also try to investigate the root cause of this! @zzzaries , Thank you. The directory structure looks like:
```
logs/
  -plugins
       -profile
                 -2023_11_17_10_16_20
                                  -ip-172-31-93-51.xplane.pb

```
This how the directory structure looks like.

Thank You @akote123, thank you very much for your sharing. @zzzaries , Do you have any example inference code where tensorboard shows the profile data? 
Can you please share any working example for model inference tensorboard code, so that it will be helpful for us to understand whether we are doing any mistake while using profiler API.

Thanks I used the following code to investigate the root cause of this issue.
Reference: [Extract features with VGG16](https://keras.io/api/applications/#extract-features-with-vgg16)

```python
from pathlib import Path                                                                                                                                                                                                                                                                                                                                                            

import numpy as np

import tensorflow as tf
from tensorflow.keras.applications.vgg16 import VGG16
from tensorflow.keras.applications.vgg16 import preprocess_input
from tensorflow.keras.preprocessing import image


def main(args):
    model = VGG16(weights='imagenet', include_top=False)
    img_path = 'common/elephant.jpg'
    img = image.load_img(img_path, target_size=(224, 224))
    x = image.img_to_array(img)
    x = np.expand_dims(x, axis=0)
    x = preprocess_input(x)
    log_path = Path(args.log_path)
    if not log_path.exists():
        raise FileNotFoundError(str(log_path))
    with tf.profiler.experimental.Profile(str(log_path)):
        model.predict(x)


if __name__ == '__main__':
    from argparse import ArgumentParser
    arg_parser = ArgumentParser()
    arg_parser.add_argument("--log_path", required=True)
    args = arg_parser.parse_args()
    main(args)
``` I got the following results. I used each version of TF official container image from Docker Hub.
https://hub.docker.com/r/tensorflow/tensorflow/tags?page=1

|TF version|eventfile|profiler results|
|:---:|:---:|:---:|
|2.8.0|○|○|
|2.8.1|○|○|
|2.8.2|○|○|
|2.8.4|○|○|
|2.9.0|○|○|
|2.9.1|○|○|
|2.9.2|○|○|
|2.9.3|○|○|
|2.10.0|○|○|
|2.10.1|○|○|
|2.11.0|○|✕|
|2.11.1|○|✕|
|2.12.0|✕|✕|
|2.13.0|✕|✕|
|2.14.0|✕|✕|
|2.15.0|✕|✕|

The result eventfile: ○, profiler results: ○ means the following dir structure.

```bash
logs/
├── events.out.tfevents.1700568471.a0a96e93ba93.profile-empty
└── plugins
    └── profile
        └── 2023_11_21_12_07_51
            ├── a0a96e93ba93.input_pipeline.pb
            ├── a0a96e93ba93.kernel_stats.pb
            ├── a0a96e93ba93.memory_profile.json.gz
            ├── a0a96e93ba93.overview_page.pb
            ├── a0a96e93ba93.tensorflow_stats.pb
            ├── a0a96e93ba93.trace.json.gz
            └── a0a96e93ba93.xplane.pb

3 directories, 8 files
```

The result eventfile: ○, profiler results: ✕ means the following dir structure.

```bash
logs/
├── events.out.tfevents.1700520805.fa34d7d6c817.profile-empty
└── plugins
    └── profile
        └── 2023_11_20_22_53_25
            └── fa34d7d6c817.xplane.pb

3 directories, 2 files
```

The result eventfile: ✕, profiler results: ✕ means the following dir structure.

```bash
logs/
└── plugins
    └── profile
        └── 2023_11_21_05_12_44
            └── d12c6b2d4b00.xplane.pb

3 directories, 1 file
``` thanks @y-vectorfield  for the detailed debugging and that's really helpful!

for the 3 types of directory structure you showed, the 1 and 2 should both work (2 is intended change), meaning until 2.11 it should be working fine. We've identified some suspicious code snippet and has been working on a fix for this, will update you ASAP.   > @zzzaries , Do you have any example inference code where tensorboard shows the profile data? Can you please share any working example for model inference tensorboard code, so that it will be helpful for us to understand whether we are doing any mistake while using profiler API.
> 
> Thanks

@akote123 I'm not an expert in performance tuning, but from what I know, you are not doing anything wrong, and this should be a known issue on our side.

However want to double check, as you mentioned 
> We also tried with
> tf.profiler.experimental.start('logs')
> output = model.predict(encoded_input, verbose=False)[0]
> when I do tensorboard --logdir=logs
> I get profile page as attached .

does the above code create the directory of following but you can see the data in the profiler?
and this is all the files within /logs dir?

```
logs/
  -plugins
       -profile
                 -2023_11_17_10_16_20
                                  -ip-172-31-93-51.xplane.pb
``` Hello, @sachinprasadhs, @zzzaries, 
I think we should attach the bug label to this issue and remove the support label.
What do you think of my opinion? @zzzaries, thank you very much for your advice.

>  the 1 and 2 should both work (2 is intended change), meaning until 2.11 it should be working fine. 

OK, I will check this using TensorBoard.
By the way, I would like to contribute to solving this problem.
I will investigate the source codes deeply and submit some modification codes(Pull Request) as soon as possible.
OK? @sachinprasadhs, thank you! > > @zzzaries , Do you have any example inference code where tensorboard shows the profile data? Can you please share any working example for model inference tensorboard code, so that it will be helpful for us to understand whether we are doing any mistake while using profiler API.
> > Thanks
> 
> @akote123 I'm not an expert in performance tuning, but from what I know, you are not doing anything wrong, and this should be a known issue on our side.
> 
> However want to double check, as you mentioned
> 
> > We also tried with
> > tf.profiler.experimental.start('logs')
> > output = model.predict(encoded_input, verbose=False)[0]
> > when I do tensorboard --logdir=logs
> > I get profile page as attached .
> 
> does the above code create the directory of following but you can see the data in the profiler? and this is all the files within /logs dir?
> 
> ```
> logs/
>   -plugins
>        -profile
>                  -2023_11_17_10_16_20
>                                   -ip-172-31-93-51.xplane.pb
> ```

@zzzaries ,these are the  files created and in the profiler its not showing any data. Im expperimenting the same issue I tryed the solution provided by @TillBeemelmanns but this still doesn't work for me

> So I could resolve the problem. The solution for me was to manually copy and paste this file https://github.com/tensorflow/profiler/blob/master/demo/events.out.tfevents.1583461681.localhost.profile-empty
> 
> into the Tensorboard log directory of my experiment, then suddenly Tensorboard and the profiler-plugin load the results of the Tensorflow profiler.


I used the TensorFlow profiler to measure large-scale DL models' CPU/GPU performance. However, this tool did not work correctly. I did not see a graph for the results of measurement using TensorBoard. Hence, I tried a tutorial of [this](https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/ja/tensorboard/tensorboard_profiling_keras.ipynb?hl=ja), and this problem occurred again. I think the current version of the TensorFlow profiler does not save all of the files for measurement results.

The following image shows the current status.
<img width="1986" alt="TensorBoard" src="https://github.com/tensorflow/tensorflow/assets/30323722/b043d495-6999-4383-bac9-6f4fbe255885">
