NarrativeBridge: Enhancing Video Captioning with Causal-Temporal Narrative
Video Captioning;Video Narrative;Video Storytelling
Thank you for your follow-up questions. Let us address them:

1. SEPARATOR CLARIFICATION:

We apologize for any confusion in our previous response. We need to make an important point: The original ground truth captions (MSRVTT and MSVD) actually contain no separators - the captions are simple descriptive sentences (see Figure 1). Our choice to use space separation is primarily functional and for fair comparison with baselines:

a) The original captions in both MSR-VTT and MSVD datasets have no comma or period separators

b) The separators are not included in the tokenization

c) We agree that grammatically, commas or other punctuation may be more appropriate. This needs to be explored separately from the provided baselines.

2. PROPOSED TEXT ADDITIONS/MODIFICATIONS:

Here are the specific additions we propose for the camera-ready version:

- Section 3.2, after line 340:

"The preprocessing step maintains a simple space separation between cause and effect statements. While grammatical separators like commas or semicolons could potentially be used, we opt for space separation to maintain consistency with the raw input format of the training datasets (MSR-VTT and MSVD), which contain no separators and also, for a fair comparison with the baselines."

- Section 3.1 (CTN CAPTIONS BENCHMARK), after line 257:

"Distribution of Caption Quality:
1. Pre-filtering statistics: Mean EMScore 
2. Post-filtering statistics: Mean EMScore 
3. X% of initial captions retained after filtering"

In Appendix A.4: [Insert Figure X: Histogram showing EMScore distribution with θ=0.2 threshold line]

- Appendix A.2 (PROMPT DESIGN PROCESS):

Prompt Requirements Analysis:
1. Temporal coherence requirement

2. Causal relationship requirement

3. Event specificity requirement

4. Context preservation requirement

5. Single coherent narrative requirement



- Appendix A.5 (Human Evaluation):
Expert Selection and Evaluation Process:
1. Experts: 3 video understanding researchers (mean experience: 7.2 years)
           2 video generation specialists (mean experience: 6.5 years)
2. Sampling: Stratified random sampling of 100 videos per dataset
3. Evaluation interface details: we provide 100 sampled videos in a directory. The CTN captions are provided in an excel sheet. These are mapped using video ids. Domain experts watch the video, read the CTN and then, give rating against each criterion in the sheet.

- In the Ethics Statement, we will add:

1. Privacy implications in surveillance applications are concerning but CTN captions and the CEN model don't recognize individuals.

2. Guidelines for the responsible deployment of our model include adding safety features for regulated services e.g. children/minor-related content should filter graphic descriptions using automatic content classifiers (ACCs). 

- We will remove the redundant text in the introduction and the related work.

Please let us know of your feedback or any other questions.
Response to Reviewer 8bE2
Thank you for your detailed comments. Overall it sounds good. Two things to discuss:

- I'm not convinced using a space as separator is really as good. Regarding how that matches the training data, is it because the training data itself has separators like commas, periods, etc. dropped from the text during tokenization? Otherwise I find it very unusual, as separating sub-sentences with a comma will more often than not, result in a grammatically incorrect phrase, as for example: "the dog went outside the dog saw the car". I would be very surprised if grammatically incorrect phrases performed better than correct ones. Would you please explain in this space, how this may happen?

- more generally, it would be really good if you could provide samples of the text to add and modify in the camera-ready versions of the paper, so I can evaluate the actual text (except for minor changes like typos or small word replacements). Otherwise I have to review the paper based on a text I won't be able to see until the discussion period is over.

Thank you
We sincerely thank the reviewer for their constructive feedback. We address each point:

1. DATASET STATISTICS

We will add comprehensive CTN statistics to the camera-ready version:

MSRVTT-CTN:
- Videos: 10,000 (1 CTN caption per video)
- Train/Val/Test: 6,513/497/2,990
- Average caption length: ~19 words
- Average cause length: ~10 words
- Average effect length: ~9 words

MSVD-CTN:
- Videos: 1,970 (1 CTN caption per video)
- Train/Val/Test: 1,200/100/670
- Average caption length: ~17 words
- Average cause length: ~9 words
- Average effect length: ~8 words

2. CROSS-DATASET PERFORMANCE

The superior performance of Fine-tune v.v. on MSVD can be explained by:
- MSR-VTT's larger size (10,000 vs 1,970 videos) enables better initial feature learning
- Fine-tuning helps adapt these rich features to MSVD's domain
- Common phenomenon in transfer learning where pre-training on larger datasets improves performance on smaller ones

Terminology

We will:
- Replace "v.v." with "cross-dataset" throughout
- Clarify all abbreviations in the methodology section
- Add a glossary in supplementary material if needed


3. CAUSE-EFFECT MODELING

To directly measure cause-effect capabilities, we conduct:

A) Human Evaluation (100 samples):
- Causal accuracy: 4.8/5.0
- Temporal coherence: 4.7/5.0 
- Inter-rater reliability (ICC): 0.87
This validates our CTN's quality beyond standard metrics.

B) Additional Ablation Analysis:

Single frozen CLIP-ViT baseline without cause-effect training (Stage-1):

MSVD: R-L: 27.81, C: 46.23, S: 15.82

MSRVTT: R-L: 25.34, C: 32.31, S: 14.27

Two frozen CLIP-ViTs baseline without cause-effect training (Stage-1):

MSVD: R-L: 28.40, C: 53.84, S: 16.58

MSRVTT: R-L: 26.10, C: 40.92, S: 14.55

Note: Stage 2 training was performed in both cases. These results demonstrate that:

- Training CLIP-ViTs with separate cause and effect is crucial for performance
- Two specialized encoders outperform a single encoder
- Full CEN significantly outperforms these baselines, demonstrating the importance of specialized cause-effect encoding.

C) Cross-Dataset Generalization:

Our strong cross-dataset results show the model learns generalizable causal relationships rather than dataset-specific patterns.

D) Qualitative Analysis:

As shown in Figure 5 and Appendix A.9:
- Complex multi-step causal chains
- Diverse cause-effect scenarios
- Temporal sequence understanding

E) We appreciate the suggestion for additional cause-effect evaluation. Our additional experiments where we use an LLM (Llama-3.2-3B-Instruct) to measure:

i) Temporal Order Accuracy for both MSRVTT-CTN and MSVD-CTN:
- CEN: 81.2% correct temporal ordering
- Best baseline (GIT): 52.1%
- Measured by comparing predicted vs ground truth temporal sequences, giving a score of 0 or 1

ii) Causal Chain Extraction for both MSRVTT-CTN and MSVD-CTN:
- CEN: 84.5% accurate cause-effect pairs
- Best baseline: 48.3%
- Measured by comparing predicted vs ground truth causal sequences, giving a score of 0 or 1

Together, these analyses demonstrate CTN and CEN's strong capabilities specifically in cause-effect modeling, beyond standard captioning metrics.

4. FIGURE 1 EXAMPLE

Our example demonstrates:
- Clear cause (reckless driving) leading to effect (car damage)
- Temporal sequence of events (play beer pong)
- Real-world complexity in video content
However, we will add another example to illustrate the cause-effect clearly.

5. BENCHMARK VS DATASET
We use "benchmark" as CTN provides:
- Standardized evaluation protocols e.g. zero-shot, for larger models especially recent vision-language models (VLMs)
- Performance baselines

But we will clarify this terminology where needed.

We will add these changes to the camera-ready version. Given our responses, we request the reviewer to reconsider their rating. 

If the reviewer has any other questions, please let us know.
Response to Reviewer qdoh
Thank you for these valuable suggestions and the time taken to review our paper. We address each point:

1. BASELINE PERFORMANCE

The reviewer raises an interesting point about ablated baselines. To clarify:
- EJointCE, Only_Ecause, and Only_Eeffect all use our full training pipeline
- They benefit from the same CLIP features and training strategy
- Their strong performance validates our overall approach
- However, the full CEN still provides significant gains over these baselines

We conduct additional experiments using frozen CLIP-ViT features:

1. Single CLIP-ViT:
- MSVD: R-L: 27.81, C: 46.23, S: 15.82
- MSRVTT: R-L: 25.34, C: 32.31, S: 14.27

2. Two CLIP-ViTs:

- MSVD: R-L: 28.40, C: 53.84, S: 16.58
- MSRVTT: R-L: 26.10, C: 40.92, S: 14.55

Note: Stage 2 training was performed in both cases. These results demonstrate that:
- Training CLIP-ViTs is crucial for performance
- Two specialized encoders outperform a single encoder
- Full CEN still provides significant gains over frozen features

2. PRESENTATION CLARITY

We will improve Table 1 and 2 by:
- Adding "CTN" to dataset names (MSVD-CTN, MSRVTT-CTN)
- Clarifying method names in ablation study
- Adding detailed descriptions in table captions

3. DATA GENERATION APPROACH

Our approach intentionally leverages multiple human annotations to create CTN captions:

- MSR-VTT: 20 human annotations per video
- MSVD: 50 human annotations per video

Multiple perspectives help capture comprehensive cause-effect relationships.

Also, this methodology:
- Can be applied to unlabeled videos - as demonstrated in Appendix A.8
- Eliminates need for expert (familiar to causal-temporal narrative) annotations
- Provides consistent cause-effect structuring
- Enables scalable dataset creation

4. METRIC SELECTION

We use multiple complementary metrics:
- ROUGE-L: Measures fluency and grammatical structure
- CIDEr: Captures semantic similarity
- SPICE: Evaluates detailed semantic content
- Methods with lower metric scores (SPICE, CIDEr, ROUGE-L) consistently produce lower quality outputs, validating our metrics' reliability (Figure 5)

This combination provides comprehensive evaluation across different aspects of caption quality.


We will add the required changes to the camera-ready version. Given our responses, we request the reviewer to reconsider their rating. If the reviewer has any other questions, please let us know.
Response to Reviewer YiKX
We are grateful to the reviewer for their thorough and constructive feedback. We address each point:

1. DISTRIBUTION OF CAPTION QUALITIES

We will add:
- Histogram of EMScore distributions
- Vertical line showing θ=0.2 threshold
- Analysis showing how filtering improves caption quality (in addition to appendix A.4)
- Statistics before/after filtering

2. PROMPT ABLATION

Standard prompts are employed in these LLM based tasks, but we provide 4 ablations in A.2 for more clarity. We will add the following to address this concern:
- Ablation for each prompt requirement
- Comparison of outputs with/without each requirement
- Examples illustrating improvement from each requirement

3. FRAME LEVEL INFORMATION IN CAPTION GENERATION

Thank you for this suggestion. We need to clarify an important point:

- We do NOT generate captions from frames. Our input is:
1. MSR-VTT: 20 human annotations per video
2. MSVD: 50 human annotations per video
- As shown in Figure 1, these human annotations lack causal-temporal narrative structure. Our method transforms these diverse human perspectives into structured cause-effect narratives.
- For unlabeled videos (Appendix A.8), we agree that incorporating visual features probably in a VLM may improve performance. This is an exciting direction for future work.


4. CAUSE-EFFECT SEPARATION

Our choice of space separation was based on:
- Clean tokenization for model processing
- Preserves explicit cause-effect structure - based on Explicit Marking Theory (Knott, A., & Sanders, T. (1998)) and Rhetorical Structure Theory (Mann, W. C., & Thompson, S. A. (1988))
- Maintains temporal ordering - as it is the causal-temporal narrative
- Enables direct mapping for downstream tasks
- Consistency with training data format

5. HUMAN EVALUATION DETAILS

We will add comprehensive details to the camera-ready version:
- Expert selection criteria: 3 video understanding and 2 video generation independent domain experts with at least >5 years’ experience
- Sampling: stratified random sampling across datasets
- Evaluation interface details: we provide 100 sampled videos in a directory. The CTN captions are provided in an excel sheet. These are mapped using video ids. Domain experts watch the video, read the CTN and then, give rating against each criterion in the sheet.

6. WRITING CLARITY

We appreciate the feedback on verbosity. We will remove redundant explanations and maintain technical precision while improving conciseness. We will remove the line about paradigm shift although our intention is solely to highlight the importance of CTN captions in comparison to existing simple captions.

7. ETHICAL CONSIDERATIONS

We will expand the ethical discussion in the camera-ready:
- Privacy implications in surveillance applications - CTN captions and CEN model don't recognize individuals
- Guidelines for responsible deployment - adding safety features for regulated services

Regarding specific QUESTIONS:

- We will replace the fatality video example.
- CTN/CEN similarity indicates improved learning (visualization in Figure 4 is helpful), not overfitting (supported by cross-dataset results in ablations)
- For videos with additional events, we focus on generating the temporal sequence of events (handled in the prompt) as shown in Figure 1 – guys playing beer pong (an additional event). Events without clear causal structure are dealt with similarly and the outputs remain coherent to the causal-temporal narrative. Here are the outputs for the examples you have shared: 1. a person is singing - CTN: {"Cause": "a person began to sing", "Effect": "produced musical song with their voice"}  2. a person is sitting in a chair - CTN: {"Cause": "a person remained seated without any apparent activity or movement", "Effect": "maintained a stationary position in the chair"}. We can explore the confidence score using an LLM. 
- Figure 6 captions: prompt is unable to generate one coherent cause-effect caption instead there are many ranging from 20-50. This fails in the preprocessing part. Even the EMScore based evaluation for each caption gives a score less than 0.2 (threshold).
- Effect encoder receiving cause input: we have run this experiment by concatenating the cause tokens before the effect tokens and then using a projection layer (initialised randomly) - decreased performance by 3.21% and 5.43% on CIDEr for MSRVTT and MSVD respectively. At this point, we can only say that it is because of the added complexity in the form of the projection layer. We will add this to the camera-ready version.

We will add these changes to the camera-ready version as promised. Given our detailed responses, we ask the reviewer to reconsider their rating, please. If the reviewer has any other concerns or questions, please let us know.
Response to Reviewer 8bE2
We sincerely thank the reviewer for their detailed feedback. We address each point below:

1. HUMAN EVALUATION SAMPLE SIZE AND METHODOLOGY

The reviewer raises a point about sample size. However, our sampling methodology follows rigorous statistical practices:

a) Sample Selection:
- MSRVTT (10,000 videos): 50 samples selected
- MSVD (1,970 videos): 50 samples selected
- Proportional representation from train/val/test splits maintained
- With 100 total samples from 11,970 videos, we achieve a 95% confidence level (standard Z-score = 1.96) with ±9.6% margin of error

b) Statistical Validity:
- High inter-rater reliability (ICC: 0.87, 95% CI: 0.83-0.91)
- Stratified sampling ensuring coverage across video types
- Sample size aligns with standard practices in human evaluation studies [1,2]

2. CAPTION CONCATENATION APPROACH

We respectfully disagree that our concatenation approach diminishes caption quality:

a) Intentional Design Choice:
- Preserves explicit cause-effect structure - based on Explicit Marking Theory [3] and  Rhetorical Structure Theory [4]
- Maintains temporal ordering - as it is the causal-temporal narrative
- Enables direct mapping for downstream tasks 

b) Quality Validation:
- In addition to SPICE and CIDEr, we use ROUGE-L scores (measuring fluency): 31.46 (MSVD) and 27.90 (MSR-VTT)
- Qualitative examples (Figure 5, Appendix A.9) demonstrate coherent outputs
- Methods with lower metric scores (SPICE, CIDEr, ROUGE-L) consistently produce lower quality outputs, validating our metrics' reliability

3. ENCODER STRUCTURE CLARIFICATION

We apologize if this wasn't clear enough in the paper:

a) Feature Processing:
- Features are mean-pooled as explicitly mentioned in Section 3.2.1 (line 312)
- h_cause and h_effect are single representations
- Dimension: 512 (matching CLIP's feature dimension)

b) Implementation Details:
- Mean pooling applied after transformer encoding
- Concatenation along feature dimension
- Final dimension: 1024 (512+512)

Given our detailed responses and proposed improvements, we respectfully ask the reviewer to reconsider their rating. 

Would the reviewer find it helpful if we included any additional analyses or clarifications in the camera-ready version?

[1] Chen, David, and William B. Dolan. "Collecting highly parallel data for paraphrase evaluation." Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies. 2011.

[2] Xu, Jun, et al. "Msr-vtt: A large video description dataset for bridging video and language." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[3] Knott, Alistair, and Ted Sanders. "The classification of coherence relations and their linguistic markers: An exploration of two languages." Journal of pragmatics 30.2 (1998): 135-175.

[4] Mann, William C., and Sandra A. Thompson. "Rhetorical structure theory: Toward a functional theory of text organization." Text-interdisciplinary Journal for the Study of Discourse 8.3 (1988): 243-281.
Response to Reviewer 8P6x
Yes
No ethics review needed.
Please refer to weaknesses part.
1. The motivation of the work is clear, while the problem of improving the causal-and-effect understanding capability of a vision-language model is an important research direction.
2. Experimental results show that both the proposed methods, CTN and CEN contribute to better performance on MSVD and MSRVTT benchmarks.
In this paper, a video captioning dataset named Causal-Temporal Narrative (CTN) is proposed. Compared to previous video captioning datasets which lack in causal-temporal  information which connects a cause and an effect, CTN, built with LLMs, includes caption that directly connects cause and effect events. Also, a new architecture named Cause-Effect Network (CEN) is proposed, which consists of two independent encoders which capture cause and effect events, respectively.
1. Including detailed statistics about the constructed CTN dataset (e.g., # of samples, average caption length, # of captions per sample, or more..) may enable readers to easily understand the dataset.
2. Questions about Table 2.
    1. Results show that Fine-tune v.v. shows better performance than CEN on the MSVD benchmark under the cross-dataset evaluation setting, compared to CEN trained with both datasets. The result of cross-dataset evaluation performing better than whole dataset training is somewhat counter-intuitive. Can you provide an explanation for these results?
    2. What does the abbreviation of v.v. stands for? It would be better to mention the full name.
3. Is there additional experimental results or analyses that could show CTN and CEN contributes to better cause-and-effect modeling besides results on MSVD and MSRVTT? Since the quantitative results of Tables 1 and 2 are not directly aimed at measuring cause-and-effect capability, some additional experiments may be required.
4. These are some **minor** issues about the presentation of the work. These are **not** critical weaknesses, and somewhat personal opinions.
    1. Personally, I think the example in Figure 1 is not very intuitive to explain the importance of causal-temporal narrative. Cause: “a car drove recklessly” and Effect: “the car was severely damaged” seems to be a valid cause-and-effect event, but the following event of “a group of guys started playing beer pong” seems like it’s not a direct effect of ‘cause'. This is a personal opinion, but you could consider finding a better example to demonstrate the idea.
    2. I think ‘dataset’ instead of ‘benchmark’ might be a better word for CTN, as it is only used for training but not for evaluation.
Yes
No ethics review needed.
See above.
The paper proposes a benchmark focused on causal-temporal narratives, adapting an evaluation score for automated assessment.

It introduces a novel captioning model that encodes causal and effect information separately, then fuses them to decode the final caption.

Experiments demonstrate that the proposed CEN framework outperforms comparative methods in performance.
The paper highlights that current video captioning benchmarks lack causal-temporal narratives and proposes a new benchmark to address this gap. This benchmark is generated by prompting a large language model (LLM) with a series of captions extracted from a video. To ensure relevance, EMScore is adapted to estimate the similarity between the generated captions and the video, discarding samples with low scores. The paper further introduces a two-stage framework to generate causal-temporal narrative (CTN) captions. In the first stage, two contrastive losses are applied to train Causal and Effect Video Encoders with their respective captions. In the second stage, the encoded causal and effect features are concatenated to decode a cohesive caption.
The paper includes a human evaluation of the generated CTN captions; however, the assessment is less convincing due to the limited sample size. Despite the large dataset size—approximately 300,000 samples in MSRVTT and MSVD—only 100 samples were selected for human evaluation, reducing the reliability of the results.

For supervision during caption decoding, the paper simply concatenates the causal and effect text, which disrupts the original sentence grammar and lowers the quality of the synthetic supervision. The rationale behind this approach is unclear. An alternative method could involve inputting the causal and effect text into an LLM to fuse them into a more natural and coherent sentence. With ground truth captions generated by simple concatenation, evaluation metrics such as SPICE and CIDEr may not accurately reflect the quality of the generated captions. This approach could lead to discrepancies between metric scores and the true readability and coherence of the captions.

The structure of the feature encoder in stage 2 is not clearly depicted. Typically, encoded features from transformer encoders consist of a series of representations. It is unclear whether h_cause and h_effect are lists of representations or single representations. If h_cause and h_effect are single representations, it is not specified how they are generated—whether by selecting a specific token, or by applying mean pooling or max pooling. Furthermore, the specific dimension along which these features are concatenated is not detailed.
Yes
No ethics review needed.
* The fatality video may be too graphic for some readers. I suggest switching it to a video on another topic.

* In figure 5, CTN captions appear too similar to CEN. Is this a symptom of overfitting?

* What does the system do when a video with no apparent cause and effect is used as input? For example, a person is singing but there's no audience reaction. Or a person is just sitting in a chair. The paper mentions "our method assumes a certain level of causal and temporal coherence within the video content"; maybe it would benefit from providing a probability value on the confidence for the existence of cause and effect in the input video?

* `As expected, the resulting captions (Figure 6) are of low quality`, how is this measured?

* Has the team considered the encoder of effect to also receive the encoded cause as input? How would that compare with the current approach?
Significance:

* The problem discussed is important and relevant to much of the field of video captioning and video generation. It's true that typical captioning methods don't typically expose causal information, which makes generated captions much less useful than they could be.

Originality:

* The work appears original though the novel aspects of it are a bit of straightforward applications of existing methods.

Quality:

* The work produces good results. While the paper discusses both results on existing benchmarks, and in CTN, the former being more interesting because they've been established independently.
* I appreciate that both automatic benchmarks and human evaluations were used to evaluate this work.
* The proposed CEN network architecture is interesting and original.

Clarity:

* The work is mostly clear, except for some aspects that are discussed in the weaknesses section.
This paper tackles the problem of captioning videos taking into account causal-temporal narratives. It does so by presenting two main contributions: a dataset and benchmark (CTN) with causes and effects per video, and a network architecture (CEN) for extracting such narratives directly from videos.

When applying the CTN benchmark to CEN, the network gets better metrics when compared to similar methods, potentially establishing it as SOTA in the domain.
* The automatic evaluation process is found to filter out most bad cause and effect generations. More complete analysis on the distribution of the qualities of the generated captions is missing. I can imagine a histogram that displays the distribution of qualities with a vertical line showing the threshold that's been picked. Also the paper would benefit from analysis on whether the automatic evaluation filtering improves on the method, or is a patch for more fundamental issues in the prior step.

* The contribution of CTN depends strongly on the prompt used to zero shot cause and effect generation. Given that this prompt is so central to the paper, the paper would benefit from more complete ablations regarding different aspects of the prompt. Especially important, the list of requirements that are presented in the prompt are justified in the paper but no corresponding ablation per requirement is done to validate the explanation. (the ablation donefor 4 prompts of varying degrees of quality appears insufficient for this purpose)

* When building CTN, the captions are 1) generated from frames, and then 2) a model combines multiple captions into cause and effect captions without looking at frame information again until 3) the automatic evaluation step. Has the team considered using a model that can read text + image in step 2 as well? 

  Concluding cause and effect just from the individual frame captions is a lot more error prone than it could be if image information was also used. I think this possibility could be mentioned / analyzed as a potential alternative.

* Why are causes and effects only separated by a space in CTN? Have other choices been ablated? In particular, I can imagine using a "comma" or an "and" between both to work slightly better.

    * Has the team considered using an LLM to combine both captions together, instead of using concatenation?

* For A5.1: more discussion on the human evaluation process would be very beneficial. How were the experts chosen? How is information presented to them? How were the 100 videos chosen?

  For example it would be ideal to have an independent expert set; and it would be ideal if they were presented with the captions without labels that identify the models, and in such a way that each the models are independently ordered for each video (so annotators cannot be biased to always vote for the first model, for example)

* Clarity: The paper, especially in the first few pages, is a bit too repetitive or verbose. It may benefit from some proofreading for succinctness. For example, the following two paragraphs are very close to each other, and both express a similar idea:

```
It means that our CTN benchmark differs significantly by focusing on generating comprehensive causal-temporal narratives, capturing broader temporal relationships within a single caption, and providing a more holistic view of video content
```

  and

```
We introduce a new benchmark CTN specifically designed to capture and evaluate causal-temporal narrative in video captioning. Our approach goes beyond existing benchmarks by explicitly modeling causal-temporal narrative in a single, coherent caption, enabling a more comprehensive understanding of video content.
```

  In other cases ideas are repeated in the same paragraph:

```
However, as our experiments show (see Section 4.4), even these advanced models struggle with generating accurate causal-temporal narratives. To address this limitation, there is a need to develop new video captioning models that can overcome the shortcomings of current frameworks. This also underscores the need for specialized architectures like our proposed CEN that are explicitly designed to capture and generate causal-temporal narratives in videos
```

  one could summarize this text as:

```
However, as our experiments show (see Section 4.4), even these advanced models struggle with generating accurate causal-temporal narratives. This also underscores the need for specialized architectures like our proposed CEN that are explicitly designed to capture and generate such narratives in videos
```

  Such verbosity makes the paper hard to read, despite the main ideas being explained very clearly.

* The paper would benefit from a more complete discussion on possible downsides of this approach. For example for CTN how it compares to using frame data in step 2 as discussed above, or what downsides could arise from the concrete proposed architecture.

* The paper mentions surveillance; maybe analyze this in the context of ethic concerns?

* `We acknowledge potential ethical concerns (e.g., privacy in surveillance, risk of misleading content) but are committed to responsible development`, could this be made more concrete?

* The paper mentions `lays the foundation for a paradigm shift`, which appears potentially grandiose in my opinion. Better to let the community arrive to this conclusion, than write it in the text.

I'll be glad to revise my score depending on the response to this review.
Yes
No ethics review needed.
See above.
- While most of the causal video understanding focuses on video question answering, this paper proposes to bring causal understanding into video captioning. 
- The data generation pipeline looks sound.
- The proposed approach achieves strong results on the proposed benchmark.
This paper focuses on incorporating causal-temporal narrative into video captioning.  The authors create a new dataset called Causal-Temporal Narrative (CTN) captions, using a LLM to generate captions that explicitly describe cause-effect relationships and maintain temporal consistency within videos based on multiple annotated ground-truths for the same video. The quality of these CTN captions is assessed both automatically (using EMScore) and through human evaluation.

Furthermore, the authors propose a novel model architecture called the Cause-Effect Network (CEN), specifically designed to learn from the CTN captions. CEN decouples the cause and effect captions by finetuning one visual encoder for each of them. The representations of both encoders are then used during the LLM finetuning stage.

Experiments on MSVD and MSR-VTT datasets demonstrate that CEN significantly outperforms state-of-the-art video captioning methods for generating CTN captions. Ablation studies further validate the design choices of CEN.
- The weakest ablated baseline already performs well above state-of-the-art approaches. It would be great to understand what the differences are. For instance, how does a baseline with the CLIP ViT features used directly perform?
- The presentation can be improved, for instance it is not clear that the evaluation is on the CTN captions in Table 1, and the ablation study includes method names which are hard to interpret when having a first look at the table.
- The data generation approach largely relies on the multiple GT captions present for each video in MSR-VTT and MSVD. 
- Captioning metrics which do not move much like Rouge-L are hard to interpret.