Feynman: Knowledge-Infused Diagramming Agent for Scaling Visual Reasoning Data
Vision-Language Dataset;Synthetic Data;Visual Reasoning Benchmark
Yes
No ethics review needed.
- What are the visual judges?
- Why were there no human evaluations even at the very least for sanity checking conducted?
- Why choose the QA format after all the hard work? In addition, how do we envision the community would benefit from this dataset or any of the proposed methods?
- The whole approach of the proposed Feyman is sound.
- The work is tackling a challenging domain and indeed requires a much better scalable solution.
- The feedback mechanism and deduplication is interesting.
This work proposes the Feyman framework, which is tackling the challenges of scalable generations of knowledge-grounded diagrams. Feyman follows a code-generation and diversification fashion. At first, an LLM is used to propose some “ideas”, and then these ideas will be translated into codes following the Penrose template. The Penrose code can be formulated as a constraint optimization problem to derive diagrams. Then equipped with a visual feedback mechanism, Feyman generates visual diagrams with the codes and refine upon satisfaction. Eventually Feyman is used to generate knowledge-grounded visual QAs.
The authors then conduct some extensive studies on their synthetically generated data with various open-source and proprietary visual-language models.
- More detailed statistics are needed for the proposed dataset. It is clueless to judge its quality, complexity, as well as how it would contribute/drive the research community.
- While Table 2 is much appreciated (for its comprehensiveness), what is the current plausible upper bound? Of course 100% correctness would be the ultimate ideal performance, but what is the realistic upper bound for the community to judge the progress?
- More details of the visual judges are needed. For example, what exactly are they? Are they fine-tuned, are they evaluated with a certain type of performance measurement? How do we know if the judges are faithful enough?
Yes
No ethics review needed.
As I mentioned above.

- What exactly is the significance of Figure 8? 
    - Can the relationship between tokens and the number of programs demonstrate scalability?
    - There are too few data points shown in the figure. It seems that it is difficult to observe obvious patterns when fitting these lines.
    - It shows the relationship between total input/output tokens and the number of generated images after de-duplication. Does it consider the number of tokens consumed in the iterative process?

- In Algorithm 1 
    - Does setting different rendering seeds affect the scoring? 
    - Is it a binary score (good/bad), or is it a score given within a specific range?
 
- If we remove Iterative Visual-Refine based on VLMs and only reflect based on feedback by the compiler. What will be the effect? How many samples can it generate?

- Please elaborate in more detail on what the innovation of the paper is? It seems that generating charts through code as a medium has been explored by some works.

- FEYNMAN generated over 106,930 diagram-caption pairs, yet DIAGRAMMA only contains 1,058 multiple-choice questions？ How many testing set in total?

- Could you provide a complete example, including each step of the pipeline’s output and the iteration process?

- Since no anonymous open-source link has been provided, the quality of the data cannot be determined. Hope to see the provision of an anonymous link in the subsequent rebuttal.
- The paper conducted comprehensive experiments and analyses. 

    - It studied the generation capabilities of the pipeline across different subjects, evaluated the performance of 17 MLLMs on DIAGRAMMA. 

    - Additionally, compared FEYNMAN with other methods for generating knowledge-rich images, and  an ablation study was performed to examine the impact of different pipeline components. 

     - The appendix provides detailed prompt designs for each component of the pipeline.

- The paper uses iterative visual refinement to obtain programs that better represent knowledge. 

    - During the iterations, VLMs act as judges, providing visual judgments and feedback to refine the program and produce better images with correct representation, proper relationships, legible text, simplicity, no redundancy and other essential qualities.
This paper presented a scalable data generation pipeline with a diagramming agent, FEYNMAN. Through four steps, idea, plan, iterate and render, FEYNMAN decouples knowledge elicitation and visual production to create programs, which are then rendered into different layout diagram variations using penrose.


FEYNMAN generated 10693 programs, further resulting in 106,930 diagram-caption pairs. Additionally, it curated a scientific benchmark, DIAGRAMMA,  which contains 1,058 multiple-choice questions focused on visual understanding and reasoning. 

The paper experimented with the generation capabilities of FEYNMAN pipeline across different subjects and the Appendix D.3 further discusses the diversity between programs and images under identical and distinct substances or domains. Furthermore, the paper evaluated DIAGRAMMA on 17 different MLLMs, confirming its validity for measuring the scientific visual reasoning capabilities of MLLMs.


Additionally, it qualitatively compared different diagramming approaches, discussing the relationship between the number of generate images and the total input/output tokens. At the same time, an ablation study was conducted to examine the impact of various pipeline components on the accuracy and quality of the generated programs.
- How many data were generated exactly? Is this in Section 3.1 mentions generating 10,693 unique substance programs across different subjects through the pipeline?

- To demonstrate the scalability of the pipeline, Section 4.2 discusses the relationship between the total input/output tokens and the number of generated images after de-duplication.  I don't quite understand the significance of this figure. The more samples are generated, the more tokens are naturally consumed. This seems like correct but useless talk.

- In Section 4.3, ablation experiments were conducted on 10 subdomains, but the experimental conditions and analysis are somewhat vague.
    - What exactly is the specific experimental design? How are these several pipelines compared? 
    - What is the specific number of samples in each sub-domain?
    - How exactly were KP, CP, and S ablated? If CP is missing, how is code generation achieved?
    - In Table 3, Why does the Yield and Compile of CP + S differ by about 10% when KP is absent?
    - Were these evaluation metrics designed by the author (PR, LR,...) all given by VLMs? Is this accurate enough?

- In the iterative process, feedback is provided by GPT-4o, Claude-3.5-sonnet, and Gemini-Pro-1.5. However, what I want to say is that this evaluation seems to be inaccurate. The benchmark generated in this paper evaluates the performance of VLMs on such math-related charts. Judging from Table 2, the highest performance is only 60%. It seems unreliable to use these VLMs for evaluation and provide feedback.
Yes
No ethics review needed.
Please refer to the Weakness section.
There are several noticeable strengths in this paper:
* This paper presents a well-justified motivation for introducing LLMs into the pipeline of automatic diagram generation process.
* The manuscript is commendable for its clarity and structured writing style, which greatly facilitates reader comprehension.
* Additionally, the inclusion of clear and illustrative figures and tables is a notable strength, as it significantly aids in conveying the main claims of the paper to the audience.
This submission presents FEYNMAN, a knowledge-infused diagramming agent designed to generate high-quality, well-aligned diagram-caption pairs at scale. ​ The agent decouples knowledge elicitation from visual production, leveraging LLMs for domain-specific knowledge enumeration and the PENROSE diagramming system for rendering. ​ The authors synthesized a dataset of over 100k diagram-caption pairs and curated a visual-language benchmark, DIAGRAMMA, to evaluate the visual reasoning capabilities of vision-language models.
Since this submission is one of the first studies on MLLM-assisted diagram generation agents, it could offer even greater value to future research if the authors provided a more comprehensive error analysis. Such analysis would help identify bottlenecks in the current pipeline and suggest directions for improvement. 

Specifically, in the four steps of the FEYNMAN pipeline—idea, plan, iterate, and render—what types of errors occur? How are these errors distributed across the steps, and what might be their underlying causes? For example, are certain errors attributable to the limitations of the PENROSE program, or are others due to knowledge gaps or inaccuracies inherent in the LLMs? A detailed breakdown of these errors would provide valuable insights for future work.
Yes
No ethics review needed.
- The Penrose diagram tool should be cited (with URL) the first time it’s mentioned.
- In Sec. 2, the subsection names do not directly match with the four steps mentioned in the first paragraph of Sec 2: idea, plan, iterate, and render. For example, Sec 2.2 is ‘Knowledge Planning' instead of ‘idea'; there's no subsection for the last step - rendering. Explicitly matching the subsection names with the four steps would make the readers easily match which subsection describes which step.
- Is Sec 2.5 part of the ‘iterate’ step? it is not clear how the QA data is generated and used. The step needs to be explained in more detail.
- Introduction of Feynman, an LLM-based diagram generation framework that uses Penrose as a backend.
- Introduction Diagramma, a VQA dataset to evaluate  diagram understanding capability of MLLMs where the diagram images are generated with Feynman
The authors introduce Feynman, a diagram-generation agent.  Feynman leverages Penrose, a diagramming tool that generates vector graphic diagrams from its own programming codes. Feynman’s diagram synthesis has four steps: idea, plan, iterate, and render.

- In idea step, an LLM is given a text description of target diagram and generates relevant ideas/concepts in plain text.
- In plan step, an LLM is given the documentation of Penrose and in-context examples and generates visual elements.
- In iterate step, a panel of visual judges (MLLMs) iteratively provide the feedback and the generation plan gets refined.
- Render step is not explicitly described, but perhaps Penrose tool would render the iteratively refined plan from the previous step into a diagram.

The authors also provide a benchmark called Diagramma, which contains 1058 multiple-choice QAs on the diagrams generated with Feynman.

The authors provide a brief qualitative comparison of Feynman and baselines (AutomaTikz and Flux-Pro). They also provide the evaluation of different MLLMs on Diagramm, and ablation of different planning components within Feynman.
- **Unclear discussion about Penrose vs. TikZ.** In the introduction, the authors mention that “AutomaTikz still exhibits efficiency overhead in synthesizing scientific diagrams at scale, due to the inherent complexity of both the TikZ language and visual design.” However, the authors do not compare using TikZ or Penrose as a rendering backbone. There must be different scenarios when Tikz and Penrose are more useful, and the authors should discuss them.
- **Weak comparison with baselines.** The authors only provide brief qualitative examples in Fig. 7, but it is unclear how the five input prompts are selected. In addition, compared to AutomaTikz, Feynman uses stronger LM (GPT-4) than Llama 2 used in AutomaTikz. This makes it hard to understand the fundamental advantage of Feynman compared to baselines such as AutomaTikz.
- **Missing related work.** DiagrammerGPT (COLM 2024; https://arxiv.org/abs/2310.12128) seems to be highly relevant, as it also uses an LLM for diagram layout planning (w/ iterative refinement) and uses diffusion or different vector graphic tools (e.g., PowerPoint/Inkscape/illustrator) for rendering. They also provide a training annotation and benchmark in the scientific domain - AI2D caption. I highly encourage authors to add discussion and comparison to this work.
- **Weak analysis of the correctness of the generated diagrams.** The highest performance among its ablation study (in Table 5) does not mean that the generated diagrams are correct. The authors should provide a more rigorous analysis of the correctness of the diagrams generated by Feynman (e.g., human evaluation). It is unclear whether Diagramma can be a meaningful evaluation benchmark of MLLMs without such analysis.
We create a scalable vision-language data generation pipeline that produces knowledge-infused images.
@inproceedings{
anonymous2024feynman,
title={Feynman: Knowledge-Infused Diagramming Agent for Scaling Visual Reasoning Data},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=jNmsuEE4Gf},
note={under review}
}
Visual reasoning is an essential ability of state-of-the-art multi-modal AI systems. Improving these systems requires high-quality vision-language data at scale. Despite the abundance of internet image and text data, knowledge-rich and well-aligned image-text pairs are rare. In this paper, we present a scalable data generation pipeline built with our diagramming agent, **Feynman**. To create diagrams, Feynman first enumerates domain-specific knowledge components ("ideas") and performs code planning based on the ideas. Given the plan, Feynman translates ideas into simple declarative programs and iterates to receives feedback and visually refine diagrams. Finally, the declarative programs are rendered by the Penrose diagramming system. The optimization-based rendering of Penrose preserves the visual semantics while injecting fresh randomness into the layout, thereby producing diagrams with visual consistency and diversity. As a result, Feynman can author diagrams along with grounded captions with very little cost and time. Using Feynman, we synthesized a dataset with more than 100$k$ well-aligned diagram-caption pairs. We also curate a visual-language benchmark, **Diagramma**, from freshly generated data.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Vision-Language Dataset Synthetic Data Visual Reasoning Benchmark
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/ecff4497f3c892daa75c51cdd799d1fcdb4a2a34.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Feynman: Knowledge-Infused Diagramming Agent for Scaling Visual Reasoning Data
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission