Tag: conference papers

Posted on

CCG Papers at ACL and ICML

The 64th Annual Meeting of the Association for Computational Linguistics (ACL) is underway in San Diego, followed next week by the Forty-Third International Conference on Machine Learning (ICML) in Seoul, South Korea. We’re excited to share the works that will be presented by the group and our collaborating authors. You can find links to our ACL and ICML 2026 papers below!

Papers from CCG collaborations:

Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say “I Don’t Know” (ACL Findings)

This paper examines how different prompting strategies influence the reliability of large language models (LLMs) in closed-book question answering, with a focus on their tendency to produce confident hallucinations. It compares model responses to direct questions with those generated using question decomposition: a multi-step reasoning plan for solving the original question. The results show that discrepancies between these two prompting approaches are highly indicative of hallucinations, offering a stronger signal than prior methods such as self-consistency. 

Dhruv Madhwal and Lyuxin David Zhang and Dan Roth and Tomer Wolfson and Vivek Gupta, Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say “I Don’t Know” ACL Findings (2026)

Sunday, July 5, 14:00-15:30 — Coronado

Toward Robust Evaluation for Multilingual Grammatical Error Correction: Can Large Language Models Replace Human References? (ACL)

Prior research has shown that using a <i>closest-gold reference</i> – i.e., a human reference generated with respect to the system output rather than the original text – yields more accurate performance estimates. This paper introduces <i>an automated method for generating closest-gold references</i> by prompting a large language model (LLM) with system outputs. We find that performance scores computed using automatic closest-gold references correlate well with human closest-golds, whereas standard reference-based evaluations show weak or no correlation. Building on this insight, we use both fixed human references and closest-gold references generated by Claude and Llama to compare the performance of supervised models and GPT-4 across 14 benchmarks spanning 12 languages.

Alla Rozovskaya and Dan Roth, Toward Robust Evaluation for Multilingual Grammatical Error Correction: Can Large Language Models Replace Human References? ACL (2026)

Monday, July 6, 09:00-10:30 — Grand Hall & Coronado, Poster Session D

MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents (TACL, presented at ACL)

The landscape of AI-powered web search is being transformed: evolving from static large language models to LLM-powered “agents” which surface hard-to-find information through multiple rounds of reasoning and retrieval. Evaluating these agents requires measuring their ability to formulate and execute long-horizon plans that demand integrating information from dozens or even hundreds of web pages. Yet existing benchmarks are largely limited to tasks involving only a handful of sources, failing to capture the recall-intensive challenges found in many real-world applications.
This paper introduces MoNaCo, the first benchmark that evaluates AI agents’ ability to answer long-horizon questions that span dozens—and in some cases hundreds—of web pages. MoNaCo consists of 1,315 human-authored questions, each paired with ground-truth answers, supporting evidence, and gold-standard reasoning chains. We evaluate 15 frontier LLMs and identify substantial performance gaps: the top-performing model, o3, achieves a perfect score on just 38.7% of the benchmark.
Using MoNaCo, we show that deep reasoning across large document collections remains an open challenge for current AI agents, and discuss promising directions for bridging this gap.

Tomer Wolfson and Harsh Trivedi and Mor Geva and Yoav Goldberg and Dan Roth and Tushar Khot and Ashish Sabharwal and Reut Tsarfaty, MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents TACL (2026)

Monday, July 6, 09:00-10:30 — Regatta, Oral Session D: Question Answering 1

REaR: Retrieve, Expand and Refine for Effective Multitable Retrieval (ACL)

This paper introduces REAR (Retrieve, Expand and Refine), a three-stage, LLM-free framework for efficient, high-fidelity multi-table retrieval. REAR (i) retrieves query-aligned tables, (ii) expands these with structurally joinable tables via fast, precomputed column-embedding comparisons, and (iii) refines them by pruning noisy or weakly related candidates.Despite being LLM-free, it delivers performance competitive with state-of-the-art LLM-augmented retrieval systems while achieving much lower latency and cost.

Rishita Agarwal and Himanshu Singhal and Peter Baile Chen and Manan Roy Choudhury and Dan Roth and Vivek Gupta, REaR: Retrieve, Expand and Refine for Effective Multitable Retrieval ACL (2026)

Tuesday, July7, 09:00-10:30 — Grand Hall

Is Code Better Than Language for Algorithmic Reasoning? (ICML)

This paper provides a methodology to compare two types of problem solving strategies with LLMs, 1) directly use CoT reasoning 2) call a tool (e.g. external runtime) by writing a piece of code. The methodology shows how tool calling is dominant through a theoretical perspective, and backs up the intuition with experimental results. 

Terry Tong and Yu Feng and Surbhi Goel and Dan Roth, Is Code Better Than Language for Algorithmic Reasoning? ICML (2026)

A selection of further papers co-authored by Dan Roth

ToolScope: Enhancing LLM Agent Tool Use through Tool Merging and Context-Aware Filtering (ACL)

Large language model (LLM) agents rely on external tools to solve complex tasks, but real-world toolsets often contain noisy and overlapping tool options. This paper introduces ToolScope, which analyzes external toolsets to reduce redundancy and to select the most relevant tools for each query. 

Marianne Menglin Liu and Daniel Garcia and Fjona Parllaku and Vikas Upadhyay and Syed Fahad Allam Shah and Dan Roth, ToolScope: Enhancing LLM Agent Tool Use through Tool Merging and Context-Aware Filtering ACL (2026)

SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks (ACL)

Large language models (LLMs) have achieved strong performance on natural language to SQL (NL2SQL) benchmarks, yet their reported accuracy may be inflated by contamination from benchmark queries or structurally similar patterns seen during training. We introduce SPENCE (Syntactic Probing and Evaluation of NL2SQL Contamination Effects), a controlled syntactic probing framework for detecting and quantifying such contamination.

Mohammadtaher Safarzadeh and Hitesh Laxmichand Patel and Afshin Orojlooyjadid and Graham Horwood and Dan Roth, SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks ACL (2026)

When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias (ACL)

This paper addresses a fundamental flaw in current VLM-as-a-Judge systems, namely informativeness bias, the tendency to blindly favor more informative answers while giving little attention to the image. To improve reliability, we propose BIRCH, which compares candidate answers against a self-generated anchor that corrects or verifies image details, shifting the judge’s focus back to image content and balancing informativeness with correctness.

Xiaohan Zou and Roshan Sridhar and Mohammadtaher Safarzadeh and Dan Roth, When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias ACL (2026) 

JTPRO: A Joint Tool–Prompt Reflective Optimization Framework for Language Agents (ACL Findings)

Large language model (LLM) agents augmented with external tools often struggle as number of tools grow large and become domain-specific. This paper introduces Joint Tool-Prompt Reflective Optimization (JTPRO), a framework for improving tool-calling reliability in trace-supervised settings by iteratively using rollout-driven reflection to co-optimize global instructions and per-tool schema/argument descriptions for accurate tool selection and argument instantiation in large tool inventories.

Sandip Ghoshal and Anshul Mittal and Jyotika Singh and Miguel Ballesteros and Weiyi Sun and Fang Tu and Shailender Singh and Yassine Benajiba and Fahad Shah and Sujeeth Bharadwaj and Sujith Ravi and Dan Roth, JTPRO: A Joint Tool-Prompt Reflective Optimization Framework for Language Agents ACL Findings (2026)

MT-OSC: Path for LLMs that Get Lost in Multi-Turn Conversation (ACL Findings)

The routine approach of appending full chat history to prompts for multi-turn interactions rapidly exhausts LLM context windows, leading to diminishing returns as conversations extend. This paper introduces MT-OSC, a One-off Sequential Condensation framework that efficiently and automatically condenses chat history in the background without disrupting the user experience. The results establish MT-OSC as a scalable solution for multi-turn chats, enabling richer context within constrained input spaces, reducing latency and operational cost, while balancing performance.

Jyotika Singh and Fang Tu and Miguel Ballesteros and Weiyi Sun and Sandip Ghoshal and Michelle Yuan and Yassine Benajiba and Sujith Ravi and Dan Roth, MT-OSC: Path for LLMs that Get Lost in Multi-Turn Conversation ACL Findings (2026)

The Crowded Embedding Space: A Mean-Field Mechanism for Emergent Marginalization in Retrieval-Augmented Agents (ICML)

This paper introduces a formal framework to analyze how  goal collisions in dense retrieval by RAG agents induce fundamental performance limits and emergent fairness issues inherent to spatial crowding. In our static analysis, we demonstrate that for a fixed embedding space, a phase transition occurs where minority user goals suffer a catastrophic collapse in performance as the density of majority goals increases. We then extend this to a dynamic model and derive a non-linear Fokker-Planck equation that governs the evolution of document embeddings as the agent updates them to maximize retrieval accuracy.  The results provide a theoretical foundation for understanding a critical grounding failure mode in retrieval-augmented agents.

Shwan Ashrafi and Dan Roth, The Crowded Embedding Space: A Mean-Field Mechanism for Emergent Marginalization in Retrieval-Augmented Agents ICML (2026)

Posted on

CCG Papers at EMNLP and *SEM 2025

The Conference on Empirical Methods in Natural Language Processing (EMNLP), celebrating its 30th anniversary, is underway in Suzhou, China, and the co-located 14th Joint Conference on Lexical and Computational Semantics (*SEM) kicks off tomorrow. We’re excited to share the works that will be presented by the group and our collaborating authors. You can find links to our EMNLP and *SEM 2025 papers below!

[Some summaries generated by AI.]

*SEM 2025 (11/08-11/09, co-located with EMNLP)

Cross-lingual Extractive Question Answering with Unanswerable Questions

This paper extends cross-lingual extractive QA, where models need to find answers in passages written in a language different from the question, to cases where no answer exists within the given context. Proposing two novel datasets and performing extensive experiments, we analyze the strengths and weaknesses of different language models and training strategies for this task as well as the effect of the language identity on the performance. 

Yuval Gorodissky and Elior Sulem and Dan Roth, Cross-lingual Extractive Question Answering with Unanswerable Questions *SEM (2025)

Sunday, November 9, 09:00-10:30 local time — Virtual Poster Session 5

EMNLP 2025 (11/5-11/9)

Conflicts in Texts: Data, Implication, Challenges

This survey examines how conflicting information arises in NLP—from inconsistencies in natural texts, to annotation disagreements, to model-level hallucinations and knowledge clashes—and unifies them under a common framework. We analyze their implications, highlight mitigation strategies, and chart future directions for building conflict-aware NLP systems that can better reconcile contradictions.

Siyi Liu and Dan Roth, Conflicts in Texts: Data, Implication, Challenges EMNLP-Findings (2025)

AutoCT: Automating Interpretable Clinical Trial Prediction with LLM Agents

The paper introduces AutoCT, a framework that integrates large language models with classical machine learning to predict clinical trial outcomes. By autonomously proposing, constructing, and refining features through Monte Carlo Tree Search, AutoCT achieves competitive performance with state-of-the-art methods while maintaining interpretability and reducing reliance on human input.

Fengze Liu and Haoyu Wang and Joonhyuk Cho and Dan Roth and Andrew Lo, AutoCT: Automating Interpretable Clinical Trial Prediction with LLM Agents EMNLP (2025)

LogiCoL: Logically-Informed Contrastive Learning for Set-based Dense Retrieval

LogiCoL is a new training method for dense retrieval models that helps them better handle queries containing logical connectives (like “and,” “or,” “not”) by incorporating logical constraints directly into the learning process through contrastive learning. The authors demonstrate that this approach improves both retrieval accuracy and logical consistency when retrieving sets of Wikipedia entities that must satisfy complex logical relationships specified in queries.

Yanzhen Shen and Sihao Chen and Xueqiang Xu and Yunyi Zhang and Chaitanya Malaviya and Dan Roth, LogiCoL: Logically-Informed Contrastive Learning for Set-based Dense Retrieval EMNLP (2025)

Evaluating NL2SQL via SQL2NL

We propose a novel schema-aligned paraphrasing framework that leverages SQL-to-NL (SQL2NL) to automatically generate semantically equivalent, lexically diverse queries while maintaining alignment with the original schema and intent. This enables the first targeted evaluation of NL2SQL robustness to linguistic variation in isolation-distinct from prior work that primarily investigates ambiguity or schema perturbations.

Mohammadtaher Safarzadeh and Afshin Oroojlooyjadid and Dan Roth, Evaluating NL2SQL via SQL2NL EMNLP-Findings (2025)

Weaver: Interweaving SQL and LLM for Table Reasoning

Weaver, a novel framework for question answering over tables where questions are complex and require reasoning, when you have unstructured columns, or need to combine logical operations with semantic understanding. Weaver dynamically combines SQL for retrieving and aggregating, together with the semantic reasoning capabilities of LLMs.

Rohit Khoja and Devanshu Gupta and Yanjie Fu and Dan Roth and Vivek Gupta, Weaver: Interweaving SQL and LLM for Table Reasoning EMNLP (2025)

Rethinking LLM Uncertainty: A Multi-Agent Approach to Estimating Black-Box Model Uncertainty

We show in this paper that LLMs are inconsistent on factual questions and can easily be biased by the additional context in the input, and we propose a multi-agent approach to improve answer consistency and uncertainty quantification.

Yu Feng and Phu Mon Htut and Zheng Qi and Wei Xiao and Manuel Mager and Nikolaos Pappas and Kishaloy Halder and Yang Li and Yassine Benajiba and Dan Roth, Rethinking LLM Uncertainty: A Multi-Agent Approach to Estimating Black-Box Model Uncertainty EMNLP-Findings (2025)

Posted on

CCG Papers at ICML 2025

The Forty-Second International Conference on Machine Learning (ICML) starts off today in Vancouver, Canada. We’re excited to share the works that will be presented by the group and our collaborating authors. You can find links to our ICML 2025 papers below!

ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding 

This work presents ReFocus, a framework that equips multimodal LLMs with the ability to generate “visual thoughts” by performing visual editing on the input image through code, shifting and refining their visual focuses. With experiments on a wide range of structured image understanding tasks involving tables and charts, we present an in-depth analysis of the effects of different visual edits and the ways ReFocus can edit the input image until an answer is reached.

Xingyu Fu, Minqian Liu, Zhengyuan Yang, John Corring, Yijuan Lu, Jianwei Yang, Dan Roth, Dinei Florencio, and Cha Zhang, ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding ICML (2025).

Tuesday, July 15, 4:30 pm PDT — Poster Session 2 West

GIVE: Structured Reasoning of Large Language Models with Knowledge Graph Inspired Veracity Extrapolation

This paper introduces GIVE (Graph Inspired Veracity Extrapolation), a framework that enhances Large Language Models’ reasoning on knowledge-intensive tasks by combining their internal knowledge with minimal, structured external knowledge graph cues. The method enables smaller LLMs to achieve or exceed the performance of larger models on complex scientific reasoning tasks, while also reducing hallucination.

Jiashu He, Mingyu Ma, Jinxuan Fan, Dan Roth, Wei Wang, and Alejandro Ribeiro, GIVE: Structured Reasoning of Large Language Models with Knowledge Graph Inspired Veracity Extrapolation ICML (2025).

Thursday, July 17, 4:30 pm PDT — Poster Session 6 East

Includes some text generated by AI.

Posted on

CCG Papers at ICLR 2025

The 2025 International Conference on Learning Representations (ICLR) is happening this week in Singapore. We’re excited to share the work that will be presented and published by the group and our collaborating authors. You can find links to our ICLR 2025 papers below!

BIRD: A Trustworthy Bayesian Inference Framework for Large Language Models

Language models often struggle with reliable and consistent decisions under uncertainty, partially because they can’t reliably estimate the probability of each choice. We propose BIRD, a framework that significantly enhances LLM decision making under uncertainty. BIRD leverages LLMs for world modeling and constructs a Bayesian network using LLM-generated variables, enabling interpretable and trustworthy probability estimates.

Yu Feng, Ben Zhou, Weidong Lin, and Dan Roth, BIRD: A Trustworthy Bayesian Inference Framework for Large Language Models ICLR (2025)

MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding

We introduce MuirBench, a comprehensive benchmark that focuses on robust multi-image understanding capabilities of multimodal LLMs. MuirBench consists of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that involve 10 categories of multi-image relations (e.g., multiview, temporal relations). For reliable assessment, each standard instance in MuirBench is paired with an unanswerable variant that has minimal semantic differences.

Fei Wang, Xingyu Fu, James Y. Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, Tianyi Lorena Yan, Wenjie Jacky Mo, Hsiang-Hui Liu, Pan Lu, Chunyuan Li, Chaowei Xiao, Kai-Wei Chang, Dan Roth, Sheng Zhang, Hoifung Poon, and Muhao Chen, MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding ICLR (2025)

Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge

The effectiveness of automatic evaluation of generative models is typically measured by comparing labels generated via automation with labels by humans using correlation metrics. In this paper, we show how *relying on a single aggregate correlation score* can obscure fundamental differences between human labels and those from automatic evaluation, including LLM-as-a-Judge. Based on these findings, we propose stratifying data by human label uncertainty to provide a more robust analysis and introduce a new metric to better measure the effectiveness of automatic evaluations.

Aparna Elangovan, Lei Xu, Jongwoo Ko, Mahsa Elyasi, Ling Liu, Sravan Bodapati, and Dan Roth, Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge ICLR (2025)

Posted on

CCG Papers at NAACL 2025

Red NAACL logo with text "NAACL 2025"

The 2025 Conference of the Nations of the Americas* Chapter of the Association for Computational Linguistics (NAACL) will be coming up in May, in Albuquerque, New Mexico. We’re excited to share the work that will be presented and published by the group and our collaborating authors. You can find links to our NAACL 2025 papers below!

Updated with presentation times and dates (note: not chronological, even though the first one is on April 30; so is the last).
___
* Here’s a blog post from the NAACL organizing committee about their name change.

On Reference (In-)Determinacy in Natural Language Inference

This paper introduces RefNLI, a diagnostic benchmark for identifying reference ambiguity in Natural Language Inference examples. We provide insight into how the reference determinacy assumption (the assumption that premise and hypothesis refer to the same context) impacts the downstream utility of NLI models, and discover that the existence of reference ambiguity in NLI examples can in part explain the inherent human disagreements in NLI.

Sihao Chen, Chaitanya Malaviya, Alex Fabrikant, Hagai Taitelbaum, Tal Schuster, Senaka Buthpitiya, and Dan Roth, On Reference (In-)Determinacy in Natural Language Inference. NAACL Findings (2025)

Wednesday, April 30, Session C: Oral/Poster 2, Hall 3, 14:00-15:30.

Towards Long Context Hallucination Detection

This paper studies hallucination detection where the context length is long (>=512 tokens). We construct a dataset to evaluate the task and propose a method to approach it.

Siyi Liu, Kishaloy Halder, Zheng Qi, Wei Xiao, Nikolaos Pappas, Phu Mon Htut, Neha Anna John, Yassine Benajiba, and Dan Roth, Towards Long Context Hallucination Detection. NAACL Findings (2025)


Friday, May 2, Session J: Oral/Poster 7, Hall 3, 9:00-10:30.

Open Domain Question Answering with Conflicting Contexts

We study open domain question answering when there is conflicting evidence presented on the web. We demonstrate that by finetuning LLMs to explain their answers, we can introduce richer information into their training that guides them through the process of reasoning with conflicting contexts.

Siyi Liu, Qiang Ning, Kishaloy Halder, Zheng Qi, Wei Xiao, Phu Mon Htut, Yi Zhang, Neha Anna John, Bonan Min, Yassine Benajiba, and Dan Roth, Open Domain Question Answering with Conflicting Contexts. NAACL Findings (2025)

Friday, May 2, Session J: Oral/Poster 7, Hall 3, 9:00-10:30.

H-STAR: LLM-driven Hybrid SQL-Text Adaptive Reasoning on Tables

Existing approaches to tabular reasoning employ either textual reasoning, which excels in semantic interpretation but struggles with mathematical operations, or symbolic reasoning, which handles computations well but lacks semantic understanding. H-STAR, a novel method introduced in this paper, integrates text comprehension with SQL-like logic to effectively answer queries from structured tables.

Nikhil Abhyankar, Vivek Gupta, Dan Roth, and Chandan K. Reddy, H-STAR: LLM-driven Hybrid SQL-Text Adaptive Reasoning on Tables. NAACL (2025)

Thursday, May 1, Session H: Oral/Poster 5, Hall 3, 14:00-15:30.

TRANSIENTTABLES: Evaluating LLMs’ Reasoning on Temporally Evolving Semi-structured Tables

The ability to reason over time allows us to identify future steps and to understand the effects of decisions on our lives. However, large language models are typically trained on static datasets, limiting their ability to perform effective temporal reasoning. To assess the temporal reasoning capabilities of LLMs, this paper presents the TRANSIENTTABLES dataset, with questions derived from over 14,000 tables, spanning multiple time periods.

Abhilash Shankarampeta, Harsh Mahajan, Tushar Kataria, Dan Roth, and Vivek Gupta, TRANSIENTTABLES: Evaluating LLMs’ Reasoning on Temporally Evolving Semi-structured Tables. NAACL (2025)

Friday, May 2, Session K: Oral/Poster 8, 11:00-12:30 (presentation 11:45).

Enhancing Temporal Understanding in LLMs for Semi-structured Tables

We introduce the C.L.E.A.R. prompting framework and auxiliary cross-format training to enhance LLM performance in temporal tabular reasoning. Our findings demonstrate that our method improves evidence-based reasoning across various models. Additionally, our experimental results reveal that indirect supervision with auxiliary unstructured data (TRAM) substantially boosts model performance.

Irwin Deng, Kushagra Dixit, Vivek Gupta, and Dan Roth, Enhancing Temporal Understanding in LLMs for Semi-structured Tables. NAACL Findings (2025)

Thursday, May 1, Session H: Oral/Poster 5, Hall 3, 14:00-15:30.

MAPWise: Vision-Language Models for Advanced Map Queries

Revised maps with counterfactual data, forcing the models to rely on the maps themselves.

This paper introduces a new benchmark for evaluating vision-language models (VLMs) on choropleth map question answering, featuring diverse maps and question types across multiple geographic regions. Evaluation of several VLMs reveals significant performance gaps, highlighting the need for further research in this area and providing a resource for future model development.

Srija Mukhopadhyay, Abhishek Rajgaria, Prerana Khatiwada, Manish Shrivastava, Dan Roth, and Vivek Gupta, MAPWise: Evaluating Vision-Language Models for Advanced Map Queries. NAACL (2025)

Thursday, May 1, Session F: Oral/Poster 4, Ballroom B, 10:30-12:00 (presentation 10:45).

NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Models

NTSEBENCH is a Vision-Language Model benchmark dataset with 2,728 questions and 4,642 images from India’s NTSE exam, to evaluate VLMs on cognitive multimodal reasoning.

Pranshu Pandya, Vatsal Gupta, Agney Talwar, Tushar Kataria, Dan Roth, and Vivek Gupta, NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Models. NAACL Findings (2025)

Friday, May 2, Session K: Oral/Poster 8, Hall 3, 11:00-12:30.

Aligning to Constraints for Data-Efficient Language Model Customization

This paper proposes ACT (Aligning to ConsTraints), a unified and efficient Language Model customization framework using automatic constraint verifiers to provide supervision signals for adapting models to downstream tasks.

Fei Wang, Chao Shang, Sarthak Jain, Shuai Wang, Qiang Ning, Bonan Min, Vittorio Castelli, Yassine Benajiba, and Dan Roth, Aligning to Constraints for Data-Efficient Language Model Customization. NAACL Findings (2025)

Tuesday, May 6, Gather Session 2, online, 15:00-16:30.

Leveraging LLM For Synchronizing Information Across Multilingual Tables

We explored the application of large language models (LLMs) for multilingual information synchronization, focusing on improving the accuracy and coherence of updates to Wikipedia tables in low-resource languages.

Siddharth Khincha, Tushar Kataria, Ankita Anand, Dan Roth, and Vivek Gupta, Leveraging LLM For Synchronizing Information Across Multilingual Tables. NAACL (2025)

Wednesday, April 30, Session D: Oral/Poster 3, Ballroom B, 16:00-17:30 (Presentation 17.00).

Posted on

CCG papers at ACL 2024

The 2024 Annual Meeting of the Association for Computational Linguistics (ACL) is underway in Bangkok! We’re excited to share the work that’s being presented and published from CCG and our collaborating authors. You can find links to our ACL papers below!

ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models

In this position paper, we argue that human evaluation of generative large language models (LLMs) should be a multidisciplinary undertaking that draws upon insights from disciplines such as user experience research and human behavioral psychology to ensure that the experimental design and results are reliable. To design an effective human evaluation system in the age of generative NLP, we propose the ConSiDERS-The-Human evaluation framework, consisting of 6 pillars — Consistency, Scoring Criteria, Differentiating, User Experience, Responsible, and Scalability.

Aparna Elangovan, Ling Liu, Lei Xu, Sravan Bodapati, and Dan Roth, ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models (ACL 2024)

Winner of the Outstanding Paper Award at the ACL2024 Workshop on Knowledgeable LMs
Is Table Retrieval a Solved Problem? Exploring Join-Aware Multi-Table Retrieval

Retrieving relevant tables containing the necessary information to accurately answer a given question over tables is critical to open-domain question-answering (QA) systems. However, many questions require retrieving multiple tables and joining them through a join plan that cannot be discerned from the user query itself. In this paper, we introduce a method that uncovers useful join relations for any query and database during table retrieval. We use a novel re-ranking method formulated as a mixed-integer program that considers not only table-query relevance but also table-table relevance that requires inferring join relationships.

Peter Baile Chen, Yi Zhang, and Dan Roth, Is Table Retrieval a Solved Problem? Exploring Join-Aware Multi-Table Retrieval (ACL 2024)

FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts

This paper introduces FlowVQA to overcome the shortcomings of existing visual question answering benchmarks in visual grounding and spatial reasoning. FlowVQA features 2,272 flowchart images and 22,413 question-answer pairs to evaluate tasks like information localization, decision-making, and logical reasoning. The evaluation of various multimodal models highlights FlowVQA’s potential to advance multimodal modelling and improve visual and logical reasoning skills.

Shubhankar Singh, Purvi Chaurasia, Yerram Varun, Pranshu Pandya, Vatsal Gupta, Vivek Gupta, and Dan Roth, FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts (ACL-Findings 2024)

Evaluating LLMs’ Mathematical Reasoning in Financial Document Question Answering

In this paper, we assess LLM robustness in complex mathematical reasoning with financial tabular datasets, revealing that LLMs struggle with increasing table and question complexity, especially with multiple arithmetic steps and hierarchical tables. The new EEDP technique enhances LLM accuracy and robustness by improving domain knowledge, extracting relevant information, decomposing complex questions, and performing separate calculations.

Pragya Srivastava, Manuj Malik, Vivek Gupta, Tanuja Ganu, and Dan Roth, Evaluating LLMs’ Mathematical Reasoning in Financial Document Question Answering (ACL-Findings 2024)

Posted on

CCG Papers at NAACL 2024

The 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) is coming up, from June 16-21 in Mexico City. We’re excited to share the work that’s being presented and published from CCG and our collaborating authors. You can find links to our NAACL papers below!

Sub-Sentence Encoder: Contrastive Learning of Propositional Semantic Representations

Figure 1: Given a proposition in a sentence (represented by a highlighted subset of tokens), the sub-sentence encoder produces a contextual embedding for the meaning of the proposition.

Text embeddings typically produce one embedding for the entire text sequence, but what if the text is long and says many things? Check out Sub-Sentence Encoder — A contextual text encoder model that learns to embed individual pieces of meaning in text. 

Sihao Chen, Hongming Zhang, Tong Chen, Ben Zhou, Wenhao Yu, Dian Yu, Baolin Peng, Hongwei Wang, Dan Roth, and Dong Yu, Sub-Sentence Encoder: Contrastive Learning of Propositional Semantic Representations (NAACL 2024).

SocREval: Large Language Models with the Socratic Method for
Reference-Free Reasoning Evaluation

The paper develops a reference-free evaluation of reasoning abilities of LLMs, surpassing the abilities of GPT-4 to evaluate reasoning abilities.

Hangfeng He, Hongming Zhang, and Dan Roth, SocREval: Large Language Models with the Socratic Method for Reference-free Reasoning Evaluation (NAACL Findings 2024).

What if you said that differently?: How Explanation Formats Affect Human Feedback Efficacy and User Perception

In this work, we study the effect of intermediate explanation formats on the effectiveness of human feedback for correcting QA model responses. Further, we investigate the properties of explanations which allow users to understand and trust responses.

Chaitanya Malaviya, Subin Lee, Dan Roth, and Mark Yatskar, What if you said that differently? How Explanation Formats Affect Human Feedback Efficacy and User Perception (NAACL 2024).

ExpertQA: Expert-Curated Questions and Attributed Answers

This work conducts expert evaluation of responses to domain-specific questions according to various axes of attribution and factuality. Based on our evaluation, we present ExpertQA, a high-quality long-form QA dataset with 2177 questions spanning 32 fields, along with verified answers and attributions for claims in the answers.

Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth, ExpertQA: Expert-Curated Questions and Attributed Answers (NAACL 2024).

ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented
Large Language Models via Transferable Adversarial Attacks

Figure 1: An example of how original evidence is edited by ReEval. The question is “When did Athens emerge as the wealthiest Greek city state?” The desirable answers, respectively, for answer swapping (Category 1) and context enriching (Category 2) are “the early 4th century BCE” and “the late 6th century BCE”. ChatGPT answers are next to the emoji.

Despite remarkable advancements in mitigating hallucinations in large language models (LLMs) by retrieval augmentation, it remains challenging to measure the reliability of LLMs using static question-answering (QA) data. Inspired by adversarial machine learning, we investigate the feasibility of automatically perturbing existing static benchmarks for dynamic evaluation. Specifically, this paper presents ReEval, an LLM-based framework using prompt chaining to perturb the original evidence and generate new test cases for evaluating the LLMs’ reliability in using new evidence for answering. 

Xiaodong Yu, Hao Cheng, Xiaodong Liu, Dan Roth, Jianfeng Gao, ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks (NAACL Findings 2024).