ACL 2026 — Datasets & Benchmarks | Rajat Kanti Bhattacharjee

Datasets & Benchmarks

340 papers Links not yet available — ACL proceedings pending on ACL Anthology.

Benchmarking LLM’s Capability in Reasoning over Conflicting Web References
MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Long-tail Knowledge
SenseRel: A Sense-Level Benchmark for Denotational and Connotational Meaning Relations
Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark
VALU: A Benchmark for Video Anomaly Temporal Localization and Understanding at Multiple Semantic Levels
Is EEG-to-Text Feasible in Real-World Scenarios? An In-Depth Analysis Using a Neuropsychology-Inspired Benchmark
SAGE: A Search-AuGmented Evaluation of Large Language Models on Free‑Form QA
A Functionality-Grounded Benchmark for Evaluating Web Agents in E-commerce Domains
Psychological Steering in LLMs: An Evaluation of Effectiveness and Trustworthiness
AraVQA: Building a New Arabic Factoid Visual Question Answering Dataset from Wikipedia
SATQuest: A Verifier for Logical Reasoning Evaluation and Reinforcement Fine-Tuning of LLMs
MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generation
Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examiner
RealTalk-CN: A Realistic Chinese Speech Task-Oriented Dialogue Benchmark with Cross-Modal Analysis
Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders
KG-MuLQA: A Framework for KG-based Multi-Level QA Extraction and Long-Context LLM Evaluation
A Comprehensive Survey of Process Reward Models: Data Generation, Model Construction, and Usage
SDiaReward: Modeling and Benchmarking Spoken Dialogue Rewards with Modality and Colloquialness
Looking at Radiology Report Generation through a Causal Lens: A Survey
Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods
CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation
Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding
Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation
Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models
MedMCP-Calc: Benchmarking LLMs for Realistic Medical Calculator Scenarios via MCP Integration
Rethinking Meeting Effectiveness: A Benchmark and Framework for Temporal Fine-grained Automatic Meeting Effectiveness Evaluation
CEDAR: A Chinese Evaluation Dataset for Computational Argumentation
MSEarth: A Multimodal Benchmark for Earth Science Phenomenon Discovery with MLLMs
GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidance
Par-ITA: Benchmarking Seq2Seq and LLMs on a Human-Supervised Parallel Corpus for Italian Hyperpartisan Neutralization
ROSE: An Intent-Centered Evaluation Metric for NL2SQL
Demystifying Uncertainty in LLMs: Active Calibration between Concepts and Human Evaluations
OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding
MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments
AutoTaskEval: Towards Domain-Specific and Fine-Grained Evaluation for LLMs
UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents
Annotating Dimensions of Social Perception in Text: A Sentence-Level Dataset of Warmth and Competence
Efficient Self-Evaluation for Diffusion Language Models via Sequence Regeneration
MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agents
HoWToBench: Holistic Evaluation for LLM’s Capability in Human-level Writing using Tree of Writing
DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints
AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts
arXiv2Table: Toward Realistic Benchmarking and Evaluation for LLM-Based Literature-Review Table Generation
Responsible Evaluation of AI for Mental Health
SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluation
Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs
Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents
A Survey of Large Language Model-Based Search Agents
AgentCoMa: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarios
Reasoning with OmniThought: A Large CoT Dataset with Verbosity and Cognitive Difficulty Annotations
Beyond Monolingual Assumptions: A Survey on Code-Switched NLP in the Era of Large Language Models across Modalities
From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level
RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository
CityCube: Benchmarking Cross-view Spatial Reasoning on Vision-Language Models in Urban Environments
“Newspaper Eat” Means “Not Tasty”: A Taxonomy and Benchmark for Coded Language in Real-World Chinese Online Reviews
One Battle After Another: Probing LLMs’ Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework
Benchmarking and Learning Real-World Customer Service Dialogue
DREAM: Deep Research Evaluation with Agentic Metrics
ACIArena: Toward Unified Evaluation for Agent Cascading Injection
PLAWBENCH: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice
Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environments
Beyond Word Boundaries: A Hebrew Coreference Benchmark and an Evaluation Protocol for Morphologically Complex Text
All That Glisters Is Not Gold: A Benchmark for Reference-Free Counterfactual Financial Misinformation Detection
Musical Score Understanding Benchmark: Evaluating Large Language Models’ Comprehension of Complete Musical Scores
Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectification
Limited Linguistic Diversity in Embodied AI Datasets
CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval
WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics
LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection
BhashaSutra: A Task-Centric Unified Survey of Indian NLP Datasets, Corpora, and Resources
Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation
Author-in-the-Loop Response Generation and Evaluation: Integrating Author Expertise and Intent in Responses to Peer Review
Reward Modeling for Scientific Writing Evaluation
Decoding Scientific Experimental Images: The SPUR Benchmark for Perception, Understanding, and Reasoning
Stereotype Bias in a Bilingual Setting: A Culturally Grounded Evaluation in Kazakhstan
FinCall-Surprise: A Large Scale Multi-modal Benchmark for Earning Surprise Prediction
FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Models
From Charts to Code: A Hierarchical Benchmark for Multimodal Models
Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation
Omni-RewardBench: Toward a Comprehensive Evaluation of Generative Reward Models Across Modalities
Iterative Dual-Model Alignment for Story Evaluation
FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning
YIELD: A Large-Scale Dataset and Evaluation Framework for Information Elicitation Agents
Rethinking Evaluation for LLM Hallucination Detection: A Desiderata, A New RAG-based Benchmark, New Insights
PRIME: A Process-Outcome Alignment Benchmark for Verifiable Reasoning in Mathematics and Engineering
AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMs
LiveCLKTBench: Towards Reliable Evaluation of Cross-Lingual Knowledge Transfer in Multilingual LLMs
VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models
ReproEvalCard: A Reporting Standard for Reproducible Evaluation of LLM Pipelines
BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks
MediEval: A Unified Medical Benchmark for Patient-Contextual and Knowledge-Grounded Reasoning in LLMs
AlignUSER: Human-Aligned LLM Agents via World Models for Recommender System Evaluation
ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios
Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasets
Quantifying Metric and Model Agreement in Bias Evaluation of Large Language Models
MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Application
Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection
From Query to Counsel: Structured Reasoning with a Multi-Agent Framework and Dataset for Legal Consultation
Subject-level Inference for Realistic Text Anonymization Evaluation
VOYAGER: A Training Free Approach for Generating Diverse Datasets using LLMs
CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval
Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation
Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models
Tracing Logit Trajectories Across Layer Depth: Dataset-Level Explainability for Language Models
AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following
PRiSM: Benchmarking Phone Realization in Speech Models
Your Reasoning Benchmark May Not Test Reasoning: Revealing Perception Bottleneck in Abstract Reasoning Benchmarks
Persona-Grounded Safety Evaluation of AI Companions in Multi-Turn Conversations
Beyond Value Benchmarks: Measuring Value-Structure Alignment in Large Language Models via Symmetric Q-Sorts
EvolvR: Self-Evolving Pairwise Reasoning for Story Evaluation to Enhance Generation
CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks
When Bigger Isn’t Better: A Comprehensive Fairness Evaluation of Political Bias in Multi-News Summarisation
Region-Grounded Report Generation for 3D Medical Imaging: A Fine-Grained Dataset and Graph-Enhanced Framework
Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems
MTAVG-Bench: A Diagnostic Benchmark for Multi-Talker Dialogue-Centric Audio-Video Generation
SubTokenTest: A Practical Benchmark for Real-World Sub-token Understanding
Benchmarking and Enabling Efficient Chinese Medical Retrieval via Asymmetric Encoders
League of LLMs: A Benchmark-Free Paradigm for Mutual Evaluation of Large Language Models
RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora
SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks
Beyond the Last Frame: Process-aware Evaluation for Generative Video Reasoning
HSCodeComp: A Realistic and Expert-level Agent Benchmark for Hierarchical Rule Application
Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues
USB: A COMPREHENSIVE AND UNIFIED SAFETY EVALUATION BENCHMARK FOR MULTIMODAL LARGE LANGUAGE MODELS
LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models
AEGIS: A Holistic Benchmark for Evaluating Forensic Analysis of AI-Generated Academic Images
The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models
LexRel: Benchmarking Legal Relation Extraction for Chinese Civil Cases
MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluation
Community-Aware Assessment of Social Textual Engagement and Resonance: A Human-Centric Perspective on User-Generated Content Evaluation
EduBench: A Comprehensive Benchmarking Dataset for Evaluating Large Language Models in Diverse Educational Scenarios
MMAC: A Multilingual, Multimodal Alignment Framework for Cultural Grounding Evaluation
Benchmarking Large Vision-Language Models on CFMME: A Comprehensive Chinese Financial Multimodal Evaluation Dataset
Translation or Recitation? Calibrating Evaluation Scores for Machine Translation of Extremely Low-Resource Languages
Benchmarking Web Agent Safety under E-commerce Deceptive Interfaces
BrowseComp-Plus: A Fair and Disentangled Evaluation Benchmark for Deep Search Agents
LitVISTA: A Benchmark for Narrative Orchestration in Literary Text
Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Math
InsLogicBench: An Argumentation Logic Grounded Benchmark for Complex Insurance Claims Adjudication
A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions
Automated Creativity Evaluation of Language Models Across Open-Ended Tasks
Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling
From Synthesis to Clinical Assistance: A Strategy-Aware Agent Framework for Autism Intervention based on Real Clinical Dataset
MMTutorBench: The First Multimodal Benchmark for AI Math Tutoring
Detoxification for LLM: From Dataset Itself
LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models
IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation
Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation
SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios
ADAPT: Benchmarking Commonsense Planning under Unspecified Affordance Constraints
Truth or Sophistry? LoFa: A Benchmark for LLM Robustness Against Logical Fallacies
JanusMM: A Benchmark for Self-Deprecation Understanding in Real-World Multimodal Conversations
Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Models
When Efficiency Meets Safety: A Benchmark Security Analysis of KV Cache Compression in Large Language Models
GeoLaux: A Benchmark for Evaluating MLLMs’ Geometry Performance on Long-Step Problems Requiring Auxiliary Lines
IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation
Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents
CUB: Benchmarking Context Utilisation Techniques for Language Models
MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
Chart-MRAG: Benchmarking Multimodal Retrieval Augmented Generation on Chart-based Documents
Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages
CogEvolve: A Multimodal Benchmark for Evaluating Relational Reasoning in Semantic Extension
Reliable Evaluation Protocol for Low-Precision Retrieval
Inflated Excellence or True Performance? Rethinking Medical Diagnostic Benchmarks with Dynamic Evaluation
TrendFact: A Benchmark Towards Hotspot Perception in Automatic Fact-Checking
When More Words Say Less: Decoupling Length and Specificity in Image Description Evaluation
SGVEF-LOOP: Coverage-Guided Progressive Topological Exploration and Fact-Grounded Metamorphic Evaluation for MCP Agents
Benchmarking Egocentric Clinical Intent Understanding Capability for Medical Multimodal Large Language Models
ReCoQA: A Benchmark for Tool-Augmented and Multi-Step Reasoning in Real Estate Question and Answering
LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social Simulations
DR-Arena: an Automated Evaluation Framework for Deep Research Agents
Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models
Beyond Single View: A Comprehensive Benchmark for Medical Multimodal Large Language Models on Multi-Image Understanding
Neo-Classic: A Benchmark for Evaluating Linguistic-Aesthetic Reasoning in Classical Chinese Poetry
RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension
CompTab: A Comprehensive Benchmark for Real-World TableQA with Complex Reasoning and Irregular Tables
Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards
Benchmarking Deflection and Hallucination in Large Vision-Language Models
Bloom-Eval: A Hierarchical Evaluation Benchmark for Automatic Survey Generation Based on Bloom’s Taxonomy
FactVerse: A Benchmark for Factual Consistency in Interleaved Image–Text Generation
EIFFEL: a novel benchmark to measure bias of English heavy training on French idiomatic expressions
It’s High Time: A Survey of Temporal Question Answering
Beyond Itinerary Planning—A Real-World Benchmark for Multi-Turn and Tool-Using Travel Tasks
Persona-E²: A Human-Grounded Dataset for Personality-Shaped Emotional Responses to Textual Events
Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation
LongTutor: Benchmarking Large Language Models for Long-term Personalized Tutoring
GOLEMcoref: A Multilingual Coreference Dataset of Fiction
MPR-GUI: Benchmarking and Enhancing Multilingual Perception and Reasoning in GUI Agents
Controllable Contamination Detection for Reliable LLM Evaluation with Statistical Guarantees
KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital Companions
PersonalityDBench: A Dataset for Personality Disorders - from Modeling to Controlled Generation
Generating Attribution Reports for Manipulated Facial Images: A Dataset and Baseline
ArgGenBench: Benchmarking the Complex Controlled Argument Generation Capability of Large Language Models
QuantumQA: Enhancing Scientific Reasoning via Physics-Consistent Dataset and Verification-Aware Reinforcement Learning
OSCBench: Benchmarking Object State Change in Text-to-Video Generation
Evaluation Pitfalls and Challenges in Multimedia Event Extraction
Attribution, Citation, and Quotation: A Survey of Evidence-based Text Generation with Large Language Models
Can AI-Generated Persuasion Be Detected? Persuaficial Benchmark and AI vs. Human Linguistic Differences
SwissGov-RSD: A Human-annotated, Cross-lingual Benchmark for Token-level Recognition of Semantic Differences Between Related Documents
TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systems
How Controllable Are Large Language Models? A Unified Evaluation across Behavioral Granularities
RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation
A Survey of Inductive Reasoning for Large Language Models
CogToM: A Comprehensive Theory of Mind Benchmark inspired by Human Cognition for Large Language Models
CITE: Benchmarking Heterogeneous Text-Attributed Graph Models
SAFO: Stable Adaptive Fairness Optimization for LLM-Based Social Survey Simulation
GQLBench: A Large-Scale Cross-Domain, Cross-Dialect Benchmark for NL2GQL
Alexandria: A Multi-Domain Dialectal Arabic Machine Translation Dataset for Culturally Inclusive and Linguistically Diverse LLMs
Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future
MetaBench: A Multi-task Benchmark for Assessing LLMs in Metabolomics
RSMeM: Knowledge-Enhanced Memory Evolution for Remote Sensing Agents with Systematic Evaluation
The GDN-CC Dataset: Automatic Corpus Clarification for AI-enhanced Democratic Citizen Consultations
PIArena: A Platform for Prompt Injection Evaluation
ChemReason-Bench: Benchmarking Large Language Models for Procedural Reasoning in Experimental Chemistry
CL²GEC: A Multi-Discipline Benchmark for Continual Learning in Chinese Literature Grammatical Error Correction
CloneMem: Benchmarking Long-Term Memory for AI Clones
VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis
MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation
ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models
MMSciCode: Real-world Evaluation of Multilingual Multi-Discipline Scientific Research Coding
GameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agents
DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality
ClaimDB: A Fact Verification Benchmark over Large Structured Data
PluRule: A Benchmark for Moderating Pluralistic Communities on Social Media
CASS: Nvidia to AMD Transpilation with Data, Models, and Benchmark
SAHM: A Benchmark for Arabic Financial and Shari’ah-Compliant Reasoning
Towards Intrinsic Interpretability of Large Language Models: A Survey of Design Principles and Architectures
ScholaWrite: A Dataset of End-to-End Scholarly Writing
E2EDev: Benchmarking Large Language Models in End-to-End Software Development Task
Business as Rulesual: A Benchmark and Framework for Business Rule Flow Modeling with LLMs
DiningBench: A Hierarchical Multi-view Benchmark for Perception and Reasoning in the Dietary Domain
DefGen-Bench: A Benchmark for Chinese Criminal Defence Opinion Generation in LegalAI
SMART: Evaluating LLMs’ Mathematical Reasoning via a Human Cognitive Process-Inspired Benchmark
Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models
EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation
Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction
GenomeQA: Benchmarking General Large Language Models for Genome Sequence Understanding
LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient
LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluation
SAD: A Large-Scale Strategic Argumentative Dialogue Dataset
Pub-LawBench: Public-Oriented Benchmarking for LegalAI
Diversity in Unity, Theory in Practice: Hierarchical Multitask Benchmarks for Chinese Minority Languages
Test of Time: Rethinking Temporal Signal of Benchmark Contamination
IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation
VisAidMath: Benchmarking Visual-Aided Mathematical Reasoning
MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks
Beyond Static Benchmarks: Synthesizing Harmful Content via Persona-based Simulation for Robust Evaluation
Omni-I2C: A Holistic Benchmark for High-Fidelity Image-to-Code Generation
TaxPraBen: A Scalable Benchmark for Structured Evaluation of LLMs in Chinese Real-World Tax Practice
Tears or Cheers? Benchmarking LLMs via Culturally Elicited Distinct Affective Responses
HumanLLM: Benchmarking and Improving LLM Anthropomorphism via Human Cognitive Patterns
v-HUB: A Benchmark for Video Humor Understanding from Vision and Sound
G-IdiomAlign: A Gloss-Pivoted Benchmark for Cross-Lingual Idiom Alignment
HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models
EmoS: A High‑Fidelity Multimodal Benchmark for Fine-grained Streaming Emotional Understanding
HowToNarrate: A General-Domain Benchmark for Synchronized Video Narration with External Knowledge
Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment
False Friends or Cognates? A Cross-lingual Semantic Ambiguity Evaluation for Galician, Portuguese and Spanish
A Survey of Deep Learning for Geometry Problem Solving
NoisyCausal: A Benchmark for Evaluating Causal Reasoning Under Structured Noise
MedEinst: Benchmarking the Einstellung Effect in Medical LLMs through Counterfactual Differential Diagnosis
Market-Bench: Benchmarking Large Language Models on Economic and Trade Competition
Benchmarking Post-Training Quantization of Large Language Models under Microscaling Floating Point Formats
GroupToM-Bench: Benchmarking Group Theory of Mind and Nonlinear Social Emergence in MLLMs
Revisiting Evaluation of Question Answering Systems in Low-Resource Indic Languages: Bridging Human and Metric Alignment
UrbanGeoEval: A City-Scale Benchmark for Evaluating Large Language Models in Geospatial Reasoning
GROKE: Vision-Free Navigation Instruction Evaluation via Graph Reasoning on OpenStreetMap
MirrorQA: Benchmarking Multimodal LLMs on Mirror-Orientation Reasoning
DimABSA: Building Multilingual and Multidomain Datasets for Dimensional Aspect-Based Sentiment Analysis
GeoRC: A Benchmark for Geolocation Reasoning Chains
M³-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering
Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents
HarDBench: A Benchmark for Draft-Based Co-Authoring Jailbreak Attacks for Safe Human–LLM Collaborative Writing
CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Language
ProactiveEval: A Unified Evaluation Framework for Proactive Dialogue Agents
Multimodal Safety Evaluation in Generative Agent Social Simulations
Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation
Survey Response Generation: Generating Closed-Ended Survey Responses In-Silico with Large Language Models
Sign-Language Datasets at Scale: A Comprehensive Survey on Resources, Benchmarks, and Annotation Standards
LLM4Cell: Taxonomy and Evaluation of LLM and Agentic Models for Single-Cell Biology
RealChart2Code: Bridging the Gap in Real-World Chart-to-Code Generation via Multi-Task Evaluation
VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking
A Survey of Reasoning-Intensive Retrieval: Progress and Challenges
LOGICAL-COMMONSENSEQA: A Benchmark for Logical Commonsense Reasoning
PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation
Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams
A Survey of Multimodal Mathematical Reasoning: From Perception, Alignment to Reasoning
HypoEval: Hypothesis-Guided Evaluation for Natural Language Generation
GCA Framework: A GCC Countries–Grounded Dataset and Agentic Pipeline for Climate Decision Support
Explain the Synth: Interpretable Evaluation of LLM Data Synthesis
Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models
Simple Agents, Biased Judges: Efficient Multi-Party Dialogue Generation & The Evaluation Gap
Deep Research with Open-Domain Evaluation and Multi-Stage Guardrails for Safety
ServImage: An Image Generation and Editing Benchmark from Real-world Commercial Imaging Services
PodBench: A Comprehensive Benchmark for Instruction-Aware Audio-Oriented Podcast Script Generation
A Survey of Large Language Models for Text-Guided Molecular Discovery: From Molecule Generation to Optimization
OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation
CT-FineBench: A Diagnostic Fidelity Benchmark for Fine-Grained Evaluation of CT Report Generation
POLYCHARTQA: Benchmarking Large Vision-Language Models with Multilingual Chart Question Answering
ReTRE: Benchmarking LLM Transfer Robustness with Structure-Preserving Variants
Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding
Automatic and Reliable Evaluation for Academic Caption-to-Figure Generation with LMMs
AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
INFACT: A Diagnostic Benchmark for Induced Faithfulness and Factuality Hallucinations in Video-LLMs
DAC-Bench: A Decision-Aware Benchmark for Compositional Mobile GUI Tasks
Benchmarking Fine-Grained Error Detection in Multimodal Reasoning
When Benchmarks Leak: Inference-Time Decontamination for LLMs
MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Models
PolitNuggets: Benchmarking Agentic Discovery of Long-Tail Political Facts
Revealing the Seen, Imagining the Beyond: A Survey of Image-Grounded Chain-of-Thought Reasoning in Multimodal LLMs
OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Models
TwiUSD: A Benchmark Dataset and Structure-Aware LLM Framework for User Stance Detection
Perception, Understanding and Reasoning: A Multimodal Benchmark for Video Fake News Detection
JARVIS or Ultron? A Survey on the Safety and Security Threats of Computer-Using Agents
SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation
SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models
LLM-Based Multi-Agent Systems for Clinical Workflows: A Survey of AI Hospitals
Mind’s Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs
Interpretable Coreference Resolution Evaluation Using Explicit Semantics
When High Accuracy Hides Poor Calibration: Rethinking Confidence Evaluation in Transformer-Based Text Classification with Balanced Brier Score
CxMP: A Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Models
From Recognition to Reasoning: Benchmarking and Enhancing MLLMs on Real-World Receipt Document Understanding
BioTool: A Comprehensive Tool-Calling Dataset for Enhancing Biomedical Capabilities of Large Language Models
Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing
MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Events
TEA-Bench: A Systematic Benchmarking of Tool-enhanced Emotional Support Dialogue Agent
From Words to Pixels: A Comprehensive Survey on Large Language Models in Visual Segmentation
TabReX: Tabular Referenceless eXplainable Evaluation
From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLP
Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations
Toward Robust Evaluation for Multilingual Grammatical Error Correction: Can Large Language Models Replace Human References?
MathSight: A Benchmark Exploring Have Vision-Language Models Really Seen in University-Level Mathematical Reasoning?
REVEALER: Reinforcement-Guided Visual Reasoning for Element-Level Text-Image Alignment Evaluation
A Multilingual Social Bias Benchmark Incorporating Thinking Processes
NaturalGAIA: A Verifiable Benchmark and Hierarchical Framework for Long-Horizon GUI Tasks
Macaron: Controlled, Human-Written Benchmark for Multilingual and Multicultural Reasoning via Template-Filling
Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failures
JurisBench: A Deep Benchmark for Assessing Large Language Models in Professional Legal Practice