EnsemHalDet: Robust VLM Hallucination Detection via Ensemble of Internal State Detectors
Abstract
Vision-Language Models (VLMs) excel at multimodal tasks, but they remain vulnerable to hallucinations that are factually incorrect or ungrounded in the input image. Recent work suggests that hallucination detection using internal representations is more efficient and accurate than approaches that rely solely on model outputs. However, existing internal-representation-based methods typically rely on a single representation or detector, limiting their ability to capture diverse hallucination signals. In this paper, we propose EnsemHalDet, an ensemble-based hallucination detection framework that leverages multiple internal representations of VLMs, including attention outputs and hidden states. EnsemHalDet trains independent detectors for each representation and combines them through ensemble learning. Experimental results across multiple VQA datasets and VLMs show that EnsemHalDet consistently outperforms prior methods and single-detector models in terms of AUC. These results demonstrate that ensembling diverse internal signals significantly improves robustness in multimodal hallucination detection.
Type
Publication
ACL Student Research Workshop (SRW) 2026
Note
Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.
Note
Create your slides in Markdown - click the Slides button to check out the example.
Add the publication’s full text or supplementary notes here. You can use rich formatting such as including code, math, and images.
Authors
I am currently a researcher at the Japan AI Safety Institute and the University of Electro-Communications under Satoshi Hara. I received my Master’s degree from the same university under the supervision of Kei Harada.
My research interests lie in evaluation and alignment for trustworthy AI and AI Safety. At Japan AISI, I evaluate Japanese LLMs, with a particular focus on robustness to misinformation and disinformation, as well as hallucination generation. At UEC, I work on robust hallucination detection using model-internal representations, with the goal of improving methods for detecting model misalignment.
My research interests lie in evaluation and alignment for trustworthy AI and AI Safety. At Japan AISI, I evaluate Japanese LLMs, with a particular focus on robustness to misinformation and disinformation, as well as hallucination generation. At UEC, I work on robust hallucination detection using model-internal representations, with the goal of improving methods for detecting model misalignment.
Authors
Authors