Ensembling Multiple Hallucination Detectors Trained on VLLM Internal Representations
Aug 1, 2025·
,,·
1 min read
Yuto Nakamizo
Equal contribution
Ryuhei Miyazato
Equal contribution
,Hikaru Tanabe
Ryuta Yamakura
Kiori Hatanaka
Abstract
This paper presents the 5th place solution by our team, y3h2, for the Meta CRAG-MM Challenge at KDD Cup 2025. The CRAG-MM benchmark is a visual question answering (VQA) dataset focused on factual questions about images, including egocentric images. The competition was contested based on VQA accuracy, as judged by an LLM-based automatic evaluator. Since incorrect answers result in negative scores, our strategy focused on reducing hallucinations from the internal representations of the VLM. Specifically, we trained logistic regression-based hallucination detection models using both the hidden_state and the outputs of specific attention heads. We then employed an ensemble of these models. As a result, while our method sacrificed some correct answers, it significantly reduced hallucinations and allowed us to place among the top entries on the final leaderboard.
Type
Publication
In Proceedings of the KDD Cup Workshop at the 31st SIGKDD Conference on Knowledge Discovery and Data Mining (KDD2025)
Note
Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.
Note
Create your slides in Markdown - click the Slides button to check out the example.
Add the publication’s full text or supplementary notes here. You can use rich formatting such as including code, math, and images.
Authors
Authors
I am currently a researcher at the Japan AI Safety Institute and the University of Electro-Communications under Satoshi Hara. I received my Master’s degree from the same university under the supervision of Kei Harada.
My research interests lie in evaluation and alignment for trustworthy AI and AI Safety. At Japan AISI, I evaluate Japanese LLMs, with a particular focus on robustness to misinformation and disinformation, as well as hallucination generation. At UEC, I work on robust hallucination detection using model-internal representations, with the goal of improving methods for detecting model misalignment.
My research interests lie in evaluation and alignment for trustworthy AI and AI Safety. At Japan AISI, I evaluate Japanese LLMs, with a particular focus on robustness to misinformation and disinformation, as well as hallucination generation. At UEC, I work on robust hallucination detection using model-internal representations, with the goal of improving methods for detecting model misalignment.
Authors
Authors
Authors