CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs

Abstract

Recently, efficient Multimodal Large Language Models (MLLMs) have gained significant attention as a solution to their high computational complexity, making them more practical for real-world applications. In this regard, the knowledge distillation (KD) approach has emerged as a promising alternative, which transfers the rich visual and linguistic knowledge from a larger model (teacher) to a smaller model (student).

However, we observe that existing KD methods struggle to effectively distill the teacher MLLM's rich visual perception abilities to the student, a challenge that has been largely overlooked in previous studies. Through a systematic analysis, we identify visual attention misalignment between student and teacher as the main cause of this issue.

Based on this insight, we propose CompoDistill, a novel KD framework that explicitly aligns the student's visual attention with that of the teacher to enhance the student's visual perception abilities. Our extensive experiments show that CompoDistill significantly improves performance on compositional reasoning tasks while maintaining strong performance on visual question answering tasks.

Motivation: Visual Attention Misalignment

Do existing KD methods distill visual perception?

While existing KD methods for MLLMs have shown effectiveness in VQA tasks that primarily require visual recognition, they struggle with compositional reasoning tasks that demand fine-grained visual perception — such as attribute binding and spatial relationship understanding.

Compositional Reasoning Performance Gap

Figure 1. Existing KD methods fail to transfer visual perception abilities, leading to degraded compositional reasoning performance.

What causes the performance gap?

Through systematic analysis, we identify visual attention misalignment as the root cause. The student model fails to attend to the same visual regions as the teacher, resulting in degraded visual perception despite successful knowledge transfer for simpler recognition tasks.

Visual Attention Misalignment

Figure 2. Attention maps reveal misalignment between student and teacher, highlighting the root cause of degraded visual perception.

Why Is Visual Perception Not Distilled Properly?

We conduct a systematic three-step analysis to identify the root cause of the failure in distilling visual perception abilities from the teacher to the student MLLM.

Step 1 — Identifying the Key Factor

Teacher-Student Attention Similarity over Visual Tokens

We examine the layer-wise attention similarity between the student and teacher over both visual and text tokens. On VQA tasks, where KD succeeds, the student's visual attention is highly aligned with the teacher's in the visual understanding layers. However, on CR tasks, where KD fails, no such alignment exists — revealing visual attention misalignment as the key factor.

Layer-wise Attention Similarity

Figure 3. (a) Attention of the answer token over visual and text tokens. (b–e) Layer-wise teacher-student and teacher-SFT attention similarities over visual tokens (b, c) and text tokens (d, e). Visual attention similarity in the visual understanding layers is the key differentiator.

Step 2 — Direct Impact on Performance

Attention Similarity ↔ Answer Probability

We quantify the direct relationship between visual attention similarity and downstream performance. By grouping instances by their student-teacher attention similarity, we observe a clear positive correlation: higher attention similarity leads to higher answer token probability, providing direct evidence that attention alignment drives performance.

Figure 4. Higher student-teacher attention similarity correlates with higher answer token probability, providing direct evidence that attention alignment drives performance.

Step 3 — A Simple Solution

Replacing Student's Attention with Teacher's

To validate our hypothesis, we perform a direct intervention: substituting the student's visual attention with the average of the teacher's and student's attention during inference. This yields consistent performance gains on compositional reasoning sub-tasks (Swap, Replace, Add), confirming that the student benefits from incorporating the teacher's attention patterns.

Figure 5. Replacing the student's visual attention with the teacher's yields consistent performance gains on compositional reasoning sub-tasks, validating our hypothesis.

CompoDistill Framework

CompoDistill Framework Overview

Figure 6. Our framework explicitly aligns the student's visual attention with the teacher's through an attention distillation loss, enhancing the student's visual perception abilities for compositional reasoning.

CompoDistill introduces an attention distillation mechanism that explicitly aligns the student MLLM's visual attention patterns with those of the teacher. By supervising how the student attends to visual tokens, our framework ensures that fine-grained visual perception abilities — critical for compositional reasoning — are effectively transferred during distillation.

Main Results

CompoDistill is evaluated on both visual question answering (VQA) and compositional reasoning (CR) benchmarks. The student (1.8B) and teacher (4B) both use SigLIP vision encoder and Qwen 1.5 LLM.

Comparison with KD Methods and Other MLLMs

Relational Hallucination Evaluation

Beyond compositional reasoning, CompoDistill also mitigates relational hallucinations by accurately understanding object relationships — achieving performance nearly on par with the teacher on R-Bench and Reefknot benchmarks.

Qualitative Comparison

‹

›

Figure 7. Qualitative examples demonstrating improved visual attention alignment and compositional reasoning performance. CompoDistill's student attends to the correct regions, closely matching the teacher's attention patterns.

Ablation Studies

Effect of Core Components (VAT & TAF)

Fine-grained Analysis on VAT Module

Citation

If you find this work helpful, please consider citing:

@inproceedings{kim2026compodistill,
  title={CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs},
  author={Jiwan Kim and Kibum Kim and Sangwoo Seo and Chanyoung Park},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2026},
  url={https://arxiv.org/abs/2510.12184}
}