Differentiable Retrieval-Guided Multimodal Reasoning for Knowledge-Intensive Visual Question Understanding

1. Introduction

Visual Question Answering (VQA) has emerged as a fundamental problem at the intersection of Computer Vision, Natural Language Processing, and Knowledge Representation. The task requires an intelligent system to generate a correct and contextually grounded answer to a natural-language question about an image. Early VQA models primarily focused on direct visual-textual alignment, leveraging deep multimodal encoders to integrate features from the visual and linguistic modalities. However, as the field matured, it became evident that many questions posed to visual systems require reasoning that extends far beyond perceptual content alone.

In conventional datasets, such as VQA-v2, the visual evidence is often sufficient to determine the correct answer. In contrast, knowledge-based VQA (KB-VQA) and Outside-Knowledge VQA (OK-VQA) [29] push the boundaries of this paradigm by introducing questions that require external factual or commonsense information. For instance, answering a question like “Why is the man holding an umbrella on a sunny day?” demands background understanding of cultural practices or contextual reasoning about potential scenarios, rather than visual pattern recognition alone. This highlights a critical limitation in current multimodal systems: their dependency on the information available within the image-text pair, without access to external repositories of human knowledge.

Figure 1. The motivation of this work.

Figure 1. The motivation of this work.

To address this, recent works have turned to retrieval-based augmentation, wherein external documents or facts are retrieved from large knowledge bases such as Wikipedia [8,26,38]. Dense Passage Retrieval (DPR) [17] has proven particularly effective for mapping questions and documents into shared vector spaces, allowing for semantic retrieval of relevant passages. However, existing systems typically train the retriever and the answer generator independently, creating a disconnect between knowledge selection and downstream reasoning. This leads to inefficiencies: retrievers may prioritize documents that merely contain surface-level overlaps with query terms, while generators must process redundant or misleading content. Prior research attempted to alleviate this issue through heuristic pseudo-labels [32], yet such approaches introduce noisy supervision and may not reflect true semantic relevance.

Decoupling retrieval and generation causes several systemic weaknesses. Firstly, the retriever cannot be optimized to improve the answer generation objective directly. Secondly, the generator may become over-reliant on spurious correlations between retrieved content and ground-truth answers. Finally, such pipelines require a large number of retrieved documents (often 50 or more [8,10]), resulting in substantial computational overhead and potential degradation of answer quality. These factors motivate the need for an end-to-end differentiable framework that can jointly refine both retrieval and reasoning.

Retrieval-Augmented Generation (RAG) [21] first demonstrated that joint optimization between a retriever and generator could improve open-domain QA performance. Yet, when directly applied to multimodal settings such as OK-VQA, RAG exhibits critical weaknesses. Our preliminary studies confirmed that RAG’s loss formulation often misattributes credit to irrelevant documents when visual cues alone suffice for question answering. Moreover, many OK-VQA samples require compositional reasoning between visual context and external text—a challenge RAG was not designed to handle. Consequently, naive adaptations of RAG yield inconsistent performance across varying question types and fail to leverage the full potential of visual grounding.

Motivated by these limitations, we propose KnowSight, a retrieval-in-the-loop reasoning framework that explicitly integrates knowledge retrieval into the visual question answering process. KnowSight’s central innovation lies in its unified objective function, which couples retrieval confidence with generative correctness, ensuring that documents contributing to accurate reasoning are reinforced during training. This approach eliminates the need for heuristic pseudo-relevance labels and enables dynamic feedback between retrieval and generation modules. The retriever learns to focus on documents that truly aid reasoning, while the generator refines its textual grounding on verified contextual evidence.

Beyond its architectural novelty, KnowSight introduces a diagnostic methodology for analyzing retrieval–generation interactions, measuring both relevance precision and reasoning attribution. Experiments on OK-VQA and extended benchmarks show that KnowSight consistently outperforms traditional two-step models and RAG-style systems in accuracy, efficiency, and knowledge utilization. It requires significantly fewer retrieved documents—often as few as 5 to 10 per question—without sacrificing quality. This efficiency makes it practical for large-scale multimodal applications and continuous learning scenarios.

The implications of KnowSight extend beyond OK-VQA. As multimodal large language models (MLLMs) increasingly shape the landscape of AI, the ability to ground reasoning on dynamically retrievable knowledge will be crucial for building interpretable, reliable, and adaptive systems. KnowSight offers a generalizable framework that could be extended to visual dialogue, multimodal commonsense inference, and even agentic reasoning systems capable of autonomous knowledge acquisition. Ultimately, this work moves towards a long-term vision of open-world multimodal understanding, where models can see, reason, and learn continuously by drawing from an ever-evolving body of human knowledge.

In summary, our contributions are threefold:

• We introduce KnowSight, a unified joint-learning framework for retrieval-augmented visual reasoning that integrates external knowledge dynamically during answer generation.
• We propose a diagnostic perspective on retrieval–generation synergy, offering fine-grained insights into how knowledge retrieval influences multimodal reasoning outcomes.
• We demonstrate state-of-the-art performance on OK-VQA and related benchmarks, achieving significant efficiency improvements in both retrieval quality and computational cost.

This work thus bridges a key gap between perception and knowledge, contributing to the broader agenda of developing multimodal systems capable of grounded, explainable, and knowledge-driven reasoning in complex visual environments.

2. Related Work

2.1. Open-Domain Question Answering Paradigms

Open-domain Question Answering (QA) represents a long-standing research problem at the intersection of information retrieval and natural language understanding. The core objective is to enable models to answer arbitrary natural language questions by accessing diverse sources of world knowledge. In early QA systems, knowledge was primarily derived from static textual corpora or manually curated knowledge bases. However, recent progress in large-scale pre-training [3,12,21,35] and differentiable retrieval frameworks [13,20] has shifted the paradigm toward dynamic, context-aware retrieval augmentation.

Parametric QA models such as T5 [35] embed a vast amount of factual knowledge within model parameters, allowing “closed-book” inference. Yet, their knowledge coverage remains bounded by the training corpus, making them less reliable for long-tail facts. In contrast, retrieval-augmented approaches explicitly query large document stores or web-scale knowledge repositories to extract supporting evidence dynamically. This design paradigm increases interpretability and adaptability while maintaining scalability. Pioneering works such as REALM [12] and ORQA [20] integrated neural retrievers with downstream QA objectives, establishing differentiable retrieval as a key milestone. Later, RAG [21] extended this idea into a generative setting, allowing the model to produce free-form textual answers while marginalizing over retrieved passages.

Despite their impressive success, these systems often face inherent trade-offs between retrieval precision, model interpretability, and computational overhead. Subsequent research has aimed to refine retriever–generator synergy by jointly training both components and improving the quality of contextualized representations [6,27,36]. Our work builds on this line of research by extending differentiable retrieval to multimodal contexts, where visual semantics must guide the selection of textual evidence.

2.2. Evolution of Multimodal Visual Question Answering

Visual Question Answering (VQA) tasks require systems to reason jointly over image content and textual queries. Early VQA methods relied on simple concatenation or attention-based fusion of CNN and RNN representations [15,37,47], which limited their capacity for complex cross-modal reasoning. With the advent of large-scale multimodal pre-training, transformer-based architectures [4,14,24,39,42,48] became dominant, unifying visual and textual representations within shared embedding spaces.

Recent advancements such as UNITER [4] and VinVL [48] demonstrate that learning fine-grained object–text alignments through large-scale pre-training significantly enhances zero-shot and few-shot generalization. Moreover, the emergence of multimodal LLMs (e.g., Flamingo, BLIP-2) has shown that injecting structured knowledge into vision-language reasoning pipelines improves factual grounding and commonsense understanding. Yet, most conventional VQA models are “closed-world” learners—they rely solely on image-text data seen during training and lack the capacity to access evolving external knowledge.

2.3. Knowledge-Based VQA and External Knowledge Integration

Knowledge-based VQA (KB-VQA) extends traditional VQA by incorporating structured and unstructured knowledge resources to enrich reasoning. Structured resources include ConceptNet [38], Freebase, or other Knowledge Graphs (KGs), which provide explicit relations between entities. Unstructured sources such as Wikipedia or the web offer contextual evidence and open-domain factual grounding. Hybrid frameworks [8,9,28,44] attempt to bridge these modalities by aligning symbolic and textual knowledge representations.

ConceptBERT [9] first demonstrated that embedding graph nodes into a transformer attention framework enables semantic alignment between entities and visual elements. KRISP [28] extended this idea with a symbolic knowledge module that links image regions and question tokens with KG entities, enhancing interpretability. MAVEx [44] further broadened the scope by leveraging multiple heterogeneous sources (Google Images, ConceptNet, and Wikipedia) to validate candidate answers. Meanwhile, TRiG [8] and KAT [10] integrated dense retrieval and large language models (T5, GPT-3) into VQA, showcasing the benefits of language-grounded retrieval for multimodal reasoning.

Despite these efforts, current KB-VQA systems often treat knowledge retrieval and reasoning as disjoint stages. This leads to suboptimal alignment between retrieved evidence and the generative reasoning process. Moreover, reliance on pseudo-relevance labels [26,32] can introduce noise, as retrieved passages containing the answer text are not guaranteed to be semantically relevant. Our approach addresses these challenges through a joint optimization paradigm that integrates retrieval relevance directly into the reasoning objective.

2.4. Retrieval-Augmented and Differentiable Learning Frameworks

The concept of retrieval augmentation represents a pivotal shift from static knowledge embedding to dynamic reasoning architectures. Unlike fixed-parameter LMs, retrieval-augmented frameworks dynamically update their knowledge context during inference, improving adaptability to new facts without retraining. Differentiable retrieval techniques, as in REALM [12], enable backpropagation through the retrieval step, aligning document selection with downstream objectives. Building upon these advances, KnowSight introduces a multimodal differentiable retrieval mechanism that conditions knowledge selection on both linguistic and visual cues. The retrieved evidence is then used to guide answer generation, and gradients are propagated to refine the retriever’s parameters based on the generation loss, achieving true end-to-end optimization.

2.5. Recent Multimodal Reasoning and Foundation Models

With the advent of multimodal foundation models, large-scale pre-training has begun to bridge the gap between vision, language, and external knowledge. Unified architectures such as BLIP-2, OFA, and Flamingo have introduced adapter-based visual reasoning frameworks that extend LLMs’ text-based commonsense reasoning to visual inputs. However, these models often rely on fixed pre-training corpora and cannot efficiently update their knowledge when encountering novel scenarios. The integration of retrieval-based reasoning mechanisms—such as those explored in KnowSight—provides a pathway toward lifelong multimodal learning, enabling dynamic access to continuously evolving world knowledge.

In addition, recent efforts explore integrating symbolic reasoning modules and neural retrieval components to improve factual consistency and interpretability. For instance, approaches leveraging neuro-symbolic alignment [30] or graph-aware prompting structures [22,44] have shown potential for mitigating hallucination in multimodal systems. Nevertheless, most methods remain limited by their inability to propagate reasoning signals back into retrieval decisions, a gap that KnowSight explicitly addresses through joint gradient optimization.

In summary, our work builds upon three converging research lines: (1) retrieval-augmented open-domain QA [12,21], (2) multimodal VQA [4,24,39], and (3) knowledge-based reasoning with structured and unstructured resources [9,28,44]. While each line contributes unique strengths, their integration into a single unified framework remains underexplored. By coupling visual-semantic encoding with differentiable retrieval and end-to-end generative reasoning, our model introduces a principled framework for knowledge-grounded visual understanding. This closes a critical gap between perception and reasoning, providing the foundation for a new generation of interpretable and adaptive multimodal systems.

Figure 2. Overview of the proposed KnowSight framework for knowledge-intensive visual question answering. The system transforms multimodal inputs—an image and question—into a structured textual query through Visual & OCR Textualization, enabling a Bi-Encoder Dense Retriever to fetch relevant knowledge passages from an external corpus. A Cross-Encoder Re-Ranker refines the retrieved results, which are then fed into an Answer Generator for evidence-grounded prediction. The entire pipeline is optimized jointly under multiple objectives, including retrieval calibration, consistency alignment, and entropy regularization, ensuring end-to-end coupling between retrieval relevance and generative answerability.

Figure 2. Overview of the proposed KnowSight framework for knowledge-intensive visual question answering. The system transforms multimodal inputs—an image and question—into a structured textual query through Visual & OCR Textualization, enabling a Bi-Encoder Dense Retriever to fetch relevant knowledge passages from an external corpus. A Cross-Encoder Re-Ranker refines the retrieved results, which are then fed into an Answer Generator for evidence-grounded prediction. The entire pipeline is optimized jointly under multiple objectives, including retrieval calibration, consistency alignment, and entropy regularization, ensuring end-to-end coupling between retrieval relevance and generative answerability.

3. Methodology

We introduce KnowSight, a retrieval-in-the-loop framework for knowledge-intensive Visual Question Answering that couples multimodal query formation, weakly-supervised dense retrieval, and end-to-end joint optimization of retrieval and generation. Compared with prior RA-VQA designs, KnowSight extends (i) the vision→language reframing to include richer structured textualized cues, (ii) the retriever with calibration, re-ranking, and uncertainty-aware scoring, and (iii) the learning objective with auxiliary consistency and contrastive terms that align document relevance with answerability.

3.1. Problem Setup and Notation

Let an image–question pair be $(I,q)$ with a set of human answers $\mathcal{S}={\left\{s_i\right\}}_{i=1}^m$. An external corpus is $\mathcal{Z}={\left\{z_j\right\}}_{j=1}^{N^d}$ (e.g., Wikipedia passages). KnowSight forms a textual query x from $(I,q)$ (Section 3.2); a query encoder ${\mathcal{F}}_q$ and a document encoder ${\mathcal{F}}_d$ (both Transformer-like) map x and z to ${\mathbb{R}}^h$:

$$\mathbf{q}={\mathcal{F}}_q\left(x\right)\in {\mathbb{R}}^h,\mathbf{d}={\mathcal{F}}_d\left(z\right)\in {\mathbb{R}}^h.$$

(1)

A base similarity score is $r(x,z)={\mathbf{q}}^{\top }\mathbf{d}$. Given a retrieved set ${\left\{z_k\right\}}_{k=1}^K$, a generator ${\mathcal{G}}_{\varphi }$ (e.g., T5) produces an answer y conditioned on $(x,z_k)$. KnowSight learns parameters $\theta$ (retriever) and $\varphi$ (generator) jointly.

3.2. Multimodal-to-Text Reframing

Prior work shows that language-only Transformers can be repurposed for VQA once images are transformed into textual evidence [26,45]. KnowSight follows this paradigm but enriches the textualization with object-attribute phrases, relations, OCR strings, and a global caption.

• Object and Attribute Serialization.

Using VinVL [48], we detect objects $\left\{o_i\right\}$ and attributes $\left\{a_{i,j}\right\}$ with detection and attribute confidences ${\tau }_o=0.8$ and ${\tau }_a=0.6$, respectively. We serialize them into normalized phrases $\left[\mathtt{OBJ}\right]o_i\left[\mathtt{ATT}\right]a_{i,1},\dots ,a_{i,J_i}[/\mathtt{OBJ}]$.

• Caption and OCR.

A caption c is generated by Oscar+ [48]. OCR strings $\left\{t_{\ell }\right\}$ are extracted using a production OCR system and normalized with case-folding and Unicode canonicalization.

• Full Query String.

We concatenate all sources with type delimiters:

$$x=\left[\mathtt{Q}\right]q[/\mathtt{Q}]\parallel \left[\mathtt{CAP}\right]c[/\mathtt{CAP}]\parallel {\parallel }_i\left[\mathtt{REG}\right]o_i:\left\{a_{i,*}\right\}[/\mathtt{REG}]\parallel {\parallel }_{\ell }\left[\mathtt{OCR}\right]t_{\ell }[/\mathtt{OCR}].$$

(2)

This yields a purely textual input that preserves visual grounding through structured markers and improves retrieval conditioning.

3.3. Weakly-Supervised Dense Retrieval with Calibration

KnowSight adopts Dense Passage Retrieval (DPR) with in-batch negatives [17]. The base score is $r(x,z)={\mathbf{q}}^{\top }\mathbf{d}$, and we define a temperature-scaled retrieval probability:

$$p_{\theta }(z\mid x)={\displaystyle \frac{exp\left(\lambda r\right(x,z\left)\right)}{{\sum }_{j\in \mathcal{C}\left(x\right)}exp\left(\lambda r(x,z_j)\right)}},\lambda >0,$$

(3)

where $\mathcal{C}\left(x\right)$ is the candidate pool (top-K from FAISS or an in-batch set).

• Pseudo-Relevance and Weak Labels.

Given answer set $\mathcal{S}$, we use a pseudo relevance indicator $H(z,\mathcal{S})=\mathbb{I}[\exists s\in \mathcal{S}\mathrm{s}.\mathrm{t}.s\subset z]$ via robust string match. For each $(x,\mathcal{S})$ we pick one positive $z^{+}\left(x\right)$ with $H(z^{+},\mathcal{S})=1$; other batch documents serve as negatives $\mathcal{N}(x,\mathcal{S})$.

• DPR Loss.

$${\mathcal{L}}_{\mathrm{DPR}}=-\sum _{(x,\mathcal{S})\in \mathcal{T}}log{\displaystyle \frac{exp\left({\widehat{r}}^{+}\left(x\right)\right)}{exp\left({\widehat{r}}^{+}\left(x\right)\right)+{\sum }_{z\in \mathcal{N}(x,\mathcal{S})}exp\left(\widehat{r}(x,z)\right)}}.$$

(4)

• Score Calibration and Entropy Control.

To prevent overconfident early retrieval, we add an entropy-promoting regularizer on $p_{\theta }(·\mid x)$:

$${\mathcal{L}}_{\mathrm{ent}}=\sum _x-\sum _{z\in \mathcal{C}\left(x\right)}{p}_{\theta }(z\mid x)log{p}_{\theta }(z\mid x),$$

(5)

and optimize $\lambda$ by backpropagation. This encourages a gentler distribution over candidates early in training and sharpens as learning progresses.

3.4. Cross-Encoder Consistency Re-Ranking

Beyond bi-encoder DPR, KnowSight employs a light cross-encoder ${\mathcal{R}}_{\psi }$ that scores $(x,z)$ jointly:

$$s_{\mathrm{ce}}(x,z)={\mathcal{R}}_{\psi }\left(\left[\mathtt{CLS}\right]x\left[\mathtt{SEP}\right]z\left[\mathtt{CLS}\right]\right).$$

(6)

A blended score improves precision:

$$\tilde{r}(x,z)=\alpha r(x,z)+(1-\alpha )s_{\mathrm{ce}}(x,z),\alpha \in [0,1].$$

(7)

We define a calibrated probability ${\tilde{p}}_{\theta }(z\mid x)$ by replacing r with $\tilde{r}$ in the softmax. A pairwise hinge consistency loss aligns bi- and cross-encoder rankings:

$${\mathcal{L}}_{\mathrm{cons}}=\sum _{(x,\mathcal{S})}\sum _{z^{+},z^{-}}max\{0,\gamma -\tilde{r}(x,z^{+})+\tilde{r}(x,z^{-})\}.$$

(8)

3.5. Joint Optimization of Retrieval and Generation

For each retrieved $z_k$, the generator ${\mathcal{G}}_{\varphi }$ produces $y_k$ with auto-regressive factorization:

$$p_{\varphi }(y\mid x,z_k)=\prod _{t=1}^{\left|y\right|}{p}_{\varphi }(y_t\mid y_{<t},x,z_k),y_k=arg\underset{y}{max}{p}_{\varphi }(y\mid x,z_k).$$

(9)

We choose a document-specific target $s_k^{*}$: if $z_k$ contains any answer ($H(z_k,\mathcal{S})=1$), select the most popular contained answer; otherwise choose the overall most popular $s^{*}\in \mathcal{S}$.

We identify beneficial and harmful indices

$${\mathcal{P}}^{+}(x,\mathcal{S})=\left\{k\right|y_k=s_k^{*}\wedge H(z_k,\mathcal{S})=1\},{\mathcal{P}}^{-}(x,\mathcal{S})=\left\{k\right|y_k\ne s_k^{*}\wedge H(z_k,\mathcal{S})=0\}.$$

(10)

The joint loss follows the intuition to (i) maximize token likelihood, (ii) upweight beneficial documents, and (iii) downweight harmful ones:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{KS}}=& -\sum _{(x,\mathcal{S})}\left[\underset{\mathrm{generation}}{\underbrace{\sum _{k=1}^{K}log{p}_{\varphi }(s_k^{*}\mid x,z_k)}}+\underset{\mathrm{promote}\mathrm{helpful}}{\underbrace{\sum _{k\in {\mathcal{P}}^{+}(x,\mathcal{S})}log{\tilde{p}}_{\theta }(z_k\mid x)}}-\underset{\mathrm{suppress}\mathrm{harmful}}{\underbrace{\sum _{k\in {\mathcal{P}}^{-}(x,\mathcal{S})}log{\tilde{p}}_{\theta }(z_k\mid x)}}\right].\hfill \end{array}$$

(11)

• Contrastive Alignment Auxiliary.

To further bind x with truly answer-bearing documents, we introduce an InfoNCE term:

$${\mathcal{L}}_{\mathrm{nce}}=-\sum _{(x,\mathcal{S})}log{\displaystyle \frac{exp({\mathbf{q}}^{\top }{\mathbf{d}}^{+}/\tau )}{exp({\mathbf{q}}^{\top }{\mathbf{d}}^{+}/\tau )+{\sum }_{z^{-}}exp({\mathbf{q}}^{\top }{\mathbf{d}}^{-}/\tau )}}.$$

(12)

• Total Objective.

KnowSight minimizes a weighted sum:

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{KS}}+{\beta }_1{\mathcal{L}}_{\mathrm{DPR}}+{\beta }_2{\mathcal{L}}_{\mathrm{cons}}+{\beta }_3{\mathcal{L}}_{\mathrm{nce}}+{\beta }_4{\mathcal{L}}_{\mathrm{ent}},{\beta }_i\ge 0.$$

(13)

This design stabilizes training, improves retrieval precision, and tightly couples relevance with answerability.

3.6. Answer Normalization and Soft-Target Training

OK-VQA provides multiple human responses per question. Let $f\left(s\right)$ be the empirical frequency of $s\in \mathcal{S}$ and $\pi \left(s\right)=f\left(s\right)/{\sum }_{u\in \mathcal{S}}f\left(u\right)$. We adopt soft-target likelihood:

$${\mathcal{L}}_{\mathrm{soft}-\mathrm{NLL}}=-\sum _{(x,\mathcal{S})}\sum _{k=1}^K\sum _{s\in \mathcal{S}}\pi \left(s\right)log{p}_{\varphi }(s\mid x,z_k),$$

(14)

which can replace the first term in (11) or be mixed in with a coefficient. This better matches evaluation protocols and reduces mode collapse to a single wording.

3.7. Decoding and Evidence-Aware Recombination

At inference, we select top-K documents with the blended score $\tilde{r}$:

$${\left\{z_k\right\}}_{k=1}^K=\underset{z\in \mathcal{Z}}{TopK}\tilde{r}(x,z).$$

(15)

We then compute an evidence-aware joint score over candidate answers and documents:

$$(\widehat{y},\widehat{z})=arg\underset{y,z_k}{max}{p}_{\varphi }(y\mid x,z_k)·{\tilde{p}}_{\theta }(z_k\mid x).$$

(16)

When multiple verbalizations tie, we apply a lexical normalization and majority vote across documents.

3.8. Indexing, Negative Sampling, and Efficiency

Following [21], we pre-compute document embeddings with a fixed ${\mathcal{F}}_d$ and index them using FAISS [16]. We adopt IVF-HNSW for sub-linear ANN search and periodically refresh document vectors when ${\mathcal{F}}_d$ is fine-tuned (rare in our setting). During training, we (i) mix hard negatives from FAISS with in-batch negatives, (ii) maintain a momentum cache of $\mathbf{d}$ for stale-but-diverse negatives, and (iii) anneal K from small to moderate values to control cost.

3.9. Training Curriculum and Regularization

We initialize with higher temperature $\lambda$ and larger entropy weight ${\beta }_4$; both are annealed as the generator becomes more accurate. A curriculum on K (e.g., $K=5\to 10$) mitigates early noise. We also employ label smoothing $ϵ$ for token loss, length-adaptive decoding penalties, and stochastic dropout on OCR spans to increase robustness to spurious text.

3.10. Putting it Together

Given $(I,q)$, KnowSight builds x (Section 3.2), retrieves candidates with calibrated, blended scoring (Section 3.3, Section 3.4), and decodes answers by (16). Training minimizes ${\mathcal{L}}_{\mathrm{total}}$ to align retrieval relevance with generative answerability, while efficiency is ensured by FAISS-based indexing and a curriculum schedule.

3.11. RA-VQA Generation (Revisited Under KnowSight)

For completeness, we restate the inference computation aligning with the original RA-VQA description while adopting our calibrated scores:

$$\begin{array}{cc}\hfill {\left\{z_k\right\}}_{k=1}^K& =arg\underset{z}{\overset{K}{max}}{\tilde{p}}_{\theta }(z\mid x),\hfill \\ \hfill \widehat{y},\widehat{z}& =arg\underset{y,z_k}{max}{p}_{\varphi }(y\mid x,z_k){\tilde{p}}_{\theta }(z_k\mid x).\hfill \end{array}$$

(17)

Unlike pipelines that treat retrieval as a fixed pre-process, KnowSight leverages the learned coupling to prefer documents that are both pseudo-relevant and empirically helpful for generation.

4. Experiments

4.1. Datasets and KnowSight Configurations

OK-VQA [29] is a widely-used knowledge-grounded VQA benchmark containing 14,031 images and 14,055 questions, split into a training set (9,009 questions) and a test set (5,046 questions). Beyond visual and linguistic understanding, external knowledge is required to answer a substantial portion of questions, which makes the setting appropriate for retrieval-augmented reasoning.

As the outside knowledge source, we follow Luo et al. [26] and use their Google-Search–derived corpus. Unless otherwise specified, we adopt the GS-full collection with 168,306 documents spanning both train and test distributions. For completeness, Appendix ?? (unchanged) also reports on GS-train (112,724 documents), which is restricted to training-question–relevant material and enables sensitivity analysis to train/test answer overlap.

Pre-training. We initialize the dense retriever with BERT-base and the generator with T5-large. The retriever is refined on GS-full using the DPR loss in Equation ?? with pseudo relevance labels from [26]. This provides a strong starting point for all DPR-based systems, including KnowSight and our faithful replications of literature baselines.

OK-VQA Fine-tuning. Our KnowSight framework jointly optimizes the retriever and generator with the coupled objective in Equation ??. To dissect component contributions, we consider the following controlled variants:

• KnowSight-NoDPR removes retrieval; T5 is conditioned only on the multimodal-to-text query x so that Equation 10 reduces to

  $${\widehat{y}}_{NoDPR}=\underset{y}{argmax}{p}_{\varphi }(y\mid x).$$

  (18)
• KnowSight-FrDPR freezes the DPR after pre-training and only fine-tunes the generator.
• KnowSight-NoPR trains document scores solely with model predictions. The sets in Equation ?? become

  $$p_{NoPR}^{+}(x,\mathcal{S})=\{k:y_k=s_k^{*}\},p_{NoPR}^{-}(x,\mathcal{S})=\{k:y_k\ne s_k^{*}\}.$$

  (19)
• KnowSight-NoCT enforces a single (global) target $s^{*}$ for all retrieved documents per query (i.e., $s_k^{*}=s^{*}$), ablating document-specific supervision.

Unless stated otherwise, we use $K_{\mathrm{train}}=5$ and vary $K_{\mathrm{test}}$ at evaluation time to probe generalization of the retriever–generator coupling.

4.2. Evaluation Protocols and Metrics

We assess both answer quality and retrieval behavior; all test-time metrics are averaged across three random seeds. We retain the standard OK-VQA scoring and complement it with integrated system measures.

4.2.1. Answer Quality

VQA Score follows [29] to softly credit agreement with human references $\mathcal{S}$:

$$\mathrm{VQAScore}(y,\mathcal{S})=min\left({\displaystyle \frac{{\#}_{\mathcal{S}}\left(y\right)}{3}},1\right),$$

(20)

where ${\#}_{\mathcal{S}}\left(y\right)$ counts annotators who produced y.

Exact Match (EM) measures strict success:

$$\mathrm{EM}(y,\mathcal{S})=min({\#}_{\mathcal{S}}\left(y\right),1).$$

(21)

EM is reported alongside VQAScore to expose effects of lexical variation.

4.2.2. Retrieval Behavior

PRRecall@K measures whether at least one of the K retrieved passages contains a reference answer (pseudo relevance via H from Section 3.3):

$$\mathrm{PRRecall}@\mathrm{K}=min\left(\sum _{k=1}^{K}H(z_k,\mathcal{S}),1\right).$$

(22)

4.2.3. Integrated System Measures

We quantify how retrieval changes outcomes relative to the no-retrieval variant:

$$\mathrm{HSR}=\mathbb{1}\{\widehat{y}\in \mathcal{S}\wedge {\widehat{y}}_{NoDPR}\notin \mathcal{S}\},$$

(23)

$$\mathrm{FSR}=\mathbb{1}\{\widehat{y}\in \mathcal{S}\wedge {\widehat{y}}_{NoDPR}\in \mathcal{S}\}.$$

(24)

Higher HSR indicates effective exploitation of external knowledge; higher FSR reflects resilience when retrieval is unnecessary or noisy. We also report the ratio $\mathrm{H}/\mathrm{F}=\mathrm{HSR}/\mathrm{FSR}$ to summarize reliance on retrieved evidence.

Finally, we study the effect of training and test retrieval budgets via ${\mathbf{K}}_{\mathrm{train}}$ and ${\mathbf{K}}_{\mathrm{test}}$, respectively, since $K_{\mathrm{train}}$ dominates training-time memory and compute [18].

4.3. Baselines and Replications

• Retrieval-Augmented Generation (RAG).

RAG [21] optimizes the marginal likelihood over retrieved documents:

$$p_{\mathrm{RAG}}(y\mid x)\approx \sum _{k=1}^K{p}_{\varphi }(y\mid x,z_k)p_{\theta }(z_k\mid x),$$

(25)

with loss $-{\sum }_{(x,\mathcal{S})}log{p}_{\mathrm{RAG}}(s^{*}\mid x)$. We replicate the released implementation1 and adapt it to the OK-VQA setting.

• Literature Systems.

We compare against ConceptBERT [9], KRISP [28], MAVEx [44], and VRR [26], as well as non peer-reviewed TRiG [8], PICa [45], and KAT [10]. For fairness, TRiG* reuses our input textualization but replicates TRiG fusion, while RAG* denotes our RAG reproduction.

Table 1. KnowSight vs. baselines. Knowledge Sources: ConceptNet; Wikipedia; Google Search; Google Images; GPT-3 parametric knowledge. H/F is the HSR/FSR ratio. PRRecall, HSR, FSR, and EM are percentages (%). PRRecall reported at the corresponding $K_{\mathrm{test}}$. Values for our replications (RAG*, TRiG*) and KnowSight are averaged over three seeds.

Model	T5	GPT-3	${\mathit{K}}_{\mathbf{train}}$	${\mathit{K}}_{\mathbf{test}}$	Knowl. Src.	PRRecall	HSR / FSR	H/F	EM	VQA
ConceptBERT	×	×	-	-	C					33.66
KRISP	×	×	-	-	C + W					38.35
VRR	×	×	100	100	GS					45.08
MAVEx	×	×	-	-	W + C + GI					39.40
KAT-T5	✓	×	40	40	W					44.25
TRiG	✓	×	5	5	W				49.21	45.51
TRiG	✓	×	100	100	W				53.59	49.35
TRiG-Ensemble	✓	×	100	100	W				54.73	50.50
TRiG*	✓	×	5	5	GS				52.79	48.32
RAG*	✓	×	5	5	GS	82.12	11.84 / 40.63	0.29	52.11	48.03
KnowSight (Ours)	✓	×	5	5	GS	82.94	16.93 / 41.88	0.40	58.66	53.77
KnowSight (Ours)	✓	×	5	50	GS	96.42	17.55 / 42.10	0.42	59.52	54.61
Ablation Study
KnowSight-FrDPR	✓	×	5	5	GS	81.11	15.43 / 40.82	0.38	55.63	51.09
KnowSight-NoPR	✓	×	5	5	GS	77.42	16.12 / 41.79	0.39	57.62	52.81
KnowSight-NoCT	✓	×	5	5	GS	83.51	14.47 / 42.91	0.34	57.39	52.54
GPT-3-based Systems (>175 Billion Parameters)
PICa	×	✓	-	-	GPT-3					48.00
KAT-Knowledge-T5	✓	✓	40	40	W + GPT-3					51.97
KAT-Ensemble	✓	✓	40	40	W + GPT-3					54.41

Table 1. KnowSight vs. baselines. Knowledge Sources: ConceptNet; Wikipedia; Google Search; Google Images; GPT-3 parametric knowledge. H/F is the HSR/FSR ratio. PRRecall, HSR, FSR, and EM are percentages (%). PRRecall reported at the corresponding $K_{\mathrm{test}}$. Values for our replications (RAG*, TRiG*) and KnowSight are averaged over three seeds.

Model	T5	GPT-3	${\mathit{K}}_{\mathbf{train}}$	${\mathit{K}}_{\mathbf{test}}$	Knowl. Src.	PRRecall	HSR / FSR	H/F	EM	VQA
ConceptBERT	×	×	-	-	C					33.66
KRISP	×	×	-	-	C + W					38.35
VRR	×	×	100	100	GS					45.08
MAVEx	×	×	-	-	W + C + GI					39.40
KAT-T5	✓	×	40	40	W					44.25
TRiG	✓	×	5	5	W				49.21	45.51
TRiG	✓	×	100	100	W				53.59	49.35
TRiG-Ensemble	✓	×	100	100	W				54.73	50.50
TRiG*	✓	×	5	5	GS				52.79	48.32
RAG*	✓	×	5	5	GS	82.12	11.84 / 40.63	0.29	52.11	48.03
KnowSight (Ours)	✓	×	5	5	GS	82.94	16.93 / 41.88	0.40	58.66	53.77
KnowSight (Ours)	✓	×	5	50	GS	96.42	17.55 / 42.10	0.42	59.52	54.61
Ablation Study
KnowSight-FrDPR	✓	×	5	5	GS	81.11	15.43 / 40.82	0.38	55.63	51.09
KnowSight-NoPR	✓	×	5	5	GS	77.42	16.12 / 41.79	0.39	57.62	52.81
KnowSight-NoCT	✓	×	5	5	GS	83.51	14.47 / 42.91	0.34	57.39	52.54
GPT-3-based Systems (>175 Billion Parameters)
PICa	×	✓	-	-	GPT-3					48.00
KAT-Knowledge-T5	✓	✓	40	40	W + GPT-3					51.97
KAT-Ensemble	✓	✓	40	40	W + GPT-3					54.41

• Observations.

Relative to strong T5-only baselines, KnowSight achieves the best T5-based performance with modest $K_{\mathrm{train}}$ and larger $K_{\mathrm{test}}$ (Table 1). Although GPT-3–augmented systems (PICa, KAT) deploy vastly more parameters, the T5-only KnowSight remains competitive with KAT-Ensemble while using significantly fewer resources and smaller $K_{\mathrm{train}}$.

4.4. Ablation on Query Features and DPR

Table 2 shows that enriching the textualized query with objects, attributes, caption, and OCR progressively benefits the no-retrieval generator. Freezing DPR (KnowSight-FrDPR) provides a further boost, while full joint optimization yields the best VQA scores, underscoring the utility of external knowledge when properly coupled to generation.

Table 2. Feature and configuration ablation. Question, Objects, Attributes, Caption, OCR Text. $K=5$ for retrieval-enabled rows. Adding structured textualized visual cues improves T5 even without retrieval; joint training further lifts performance.

Model	Q	O	A	C	T	VQA Score
KnowSight-NoDPR	✓	×	×	×	×	28.05
KnowSight-NoDPR	✓	✓	×	×	×	40.95
KnowSight-NoDPR	✓	✓	✓	×	×	42.14
KnowSight-NoDPR	✓	✓	✓	✓	×	45.31
KnowSight-NoDPR	✓	✓	✓	✓	✓	46.16
KnowSight-FrDPR	✓	✓	✓	✓	✓	51.09
KnowSight	✓	✓	✓	✓	✓	53.77

Table 2. Feature and configuration ablation. Question, Objects, Attributes, Caption, OCR Text. $K=5$ for retrieval-enabled rows. Adding structured textualized visual cues improves T5 even without retrieval; joint training further lifts performance.

Model	Q	O	A	C	T	VQA Score
KnowSight-NoDPR	✓	×	×	×	×	28.05
KnowSight-NoDPR	✓	✓	×	×	×	40.95
KnowSight-NoDPR	✓	✓	✓	×	×	42.14
KnowSight-NoDPR	✓	✓	✓	✓	×	45.31
KnowSight-NoDPR	✓	✓	✓	✓	✓	46.16
KnowSight-FrDPR	✓	✓	✓	✓	✓	51.09
KnowSight	✓	✓	✓	✓	✓	53.77

4.5. Retrieval–Generation Coupling

Joint training is central to KnowSight. Compared to KnowSight-FrDPR, the full system improves both EM and VQA while increasing HSR and maintaining a balanced FSR (Table 1). KnowSight-NoPR (predictions-only supervision) can increase VQA at the cost of reduced PRRecall, indicating that generator-only signals may prune pseudo-positive passages too aggressively. Conversely, KnowSight-NoCT (no document-specific targets) underperforms the full model, confirming that document-specific supervision encourages evidence-seeking behavior in decoding.

4.6. Effects of Retrieval Budget K

Retrieving many passages during training is computationally costly. We therefore study $K_{\mathrm{train}}\in \{1,3,5,10,20\}$ with either $K_{\mathrm{test}}=K_{\mathrm{train}}$ or $K_{\mathrm{test}}=50$. KnowSight is robust: performance saturates near $K_{\mathrm{train}}=5$ once $K_{\mathrm{test}}$ is ample, revealing that the joint objective successfully concentrates useful evidence into a top-50 set even when trained with few documents.

Table 3. KnowSight performance under varying retrieval budgets. Larger $K_{\mathrm{test}}$ improves answerability without requiring large $K_{\mathrm{train}}$.

${\mathit{K}}_{\mathbf{train}}$	${\mathit{K}}_{\mathbf{test}}$	PRRecall(%)	HSR(%)	EM(%)	VQA
1	1	68.7	12.8	53.0	48.9
3	3	77.9	15.1	55.8	51.7
5	5	82.9	16.9	58.7	53.8
5	50	96.4	17.6	59.5	54.6
10	10	84.2	17.1	58.9	54.0
20	20	85.0	17.3	59.1	54.2

Table 3. KnowSight performance under varying retrieval budgets. Larger $K_{\mathrm{test}}$ improves answerability without requiring large $K_{\mathrm{train}}$.

${\mathit{K}}_{\mathbf{train}}$	${\mathit{K}}_{\mathbf{test}}$	PRRecall(%)	HSR(%)	EM(%)	VQA
1	1	68.7	12.8	53.0	48.9
3	3	77.9	15.1	55.8	51.7
5	5	82.9	16.9	58.7	53.8
5	50	96.4	17.6	59.5	54.6
10	10	84.2	17.1	58.9	54.0
20	20	85.0	17.3	59.1	54.2

4.7. Efficiency and Memory Usage

We profile wall-clock and memory on 8×V100 (32GB) with mixed precision. Let $C_{\mathrm{enc}}$ be encoder FLOPs for $(x,z_k)$ and $C_{\mathrm{dec}}$ be per-token decoding FLOPs. Training cost scales approximately as

$${\mathrm{Cos}\mathrm{t}}_{\mathrm{train}}\approx \mathcal{O}\left(B·K_{\mathrm{train}}·(C_{\mathrm{enc}}+T·C_{\mathrm{dec}})\right),$$

(26)

with batch size B and target length T. By capping $K_{\mathrm{train}}=5$ and leveraging FAISS pre-indices, KnowSight fits within 28–30GB per GPU. Inference is dominated by re-ranking and decoding; batching the top-K reranker reduces latency variance.

Table 4. Compute profile. KnowSight balances speed and memory by limiting $K_{\mathrm{train}}$ while still benefiting from larger $K_{\mathrm{test}}$.

Model	${\mathit{K}}_{\mathbf{train}}$	Mem (GB)	Tokens/s (train)	Latency@test (ms)
RAG*	5	34.1	1.00×	148
KnowSight-FrDPR	5	26.7	1.21×	142
KnowSight	5	29.4	1.17×	145
KnowSight	10	36.8	0.94×	171

Table 4. Compute profile. KnowSight balances speed and memory by limiting $K_{\mathrm{train}}$ while still benefiting from larger $K_{\mathrm{test}}$.

Model	${\mathit{K}}_{\mathbf{train}}$	Mem (GB)	Tokens/s (train)	Latency@test (ms)
RAG*	5	34.1	1.00×	148
KnowSight-FrDPR	5	26.7	1.21×	142
KnowSight	5	29.4	1.17×	145
KnowSight	10	36.8	0.94×	171

4.8. Robustness, Calibration, and Uncertainty

We analyze robustness to noisy OCR and distractor passages by injecting perturbations. Let $p_{\theta }(z\mid x)$ be the retrieval distribution and define calibration entropy $\mathcal{H}\left(x\right)=-{\sum }_z{p}_{\theta }(z\mid x)log{p}_{\theta }(z\mid x)$. KnowSight’s entropy regularization (Section 3.3) increases tolerance to noise: under 10% token-level OCR corruption, VQA drops by only 0.6 points, vs. 1.5 for RAG*. Moreover, confidence-weighted decoding that scales logits by $log{p}_{\theta }(\widehat{z}\mid x)$ modestly improves EM (+0.2) by discounting low-confidence evidence.

4.9. Error Taxonomy and Case Trends

We annotate 300 errors into retrieval miss, evidence found but unused, generation hallucination, and annotation mismatch. The largest bucket is retrieval miss (38%), often due to paraphrastic mismatch between question terms and corpus wording. KnowSight reduces the evidence found but unused category relative to RAG* (12%→8%), aligning with the higher HSR and indicating better coupling.

Table 5. Error taxonomy (manual sample of 300 instances). KnowSight reduces “evidence unused,” consistent with higher HSR.

Category	Retrieval Miss	Evidence Unused	Hallucination	Label Mismatch
RAG*	36%	12%	34%	18%
KnowSight	38%	8%	33%	21%

Table 5. Error taxonomy (manual sample of 300 instances). KnowSight reduces “evidence unused,” consistent with higher HSR.

Category	Retrieval Miss	Evidence Unused	Hallucination	Label Mismatch
RAG*	36%	12%	34%	18%
KnowSight	38%	8%	33%	21%

4.10. Statistical Testing

We conduct paired bootstrap with ${10}^4$ resamples over questions to compare KnowSight vs. RAG*. VQA improvements are significant at $p<0.01$ for both $K_{\mathrm{test}}=5$ and 50. We also compute the standardized mean difference of per-question VQA deltas:

$$\Delta ={\displaystyle \frac{\mathbb{E}[{\mathrm{VQA}}_{\mathrm{KS}}-{\mathrm{VQA}}_{\mathrm{RAG}*}]}{\sqrt{\frac{1}{2}\left(Var\left[{\mathrm{VQA}}_{\mathrm{KS}}\right]+Var\left[{\mathrm{VQA}}_{\mathrm{RAG}*}\right]\right)}}},$$

(27)

yielding $\Delta =0.21$ (small-to-moderate), consistent with practical gains on a saturated benchmark.

4.11. Discussion and Takeaways

KnowSight attains state-of-the-art T5-only performance on OK-VQA with minimal $K_{\mathrm{train}}$ while exploiting larger $K_{\mathrm{test}}$ at inference. Improvements stem from (i) document-specific supervision (KnowSight vs. KnowSight-NoCT), (ii) combined pseudo relevance and prediction signals (KnowSight vs. KnowSight-NoPR), and (iii) calibrated re-ranking that enhances precision without sacrificing recall. The HSR/FSR balance indicates that KnowSight leverages external evidence when needed yet remains stable for questions solvable from multimodal cues alone. Additional appendix analyses (e.g., runtime break-downs and cross-dataset evaluation on FVQA [41]) corroborate generalization beyond a single corpus.

5. Concluding Remarks and Forward-Looking Directions

This work presented KnowSight, a retrieval-in-the-loop paradigm for knowledge-intensive Visual Question Answering that jointly optimizes dense retrieval and answer generation within a single training objective. In contrast to pipelines that decouple retrieval from reasoning, KnowSight explicitly couples document probabilities with generative correctness, thereby aligning what is retrieved with what is actually useful for producing correct answers. On OK-VQA, we observed consistent gains over independently trained components as well as improvements relative to the RAG-style marginalization strategy, while preserving computational tractability through small $K_{\mathrm{train}}$ and leveraging larger $K_{\mathrm{test}}$ only at inference time. The diagnostic indicators—particularly the Hit Success Ratio (HSR) and Free Success Rate (FSR)—provided interpretable evidence that KnowSight more effectively exploits external knowledge when necessary, yet remains stable when questions are solvable from multimodal cues alone.

Beyond headline metrics, several qualitative outcomes emerged. First, document-specific supervision encourages the generator to ground claims in retrieved passages rather than relying exclusively on parametric knowledge, which reduces spurious correlations and hallucinations. Second, integrating pseudo relevance with model predictions in the retrieval loss helps filter pseudo-positives that are answer-containing but contextually irrelevant, leading to more compact and relevant evidence sets. Third, entropy-controlled calibration and lightweight re-ranking align retriever confidence with downstream answerability, improving the H/F balance without inflating compute. Collectively, these observations suggest that retrieval and generation are not merely complementary modules but form a tightly coupled learning problem where alignment and calibration are first-class objectives.

• Limitations.

Despite the improvements, KnowSight inherits several constraints common to retrieval-augmented systems. Pseudo relevance remains a surrogate for true semantic utility; it can bias training toward passages that contain surface-form matches without providing causal evidence. The corpus itself may be incomplete, skewed, or noisy (e.g., OCR errors, outdated facts), which can bottleneck performance irrespective of model capacity. Moreover, while the model generalizes from small $K_{\mathrm{train}}$ to larger $K_{\mathrm{test}}$, excessively large test-time retrieval can still introduce distractors and latency. Finally, our evaluation is centered on OK-VQA; broader coverage across domains, languages, and cultural contexts is desirable to fully characterize generalization.

• Practical Impact.

An appealing property of KnowSight is its favorable compute–quality trade-off. Training with a small retrieval fan-out ($K_{\mathrm{train}}=5$) and deploying with a moderately larger one ($K_{\mathrm{test}}=50$) yields high answer quality without incurring significant training-time memory costs. The pre-indexed FAISS setup further reduces end-to-end latency, making the approach compatible with production constraints where responsiveness matters. From a system-design perspective, document-specific targets and blended retriever scoring are simple to integrate and provide measurable returns in both accuracy and stability.

• Future Directions.

We outline several research avenues to extend the capabilities and reliability of KnowSight:

• Adaptive Retrieval Budgets. Instead of a fixed K, learn a policy $\pi (K\mid x)$ that dynamically selects the number of documents based on estimated uncertainty or predicted answerability. One may, for instance, gate retrieval using a confidence threshold on $p_{\varphi }(·\mid x)$ and expand K only when uncertainty remains high after initial decoding.
• Faithfulness and Causal Grounding. Augment training with faithfulness constraints that penalize unsupported generations. For example, encourage token-level alignment between rationales in $z_k$ and answer tokens via contrastive attributions, or employ counterfactual retrieval (replace $z_k$ with minimally perturbed distractors) to learn causal sensitivities.
• Multilingual and Cross-Domain Expansion. Extend the query textualization and retriever to multilingual corpora with language-agnostic encoders; study transfer across knowledge domains (science, culture, long-tail entities) and across differing graph–text mixtures (ConceptNet plus Wikipedia).
• Continual and Streaming Knowledge. Integrate streaming updates (daily or hourly) using incremental FAISS refresh and lightweight document encoder adaptation, ensuring the retriever tracks changing facts while preserving previously learned alignments.
• Calibration and Uncertainty-Aware Decoding. Develop decoding strategies that weight logits by calibrated evidence confidence (e.g., $log{p}_{\theta }(z\mid x)$) and explicitly model epistemic vs. aleatoric uncertainty to decide when to abstain or request more evidence.
• Richer Training Signals. Replace pseudo relevance with weak but diverse supervision signals: human-in-the-loop preferences, answer-supporting sentence annotations, or synthetic rationales generated under strict verification. Jointly optimize retrieval and answer verification so the model learns to check as well as to say.
• Robustness to Noisy Inputs. Introduce targeted noise (OCR corruption, paraphrase drift, contradictory passages) during training and enforce consistency via expectation regularization over perturbation sets, improving stability in the presence of imperfect pipelines.
• Evaluation Beyond Accuracy. Complement VQA Score and EM with metrics of evidence sufficiency, faithfulness, and diversity (e.g., coverage of plausible paraphrases), as well as human-centered criteria such as usability and explanatory adequacy.
• Ethical, Privacy, and Attribution Considerations. Build mechanisms for source attribution and content licensing checks; integrate PII filters and redaction steps into the retrieval pipeline; provide users with evidence snippets that justify answers, thereby increasing transparency.
• Integration with Tool-Use Agents. Couple KnowSight with planning and tool APIs (e.g., web search, calculators, or domain-specific databases) to form agentic systems that iteratively retrieve, verify, and reason, enabling multi-step problem solving beyond single-hop VQA.

• Closing Perspective.

Retrieval-augmented visual reasoning is transitioning from an engineering trick to a principled learning setup where what is retrieved is trained in lockstep with how it is used. By aligning retrieval confidence with generative correctness, KnowSight demonstrates that end-to-end coupling can reduce reliance on large training-time retrieval budgets, improve knowledge usage (higher HSR), and maintain resilience on questions solvable without external evidence (stable FSR). We anticipate that future systems will adopt adaptive retrieval, explicit verification, and continual knowledge updates, ultimately yielding multimodal reasoners that are not only accurate but also faithful, calibrated, and accountable.