[TCSVT 2022] KVQG: Bridging the Gap Between Visual Perception and Human Knowledge

TL;DR

While current AI can identify a "lamp" in a photo, it often fails to ask "What is used to light up a dark room?" because it lacks common-sense context. This paper introduces a Knowledge-based Visual Question Generation (KVQG) model that retrieves non-visual facts from ConceptNet and fuses them with visual features. It breaks the "visual bottleneck," leading to a 43.37 BLEU-4 score on the KBVQA dataset, far outperforming models that rely solely on pixels.

The "Blind Spot" of Current VQG

Visual Question Generation (VQG) is the dual task of VQA—essentially teaching a machine to be curious. However, existing SOTA models suffer from "Visual Myopia." They can only describe what they see (e.g., "What color is the car?").

Humans, conversely, use Inductive Bias derived from years of learning. When we see a giraffe, we think of "Africa" or "Savannah." Research in education suggests that asking deep questions requires integrating visual stimuli with internal knowledge. Prior VQG methods ignored this external knowledge base, resulting in "shallow" questions that lack utility for real-world applications like intelligent tutoring or engaging chatbots.

Methodology: The KVQG Architecture

The proposed KVQG model functions through a sophisticated four-stage pipeline designed to mimic human cognitive processing.

1. Visual & Spatial Encoding (VCE)

The model starts with a Faster R-CNN to detect salient objects. To understand the scene layout, it employs a Graph Neural Network (GNN). Nodes represent objects, and edges represent spatial relations (overlap, inside, cover). This ensures the model understands that a "fork" isn't just an isolated object but is spatially related to "food."

2. Knowledge Retrieval & Answer-Aware Filtering (KRE)

This is the core innovation. For every detected object, the model queries ConceptNet. Since a "lamp" might have hundreds of associated triplets (e.g., UsedFor: Light, MadeOf: Metal), the authors introduce an Answer-Aware Module. By processing the target answer through a Bi-LSTM, the model learns to "attend" only to the knowledge triplets that logically lead to that specific answer.

Figure 1: The KVQG framework. Note the integration of the blue Knowledge Extraction path with the orange Visual path.

3. Feature Fusion and Decoding

The Target Object Extractor (TOE) merges the visual vector $h_{vi}$ and the knowledge vector $h_{ki}$ into a unified representation. This multimodal embedding is then fed into an LSTM decoder to generate the final natural language question.

Experimental Breakthroughs

The model was tested against strong baselines like Radial-GCN and MOAG.

The results in the table above show a massive jump in performance. The CIDEr score improvement is particularly telling; CIDEr measures the "consensus" with human-written questions, indicating that KVQG captures the specific "fact-based" essence that humans naturally include in their queries.

Figure 2: Comprehensive benchmark on FVQA and KBVQA datasets.

Ablation Insights: What matters most?

The ablation studies revealed that:

• w/o KRE: Performance dropped significantly, proving that visual features alone cannot support complex question generation.
• w/o AaM (Answer-Aware Module): Using raw knowledge without filtering reduced accuracy, highlighting that not all "facts" are relevant to a specific target answer.

Deep Insight: Beyond the Pixels

Through the case studies (Fig 3), we see clearly how KVQG outperforms others. For an image of a Trumpet, baseline models generate generic "What is this?" questions. KVQG, having retrieved the triplet <Trumpet, AtLocation, Music Store>, generates: "Which object in this image can be found in a music shop?"

Figure 3: Qualitative comparison showing KVQG's ability to utilize ConceptNet triplets.

Conclusion & Future Outlook

KVQG successfully demonstrates that non-visual knowledge is a first-class citizen in multimodal tasks. By explicitly modeling external facts, AI moves from simple "pattern matching" toward true "understanding."

Limitations: The model is still dependent on the quality of the object detector and the coverage of ConceptNet. If an object is misidentified (e.g., a "remote" seen as a "pot"), the knowledge retrieval will fail. Future work could involve replacing static knowledge bases with the latent knowledge of Large Language Models (LLMs) to handle more diverse and niche objects.