Combining Vision and Language

Combining computer vision and natural language processing is an emerging topic which has been receiving  much research attention recently. Visual Question Answering (VQA) can be seen as a proxy task for evaluating a vision system’s capacity for deeper image understanding. Current datasets, and the models built upon them, have focused on questions which are answerable by direct analysis of the question and image alone. The set of such questions that require no external information to answer is interesting, but very limited. We thus propose a new VQA dataset (FVQA) with additional supporting-facts. In response to the observed limitations of RNN-based approaches, we propose a method which is based on explicit reasoning about the visual concepts detected from images.  Second, an intriguing features of the Visual Question Answering (VQA) challenge is the unpredictability of the questions. Extracting the information required to answer them demands a variety of image operations from detection and counting, to segmentation and reconstruction. We propose a general and scalable approach which exploits the fact that very good methods to achieve these operations already exist, and thus do not need to be trained.  Last, the key challenge in Visual Dialogue is to maintain a consistent, and natural dialogue while continuing to answer questions correctly. We show how to combine Reinforcement Learning and Generative Adversarial Networks (GANs) to generate more human-like responses to questions.

Vision and Language are the core channels of communication for humans and when interacting with intelligent agents we also want to rely on these channels. In this talk I will present our recent progress to model vision and language systems, by not only by improving their performance but also increasing their grounding or interpretability. I will discuss applications in visual question answering, visual dialog, and video description.

Question-answering (QA) on video contents is a significant task for achieving human-level intelligence as it involves both vision and language in real-world settings. However, video story QA task is challenging because of its multimodal, time-series, and ambiguous properties. The task requires to extract high-level meanings from the multimodal contents, while scene frames and captions in a video are redundant, highly-complex, and sometimes ambiguous information for the task. In this talk, I will introduce various visual-linguistic story representation with attention mechanisms. They represent visual- linguistic knowledge in the form of a sentence or a distributed representation taking the entire story into account. Also, I will introduce a video QA challenge, MovieQA Challenge, that is attracting attention from researchers.

Modern advances in computer vision and natural language processing have generated considerable interest for tasks at the intersection of these two fields. Image captioning and visual question answering have emerged as opportunities to study automated, high-level reasoning over multiple modalities. While the performance of deep learning models has generated significant excitement, recent studies have pointed at limited abilities for actual reasoning of such models. In this talk, we will consider the intrinsic limitations of purely supervised, learning-based approaches to these tasks, and present some of our ongoing work to overcome them, using modular architectures, mid-level supervision, and a meta learning formulations.

Linguistic style is an essential part of written communication, with the power to affect both clarity and attractiveness. With recent advances in vision and language, we can start to tackle the problem of generating image captions that are both visually grounded and appropriately styled. One such style is descriptions with emotions, which is commonplace in everyday communication. We develop a system to describe an image with emotions, and present a model that automatically generates captions with positive or negative sentiments. We propose a novel switching recurrent neural network with word-level regularization, which is able to produce emotional image captions using only 2000+ training sentences containing sentiments. In the second part of this talk, we will discuss the problem of learning to generate styled image captions from unpaired image and text data. We present a strategy that learns styled image caption generation from a large corpus of styled text without aligned images. The core idea of this model, called SemStyle, is to separate semantics and style. One key component is a novel and concise semantic term representation generated using natural language processing techniques and frame semantics. In addition, we develop a unified language model that decodes sentences with diverse word choices and syntax for different styles. Evaluations, both automatic and manual, show captions from SemStyle preserve image semantics, are descriptive, and are style shifted. This work provides possibilities to learn richer image descriptions from the plethora of linguistic data available on the web.

Image recognition performance has dramatically improved due to the progress of machine learning, large-scale and high-quality datasets, and strong computer power. However, in most methods, adaptable environments are limited, and many problems to be solved remain to adapt to real-world issues with high uncertainty. One of the effective ways to tackle those difficult problems for a recognition system is learning by asking questions. To realize this knowledge acquisition, the following three modules are required: 1) knowing unknown objects, 2) generating appropriate questions for unknown objects, 3) learning from limited supervised information. In this talk, we will introduce recent works of our team about the above modules for acquiring knowledge by asking questions.

For the past decade computer vision research has achieved increasing success in visual recognition including object detection and video classification. Nevertheless, these achievements still cannot meet the urgent needs of image and video understanding. The recently rapid development of social media sharing has created a huge demand for automatic media classification and annotation techniques. One promising solution is to employ attribute learning to transfer the information to tasks for which no data have been observed so far. In this talk, we will summarize the our recent works on attribute learning in image/video understanding. We show the benefits of solving these challenges and limitations in our approach which thus achieves better performance than previous methods.