Modular Visual Question Answering via Code Generation

06/08/2023
by   Sanjay Subramanian, et al.
6

We present a framework that formulates visual question answering as modular code generation. In contrast to prior work on modular approaches to VQA, our approach requires no additional training and relies on pre-trained language models (LMs), visual models pre-trained on image-caption pairs, and fifty VQA examples used for in-context learning. The generated Python programs invoke and compose the outputs of the visual models using arithmetic and conditional logic. Our approach improves accuracy on the COVR dataset by at least 3 the GQA dataset by roughly 2 employ code generation.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset