Abstract & Overview
This paper studies generation of descriptive sentences from densely annotated images.Previous work studied generation from automatically detected visual information but produced a limited class of sentences, hindered by currently unreliable recognition of activities and attributes.Instead, we collect human annotations of objects, parts, attributes and activities in images.These annotations allow us to build a significantly more comprehensive model of language generation and allow us to study what visual information is required to generate human-like descriptions.Experiments demonstrate high quality output and that activity annotations and relative spatial location of objects contribute most to producing high quality sentences.* This work was conducted at Microsoft Research. 1 While object recognition is improving (ImageNet accuracy is over 90% for 1000 classes) progress in activity recognition has been slower; the state of the art is below 50% mean average precision for 40 activity classes (Yao et al., 2011).cars (Count:3) Isa: ride, vehicle,… Doing: parking,… Has: steering wheel,… Attrib: black, shiny,… children (Count:2) Isa: kids, children … Doing: biking, riding … Has: pants, bike … Attrib: young, small … bike