Action Recognition using Visual Attention


Shikhar Sharma, Ryan Kiros, Ruslan Salakhutdinov

Department of Computer Science
University of Toronto

Abstract

We propose a soft attention based model for the task of action recognition in videos. We use multi-layered Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units which are deep both spatially and temporally. Our model learns to focus selectively on parts of the video frames and classifies videos after taking a few glimpses. The model essentially learns which parts in the frames are relevant for the task at hand and attaches higher importance to them. We evaluate the model on UCF-11 (YouTube Action), HMDB-51 and Hollywood2 datasets and analyze how the model focuses its attention depending on the scene and the action being performed.

BibTeX

            @inproceedings{sharma2016actrecICLR,
  title     = {Action Recognition using Visual Attention},
  author    = {Sharma, Shikhar and Kiros, Ryan and Salakhutdinov, Ruslan},
  booktitle = {International Conference on Learning Representations (ICLR) Workshop},
  year      = {2016},
  month     = {May},
  url       = {https://arxiv.org/abs/1511.04119}
}
          

PDF URL