Access Type

Open Access Embargo

Date of Award

January 2019

Degree Type


Degree Name



Computer Science

First Advisor

Xuewen . Chen


Attention mechanism has shown promising results in many fields of machine learning such as image captioning and machine translation. In this work, we focus on attention-based models for deep reinforcement learning. We concentrate on developing deep neural networks

that are fed with a sequence of high-dimensional raw pixels. Particularly, we design attention-based models for challenging tasks including navigation, autonomous driving, and video captioning. In these tasks, deep reinforcement learning algorithms facilitate training of their sophisticated models, and the attention mechanism serves different purposes. In the navigation and autonomous driving tasks, through the attention mechanism, our model attends over different views of the environment provided by different available cameras to decide about the

best actions that should be taken at each time step. On the other hand, in the video captioning task, through the attention mechanism, the model attends over correlated events to generate captions for different temporal regions in the video. Through experiments, we illustrate that the performance of our attention-based methods for deep reinforcement learning surpasses the performance of their state-of-the-art baselines.