In the present day world, there is large scale deployment of Deep Learning based models in a variety of AI critical applications but very little work has been done to test the robustness of such models to adversarial attacks. In this work we propose a way to generate adversarial samples for the task of Visual Question Answering (VQA) by guiding our adversarial sample generation using attention maps from the underlying VQA model. We examine attacks on the state of the art VQA model proposed by Kazemi and Elqursh and demonstrate the effectivenes of our approach on the VQA dataset. Our attention guided adversarial attack model beats the prior state of the art attack model by a substantial margin and establishes a new state of the art for this task.