After a lot of tinkering, I found a solution. Downloaded version of Nvidia drivers and Cuda tool-kit compatible with my installed TensorFlow version and used pip install TensorFlow[and-gpu] which was finally able to activate and use the gpu in training