The issue was slightly different in my case. Mouse wasn't working at all. Tried these solutions and few others, nothing worked until i finally came across this blog
https://github.com/alacritty/alacritty/issues/2931
In short, trying TERM=xterm-256color vi -u NORC <file>
worked for me. So i exported this variable in my ~/.zshrc
file