After spending lot of time realized issue was due to

applicationId

present in apps build.gradle

Framework was expecting application id to have tv in it something like this:

"com.android.tv.myapplication"

it cannot be just

"com.example.myapplication"

or something random. Needs to have some prefix like:

"com.android.tv"

Not sure but seems like there are some checks in system not to send D-Pad events to any random application.