I used the solution provided by @ianhanniballake but I encountered another problem that now android system started following the navigation track. So let suppose you navigate fragments like A -> B -> C -> D -> C -> B -> A.

Now press back button B is selected and displayed, again press back button C is displayed, again press back button D is displayed, again press back button C is displayed, again press back button B is displayed, again press back button A is displayed(start destination is reached and all fragments are pooped off the stack)

If you agains press the back button then app is closed.

So solution to this problem is handle back button navigation manually in the activity where your NavController is tied by using OnBackPressedDispatcher

    onBackPressedDispatcher.addCallback(this, object : OnBackPressedCallback(true) {
        override fun handleOnBackPressed() {

            // If the current destination is not the start destination, pop the back stack
            if (navController.currentBackStackEntry?.destination?.id != navController.graph.startDestinationId) {
                navController.popBackStack()
            } else {
                // If the start destination is already reached, exit the app
                isEnabled = false
                finish()
            }
        }
    })