I had this issue with when using flutter 3.22 and after updating the flutter to 3.24 this issue was fixed. might have been something related to this commit:

Fix incorrect behavior of ScrollViewKeyboardDismissBehavior.onDrag for ScrollViewers with Drawer by @dawidope in 148948

you can find it here: https://docs.flutter.dev/release/release-notes/release-notes-3.24.0