To answer my own question:

There really seems to be no general way to achieve this, but I found the following workaround acceptable.

On API 36+, there is PRIORITY_SYSTEM_NAVIGATION_OBSERVER for OnBackInvokedDispatcher.registerOnBackInvokedCallback. From the documentation:

Callbacks registered with this priority do not consume back events. They receive back events whenever the system handles a back navigation and have no impact on the normal back navigation flow. Useful for logging or analytics.

Awesome, exactly what I was looking for. Unfortunately this value is only available from API 36 onwards, so predictive back can't be used on API 35 and lower. To achieve this, I'm not setting android:enableOnBackInvokedCallback in the AndroidManifest. It will then default to false on API <= 35 and to true on >= 36. On APIs up to 35 onBackPressed can be used as usual.