Make sure the Activity Recognition code is initialized, called and stopped from the foreground service.
Use Wakelocks if you want activity recognition to be done even screen is off.
Ensure battery optimizations are "unrestricted" or disabled for your app.
https://dontkillmyapp.com will help if you have device specific issue.