Probably lstModifiedDate is not exposed by design, to prevent developers from going in this direction. Instead, GraphAPI exposes Delta Queries which return not only modified entities but also created and deleted.

Delta Query for users example.