Using Yolo tracker I was able to solve my solution.

Adding snippet here

replace these lines:

detections_model = coco_model(frame)[0]

To this:

detections_model = coco_model.track(frame, persist=True)
track_ids = detections_model[0].boxes.id.int().cpu().tolist()