This issue was resolved here: https://devzone.nordicsemi.com/f/nordic-q-a/123400/zephyr-sd-card-remount-issue-fs_unmount-vs-disk-deinitialization-leading-to-eio-or-blocked-workqueue

I was able to solve it with these steps:

1. Did not use either of the following:

disk_access_ioctl("SD", DISK_IOCTL_CTRL_INIT, NULL);

disk_access_ioctl("SD", DISK_IOCTL_CTRL_DEINIT, NULL);

Earlier I would init the disk, mount, (do stuff) and then on pin triggered removal of SD card unmount and deinit. It seems I need to remove the init/deinit them altogether or deinit right after init if I need to access any parameters using the disk_access_ioctl command.

2. Even with the above solution for some reason everything would get blocked after at unmount. This was resolved once I moved to a lower priority workqueue. I was using the system workqueue before and it would block forever.