After some try and error, mostly errors, it seems like the answer or workaround could be something like this:

$ShareDriveItemCam=Get-MgShareDriveItem -SharedDriveItemId  $SharedEncodeURLCam   -ExpandProperty "children" 

$AllFiles=Get-MgDriveItemChild  -DriveId $ShareDriveItemCam.ParentReference.DriveId -DriveItemId $ShareDriveItemCam.Id  -All

Where $SharedEncodeURLCam is the encoded weburl of folder of intrest.

Using Get-MgDriveItemChild returns all 5000+ objects of the shared folder.