I would recommend you to use sharelink in SharePoint. If you grant permission to users, they will have access to view site contents and navigation to the site even have the least permission.
If you share the file or folder to the users with share link, they will only have the access to the item you shared. Please refer to the document