@Luuk's answer is great, but it also deletes photos without a matching movie file.

Here my commented alternative, which I have no doubt could be improved.
the Remove-Item uses the -WhatIf parameter for safety.
If it works as intented, remember to remove it.

# Let's get all the MOV files.  
$MovieList = Get-ChildItem -File -Filter '*.mov' | 
    # But we only need the base names without the extension.   
    Select-Object -ExpandProperty BaseName

# Let's get all the photos.     
$PhotoGroupArray = Get-ChildItem -File -Filter '*.jpg' | 
    # Hashing them.   
    Get-FileHash |
    # Grouping them by Hash, so 1 Hash : N Photos
    Group-Object -Property Hash | 
    # Skipping Hashes with just 1 photo: they have no copies we need to remove.   
    Where-Object { $_.Group.Count -gt 1 } |
    # Simplifying the resulting object by removing unneeded properties, though it's optional.  
    Select-Object -Property @{Name = 'FileName'; Expression = { $_.Group.Path } }


# For each group of photos.   
foreach ($PhotoGroup in $PhotoGroupArray) {
    # Check if any photo name is the same as any movie name, except for the extension of course.   
    # If not, adds the photo to the array of files to remove.   
    $RemoveList = foreach ($Photo in $PhotoGroup.FileName) {
        if ([System.IO.Path]::GetFileNameWithoutExtension($Photo) -notin $MovieList) {
            $Photo
        }
    }

    # If the number of photo to remove is the same as the full array of photos, it means there was no movie with the same name.  
    if ($RemoveList.Count -eq $PhotoGroup.FileName.count) {
        # In this case remove every photo except the shortest name one.  
        $RemoveList = $RemoveList | Sort-Object -Property Length -Descending -Top ($RemoveList.count - 1)
    }

    # Remove the compiled list.   
    $RemoveList | Remove-Item -WhatIf
  
}

#>