Yes, you must make a custom implementation.
Only the primary (left) mouse button can be used with SwiftUI's built-in gestures (TapGesture, DragGesture, LongPressGesture, etc.). They do not support right or middle mouse input.