Just to get an overview of the current situation take for instance Object A and Object B on a scene the player (user) can select either one of them (any one to be specific) as both of the objects will have the MonoBehaviour script attached to them. In the case Object A is clicked on (selected variable is set) and the offset is also determined at the time of click to keep it away from the screen (maybe on a surface or something). Now as the selected object is set the following block of code can run:

 if (Input.GetMouseButton(0) && selectedObject)  // While dragging
    {
        Vector3 screenPoint = new Vector3(Input.mousePosition.x, Input.mousePosition.y, cam.WorldToScreenPoint(selectedObject.transform.position).z);
        Vector3 newWorldPosition = cam.ScreenToWorldPoint(screenPoint) + offset;
        selectedObject.transform.position = new Vector3(newWorldPosition.x, newWorldPosition.y, originalPosition.z);  // Keep Z position fixed
    }

now the position of the Object A (selected object is updated until the mouse button is not released). The Object B (collided object) is set by using a Tag check (not recommended but it can work). And when the trigger is Exited the Object B (collided Object) is set to null i.e the Object A no longer has a reference to it. On mouse button released the position of the Object A and Object B is swapped. And if the collided Object is null the Object A ( selected Object) moves back to its orignal position. Now from what I understand you are setting the Object B (collided Object) to null when the Trigger is exited. The Objects should swap perfectly fine unless and until they are in collision with each other then the collision ends and the move away from the Trigger bounds they will not swap.

void OnTriggerExit(Collider other) {
    if (collidedObject == other.gameObject) {
        collidedObject = null;  // Reset if the selected object moves away
    }
}

removing this block of code will make the code work as intended but it will be prone to error if you intend to swap the object with any latest collided object than removing it could be the easiest solution.

Tips to make the code better:

1. Utilize the use of interfaces example "IDragable" to make the code reusable.
2. IDragable might have a function "Drag" etc which later sets the position of the objects when mouse button will be released. This approach will help you seperate two different logics i.e. the swap and the mouse control logic.

Please reply to this comment for further queries I will try my best to help. And Notify me if the problem has been solved I will be grateful. I apologize in advance if I was unable to understand the scenario to maximum detail.

Thanks.