Use the map property, which sets the maps it should be rendered on:

// remove marker from the map
markerToRemove.map = null;