You're not wrong in thinking that your model is associating the yellow/red color combination to presence of this object in the image. From your dataset, it seems you have already added some augmentations to your train data (rotations, brightness etc.). Your model may work well with positive examples but it also needs negative examples. You have to train your model to understand what is NOT this object - similar to the yellow box in your 3rd picture. You don't have to specifically label them as negative example, just having such objects in your image in the background or near the object that you want to detect will suffice. When trained, the model will optimize to not detect these as the object of interest.