Similar to using :empty you can also check if the element has no children. It comes in handy if the element has extra whitespace, but no content which causes :empty evaluate as false:

.container:not(:has(.grid-element--3 > *)) {
  margin-top: -20px;
}