Answer for Bootstrap 5 without jQuery:
if (bootstrap.Modal.getOrCreateInstance(document.getElementById('myModal'))._isShown) { // ... }
More info on BS5 with JS