The top answer from Eric was great, but I couldn't initially locate that View > Cell Toolbar in Jupyter Labs. So for anyone looking for that feature in Jupyter Labs, I found it on the right side bar, in the "Property Inspector" icon, under "COMMON TOOLS". I added the "raises-exception" Cell Tag in there, and it worked like a charm. I added a link to a screenshot of that below. Thanks Eric!
