So I partly figured it out. Turns out that I was trying to put the event listeners on the mailbox itself instead of the item, which made it so the function was never triggered since the event was fired on the item. Haven't found a solution for the title but here is the new base, seems to work just fine (will replace the console.log with the handler functions I have).

useEffect(() => {
        const mailbox = Office.context.mailbox;
        
        if(mailbox.item?.itemType==='appointment'){
          Office.context.mailbox.item?.addHandlerAsync(Office.EventType.AppointmentTimeChanged, (msg:any)=>{console.log(msg)});
          Office.context.mailbox.item?.addHandlerAsync(Office.EventType.RecipientsChanged, (msg:any)=>{console.log(msg)});  
      }
    }, []);