I had faced similar issue with community page. Instead of using NavigationMixin, please try using Lightning-Modal component.

This component helps in generating a pop up in same window.

Refer: https://developer.salesforce.com/docs/component-library/bundle/lightning-modal/documentation

Please upvote, If this helps in resolving your issue.

Let me know if you face any issue in incorporating Lightning-Modal component in your use-case.