From a quick look at the docs, it looks like Rocket.Chat can integrate with an OIDC compatible IdP (set up via the OAuth settings you mentioned).

Cognito is an OIDC compatible IdP, so you should be able to configure Rocket.Chat to use Cognito using the endpoints listed here. However, using Cognito directly as an OIDC IdP like this means using the hosted UI.

To integrate with your custom UI using the Cognito SDK, you would need to implement your own OIDC authentication server on top of Cognito.

While that's doable, it would mean a fair bit of additional complexity (and therefore risk) that you probably want to avoid. So I'd recommended customizing the hosted login UI rather than building your own auth UI + OIDC server.