Looks like this issue has been fixed in the last few versions of Angular.

Here is an example of Angular Material components working inside a component with encapsulation set to ShadowDom:

https://stackblitz.com/edit/angular-material-components-in-shadow-dom?file=src%2Fexample%2Fdate-range-picker-overview-example.ts

And if you need the contents of cdkOverlay, modals, etc to be inside the shadow dom, here is a proof of concept doing that:

https://stackblitz.com/edit/angular-overlay-inside-shadow-dom?file=src%2Fshadow-dom-with-overlay-override%2Fshadow-dom-with-overlay-override.ts