My Project was using 2 different versions of System.Memory.

In my case System.Diagnostics.DiagnosticSource 4.0.5.0 was Referencing System.Memory 4.0.1.1

I upgraded System.Diagnostics.DiagnosticSource nuget package to 8.0.0.1 and this action upgraded System.Memory to 4.0.1.2

Now my project has same version i.e 4.5.5 of System.Memory in my project

After upgrading worked well.