Took me a while to figure out but by using multiple different answers from StackOverflow I was finally able to recreate the desired behaviour.

To enable me to debug a package from a local NuGet feed I had to add the following section to my .csproj. After doing so VS 2022 would locate the correct source files.

<Project Sdk="Microsoft.NET.Sdk">
  [...]
  <PropertyGroup>
    <!-- Map 'Release' / 'Debug' environments to boolean values -->
    <IsReleaseBuild>false</IsReleaseBuild>
    <IsReleaseBuild Condition="'$(Configuration)' == 'Release'">true</IsReleaseBuild>
    <IsDebugBuild>false</IsDebugBuild>
    <IsDebugBuild Condition="'$(Configuration)' != 'Release'">true</IsDebugBuild>
    <!-- Required for SourceLink when publishing NuGet packages to shared feed online. -->
    <PublishRepositoryUrl>$(IsReleaseBuild)</PublishRepositoryUrl>
    <ContinuousIntegrationBuild>$(IsReleaseBuild)</ContinuousIntegrationBuild>
    <DeterministicSourcePaths>$(IsReleaseBuild)</DeterministicSourcePaths>
    <IncludeSourceRevisionInInformationalVersion>$(IsReleaseBuild)</IncludeSourceRevisionInInformationalVersion>
    <DebugType>Portable</DebugType>
    <!-- Required for Debugging with packages in local NuGet feed -->
    <GenerateDocumentationFile>$(IsDebugBuild)</GenerateDocumentationFile>
    <EmbedUntrackedSources>$(IsDebugBuild)</EmbedUntrackedSources>
    <EmbedAllSources>$(IsDebugBuild)</EmbedAllSources>
    <DebugType Condition="'$(Configuration)' != 'Release'">Embedded</DebugType>
  </PropertyGroup>

</Project>

You can verify the behaviour by opening the files from the "External Sources"-section during debugging:

• The tab file show a small lock-symbol
• The containing folder will be %AppData%\Local\Temp\.vsdbgsrc.

In case the symbols aren't loaded when starting the Project, try a full solution rebuild.
If that still doesn't load the correct symbols you can go to "Tools > Options > Debugging > Symbols" and change to "Search for all module symbols unless excluded", then rebuild the solution again.