The answer is that builds doesn't have this functionality.

Build append was designed when Xray didnt exist so it took some shortcuts to include links to builds and not the actual artifacts in the build info.

The best solution right now in the platform is to create a release bundle from the builds and scan it.