Found a workaround, it seems that all the metal files in the target will be built into a normal .metallib library. But adding the -fcikernel makes it build for CIKernel. I end up building one of the .metallib using command line.

xcrun -sdk iphoneos metal -c -mios-version-min=15.0 blur.metal -o blur.air

xcrun -sdk iphoneos metallib blur.air -o blur.metallib

Then add the output file to the target.

let library = ShaderLibrary(url: Bundle.main.url(forResource: "blur", withExtension: "metallib")!)

The drawback is you have to manually build it when you update the metal file and it can not work on simulator. Guess the better way to do it is separate two different metal source file into frameworks?