You could try to disallow the cleaning of those file by using GIT_CLEAN_FLAGS: -ffdx -e <folder or file> at the start of the .yml file. This is documented here
GIT_CLEAN_FLAGS: -ffdx -e <folder or file>
Or you could try to research into caching