I am not an expert on GitLab and probably not the best person to answer, but because I see no other answers here yet...

I have not seen this detail explicitly stated - GitLab cache is for files not committed as part of your project, but is for artifacts downloaded from elsewhere.

Also, it is not appropriate to commit large artifacts, they should be stored in an artifact repository instead. If you move your large artifacts to an artifact repository, then caching will be more appropriate.