finally got it to work, just leaving out the languagefield is not enough, since TYPO3 7.4 you need to explicitly set it to 0

https://docs.typo3.org/c/typo3/cms-core/main/en-us/Changelog/7.4/Feature-68191-TypoScriptSelectOptionLanguageFieldIsActiveByDefault.html

so this is the code that grabs the correct categories from the base language

table = sys_category
        select {
            # set correct id for sys folder
            pidInList = 652
            selectFields = title, uid
            join = sys_category_record_mm ON sys_category_record_mm.uid_local = sys_category.uid
            where.data = field:recordUid
            where.intval = 1
            where.wrap = uid_foreign=|
            languageField = 0
        }