To solve the issue I was having I needed to use a delegated account. I couldn't add the service account email directly due to workspace limitations but it was able to act under an email in the workspace

    credentials = service_account.Credentials.from_service_account_file(
        path.join(path.dirname(__file__), "config", "credentials.json"),
        scopes=scopes,
    )

    delegated_creds = credentials.with_subject("email@domain.com")
    self.service = build("drive", "v3", credentials=delegated_creds)