The error messages are not very clear to me, but after some testing I was able to deduce that it is related to connections being reused due to pooling, which causes an invalid state when the users language was changed.

A possible solution would be disabling pooling for the "regular" connection string MSDN:

connectionString = $"server={sqlServerName};database={testDbName};user id={testDbUsername};password={testDbPassword};Pooling=false";

Or resetting the connection pool programatically:

public void ChangeLanguage()
{
    ExecuteSql($"USE [master]; ALTER LOGIN [myuser] WITH DEFAULT_LANGUAGE=[Deutsch]");
    SqlConnection.ClearAllPools();
}

For me personally the second option worked well, since disabling pooling completely would impact performance of the whole test-suite negatively.