Looks like you might be encountering the same problem discovered in llama.cl -- hidden memory allocation, at least in the BLAS library we used, LLA. To get fast results with gemm you'll need to allocate the memory ahead of time. and use the destructive version (gemm! for example; note the allocation of the target array before the matrix multiplication)