Following the @cafce25's comments, using into_inner from Mutex and Arc, the last lines of the build_matrix function ended up like this:

let mut ret_rows = vec![];
for row in rows {
    ret_rows.push(Mutex::into_inner(Arc::into_inner(row).unwrap()).unwrap());
}
ret_rows

There's no value being copied and overhead is minimal.