I think "return [self.cdf_inv(p) for p in u]" is the root cause of slowing things down because each call to cdf_inv(p) performs scalar root finding which is actually CPU based and not gpu accelerated even after being wrapped in Tensorflow. So you should try vectorizing inverse CDF.
You might see significant speed enhancement if you eliminate the scalar root finding.