Using tolist can work, followed by np.array will correctly return a (2,3) numpy array

np.array(df["a"].values.tolist())

returns

array([[1, 2, 3],
       [4, 5, 6]])