In RandomForestRegressor() the criterion options are 'MSE' and 'MAE'. But what is this error that is being measured and optimised before splitting?

As you probably kown, random forests are a collection of decision trees. And you can find the answer of your question for decision trees in the user guide of scikit-learn: https://scikit-learn.org/stable/modules/tree.html#mathematical-formulation