Thank you to @luk2302 for directing me to the source to confirm it's not possible with S3BucketOrigin.withOriginAccessControl.

I then realised that Deny statements in S3 bucket policies always override Allow statements, so in my case, where I need to restrict access to certain "directories", I managed to add a Deny statement to the bucket policy afterwards:

bucket.addToResourcePolicy(new iam.PolicyStatement({
  effect: iam.Effect.DENY,
  principals: [new iam.ServicePrincipal('cloudfront.amazonaws.com')],
  actions: ['s3:GetObject'],
  notResources: [
    `${bucket.bucketArn}/assets/*`,
    `${bucket.bucketArn}/tmp/*`
  ],
  conditions: {
    StringEquals: {
      'AWS:SourceArn': `arn:aws:cloudfront::${this.account}:distribution/${dist.distributionId}`
    }
  }
}));

I hope this helps someone.