Reports

For "the" first case

any class or method annotated with Spring's @Transactional annotation must also be annotated with @Service

(which I think are really two structurally different cases: for @Transactional methods, you want the declaring classes to be @Services, not the methods themselves), I don't see how you could combine them if you want to have classes and methods reported as individual violations in the corresponding cases.

If you didn't care, but really preferred to combine the rules, you could in this case (when you also accept that, for a class annotated with @Transactional but not @Service, you'll get a violation for every method – also none if the class doesn't have any methods) use:

@ArchTest
ArchRule transactionalServices = methods()
    .that().areAnnotatedWith(Transactional.class)
    .or().areDeclaredInClassesThat().areAnnotatedWith(Transactional.class)
    .should().beDeclaredInClassesThat().areAnnotatedWith(Service.class);

(I've expressed that I wouldn't do that, haven't I?)

The second case

the annotation @jakarta.transaction.Transactional is not allowed on any class or method

is something that could actually be unified as a single rule for classes or methods (regarding their common functionality that they can have annotations [and have a source code location each]), cf. §7.5 Rules with Custom Concepts of the ArchUnit User Guide:

// Intersection types are tricky, cf. https://stackoverflow.com/q/6643241 .
// Please let me know if you find a better solution for this:
<HAS_ANNOTATIONS extends HasAnnotations<?> & HasSourceCodeLocation>
ClassesTransformer<HAS_ANNOTATIONS> classesOrMethodsWhichCanHaveAnnotations() {
    return new AbstractClassesTransformer<>("classes/methods") {
        @Override
        public Iterable<HAS_ANNOTATIONS> doTransform(JavaClasses classes) {
            List<HAS_ANNOTATIONS> result = new ArrayList(classes); // shortcut 🤫
            classes.forEach(javaClass ->
                result.addAll((Set<HAS_ANNOTATIONS>) javaClass.getMethods())
            );
            return result;
        }
    };
}

@ArchTest
ArchRule noJakartaTransaction = no(classesOrMethodsWhichCanHaveAnnotations())
        .should(beAnnotatedWith(jakarta.transaction.Transactional.class));

This uses the following static imports:

import static com.tngtech.archunit.lang.syntax.ArchRuleDefinition.no;
import static com.tngtech.archunit.lang.conditions.ArchConditions.beAnnotatedWith;

79151014