It appears that the difference between a and ä is considered primary. Using [before 1] you can achieve the expected result.
from icu import RuleBasedCollator
l=["a","ä"]
rbc = RuleBasedCollator('&[before 1]a < ä')
print(sorted(l, key=rbc.getSortKey))