Finally managed to make it work. lance-javas answer was mostly correct, my working module was working because i had one bean annotated with @SecurityDomain and it made the whole subdeployment work.

The valid jboss-ejb3.xml syntax that was working for me (Wildfly 37):

<?xml version="1.0" encoding="UTF-8"?>
<jboss:ejb-jar xmlns:jboss="http://www.jboss.com/xml/ns/javaee"
               xmlns="http://java.sun.com/xml/ns/javaee" version="3.2">
  <assembly-descriptor>
    <s:security xmlns:s="urn:security:1.1">
      <ejb-name>*</ejb-name>
      <s:security-domain>mySecurityDomain</s:security-domain>
    </s:security>
  </assembly-descriptor>
</jboss:ejb-jar>

Also, putting the xml in the EARs root wasn't working, WF didn't parse it (didn't throw error on the wrong syntax), and the correct syntax wans't working either. I had to put the xml in every subdeployment.