When you used InMemoryUserDetailsManager, security stored not the user object itself, but UserDetails which was safe and serialization does not have necessary. However with JPA Auth Server objects that contain OAuth2Authorization use ser via jackson there is a problem that jackson does not trust that class custom user. Consequently leading to 2 approaches, i guess, jackson Mixin like

public abstract class UserMixin {
    @JsonCreator
    public UserMixin(@JsonProperty("username") String username,
                     @JsonProperty("password") String password) {}
}

then in your config class register that Mixin. Second (much easier) add constructor to requered fields with @JsonCreator to your costr. and for every parameter @JsonProperty