I was able to make it work.
I had the custom binder configuration class annotated with @Configuration which was causing it to be automatically registered. By Removing the @Configuration I was able to use the custom binder for specific binding.