My 1st step towards the resolution was to check if olcDisallows is configured or not, if olcDisallows is configured then remove it.

My 2nd step towards the resolution was to check if olcRequires is configured or not, if olcRequires is configured then remove it.

Last step was to check if olcAllows exists or not, we need to allow it for the anonymous to work.

I have fixed it by following the above order. Also I have set my access to unlimited and works like a charm. Thanks.