According to the boost docs:

binary_traits should be instantiated with either a function taking two parameters, or an adaptable binary function object (i.e., a class derived from std::binary_function or one which provides the same typedefs). (See §20.3.1 in the C++ Standard.

a lambda doesn't meet either of those criteria, so it is not surprising that it doesn't work as-is. You will likely need to change your function to take a std::function (or applying a concept to enforce the boost::binary_traits requirements) instead of an arbitrary callable,