in v4 you trigger opening it like:

Fancybox.show([{ src: "#modalForm" }]);

in old versions

$.fancybox.open({src: '#modalForm', type: 'inline'});