My first suspicion here would be a memory-related error. These will show up in the kernel log:

$ sudo dmesg -T

as OOM events. You could also use strace on the application to look for malloc() calls that fail, but you do have to make sure you run strace on the underlying binary, not any wrapper script that might be being invoked in the rule.

If the application has just enough memory available when run outside of Snakemake then it may be the overhead of Snakemake pushing it over the edge. Also, with Snakemake are you running one job at a time or allowing it to run multiple jobs in parallel? Are you using multiple threads within the rule?