To update @vishesh answer:

There has since been implemented an isPasswordProtected() method in the PSTMessageStore class. It works exactly as described by @vishesh, by checking the 0x67FF identifier.

See (commit)

Minimal code example:

PSTFile pstFile = new PSTFile(filename);
System.out.println(pstFile.getMessageStore().isPasswordProtected());