This was asked years ago, but I still see the question popping up and didn't find very good answers.
The context is important, particularly are rare false positives acceptable? I can imagine a video game effect that doesn't impact scoring where a few extra bullets making sparks on a bad guy isn't critical.
This addresses the critical cases where you need to be right, like 3D mesh/mesh intersections or Boolean operations. It also works on concave meshes.
Ignore methods relying on random ray direction as solutions. They improve your odds of a good solution, but create difficult to reproduce failures.
First, filter out all open or non-manifold meshes - they have no inside or outside. Open meshes have one or more laminar edges. Non-manifold meshes have at least one edge which shares more than two faces. The concept of inside simply doesn't apply to these.
Assure all triangle normals are correctly oriented according the right hand half edge rule. This assures all normals point into or out of the closed region.
The worst case situation is a conical, triangular fan where the ray hits the vertex at the center of the fan. I just had to deal with this one.
Determine a global same distance tolerance for the entire model and use it everywhere. Differing same distance tolerances will result in A != B, B !=C and A == C cases if the three tests are done with different tolerances. THAT can result in crashing or infinite looping binary tree tests.
Record the distance along the ray AND if the ray's dot product with the tri normal was positive or negative in a list. Sort the list, first on distance, than on sign of the crossing.
Now, all crossing within the same distance tolerance are clustered first, and crossings of the same parity are grouped within the group.
Starting from the back of the list, remove any crossing within the same dist tolerance which has the same parity. That's a mutliple hit on the fan vertex or edge where the ray entered the boundary through two faces simultaneously. Continue until all duplicates are purged.
In the case where the ray hit the fan vertex, entered and exited legitimately, that will be counted as real entry/exit pair. All cases where the ray entered/exited with the same parity have been reduced to a single entry/exit.
Now calculate
parity = list.size modulo 2
If parity is 1, you're inside. If it's 2 you're outside.
Do this multiple times from the same start point.
Only count hits with 2 or more inside results. I still get a few cases with false outside results.
If you have 2 or more "inside" results, it's inside.
I've worked in the industry for years and even NVidia's GPU raycaster documentation admits they get occasional false positives and recommends some oversampling.
