It sounds like you’re running into the classic challenges of applying GPA + PCA to complex 3D anatomy like vertebrae. From what you describe, there are a few reasons why your ASM fitting is going “off”:

Insufficient or inconsistent correspondences

Active Shape Models (ASM) work best when each landmark has a consistent semantic meaning across all shapes. Vertebrae have complex topology, and even after Procrustes alignment, landmarks may not correspond exactly between meshes.

Using closest points for surface-based fitting can lead to mismatched correspondences, especially on highly curved or irregular regions.

Large shape variability / non-overlapping regions

If parts of your vertebrae are displaced or have high variability, the mean shape may not represent all instances well. PCA will then project shapes onto modes that don’t match the local geometry, producing unrealistic fits.

Scaling / alignment issues

You are doing similarity Procrustes alignment (scaling + rotation + translation), which is generally good, but when using surface points instead of annotated landmarks, slight misalignments can propagate and distort PCA projections.

Step size / iterative fitting

In your iterative ASM, step_size=0.5 may overshoot or undershoot. Sometimes, reducing the step size and increasing iterations helps stabilize convergence.

Too few points / too sparse sampling

Sampling only 1000 points on a vertebra mesh may not capture all the intricate features needed for proper alignment. Denser sampling or using semantically meaningful points (e.g., tips of processes, endplates) improves GPA convergence.

Flattening for PCA

Flattening 3D coordinates for PCA ignores the spatial structure. For complex anatomical shapes, methods like point distribution models (PDM) with mesh connectivity) or non-linear dimensionality reduction can sometimes work better.

Suggestions:

Increase landmark consistency: Make sure points correspond anatomically across all vertebrae. Consider manual annotation for critical points.

Refine initial alignment: Before fitting ASM, ensure the meshes are roughly aligned (translation, rotation, maybe even rigid ICP). Avoid large initial offsets.

Reduce PCA modes or increase data: If your dataset is small (7 vertebrae for landmarks, 40 for surfaces), PCA may overfit. More training shapes help.

Use robust correspondence methods: Instead of just nearest points, consider geodesic or feature-based correspondences.

Check scaling: Surface-based fitting may benefit from rigid alignment without scaling, to avoid distortion.

Visualize intermediate steps: Plot each iteration to see where it diverges—sometimes only a few points cause the misalignment.