I think it can be easier if you use an ofPolyline object to draw your triangle, apply your transformation in ofPolyline object (rotation, translation) https://openframeworks.cc/documentation/graphics/ofPolyline/#show_rotateDeg https://openframeworks.cc/documentation/graphics/ofPolyline/#show_translate and then use https://openframeworks.cc/documentation/graphics/ofPolyline/#show_inside to test your mouse coordinates and see if it is inside