Solved - the issue was never directly with poetry or with pyproject.toml. Every repo I tried to install also contained a build.py file that imported numpy first. Poetry runs this before anything else, hence the error was generated. The solution was to modify build.py so that it does not import at the top level.