The solution was to use this tool: https://www.java.com/en/download/uninstalltool.jsp

Then exit out of System Settings and re-enter it and Java should be gone.