You should stick to standard HTTP response codes with custom error messages. You mentioned malformed user-supplied data as the main cause for the errors, which should be handled with a 4xx error code. 5xx error codes are used to indicate an error from the server (like a database connection error or server code crashing).

You mentioned inconsistent and buggy behavior when using this approach, which should not happen. Can you edit your question to share some minimal code (both client and server side) that triggers this inconsistent behavior ?