REST response code for invalid data

400 is the best choice in both cases. If you want to further clarify the error you can either change the Reason Phrase or include a body to explain the error.

412 - Precondition failed is used for conditional requests when using last-modified date and ETags.

403 - Forbidden is used when the server wishes to prevent access to a resource.

The only other choice that is possible is 422 - Unprocessable entity.

I would recommend 422. It's not part of the main HTTP spec, but it is defined by a public standard (WebDAV) and it should be treated by browsers the same as any other 4xx status code.

From RFC 4918:

The 422 (Unprocessable Entity) status code means the server understands the content type of the request entity (hence a 415(Unsupported Media Type) status code is inappropriate), and the syntax of the request entity is correct (thus a 400 (Bad Request) status code is inappropriate) but was unable to process the contained instructions. For example, this error condition may occur if an XML request body contains well-formed (i.e., syntactically correct), but semantically erroneous, XML instructions.

If the request could not be correctly parsed (including the request entity/body) the appropriate response is 400 Bad Request [1].

RFC 4918 states that 422 Unprocessable Entity is applicable when the request entity is syntactically well-formed, but semantically erroneous. So if the request entity is garbled (like a bad email format) use 400; but if it just doesn't make sense (like @example.com) use 422.

If the issue is that, as stated in the question, user name/email already exists, you could use 409 Conflict [2] with a description of the conflict, and a hint about how to fix it (in this case, "pick a different user name/email"). However in the spec as written, 403 Forbidden [3] can also be used in this case, arguments about HTTP Authorization notwithstanding.

412 Precondition Failed [4] is used when a precondition request header (e.g. If-Match) that was supplied by the client evaluates to false. That is, the client requested something and supplied preconditions, knowing full well that those preconditions might fail. 412 should never be sprung on the client out of the blue, and shouldn't be related to the request entity per se.

It is amusing to return 418 I'm a teapot to requests that are obviously crafted or malicious and "can't happen", such as failing CSRF check or missing request properties.

2.3.2 418 I'm a teapot Any attempt to brew coffee with a teapot should result in the error code "418 I'm a teapot". The resulting entity body MAY be short and stout.

To keep it reasonably serious, I restrict usage of funny error codes to RESTful endpoints that are not directly exposed to the user.