Playframework (Scala) custom exception handling
You are right to not create classes for AuthFail and EmailTaken.
These really sense as instances, rather than classes,
as they are special cases of AppException.

For example if AppException was a class, then AuthFail and EmailTaken would be best as instances of that class:

class AppException(val errorCode: Int, val message: String) extends Exception {
  val success = false
}

val emailTaken = new AppException(1, "email already used")

val authFail = new AppException(2, "id/password dont match")


However,
if they are instances,
then you would need to pass them around in the code where they might be needed.
Another alternative is to make them objects,
so you can easily import them and use them where needed:

object EmailTaken extends AppException(1, "email already used")

object AuthFail extends AppException(2, "id/password dont match")


Your approach using a trait for AppException instead of a class seems fine too, but I'm fairly new to Scala myself.
Using a class like I did above seems shorter, so I would prefer it.

As for using case classes (or case objects),
in the posted code I don't see a need:
it seems that you don't need to differentiate the different cases.
If all of these cases can be handled as an AppException and result in an InternalServerError,
then there's no need to implement more.

Finally:


  I could have used isInstanceOf[AppException] to check for custom exceptions. But are there any problems with this approach?


Don't do that. It's recommended to avoid isInstanceOf as much as possible.
The alternative is using polymorphism or pattern matching.
In your case, you don't actually need any of this.
You have simple instances of AppException:
all your cases behave the same way,
their only difference is in parameters, not in behavior.