two dimensional array with 3 characters per entry?
error: too many initializers for 'char [3][8][8]'


The order is wrong:

char keys[8][8][3]


But then some of the literals are larger than int and give a warning as they are truncated.

Easiest is to define as string literals:

const char* keys[8][8] = {
  {" ","4","5","6","x","/"," "," "},
  {"x10","7","8","9"," "," "," "," "},
  {".","rcl","eng","(",")","sd","m+","^2"},
  {"0","(-)","..,,","hyp","sin","cos","tan","hyp"},
  {" ","br","wur","^2","^x","log","ln"," "},
  {" "," "," "," "," ","mode","log"," "},
  {"="," "," "," "," "," ","^3"," "},
  {"=","1","2","3","+","-"," "," "}
};


This can be further improved (stored in program memory, etc). 

Cheers!