Receiving a string via Xbee in API Mode
1) You need to intialize frame as array and it's length is the maximum frame length you may recieve. Now you are only making a pointer which point to one place then you write to bytes after that in RAM and it may cause errors. If you didn't see errors you only get lucky!!

2) Numbers is numbers 1 is the same as (0b00000001) in binary as the same as (0x01) in hex. you only need to compare as you want. For example to compare the start byte you could do one of the followings:

if(frame[0] == 0x7E){
    // Here you are comparing with Hexadicimal
}

if(frame[0] == 126){
    // Here you are comparing with Decimal
}

if(frame[0] == 0b01111110){
    // Here you are comparing with Binary
}


Note that any conditions of the above have the same machine code in AVR. You are only represent the number to yourself.

Also i have some notes on that code:

1) Why are you add statickeyword to ch , *frame , index it has no meaning at all here.

2) For frame it is better to be unsigend char or uint8_t as char alone is signed. intialize as array like uint8_t frame[10];