How is a Caesar (Shift) Cipher represented at Binary level? If I want to encrypt the letter H, represented by ASCII 01001000 in Binary, with a key of 1, represented by  00110001. We should get the letter I, represented by 01001001. (H+1 = I through a shift). What actually is the binary manipulation process to get I using the key of 1? How do we get to 01001001 using 00110001? Straight addition of the Binary values doesn't come to the answer.
A Caesar cipher commonly operates on letters. This is often true for classical ciphers. Modern ciphers instead operate on binary values; most of the time implementations consider bytes instead of bits as atomic values.

The 26 letters form the alphabet of the classical cipher. The alphabet is a sequence of characters; in this case it is simply the English alphabet or ABC. The shift of the Caesar cipher is performed using the location within the sequence, using modular addition. The modulus is the number of characters in the alphabet, in this case of course 26. For this reason it is better to use 0 as starting index for the letter A. Modular addition can simply consist of first adding the shift, and then performing the modulus operation. 

Fortunately the characters of the alphabet are already in order within the ASCII table. So instead of doing any binary arithmetic you can simply subtract the value of the letter A from the character you need to encrypt/decrypt. This way you get the location in the sequence of the alphabet. Then you do the shift. Finally you can simply add the value of the letter A to get the ciphertext back.

Creating a larger alphabet, for instance containing letters and digits, is certainly possible. In that case the letters and digits combined are the ciphertext. Determining the index in the alphabet for the cipher will of course get a bit harder (e.g. if x is a digit then the location could be x - '0' + 26 to place it after the letter Z in the alphabet). Alternatively you could write down the alphabet "abcdefghijklmnopqrstuvwxyz0123456789" and then use an indexOf function to find the location in the alphabet string.

So in the end, no binary operations are necessary. Addition, subtraction and modulus are all the operations you need.

Notes


Characters outside the alphabet are often removed before performing encryption for the Caesar classical cipher.
There are multiple ways of handling case, but simply converting everything to upper or lowercase - whichever you decide to use - before encryption should do the trick nicely.
Many languages include a remainder operator rather than a modulus operator. This can be tricky when decrypting as subtraction may result in a negative value.