Verilog modulus operator for wrapping around a range
Unless there is a specialized macro cell, non powers of 2 modulus will take a large number of gates and have relatively long propagation delays especially if done as pure combiantional logic. 

Be aware depending on your synthesizer the variables 'alphabet_start', 'alphabet_position', and 'new_alphabet_position' my be inferred latches. The way you used them is as intermediated logic, so if you don't references them outside this always block and your synthesizer has decent optimization, then it will not be a latch. To guarantee they will not be latches, they must be given default values outside the if statement.

You state that all variables except 'direction' are type 'byte', this means 'shift_by' may have a value greater than 25 or less than -25 ('byte' is a signed value by default). By using a signed values and adding three value (26 + (alphabet_position + shift_by)) before using the modulus, there is a decent changes that the mod26 will be evaluated on a 10-bit signed value. That will use more logic than if used on an 8-bit value. There is a change your synthesizer may do some optimization, but it might not be great. 

If you can guarantee 'shift_by' is less than 26 and greater than -26 ( greater or equal to 0 if unsigned), then you don't need  'alphabet_position' or 'new_alphabet_position'. Simply add or subtract the 'shift_by' and calculate if out of range. For the range check, fist check if 8'(shifted_char-26) &gt;= alphabet_start. The reason for this is to make sure we are comparing positive numbers. "z"+25 is 147 which is negative for a signed 8-bit value. The 8'() with cast it as an 8-bit unsigned value to trim any non-zero intermediate 9th+ bit(s). If an adjustment is not needed then check if hifted_char &lt; alphabet_start as now the possibility of overflowing to a negative number has been already handled.

If you cannot guarantee 'shift_by' is within range, then you have no choose by to mod it. Luckily this is an 8-bit signed value which is better than your original worse case with a 10-bit signed value. This is not ideal but the best I can offer. It is more optimal to have the driver of 'shift_by' assign a legal value then adding more logic to mod it.

Since you are using SystemVerilog, you may want to consider using fresh_char inside { ["A":"Z"] } which is functionally the same as "Z" &gt;= fresh_char &amp;&amp; fresh_char &gt;= "A". The inside is keyword is intended to be synthesizable, but I don't know if it is commonly supported.

Consider the following code. It may not be the most optimized, but it is more optimized than your original code:

always_comb
begin
  shift_by_mod26 = shift_by % 26; // %26 is not need if guaranteed asb(value) &lt; 26
  alphabet_start = (fresh_char inside { ["A":"Z"] }) ? "A" : "a";
  if ( fresh_char inside { ["A":"Z"], ["a":"z"] } )
  begin
     if (direction == "f")
       shifted_char = fresh_char + shift_by_mod26;
     else
       shifted_char = fresh_char - shift_by_mod26;

     // subtract 26 first in case shifted_char is &gt;127
     // bring back to a positive if signed (&gt;127 unsigned is negative signed)
     if (8'(shifted_char-26) &gt;= alphabet_start)
       shifted_char -= 26;
     else if (shifted_char &lt; alphabet_start)
       shifted_char += 26;
  end
  else
  begin
    /* don't touch bytes that aren't letters */
    shifted_char = fresh_char;
  end
end


Note: if 'direction' is not a type 'byte', then it must be at least a 7bits(unsigned) wide or greater (sign agnostic) to every match "f" 



Cross post answer for a cross post question