Need help determining why my voltage level drops
Your problem is that resistive divider, R2 and R4. Calculate the Thevenin equivalent. Now use the 7404 data sheet to see how much current a 7404 draws, and calculate the voltage drop across the equivalent resistance. You'll see that the current needed loads down the divider output.

Instead of using a divider, use a regulator such as a 7805.