How to "unground" a low state of a not gate
Your circuit idea is indeed going to work for just one of the decoded digits at a time. However when you try to apply the outputs for the next decoder output the corresponding inverter output is getting directly shorted to the previously wired inverter that is driving the same segments. You cannot directly inter-tie the outputs of the 74LS04 in this manner. 

There needs to be some manner to isolate the drives to each segment in a way that they can properly OR together. Here is one way that you can achieve this if you want to continue to use the 4 to 16 decoder:



Others have suggested that instead of you trying to design and build your own display decoder that you should use a ready made chip. This is all fine and dandy except for several things. 


You learn more concepts by building out circuits from ideas to functionality.
The CD4511 chip that they are suggesting is not suitable if you want to display every digit value from your four bit adder output including 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, b, C, d, E and F. I guess that since you are using a 4 to 16 decoder that you want to display all sixteen values.


There are limited available chips that can directly decode 4-bit binary (as a hex digit) to seven segment. Many of the older types that performed this function are no longer available. So these days the most likely solution involves programming a PLD chip to make the decoder or using software in a microcontroller to drive the display from pins on the MCU.