Connect a Thunderbolt display and a DisplayPort display to the same MacBook Pro
Your model MacBook Pro can power external displays at 60Hz in the following combinations:


One display at 5120 x 2880 resolution with over a billion colors
Two displays at 3840 x 2160 resolution with over a billion colors
Two displays at 4096 x 2304 resolution with millions of colors


So, to use both the 5K and 4K displays you need to be mindful of the resolution they're each at. To change the resolution:


Since you're trying to use a 5K display and a 4K display, I'd start by only connecting the 5K display first, preferably in one of the left ports
Go to Apple > System Preferences... > Displays
Make sure the Display tab is selected
Select the Scaled radio button
Now choose one of the scaled options that appear below (Hint: You can view the selected pixel dimensions below the image of your display on the left)
Once you have your display set at 4096 x 2304 (or below), exit System Preferences
Connect your 4K display to one of the right ports
Now go back to Apple > System Preferences... > Displays
Make sure the Display tab is selected
Now hold down the option key to see the additional scaled options for the second display
Now choose one of the scaled options for the 2nd display
Exit System preferences


See the screenshot below for reference:



REMINDER: If you're wanting to use billions of colours, then at Steps 5 and 11 you need to set your resolution at 3840 x 2160 or lower. Otherwise, if you're happy to use millions of colours, you  can set the resolution for each display as high as 4096 x 2304 each. Keep in mind the actual resolution choices you get will depend on your displays and the adapters you're using.

Let me know if you're still running into problems and we can troubleshoot from there.