Comparing luminosity of LEDs of different units
The LEDs you refer to are approximately 458 nm and 585 nm. 

XPEBPA-L1-0000-00D01 585 nm
XPEBRY-L1-0000-00S01 458 nm 

In general Royal Blue has the highest efficacy of all LEDs and Amber is the least efficient. It is common that an amber LED is a royal blue LED with wavelength converting phosphors (PC amber). 

LEDs emit a number of photons per second.  Each photon has a distinct amount of energy based on its wavelength (i.e. color). Each wavelength is perceived differently by the human eye. 

The International Commission on Illumination (CIE) in 1931 developed the mathematical relationships between photon energy in their Photopic Luminous Efficacy Relative Sensitivity Curve for the C.I.E. Standard Observer. All wavelengths were compared to the most sensitive color green at 555 nn where the 555 nm source = 100,000.  

The brightness of the 555 nm source was reduced until the observer felt that the two sources were equal in brightness. The fraction by which the 555 nm source was reduced, became the luminous sensitivity with respect to the second observed wavelength.

From the CIE Photopic Luminous Efficacy table:
458 nm = 5,560
586 nm = 81,350

What this is saying is 100,000 photons @ 458 nm would be perceived by the human eye to be as bright as 5,560 green 555 nm photons.

There is also a luminous efficacy of 683.00 lumens at 555 nm per 1 Watt of radiometic energy at 555 nm. By combining this factor with the CIE Photopic Luminous Efficacy table we then have a conversion factor between radiometric and photometric at each wavelength.  

Conversion factor from photometric to radiometric at 458 nm ≈ 0.026
Conversion factor from radiometric to photometric to at 458 nm ≈ 37.97
Conversion factor from photometric to radiometric at 585 nm ≈ 0.002
Conversion factor from radiometric to photometric to at 585 nm ≈ 555.62

I have verified these numbers by comparing the calculated factor with spectometer measurements here: Luminous Radiometric and Photonic Conversion.

It does not make sense to me why someone would want to compare "perceived" brightness between these two LEDs.  I would understand wanting to compare the number of photons emitted as for horticulture lighting. 

From the luminous efficacy table a royal blue at 1 watt would emit:
5,560/100,000 = 0.0556 lumens. 

x 0.65 Watts (radiant energy) @ 350 mA =  0.036 lumens vs 104 lumens amber @ 350 mA

The above is the perceived difference by the human eye. Not a fair comparison of two LEDs. 

The blue LED is emitting more radiometric energy than the amber.

From the datasheet the efficacy of the amber LED is 104 lm/W.
104 lm/W @ 350 mA where Vf = 3.05V
Electrical wall watts =  3.05 x 0.35 ≈ 1 watt 
104 lm/W x 0.002 luminous efficacy factor @ 585 nm = 208 mW radiant energy.

                 | Blue  | Amber
-------------------------------
Radiant Energy W | 0.650 | 0.208 
-------------------------------
Luminous Flux lm | 0.036 | 107 
-------------------------------