IEEE SINGLE PROCESSING LETTERS

1

Emerging from Water: Underwater Image Color
Correction Based on Weakly Supervised Color
Transfer

arXiv:1710.07084v3 [] 3 Jan 2018

Chongyi Li, Student Member, IEEE, Jichang Guo, Chunle Guo

Abstract—Underwater vision suffers from severe effects due
to selective attenuation and scattering when light propagates
through water. Such degradation not only affects the quality of
underwater images but limits the ability of vision tasks. Different
from existing methods which either ignore the wavelength dependency of the attenuation or assume a specific spectral profile,
we tackle color distortion problem of underwater image from a
new view. In this letter, we propose a weakly supervised color
transfer method to correct color distortion, which relaxes the
need of paired underwater images for training and allows for
the underwater images unknown where were taken. Inspired
by Cycle-Consistent Adversarial Networks, we design a multiterm loss function including adversarial loss, cycle consistency
loss, and SSIM (Structural Similarity Index Measure) loss, which
allows the content and structure of the corrected result the same
as the input, but the color as if the image was taken without
the water. Experiments on underwater images captured under
diverse scenes show that our method produces visually pleasing
results, even outperforms the art-of-the-state methods. Besides,
our method can improve the performance of vision tasks.

I. I NTRODUCTION
Recently, ocean engineering and research have increasingly
relied on underwater images captured from Autonomous Underwater Vehicles (AUVs) and Remotely Operated Vehicles
(ROVs) [1]. However, underwater images usually suffer from
degeneration, such as low contrast, color casts, and noise,
due to wavelength-dependent light absorption and scattering as
well as the effects from low-end optical imaging devices. The
scattering and absorbtion attenuate the direct transmission and
introduce surrounding scattered light. The attenuated direct
transmission causes the intensity from the scene to weaker
and color casts, while the surrounding scattered light causes
the appearance of the scene to be washed out. Besides, the
magnitude of attenuation and scattering depends on several
complex factors including water temperature and salinity, and
the type and quantity of particulates in the water. Serious
degeneration makes it difficult to recover the appearance and
color of underwater images. However, color is extremely important for underwater vision tasks and research [2]. Therefore,
how to effectively approximate the real color of underwater
image has become a challenging problem needed to be solved.
This work was supported in part by the National Key Basic Research
Program of China (2014CB340403), the National Natural Science Foundation
of China (61771334).
Chongyi Li, Jichang Guo and Chunle Guo are with the School of Electrical
and Information Engineering, Tianjin University, Tianjin, China (e-mail:
lichongyi@tju.edu.cn; jcguo@tju.edu.cn; guochunle@tju.edu.cn).
(Corresponding author: Chunle Guo.)

A number of methods [3]–[16] have been proposed to
improve the visual quality of underwater images, ranging
from hardware solutions to image dehazing and color correction methods [17], [18]. The hardware solutions [3]–[6]
have shown the effectiveness, but these solutions are not
applicable to dynamic acquisition. Most of single underwater
image restoration methods [7]–[13] are inspired by the outdoor
dehazing strategies [19]–[22]. For underwater image color correction, traditional color constancy methods (e.g., Gray World
[23], Max RGB [24], White Balance [25], Shades-of-Grey
[26], and et al.) and their variations are usually employed.
Compared with traditional methods based on statistical priors,
we bridge the gap between the color of underwater image
and that of air image by learning their cross domain relations.
Recently, a semi-supervised learning model for underwater
image color correction, namely WaterGAN, has been proposed
[14]. Unlike this work, our model is no need for a large
annotated dataset of images pairs. Besides, WaterGAN shows
limitation in processing the underwater images captured under
unknown sites. Contrast to translating image style [27], [28],
our model learns the semantic color of air image and preserves
the key attributes such as content and structure of input image.
In this letter, we present a novel weakly supervised model
for underwater image color correction, which maps the color
from the scenes of underwater into the scenes of air without
any explicit pair labels. Specifically, given an underwater
image as the input, our model directly outputs an image, which
has the content and structure the same as the input, but the
color as if the image was taken without the water. Though the
translated color might not be the “real” color of underwater
image, our method removes the color casts and improves the
performance of vision tasks. In fact, it is impossible that a
method recovers the appearance and color of any underwater
images when the scenes and light conditions are unknown.
Figure 1 shows two examples of our attempts.
Contributions. This letter introduces the following main
contributions: (1) Compared to existing priors/assumptions
based and semi-supervised methods, to our best knowledge,
this is the first attempt to build a weakly supervised model for
underwater image color correction. Here, “weak supervision”
means that our model relaxes the need of paired underwater
images for training and allows for the underwater images
unknown where were taken. (2) We tackle underwater image
color correction problem from a new angle, which learns a
cross domain mapping function between underwater images
and air images. (3) A multi-term loss function allows our

IEEE SINGLE PROCESSING LETTERS

2

cycle should be able to bring y back to target image. Cycle
consistency loss can be expressed as:
Lcyc (G, F ) =Ex∼pdata (x) [kF (G(x)) − x)k1 ]
+ Ey∼pdata (y) [kG(F (y)) − yk1 ].
(a)

(b)

(c)

(2)

(d)

Fig. 1. Two examples of our results. (a) and (c) are raw underwater images.
(b) and (d) are our results.

model capturing context and semantic information, which
preserves the content and structure of input image, but the
color as if the output image was taken without the water.
II. P ROPOSED M ETHOD
Our solution is based upon very recent advances in imageto-image translation networks [29]–[31], which captures special characteristics of one image collection and figures out
how these characteristics could be translated into the other
image collection, all in the absence of any paired training
examples. Our goal is to learn mapping functions between
a source domain X (i.e., underwater) and a target domain
Y (i.e., air). The inputs are unpaired training image samples
x ∈ X and y ∈ Y . Specifically, our model includes two
mappings G : X → Y (forward) and F : Y → X
N
(backward) given training data {xi }N
i=1 ∈ X and {yi }i=1 ∈ Y .
Following the CycleGAN [29], we also use two adversarial
discriminators DX and DY . The DX aims to distinguish
between images {x} and translated images {F (y)} when the
DY aims to distinguish between images {y} and translated
images {G(x)}. For the generator of forward networks, the
loss function includes three terms: adversarial loss is to match
the distribution of generated images to the distribution in the
target domain; cycle consistency loss is to prevent the learned
mappings G and F from contradiction each other; SSIM loss
[32] is to preserve the content and structure of source images.
For the generator of backward networks, the loss function also
includes three terms. The framework of the proposed method
is shown in Figure 2. Below, we just present the losses of
forward networks and vice versa for backward networks.

C. SSIM loss
To preserve the content and structure of the input image,
SSIM loss [33] is used. First, for pixel p, the SSIM between
input image x and translated image G(x) is defined as:
SSIM (p) =

2σxy + C2
2µx µy + C1
· 2
,
2
2
µx + µy + C1 σx + σy2 + C2

(3)

where p is the center pixel of an image patch, x is an image
patch ∈ X with size 13×13, y is an image patch ∈ G(x) with
size 13×13, µx is the mean of x, σx is the standard deviations
of x, µy is the mean of y, σy is the standard deviations of y,
σxy is the covariance of x and y. C1 =0.02 and C2 =0.03 are
default in SSIM loss. In this way, the SSIM value for every
pixel between input image x and translated image G(x) is
calculated. The SSIM loss can be expressed as:
1 N
Σ (SSIM (p)).
N p=1
where N is the number of pixel in an image.
LSSIM (x, G(x)) = 1 −

(4)

D. Total loss
Finally, the total loss is the linear combination of the abovementioned three losses with weights:
Lloss =λ1 LGAN (G, DY , X, Y ) + λ2 Lcyc (G, F )
+ λ3 LSSIM (x, G(x)).

(5)

A. Adversarial loss

where weights λ1 , λ2 and λ3 are 1, 1 and 10 based on heuristic
experiments on our training data, which makes the order of
magnitude of three components equal because we expect that
these three components have equally significant contributions
in the final loss function. The optimization of generator G
of forward networks is to minimize Equation (5) when the
optimization of discriminator DY is to maximize Equation (1).
In training process, we alternatively optimize G, DY , F , and
Dx , respectively.

For the mapping function G : X → Y and its discriminator
DY , the adversarial loss can be expressed as:

E. Network architecture and training details

LGAN (G, DY , X, Y ) = Ey∼pdata (y) [logDY (y)]
+ Ex∼pdata (x) [log(1 − DY (G(x))].
(1)
where G tries to generate images G(x) that look similar to
images from domain Y while DY aims to distinguish between
translated samples G(x) and real samples y.
B. Cycle consistency loss
We add a cycle consistency loss to constrain the space
of possible mapping functions. Cycle consistency means that
for each image x from domain X, the image translation
cycle should be able to bring x back to the original image,
while for each image y from domain Y , the image translation

The forward and backward networks have the same architecture. Following CycleGAN [29], we adapt the architecture
[34] for our generators and use 70 × 70 PatchGANs [35]–
[37] as our discriminators. To train our model, we collect a
dataset from Internet, which contains 3800 underwater images
and 3800 air images. After that, those images are resized to
256 × 256 based on our limited memory and the architecture
of our discriminators. Figure 3 shows several training samples.
In addition, to stabilize our model training, we replace
the negative log likelihood objective by a least square loss.
Equation (1) is rewritten as :
LGAN (G, DY , X, Y ) = Ey∼pdata (y) [(DY (y) − 1)2 ]
+ Ex∼pdata (x) [DY (G(x))2 ].

(6)

IEEE SINGLE PROCESSING LETTERS

3

Forward networks

Backward networks

Fig. 2. A diagram of our framework. Our model contains two mapping functions G : X → Y (forward) and F : Y → X (backward), and associated
adversarial discriminators DY and DX .

well restored since the assumed optical parameters do not hold
in some underwater scenes. On the contrary, our method can
totally remove the greenish and bluish tone as if our results
were taken without water, which leads to better visual quality.
(a)

(b)

(c)

(d)

Fig. 3. Our training samples. (a) and (b) are underwater images. (c) and (d)
are air images.

The discriminators DX and DY were updated by a history
of generated images rather than the ones produced by the latest
generative networks. We trained the our model using ADAM
[38] and set the learning rate to 0.0002 and momentum to 0.5.
The batch size was set to 1. We implemented our network
with the TensorFlow framework and trained it using NVIDIA
TITAN X GPU. It took 15 hours to optimize our model.
III. E XPERIMENTS
We compare the proposed model with several state-of-theart methods: image-to-image transfer method (i.e., CycleGAN
[29]), color constancy method (i.e., Gray Word (GW) [23] ),
image enhancement method (i.e., INT [39]), and underwater
image restoration methods (i.e., RED [9], UWID [11] and
UWIB [12]). In our experiments, the test images were captured
under varying underwater scenes. For fair comparisons, we did
not compare our method with WaterGAN since WaterGAN
was just available for underwater images taken under designated sites. We subjectively evaluate the visual quality of the
results of different methods. Then, we conduct a user study
since there is no metric designed for underwater image color
correction available. Finally, we carry out application test on
visual tasks in order to give additional evidence.
A. Subjective assessment
In Figure 4, we show several results of different methods. In
Figure 4, original CycleGAN produces blurring results because
it tends to translate the content and structure of air image
to the underwater image. GW method introduces color casts
for some images when its assumption is not available. INT
method just increases the brightness of underwater images
because the prior learned from natural scenes is unavailable.
Three underwater images restoration methods remove the haze
effects and improve the contrast, however, the color is not

B. User study
To perform a visual comparison in an objective way, we
randomly selected 30 underwater images from our collected
dataset for user study. The results of different methods were
randomly displayed on the screen and compared with the
corresponding raw underwater images. After that, we invited
10 participants who had experience with image processing to
score results. There was no time limitation for each participant.
Moreover, the participants did not know which results were
produced by our method. The scores ranged from 1 (worst)
to 8 (best). As baseline, we set the scores of raw underwater
images to 3. We expected that the good result has good contrast
and visibility, abundant details, in especial the color as if the
image was taken without water. On the contrary, the bad result
has low visibility, over-enhanced regions, serious artifacts and
noise, and inauthentic color. We first present in Table I the
average visual quality scores of the images shown in Figure 4.
TABLE I
T HE AVERAGE SCORES OF THE IMAGES PRESENTED IN F IGURE 4
Method
Raws
CycleGAN
GW
INT
RED
UWID
UWBI
Ours

I1
3
3.3
1.4
3.1
6.4
3.1
3.5
7.5

I2
3
2.5
1.7
3.1
3.8
4.5
4.1
7.5

I3
3
2.8
3.8
3.5
5.3
5.9
5.5
6.2

I4
3
2.9
3.5
4.9
3.5
5.4
4.5
6.2

I5
3
3.5
3
3
4.5
5.9
3.5
6.6

In Table I, our results receives best scores, which indicates
that, from a visual perspective, our method produces much
better results. Additionally, the underwater image restoration
methods also achieve good scores. The average visual quality
scores for the selected 30 underwater images are 3, 2, 3,
3.3, 3.7, 4.6, 4 and 6.3 for Raws, CycleGAN, GW, INT,
RED, UWID, UWBI, and Ours. More Results and scores
are available in https://li-chongyi.github.io/homepage.github.
io/proj Emerging water.html. Those results provide a realistic
feedback that our method generates visually pleasing results.

IEEE SINGLE PROCESSING LETTERS

(a) Raws

(b) CycleGAN

4

(c) GW

(d) INT

(e) RED

(f) UWID

(g) UWIB

(h) Ours

Fig. 4. Visual quality comparisons on varying underwater scenes. From top to bottom are image I1-I5.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Fig. 5. Saliency detection test. (a) and (b) are raw underwater images. (c)
and (d) are our results. (e) and (f) are the saliency maps of (a) and (b). (g)
and (h) are the saliency maps of (c) and (d).

(a)

(b)

(c)

(d)

Fig. 6. Keypoint matching test. (a) and (b) are keypoint matching maps of
raw underwater images. (c) and (d) are keypoint matching maps of our results.

IV. C ONCLUSION
C. Application assessment
To further demonstrate the effectiveness of the proposed
methods, we carry out several application tests including
saliency detection [40] and keypoint matching [41]. In Figures 5 and 6, we present the results of application tests before
and after using our method. For the limited space, we just
present several examples. After color corrected by our method,
the results achieve better saliency detection performance and
more matching points. Application tests give additional evidence of the effectiveness of our method.

In this letter, we presented a novel method for underwater
image color correction. Based on the learned cross domain
relations, the proposed method can remove color distortion by
weakly supervised model. It is the first attempt that correcting
the color casts of underwater image by weakly supervised
learning. Furthermore, our method can be use as a guide
for subsequent research of underwater image color correction.
Experiments including quality evaluation, user study, and
application assessment demonstrated the effectiveness of our
method.

IEEE SINGLE PROCESSING LETTERS

R EFERENCES
[1] M. Bryson, M. Johnson, O. Pizarro, and et al., “True color correction of
autonomous underwater vehicle imagery,” J. Field Robot., vol. 33, no.
6, pp. 853-874, 2016.
[2] R. Gibson, R. Atkinson, and J. Gordon, “A review of underwater stereoimage measurement for marine biology and ecology applications,”.
Oceanography and Marine Biology: An Annual Review, vol. 47, pp.
257-292, 2016.
[3] D. He and G. Seet, “Divergent-beam lidar imaging in turbid water,”Optics and Lasers in Eng., vol. 41, no. 1, pp. 217-231, 2004.
[4] Y. Schechner and N. Karpel, “Clear underwater vision,” in Proc. IEEE
Conf. Comput. Vis. Pattern Recognit., 2004, pp. xx.
[5] Y. Schechner and N. Karpel, “Recovery of underwater visibility and
structure by polarization analysis,”IEEE J. Oceanic Eng., vol. 30, no. 3,
pp.570-587, 2005.
[6] T. Treibitz and Y. Schechner, “Active polarization descattering,”IEEE
Trans. Pattern Anal. Mach. Intell., vol. 31, no. 3, pp.385-399, 2009.
[7] N. Carlevaris Bianco and A. Mohan and R. Eustice, “Initial results in
underwater single image dehazing,” in Proc. of IEEE Conf. Oceans,
2010, pp. 1-8.
[8] J. Chiang and Y. Chen, “Underwater image enhancement by wavelength
compensation and dehazing,”IEEE Trans. Image Process., vol. 21, no.
4, pp.1756-1769, 2012.
[9] A. Galdran, D. Pardo, and A. Picn, “Automatic red-channel underwater
image restoration,”J. Vis. Commun. Image R., vol. 26, pp.132-145, 2015.
[10] P. Drews-Jr, E. Nascimento, S. Botelho, and et al., “Underwater depth
estimation and image restoration based on single images,”IEEE Comput.
Graph. Appl., vol. 36, no. 2, pp. 24-35, 2016.
[11] C. Li, C. Ji, R. Cong, and et al.“Underwater image enhancement by
dehazing with minimum information loss and histogram distribution
prior,”IEEE Trans. Image Process., vol. 25, no. 12, pp.5664-5677, 2016.
[12] Y. Peng and P. Cosman, “Underwater image restoration based on image
blurriness and light absorption,”IEEE Trans. Image Process., vol. 26,
no. 4, pp.1579-1594, 2017.
[13] S. Emberton, L. Chittka, A. Cavallaro and et al., “Underwater image and
video dehazing with pure haze region segmentation,” Computer Vision
and Image Understanding, 2017.
[14] J. Li, K. Skinner, R. Eustice and et al., “WaterGAN: unsupervised
generative network to enable real-time color correction of monocular
underwater images,” IEEE Robot. and Aut. Lett., 2017.
[15] S. Zhang, T. Wang, J. Dong and et al., “Underwater image enhancement
via extended multi-scale retinex,” Neurocomputing, vol. 245, pp. 1-9,
2017.
[16] J. Ahn, S. Yasukawa, T. Sonoda and et al., “Enhancement of deep-sea
floor images obtained by an underwater vehicle and its evaluation by
crab recognition,” J. Marine Science and Technology, vol. 22, no. 4, pp.
758-770, 2017.
[17] R. Schettini and S. Corchs, “Underwater image processing: state of the
art of restoration and image enhancement methods,”EURASIP J. Adv.
Signal Process., 2010.
[18] T. Haware and P. Gumble, “A review on underwater image secne
enhancement and restoration using image processing,”Inter. J. Innovative. Research in Electrical, Electronics, Instrumentation and Control
Engineering, vol. 5, no. 9, pp. 28-31, 2017.
[19] K. He, J. Sun, and X. Tang “Single Image Haze Removal Using Dark
Channel Prior,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 12,
pp.2341-2353, 2011.
[20] Q. Zhu, J. Mai, and L. Shao, “A fast single image haze removal algorithm
using color attenuation prior,” IEEE Trans. Image Process., vol. 24,
no.11, pp. 3522-3533, 2015.
[21] K. Tang, J. Yang, and J. Wang, “Investigating haze-relevant features
in a learning framework for image dehazing,” in Proc. of IEEE Conf.
Comput. Vis. Pattern Recognit., 2014, pp. 2995-3002.
[22] B. Cai, X. Xu, K. Jia, and et al., “DehazeNet: an end-to-end system for
single image haze removal,” IEEE Trans. Image Process., vol. 25, no.
11, pp. 5187-5198, 2016.
[23] G. Buchsbaum, “A spatial processor model for object colour perception,”
J. of The Franklin Inst., vol. 310, no. 1, pp. 1-26, 1980.
[24] E. Land. “The retinex theory of color vision,” Scientific American, 1977.
[25] M. Ebner. “Color Constancy,” Wiley 1st edition, 2007.
[26] G. Finlayson and E. Trezzi“Shades of gray and colour constancy,” in
Proc. of Color Imaging, 2004, pp. 37-41.
[27] I. Goodfellow, J. Pouget-Abadie, M. Mirza and et al., “Generative
adversarial nets,” in Proc. of Adv. in Neur. Infor. Process. Sys., 2014,
PP. 2672-268.

5

[28] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation
learning with deep convolutional generative adversarial networks,” in
Proc. of Int. Conf. Learn. R., 2016.
[29] Y. Zhu, T. Park, P. Isola, and et al., “Unpaired image-to-image translation
using cycle-consistent adversarial networks,” arXiv:1703.10593, 2017.
[30] T. Kim, M. Cha, H. Kim, and et al., “Learning to discover cross-domain
relations with generative adversarial networks,” arXiv:1703.05192, 2017.
[31] M. Liu, T. Breuel, and J. Kautz, “Unsupervised image-to-image translation networks,” arXiv:1703.00848, 2017.
[32] Z. Wang, A. Bovik, H. Kautz, and et al. “Image quality assessment: from
error visibility to structural similarity,” IEEE Trans. Image Process.,
2004.
[33] H. Zhao, O. Gallo, I. Frosio, and et al. “Loss functions for neural
networks for image processing,” arXiv:1511.08861, 2015.
[34] J. Johnson, A. Alahi, and L. Fei-Fei. , “Perceptual losses for real-time
style transfer and super-resolution” in Proc. of Eur. Conf. Comput. Vis.
, 2016, pp. 694-711.
[35] P. Isola, J. Zhu, T. Zhou, and et al., “Image-to-image translation with
conditional adversarial networks,” arXiv:1611.07004, 2016.
[36] C. Ledig, L. Theis, F. Huszar, and et al., “Photo-realistic single image superresolution using a generative adversarial network,”
arXiv:1609.04802, 2016.
[37] C. Li and M. Wand., “Precomputed real-time texture synthesis with
markovian generative adversarial networks” in Proc. of Eur. Conf.
Comput. Vis. , 2016, pp. 702-716.
[38] D. Kingma and J. Ba, “Adam: a method for stochastic optimization,”
arXiv:1412.6980, 2017.
[39] Y. Gong and F. Sbalzarini, “A natural-scene gradient distribution prior
and its application in light-microscopy image processing,” IEEE J. Sel.
Topics Signal Process., vol 10, no. 1, pp. 99-114, 2016.
[40] H. Peng, B. Li, H. Ling, and et al., “Salient object detection via structured matrix decomposition,”IEEE Trans. Pattern Anal. Mach. Intell.,
vol. 39, no. 4, pp.818-832, 2017.
[41] H. Bay, T. Tuytelaars, and L. Van, “Surf: Speeded up robust features,”
in Proc. of Eur. Conf. Comput. Vis., 2006, pp. 404-417.

