A Generalization Method of
arXiv:1802.02987v1 [cs.NE] 8 Feb 2018

Partitioned Activation Function for
Complex Number
HyeonSeok Lee ∗, Hyo Seon Park

†

February 9, 2018

A method to convert real number partitioned activation function into complex
number one is provided. The method has 4em variations; 1 has potential to get
holomorphic activation, 2 has potential to conserve complex angle, and the last 1
guarantees interaction between real and imaginary parts. The method has been
applied to LReLU and SELU as examples.
The complex number activation function is an building block of complex number ANN, which has potential to properly deal with complex number problems.
But the complex activation is not well established yet. Therefore, we propose a
way to extend the partitioned real activation to complex number.

∗
†

1st author: Yonsei Univ, Seoul, Korea
Corresponding Author: Yonsei Univ, Seoul, Korea, hspark@yonsei.ac.kr

1

1 Motivation
Complex Number Artificial Neural Net (CANN) is a natural consequence of using ANN for
complex number, while complex number appear in many problems. The complex number
problems include (Fourier, Laplace, Z, etc) transform based methods [Haberman, 2013] ,
steady-state problems [Banerjee, 1994] , mapping 2D problem to complex plane [Greenberg,
1998] , and electromagnetic signals such as MRI image [Virtue et al., 2017] . Due to such necessity, CANN appears from 1990’s even long before the days of deep learning, as discussed in
[Reichert and Serre, 2013] . As the result of various efforts, building blocks of CANN has been
developed; Convolution, Fully-connection, Back-propagation, Batch-normalization, Initialization, Pooling, and Activation. Among them, Complex Convolution and Fully-connection
are essentially multiplication and addition, which are clearly established for complex number
in mathematics [Trabelsi et al., 2017] . Back-propagation for complex number is developed
for general complex number function [Georgiou and Koutsougeras, 1992, Nitta, 1997], and
for holomorphic function [La Corte and Zou, 2014] . Complex Batch-normalization is also
developed following the idea of real Batch-normalization [Trabelsi et al., 2017] . The complex
number Initialization is suggested to be done in polar-form [Trabelsi et al., 2017] , not Cartesian. Max Pooling for complex number is another necessity, but the authors cannot see any
research. Complex Activation is in active development [Arjovsky et al., 2015, Guberman,
2016, Virtue et al., 2017, Georgiou and Koutsougeras, 1992]; see [La Corte and Zou, 2014]
for brief review about various types of complex activation functions.
For complex activation, holomorphic (a.k.a. analytic) function has been an important
topic [Hitzer, 2013, Jalab and Ibrahim, 2011, Vitagliano et al., 2003, Kim and Adali, 2001,
Mandic and Goh, 2009, Tripathi and Kalra, 2010] , since it makes the back-propagation
simpler thus training becomes faster [Amin et al., 2011, La Corte, 2014] . The importance of
Complex Phase Angle of CANN has been suggested, including similarity to biological neuron
[Reichert and Serre, 2013]. The importance leads to the idea of phase-preserving complex
activation function [Georgiou and Koutsougeras, 1992, Virtue et al., 2017]. In contrast,

2

[Hirose, 2013, Kim and Adalı, 2003] suggested that phase-preservation makes training more
difficult.
We focus on the complex activation, and propose a methods to generalize partitioned
activation such as LReLU and SELU for complex number. There are 4 variations in the
method to accommodate various cases; 1 of them is potentially holomorphic and alter phase,
1 is not holomorphic and alter phase while guarantees interaction between real and imaginary
parts, and 2 are not holomorphic and potentially keep complex phase angle.
This article is structured as follows. Sec. 2 is a brief review about existing Complex
Activation functions. Then Sec. 3 presents our generalization method along with examples
of LReLU [Maas et al., 2013] and SELU [Klambauer et al., 2017] . Finally, Sec. 4 summarize
the article.

2 Review of Current Complex Activation Functions
Certain complex-argument activation functions are derived from real-argument counterparts,
using 3 ways depending on the characteristic of each function; No change, Modification, and
generalization. We will briefly discuss them with particular weight on the generalization,
since our approach is to generalize.

2.1 No change
Certain no-partitioned activation functions can be used for complex-argument without any
change. Such functions include logistic, tanh, tan−1 , etc. These are already defined for
complex-argument, and we can just use them. But they suffer from poles near the origin
[La Corte and Zou, 2014] .

3

2.2 Modification
Partitioned activation functions are not straightforward for complex-argument, due to the
partition points on the real-axis. So approach of “just take the idea of real activation, and
create a corresponding complex activation” is tried. We call it modification. One example is
modReLU [Arjovsky et al., 2015] as
modReLU(z) = ReLU (|z| + b) ei θ

z


(|z| + b)
|z| + b ≥ 0
|z|
=


0
|z| + b < 0

(1)

where z ∈ C, θ is the phase angle of z, and b ∈ R is a trainable parameter. The modReLU
takes the idea of inactive region from ReLU, and forms region around origin. Unfortunately,
the modReLU is not holomorphic, while keeps the complex phase.

2.3 Generalization
We call a complex activation Generalized , if the values on real-axis coincide with the realargument counter-part. The easiest generalization of partitioned activation function would
be to separately apply activation function to real and imaginary parts [Nitta, 1997, Faijul Amin and Murase, 2009] . One example is Separate Complex ReLU (SCReLU) as
SCReLU(z) = ReLU (x) + i ReLU (y)

(2)

where z = x + i y . Another simple generalization of ReLU is to activate only when both
real and imaginary parts are positive, which is called zReLU [Guberman, 2016]



z x > 0, y > 0
zReLU(z) =


0 otherwise

(3)

Both SCReLU and ReLU are holomorphic, and are essentially not complex but 2 separate

real activations [La Corte and Zou, 2014] . Another generalization of ReLU is Complex
Cardioid (CC) [Virtue et al., 2017]

4

CC(z) =

1
{1 + cos θ} z
2

(4)

which keeps the complex angle, but is not holomorphic. Our approach is inspired by the
CC.

3 New Generalization Method
Consider a partitioned activation function for real number, which has the typical form



f0 (x) x < 0
f (x) =
(5)


f1 (x) x ≥ 0
where f0 and f1 are local functions for positive and negative regions. Activation functions

like LReLU and SELU are examples of the above case. Now, replace partitions with Heaviside
unit-step function H(x), and we can rewrite f (x) as
f (x) = f0 (x) H(−x) + f1 (x) H(x)

(6)

Next, select a complex-argument function, whose real axis values coincide with the H(x)
and H(−x) . We pick

1
1 + ei (2n+1)θ → H(x)
2

(7a)


1
1 − ei (2n+1)θ → H(−x)
2

(7b)

where θ is the phase angle of complex number z = x + i y, and n ∈ Z (is an integer) is a
parameter. Then, we can get a generalized complex-argument function
f (z) =



1
1
1 − ei (2n+1)θ f0 (z) +
1 + ei (2n+1)θ f1 (z) (z)
2
2

(8)

This approach can be easily extended to more complicated cases, like S-shaped ReLU
(SReLU) which has 3 partitions. The typical form

5

f (x) =

is rewritten as





f0 (x)








f1 (x)





x < x0
x0 < x ≤ x1

..
.







fn−1 (x) xn−2 < x ≤ xn−1







fn (x)
xn−1 < x

f (x) = f0 (x) H(x − x0 ) + f1 (x) {H(x − x0 ) − H(x − x1 )} + · · ·
+ fn−1 (x) {H(x − xn−2 ) − H(x − xn−1 )} + fn (x) H(−x + xn−1 )

(9)

(10)

, then generalized to

f (z) =



1
1 i (2n+1)θ1
1 − ei (2n+1)θ0 f0 (z) +
e
− ei (2n+1)θ0 f1 (z) + · · ·
2
2


1 i (2n+1)θn−1
1
+
e
− ei (2n+1)θn−2 fn−1 (z) +
1 + ei (2n+1)θn−1 fn (z)
2
2

(11)

where θp denotes the phase angle of complex number zp = z − xp = (x − xp ) + i y, and xp
is the location of partition boundary p.
In case real valued scale is preferred (eg: to keep the complex angle), we made simple
modification for making replacement of H(x) real valued. The modifications are
1
[1 + cos {(2n + 1) θ}] → H(x)
2

(12a)

1
|1 + ei (2n+1)θ | → H(x)
2

(12b)

and

and generalization using them are
f (z) =

1
1
[1 − cos {(2n + 1) θ}] f0 (z) + [1 + cos {(2n + 1) θ}] f1 (z)
2
2

(13a)

and
1
1
f (z) = |1 − ei (2n+1)θ | f0 (z) + |1 + ei (2n+1)θ | f1 (z) (z)
2
2

6

(13b)

respectively.
Another approach is to approximate H(x − xp ) on real axis with a complex-argument
function. A few well known such functions are based on tanh (hyperbolic tangent) , tan−1
(arc tangent), Si (sine integral), and erf (error function) functions. Among them, we pick
erf based Sigmoid function as below, since tanh and tan−1 has poles on imaginary axis, and
Si is oscillatory on real axis.
1
S(z) =
2





z
1 + erf √
2σ



(14)

where σ > 0 is a just a parameter such that smaller σ more closely approximates H(x) on
the real axis. Then approximately generalized functions are
fe(z) = S(−z) f0 (z) + S(z) f1 (z)

(15)

fe(z) = S(−z0 ) f0 (z) + (S(z1 ) − S(z0 )) f1 (z) + · · · + (S(zn−1 ) − S(zn−2 )) fn−1 (z) + S(zn ) fn (z)
(16)
respectively for (15) and (16) cases. An important advantage of this approximate generalization is that the fe(z) is holomorphic , if all the fp (z) are holomorphic. The reason is

simple; a) logistic function is holomorphic, b) product and sum of holomorphic functions are
also holomorphic. Then, each terms in (15) and (16) are holomorphic which are products

of holomorphic, and fe(z) is holomorphic which is sum of holomorphic. With using normal-

ization (eg: Batch-Renormalization [Ioffe, 2017]) is suggested to prevent exploding feature

values toward ± i ∞.

3.1 Example 1: Leaky Rectified Linear Unit (LReLU)
The LReLU [Maas et al., 2013] is the one of the most popular activation function (The ReLU
is just a LReLU with α = 0). The real-argument LReLU is



αx x < 0
LReLU(x) =


x
x≥0
7

(17)

and is generalized using (8) with n=0 to “Complex LReLU” (CLReLU) as
1
CLReLU(z) =
(1 + α) + (1 − α) ei θ z
2

(18)

which alters both phase and magnitude of input z and has cross-influence between real
and imaginary components. To keep the complex angle of argument, we use (13a) and (13b)
with n=0 to get
1
{(1 + α) + (1 − α) cos θ} z
2

(19a)


1
α|1 − ei (2n+1)θ | + |1 + ei (2n+1)θ | z
2

(19b)

cLReLU(z) =
and
aLReLU(z) =

which we call “cos LReLU” (cLReLU) and “abs LReLU” (aLReLU), respectively. Meanwhile, a “Holomorphic LReLU” (HLReLU) can be derived using (15) as



z
1
z
(1 + α) + (1 − α) erf √
HLReLU(x) =
2
2σ

(20)

which is holomorphic, since both the f0 (z) = αz and f1 (z) = z are polynomials which
are all holomorphic. Please note that the HLReLU alters phase angle of argument, since the
argument z is multiplied with a complex number.

3.2 Example 2: Scaled Exponential Linear Unit (SELU)
The SELU [Klambauer et al., 2017] was recently developed, and rapidly becomes popular
due to its self-normalizing feature (The ELU is just a SELU with λ = 1 ). The real-argument
SELU is
SELU(x) = λ




α(ex − 1) x < 0


x

(21)

x≥0

where λ > 1and α are parameters with optimum values λ = 1.0507 and α = 1.67326. The
SELU is generalized using (8), (13a), and (13b) with n=0 to “Complex SELU” (CSELU),
“cos SELU” (cSELU) and, “abs SELU” (aSELU), respectively as


λ
CSELU(z) =
1 − ei θ α(ez − 1) + 1 + ei θ z
2
8

(22a)

cSELU(z) =

λ
{(1 − cos θ) α(ez − 1) + (1 + cos θ) z}
2

(22b)

λ
|1 − ei θ |α(ez − 1) + |1 + ei θ | z
2

(22c)

aSELU(z) =

Complex phase is altered with all 3 CSELU, cSELU, and aSELU, unlike cLReLU (19a)
and aLReLU (19b). Instead, all has interaction between real and imaginary components of
argument.
Meanwhile, a Holomorphic “SELU” (HSELU) can be derived using (15) as
HSELU(z) = λ {S(−z) α(ez − 1) + S(z) z}

(23)

which is also holomorphic, since both the scaled shifted exponential f0 (z) = λα(ez − 1)
and polynomial f1 (z) = λz are holomorphic.

4 Concluding Remark
A generalization of partitioned real number activation function to complex number has been
proposed. The generalization process has 2 steps; a) Replace partition on activation with
H(x), b) Generalize H(x) with a complex number function. The generalization has 4 variations, and 1 of them is potentially holomorphic. The generalization scheme has been
demonstrated using 2 popular partitioned activations; LReLU and SELU. The properties
of generalized complex activations are summarized on table (1) .
Furthermore, the method can be used for various partitioned real activation to make them
into complex activation. He hope that this humble research adds another building block for
complex ANN, which is important for complex number problems.

References
Md Faijul Amin, Muhammad Ilias Amin, A. Y. H. Al-Nuaimi, and Kazuyuki Murase.
Wirtinger Calculus Based Gradient Descent and Levenberg-Marquardt Learning Algorithms in Complex-Valued Neural Networks. In Neural Information Processing, Lecture

9

Original Real
Activation

Generalized Complex
Activation
CLReLU (18)
cLReLU (19a)
aLReLU (19b)
HLReLU (20)
CSELU (22a)
cSELU (22b)
aSELU (22c)
HSELU (23)

LReLU (17)

SELU (21)

Holomorphic

Real-Complex
Interaction
O
X
X
O
O
O
O
O

X
X
X
O
X
X
X
O

Phase
Preverving
X
O
O
X
X
X
X
X

Table 1: Properties of the Generalized Complex Activation
Notes in Computer Science, pages 550–559. Springer, Berlin, Heidelberg, November 2011.
ISBN 978-3-642-24954-9 978-3-642-24955-6. doi: 10.1007/978-3-642-24955-6_66. URL
https://link.springer.com/chapter/10.1007/978-3-642-24955-6_66. 1
Martin Arjovsky,

Amar Shah, and Yoshua Bengio.

rent Neural Networks.

arXiv:1511.06464 [cs, stat],

Unitary Evolution RecurNovember 2015.

URL

http://arxiv.org/abs/1511.06464. arXiv: 1511.06464. 1, 2.2
Prasanta Kumar Banerjee. The Boundary Element Methods in Engineering. Mcgraw-Hill
College, London ; New York, rev sub edition edition, January 1994. ISBN 978-0-07-7077693. 1
Md. Faijul Amin and Kazuyuki Murase.

Single-layered complex-valued neural
Neurocomputing, 72(4):945–955,

network for real-valued classification problems.
January 2009.

ISSN 0925-2312.

doi:

10.1016/j.neucom.2008.04.006.

URL

http://www.sciencedirect.com/science/article/pii/S0925231208002439. 2.3
G. M. Georgiou and C. Koutsougeras.

Complex domain backpropagation.

IEEE

Transactions on Circuits and Systems II: Analog and Digital Signal Processing,
39(5):330–334, May 1992.

ISSN 1057-7130.

doi:

http://ieeexplore.ieee.org/document/142037/. 1

10

10.1109/82.142037.

URL

Michael Greenberg.

Advanced Engineering Mathematics.

Prentice Hall, Upper Saddle

River, N.J, 2 edition edition, January 1998. ISBN 978-0-13-321431-4. Google-Books-ID:
CpoUngEACAAJ. 1
Nitzan Guberman. On Complex Valued Convolutional Neural Networks. arXiv:1602.09046
[cs], February 2016. URL http://arxiv.org/abs/1602.09046. arXiv: 1602.09046. 1,
2.3
Richard Haberman. Applied Partial Differential Equations: With Fourier Series and Boundary Value Problems. PEARSON, 2013. ISBN 978-0-321-79705-6. Google-Books-ID: hGNwLgEACAAJ. 1
Akira Hirose, editor. Complex-valued neural networks: advances and applications. IEEE
Press series on computational intelligence. John Wiley & Sons Inc, Hoboken, N.J, 2013.
ISBN 978-1-118-34460-6. OCLC: ocn812254892. 1
Eckhard Hitzer. Non-constant bounded holomorphic functions of hyperbolic numbers - Candidates for hyperbolic activation functions. arXiv:1306.1653 [cs, math], June 2013. URL
http://arxiv.org/abs/1306.1653. arXiv: 1306.1653. 1
Sergey Ioffe.

Batch Renormalization:

in Batch-Normalized Models.

Towards Reducing Minibatch Dependence

arXiv:1702.03275 [cs],

February 2017.

URL

http://arxiv.org/abs/1702.03275. arXiv: 1702.03275. 3
Hamid A. Jalab and Rabha W. Ibrahim. New activation functions for complex-valued neural
network. International Journal of Physical Sciences, 6(7):1766–1772, 2011. 1
Taehwan Kim and Tülay Adali. Complex backpropagation neural network using elementary
transcendental activation functions. In Acoustics, Speech, and Signal Processing, 2001.
Proceedings.(ICASSP’01). 2001 IEEE International Conference on, volume 2, pages 1281–
1284. IEEE, 2001. 1

11

Taehwan

Kim

tilayer

and

Tülay

Adalı.

Neural

perceptrons.

Approximation

computation,

by

fully

15(7):1641–1666,

complex
2003.

mulURL

https://www.mitpressjournals.org/doi/abs/10.1162/089976603321891846. 1
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter.
Normalizing Neural Networks.

arXiv:1706.02515 [cs, stat], June 2017.

SelfURL

http://arxiv.org/abs/1706.02515. arXiv: 1706.02515. 1, 3.2
Diana Thomson La Corte. Newton’s Method Backpropagation for Complex-Valued Holomorphic Neural Networks: Algebraic and Analytic Properties. Theses and Dissertations,
August 2014. URL https://dc.uwm.edu/etd/565. 1
Diana Thomson La Corte and Yi Ming Zou. Newton’s Method Backpropagation for ComplexValued Holomorphic Multilayer Perceptrons. 2014 International Joint Conference on Neural Networks (IJCNN), pages 2854–2861, June 2014. doi: 10.1109/IJCNN.2014.6889384.
URL http://arxiv.org/abs/1406.5254. arXiv: 1406.5254. 1, 2.1, 2.3
Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. Rectifier nonlinearities improve
neural network acoustic models. In Proc. icml, volume 30, page 3, 2013. 1, 3.1
Danilo P. Mandic and Vanessa Su Lee Goh. Complex Valued Nonlinear Adaptive Filters:
Noncircularity, Widely Linear and Neural Models. John Wiley & Sons, April 2009. ISBN
978-0-470-74263-1. Google-Books-ID: MaW8MaIkztUC. 1
Tohru

Nitta.

Complex
1997.

An

Extension

Numbers.
ISSN

0893-6080.

of

Neural

the

Back-Propagation

Networks,

doi:

Algorithm

10(8):1391–1415,

to

November

10.1016/S0893-6080(97)00036-1.

http://www.sciencedirect.com/science/article/pii/S0893608097000361.

URL
1,

2.3
David P. Reichert and Thomas Serre.

Neuronal Synchrony in Complex-Valued

12

Deep Networks.

arXiv:1312.6115 [cs,

q-bio,

stat],

December 2013.

URL

http://arxiv.org/abs/1312.6115. arXiv: 1312.6115. 1
Chiheb Trabelsi, Olexa Bilaniuk, Ying Zhang, Dmitriy Serdyuk, Sandeep Subramanian,
João Felipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher J. Pal.

Deep Complex Networks.

arXiv:1705.09792 [cs], May 2017.

URL

http://arxiv.org/abs/1705.09792. arXiv: 1705.09792. 1
B. K. Tripathi and P. K. Kalra. High Dimensional Neural Networks and Applications.
In Intelligent Autonomous Systems, Studies in Computational Intelligence, pages 215–
233. Springer, Berlin, Heidelberg, 2010.

ISBN 978-3-642-11675-9 978-3-642-11676-6.

URL https://link.springer.com/chapter/10.1007/978-3-642-11676-6_10.

DOI:

10.1007/978-3-642-11676-6_10. 1
Patrick Virtue, Stella X. Yu, and Michael Lustig.

Better than Real:

Complex-

valued Neural Nets for MRI Fingerprinting. arXiv:1707.00070 [cs], June 2017. URL
http://arxiv.org/abs/1707.00070. arXiv: 1707.00070. 1, 2.3
Francesca Vitagliano, Raffaele Parisi, and Aurelio Uncini. Generalized splitting 2d flexible
activation function. In Italian Workshop on Neural Nets, pages 85–95. Springer, 2003. 1

13

