2014 World Congress on Computing and Communication Technologies

Sparse Coding: A Deep Learning using Unlabeled Data for High - Level
Representation
Mrs. R. Vidya M.Sc., M.Phil,
Dr.G.M.Nasira

Asst. Prof / PG and Research Department of Computer
science
St. Joseph’s college of Arts and Science (Autonomous)
Cuddalore-1&
Research scholar in MS University, Tirunelveli
vidya.sjc@gmail.com

Asst.Prof/Department of Computer
Applications/Chikana Govt. Arts College.
Tirupur
nasiragm99@yahoo.com

Ms. R. P. Jaia Priyankka
M.Phil Computer Science Scholar
St. Joseph’s college of Arts and Science
(Autonomous)
Cuddalore-1
jaiapriyankka@yahoo.com

Abstract- Sparse coding algorithm is an learning algorithm
mainly for unsupervised feature for finding succinct, a little
above high - level Representation of inputs, and it has
successfully given a way for Deep learning. Our objective is
to use High - Level Representation data in form of unlabeled
category to help unsupervised learning task. when compared
with labeled data, unlabeled data is easier to acquire
because, unlike labeled data it does not follow some
particular class labels. This really makes the Deep learning
wider and applicable to practical problems and learning.
The main problem with sparse coding is it uses Quadratic
loss function and Gaussian noise mode. So, its performs is
very poor when binary or integer value or other NonGaussian type data is applied. Thus first we propose an
algorithm for solving the L1 - regularized convex
optimization algorithm for the problem to allow High - Level
Representation of unlabeled data. Through this we derive a
optimal solution for describing an approach to Deep learning
algorithm by using sparse code.

Optimization algorithm, which will solve this nonGaussian error and this optimization procedure can be
applied to other L1 Regularized Optimization problem and
it is especially efficient for the problems which have very
sparse optimal solution
We successfully apply this model to Deep learning
problems with unlabeled data for High - Level
Representation. labeled data for machine learning is very
difficult and expensive to obtain but the ability to use
unlabeled data makes a significance growth in the terms
of learning method[3]. To motivate our discussion let's
consider some example of "Giraffe".
In Fig 1,when certain pictures of giraffe were used it is
easy to find out labeled data whereas other type of
unlabeled data remains to be a invariant recognition in
natural images. In the image it is not easy to identify
giraffe, because it is not in the form of reality[6]. we pose
Deep learning problem mainly to machine learning
framework which will then have the capacity to make
learning easier. Our approach uses the unlabeled data to
learn high level representation.

Keywords: Deep Learning, High - Level Representation,
Sparse Coding, Unlabeled Data, Neural Network.

I.

INTRODUCTION

Deep Learning problem arises, when we have limited
labeled data for classification task and also large amount
of unlabeled data available which is very mildly related to
the task. Like labeled data, unlabeled data does not share
same class label or some distribution but with specific
Sparse coding we can convert this mildly related
unlabeled data to perform some classification task[1].
Sparse coding model input consist of real valued vector
and it can be described well using Gaussian model and
this Gaussian's Sparse coding is really best one among it.
When Sparse coding is subjected to unlabeled data it
comes to be a great problem and it turns into nonGaussian form[2].
Our parameter makes the learning problem significantly
harder. However this problem can be solved by an
optimization process of L1 - Regularized Convex
978-1-4799-2876-7/13 $31.00 © 2014
978-1-4799-2876-7/14
978-1-4799-2877-4/14
2013 IEEE
DOI 10.1109/WCCCT.2014.69

1

2

3

4

Fig 1. 1st Image is a unlabeled image which can be identified during
normal search where as other unlabeled images cannot be identified.

II.

LEARNING : UNSUPERVISED & SUPERVISED

Unsupervised Learning under machine learning refers
to be a problem of finding the structure in the unlabeled
data which is hidden. The paradigm given to the learners
124

Fig 2 . Show the differences in the causal structure in both the learning

are in the form of unlabeled data, so there are no possible
ways to find out the potential solution[10]. This totally
distinguishes the supervised learning from unsupervised
learning. Unsupervised method is intimately linked to the
inference of thickness or density in statistics. In
unsupervised methodology there are variety of techniques
which summarize the main features in data.

Latent Variables

A. Supervised Vs Unsupervised
Theoretical point of view says that Supervised and
Unsupervised learning differs mainly from their model
structure. According to supervised the observations are
separated into two types. First set is input which occurs in
the commencement and the next set is output which takes
place in the last part of causal chain. Mediating variables
can be incorporated in the model between input and
output. All the observation are assumed by latent
variables. Basically the supervised models leaves the
inputs indetermined. If input is accessible then this model
is not required. if input values are missing then we cannot
finish off anything about output. If input is modeled then
the lost input is not a difficulty because it is measured as
latent variables which approaches under unsupervised
learning[10].
Fig 2 Shows the differences in the casual structure. it is
also feasible to have a blend of the both learning because
the output observation are understood to be rooted by both
latent variables and input observation. When evaluated
with supervised learning there is additional opportunities
to study better and complex model in unsupervised
learning. This is because in supervised learning we are
trying to discover the relation between the two sets of
observation, so the complication in learning the
assignment amplified leads to fewer number of steps
between two sets. This is why supervised learning in
practice cannot find out deep hierarchies model.
In unsupervised learning it can still even carry on from
the launch of observation to the conclusion even more
deeply in the level of representation[9]. Each spared steps
hierarchy needs to find out only one thing so the learning
time enlarges which results in the raise of the level in the
model.
In the fundamental relationship between the input and
the output the observation is complex and there
approaches a big gap and it can be simply bridge using
unsupervised learning. This is evidently shown in the fig
3. Instead of finding the relationship between the model
we can build a model inside it from upwards of both sets,
which results the observation in every level of abstraction
and the gap seems to be easier to be filled.
Supervised Learning
Observations (inputs)

Observations (output)
Observations (inputs)
Fig 3. Represents that the unsupervised learning can be used to fill the
gap in-between inputs and outputs.

III. GROUND RULES
With a small labeled training set we consider a
classification problem xl(1) , y (1) , xl( 2 ) , y (2 ) ,..., xl(m ) , y ( m ) .
Every input xl(i ) ∈ χ is a class label assigned to

{(

) (

)}

y (i ) ∈ y = {1,2,..., k }. For the use of non-Gaussian data, the

binary vector can be represented as xl(i ) ∈ χ = {0,1}k or
integer vector xl(i ) ∈ χ = {0,1,2...}k . For the Deep Framework
we should also assume like large set of unlabeled data
without subscript x (1) , x (2 ) ,..., x (r ) .

{

}

IV. SPARSE CODING - GAUSSIAN
Generally Gaussian's Sparse uses unlabeled input
vector x ∈ ℜ k to identify basis b1 , b2 ,..., bn ∈ ℜ k vector
where x is represented as linear combination of basis
vector x ≈ ¦ b j s j . Here s ∈ ℜ n called activations for
j
the input x [5] . Vector x is originated from Gaussian
distribution mean η = ¦ b j s j and σ 2 Ι covariance. It
j

follows Laplacian Prior P (s )∞ ∏ j exp (− β | s j |) some

constant β . Given unlabeled {x1 , x 2 ,..., x (r ) } corresponding
activations

{s( ) ,..., s ( )}
i

r

and the basic vector {b j } is

obtained by

max
{ }{ }
b j , s (i )

∏ P(x ( ) | {b }, {s ( )})P(s ( ) )
i

i

i

j

i

The Optimization Problem

Unsupervised Learning
Latent Variable

min
{ }{ }
b j , s(i )

1
2σ

¦ ||x ( ) − ¦
i

2

i

n
j =1

b j s (ji ) ||2 + β ¦i , j | s (ji ) |

|| b j ||2 ≤ C , ∀ j = 1,..., n
This optimization is Convex for the basic vector and it
helps the use of unlabeled data mainly for High - Level
Representation[7].

Subj.To

Observations (output)

Observations

125

V.

DISCRETE INPUTS FOR SELF TAUGHT
LEARNING

Here

xu(i )

To apply the above algorithm to Deep Learning,
binary input vector x ∈ {0,1}k . Probabilistic assumption for
Gaussian Sparse P x | η = ¦ b j s j . It is a Poor fit to non-

(

Deep Algorithm using Sparse Coding:

)

j

input Labeled Set

Gaussian data. To enforce decomposition x ≈ ¦ b j s j .
j

Thus it leads to useful basic vector x ≈ σ

{(

j

1
ª 1
º
,
,... . This makes a promise to better
−v
−v
¬1 + e 1 1 + e 2 »¼

G
s := 0 .
Β ∈ ℜ k ×n , x ∈ ℜ k , Threshold ∈

( )

aˆ xl(i ) = arg min a (i ) xl(i ) − ¦ j a (ji )b j 22 + β a (i )

with

Λ ii = a '' ((Bs )i )

return learned classifier c

VIII.

sˆ = arg min s ' || Λ1/ 2 Βs ' − Λ1/ 2 z ||2 + β || s ' ||1
set s := (1 − t ) s + tsˆ where t is established by

DEEP LEARNING APPLICATION

{(x ( ) , y ( ) ), (x ( ) , y ( ) ),..., (x (

)}

labeled

set

xl ∈ ℜ ,where
"l" indicates labeled example. Given a set k unlabeled
1

1

2

l

2

l

l

m)

, y (m ) . Each

n

example xu(1) , xu(2 ) ,..., xu(k ) ∈ ℜ n
Given labeled and unlabeled set Deep Learning
proposes a hypothesis h : ℜ n → {1,..., C}

VII.

HIGHER - LEVEL REPRESENTATION

We use a little bit of modified version of Sparse
coding algorithm for learning Higher -Level
Representations[8].
Given unlabeled data {xu(1) ,..., xu(k ) } ,each xu(i ) ∈ ℜ n .
The optimization problem

min b,a

¦ || x ( ) − ¦

Subj . to

|| b j ||2 ≤ 1, ∀ j ∈1,..., s

i

i
u

j

vector each b j ∈ ℜ
(i )

each a ∈ ℜ

n

{

(1)

IX.

a (ji )b j ||22 + β || a (i ) ||1

The optimization variables b = {b1 , b2 ,..., bs }
and a = a ,..., a

(k )

}

CLASSIFIER : NEURAL NETWORK

Our goal is to use Sparse coding and to find high Level representation of unlabeled data in deep learning.
The task is to find the best classifier for the
implementation. This approach is mainly used for Sparse
algorithm which is designed for deep learning.
Neural Network Classifier Consists of Neurons i.e
Units. The units are arranged in a layer which converts the
input vector into output. Each and every unit first takes an
input and applies some functions on it and then it passes
the input to next level. Networks are said to be feed forward, which means it feeds the first input's output to
the next level which is input to the next level. It does not
gives the feedback to the preceding stage so it is entitled
as feeding forward. weightings is also applied to the each
units signals which passes from one level to another.
These weighting actually help or tune the particular
problem to adapt the neural network.
Using the Neural Network classifier data is applied in
the Matlab. To increase the accuracy in classifier the input
latent variables is reduced as vector for preserving
correlations which are important for the original data set.
By reducing the Dimensionality in the sample data the
result seems to be more accurate.

backtracking line-search to reduce the objective function
end while

(i )

1

c is obtained by applying algorithm, to the
unlabeled training set Tˆ
a classifier

Computational Vector z = Λ−1 (T (x ) − a ' (Bs )) + Bs
Initialize search feature - sign at s, compute

m

)}

where

While objective value decrease at last step >∈ do
Diagonal
Matrix
Λ compute

given

(k )

i =1

l

Input: Initialize

Learning

(2 )

problem to obtain b
compute feature for classification task for new unlabeled
training set Tˆ = {(aˆ (x (i ) ), y (i ) )} m

L1 - Regularized Convex Optimization Algorithm

Deep

(1)

{ }

form of Sparse Algorithm[4]

In

) (

Unlabeled data {xu , xu ,..., xu }
Output a Learned Classifier for classification task
Algorithm: unlabeled data xu(i ) solve the optimization

j

σ (v ) = «

VI.

)(

T = xl(1) , y (1) , xl(2 ) , y (2 ) ,..., xl(m ) , y (m )

(¦ b s ), where
j

a (ij ) is activation of b j (basis) for input

DISCUSSION

In this paper we presented a general method to use
Sparse coding for unsupervised learning with the help of
unlabeled data mainly for High - Level Representation.
Since Sparse coding does not support unlabeled data or
non-Gaussian data we derived an algorithm for the
optimization usage which came out as L1-regularized
convex optimization algorithm. With the help of this

is basis

is activation

s

126

algorithm we derived an Deep Learning algorithm for
Higher -Level Representation. Overall our result suggest
that L1- regularized convex optimization algorithm can
help Higher - Level Representation of documents form
unlabeled data and this knowledge is useful in
classification problem. We believe this model could be
applied generally to other problems, where large amount
of unlabeled data are available.
X.

FUTURE WORK

The model represented in the paper consists of two
algorithm, which indirectly or directly related the
Gaussian Sparse coding with Deep Learning. We have
only implemented the model to the images, whereas we
like to extend it to Text, Audio and Video related data. It
really Seems to be challenging yet this extension when
fully related and applied to all the types of multimedia
content, It can be applied in Robotics and mainly in their
Perception Task.

REFERENCES
[1]

[2]
[3]

[4]
[5]
[6]

[7]
[8]

[9]

[10]

Brian Quanz, Jun Huan, Meenakshi Mishra " Knowledge
Transfer with Low-Quality Data : a Feature Extraction
Issue" University of Kansas.
Andreas, Pontil, Bernardino, " Sparse Coding for
Multitasking & Transfer Learning" University of london.
Andro, R.K., & Zhang, T.(2005). "A framework for
learning predictive structures from multiple tasks and
unlabeled data." JMLR,6, 1817 - 1853
Ng, A. Y. (2004). Feature Selection, "L1 vs L2
regularization, and rotational invariances." ICML
Lee, H., Battle, A., Raina, R., & Ng,A. Y. (2007).
"Efficient sparse coding algorithm". NIPS.
R.Raina, A.Battle, H.Lee, B.Packer, and A. Y. Ng. "Self Taught learning: Transfer learning from unlabeled data". In
ICML, 2007
S.Boyd and L. Vandenberghe. "Convex Optimization".
Cmbridge University Press 2004
Lee, Raina, Teichman, Ng "Exponential family sparse
coding with applications to self - taught learning" IJCAI
2009
Lee, Grosse, Ranganath, Ng "Deep Belief Network for
scalable
unsupervised learning of hierarchical
representations" ICML 2009
Socher, R., Manning, C.D., and Ng, A.Y. "Unsupervised
Feature Learning and Deep Learning". In Deep learning
and unsupervised feature learning workshop, 2010

127

