   ANN problem
  Hi everyone,I am getting my hands dirty with artificial neural networks. To start off, I am trying to do a basic handwriting recognition and got a neural network structure that learns the weights and biases by backpropagation.The learning process works when I have two outputs (like 0 and 1) however; it completely messes up when there is more than two (for instance when I try to teach 0, 1 and 2).I saved a number of samples for each number and the algorithm loops through the samples iteratively as in: first zero, update weights, first one, update weights, first two, update weights, second zero, update weights, second one etc. I also tried epoch updating where you accumulate weight changes per entire set and then update weights at the end however; that did not help either.I have checked from many sources and I'm almost completely sure that there is nothing wrong with the algorithm. Can any of you guys give an insight to why this problem may occur?Thanks.   
Quote:Original post by DynxI am getting my hands dirty with artificial neural networks. To start off, I am trying to do a basic handwriting recognition and got a neural network structure that learns the weights and biases by backpropagation.The learning process works when I have two outputs (like 0 and 1) however; it completely messes up when there is more than two (for instance when I try to teach 0, 1 and 2).I assume that you are using 0, 1 and 2 to represent 3 different letters?  If so, then you will find it difficult to train this model since your class representation implies a meaningless ordering.  The two most common ways of addressing this problem are:1. Use multiple 0/1 target variables with a single model, one for each class (though technically the last one is unnecessary)2. Build multiple multiple 0/1 models, one for each classThe advantage of solution 1 is that information (hopefully) can be shared among classes.  The disadvantage is that training is more complex, especially since it is likely that some classes will achieve optimum fit before others, implying that some classes will be underfit and others, overfit.The advantage of solution 2 is simplified training, although there is more logistical work to do outside of the models themselves, both before training and during model recall.-Will DwinnellData Mining in MATLAB;  No, I meant I actually try to make the system recognize the characters '0', '1' and '2'.I have 3 outputs if that is what you meant with no specific ordering, but ofcourse the samples are given to the system in order. I am not quite sure if I understood your suggestions though.   ;Quote:Original post by DynxNo, I meant I actually try to make the system recognize the characters '0', '1' and '2'.I have 3 outputs if that is what you meant with no specific ordering, but ofcourse the samples are given to the system in order. I am not quite sure if I understood your suggestions though.Scrambling the order of the observations is usually preferable to feeding the neural network one class at a time.I was getting at the representation of the target variables, though.  Two popular choices are: 1-of-many encoding (1 0 0 = 0, 0 1 0 = 2, 0 0 1 = 3), or simply 0/1 for each class, with separate neural networks for each class.-Will DwinnellData Mining in MATLAB;  I'm kind of curious...If you have one network per pattern you are trying to recognize, how would you handle the case of when multiple networks return 1 saying "yes it's one of my type"? (:   ;Quote:Original post by PredictorI was getting at the representation of the target variables, though.  Two popular choices are: 1-of-many encoding (1 0 0 = 0, 0 1 0 = 2, 0 0 1 = 3)...Oh I see,I'll try that and let know of the results. What do you mean by the information being shared among classes? Or how is it different than assigning one output node to one class as in ( 0 : 1 0 0, 1: 0 1 0, 2: 0 0 1 )?;Quote:Original post by Atrix256I'm kind of curious...If you have one network per pattern you are trying to recognize, how would you handle the case of when multiple networks return 1 saying "yes it's one of my type"? (:Usually, the individual, class-specific models will return a quantity between 0.0 and 1.0.  These can be combined (as by the softmax procedure) to yield probabilities for each class.-Will DwinnellData Mining in MATLAB;Quote:Original post by DynxQuote:Original post by PredictorI was getting at the representation of the target variables, though.  Two popular choices are: 1-of-n encoding (1 0 0 = 0, 0 1 0 = 2, 0 0 1 = 3)...Oh I see,I'll try that and let know of the results. What do you mean by the information being shared among classes? Or how is it different than assigning one output node to one class as in ( 0 : 1 0 0, 1: 0 1 0, 2: 0 0 1 )?The 1-of-n encoding is the same thing as using multiple output nodes.  The idea behind information sharing is that (hopefully) information learned in training one class will assist (via shared weights within the neural network) the training of other classes.  How much benefit this provides in practice, I don't know.-Will DwinnellData Mining in MATLAB;  Ok I thought so. I tried something similar to bit encoding for output values anyway and I just cannot get the errors to converge to zero. I thought at first it could be that 3 outputs may require longer time to converge and to give you an idea: I can get pretty distinct results with 50 loops through 60 sample cases (30 each) while I can't get anything at all with 1000 loops for 90 sample cases. I also randomized the ordering of the samples and still no effect.I mean I tried pretty much everything basic and I just don't get why it wouldn't work. Absolutely frustrating.[Edited by - Dynx on April 29, 2009 8:51:37 PM];  What is your input data (I'm guessing a 2D array of binary values - how many)? And how many types of data do you present? (You say there are 3 values but you mention 4 blocks of 15 sample cases.) How many hidden nodes do you have, and are they fully interconnected to inputs and outputs? Does your data set for each class include dirty or varying inputs? What's the difference between the first 60 samples and the next 30 that you add?   