▁will ▁be ▁encoded ▁by ▁the ▁same ▁hidden ▁units ▁and ▁weights , ▁and ▁so ▁will ▁decrease ▁the ▁chance ▁of ▁inter ference . ▁ ▁Indeed , ▁a ▁number ▁of ▁the ▁proposed ▁solutions ▁to ▁cat ast roph ic ▁inter ference ▁involve ▁reducing ▁the ▁amount ▁of ▁overlap ▁that ▁occurs ▁when ▁storing ▁information ▁in ▁these ▁weights . ▁ ▁Many ▁of ▁the ▁early ▁techniques ▁in ▁reducing ▁represent ational ▁overlap ▁involved ▁making ▁either ▁the ▁input ▁vectors ▁or ▁the ▁hidden ▁unit ▁activation ▁patterns ▁orthogonal ▁to ▁one ▁another . ▁Lew and ow sky ▁and ▁Li ▁( 1 9 9 5 ) ▁noted ▁that ▁the ▁inter ference ▁between ▁sequ entially ▁learned ▁patterns ▁is ▁minim ized ▁if ▁the ▁input ▁vectors ▁are ▁orthogonal ▁to ▁each ▁other . ▁Input ▁vectors ▁are ▁said ▁to ▁be ▁orthogonal ▁to ▁each ▁other ▁if ▁the ▁pair wise ▁product ▁of ▁their ▁elements ▁across ▁the ▁two ▁vectors ▁sum ▁to ▁zero . ▁For ▁example , ▁the ▁patterns ▁[ 0 , 0 , 1 , 0 ] ▁and ▁[ 0 , 1 , 0 , 0 ] ▁are ▁said ▁to ▁be ▁orthogonal ▁because ▁( 0 × 0 ▁+ ▁ 0 × 1 ▁+ ▁ 1 × 0 ▁+ ▁ 0 × 0 ) ▁= ▁ 0 . ▁One ▁of ▁the ▁techniques ▁which ▁can ▁create ▁orthogonal ▁representations ▁at ▁the ▁hidden ▁layers ▁involves ▁b ip olar ▁feature ▁coding ▁( i . e ., ▁coding ▁using ▁- 1 ▁and ▁ 1 ▁rather ▁than ▁ 0 ▁and ▁ 1 ). ▁Orth ogonal ▁patterns ▁tend ▁to ▁produce ▁less ▁inter ference ▁with ▁each ▁other . ▁However , ▁not ▁all ▁learning ▁problems