▁pre - tra ined ▁network ▁the ▁represent ational ▁overlap ▁between ▁C VC s ▁was ▁reduced ▁compared ▁to ▁the ▁na ï ve ▁network . ▁The ▁pre - tra ined ▁network ▁also ▁retained ▁some ▁similarity ▁information ▁as ▁the ▁represent ational ▁overlap ▁between ▁similar ▁C VC s , ▁like ▁" J EP " ▁and ▁" ZE P ", ▁was ▁greater ▁than ▁for ▁dis similar ▁C VC s , ▁such ▁as ▁" J EP " ▁and ▁" Y UG ". ▁This ▁suggests ▁that ▁the ▁pre - tra ined ▁network ▁had ▁a ▁better ▁ability ▁to ▁general ize , ▁i . e . ▁notice ▁the ▁patterns , ▁than ▁the ▁na ï ve ▁network . ▁ ▁Most ▁important ly , ▁this ▁reduction ▁in ▁hidden ▁unit ▁activation ▁and ▁represent ational ▁overlap ▁resulted ▁in ▁significantly ▁less ▁forget ting ▁in ▁the ▁pre - tra ined ▁network ▁than ▁the ▁na ï ve ▁network , ▁essentially ▁elimin ating ▁cat ast roph ic ▁inter ference . ▁Ess entially , ▁the ▁pre - training ▁acted ▁to ▁create ▁internal ▁orthogonal ization ▁of ▁the ▁activ ations ▁at ▁the ▁hidden ▁layer , ▁which ▁reduced ▁inter ference . ▁Thus , ▁pre - training ▁is ▁a ▁simple ▁way ▁to ▁reduce ▁cat ast roph ic ▁forget ting ▁in ▁standard ▁back prop ag ation ▁networks . ▁ ▁Pse udo - rec urrent ▁networks ▁ ▁French ▁( 1 9 9 7 ) ▁proposed ▁the ▁idea ▁of ▁a ▁pseudo - rec urrent ▁back prop ag ation ▁network ▁in ▁order ▁to ▁help ▁reduce ▁cat ast roph ic ▁inter ference ▁( see ▁Figure ▁ 2 ). ▁In ▁this