, ▁this ▁rule ▁can ▁only ▁apply ▁to ▁auto - enc oder ▁or ▁auto - associ ative ▁networks , ▁in ▁which ▁the ▁target ▁response ▁for ▁the ▁output ▁layer ▁is ▁identical ▁to ▁the ▁input ▁pattern . ▁This ▁is ▁because ▁the ▁novel ty ▁vector ▁would ▁be ▁meaning less ▁if ▁the ▁desired ▁output ▁was ▁not ▁identical ▁to ▁the ▁input ▁as ▁it ▁would ▁be ▁impossible ▁to ▁calculate ▁how ▁much ▁a ▁new ▁input ▁differ ed ▁from ▁the ▁old ▁input . ▁ ▁Pre - training ▁networks ▁ ▁Mc R ae ▁and ▁H ether ington ▁( 1 9 9 3 ) ▁argued ▁that ▁humans , ▁unlike ▁most ▁neural ▁networks , ▁do ▁not ▁take ▁on ▁new ▁learning ▁tasks ▁with ▁a ▁random ▁set ▁of ▁weights . ▁R ather , ▁people ▁tend ▁to ▁bring ▁a ▁wealth ▁of ▁prior ▁knowledge ▁to ▁a ▁task ▁and ▁this ▁helps ▁to ▁avoid ▁the ▁problem ▁of ▁inter ference . ▁They ▁proposed ▁that ▁when ▁a ▁network ▁is ▁pre - tra ined ▁on ▁a ▁random ▁sample ▁of ▁data ▁prior ▁to ▁starting ▁a ▁sequ ential ▁learning ▁task ▁that ▁this ▁prior ▁knowledge ▁will ▁naturally ▁const rain ▁how ▁the ▁new ▁information ▁can ▁be ▁incorpor ated . ▁This ▁would ▁occur ▁because ▁a ▁random ▁sample ▁of ▁data ▁from ▁a ▁domain ▁which ▁has ▁a ▁high ▁degree ▁of ▁internal ▁structure , ▁such ▁as ▁the ▁English ▁language , ▁training ▁would ▁capture ▁the ▁regular ities , ▁or ▁rec urr ing ▁patterns , ▁found ▁within ▁that ▁domain . ▁Since ▁the ▁domain ▁is ▁based ▁on ▁regular ities , ▁a ▁newly ▁learned ▁item ▁will ▁tend ▁to ▁be ▁similar ▁to ▁the ▁previously ▁learned ▁information ,