ty ▁rule ▁changes ▁only ▁the ▁weights ▁that ▁were ▁not ▁previously ▁dedicated ▁to ▁storing ▁information , ▁thereby ▁reducing ▁the ▁overlap ▁in ▁representations ▁at ▁the ▁hidden ▁units . ▁Thus , ▁even ▁when ▁inputs ▁are ▁somewhat ▁similar ▁to ▁another , ▁dis similar ▁representations ▁can ▁be ▁made ▁at ▁the ▁hidden ▁layer . ▁In ▁order ▁to ▁apply ▁the ▁novel ty ▁rule , ▁during ▁learning ▁the ▁input ▁pattern ▁is ▁replaced ▁by ▁a ▁novel ty ▁vector ▁that ▁represents ▁the ▁components ▁that ▁differ . ▁The ▁novel ty ▁vector ▁for ▁the ▁first ▁layer ▁( input ▁units ▁to ▁hidden ▁units ) ▁is ▁determined ▁by ▁taking ▁the ▁target ▁pattern ▁away ▁from ▁the ▁current ▁output ▁of ▁the ▁network ▁( the ▁delta ▁rule ). ▁For ▁the ▁second ▁layer ▁( hidden ▁units ▁to ▁output ▁units ) ▁the ▁novel ty ▁vector ▁is ▁simply ▁the ▁activation ▁of ▁the ▁hidden ▁units ▁that ▁resulted ▁from ▁using ▁the ▁novel ty ▁vector ▁as ▁an ▁input ▁through ▁the ▁first ▁layer . ▁We ight ▁changes ▁in ▁the ▁network ▁are ▁computed ▁by ▁using ▁a ▁modified ▁delta ▁rule ▁with ▁the ▁novel ty ▁vector ▁replacing ▁the ▁activation ▁value ▁( sum ▁of ▁the ▁inputs ): ▁▁▁ Δ wij ▁= ▁k δ i ▁di ▁▁▁▁▁▁ Δ wij ▁= ▁weight ▁change ▁between ▁nodes ▁i ▁and ▁j ▁▁▁ ▁k ▁= ▁learning ▁rate ▁▁ δ i ▁= ▁error ▁signal ▁▁ ▁di = ▁nov ely ▁vector ▁ ▁When ▁the ▁novel ty ▁rule ▁is ▁used ▁in ▁a ▁standard ▁back prop ag ation ▁network ▁there ▁is ▁no , ▁or ▁less ened , ▁forget ting ▁of ▁old ▁items ▁when ▁new ▁items ▁are ▁presented ▁sequ entially . ▁However