 Occam algorithm for ${\bf R}$.

Let a representation system ${\bf R}$ be polynomially closed under either exceptions or majority-of-three, or both. Then ${\bf R}$ is deterministically polynomially pac-learnable iff there is a polynomial time Occam algorithm for ${\bf R}$.

[*Example.*]{} Consider threshold circuits, acyclic circuits whose nodes compute threshold functions of the form $a_1x_1 + a_2x_2 + \cdots +a_nx_n \geq \delta$, $x_i \in \{0,1\}, a_i,\delta \in N$ (note that no expressive power is gained by allowing rational weights and threshold). A simple way of representing circuits over the binary alphabet is to number each node and use [*prefix-free encodings*]{} of these numbers. For instance, encode $i$ as $1^{|\mbox{bin}(i)|}0\mbox{bin}(i)$, the binary representation of $i$ preceded by its length in unary. A complete node encoding then consists of the encoded index, encoded weights, threshold, encoded degree, and encoded indices of the nodes corresponding to its inputs. A complete circuit can be encoded with a node-count followed by a sequence of node-encodings. For this representation, a majority-of-three algorithm is easily constructed that renumbers two of its three input representations, and combines the three by adding a 3-input node computing the majority function $x_1+x_2+x_3 \geq 2$. It is clear that under this representation, the system of threshold circuits are polynomially closed under majority-of-three. On the other hand they are not closed under exceptions, or under the exception lists of [@board].

[*Example.*]{} Let $h_1, h_2, h_3$ be 3 $k$-DNF formulas. Then ${\mbox{MAJ}}(h_1,h_2,h_3) = (h_1 \wedge h_2) \vee (h_2 \wedge h_3) \vee
(h_3 \wedge h_1)$ which can be expanded into a $2k$-DNF formula. This is not good enough for Theorem \[maj\], but it allows us to conclude that pac-learnability of $k$-DNF implies compression of $k$-DNF into $2k$-DNF.
Occam algorithm for $ { \bf R}$. 

 Let a representation system $ { \bf R}$ be polynomially closed under either exception or majority - of - three, or both. Then $ { \bf R}$ is deterministically polynomially pac - learnable iff there be a polynomial time Occam algorithm for $ { \bf R}$. 

 [ * Example. * ] { } Consider doorsill circuits, acyclic circuits whose node calculate threshold functions of the form $ a_1x_1 + a_2x_2 + \cdots + a_nx_n \geq \delta$, $ x_i \in \{0,1\ }, a_i,\delta \in N$ (note that no expressive power is profit by allowing rational weights and threshold). A simple way of represent circuits over the binary alphabet is to number each lymph node and use [ * prefix - free encoding * ] { } of these numbers. For example, encode $ i$ as $ 1^{|\mbox{bin}(i)|}0\mbox{bin}(i)$, the binary representation of $ i$ precede by its length in unary. A complete node encoding then consist of the encoded index, encoded weights, threshold, encoded degree, and encoded indices of the nodes corresponding to its inputs. A complete circuit can be encoded with a node - count followed by a sequence of lymph node - encoding. For this representation, a majority - of - three algorithm is easily constructed that renumbers two of its three stimulation representations, and combines the three by adding a 3 - stimulation node computing the majority function $ x_1+x_2+x_3 \geq 2$. It is clear that under this theatrical performance, the system of threshold circuits are polynomially closed under majority - of - three. On the other hand they are not closed under exceptions, or under the exception lists of [ @board ]. 

 [ * Example. * ] { } Let $ h_1, h_2, h_3 $ be 3 $ k$-DNF formulas. Then $ { \mbox{MAJ}}(h_1,h_2,h_3) = (h_1 \wedge h_2) \vee (h_2 \wedge h_3) \vee 
 (h_3 \wedge h_1)$ which can be extend into a $ 2k$-DNF formula. This is not dear enough for Theorem   \[maj\ ], but it allows us to conclude that pac - learnability of $ k$-DNF implies compression of $ k$-DNF into $ 2k$-DNF.
 Ocfam algorithm for ${\bf R}$.

Leu a representatiou systek ${\bf R}$ be polyvomially closed under either ezceptuons or majority-of-thred, or both. Then ${\bf R}$ iw deterministically polynklialnb pac-learnable lff there iv a polynomial tkmz Occam algorithm for ${\bf R}$.

[*Example.*]{} Cogsider yhgeshold circuijs, acjcjic dprguits whose nodes compute threshkld funbtions of the fork $a_1x_1 + a_2x_2 + \cdots +a_nx_n \geq \dflta$, $x_i \in \{0,1\}, a_i,\delta \in N$ (note thaj no wxpressive puwer is gapued by allosing rational weights and thresfold). C simple wai kf tepresenting circlits over the binary dlphabey is to number earh nide and use [*prefix-frex encodings*]{} of these numbers. Xox instance, encode $i$ aw $1^{|\nbox{bhn}(i)|}0\mtox{bkb}(i)$, ghe bmnady repgesxntation of $i$ preceded by its length in umawj. A complete hode egcjding then consists of the encoded indeq, endoded weights, threshold, encoded degree, and ejcoded inqices of the nodes corresponding to its inputs. A womplxtd cnvguit xaj be encoded with a node-count followed by a sqsutncv of node-encodingf. For this teorrfentation, a mxjoritv-kf-fhree algorithm is easily consrructed trat tenumbers two of its three unput represvntarions, and combines the three yy addong a 3-input node computing tke majkrity functlon $x_1+x_2+x_3 \gsd 2$. It is clear tfat utder this representation, tre system of chreshola citcuits wre polynolialln closed under majogity-oy-threa. On the ohher hand they are not closed uiver exceptionx, mr lnder the exceition lists of [@foard].

[*Example.*]{} Lgt $h_1, h_2, h_3$ ye 3 $k$-DVF formulaa. Then ${\kbox{MAJ}}(h_1,h_2,h_3) = (h_1 \wedge h_2) \vea (h_2 \wedge h_3) \vxe
(h_3 \wedge h_1)$ wyich can be dxpanded into s $2k$-DNF fogmbla. This us not good enough fot Fheorem \[maj\], but nu aolows us to convluae ehwt pww-learnabilitf of $k$-DVG impuies comprewrion of $k$-DNF into $2k$-DNF.
Occam algorithm for ${\bf R}$. Let a ${\bf be polynomially under either exceptions ${\bf is deterministically polynomially iff there is polynomial time Occam algorithm for ${\bf [*Example.*]{} Consider threshold circuits, acyclic circuits whose nodes compute threshold functions of the $a_1x_1 + a_2x_2 + \cdots +a_nx_n \geq \delta$, $x_i \in \{0,1\}, a_i,\delta \in (note no power gained by allowing rational weights and threshold). A simple way of representing circuits over the binary is to number each node and use [*prefix-free of these numbers. For encode $i$ as $1^{|\mbox{bin}(i)|}0\mbox{bin}(i)$, the representation $i$ preceded its in A complete node then consists of the encoded index, encoded weights, threshold, encoded degree, and encoded indices of the nodes to its complete circuit be with node-count followed by of node-encodings. For this representation, a easily constructed that renumbers two of its three representations, and the three by adding a 3-input computing the majority function $x_1+x_2+x_3 \geq 2$. It clear that under this representation, the system of threshold circuits are polynomially closed under majority-of-three. other hand they are closed under exceptions, under exception of [*Example.*]{} Let h_2, h_3$ be 3 $k$-DNF formulas. Then ${\mbox{MAJ}}(h_1,h_2,h_3) = (h_1 \wedge \vee (h_2 \wedge h_3) \vee (h_3 \wedge h_1)$ which can into $2k$-DNF formula. This not good enough for \[maj\], it allows us to pac-learnability $k$-DNF $k$-DNF $2k$-DNF.
 Occam algorithm for ${\bf R}$.

Let a rEpresentatIon sySteM ${\bf r}$ bE polYnomIally closed undER eitHer exceptions or majoritY-of-thReE, Or boTH. THen ${\bf r}$ is deteRMiNIStiCaLlY poLyNOmIally Pac-LearnabLe iff there Is a PoLynomial time oCcAm algorithM foR ${\bf R}$.

[*Example.*]{} COnsIder thReShoLD circUitS, acycLic cirCUits whOse nodes cOmPUte thrEShold fuNCTiOns oF the form $a_1x_1 + a_2x_2 + \cdots +A_Nx_N \Geq \delta$, $x_i \in \{0,1\}, a_i,\Delta \iN N$ (NOtE THat No eXpressive pOwEr is gAIned by aLLoWING raTIonal weights aNd threshold). a SimPle way Of RepREsentiNg cirCuITs oVer the binarY alpHabet is to Number EAch node ANd use [*prEfix-frEe eNcoDingS*]{} Of ThEse NuMBerS. foR inSTanCe, encode $I$ aS $1^{|\mBox{biN}(i)|}0\mbOX{BIN}(i)$, thE biNary RepreSentation of $i$ pRecEded BY itS lengTh in uNary. a cOmpleTe node EncodInG then consists of The eNcoded indEx, eNcOdeD wEightS, ThreshOld, EncOded degRee, and eNCodEd INDIcEs of the nodes corresPoNDInG to its inPuts. A cOMpLeTE circuit CaN be EncoDED with A nodE-CoUnt folloWed by a SEqUeNce of noDe-EncodiNgS. FoR thIs repREsenTation, A majoritY-of-thREe algorithm is eASily constructED tHAT rENumbErs Two of its thrEe inPUt rePresENtAtiONs, and CombiNeS ThE Three by adding a 3-input NoDe compUting The majority fuNction $x_1+x_2+x_3 \gEQ 2$. iT is clear That UNdER this representAtion, The system oF ThresholD circUits are pOlynomialLY Closed unDer MajOriTy-oF-THrEe. On the other hAND theY aRe not clOseD under eXcePtiOns, Or uNdEr the excePtion lisTs Of [@BoArD].

[*ExAmple.*]{} lEt $h_1, h_2, h_3$ be 3 $k$-dNf foRmUlaS. Then ${\MBox{MAJ}}(H_1,h_2,h_3) = (h_1 \wEdge H_2) \vEe (H_2 \WedGe h_3) \vee
(h_3 \WEdGE H_1)$ whiCh CaN be eXpaNdEd intO a $2k$-Dnf foRmula. ThIs is not goOd eNOugh FoR THeorem \[mAj\], but it allows Us To conclude ThAt pAc-learNABility of $K$-DNF implies compression oF $K$-DNF intO $2k$-DnF.
 Occam algorithm for ${\bf  R}$.

Let  a re pre sen ta tion  sys tem ${\bf R}$b e po lynomially closed unde r eit he r  exc e pt ionsor majo r it y - of- th re e,or bo th. T hen  ${\bfR}$ is det erm in istically po l yn omially pa c-l earnable iff  th ere is  a  po l ynomi altimeOccama lgorit hm for ${ \b f  R}$.[*Examp l e .* ]{}Consider threshol d  c i rcuits, acycli c circ ui t sw h ose  no des comput ethres h old fun c ti o n s  of the form $a_1 x_1 + a_2x_ 2  +\cdots  + a_n x _n \ge q \de lt a $,$x_i \in \{ 0,1\ }, a_i,\d elta \ i n N$ (n o te that  no ex pre ssi ve p o we risga i ned by  al l owi ng ratio na lweigh ts a n d t hres hol d).A sim ple way of re pre sent i ngcircu its o verth e bin ary al phabe tis to number ea ch n ode and u se[* pre fi x-fre e  encod ing s*] {} of t hese nu m ber s. F o rinstance, encode $ i$ a s$1^{|\mb ox{bin } (i )| } 0\mbox{b in }(i )$,t h e bin aryr ep resentat ion of $i $precede dby its  l eng thin un a ry.A comp lete nod e enc o ding then cons i sts of the en c od e d  i n dex,  en coded weigh ts,t hres hold ,  e nco d ed de gree,  a n de ncoded indices of t he  nodes  corr esponding toits inputs . A  complet e ci r cu i t can be encod ed wi th a node- c ount fol lowed  by a se quence of n ode-enco din gs.  Fo r t h i srepresentatio n ,  a m aj ority-o f-t hree al gor ith m i s e as ily const ructed t ha tre nu mbe rs tw o  of itsth ree  i npu t rep r esenta tions , an dco m bin es thet hr e e  byad di ng a  3- in put n odec omp uting t he majori tyf unct io n$x_1+x_ 2+x_3 \geq 2$ .It is clea rtha t unde r this rep resentation, the system of thre sho ld ci rcui ts are po lyn omiall y c l osed u nder m ajori ty -of - t hree. O nthe  o ther handt h eyare n ot  clo sed und er exceptions, oru nde r the excepti onlist s of  [@ b oa r d]. 
[ *Ex a m ple.*]{} Let $h _1, h_2, h _3 $  b e 3 $k$-DN F  fo rm ulas. T hen ${\ mbox{ M AJ}}(h_ 1,h_2,h_3 ) = (h_1\w edge h _2)  \vee (h_2  \wedgeh_3) \vee (h_3\ we dge h _1) $ whic hcan  be e xpande d  in to a$2k$-D NF  formu la. T hi s is not  good enough for Theore m \[ma j\],but  it allow s u s  to  conclude  tha t pac-lear nab ili ty of  $k $ -DNFimpl i es  co m press iono f $k$-DNF in to$ 2 k$ -DNF.
 Occam_algorithm for_${\bf R}$.

Let a representation_system ${\bf_R}$_be polynomially_closed_under either exceptions_or majority-of-three, or_both. Then ${\bf R}$_is deterministically polynomially_pac-learnable_iff there is a polynomial time Occam algorithm for ${\bf R}$.

[*Example.*]{} Consider threshold circuits,_acyclic_circuits whose_nodes_compute_threshold functions of the form_$a_1x_1 + a_2x_2 + \cdots_+a_nx_n \geq_\delta$, $x_i \in \{0,1\}, a_i,\delta \in N$ (note_that_no expressive power_is gained by allowing rational weights and threshold). A_simple way of representing circuits over_the binary alphabet_is_to_number each node and_use [*prefix-free encodings*]{} of these numbers._For instance, encode $i$ as $1^{|\mbox{bin}(i)|}0\mbox{bin}(i)$,_the binary representation of $i$ preceded by_its length in unary. A complete_node encoding then consists of_the encoded_index, encoded weights, threshold, encoded_degree, and encoded_indices of_the nodes corresponding_to its inputs. A complete circuit_can be encoded_with a node-count followed by a_sequence_of node-encodings. For_this_representation,_a majority-of-three_algorithm is easily_constructed_that renumbers_two_of its three input representations, and_combines_the three by adding a 3-input node_computing the majority function_$x_1+x_2+x_3_\geq 2$. It is_clear that under this representation,_the system of threshold circuits are_polynomially closed_under majority-of-three._On the other hand they are not closed under exceptions, or_under the exception lists of [@board].

[*Example.*]{}_Let $h_1, h_2, h_3$_be 3_$k$-DNF_formulas. Then ${\mbox{MAJ}}(h_1,h_2,h_3)_=_(h_1 \wedge_h_2) \vee (h_2 \wedge h_3) \vee
(h_3 \wedge_h_1)$ which_can be expanded into a $2k$-DNF_formula. This is not_good_enough for Theorem \[maj\], but it allows_us to conclude that pac-learnability of_$k$-DNF implies compression of $k$-DNF_into_$2k$-DNF.