$ of the sets, and try and prove the lemmas in the remaining set. This process repeats until each set has been used exactly once as the collection of lemmas to be proved.

For each proof attempt, we allow 10,000 tactics to be applied before reporting a failure. The results presented in this paper are from using $k=10$, a standard value for $k$-folds cross-validation [@KohaviIJCAI]. Other values of $k$ have been investigated and the full set of results are online.

As well as capturing whether a proof attempt was successful or not, when a proof is found we analyse how “interesting" the proof is. First, we check and see whether a proof is shorter than the corresponding hand-curated proof. We also check whether the sequence of tactics was new (i.e. not present in the examples the model was inferred from). These provide us with answers to RQ2.

To investigate RQ3 we also run the Coq automated tools to try and prove each lemma. The following command is issued to Coq: `auto with * || eauto with * || tauto || firstorder || trivial`. This simply attempts to prove a goal by trying all of the automated tactics. The default search depth is used in all cases. Where we can specify lemma databases, we allow any available database to be used during proof search.

Results
-------

The full results from our experiments are shown in Table \[tab:res\]. The results are presented for each theory, grouped by library. The remainder of this section provides some answers to the research questions defined earlier.

  ------------------------ -------------- ---------- ----------- --------- ------------- --------------
                                                                                         
  (r)[4-6]{} **Library**   **Theory**       **Size**   **Total**   **New**   **Shorter**   **Coq-Tacs**
                           ssrnat                341   135 (39%)        14             9       59 (17%)
                           ssrbool               240   120 (50%)        17            10       60 (25%)
                           seq                   394    94 (24%)        14             6        18 (4%)
                           fintype               243    42 (17%)        15             1         0 (0%)
                           eqtype                 82    36 (44%)        18             2       10 (12%)
                           choice                 30     6 (20
$ of the sets, and try and prove the lemmas in the remain hardening. This process repeats until each set has been use exactly once as the collection of lemma to be proved. 

 For each proof attempt, we let 10,000 tactics to be applied before report a failure. The results stage in this paper are from using $ k=10 $, a standard value for $ k$-folds cross - validation [ @KohaviIJCAI ]. Other values of $ k$ have been investigated and the entire set of results are online. 

 equally well as capturing whether a proof attempt was successful or not, when a validation is found we analyse how “ interesting " the validation is. First, we check and see whether a proof is shorter than the corresponding bridge player - curated proof. We also check whether the sequence of tactics was new (i.e. not present in the examples the model was inferred from). These provide us with answers to RQ2. 

 To investigate RQ3 we also run the Coq automated tools to try and prove each lemma. The follow command is issued to Coq: 'auto with * || eauto with * || tauto || firstorder || fiddling `. This simply attempts to prove a finish by trying all of the automatize tactics. The default search depth is use in all cases. Where we can specify lemma databases, we allow any available database to be used during proof search. 

 Results 
 ------- 

 The broad results from our experiments are shown in Table \[tab: res\ ]. The results are portray for each hypothesis, group by library. The remainder of this section provides some answers to the inquiry questions defined earlier. 

   ------------------------ -------------- ---------- ----------- --------- ------------- -------------- 
                                                                                         
   (r)[4 - 6 ] { } * * Library * *    * * Theory * *        * * Size * *    * * Total * *    * * New * *    * * Shorter * *    * * Coq - Tacs * * 
                            ssrnat                 341    135 (39 %)         14              9        59 (17 %) 
                            ssrbool                240    120 (50 %)         17             10        60 (25 %) 
                            seq                    394     94 (24 %)         14              6         18 (4 %) 
                            fintype                243     42 (17 %)         15              1          0 (0 %) 
                            eqtype                  82     36 (44 %)         18              2        10 (12 %) 
                            choice                  30      6 (20
$ of the sets, and try and pruve the lemmas nb the cemainihg set. Tfis process repeats until earh swt haw been used exactly onze as the collectuon id lemmas to be proyzd.

For cach 'riof attempt, we allow 10,000 tawtics to be ap[lkeb before reporting a failure. The restlts prrsfnted in this kaper wre rgon using $k=10$, a standard value fod $k$-foldv cross-validayion [@KohaviIJCAI]. Other valkes lf $k$ have been invfstigated abd trw full set ow results are online.

As well as capturing whether a pruof actempt was wuxcedvful or not, when a proof is npund wa analyxe how “interesbing" vhe proof is. First, we cherk and see whether a proof is skorter than the correwpindinc hatd-cufqtea pdopf. We aldo rheck whethsr the sequwnce of tactics was nqq (i.e. not pressnt in tre examples the model was inferred from). Thsse provide us with ansqers to RQ2.

To investigwte RQ3 we also run the Coq automated tools to try and prova eaci uemna. Gye following command is issued to Coq: `auto with * || tauno with * || tauto || nirstorder || triviak`. Hhof simply atteopts to pdove a goal by trylng all of tye automaued tsctics. The default search dwpth is used un all cases. Where we can speeify lgmma dstabases, we allow any arailabme database to be ussa during proof sdarbh.

Revults
-------

The full results from our expecimencs are sfown in Tafle \[tab:res\]. The vasults are presentfd fot each theory, grluped by library. The remainder of this sectiom [ronides somz answcrs to the resewrch questions defineb earlker.

  ------------------------ -------------- ---------- ----------- --------- ------------- --------------
                                                                                         
  (r)[4-6]{} **Library**   **Theory**       **Side**   **Jofal**   **New**   **Shoxuer**   **Coq-Tacs**
                           ssrtat                341   135 (39%)        14             9       59 (17%)
                           sstbool               240   120 (50%)        17            10       60 (25%)
                           seq                   394    94 (24%)        14             6        18 (4%)
                           fiitype               243    42 (17%)        15             1         0 (0%)
                           eqtypw                 82    36 (44%)        18             2       10 (12%)
                           choice                 30     6 (20
$ of the sets, and try and lemmas the remaining This process repeats used once as the of lemmas to proved. For each proof attempt, we 10,000 tactics to be applied before reporting a failure. The results presented in paper are from using $k=10$, a standard value for $k$-folds cross-validation [@KohaviIJCAI]. Other of have investigated the full set of results are online. As well as capturing whether a proof attempt was or not, when a proof is found we how “interesting" the proof First, we check and see a is shorter the hand-curated We also check the sequence of tactics was new (i.e. not present in the examples the model was inferred from). provide us to RQ2. investigate we run the Coq to try and prove each lemma. is issued to Coq: `auto with * || with * tauto || firstorder || trivial`. This attempts to prove a goal by trying all the automated tactics. The default search depth is used in all cases. Where we can databases, we allow any database to be during search. ------- full results our experiments are shown in Table \[tab:res\]. The results are presented each theory, grouped by library. The remainder of this section answers the research questions earlier. ------------------------ -------------- ---------- --------- -------------- (r)[4-6]{} **Library** **Theory** **New** **Coq-Tacs** (39%) 9 (17%) ssrbool 240 120 17 10 60 (25%) seq 94 (24%) 14 6 (17%) 15 1 0 (0%) eqtype 82 36 18 2 10 (12%) choice 30 6
$ of the sets, and try and prove thE lemmas in tHe remAinIng SeT. ThiS proCess repeats untIL eacH set has been used exactly Once aS tHE colLEcTion oF lemmas TO bE PRovEd.

foR eaCh PRoOf attEmpT, we alloW 10,000 tactics to Be aPpLied before rePOrTing a failuRe. THe results preSenTed in tHiS paPEr are FroM usinG $k=10$, a staNDard vaLue for $k$-foLdS Cross-vALidatioN [@kOhAviIjCAI]. Other values of $K$ HaVE been investigaTed and ThE FuLL Set Of rEsults are oNlIne.

As WEll as caPTuRING whETher a proof attEmpt was succESsfUl or noT, wHen A Proof iS founD wE AnaLyse how “inteRestIng" the proOf is. FiRSt, we cheCK and see WhetheR a pRooF is sHOrTeR thAn THe cORrEspONdiNg hand-cuRaTeD prooF. We aLSO CHeck WheTher The seQuence of tactiCs wAs neW (I.e. nOt preSent iN the ExAmpleS the moDel waS iNferred from). ThesE proVide us witH anSwErs To rQ2.

To iNVestigAte rQ3 wE also ruN the Coq AUtoMaTED ToOls to try and prove eaCh LEMmA. The follOwing cOMmAnD Is issued To coq: `Auto WITh * || eauTo wiTH * || tAuto || firsTorder || TRiViAl`. This sImPly attEmPts To pRove a GOal bY tryinG all of thE autoMAted tactics. The DEfault search dEPtH IS uSEd in All Cases. Where wE can SPeciFy leMMa DatABases, We allOw ANy AVailable database to bE uSed durIng prOof search.

ResuLts
-------

The full RESUlts from Our eXPeRIments are shown In TabLe \[tab:res\]. ThE Results aRe preSented foR each theoRY, Grouped bY liBraRy. THe rEMAiNder of this secTIOn prOvIdes somE anSwers to The ResEarCh qUeStions defIned earlIeR.

  ------------------------ -------------- ---------- ----------- --------- ------------- --------------
                                                                                         
  (r)[4-6]{} **liBrAry**   **theorY**       **size**   **TotaL**   **NEw**   **SHoRteR**   **Coq-TACs**
                           ssrnAt                341   135 (39%)        14             9       59 (17%)
                           ssrBool               240   120 (50%)        17            10       60 (25%)
                           SeQ                   394    94 (24%)        14             6        18 (4%)
                           fINtyPe               243    42 (17%)        15             1         0 (0%)
                           eqtypE                 82    36 (44%)        18             2       10 (12%)
                           ChOICe                 30     6 (20
$ of the sets, and try and  prove the  lemm asinth e re main ing set. Thisp roce ss repeats until eachset h as been us ed ex actly o n ce a s t he  c oll ec t io n oflem mas tobe proved. 

F or  each proofa tt empt, we a llo w 10,000 tac tic s to b eapp l ied b efo re re portin g  a fai lure. The  r e sultsp resente d in  thi s paper are fromu si n g $k=10$, a st andard  v a lu e for  $k $-folds cr os s-val i dation[ @K o h a viI J CAI]. Other v alues of $k $  ha ve bee ninv e stigat ed an dt hefull set of  res ults areonline . 

As we l l as ca pturin g w het hera  p ro ofat t emp t  w ass ucc essful o rno t, wh en a p r o of i s f ound  we a nalyse how “i nte rest i ng"  theproof  is.  F irst,  we ch eck a nd  see whether aproo f is shor ter  t han  t he co r respon din g h and-cur ated pr o of.  W e a ls o check whether th es e qu ence oftactic s  w as new (i.e .not  pre s e nt in  the ex amples t he mod e lwa s infer re d from ).  Th ese  prov i de u s with  answers  to R Q 2.

To investi g ate RQ3 we al s or u nt he C oqautomated t ools to t ry a n dpro v e eac h lem ma .  T h e following command  i s issu ed to  Coq: `auto w ith * || e a u t o with *  ||t au t o || firstorde r ||trivial`.T his simp ly at tempts t o prove a g oal by t ryi ngall  of t he  automated ta c t ics.  T he defa ult  search  de pth  is  us ed  in all c ases. Wh er ewe  c anspeci f y lemmada tab as es,  we a l low an y ava ilab le  d a tab ase tob eu s ed d ur in g pr oof  s earch .

R e sul ts
---- ---

Theful l  res ul ts  from o ur experiment sare shownin  Ta ble \[ t a b:res\].  The results are presen t ed foreac h the ory,  groupedbylibrar y.T he rem ainder  of t hi s s e c tionp r ov ide ssome answe r s  to  there sear ch ques tions defined earl i er. 

  --------- --- ---- - - -- --- -  - - --- -- - --- - - - ---------- -- ----------- - -- ---- ----- - --- -- -- ---- ------- ---
                                                                       ( r)[4- 6]{} * * Lib rary* *   ** Th eory**      **Size**    **Total**   **New**  **Sh orter **  **Coq-T acs * *
                      ssr nat          34 1 135 ( 39%)        14        9      5 9 (17 %)
                      ssrboo l               240  12 0 ( 5 0%)      17         1 0        6 0(25%)
                     seq             39 4   94  (24%)     1 4         6      1 8 (4%) 
                      f intyp e           24 3    42 (17% )        1 5            1       0(0%)
                    eqtype             82  36 (44 %)     1 8          2   1 0(1 2 %)                     choice            3 0    6  ( 2 0 
$ of_the sets,_and try and prove_the lemmas_in_the remaining_set._This process repeats_until each set_has been used exactly_once as the_collection_of lemmas to be proved.

For each proof attempt, we allow 10,000 tactics to be_applied_before reporting_a_failure._The results presented in this_paper are from using $k=10$,_a standard_value for $k$-folds cross-validation [@KohaviIJCAI]. Other values of_$k$_have been investigated_and the full set of results are online.

As well_as capturing whether a proof attempt_was successful or_not,_when_a proof is found_we analyse how “interesting" the proof_is. First, we check and see_whether a proof is shorter than the_corresponding hand-curated proof. We also check_whether the sequence of tactics_was new_(i.e. not present in the_examples the model_was inferred_from). These provide_us with answers to RQ2.

To investigate_RQ3 we also_run the Coq automated tools to_try_and prove each_lemma._The_following command_is issued to_Coq:_`auto with_*_|| eauto with * || tauto_||_firstorder || trivial`. This simply attempts to_prove a goal by_trying_all of the automated_tactics. The default search depth_is used in all cases. Where_we can_specify lemma_databases, we allow any available database to be used during proof_search.

Results
-------

The full results from our experiments_are shown in Table_\[tab:res\]. The_results_are presented for_each_theory, grouped_by library. The remainder of this section_provides some_answers to the research questions defined_earlier.

  ------------------------ --------------_----------_----------- --------- ------------- --------------
  _     _    ___    _     _  _      __     __     ___    _     _        _  _    ___      
  (r)[4-6]{} **Library**_ _**Theory**  _    **Size**   **Total**  _**New**   **Shorter**   **Coq-Tacs**
_          _       _      _ssrnat   __      ____ 341 _ 135 (39%)_  _ _  14_ _  __      9       59 (17%)
   _     _    __  _   _    ssrbool   _        __ 240_  120 (50%)_ _ _  _17 _ _ _      10__     60 (25%)
  ____  _  _  _  __     _  seq_   _         _    394    94 (24%)       _14 _   _ __    6 _ _    18 (4%)
  __           ___  __     fintype_____     _    243  __42 (17%)   _ __ 15  _   _      1    __ __0_(0%)
_   _              __    _ eqtype    __   _       82_   36 (44%)_   _   18             2    _  10 (12%)
 _    _  __  _   _       _ choice                _30     6 (20