 have at the moment to trust the systems we developed. Allowing the system more degrees of freedom thus hinders the developers’ ability to estimate the degree of maturity of the system they design, which poses a severe difficulty for the engineering progress, when the desired premises or the expected effects of classical engineering tasks on the system-under-development are hard to formulate.

To aid us control the development/adaptation progress of the system, we define a set of *principles*, which are basically patterns for process models. They describe the changes to be made in the engineering process for complex, adaptive systems in relation to more classical models for software and systems engineering.

\[con:parallelism\] The system and its test suite should develop in parallel from the start with controlled moments of interchange of information. Eventually, the test system is to be deployed alongside the main system so that even during runtime, on-going online tests are possible [@calinescu2012self]. This argument has been made for more classical systems as well and thus classical software test is, too, no longer restricted to a specific phase of software development. However, in the case of self-learning systems, it is important to focus on the evolution of test cases: The capabilities of the system might not grow as experienced test designers expect them to compared to systems entirely realized by human engineering effort. Thus, it is important to conceive and formalize how tests in various phases relate to each other.

\[con:antagonism\] Any adaptive systems must be subject to an equally adaptive test. Overfitting is a known issue for many machine learning techniques. In software development for complex adaptive systems, it can happen on a larger scale: Any limited test suite (we expect our applications to be too complex to run a complete, exhaustive test) might induce certain unwanted biases. Ideally, once we know about the cases our system has a hard time with, we can train it specifically for these situations. For the so-hardened system the search mechanism that gave us the hard test cases needs to come up with even harder ones to still beat the system-under-test. Employing autonomous adaptation at this stage is expected to make that arms race more immediate and faster than it is usually achieved with human developers and testers alone.

\[con:automated\] Since the realization of tasks concerning adaptive components usually means the application of a standard machine learning process, a lot of the development effort regarding certain tasks tends to shift to an earlier phase in the process model. The most developer
have at the moment to trust the systems we develop. give up the system more academic degree of exemption thus hinders the developers ’ ability to calculate the academic degree of maturity of the system they design, which poses a severe trouble for the engineering progress, when the desired premises or the expect effects of authoritative engineering undertaking on the arrangement - under - development are hard to formulate. 

 To help us control the development / adaptation progress of the system, we specify a set of * principles *, which are basically patterns for process models. They identify the changes to be made in the engineering process for complex, adaptive systems in relation to more classical models for software and systems engineering. 

 \[con: parallelism\ ] The system and its test suite should develop in parallel from the start with controlled moment of interchange of information. finally, the trial organization is to be deployed alongside the main arrangement so that even during runtime, on - going online tests are possible [ @calinescu2012self ]. This controversy has been made for more classical systems as well and thus classical software examination is, too, no longer restricted to a specific phase of software development. However, in the case of self - learn system, it is authoritative to focus on the evolution of test subject: The capabilities of the system might not grow as experienced trial designers expect them to compared to systems wholly realized by human engineering effort. Thus, it is important to gestate and formalize how trial in various phase relate to each early. 

 \[con: antagonism\ ] Any adaptive systems must be subject to an equally adaptive examination. Overfitting is a known topic for many machine eruditeness techniques. In software development for complex adaptive systems, it can happen on a big plate: Any limited test cortege (we expect our applications to be too complex to hunt a complete, exhaustive test) might induce certain undesirable biases. Ideally, once we know about the cases our system has a hard time with, we can train it specifically for these situation. For the thus - hardened system the search mechanism that gave us the hard test cases necessitate to come up with even harder ones to even beat the system - under - test. Employing autonomous adaptation at this stage is expect to make that arms race more immediate and faster than it is usually achieve with human developers and testers alone. 

 \[con: automated\ ] Since the realization of tasks concerning adaptive components usually means the application of a standard machine learning process, a lot of the development effort regarding certain tasks tends to careen to an earlier phase in the process model. The most developer
 hage at the moment to trusu the systems we bwvelopxd. Allosing the system more degrees of freevom rhus yinders the developers’ ability no estimare tie degree of matndity of the sgdtem vhey design, whigh poses a vevere difficunth yor the engineering progress, when thq desirrd premises or tre eqpqctes effects of classical engineering tasks mn the system-inder-development are hard ho flrmulate.

To aid us fontrol the devqoopment/adaptxtion proggzss of the aystem, we define a set of *princkples*, which are bqsifdlly patteris for process modcks. Thef descrobe the changex tm bw made in the engineecing process for comklex, adapthvz systems in relation ti morg clavsicxo mudems fkr sofhwace and systsms engineeeing.

\[con:parallelism\] Uhe wystem and ita test stite should develop in parallel from tht stadt with controlled momebts of interchange of informatyon. Eventually, the test system is to be deployed dlongakde tme mxun system so that even during runtime, on-going ogmime tests are posfible [@calinrsfu2012xglf]. This argumgnt has besn made for more cpassicaj sysrems as wtll amd thus classical software rest is, too, uo oonger restricted co a specifie phasg of spftware development. Howzver, ih the case lf self-lezfning systems, it is ikportant to focus on the edolution if tzst caser: Thg capabylities of the system might not grlw as efperienced test designers expect them to rpmpared to sistams entirelv reallzed by human eggineering effprt. Thuf, it ks importaht to cmnceive and formalize hof tests in vacious phafes eelare to exzh other.

\[con:antsgonism\] Auv adaptivw systems must be xubgsct to an equaloy qdaptive test. Oferwiteijg if a known issge fur osny mxchine learkine tevhniques. In software devslopment for complrx adaptivg systems, it can happem on a larger scalt: Any nimmted trst suite (we expect our applicatikns to be toj complex to run a complete, zxhaustive test) might induce certain unwented biases. Ideally, onxe we know about thg csses our syxtem ras a harg time with, we can teain it specificakly for these situatikns. Fos the so-hardened system the search mechanism that gave us the hard test cases beeds vo come up wifh efen hdrber ones eo sviul beat the systvm-under-test. Employing autonomous adaptathou at this stage is expected tp oake that armr race more immediate ans faster than it is usually achieved wiyh human developers and besters aoone.

\[ckn:automatev\] Since the realizatoon of taskw conceeninn adaptive comoonrnbs usually meais che application of a standard macyine lwagning process, a not of the ftvelopmgnt effott regarding certzin fwsks jends to shift to an earlier phase in the procesv iodel. The most seveloper
have at the moment to trust the developed. the system degrees of freedom to the degree of of the system design, which poses a severe difficulty the engineering progress, when the desired premises or the expected effects of classical tasks on the system-under-development are hard to formulate. To aid us control the progress the we a set of *principles*, which are basically patterns for process models. They describe the changes to made in the engineering process for complex, adaptive in relation to more models for software and systems \[con:parallelism\] system and test should in parallel from start with controlled moments of interchange of information. Eventually, the test system is to be deployed alongside main system even during on-going tests possible [@calinescu2012self]. This been made for more classical systems thus classical software test is, too, no longer to a phase of software development. However, in case of self-learning systems, it is important to on the evolution of test cases: The capabilities of the system might not grow as designers expect them to to systems entirely by engineering Thus, is important conceive and formalize how tests in various phases relate to each \[con:antagonism\] Any adaptive systems must be subject to an equally Overfitting a known issue many machine learning techniques. software for complex adaptive systems, happen a limited suite expect our applications to too complex to run a exhaustive test) might induce we know about the cases our system has hard time with, we can train it for these situations. For the so-hardened system the search mechanism that gave the hard needs to come up with even harder ones still beat the system-under-test. autonomous adaptation at this stage is expected to make arms more immediate faster than it usually achieved with developers and testers Since the of concerning means the application of a standard learning a lot of the development regarding tends to shift to an earlier phase in the process model. The most developer
 have at the moment to trust the Systems we dEveloPed. allOwIng tHe syStem more degreeS Of frEedom thus hinders the devElopeRs’ ABiliTY tO estiMate the DEgREE of MaTuRitY oF ThE systEm tHey desiGn, which posEs a SeVere difficulTY fOr the enginEerIng progress, wHen The desIrEd pREmiseS or The exPected EFfects Of classicAl ENgineeRIng taskS ON tHe syStem-under-developmENt ARe hard to formulAte.

To aId US cONTroL thE developmeNt/AdaptATion proGReSS OF thE System, we definE a set of *prinCIplEs*, whicH aRe bASicallY pattErNS foR process modEls. THey descriBe the cHAnges to BE made in The engIneEriNg prOCeSs For CoMPleX, AdAptIVe sYstems in ReLaTion tO morE CLASsicAl mOdelS for sOftware and sysTemS engINeeRing.

\[cOn:parAlleLiSm\] The System And itS tEst suite should dEvelOp in paralLel FrOm tHe Start WIth conTroLleD momentS of inteRChaNgE OF InFormation. EventuallY, tHE TeSt system Is to be DEpLoYEd alongsIdE thE maiN SYstem So thAT eVen durinG runtiME, oN-gOing onlInE tests ArE poSsiBle [@caLInesCu2012self]. this arguMent hAS been made for moRE classical sysTEmS AS wELl anD thUs classical SoftWAre tEst iS, ToO, no LOnger RestrIcTEd TO a specific phase of soFtWare deVelopMent. However, in The case of sELF-Learning SystEMs, IT is important to Focus On the evoluTIon of tesT caseS: The capaBilities oF THe system MigHt nOt gRow AS ExPerienced test DESignErS expect TheM to compAreD to SysTemS eNtirely reAlized by HuMaN eNgIneEring EFfort. ThuS, iT is ImPorTant tO ConceiVe and FormAlIzE How Tests in VArIOUs phAsEs RelaTe tO eAch otHer.

\[cON:anTagonisM\] Any adaptIve SYsteMs MuSt be subJect to an equalLy Adaptive teSt. oveRfittiNG Is a known Issue for many machine learNIng techNiqUes. In SoftWare develOpmEnt for ComPLex adaPtive sYstemS, iT caN HAppen ON A lArgEr Scale: Any liMITed Test sUiTe (we Expect oUr applications to be TOo cOmplex to run a cOmpLete, EXHaUstIVe TEst) MiGHt iNDUce certain unwanTed biases. IDeALlY, once we knoW AboUt The caseS our sysTem haS A hard tiMe with, we cAn train it SpEcifICAllY for these sItuationS. For the so-HArdenED sYstem The Search MeChaNism tHat gavE Us tHe harD test cAsEs needS to coMe Up with evEn harder ones to still beat The sysTem-unDer-Test. EmploYinG AutOnomous adAptaTion at this StaGe iS expeCteD To makE thaT ArMs rACe morE immEDiate and fAStEr tHAN iT is usually aCHIEveD with HumAN develOperS and testers alone.

\[cON:automated\] SincE the REAliZatIOn of TaSks concerning aDapTiVE ComponenTs Usually meanS the applIcATion oF a stanDard maChine leARNiNG proceSs, a lOt oF the develOpmEnT Effort rEgArDIng cerTain TaSks tenDs to shIFt to AN Earlier phase in thE procESS modeL. the Most dEvEloper
 have at the moment to tru st the sys temswedev el oped . Al lowing the sys t em m ore degrees of freedom  thus  h i nder s  t he de veloper s ’a b ili ty  t o e st i ma te th e d egree o f maturity  of  t he system th e ydesign, wh ich  poses a sev ere  diffi cu lty for t heengin eering progre ss, whenth e  desir e d premi s e sor t he expected effec t so f classical en gineer in g  t a s ksonthe system -u nder- d evelopm e nt a r e h a rd to formula te.

To aid uscontro lthe develo pment /a d apt ation progr essof the sy stem,w e defin e  a setof *pr inc ipl es*, wh ic h a re bas i ca lly pat terns fo rpr ocess  mod e l s .  The y d escr ibe t he changes to  be  mad e  in  theengin eeri ng  proc ess fo r com pl ex, adaptive sy stem s in rela tio ntomo re cl a ssical  mo del s for s oftware and  s y s t em s engineering.

\[ co n : pa rallelis m\] Th e  s ys t em and i ts  te st s u i te sh ould de velop in  paral l el  f rom the  s tart w it h c ont rolle d  mom ents o f interc hange of information .  Eventually,t he t es t  sys tem  is to be d eplo y ed a long s id e t h e mai n sys te m  s o  that even during r un time,on-go ing online te sts are po s s i ble [@ca line s cu 2 012self]. This  argu ment has b e en madefor m ore clas sical sys t e ms as we lland  th usc l as sical softwar e test  i s, too,  no  longer  re str ict edto  a specif ic phase  o fso ft war e dev e lopment.  H owe ve r,in th e  caseof se lf-l ea rn i ngsystems ,  i t is i mp or tant  to  f ocuson t h e e volutio n of test  ca s es:Th ecapabil ities of thesy stem might  n otgrow a s experien ced test designers expe c t themtocompa redto system s e ntirel y r e alized  by hu man e ng ine e r ing e f f or t.Th us, it isi m por tantto  con ceive a nd formalize how t e sts  in various p has es r e l at e t o  e a chot h er. \[con:antagonis m\] Any ad ap t iv e systemsm ust  b e subje ct to a n equ a lly ada ptive tes t. Overfi tt ingi s  aknown issu e for ma ny machin e  lear n in g tec hni ques.In  so ftwar e deve l opm ent f or com pl ex ada ptive  s ystems,it can happen on a larg er sca le: A nylimited t est sui te (we ex pect  our appli cat ion s tobet oo co mple x  t o r u n a c ompl e te, exhau s ti vet e st ) might ind u c e  ce rtain  un w antedbias es. Ideally, once we know aboutthec a ses  ou r  sys te m has a hard t ime  w i t h, we ca ntrain it sp ecifical ly for t hese s ituati ons. Fo r th e  so-ha rden edsystem th e s ea r ch mech an is m  thatgave  u s thehard t e st c a s es needs to come  up w i t h eve n  ha rderon es to s t illbeat the s ystem-under -test.  Emp loyin g auton om ous ad apt at ion at thi s  stage is  expe cted to  m aketha t arms  rac e moreimme di ate  and fast e r  t h an  i t  is  usu allyac hiev ed with h u man deve lop e rs andte ste r s  alone . 
\ [ con:automa ted \] Si n c e the real i zati o no f tas ks con cernin g adapt i veco mponent s u s u ally mean s the app l ica ti on o f a stan da rd m ac hin elearning proces s, a lot  of thed e velopm ent effo r t rega rd ing certaintasks  t e nds tos h ift  to ane arl ier  ph aseinthep rocess model. The most d evel oper
 have_at the_moment to trust the_systems we_developed._Allowing the_system_more degrees of_freedom thus hinders_the developers’ ability to_estimate the degree_of_maturity of the system they design, which poses a severe difficulty for the engineering_progress,_when the_desired_premises_or the expected effects of_classical engineering tasks on the_system-under-development are_hard to formulate.

To aid us control the development/adaptation_progress_of the system,_we define a set of *principles*, which are basically_patterns for process models. They describe_the changes to_be_made_in the engineering process_for complex, adaptive systems in relation_to more classical models for software_and systems engineering.

\[con:parallelism\] The system and its_test suite should develop in parallel_from the start with controlled_moments of_interchange of information. Eventually, the_test system is_to be_deployed alongside the_main system so that even during_runtime, on-going online_tests are possible [@calinescu2012self]. This argument_has_been made for_more_classical_systems as_well and thus_classical_software test_is,_too, no longer restricted to a_specific_phase of software development. However, in the_case of self-learning systems,_it_is important to focus_on the evolution of test_cases: The capabilities of the system_might not_grow as_experienced test designers expect them to compared to systems entirely realized_by human engineering effort. Thus, it_is important to conceive_and formalize_how_tests in various_phases_relate to_each other.

\[con:antagonism\] Any adaptive systems must be_subject to_an equally adaptive test. Overfitting is_a known issue for_many_machine learning techniques. In software development_for complex adaptive systems, it can_happen on a larger scale:_Any_limited_test suite (we expect our_applications to be too complex to_run a complete,_exhaustive test) might induce certain unwanted biases._Ideally,_once we know about the cases_our_system has a hard time with,_we_can_train it specifically for these_situations. For the so-hardened system the_search mechanism that gave us the hard test cases_needs to come_up with even harder ones_to_still_beat the system-under-test. Employing autonomous adaptation at this stage is_expected to_make that arms_race more immediate and faster than it is usually achieved_with human developers and testers alone.

\[con:automated\] Since_the realization of tasks concerning adaptive components usually means the application_of a standard machine learning process, a lot_of the development effort regarding certain tasks_tends to shift to_an_earlier phase in the process model. The_most_developer