Florentin Smarandache & Ştefan Vlăduţescu: Neutrosophic emergences and incidences in communication and information – Book review

Neutrosophic emergences and incidences in communication and information, published in Germany/Saarbrücken: LAP LAMBERT Academic Publishing in 2013, is a book that constitutes a new trend, a new approach in the science of logic, philosophy, communication theory, information, an approached and exemplification that it was wanted, it was intuited, it was expected but it has no mathematical foundation, that is neutrosophy. It is a book that presents avant-garde information. It is written by two appreciated researchers and inexhaustible perpetual idea machines: Professors Florentin Smarandache and Ştefan Vlăduţescu. First of them is professor of mathematics at University of New Mexico/ USA, the second is professor of communication theory at University of Craiova/ Romania. Both of them have published a lot of books and papers in domain of philosophy, communication, sociology, psychology, mathematics and literature. Logic started in Ancient with Classical Logic of Aristotle, developed and covered by Three Valued Logic of Lukasiewicz, next ring being Fuzzy Logic of Zadech, finally the comprehensive Neutrosophic Logic of Smarandache.

GENERAL ENVIRONMENT
The fundamental thesis of neutrosophy is that every idea has not only a certain degree of truth, as is generally assumed in many-valued logic contexts, but also a falsity degree and an indeterminacy degree that have to be considered independently from each other. Smarandache seems to understand such "indeterminacy" both in a subjective and an objective sense, i.e. as uncertainty as well as imprecision, vagueness, error, doubtfulness etc. Neutrosophy has laid the foundation for a whole family of new mathematical theories generalizing both their classical and fuzzy counterparts, such as neutrosophic set theory, neutrosophic probability, neutrosophic statistics and neutrosophic logic. The neutrosophic framework has already found practical applications in a variety of different fields, such as relational database systems, semantic web services, financial data set detection and new economy growth and decline analysis. Neutrosophic emergences are the unexpected occurrences of some major neutrosophic effects from the interaction of some minor qualitative elements. Emergence would be seen as a major phenomenon occurrence, important and significant from the reaction of two or more minor unimportant, insignificant elements. On the other hand, we describe the incidence as the application of a law, of an axiom, of an idea, of a conceptual accredited construction on an unclear, indeterminate phenomenon, contradictory to the purpose of making it intelligible. The incidence is the intelligibilization procedure using the concept: basically apply a previous theoretically validated concept is practically applied. If the emergence is a variant of the cognitive bottom -up processing, the incidence is a variant of the top -down cognitive processing.
Neutrosophy had been in the emergence phase since 1995. With its certification by the scientific community, Neutrosophy has become a type of incident knowledge, i.e. applicable in different fields. Neutrosophy legitimating was achieved by developing some doctoral research, through learning theory as a way of description, explanation and forecast and implementation of neutrosophic congress and conferences.
It can be said that the emergence and incidence are specific neutrosophic concepts: the appearance-occurrence and the imposing-application apply, above all, to Neutrosophy itself. Like a good doctor, neutrosophy applies its most toxic treatments to itself. It is well-known that, in their way, the best medicines are poisons with no remedy, too: medicines are some toxins without counter medicines. There are not drugs against drugs.
So Neutrosophy makes the emergence and the incidence visible as neutrosophic events and it also holds itself in the universe of emergences and incidences.
Neutrosophy handles all neutralities. In the neutrosophic taxonometry, a class of neutralities is represented by the neutralities that, without turning into contradiction, generate qualitative leaps. The emergence is the cognitive phenomenon in which, from two or more connected neutralities, without contradiction, a change of quality or a qualitative leap result. Thinking in Hegelian terms has an axiom the idea that the qualitative change, qualitative emergences may arise from related neutral items.
The studies in this book show either how neutrosophic effects burst in different worlds of spirituality, of thought, of knowledge, of experience, or how neutrosophic elements get influence-incidence in so varied universe.

THE STRUCTURE AND IDEAS
The volume is structured on three chapters approaching the neutrosophy effects from theory to pragmatic fields. Chapter 1, developed by professor Florentin Smarandache, is the visible path of Neutrosophy emergence. He presents the joints of Neutrosophy, describes neutrosophics phenomena, explains the operations of neutrosophics analytics and establish the platform for neutrosophic thinking.
Neutrosophy according to the dictionary definition is a branch of philosophy, which studies the origin, nature, and scope of neutralities, as well as their interactions with different ideational spectra. Neutrosophy considers a proposition, theory, event, idea, concept, or entity, "A" in relation to its opposite, "Anti-A" and that which is not A, "Non-A", and that which is neither "A" nor "Anti-A", denoted by "Neut-A". Neutrosophy is the basis of neutrosophic logic, neutrosophic probability, neutrosophic set, and neutrosophic statistics. The fundamental Thesis: Any idea is T % true, I % indeterminate, and F % false, where T, I, F are standard or non-standard subsets included in -0, 1 + .
The fundamental Theory: Every idea <A> tends to be neutralized, diminished, balanced by <Non-A> ideas (not only <Anti-A>, as Hegel asserted) -as a state of equilibrium.
Neutrosophy is the base of neutrosophic logic, a multiple value logic that generalizes the fuzzy logic, of neutrosophic set that generalize the fuzzy set, and of neutrosophic probability and neutrosophic statistics, which generalize the classical and imprecise probability and statistics respectively.
Did humanity really reach its insensitivity limit where the only reason, where sensible permanently lost his existential value? If so, that means Albert Camus is right: the only logical solution is suicide. To escape the darkness of death, the nightmares that we set ourselves in her name we have several solutions including apparent suicide, or why not the life performing optimism. Suicide is <anti-A>; to genuine optimism represented by neutrosophic <A>. If we accept the suicide or its equivalent or <anti-A> is such as if we should accept to cut off one's nose to spite one's face. So says also Brancusi, he doesn't create the beauty, he just removes unnecessary material to be easier for us to discover new beauty next to him. Similarly we define (is removed) <anti-A> for beauty and for its sense, to be visible the beauty of our existence in front of nonexistence. Of nonexistence fears any existence, even the Universe itself, maybe nonexistence in itself is not afraid of itself, or people who in their existence forget or do not know that they exist there. Similarly we define (we remove) <anti A> for the beauty and its sense, to be visible the beauty of our existence in front of the nonexistence. Chapter 2, developed by Florentin Smarandache and Ştefan Vlăduţescu, highlights the emergence and the development of a communication-collaboration neutrosophic function of universities (any communication-collaboration includes a mandatory component of neutrality that allows entities to put something together or to create something together).
The universities are perceived as institutions that along their evolution and development in time performed and developed such features like: understanding the university being conceptual elusive understanding, university as very complex organization, university as a system (input, output, process), university as teaching-learning, research entities, university as communication, collaboration and integration functions.
In Information Age, university system is structured mainly by actions of factors like: computing, technology, rapid innovation determined "the computing power of microchips double every 18 months", accelerate extension of the information-communication systems generating "the total bandwidth of communication systems triples every 12 months", development and accreditation of a collaborative and disseminating academic environment applying the principle "shared knowledge", liberalization of knowledge flows and setting the new dissemination channels, in a moment when "globalization and digitalization" are "the two most profound social and economic trends of our age". Professors Vlăduţescu and Smarandache through their extensive experience in academicals, initiate and support the fifth function assuring: "We value and fight, for strengthening and developing the communicative-collaborative-integrative functions of the global university system. If the Digital Age brings, however, globalization and interdependence, we should not expect that they be imposed, but we should welcome them. It is good to settle all opportunities from challenging. It would be a beneficial and wondering feed-forward response. In fact, some steps towards this emerging fifth function are already taken".
In Chapter 3, professor Ştefan Vlăduţescu explores the world of multi-field and multistructured information. Zetetic objective is to define the coordinates of the third of the fundamental elements of the world, along with the matter and energy: information.

96
Volume 38 Ontological examination shows that concerning configuration of the information have expressed a lot of opinions. The argumentation is oriented in direction of four coordinate reference system: a coordinate, the metric is quantitative and three coordinates (semantic, structural, pragmatic) are qualitative. Information makes order and input an organization. Semantic side of information includes, the transmission of knowledge horizon. Investigation of this aspect must find or even look for that tells the receiver message about existence. On structural coordinate, information appears as element of knowledge in relation with the structure and functioning of the world. Information, as event of gnosis practice, doesn't find the goal in itself, so that the utility is one of its essential features. The pragmatic quality, utility extends therefore between pleasure and achieving the goal and is manifested in making decisions and performing processes.
Concerning the novelty of information, the author asserts, "as "in any true poetry is designed a sacred action", in any communication is projected a communication action. Each of these lives as a reality in which peace, stability and balance are turned to the essence of novelty of the world: non-original meaning we are gradually revealed, it becomes order, meaning and sense. The man introduces sense. He is, however, first a disorientation that causes a tenseness leading to the deviations. Novelty increase from a significant tension that can take different forms: dissonance, deviation, surprise, breaking, difference".
Information limits are obscure, ambiguous, uncertainty. The border that separates the world of chaos is the same. This is not because the information is itself ambiguous and insignificant, but because its limits are very large, difficult to fix. Against appearances, however, the information sets limits of the world. "I am a human being I consider nothing that is human alien to me" said Terence. In a paraphrase: I am human and none of what is informative I am not foreign, because humanity, the world is information.

CONCLUSIONS
The book, Neutrosophic emergences and incidences in communication and information by Florentin Smarandache and Ştefan Vlăduţescu, represents the buoy that marks the transition of neutrosophy from the emergence phase to the incidence phase. Like any entity, ideas are born and acquire emergence. Some ideas impose and pass to the stage of incidence. Other ideas fade away. For some ideational configurations, the emergence matures into incidence, for others the emergence gets into implosion. In other words, some emergences are validated and accredited, and others get into the implosion. Any emergence is a promise. When they keep their promises, they are felt as strong ideas, as influential ideas and, thus, they have an impact in different areas of the universe of knowledge. Emergence is a tectonic development. Incidence is emergence applicability and its certification. The book is underlain on a solid ideational, well-founded argumentative, bringing new concepts in communication, coherent and cohesive written. Remaining into the neutrosophical coordinates, we have to remind the C. J. Jung's statement: "The pendulum of the mind oscillates between sense and nonsense, not between right and wrong".
International Letters of Social and Humanistic Sciences Vol. 38


Harmonising Research Reporting in the UK - Experiences and Outputs from UKRISS

Abstract The Jisc-funded UK Research Information Shared Service (UKRISS) project investigated the reporting of research information across the UK HE sector and assessed the feasibility of a national infrastructure based on CERIF with the objective of increasing the efficiency, productivity and reporting quality across the sector. A core reporting profile was developed that would enable harmonised reporting on RCUK-funded research, taking into account the HE-BCI survey as well as REF reporting elements. In this paper we describe the UKRISS modelling approach and provide some insight into the UKRISS reporting objects to support understanding of their formal CERIF representations, i.e. the selection of underlying CERIF entities; the challenges with managing objects and aggregations in CERIF. Example data extracts demonstrate the work.

Introduction and Background
This paper presents some of the findings of a feasibility and proof-of-concept study into the reporting of research information at a national level within the UK, based on CERIF. The study 1 was carried out by the Jisc-funded UK Research Information Shared Service (UKRISS) project 2 .
The reporting of research information is a complex and expensive activity for research organisations. The primary focus of UKRISS was reporting on research projects to the seven government-funded research councils within the RCUK umbrella organisation. Each council has responsibility for a specific area of research, such as engineering and science (EPSRC) or medical (MRC), and budgets are managed largely autonomously. In the past several years, five of the research councils (AHRC, BBSRC, EPSRC, ESRC, NERC) have implemented an in-house Current Research Information System (CRIS) called the Research Outcomes System (ROS), whereas the remaining two (MRC and STFC) deployed a commercial CRIS called Research Fish. The research information gathered by the RCUK funders is used for a variety of purposes. These include demonstration to government of the impact and value of the funded activities, scoping of future funding programmes and public dissemination.
The data models used across, and even within the reporting systems are substantially different for a number of reasons. At the highest level, the research councils often have their own business objectives for the information fields collected and the way it should be prioritised. There are substantial differences related to the specific disciplines such as the relevance of commercial exploitation of the outputs and differing terminology. Even where there is close alignment of the reporting fields, there are detailed semantic differences that make it difficult to combine and reuse information gathered. Another significant difference between the ROS and Research Fish approaches is that Research Fish relies primarily on direct entry by project PIs, whereas ROS has increasingly moved to harvest information from institutional systems. The data quality of the research information is in some areas a major concern, particularly with the heavy reliance on manual data entry.
As well as the seven government-funded Research Councils (RCUK), a large amount of public research in the UK is funded by charities as well as private organisations. The fields collected relating to research outputs tend to overlap with the information collected by research councils. However, there are also differences. For example, charities tend to place a greater focus on impact narratives that can be used to support future fund-raising.
Research information is also collected approximately every six years in the UK by HEFCE for the large scale national research assessments known as the Research Excellence Framework (REF). HESA also collect for the annual HE-BCI knowledge transfer survey. Both of these were referenced in the harmonisation analysis.
The European Commission is a major funder of research in the UK, but was not in the scope of the current study.
Although UKRISS was focused on research information reporting within the UK, the lack of harmonisation between reporting requirements is also an issue in other countries with multiple funding agencies. For example, Germany has recently launched a project 3 that aims to investigate harmonisation of national reporting. Smaller countries such as Norway 4 and the Czech Republic 5 have less complex funding models and have demonstrated the value and efficiency savings of being able to report and analyse research information at a national level.
The initial remit of the UKRISS project was to assess the feasibility of a national reporting infrastructure. A wide-ranging feasibility study was conducted involving interviews with a representative sample of over forty stakeholders from across the sector including funders, institutions, government bodies, umbrella bodies, CRIS vendors and charity funders. A detailed summary of the finding of the study is described in 1 . The study explored the drivers for harmonisation and requirements for a national reporting infrastructure. The study made three main recommendations: 1. Specification, standardisation and adoption of a core CERIF profile for reporting of research information in UK HEIs. 2. Implementation of a national reporting infrastructure and associated shared services to facilitate the exchange of research information between IT systems within institutions, funders and statutory bodies. 3. Provision of benchmarking tools that enable comparison and analysis of research information generated by multiple organisations for management information purposes. It is important to make the distinction between reporting services and a reporting system. Stakeholders were not in favour of a national reporting system for a number of reasons. In particular, substantial investments in infrastructure by funders, institutions and government have already been made and there was a wish that any solution should interoperate with these existing systems rather than replacing them.
Recommendation 1 was considered to be a fundamental prerequisite for implementing recommendations 2 and 3. At the same time, there was a need both to demonstrate the feasibility of a national infrastructure as well as some of the potential benefits such as demonstration of enhanced benchmarking and business intelligence functionalities.
With respect to recommendation 1, UKRISS defined the full profile as an aggregation of all reporting fields collected by an agreed set of sector bodies such as the research councils. The core profile is defined as a set of fields that are common or sufficiently similar in order to be mapped to a single reporting field. The main aim of the modelling work was to investigate the degree to which harmonisation between initially the RCUK reporting fields could be achieved, to highlight areas where there were significant discrepancies and the reasons for those differences, and where possible to make suitable harmonised definitions.
The Common European Research Information Format (CERIF) was used as the basis for the harmonisation work 6,7 . CERIF has emerged as the preferred format for expressing research information across Europe 8,9,10 , and was recommended for adoption by the UK HE sector by the Jisc-funded EXRI-UK report of 2009 11 . Since then CERIF has been piloted for specific applications, but not as a format for reporting requirements across all UK research organisations. Although CERIF allows for a formal description of the UKRISS reporting model, it does not initially provide all the semantics; i.e. object boundary specifications and applicable vocabularies. The UKRISS project has been working closely with the UK chapter of CASRAI, to move towards a standardised set of vocabularies, which are also aligned as far as possible with international conventions 12 .
The remainder of the paper is organised as follows. Section 2 contains an analysis of the approach to harmonisation of the RCUK reporting fields and the UKRISS core profile. Section 3 describes validation tools that were developed to demonstrate the value of the core profile for improving data quality. In section 4, we describe the UKRISS Crosswalk connector that provides a lightweight tool for extracting information from multiple institutional IT systems, packaging the information in CERIF format and mapping to a reporting template, which can then be transferred to a funder reporting system. Section 5 presents the lessons learned and section 6 the summary and conclusions.  Reporting records are instantiations of UKRISS reporting objects and assignments to upper level reporting concepts do not necessarily follow a linear process or sequence. In practice, a clear or single assignment also varies according to the use case. For example, an institution could view Staff Development as a Measurement whereas funders view it as a Research Outcome. Different and multiple assignments may thus be required or even suggested in support of and depending on multiple stakeholder viewpoints. Figure 1 anticipates a funder's viewpoint.

Harmonisation Analysis
UKRISS defines harmonisation as 'semantic similarity' between comparable entities where entities can be objects, fields or applicable vocabularies. Similarity at field level would for example be given with Volume or Volume Number, or likewise with Publication Type and Type of Publication within the boundaries of the reporting object Publication. Figure 2 illustrates harmonisation degrees between ROS (blue) and Research Fish (red) reporting objects before (green) and after (purple) the introduction of the UKRISS model. The harmonised UKRISS reporting objects count less string fields and apply a number of relationships instead. Following a CERIF structure this implies the use -and in an optimum case, the re-use -of controlled vocabularies (turquois), and unique identifiers such as e.g. DOI and ORCID. UKRISS investigated inherent and potential degrees of harmonisation within the mentioned UK reporting systems and suggested a gradual approach towards harmonisation guided by conceptual clarity and structural changes in support of sustainability, technological standardisation and use of shared identifiers. In addition to existing reporting objects, UKRISS introduces Dataset and Event as new reporting objects, and recommends that a future Impact object should allow for the recording and time tracking beyond project boundaries. Furthermore Equipment and Education should be added as future reporting objects to bridge gaps with Higher Education reporting and Research Infrastructures.
Before any implementation UKRISS recommends a clear harmonisation of underlying business requirements across the sector -involving a maximum number of key stakeholders to guide the priorities and the approach towards implementation of proposed extensions -and the rules that apply.

Core Information Reporting Profile
The introduced UKRISS Core Information Reporting Profile upper level reporting concepts were understood as follows (inspired by the ATN approach) 13 : Measurement: Yet to be defined following agreed indicators. Considered important for future comparison amongst data users and data producers.
The entire UKRISS Reporting Profile and its underlying CERIF aggregations are presented in figure 3, where reporting types are subsumed by record types; that is, Publication is subsumed under Research Output, Spin-Out and Further Funding is subsumed under Research Outcome.

Fig. 3. UKRISS Core Research Information Reporting Profile in CERIF including aggregations
representing the use case "institution submits report to funder". Figure 3 reflects the use case through the top two CERIF organisation entities cfOrganisationUnit with arrows showing the information flow through a report object -an aggregation of multiple CERIF entities and vocabularies -from an institution to a funder. This ukriss:report object in figure 3 aggregates all UKRISS reporting objects and vocabularies, while each UKRISS reporting object is itself an aggregation of CERIF entities and vocabularies.
The entire UKRISS Core Information Reporting Profile is available in an Excel sheet from the UKRISS project website: http://ukriss.cerch.kcl.ac.uk/, with CERIF mappings for each reporting object. More detailed documentation is provided with the final UKRISS report and CERIF XML example files for each UKRISS reporting object 14 . Where Research Output objects such as Publication, Dataset, Intellectual Property, Event, or Award are explicit entities in the CERIF model -namely cfResultPublication, cfResultProduct, cfResultPatent, cfEvent, cfPrize -this is not the case with Research Transfer objects or Research Outcome objects, which are modelled as cfResultProduct in a first instance, while aggregating functionally related entities in a second instance. For example, Staff Development is at first considered an outcome cfResultProduct and secondly a person and its aggregates. The same holds for Collaboration at first considered an outcome, cfResultProduct and secondly an organisation and its aggregates (see also figures 1 and 3).

Example UKRISS Reporting Object Extracts
The UKRISS model anticipates the recommended harmonisation changes. Some example extracts of reporting objects in CERIF XML illustrate the underlying model and its implementation.  The modelling of the UKRISS reporting objects did not require any extensions to the CERIF model. However, it encouraged consideration of a more standardised / formal approach with respect to CERIF object boundary definitions and their application according to specified requirements or 'business rules'. These will additionally support any related tool implementation and data validation.

UKRISS Validation, Visualisation and Aggregation Tools
UKRISS developed a number of practical proof-of-concept tools based on the core profile. These tools demonstrate the wider range and power of harmonised models to support research information interchange and reuse.
Harmonisation has the potential to improve data quality, enhance understanding through visualisation tools and enable advanced analytics through data aggregation.
Validation is the process of ensuring compliance with the model and the quality of the data represented therein. Having a well-understood document format, with known data types in known fields (e.g. knowing when a field is supposed to be a date, and even what format the date should be represented in) means that generic validation software can be developed, which can then be used by and to the benefit of an organisation that works with the information. During the course of the UKRISS project a tool was developed to validate individual elements of the model (such as checking that an ISSN is an ISSN). It can also look up data in external data-sources and crossreference with the document being validated in support of consistency with the metadata across the community.
Validated data records can then be visualised according to the UKRISS model based on formal CERIF (see figure 6). A well-structured and a wellunderstood data model is also a pre-requisite for quality data aggregation.

UKRISS Crosswalk Connector
An open source connecter has been developed to extract research data from existing institutional systems, transform the data into the UKRISS CERIF format, and load the data securely to a location accessible by the external data recipient, e.g. the funding body as shown in figure 7.
The guiding development principles were that it should be free to download, with a low barrier to entry in that it should be easy to install and intuitive to use without need for extensive technical resources or expertise, and thus be valuable not only to research intensive institutions, but also to smaller organisations. Fig. 7. Schematic of the Crosswalk Connector interfaces.
One of the principal benefits of aggregating data from multiple sources derives from how effective this process is at highlighting errors and inconsistencies with source data. More information is available in the final report 14 .

Lessons Learned
We have demonstrated the feasibility and potential benefits of harmonised research reporting guided by a formal domain model for the multiple involved organisations in the research ecosystem. Such a model ensures the consistent usage and re-usage of defined reporting objects and sub-objects. All stakeholders need to participate in the definition of requirements to ensure a conceptual clarity and as well support with the process of implementation, if harmonisation is to benefit the national, and subsequently the global research ecosystem.

Summary and Conclusion
Harmonised national research reporting can only be achieved through agreement between the stakeholders involved. Clearly defined requirements are the pre-requisite for sustainable conceptual descriptions and enable a consistent and valid implementation nationally and internationally. CERIF guided the harmonisation process by providing a research domain model that allows for the formal representation of any context, profile or object. However, CERIF needs to dedicate further thoughts (such as rules) towards formal definitions of contextual or object boundaries and their applicable vocabularies.


extfieldsofstudy:
Computer Science
Mathematics

text:
Sequential Model-Based Ensemble Optimization

One of the most tedious tasks in the application of machine learning is model selection, i.e. hyperparameter selection. Fortunately, recent progress has been made in the automation of this process, through the use of sequential model-based optimization (SMBO) methods. This can be used to optimize a cross-validation performance of a learning algorithm over the value of its hyperparameters. However, it is well known that ensembles of learned models almost consistently outperform a single model, even if properly selected. In this paper, we thus propose an extension of SMBO methods that automatically constructs such ensembles. This method builds on a recently proposed ensemble construction paradigm known as agnostic Bayesian learning. In experiments on 22 regression and 39 classification data sets, we confirm the success of this proposed approach, which is able to outperform model selection with SMBO.

Introduction
The automation of hyperparameter selection is an important step towards making the practice of machine learning more approachable to the non-expert and increases its impact on data reliant sciences. Significant progress has been made recently, with many methods reporting success in tuning a large variety of algorithms (Bergstra et al., 2011;Hutter et al., 2011;Snoek et al., 2012). One successful general paradigm is known as Sequential Model-Based Optimization (SMBO). It is based on a process that alternates between the proposal of a new hyperparameter configura-tion to test and the update of an adaptive model of the relationship between hyperparameter configurations and their holdout set performances. Thus, as the model learns about this relationship, it increases its ability to suggest improved hyperparameter configurations and gradually converges to the best solution.
While finding the single best model configuration is useful, better performance is often obtained by, instead, combining several (good) models into an ensemble. This was best illustrated by the winning entry of the Netflix competition, which combined a variety of models (Bell et al., 2007). Even if one concentrates on a single learning algorithm, combining models produced by using different hyperparameters is also helpful. Intuitively, models with comparable performances are still likely to generalize differently across the input space and produce different patterns of errors. By averaging their predictions, we can hope that the majority of models actually perform well on any given input and will move the ensemble towards better predictions globally, by dominating the average. In other words, the averaging of several comparable models reduces the variance of our predictor compared to each individual in the ensemble, while not sacrificing too much in terms of bias.
However, constructing such ensembles is just as tedious as performing model selection and at least as important in the successful deployment of machine-learning-based systems. Moreover, unlike the model selection case for which SMBO can be used, no comparable automatic ensemble construction methods have been developed thus far. The current methods of choice remain trial and error or exhaustive grid search for exploring the space of models to combine, followed by a selection or weighting strategy which is often an heuristic. arXiv:1402.0796v1 [cs.LG] 4 Feb 2014 In this paper, we propose a method for leveraging the recent research on SMBO in order to generate an ensemble of models, as opposed to the single best model. The proposed approach builds on the agnostic Bayes framework (Lacoste et al., 2014), which provides a successful strategy for weighting a predetermined and finite set of models (already trained) into an ensemble. Using a successful SMBO method, we show how we can effectively generalize this framework to the case of an infinite space of models (indexed by its hyperparameter space). The resulting method is simple and highly efficient. Our experiments on 22 regression and 39 classification data sets confirm that it outperforms the regular SMBO model selection method.
The paper develops as follows. First, we describe SMBO and its use for hyperparameter selection (Section 2). We follow with a description of the agnostic Bayes framework and present a bootstrap-based implementation of it (Section 3). Then, we describe the proposed algorithm for automatically constructing an ensemble using SMBO (Section 4). Finally, related work is discussed (Section 5) and the experimental comparisons are presented (Section 6).

Hyperparameter selection with SMBO
Let us first lay down the notation we will be using to describe the task of model selection for a machine learning algorithm. In this setup, a task D corresponds to a probability distribution over the input-output space X ×Y. Given a set of examples S ∼ D m (which will be our holdout validation set), the objective is to find, among a set H, the best function h : X → Y. In general, H can be any set and we refer to a member as a predictor. In the context of hyperparameter selection, H corresponds to the set of models trained on a training set T ∼ D n (disjoint from S), for different configurations of the learning algorithm's hyperparameters γ. Namely, let A γ be the learning algorithm with a hyperparameter configuration γ ∈ Γ, we will note h γ = A γ (T ) the predictor obtained after training on T . The set H contains all predictors obtained from each γ ∈ Γ when A γ is trained on T , i.e. H def = {h γ |γ ∈ Γ}.
To assess the quality of a predictor, we use a loss function L : Y × Y → R that quantifies the penalty incurred when h γ predicts h γ (x) while the true target is y. Then, we can define the risk R D (h γ ) as being the expected loss of h γ on task D, i.e. in hyperparameter selection has been testing a list of predefined configurations and selecting the best according to the loss function L on some holdout set of examples S. When a learning algorithm has more than one hyperparameter, a grid search is required, forcing |Γ| to grow exponentially with the number of hyperparameters. In addition, the search may yield a suboptimal result when the minimum lies outside of the grid or when there is not enough computational power for an appropriate grid resolution. Recently, randomized search has been advocated as a better replacement to grid search (Bergstra and Bengio, 2012). While it tends to be superior to grid search, it remains inefficient since its search is not informed by results of the sequence of hyperparameters that are tested.
To address these limitations, there has been an increasing amount of work on automatic hyperparameter optimization (Bergstra et al., 2011;Hutter et al., 2011;Snoek et al., 2012). Most rely on an approach called sequential based optimization (SMBO). The idea consists in treating R S (h γ ) def = f (γ) as a learnable function of γ, which we can learn from the observations {(γ i , R S (h γi ))} collected during the hyperparameter selection process.
We must thus choose a model family for f . A common choice is a Gaussian process (GP) representation, which allows us to represent our uncertainty about f , i.e. our uncertainty about the value of f (γ * ) at any unobserved hyperparameter configuration γ * . This uncertainty can then be leveraged to determine an acquisition function that suggests the most promising hyperparameter configuration to test next.
Namely, let functions µ : Γ → R and K : Γ × Γ → R be the mean and covariance kernel functions of our GP over f . Let us also denote the set of the M previous evaluations as (1) where R S (h γi ) is the empirical risk of h γi on set S, i.e. the holdout set error for hyperparameter γ.
The GP assumption on f implies that the conditional dis- where N (f (γ * ); µ(γ * ; R), σ 2 (γ * ; R) is the Gaussian density function with mean µ(γ * ; R) and variance σ 2 (γ * ; R). We also have vectors There are several choices for the acquisition function. One that has been used with success is the one maximizing the expected improvement: (2) which can be shown to be equal to where Φ is the cumulative distribution function of the standard normal and The acquisition function thus maximizes Equation 3 and returns its solution. This optimization can be performed by gradient ascent initialized at points distributed across the hyperparameter space according to a Sobol sequence, in order to maximize the chance of finding a global optima. One advantage of expected improvement is that it directly offers a solution to the exploration-exploitation trade-off that hyperparameter selection faces.
An iteration of SMBO requires fitting the GP to the current set of tested hyperparameters R (initially empty), invoking the acquisition function, running the learning algorithm with the suggested hyperparameters and adding the result to R. This procedure is expressed in Algorithm 1. Fitting the GP corresponds to learning the mean and covariance functions hyperparameters to the collected data. This can be performed either by maximizing the data's marginal likelihood or defining priors over the hyperparameters and sampling from the posterior using sampling (see Snoek et al. (2012) for more details).

Algorithm 1 SMBO hyperparameter optimization with GPs
While SMBO hyperparameter optimization can produce very good predictors, it can also suffer from overfitting on the validation set, especially for high-dimensional hyperparameter spaces. This is in part why an ensemble of predictors are often preferable in practice. Properly extending SMBO to the construction of ensembles is, however, not obvious. Here, we propose one such successful extension, building on the framework of Agnostic Bayes learning, described in the next section.

Agnostic Bayes
In this section, we offer a brief overview of the Agnostic Bayes learning paradigm presented in Lacoste et al. (2014) and serving as a basis for the algorithm we present in this paper. Agnostic Bayes learning was used in Lacoste et al. (2014) as a framework for successfully constructing ensembles when the number of predictors in H (i.e. the potential hyperparameter configurations Γ) was constrained to be finite (e.g. by restricting the space to a grid). In our context, we can thus enumerate the possible hyperparameter configurations from γ 1 to γ |Γ| . This paper will generalize this approach to the infinite case later.
Agnostic Bayes learning attempts to directly address the problem of inferring what is the best function h in H, according to the loss function L. It infers a posterior p h (h γ |S), i.e. a distribution over how likely each member of H is the best predictor. This is in contrast with standard Bayesian learning, which implicitly assumes that H contains the true data-generating model and infers a distribution for how likely each member of H has generated the data (irrespective of what the loss L is). From p h (h γ |S), by marginalizing h and selecting the most probable prediction, we obtain the following ensemble decision rule: To estimate p h (h γ |S), Agnostic Bayes learning uses the set of losses l γ,i def = L(h γ (x i ), y i ) of each example (x i , y i ) ∈ S as evidence for inference. In Lacoste et al. (2014), a few different approaches are proposed and analyzed. A general strategy is to assume a joint prior p(r) over the risks r γ def = R D (h γ ) of all possible hyperparameter configurations and choose a joint observation p(l γ,i ∀γ ∈ Γ|r) for the losses. From Bayes rule, we obtain the posterior p(r|S) from which we can compute with a Monte Carlo estimate. This would result in repeatedly sampling from p(r|S) and counting the number of times each γ has the smallest sampled risk r γ to estimate p h (h γ |S). Similarly, samples from p h (h γ |S) could be obtained by sampling a risk vector r from p(r|S) and returning the predictor h γ with the lowest sampled risk. The ensemble decision rule of Equation 4 could then be implemented by repeatedly sampling from p h (h γ |S) to construct the ensemble of predictors and using their average as the ensemble's prediction.
Among the methods explored in Lacoste et al. (2014) to obtain samples from p(r|S), the bootstrap approach stands out for its efficiency and simplicity. Namely, to obtain a sample from p(r|S), we sample with replacement from S to obtain S and return the vector of empirical risks [R S (h γ1 ), . . . , R S (h γ |Γ| )] as a sample. While bootstrap only serves as a "poor man's" posterior, it can be shown to be statistically related to a proper model with Dirichlet priors and its empirical performance was shown to be equivalent (Lacoste et al., 2014).
When the bootstrap method is used to obtain samples from p h (h γ |S), the complete procedure for generating each ensemble member can be summarized by where h is a returned sample.

Agnostic Bayes ensemble with SMBO
We now present our proposed method for automatically constructing an ensemble, without having to restrict Γ (or, equivalently H) to a finite subset of hyperparameters.
As described in Section 3, to sample a predictor from the Agnostic Bayes bootstrap method, it suffices to obtain a bootstrap S from S and solve the optimization problem of Equation 6. In our context where H is possibly an infinite set of models trained on the training set T for any hyperparameter configuration γ, Equation 6 corresponds in fact to hyperparameter optimization where the holdout set is S instead of S.
This suggests a simple procedure for building an ensemble of N predictors according to agnostic Bayes i.e., that reflects our uncertainty about the true best model h . We could repeat the full SMBO hyperparameter optimization process N times, with different bootstrap S j , for j ∈ {1, 2, . . . , N }. However, for large ensembles, performing N runs of SMBO can be computationally expensive, since each run would need to train its own sequence of models.
We can notice however that predictors are always trained on the same training set T , no matter in which run of SMBO they were trained on. We propose a handy trick that exploits this observation to greatly accelerate the construction of the ensemble by almost a factor of N . Specifically, we propose to simultaneously optimize all N problems in a round-robin fashion. Thus, we maintain N different histories of evaluation R j , for j ∈ {1, 2, . . . , N } and when a new predictor h γ = A γ (T ) is obtained, we update all R j with (γ, R S j (h γ )). Notice that the different histories R j contain the empirical risks on different bootstrap holdout sets, but they are all updated at the cost of training only a single predictor. Also, to avoid recalculating multiple times L(h γ (x i ), y i ), these values can be cached and shared in the computation of each R j . This leaves the task of updating all R j insignificant compared to the computational time usually required for training a predictor. This procedure is detailed in Algorithm 2.

Algorithm 2 Agnostic Bayes Ensemble With SMBO
By updating all R j at the same time, we trick each SMBO run by updating its history with points it did not suggest. This implies that the GP model behind each SMBO run will be able to condition on more observations then it would if the runs had been performed in isolation. This can only benefit the GPs and improve the quality of their suggestions.

Related Work
In the Bayesian learning literature, a common way of dealing with hyperparameters in probabilistic predictors is to define hyperpriors and perform posterior inference to integrate them out. This process often results in also constructing an ensemble of predictors with different hyperparameters, sampled from the posterior. Powerful MCMC methods have been developed in order to accommodate for different types of hyperparameter spaces, including infinite spaces.
However, this approach requires that the family predictors in question be probabilistic in order to apply Bayes rule. Moreover, even if the predictor family is probabilistic, the construction of the ensemble will entirely ignore the nature of the loss function that determines the measure of performance. The comparative advantage of the proposed Agnostic Bayes SMBO approach is thus that it can be used for any predictor family (probabilistic or not) and is loss-sensitive.
On the other hand, traditional ensemble methods such as Laviolette et al. (2011), Kim and Ghahramani (2012), and (Zhang et al., 2006) require a predefined set of models and are not straightforward to adapt to an infinite set of models.

Experiments
We now compare the SMBO ensemble approach (ESMBO) to three alternative methods for building a predictor from a machine learning algorithm with hyperparameters: • A single model, whose hyperparameters were selected by hyperparameter optimization with SMBO (SMBO).
• A single model, whose hyperparameters were selected by a randomized search (RS), which in practice is often superior to grid search (Bergstra and Bengio, 2012).
• An Agnostic Bayes ensemble constructed from a randomly selected set of hyperparameters (ERS).
Both ESMBO and SMBO used GP models of the holdout risk, with hyperparameters trained to maximize the marginal likelihood. A constant was used for the mean function, while the Matérn 5/2 kernel was used for the covariance function, with length scale parameters. The GP's parameters were obtained by maximizing the marginal likelihood and a different length scale was used for each dimension 2 .
Each method is allowed to evaluate 150 hyperparameter configurations. To compare their performances, we perform statistical tests on several different hyperparameter spaces over two different collections of data sets.

Hyperparameter Spaces
Here, we describe the hyperparameter spaces of all learning algorithms we employ in our experiments. Except for a custom implementation of the multilayer perceptron, we used scikit-learn 3 for the implementation of all other learning algorithms.

Support Vector Machine
We explore the soft margin parameter C for values ranging from 10 −2 to 10 3 on a logarithmic scale. We use the RBF kernel K(x, x ) = e γ||x−x || 2 2 and explore values of γ ranging from 10 −5 to 10 3 on a logarithmic scale.

Support Vector Regressor
We also use the RBF kernel and we explore the same values as for the Support Vector Machine. In addition, we explore the -tube parameter (Drucker et al., 1997) for values ranging between 10 −2 and 1 on a logarithmic scale.

Random Forest
We fix the number of trees to 100 and we explore two different ways of producing them: either the original Breiman (2001) method or the extremely randomized trees method of Geurts et al. (2006). We also explore the choice of bootstrapping or not the training set before generating a tree. Finally, the ratio of randomly considered features at each split for the construction of the trees is varied between 10 −4 and 1 on a linear scale.
Gradient Boosted Classifier This is a tree-based algorithm using boosting (Friedman, 2001). We fix the set of weak learners to 100 trees and take the maximum depth of each tree to be in {1, 2, . . . , 15}. The learning rate ranges between 10 −2 and 1 on a logarithmic scale. Finally, the ratio of randomly considered features at each split for the construction of the trees varies between 10 −3 and 1 on a linear scale.
Gradient Boosted Regressor We use the same parameters as for Gradient Boosted Classifier except that we explore a convex combination of the least square loss function and the least absolute deviation loss function. We also fix the ratio of considered features at each split to 1.

Multilayer Perceptron
We use a 2 hidden layers perceptron with tanh activation function and a softmax function on the last layer. We minimize the negative log likelihood using the L-BFGS algorithm. Thus there is no learning rate parameter. However, we used a different L2 regularizer weight for each of the 3 layers with values ranging from 10 −5 to 100 on a logarithmic scale. Also, the number of neurons on each layer can take values in {1, 2, . . . , 100}. In total, this yields a 5 dimensional hyperparameter space.

Comparing Methods On Multiple Data Sets
The different methods presented in this paper are generic and are meant to work across different tasks. It is thus crucial that we evaluate them on several data sets using metrics that do not assume commensurability across tasks (Demšar, 2006). The metrics of choice are thus the expected rank and the pairwise winning frequency. Let A i (T j , S j ) be either one of our K = 4 model selection/ensemble construction algorithms run on the j th data set, with training set T j and validation set S j . When comparing K algorithms, the rank of (best or ensemble) predictor h i = A i (T j , S j ) on test set S test j is defined as Then, the expected rank of the i th method is obtained from the empirical average over the L data sets i.e., . When comparing algorithm A i against algorithm A l , the winning frequency 4 of A i is In the case of the expected rank, lower is better and for the winning frequency, it is the converse. Also, when K = 2, When the winning frequency ρ i,l > 0.5, we say that method A i is better than method A l . However, to make sure that this is not the outcome of chance, we use statistical tests such as the sign test and the Poisson Binomial test (PB test) (Lacoste et al., 2012). The PB test derives a posterior distribution over ρ i,l and integrates the probability mass above 0.5, denoted as Pr(A B). When Pr(A B) > 0.8, we say that the result is significant and when Pr(A B) > 0.9, we say that it is highly significant. Similarly for the sign test, when the p-value is lower than 0.1, it is significant and when lower than 0.05, it is highly significant.
To build a substantial collection of data sets, we used the AYSU collection (Ulaş et al., 2009) coming from the UCI and the Delve repositories and we added the MNIST data set. We also converted the multiclass data sets to binary classification by either merging classes or selecting pairs of classes. The resulting benchmark contains 39 data sets. We have also collected 22 regression data sets from the Louis Torgo collection 5 .

Table Notation
The result tables present the winning frequency for each pair of methods, where grayed out values represent redundant information. As a complement, we also add the expected rank of each method in the rightmost column and sort the table according to this metric. To report the conclusion of the Sign test and the PB test, we use colored dots, where orange means significant and green means highly significant. The first dot reports the result of the PB test and the second one, the Sign test. For more stable results, we average the values obtained during the last 15 iterations.

Analysis
Looking at the overall results over 7 different hyperparameter spaces in Table 1 and Table 2, we observe that ESMBO is never significantly outperformed by any other method and often outperforms the others. More precisely, it is either ranked first or tightly following ERS. Looking more closely, we see that the cases where ESMBO does not significantly outperform ERS concerns hyperparameter spaces of low complexity. For example, most hyperparameter configurations of Random Forest yield good generalization performances. Thus, these cases do not require an elaborate hyperparameter search method. On the other hand, when looking at more challenging hyperparameter spaces such as Support Vector Regression and Multilayer Perceptrons, we clearly see the benefits of combining SMBO with Agnostic Bayes.
As described in Section 4, ESMBO is alternating between N different SMBO optimizations and deviates from the natural sequence of SMBO. To see if this aspect of ESMBO can influence its convergence rate, we present a temporal analysis of the methods in Figure 1 and Figure 2. The left columns depict Pr(A B) for selected pairs of methods and the right columns present the expected rank of each method over time.
A general analysis clearly shows that there is no significant degradation in terms of convergence speed. In fact, we generally observe the opposite. More precisely, looking at Pr(ESMBO SMBO), the green curve of the left columns, it usually reaches a significantly better state right at the beginning or within the first few iterations. A notable   exception to that trend occurs with the Multiplayer Perceptrons, where SMBO is significantly better than ESMBO for a few iterations at the beginning. Then, it gets quickly outperformed by ESMBO.

Conclusion
We described a successful method for automatically constructing ensembles without requiring hand-selection of models or a grid search. The method can adapt the SMBO hyperparameter optimization algorithm so that it can produce an ensemble instead of a single model. Theoretically, the method is motivated by an Agnostic Bayesian paradigm which attempts to construct ensembles that reflect the uncertainty over which a model actually has the smallest true risk. The resulting method is easy to implement and comes with no extra computational cost at learning time. Its generalization performance and convergence speed are also dominant according to experiments on 22 regression and 39 classification data sets. Table 2. Pairwise win frequency for the 3 different classification hyperparameter spaces (Refer to Section 6.3 for the notation).


s2fieldsofstudy:
Computer Science
Mathematics

text:
Observations Concerning the probability of the existence of annihilators for balanced boolean functions

LFSR-based stream ciphers with nonlinear filters or combiners are susceptible to algebraic attacks using linearization methods to solve an overdefined system of nonlinear equations. And this process is greatly enhanced if the filtering or combining function has a low degree annihilator. To prevent such an attack, one would choose the parameters of that function so that the degree of its annihilator becomes large enough. As computing power is continuously increasing, a choice that seems secure today, becomes insecure tomorrow. Therefore, a tool is needed to estimate the probability of the existence of annihilators for balanced boolean functions with parameters that are beyond the current computing power. Based on experimental and calculational observations, we give in this paper an almost exact estimate of that probability, which represent a great improvement over the upper bound previously known.

Introduction
LFSR stream ciphers with combining nonlinear functions are very efficient in terms of their speed of generating the keystream and their ability to mathematical analysis with respect of controlling the period of their generated sequences. Although these ciphers are nonlinear, they are not immune against algebraic attacks. Indeed, there exists efficient algorithms for linearization and solving nonlinear sets of equations [6], when the degree of the combining function, which is ultimately determined by the number of its inputs, is not too large. Moreover, there also exist algorithms that can find low degree annihilators of such functions so that they can be used to simplify the linearization process, and consequently, recover the encryption key [1,5]. To escape such an attack, one would increase n, the number of the inputs to the combining function, which leads to a larger degree d of its annihilators, and therefore increases its algebraic immunity. The question now is how big n and d should be in order to have a secure system. Using the annihilator's finding algorithms, one can check whether a given combining function has an annihilator, but this process is not sufficient to make a concrete decision about the security of a system. A choice of n and d, that is secure today according to the mentioned checking process, will be insecure tomorrow due to the increasingly growing computing power. And choosing large number of inputs to combining function arbitrarily is not practical. True, the prices of bits are decreasing fast, but there are practical limits that can not be exceeded, especially because the memory requirements of the system grow exponentially with that number. In [5], an upper bound of the probability of the existence of balanced boolean functions annihilators has been derived, so one can predict at what range that number should be. Although that upper bound is good for practical purposes, it suffers from a fatal drawback. In fact, the value of the bound is very overestimated, giving probability values "greater" than 1 for certain values of n an d. And in [2], a better upper bound is derived. In what follows, we present an almost exact formula for the probability of the existence of annihilators of a combining nonlinear boolean function, based on some empirical and calculational observations, but without a regirous mathematical proof. Our interest here is focused on balanced boolean functions, because balanced functions are a requirement in cipher systems.
The paper is organized as follows. In section 2 some definitions are given. In section 3, the algorithms of finding annihilators are quoted from [5]. In section 4, some experimental observations concerning the behavior of annihilators are described. Finally, in section 5, an analysis of the probability of the existence of annihilators for balanced boolean functions is made, following the approach of [5].

Definitions
We present some definitions that are necessary for the rest of the paper.
be a boolean function of n inputs. We call

Finding the annihilators of f
. A necessary and sufficient condition for g to be an annihilator of f, i.e. for 0   g f , is that g vanishes for all arguments x for which 1 ) ( . This leads to the following algorithm: Algorithm 1 (quoted from [5]): Of course, the number of unknowns in this system of linear equations is the number of the coefficients in the ANF of g, which equals s. Since we are interested here in balanced functions only, the number of equations that we can form, i.e. the value of N , will be 1 2  n . Note that this system of linear equations is Algorithm 1 is the basic algorithm for finding the annihilators of a boolean functions f . In [5], there is another, more computationally efficient algorithm, but in essence, it is the same as algorithm 1, although it may sometimes give wrong results.
Algorithm 2 (quoted from [5]): Note that if algorithm 2 is used to find an annihilator of a balanced function f, then all the 1 2  n equations must be involved in the process of solving the system. If one takes, for example, only m equations such that , then the result will be for a function f whose weight is m, not 1 2  n . But there is nothing in algorithm 2 that ensures that it takes all those 1 2  n equations into account. The double checking process of the solutions may partially compensate for that, but it is not sufficient on its own unless 0

Empirical Observations
In [5], an upper bound is given for the probability of the existence of annihilators for balanced boolean functions by theorem 4. This upper bound suffers from being overestimated, and worse, it gives in certain cases probability values "greater" than 1, whereas it is much less than 1, e.g. when 10  n and 4  d [5].
Searching for annihilators for 4 and 10   d n , by repetitively choosing random balanced functions f and applying algorithm 1, shows that this probability must be very small. Therefore there must be something wrong in theorem 4 of [5].
During the search for annihilators, the following observations have been made: , it was noticed that the weights of the annihilators equal, almost always, the minimum weight d n 2 , and very rarely, as shown in Fig. 1. Table 1 shows the curves parameters.
As can be seen from Fig. 1 Fig. 2 for all pairs of n and d listed in table 1. As can be seen from Fig. 2, the average rank r approaches the number of unknowns s minus 1 as the weight of f increases towards 1 2  n , which suggests that a balanced boolean function f admits only one annihilator if its degree .

For n odd and
, we found that many annihilators of different weights annihilate balanced boolean functions f. ) (

The probability of the existence of annihilators for random balanced boolean functions f of n variables
Before proceeding with the discussion of the probability, it is worth mentioning that algorithm 1 leads to the following two lemmas.
It should be clear that f N is the number of balanced functions f admitting an annihilator g of weight    reveals that its terms decrease very rapidly with increasing w. This fact, together with observation 1, leads us to conjecture again that the terms of 3  are also vanishingly negligible, making 1  , i.e. the first term of equation (2), the dominant one. This allows by all means expressing the final result that gives the probability of the existence of an annihilator g of degree d for a balanced function f with n inputs, when , in the following form with a very good accuracy:     1 2   n s , i.e. the number of available equations for algorithm 1 is equal to the number of unknowns. But by lemma 1 and lemma 2, these equations are not necessarily all linearly independent, and therefore the possibility of there being some non trivial solutions exists. On the other hand, removing just one equation from these equations leads always to non trivial solutions, i.e. the system of equations is on the verge of avalanche. This suggests that the probability of the existence of annihilators must be high.
According to observation 3, the annihilators can have many different weights whose enumerators are unknown, and therefore no further concrete statement can be made about the probability of annihilators. However, experiments suggest that:  Table 6 5.3 n even and 2 n d  By corollary 1-d, 1 2   n s , i.e. the number of available equations is less than the number of unknowns. This makes the system of linear equations in algorithm 1 underdefined, and therefore there is always non trivial solutions, giving: