abstract: Lev T. Kuzin (1928–1997) is one of the founders of modern cybernetics and information science in Russia. After getting his Doctor of Technical Sciences degree in the industrial research sphere he joined the Moscow Engineering Physical Institute at the time when there were a few specialists in cybernetics in Russia. A few years later he became a founder of Cybernetics department and a Chair of the section in Applied Problems for Cybernetics in Moscow House for Scientific and Technical Knowledge Propagation. Almost the same time he became a project principal researcher (project *Data Bank* promoted by the Russian Academy of Sciences), and attracted the teams of researchers to apply the computational models to a field of data and knowledge bases, making the pioneering research in Artificial Intelligence.
  .
  He was awarded and honored the USSR State Prize for inspiring vision into the future of technical cybernetics and his invention and innovation of key technologies.
  .
  The last years he interested in the computational models of geometrical and algebraic nature and their applications in various branches of computer science and information technologies. In the recent years the interest in computation models based on object notion has grown tremendously stimulating an interest to Kuzin's ideas. This year of 50th Anniversary of Cybernetics and on the occasion of his 70th birthday on September 12, 1998 seems especially appropriate for discussing Kuzin's Research Program.
author: Viacheslav Wolfengagen
Institute for Contemporary Education "JurInfoR-MSU",
Vorotnikovsky lane, 7, bld. 4
Moscow, 103006 Russia email@example.com
title: **L.T.Kuzin: Research Program**

Workshop on Computer Science and Information Technologies CSIT'99, Moscow, Russia, 1999

Dedicated to the memory of Professor Lev Kuzin

on the occasion of his 70th Birthday

# Introduction

The explanation to the term *research program of L.Kuzin* arose from the attempts to analyze the trends, directions and stages of his scientific activity. Kuzin did not used this expression in its own sense but often preferred some way of making the research and development work.

The obvious and simplest way is to declare that the general topic of his research was cybernetics and information science. Nevertheless, this explanation is not precise and exact. For better acceptance and understanding of his efforts, interests and activity it is important to pay attention to his basic aims and approaches. In case we forgot this general direction we have a chance to loose this understanding.

There has been some general way to understand and select out his research road. But there has been misunderstanding of his main aims and general research approach. One of the reasons for this is that many of his, especially early, tutorials, invited talks and keynote papers appeared in a few copies published in widely inaccessible paper collections or even with a limited access to copies. His ideas have changed with a time but the later ideas, as a rule, arose from the earlier preliminary contributions and projects. The evolution of his ideas with a convenience can be divided into four periods: 1) till early 60s, 2) from 1963 till 1975, 3) from 1975-1976 till 1983, and 4) after 1983-1986.

# Period till early 60s

Kuzin earned M.S. in Radio-physics as an Engineering Mathematician from Gorky State University in 1951. For 15 years he worked as Senior Scientist and Principal Scientist at the military research institutes where he earned his Ph.D. degree and Doctor of Technical Sciences degree.

This ten years period he worked in military research institutions at the governmental tasks in a field of technical cybernetics. He characterized his position as the "fitter and faultfinder of 10 thousands electronic tubes". He acquired a good experience in developing, fitting and inculcating of the large complex technical systems. But he felt a discomfort with a gap between the theory and engineering practice. On the one hand a theory was extremely abstract, long distant from the practical needs. On the other hand the developers of the complex systems had no effective engineering methods because of their lack. In particular, the educational background of the specialists was insufficient for a success of the development, they had an extreme needs for good background knowledge in system analysis, operation research, and cybernetics. That is why the research and development efforts were often directed to a blind alley, when this became obvious they were canceled out and re-initiated from the starting point. During five years he worked as a principal investigator in a theory of large complex technical systems, and he observed a frequent change of the senior managers for their failed projects.

All of this inspired his decision to change his position, join an educational institute and start the academic activity to contribute to the educational programs for managers, developers of the complex technical systems with well balanced engineering and theoretical ground. This time he was invited to join the Moscow Engineering Physical Institute and found a new department. In establishing the educational schedules he used his experience from Gorky State University, where due to the school of academician A. Andronov, his Professor, in 1946 the first in USSR technical radio-physics faculty aimed to educate the engineering mathematicians was established. A feature of this educational direction was the accommodation of existing methods and developing the new theoretical approaches to solving the real tasks arising during the development of a large complex technical system. The educational system has a lack of the clear written source books in cybernetics which were written at a good engineering level without a lost of mathematical generality, and Kuzin decided to start this new stage of his career.

# Period from 1963 to 1975

Lev Kuzin was awarded the USSR State Prize. Kuzin was honored for inspiring vision into the future of technical cybernetics and his invention and innovation of key technologies.

For nearly 35 years he worked permanently at Moscow Engineering Physical Institute where he founded the department initially named "Managing computing machines" and later renamed as "Cybernetics".

During the first two years he concentrated his efforts at the detailed preparations of the academic programs, plans and schedules in the new speciality to launch the long term activity for educating the engineering mathematicians.

He and his collaborators worked closely with industry, academia and governmental structures to promote a principal improvement of information infrastructure strongly oriented towards processing of the very large information bulks and later very large data bases and knowledge processing.

This was a stage in his career when he interested in interactive computing. He observed that in 60s and 70s automatic computing in Russia was mainly valued for data processing by the industrial communities. In 70s his interests moved permanently into the computer field, he paid a lot of attention of humans using computers interactively via CRT displays. After that he had become convinced that computers could make a difference between the humans way of thinking, everyday engineering reasoning and computational plans, so that he decided to contribute in handling the volumes of professional knowledge. He felt that this direction of research has a long range future and perspective and a lot of impacts towards a higher education.

# Period of 1975–1983

In the 1973–1976 period he wrote a series of reports, in which he set out a conceptual framework about how to organize the medium and environment where the humans could actually represent their knowledge. Later he was under an influence of ARPA whose leaders committed to the idea of establishing interactive computing with time-sharing.

By 1980 he was one of the principal investigators with a series of industrial projects and started working on a project on 'intelligent computing'. This time he was deeply interested in trying to recruit a vast community of people to start learning how to do the collaborative and multistage investigation of knowledge discovering and representation. He had some good teams of volunteers who started the partial research in a field. Often they worked days-and-nights to concentrate on fastest achievement even of the intermediate results and working software prototypes.

One of his proposals in 80s was to develop an extensible knowledge representation environment. His advice to the research community was to implement an enhanced OPS Information Technology, and a promising project was implemented by his collaborator Oleg Balovnev. This was the full scale OPS computational environment, the toolkit, information technology and corresponding particular applications were used for knowledge systems at the industrial enterprises giving rise to a fruitful research stream.

Nevertheless, in the first half of 70th most of the computing ideas moved more in the direction how to automate the industrial routine work with the huge databases than how to represent and discover the knowledge and establish the knowledge base which would be able to communicate with the databases. This period he launched several Ph.D. research topics to cover the problems of long-range future. He kept thinking about the humans with a high and deep professional training whose 'natural' knowledge could be transformed with the aid of knowledge engineers into a suitable computing schemata. He thought both of declarative and procedural approach trying to combine them into the uniform computational environment.

# After 1983

He felt that the professional in all fields could boost the quality of their knowledge work using the knowledge representation engines. But for a long time it was the single focus of his efforts, because there were a lot of years when he kept trying to save and continue the most promising working projects which were already launched and running in industry.

During the time in the 70s till the mid-80s he accumulated his experience, knowledge and intuition in a field of information systems in general. He observed the most important projects in government, academy, various research fields, industry to understand the scale and rate of their changes with a time. In each of the branches the changes were with external/internal knowledge schemata and operations, the challenge was for interoperable information exchange using, possibly, the relational databases and meta-relational knowledge bases. He was influenced by the 5th Generation Computing Systems project and tried to select out the basic, but outstanding and challenging ideas.

He used his Ph.D. students research as a platform from which, using the prototype software systems, to launch the promising computational ideas into industry. He felt importance of collaborative efforts with different participating organizations to cooperate in order to explore the possible and actual ways in which the technical systems of artificial intelligence can be done.

Intermediately he and his collaborators ended up generating a concept that they called the "Intelligent Data Bank", or shortly, IDB. He described the real phenomenon of transactions between the counterparts of this poli-based information human-machine system. This time Kuzin and his team launched a series of conferences and symposia to broader discuss the aims, targets and results in this direction. His main idea was based on the effects of group of experts behavior when they act collaboratively and try to get familiar with their environment and to coordinate the common resources with a purposeful efforts and actions. Under these circumstances they must integrate their fuzzy and incomplete knowledge into the entire IDB.

From the mid-80s with a group of supporters he kept pitching the topics of the intermediate direction between the computer science and information technologies. In particular, he was interested in an idea to apply the categorical universe of discourse to the field of data models which erased in the late 70th – early 80th.

He paid special attention to discuss applications of categories to the field of data modeling[^1]. His early idea was to use some kind of untyped calculi and impose some weak restrictions to capture the intutitive reasons concerning the data models. This approach tends to taking in mind a family of *computation models* as the restrictions of the untyped combinatory logic or untyped $\lambda$-calculus.

During more than a decade he worked as a project principal researcher (project *Data Bank* promoted by the Russian Academy of Sciences during 70th – early 80th), he attracted the teams of researchers to apply the computational models to a field of data and knowledge bases. Having launched in Russia some research projects in a field he discovered that the vital point of a general data model is to build an access to the computation environment. His proposal was to separate the *function* symbol and an *argument* symbol (when the functional expression is included within a query), then to generate access to the domain for argument and to the domain for function separately. The most of the dufficulties at this way were observed and some of them were resolved.

Later the more neutral approach was established, roughly speaking, to deal with all the *both-syntax-and-semantic* entities as with the homogeneous *objects*. The objects are interlinked by the relations which correspond to the *scripts* (script-driven data model). This model was extended to capture more *dynamics*. Studying the transitions between the scripts he verified some pure categorical models (with the explicit notion of a *state* in a computation model).

Nevertheless, the *semantics* of the computation in a category was not yet covered. During the last period of his research he was interested in discovering more and more facts which fit in the mathematical ideas beyond a category theory. In his draft research schedule he had done some improvements concerning the feasibility of a categorical abstract machine to evaluate the queries.

# Selected publications of Lev T.Kuzin

This is not a complete list of Kuzin's contributions, reports, papers, and books. A lot of his reports were published just as the very short theses in hardly available and accessed collections. A part of them were known in details as manuscripts. Another part was represented only as a collections of transparencies which he used during his regular reports at the monthly seminar "Artificial Intelligence" in Moscow House for Scientific and Technical Knowledge Propagation which existed more than a decade. Many of his papers were contributed with the collaborating co-authors. This list is compiled to reflect Kuzin's most promising ideas and challenges, and corresponds to the list of the event which he inspired or stimulated.

In addition, Kuzin was Editor-in-Chief of the periodical "Engineering matematical methods for physics and cybernetics" (Atomizdat, Moscow, Russia). His aims were to stimulate, step by step, the research activity and publications from an open position of Information Technologies. The main target was to launch the development of engineering mathematical methods based on the computer technologies in the various scientific fields. He applied efforts to select out and review the contributions in a field of cybernetics. This periodical has attracted the research communities and stimulated their activity in a manner and in an area of engineering mathematicians.

# Some of the prominent events stimulated, initiated and/or inspired by Lev Kuzin

This list includes some of the most important events which were inspired, stimulated and/or initiated by Lev T. Kuzin's and is not complete. The comments to his publications and contributions are shortly given because some of his works were never published as a complete papers, or were published in a minimal amount of copies. His most important publications are printed in Russian.

**1963**

He founded at Moscow Engineering Physical Institute the department No 22 "Controlling electronic computing machines" later renamed as "Cybernetics". A few time later he was awarded the USSR State Prize for contributing in technical cybernetics.

**1971**

Seminar at the Moscow House for Scientific and Technical Knowledge Propagation (MHSTKP)

**1973**

He analyzed the state of implementing and applications of Management Information Systems (MISs) in the USSR at the seminar with government representatives. He gave a system analysis of MIS in a scale of the big city like Moscow. This was aimed to effective using of computers and mathematical methods in industry and research institutions.

**1974**

. Seminar at the Moscow House for Scientific and Technical Knowledge Propagation (MHSTKP)

**1976**

He was one of the principal officers at the USSR conference Management Information Systems on May 10-13, Tbilisi, Georgia.

This was a forum for the scientists of the country. He had the invited talk on the problems of modeling for MIS. He summed up the results of applying the continuous mathematics under the new conditions in developing MISs. Kuzin paid attention to a strong necessity to use and establish the mathematical models of other types which previously were unknown because "…*the actual and existing industrial conditions need the frequent changes of criteria, the ranges of possible values for controlled parameters, and the functional structure and restrictions are changeable. The usual means from the general mathematical ware for computer do not give to the specialist from industry an ability to find out the controls which are optimal under the ad hoc industrial situation, without troublesome interference of the mathematicians and programmers*".

One of the possible solutions on his opinion were DBMSs. In particular, he had a serious interest to the relational databases, he inspired and personally assisted the research in this area. Usually he gave to the beginner in this field all the needed support helping to overcome both the principle scientific and purely practical difficulties.

At this event he moderated some round table discussions. Many of his listeners later became the serious researchers and high skilled specialist both in MIS software and in relational DBMS development and applications.

**1977**

. He was a General Chair at the 1-st USSR workshop "Intelligent Data Banks", Sukhumi, Abkhazia.

In his tutorial Kuzin started up generating a concept of the Intelligent Data Bank (IDB). He determined this kind of information systems as "…*having an ability to generate or derive the new information which was not previously explicitly present, and having the following properties: interface via professionally restricted natural language, multi-user mode, self-learning via open dialog interface*".

In fact, this was a premier workshop in USSR in this direction which was aimed to disseminate the AI ideas all over the country. In particular, Professors G.G.Chogowadze, G.G.Gogichaishwili re-directed their collaborators in Tbilisi, Georgia, towards more close usage of the AI principles and approaches. They were among the founders of the Georgian national school of Artificial Intelligence.

**1978a**

. He was a moderator and one of the General Chairs at the Moscow seminar "Computer Aided Design" at the Moscow House for Scientific and Technical Knowledge Propagation (MHSTKP).

Kuzin inspired this large scale seminar where many of the known specialists and the beginners in a field took part. A dominant trend was to discuss the results of finished and not-finished research projects and difficulties with the implementations and applications. Among them was a series of projects which were implemented by Kuzin's collaborators, this was an actual outcome of Kuzin's school.

In his tutorial he discussed a conceptual framework for the CAD system with the intelligent behavior, interface and other properties. This time he actively worked in this area and was a principal researcher in this branch. He sought for the opportunities to integrate the implementations of various logic means with the methods. This time he started to formulate the software design principles in terms of objects and scripts, he initiated and started the implementation of a tool kit based on event-driven doctrine.

He understood an importance to develop the general technology and discipline of programming, usage of the logical filters which would give more freedom to a usual or causal user. Most of his ideas were accepted by the USSR's specialists, they were used very naturally and in the most of the research and industrial communities.

This conceptual transparency of his tutorial, extremely natural in its origin, caused the wide dissemination, within the whole country, of the approaches and methods discovered by the collaborative groups of the researches, and first of all by Kuzin's school.

**1978b**

. He was a General Chair at the Moscow seminar Information ware and software for systems at the Moscow House for Scientific and Technical Knowledge Propagation (MHSTKP)

He was not only the principal scientific leader but also a generator of the scientific ideas, moderator of the discussions and demonstrations of the implemented toolkits and applications. His tutorial was listened by some hundreds of humans, they occupied not only the seats in the Big Red Conference Hall, but all the adjacent rooms and halls where the radio translation was turned on.

During the tutorial duration there were no noise at all, this was an actual, highly accented interest to the topic. This was the best school of a live scientific investigation, later many of the beginners and Ph.D. students were influenced to work in this area. Kuzin analyzed the advances and perspectives in an area of the information systems which could be treated as the Artificial Intelligence Systems (AIS). He established the criteria and classification which are applicable to characterize AIS among the manifold of the Information Systems (IS), he analyzed the nature of difficulties attempting the practical Knowledge Representation. He did his tutorial in the engineering style, or using his own terminology, engineering mathematician, entirely in a constructive and even practical way, avoiding the troublesome details of the ontology.

According his tutorial, the future of a matter was extremely promising and scientifically interesting, and possible results of a work were significant. And all of this were in detailed and rigorous correspondence with the known facts and trends. He gave the proper engineering approximation of a rather complicated idea for the properties of a Knowledge Representation Language. He demonstrated his approach to develop the software and to evolve the Software Engineering, he pointed out what are the promising challenges in the world known research projects and what are the palliatives and followers of the ad hoc trends in development and fundamental research.

Kuzin highly estimated an elegance in the mathematical results, but did not like the unlimited usage of this elegance principle. Often he accepted and estimated the results of implementation, even without a rigorous mathematical background, which was based on the sound intuitive ground. But this was just a first stage, after which he, as a rule, inspired the mathematical verification up to establishing the computational model.

**1979a**

Kuzin published, under his edition, the book of selected papers in AI-field reflecting the state-of-art in USSR. The title: Topics for Cybernetics. Intelligent Data Banks, vol.55, the USSR Academy of Sciences, Moscow, 1979. This was an extremely popular book which was mentioned among the specialists just as "TC-55" during half a decade. Kuzin selected out, reviewed and edited the papers, with an editing board, during almost half an year making his duties with a care.

This book gave insight and guidelines for both a theory and practice of Intelligent Data Banks and other advanced Information Systems which borrow the AI ideas. Kuzin formulated in depth the concept and notion of IDB which later were used and enriched by the hundreds of researches during the years. He accented that IDB is a kind of Cybernetics System which includes the range of bases: target base, knowledge base, data base, and, possibly some additional bases.

Kuzin concluded that the Cybernetics Systems of this kind need their own, newly established, mathematical foundations which, possibly, have no immediate predecessors and violate the traditional, e.g., continuous mathematics. He was interested in establishing a measure or unit of knowledge which give a sound basis for practical representation. He had a durable interest to the frames' research area which gave a ground to the frameworks based on scripts and situations selected out in the problem domains. One of his ideas was to iterate some simple structure to obtain the actual framework. He proposed to use the models for non-classical logics, but, possibly, had not yet enough intuition how to overcome the contradictions in the large bulk of information. In AI-field the research activity in this direction just started, the difficulties were hardly understood by the specialists, but no accepted constructive theoretical framework was established. Due to John McCarthy et. al the systems of non-monotonic reasoning with varying assumptions were established, so that some possibility to eliminate the contradictions did exist. But Kuzin preferred to find out some special algorithmic procedures or algebraic ground to re-solve and avoid the contradictions towards consistency of the represented information, instead of using and exploring the deductive problem solvers.

His principle paper on IDB inspired a great interest in Russia, its importance is acknowledged till the current years. It was conceptual in depth, enforcing the researchers to think over the range, scale and scope of the possible theory for Intelligent Systems, as well as over the limitations of the traditional Cybernetics and mathematical models, where they cancel out their working scope and the modern trends in cybernetics and Information Systems would affect.

Retrospectively speaking, many the researchers of 70s in AI-field took their own roads and tracks of investigations, which previously were hardly known, their theories met the scepticism from the orthodox mathematicians. The discussions, initiated at the conferences, had a lot of continuations in the voluminous publications of the academic periodicals.

Some of the TC-55 papers were invited, and Kuzin personally gave a lot of comments to the authors, improving the contents. He stressed, that each of these papers must contain the elements of foundations and guides to the future investigations. After the years passed, the Kuzin's strategical plan was succeeded, attracting both a lot of single volunteers and scientific groups. Almost all the young authors of the papers became the known specialists, in some cases founding their own scientific schools and directions of the research and development. Their origin have an easy recovering up to IDB book.

**1979b**

. This event was on October, 1979 in Ruza. Kuzin was invited by academician Germogen Pospelov from the USSR Academy of Sciences to organize the section. According the agreement, he got the duties of the principal coordinator for the main Artificial Intelligence stream, Knowledge Representation and Intelligent Data Banks. One of the main targets was to start up in the USSR the practical development and applications of the intelligent information systems, hence for Kuzin the organizing duties were dominant. And he succeeded in his efforts.

An atmosphere of this Symposium was stimulating, the participants could enjoy the golden autumn walking across the nearest forests. All the sessions started at 10:00 AM but Kuzin was in the habit to awake at 7:00 AM, so that all his assistants did the same. The working day of Kuzin's team started with a light running along the bank of the forest river, then sitting at the bench, they discussed the main topics of the current day session, brief outline of the reports and possibilities to organize the evening panels. It was a cool morning, the thoughts and proposals were generated distinctly, becoming in the years the valid directions of a scientific research.

Some of the evening panels were moderated while most of them were the natural continuations of the daytime sessions. The participants were attracted by Kuzin, his vision of the AI trends when he estimated and analyzed the topics. In the late evenings, when the participants were tired, he moderated the Russian romance concerts singing solo "Shine, shine, my star", "I met you", and a lot of others. His strong and deep voice attracted the additional participants and soon the hotel hall became overfull. The singing was accompanied by the piano music, the other singers changed Kuzin or just the people listened to the classical music.

All of this was happened naturally, without preliminary planning, all the participants enjoyed the intelligent freedom. But in a time the concert again was changed by the multiple local discussions on how to build the Knowledge Systems. Kuzin at ease directed all the discussions simultaneously, sometimes changing the topic or taking his own initiative. These schools of contacts was stimulating for the researchers, especially for Ph.D. students.

With a short time the people created the research teams, exchanging the papers, reports and manuscripts between the teams. They were proud of belonging to Kuzin's school.

Kuzin's tutorial at the session gathered a lot of participants, the conference hall was full. He classified the numerous works and publications, using the qualitative evaluations, the characteristics of efficiency and throughput, generalized and concluded, and estimated the possible benefits and trends. He was confident that the Intelligent CAD systems should contain, besides the database and knowledge base, as a focal subsystem the target base. The most important was to evolve its structure and features, and to solve the most complicated task to establish, implement and release the mathematical model. His tutorial stimulated the research in the developing the target bases which was done by many research teams.

At the panel discussions some speakers pointed out an interest for the knowledge representation units, the frames, comparing the authors and their approaches. There were a lot of criticism and sceptic attitude. Kuzin did not share this position, it seemed that he has an intuitive vision of the in-the-years perspective and long term trends. He usually was right when selecting out what to investigate, what has a perspective and what lost the rate and is a blind alley. In his research predictions he practically never was failed, this was known for other researchers, and as a consequence his Ph.D. students had a good orientation and a starting position to succeed in their own research. At this stage he even enforced the Ph.D. students to develop and implement the knowledge systems based on the distinct notion of frame. He highly estimated the results of a good experiment work which was compared with the same of the other authors. His strategy was to accumulate a range of working prototypes of the intelligent information systems with the aim to develop a full scale release for the industrial applications.

The same year (1979) he published his famous 2nd volume of "Foundations of Cybernetics" with a special subtitle "Models for Cybernetics". This book was written with a great inspiration and confidence in a good perspective for the Artificial Intelligence models during at least 10 years period.

In his book the most important and prominent models for knowledge based systems were selected out and classified. The readership level was as usually the engineering mathematician students, graduate and undergraduate. He applied special efforts to avoid the complicated formal derivations and used instead the trasparent intuitive reasons, the feasibility and qualitative measures, hints and explanations. He described and estimated the scopes of the models and the possible modifications to capture more meaning under the renewed conditions.

His approach attracted a lot of successors and inspired a series of scientific discussions.

**1980**

. This Workshop was on March 10–13, 1980 in Tsakhadzor, Armenia. He was General Co-Chair sharing his duties with Dr. Ashot Hovanesyan, a director of Medical Information and Computing Center of Armenia. The efforts of joint organizing team to prepare this event, disseminate call for papers, book the conference hall took several months. Kuzin's position helped the organizers to overcome all the difficulties and involve a lot of scientists from various research institutions of the country.

This event was hosted by the team from Medical Information and Computing Center of Armenia. This was due to the attractiveness of the topics and the live interest of the specialists in Medical Cybernetics to develop and apply the intelligent information systems for the health care. The computer-aided working place for the physician was one of the nearest targets. The specialists in medical cybernetics were granted by the government to make a progress in this area. Most of them were highly interested to achieve the desired results, there was a stimulating and collaborative atmosphere.

The participants were located in the hotel at a nice mountain resort, they were vivid and highly collaborative. Among them were the known specialists with invited reports, researchers with different skill, Ph.D. students, specialists from the industry.

Kuzin always was in a center of events, nobody had a free time. The working day started early in the morning with the mountain skies and sliding down the nearest hills. The participants used this opportunity to establish the personal contacts which later generated the good research teams.

The regular sessions were started after lunch time and had a duration till the late evenings while the participants just were tired to continue the panels. A few late evenings were used for the on-fly concerts which were in a Grand Red Hall. The participants could enjoy the classical piano music and Armenian songs of the highly musically talented hosts. Kuzin changed them with the Russian romances.

The topic of his tutorial and moderated panels dealt with the augmentation of the concepts for an information system based on the Intelligent Data Bank. The ways of implementing the adequate data models were at a central position. He gave the feature analysis of the IDB projects and prototypes. Some of the implementations were based on the dialects of Frame Representation Language (FRL), in a few projects the ideas of Nick Roussoupolos to separate frames and relational database were used, others were the followers of deductive models. The team from Medical Information and Computing Center of Armenia implemented and applied to the practical tasks the IDB based on IMS, this was a balanced and successful implementation, in their reports the results of two years experience were discussed.

This was a highly stimulated event with the experience exchange, the relations between many of the institutions were established. This allowed later to share the job of collaborators for the project "Data Bank".

**1981**

. This Symposium was held on April, 1981 in Suzdal, its aims and scopes were analogous to the 1st Symposium in 1979. Kuzin was a Chair of the Section on Knowledge Representation in Intelligent Systems. He decided to attract the contributions in an area of CAD, and especially in Advanced Programming Technology (APT). His aims were to discuss and analyze in depth the Intelligent Programming Systems and Intelligent CAD. More briefly, he concentrated the efforts on the target description generation for the system under development. His efforts at the 1st Symposium helped him to attract to this event the high quality submissions. Many of the researchers, under his influence, started to develop their frame-based software, dropping the initial system description down to the elementary units of knowledge representation. His school now had the branches with the research teams from Armenia, Baltic States, Georgia, the Ukraine. The new results were reported by the team from Armenia, they launched in the USSR the applied Intelligent Information Systems for Medical Cybernetics. Later the joint project was launched by the teams from Medical Information and Computing Center (MICC) of Armenia and from MICC of the Russian Federation.

**1982**

. This workshop was in Georgia. He was a General Co-Chair sharing his duties with Profs. G.Chogowadze and G.Gogichaishwili from the Georgian Politechnical Institute in Tbilisi, Georgia. This was a joint forum where the Georgian school in Artificial Intelligence was widely presented. The topics were as follows: knowledge bases and their development, augmenting the relational databases with the intelligent access, interfaces, intelligent CAD, and various applications. Most of the Georgian research teams shared Kuzin's approach and worked in a close collaboration with Kuzin's school.

In his tutorial Kuzin paid attention to the computational environment of IDB and its properties. He proposed to organize on-fly the series of panel discussions and a round table to better coordinate the mutual research activity. He felt a good reason and opportunity to launch the concert projects which were aimed towards development the full scale prototypes of IDB. Some of them must use the various data models and knowledge representation schemata, while others used the specific problem orientation. The agreements between the institutions were achieved and in the nearest future the projects were launched. This was a great advance in the sphere of Artificial Intelligence in the USSR which strongly influenced the research activity in the country.

**1983**

. This workshop was on October 25-27, 1983 in Uzhgorod. Kuzin was a Co-Chair as the principal investigator for the project "Data Bank". This event was addressed to further stimulating of the younger generation researchers. The reports were concentrated on the discussions and feature analysis and trends for developing the advanced data models.

**1984a**

. He was a General Co-Chair with Georgian partners the same way as at 3rd Workshop. This Workshop was in Terscol, Naltchik and was planned in 1982. Its aim was to estimate the results of joint projects in an area of IDB.

In his tutorial he analyzed the most advanced results of the research teams pointing out the success in applying the relational databases. The Georgian teams accumulated a good experience developing the intelligent applications for the industry. One of the projects from the Russian team resulted in a toolkit as the extensible computation environment with the embedded knowledge base system and relational database system. The basic primitives were objects within the host programming system LISP, implemented by the system programmer Anatoly Panteleev. The features of implementation and applications were discussed at the evening panels.

Among the interesting theoretical issues on data models were the categorical model and tensor model. The research teams which worked in this directions believed in the promising results and demonstrated the embedding of the relational algebra into categorical or tensor framework. They were at the beginning stage with the experiments, only partial implementations were released.

As was happened, one of the main topics for this Workshop were the embedded systems.

**1984b**

. This event was on February, 1984 in Moscow, hosted by the International Center for Scientific and Technical Information. One of the main topics was the AI methods for the New Information Technologies. He was an invited speaker and in his tutorial gave in depth analysis of the most prominent results in AI field and their fruitfulness for the Advanced Information Technologies.

**1986**

. This workshop on November 19-21, 1986 was in Dilijan, Armenia, with Kuzin as a General Co-Chair. Dr. Ashot Hovanesyan was both a General Co-Chair and Program Committee Chair and attracted a set of submissions reflecting the state-of-art in an area. Kuzin's school was presented by the numerous reports. All the relative research teams also joined this event. In fact, this was the 5th Joint Workshop on Intelligent Data Banks which continued the series of previous four workshops.

Kuzin's tutorial "Evolution, advances and trends in an area of intelligent access for data banks" later was disseminated in manuscript within the research community. One of his ideas was to use objects as data and knowledge units which are equipped with the attached procedures. Under this assumption the information system would be extensible. This time he worked at the extensible data models and gave their feature analysis in depth.

# Acknowledgements

I would like to thank all the colleagues and collaborators for valuable help and advice during the preparation of this paper.

[^1]: See, for example, his books: L.T.Kuzin, *Foundations of Cybernetics*, Vol. 2, Moscow, Energija, 1979, and L.T.Kuzin, *Foundations of Cybernetics*, Vol. 1, Moscow, Energija, 1994, both printed in Russian.


abstract: Ranking is a natural and ubiquitous way to facilitate decision-making in various applications. However, different rankings are often used for the same set of entities, with each ranking method placing emphasis on different factors. These factors can also be multi-dimensional in nature, compounding the problem. This complexity can make it challenging for an entity which is being ranked to understand what they can do to improve their rankings, and to analyze the effect of changes in various factors to their overall rank. In this paper, we present RankBooster, a novel visual analytics system to help users conveniently investigate ranking predictions. We take university rankings as an example and focus on helping universities to better explore their rankings, where they can compare themselves to their rivals in key areas as well as overall. Novel visualizations are proposed to enable efficient analysis of rankings, including a *Scenario Analysis View* to show a high-level summary of different ranking scenarios, a *Relationship View* to visualize the influence of each attribute on different indicators and a *Rival View* to compare the ranking of a university and those of its rivals. A case study demonstrates the usefulness and effectiveness of RankBooster in facilitating the visual analysis of ranking predictions and helping users better understand their current situation.
author:
Abishek Puri$^{1}$`<!-- -->`{=html}, Bon Kyung Ku$^{1}$`<!-- -->`{=html}, Yong Wang$^{1}$`<!-- -->`{=html}, Huamin Qu$^{1}$`<!-- -->`{=html}

$^1$ Hong Kong University of Science and Technology, Hong Kong
bibliography: egbibsample.bib
title: RankBooster: Visual Analysis of Ranking Predictions

# Introduction

Ranking is a natural and ubiquitous way to help people make decisions in various applications, for example, finding a good restaurant and hotel for a trip, choosing the best university, and buying a suitable car. Many studies have been conducted to help people easily interpret rankings and make better decisions through visualization. Few studies, however, have focused on helping the entities (or items) being ranked, which we call "rankees".

Due to the wide usage of ranking companies, the impact of a low ranking can be substantial. For example, universities with a lower ranking receive less applications and private . In this paper, we aim to tackle the following question *how can we use visualization techniques to help universities efficiently interpret their rankings and analyze the effect of their actions on their ranking?*

There are three reasons why this is a challenging question: multiple ranking systems and large number of entities, prediction uncertainties, and users' unfamiliarity with ranking prediction and visualization.

First, the sheer number of systems and methodologies for creating rankings creates complexity. Second, predictions are innately uncertain, and so the system must show the user the uncertainty. Third, many of the university administrators are not familiar with predictive analytics and visualization, requiring the system to be simple yet comprehensive.

Our research problem can be regarded as a Multi-Criteria-Decision-Making (MCDM) problem with the added complexity of uncertainty visualization. There have been many prior studies in the field of MCDM over the past decade , but they have generally focused on either high-level analysis of the tasks required to be solved or aiding decision-making without any uncertainty involved in the variables . To the best of our knowledge, in the domain of ranking visualization, there has been no work attempting to aid rankees in their decision-making issue.

In this paper, we propose a visual analytics system, RankBooster, built in collaboration with university ranking experts. During this collaboration, we were able to create a set of tasks that could be generalized to other sets of entities who need to understand how to improve their ranking position in their ranking systems.

Our visual analytics system was used by our university to analyze their ranking position prior to their data submission. Our case study with the domain experts also shows the effectiveness of our approach.

Our major contributions can be summarized as follows:

-   A visual analytics system, RankBooster.

-   A novel glyph visualization to show the relationship between the Indicators and the Attributes in ranking systems.

-   A case study with the domain experts highlighting the usability of the system.

# Related Work

As our work comes from the ranking visualization domain, we will list out key work in ranking visualization.

Part of the literature consists of works that aim to allow users to interact with the ranking system by modifying the weights or inputting their preferences. Works in this area include LineUp , Podium , and Rankit . However, they do not attempt to visualize any predictions of established rankings, focusing on existing rankings and making them more interactive and accessible. Other work has focused on looking at rankings over time, allowing users to observe trends and patterns. An example work in this area is RankExplorer . While such works allow rankees to evaluate historical trends visually, they do not provide a way to evaluate scenarios in the future, or evaluate the effect of decisions made now. With regards to university rankings, which is the domain we will focus on in the paper, we could find only one other work that uses university rankings, but again their focus is on helping students make the decision, not the universities. Therefore, we claim that our perspective on this domain is a novel one.

# Background

## Terminology

In this subsection, we explain key terminology used in the paper, through an example of a ranking system submission and result.

First, raw data is collected on the rankees. These are the *Attributes*. Then, the data is split into different groups and normalized across all rankees. These normalized groups are the *Indicators*, and the normalized value for a given rankee in an Indicator is their *Indicator Score*. Finally, all of a rankees Indicator Scores are inputted into a function which then gives them their *Final Score*. These final scores are the basis of a rankees final rank.

The question that is of interest to the rankee is *What will be the Final Score given the values of the Attributes I am submitting?* To frame the question, we create a *Scenario*, which consists of a set of Attributes and their values, the predicted Indicator Scores, and the predicted Final Score. Our system allows the user to evaluate scenarios individually or collectively.

## Predictive Model Requirements

Our system assumes the user already has a model that can generate predictions of the Indicator Scores, along with some measure of uncertainty. It also assumes that the user already has a model that can generate predictions for other universities, along with some measure of uncertainty.

During our collaboration, an ensemble model which produced 100 predictions for each Indicator Score was used, with the range of the predictions providing us with the measure of uncertainty.

## Example Of A Scenario

As an example, we imagine a ranking system with 3 Attributes, coalesced into 2 Indicators, eventually creating a final score. In a given scenario, we have known values for the 3 Attributes. Then, we have predictions for the 2 Indicator Scores, as well as a prediction of the Final Score. These predictions are normalized to be between 1 and a 100 inclusive. For each prediction, we also have a measure of uncertainty. In our case, the uncertainty is the range of the 100 values produced by the model for each prediction.

## Dataset Access

Our dataset includes 5 years of data on over 400 universities. While the Indicator and Final Score are publicly available through the respective ranking systems websites, the Attribute values are not publicly available, requiring us to get the dataset from our collaborators.

# Requirements Analysis

Based on a survey of ranking experts, we were able to identify 3 main requirements of the system.

-   **R1. Explore Scenarios Individually** The system should be able to track a scenario from the submitted data to the final score. It should also show the relevant uncertainties, and identify Attributes and Indicators that will have the most impact on their rankings.

-   **R2. Explore Scenarios As A Group** The system should be able to compare Attributes, Indicators, Final Scores, and uncertainties between scenarios. Users should also be able to filter and constrain the set of scenarios, and visualize a summary.

-   **R3. Compare with Rival Universities** The system should be able to directly compare a selected university with its rivals, focusing on predicted differences given a particular scenario.

# Visual Design

This section describes the usage of each view in our system, the design rationale for each view, and the interactions in each view.

## Scenario List View

This view, shown in Fig.(E), is a table, with each row of the table showing a summary of one scenario. This view partially satisfies **R2**, specifically providing a filterable and sort-able summary of all scenarios. It gives the user a simple interface by which to select particular scenarios for further analysis in the other views of the system.

We show the change in the Attributes and the predicted change in Indicator Scores and Final Scores as bars. Here, the change is relative to the previous years values and scores respectively. The color represents whether the change is positive (green) or negative (red).

## Scenario Analysis View

This view, shown in Fig.(A), consists of three columns of glyphs. The leftmost column has a glyph per Attribute, the middle column has a glyph per Indicator, and the rightmost column has a glyph each for the Final Score and the Final Rank. This view partially satisfies **R2**, showing a group summary of all the scenarios along with the uncertainty related to the predictions being made. It also partially satisfies **R3**, as multiple entities can be compared in this view.

Each glyph shows all the values for that object across all the selected scenarios, relative to the previous value of that object. For example, a glyph in the Attribute column would show all the values for a particular Attribute across all the scenarios, relative to the value of the Attribute in the previous year. This is encoded with a line chart, with the x-axis containing the relative values and the y-axis containing the frequency of appearance in the scenarios.

We encode uncertainty by enhancing the ensemble bar from Chen et al. . The enhancement consists of allowing multiple entities to be compared by superimposing each entities uncertainty on each other, whilst giving each entity a different color. The length of the bar along the glyph corresponds to the range of the uncertainty, with the lowest and highest relative values in the range of the prediction shown at the left and right of the bar respectively. In this encoding, negative values depict being below the previous years values.

## Relationship View

The view, shown in Fig.(B), is a table of glyphs. Each row of the table represents a selected scenario, with each glyph in the row encoding information about the Attributes in the scenario and one Indicator in the scenario. Each column of the table represents one particular Indicator. This view partially satisfies **R1** and **R2**, specifically allowing the user to analyze the effects of the Attribute values of a scenario on the predicted Indicator Score and Final Score, whilst also allowing multiple scenarios to be compared with each other along the Indicator axis.

Any given glyph in this view is linked to a particular scenario and a particular Indicator. In Fig.(B), the top-left glyph is linked to Scenario 0 and Indicator 1. The square shape of the glyph is due to the number of attributes; if there are more attributes, the shape would be a higher-sided polygon. The bar on the side of the glyph shows the value of a particular attribute in that scenario relative to the previous years value. The polygon is internally split equally amongst the attributes, with the shade signifying the relationship between that attribute and the indicator. The darker green the color of the section, the more the Indicator Score increases when the Attribute increases. The opposite is true if the color is a dark red. The curved bar on top of the glyph shows the predicted Indicator Score for that scenario with a line, and the maximum and minimum value of the Indicator Score is the total length of the bar.

We determine the effect of an Attribute on the Indicator Score by perturbing the Attribute value and observing how that affects the models' Indicator Score prediction. The effect is normalized across all scenarios selected to be in the view.

## Rival View

This view has a heat map, shown in Fig.(C), and a radar chart, shown in Fig.(D). This view satisfies **R3**, allowing the user to compare predictions for different universities. The heat map encodes the probability that the users university will have a higher Indicator or Final Score than a rival university, given a particular scenario. Our users had 3 different methods for predicting the Indicator or Final Score for the rival universities; each method is given a row in the heat map.

The radar chart encodes information about one method for predicting rival university Indicator and Final Scores, with the method selected via a drop-down list. Once a method is selected, the chart shows the predicted values, with the violin plot showing the uncertainty of that prediction as a distribution across all possible scores for that Indicator. This allows the user to compare the rival universities expected value with the range of predictions made in the selected scenario. We can select a particular rival to have their values highlighted in the radar chart, for ease of use.

Having two views allows the user to go from a higher-level view of all methods across all rivals in the heat map, to a more focused view of one method in the radar chart. This abides by the "detail on demand" principle, as the user only goes to the radar char once they have noticed a method they want to focus on in the heat map.

## Implementation Details

Our visual system was built using Vue.js, with a Python backend server. All the models were built using Python with scikit-learn.

# Evaluation

To evaluate our system, we conducted a case study and an interview with the experts to evaluate the overall effectiveness of the system.

## Finding Optimal Data Submission For a Ranking System

University A ("Uni") was in the process of submitting data to QS. They have a range of values that could be submitted for each attribute, and wanted to use the system to decide what to submit. The user was a female expert from Uni.

She first selected Uni and inputted her submission range in the panel at the far left of Fig., which generated 1511 scenarios **(R2)**.

She noticed that the expected changes in Student Faculty Ratio Indicator ("SFRI") and International Student Ratio Indicator ("ISRI") had a lot of variation, while the other indicators didn't **(R2)**. She therefore decided to keep only those scenarios that have a predictive value larger than 0 for those two indicators, using the Scenario Analysis View. She then noticed that there are two peaks of predicted values for ISRI, so she focused in on the second peak **(R2)**. The filtering had left 108 scenarios Fig.(A). She expressed surprise that there was such a large spread of values for SFRI and ISRI, stating this was an interesting insight.

She now focused on the differences between the scenarios that remained after filtering. She decided to focus on the influence of the first attribute, as Uni had the most leeway on this attribute, so she sorted by this attribute and selected the first and last scenario to analyse more closely **(R2)** Fig.(B). She immediately noticed that the Local Influence was different in I4, I5, and I6.The change in the attribute had also caused the influence of other attributes to change, with some increasing and others decreasing Fig.(B). In particular, when the chosen attribute was the main influencer of I6 only when it had a negative change **(R1)**. She noted it down for further analysis with her team later.

She now wanted to focus on the values for submission, so she selected the scenario matching one of the discrete values, to view it in the Scenario Analysis View in context of all scenarios **(R2)**Fig.(C). She immediately noticed that for SFRI, her selected scenario seemed to be predicting the average value, but the uncertainty was heavily biased in the negative direction **(R1)**. This bias in the negative direction implies that there is a significant possibility of the real outcome being below the predicted value, and so it could be risky to completely rely on the predicted value. She further investigated the scenario using the radar chart Fig.(D). Here she noticed that for I1, there is a bottom heavy distribution, so in her view that was enough evidence to discount this scenario with regards to I1. Similarly, the other values are viewed and, based on analysis in the same vein as above, she decided on a particular value set for data submission.

The case study shows how insights can be gained with regards to the optimality of the data points in the set of submittable values, as well as with regards to the relationship between the attributes and the indicators.

# Discussion and Conclusion

In this paper, we propose RankBooster, an interactive visual analytics system to help users conveniently investigate ranking predictions to understand the reasons of the ranking. We take university rankings as an example, working closely with the domain experts to generate requirements and build out the system. Our case study demonstrates how RankBooster can be be used to analyze scenarios and decide on the optimal data submission.

However, it is not without limitations. First, our analysis assumes that the attributes are independent of each other. Whilst this assumption is used in our domain, it may not always hold in other domains. Second, our evaluation is limited to universities specifically, further evaluation would need to be done with other domains to fully verify its usability in a more general setting. Tackling these issues will form the bulk of our future work in this area.


abstract: Application of models to data is fraught. Data-generating collaborators often only have a very basic understanding of the complications of collating, processing and curating data. Challenges include: poor data collection practices, missing values, inconvenient storage mechanisms, intellectual property, security and privacy. All these aspects obstruct the sharing and interconnection of data, and the eventual interpretation of data through machine learning or other approaches. In project reporting, a major challenge is in encapsulating these problems and enabling goals to be built around the processing of data. Project overruns can occur due to failure to account for the amount of time required to curate and collate. But to understand these failures we need to have a common language for assessing the readiness of a particular data set. This position paper proposes the use of data readiness levels: it gives a rough outline of three stages of data preparedness and speculates on how formalisation of these levels into a common language for data readiness could facilitate project management.
author: Neil D. Lawrence
Amazon Research Cambridge and University of Sheffield
date: 6th April 2017
title: Data Readiness Levels

# Introduction

The universal formula of machine learning is $$\text{data} +
\text{model} \rightarrow \text{prediction}$$ with a high quality of prediction being dependent on both good models and high quality data. Much of the focus on machine learning in the academic world of machine learning is on the quality of the model. This focus arises because data is normally from benchmarks or publicly available data sets, so performance in a task is improved by exercising control over model.

The greater interconnectedness of modern society and apparent ease with which we digitise or record data places us in a particular position. We need to develop a lot more control over the quality of our data. Whether it be in the manner in which we choose to collect, or how we choose to annotate. Currently the academic or empirical study of modeling (e.g. support vector machines, neural networks, Gaussian processes) is prominent in the education of the graduates we produce, but approaches for understanding the quality of data are less widely used.

# Data Readiness Levels

In this position paper we introduce the idea of *data readiness levels*. Data readiness levels are designed to deal with a challenge for human cognitive information processing. It's difficult for us to reason about concepts when we haven't developed a language to describe them. The idea of data readiness levels is to correct this issue and make it easier for us to reason about the state of our data.

The challenges of data quality arise before modeling even starts. Both questions and data are badly characterized. This is particularly true in the era of Big Data, where one gains the impression that the depth of data-discussion in many decision making forums is of the form "We have a Big Data problem, do you have a Big Data solution?", "Yes, I have a Big Data solution." Of course in practice it also turns out to be a solution that requires Big Money to pay for because no one bothered to scope the nature of the problem, the data, or the solution.

Data scientists and statisticians are often treated like magicians who are expected to wave a model across a disparate and carelessly collated set of data and with a cry of 'sortitouticus' a magical conclusion is drawn. It is apt to think of it as ocean of data, to paraphrase "water, water everywhere and not a drop to drink", we have "data, data everywhere and not a set to process". Just as extracting drinkable water from the real ocean requires the expensive process of desalination, extracting usable data from the data-ocean requires a significant amount of processing.

For any data analyst, when embarking on a project, a particular challenge is assessing the quality of the available data, how much processing is required? This difficulty can be compounded when project partners do not themselves have a deep understanding of the process of data analysis. If partners are not data-savvy they may not understand just how much good practice needs to be placed in the curation of data to ensure that conclusions are robust and representative. Just as water quality should be measured before consumption, so should data quality.

When scoping a project, in most proposal documents, very scant attention is paid to these obstacles, meaning in practice the process of improving data quality is under appreciated and under resourced.

One difficulty is that the concept of "data", for many people, is somehow abstract and diffuse. This seems to mean that it is challenging for us to reason about. Psychologists refer to the idea of *vivid* information as information that is weighted more heavily in reasoning than non-vivid or *pallid information*. In this sense data seems to be rendered *vivid* to be properly accounted for in planning. This may relate to availability heuristics\[1\].

This abstract nature is also true of other terms, for example for many people the idea of "technology" is also similarly diffuse, it is pallid information (although I've never heard anyone remark "we have a big technology problem, we need a big technology solution"). Perhaps to deal with this challenge, in large scale projects, when deploying technology, we are nowadays guided to consider its *readiness stage*. Technology readiness levels arose in NASA \[2\]. The readiness of the technology is made manifest through a set of numbers which describe its characteristics:[^1] is it lab tested only? Is it ready for commercialization? Is it merely conceptual?

No doubt there are pros and cons of such readiness levels, but one of the pros is that the manifestation of the technological readiness pipeline ensures that some thought is given to that process. The technology is rendered more vivid even through a shared characterization.

It would therefore seem very useful to have a scale to make *data readiness* manifest. This idea would allow analysts to encourage better consideration of the data collection/production and consolidation, with a set of simple questions, "And what will the data readiness level be at that point?". Or "How will that have progressed the data readiness?". Or to make statements, "we'll be unable to deliver on that integration unless the data readiness level is at least B3.."[^2]

This paper aims to trigger a discussion in statistics and data science communities by proposing an initial set of descriptors for data readiness.

The initial proposal is that data readiness should be split into three *bands*. Each band being represented by a letter, A, B and C. These bands reflect stages of data readiness which would each likely have some sub-levels, so the best data would be A1 and the worst data might be C4. The aim here is to avoid being fine-grained too early. We therefore begin the discussion by focussing on three bands of data readiness.

## Band C

Band C is about the accessibility of a data set. The lowest sub-level of Band C (let's label it as C4) would represent a belief that the data may exist, but its existence isn't even verified. Signs that data is C4 might include statements like "The sales department should have a record of that." Or "The data should be available because we stipulated it in the software requirements." We might think of it as *hearsay* data. Data that you've heard about so you say it's there. Problems with hearsay data might include

-   whether it really is being recorded

-   the format in which it's being recorded (e.g. handwritten log book, stored in PDF format or old machine formats)

-   privacy or legal constraints on the accessibility of the recorded data, have ethical constraints been alleviated?

-   limitations on access due to topology (e.g. the data's distributed across a number of devices)

So when we are first told a data set exists, when we have hearsay data, then it is at band C4. For data to arrive at C1, then it would have all these considerations dealt with.

When data arrives at C1 it's ready to be loaded into analysis software, or it can be made available for others to access (e.g. via a data repository such as OpenML \[3\]). It is machine readable and ethical procedures for data handling have been addressed. Bringing data to C1 is often a significant effort itself involving many lines of bespoke software and human understanding of systems, ethics and the law.

Some parts of Band C are sometimes referred to as "data munging" or "data wrangling", but those aren't the only components of this band, there are additional challenges such as ethical and legal that need to be resolved.

## Band B

Band B is about the faithfulness and representation of the data. Now that it's loaded into the software, is what is recorded matching what is purported to be recorded? How are missing values handled, what is their encoding? What is the noise characterization (for sensors) or for manual data are there data entry errors? Are any scientific units correctly formulated?

Tukey's approach of "Exploratory Data Analysis" also fits within Band B. Visualizations of the data should be carried out to help render the data vivid and to ensure decision makers, who may not be data aware, can become involved in the analysis process. Decision makers (e.g. project managers, or the client) should also begin to get a sense of the limitations of their data set through appropriate visualisation.

As part of Band B the characteristics of the collection process should also be verified, was data collection randomized, is it biased in any particular way?

Other things to watch for at this stage include:

1.  If the data has been agglomerated at some point (for example, for privacy) how were missing values dealt with before agglomeration? If they weren't dealt with then that entire section of the data may be invalidated

2.  If the data has been through a spreadsheet software, can you confirm that no common spreadsheet analysis errors were made? For example, was a column or columns accidentally perturbed (e.g. through a sort operation that missed one or more columns)? Or was a [gene name accidentally converted to a date](http://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-5-80)?

By the end of Band B, when data is B1, a broad idea of limitations in the data should be present in the expert's mind. Data at C4 was hearsay data, someone heard the data existed and they said what they thought it might be good for. At B1 the analyst knows how faithful the data is to that description. This is the significant challenge for a data scientist. What people believe they have in their data versus what's actually there. Only at the end of Stage B would the analyst begin to have an intuition about what my really be possible with the data set. Getting to this point is often the most expensive part of the project, but we do not yet have good methods to guage progress, or share the status of a particular data set.

## Band A

Band A is about data in context. It is at Band A that we consider the appropriateness of a given data set to answer a particular question or to be subject to a particular analysis.

The context must be defined. For example OpenML \[3\] defines *tasks* associated with data sets. A data set can only be considered in Band A once a task is defined. A task could be "Use the data to predict a user preference" or "Use this data to prove the efficacy of a drug" or "Use this data verify the functioning of our rocket engine".

Once data has been considered alongside a task and any remedial steps have been taken, then the data is in A1 condition. It is ready to be deployed in the context given and it can be used to make predictions with the data.

Because A1 is about data in context, it is possible for a data set to be A1 for one question (e.g. predicting customer churn) but only B1 for a different question (e.g. predicting customer susceptability to a particular special offer). So the definition of the context or task is an important pre-requisite for this band.

To bring a data set up to A1, there may be a need for significant annotation of the data by human expert. There may be a realisation that new data needs to be actively collected to get the answers required. In that sense, Band A has some characteristics of a classical statistical analysis where the question would normally preceed the data collection. It is in Band A where you should be carefully thinking about the statistical design because it is only when you have the question you wish to answer that you can really unpick how your data may be biased or what information is missing.

# The Analysis Pipeline

A common mistake in data analysis is to not acknowledge the different processes above. They have different pre-requisites and require different skill sets to carry out. However, this path cannot be completely disconnected. Anyone performing an analysis at Band A also needs to be intimately familiar with the collection process so that any biases in data collection can be understood. Sharing information about decisions taken at Bands B and C will also be critical to achieving a good result.

What happens if we bring two data sets together to form a new data set? Some assesment of data readiness would still need to be performed on the new data set, even if the two other data sets had already been assessed. Why? Well, one example is that there may be ethical issues with combining data sets. For example, medical data can be easily deanonymized if it is combined with other related data sets. That would mean that some assesment would need to be made at Band C to see if it is ethically responsible to combine the two data sets in the same analysis package. Futher assesments would also need to be done at Band B to understand differences in collection protocols that might make the two data sets hard to combine.

## Potential Results

The idea of these levels is to increase the accountability of the process and allow the nature of the data to be manifest. With data readiness levels in place you can now imagine conversations that would include statements like the following:

> Be careful, that department claims to have made 10,000 data sets available, but we estimate that only 25% of those data sets are available at C1 readiness.

> The cost of bringing the data to C1 would be prohibitive for this study alone, but the company-wide data audit is targeting this data to be C1 by Q3 2017 which means we can go ahead and recruit the statisticians we need.

> The project failed because we over recruited statistical expertise and then deployed them on bringing the data set to C1 readiness, a job that would have been better done by building up our software engineering resource.

> What's the data readiness level? My team will be ineffectual until it's B1 and at the moment I see no provision in the plan for resource to bring it there.

> We estimate that it's a \$100,000 dollar cost to bring that data to B1, but we can amortize that cost across four further studies that also need this data.

> I gave them the data readiness levels to go through and they realized they hadn't yet got the necessary ethics approval for sharing the zip codes. We'll revisit when they've got through that and can assure us they can share a C1 set.

> While their knowledge of the latest methodologies wasn't as good as I'd hoped, the candidate had a lot of experience of bringing data from C1 to B1, and that's a skill set that we're in dire need of.

> The project came in under budget because they found a team with experience of getting a closely related data set to A1. Many of the associated challenges were the same and they could even reuse some of that team's statistical models.

# Case Studies

How useful would data readiness levels be? That's difficult to give a quantitative answer to, large scale analysis of the scale of this problem requires meta data science, i.e. the study of data science itself through acquiring data about data science. That would be a very worthwhile endeavour. In the absence of quantitative information, we provide two anecdotal case studies below. The first is on the cahllenge of extracting data from a popular machine learning conference proceedings.

## Proceedings of Machine Learning Research

The Proceedings of Machine Learning Research[^3] (PMLR) were begun in 2006 (as JMLR Workshop and Conference Proceedings) to provide a convenient way to publish machine learning conference proceedings without the overhead of a conventional publisher.

There are now 69 volumes of proceedings published and planned. They contain over 3,198 papers. The original website was manually curated to mimic the JMLR website, but since Volume 26 an automated proceedings production process which relies on editors providing a zip file of PDFs and a bibtex bibliography reference file (referred to below as 'bib files') specifying author names, abstracts and titles has been used.

In early 2017, as part of a rebranding process, the original website was moved to github and a new process for receiving proceedings and publishing was set up. For the rebranding the old web-sites needed to be scraped, converted to bib files.

A key aim in the rebanding was to make abstracts and titles easily available for analysis with an initial target language of Python.

This entire process can be seen as taking the data readiness of website data from C4 to C1. In other words, get to the point where the data could be loaded into an analysis software, our target here was to load it into the pandas framework within Python.

The *original* plan was to complete the work in some idle hours at the weekend. The actual work took much longer than projected. The major github commits of code along with a description of the effort involved are listed below.

1.  An initial two day effort to create bibliography files for the first 26 volumes which were published before the publication process was first automated. Github commit: <https://github.com/mlresearch/papersite/commit/daa51a0da82b5e532214a732a3d32d94ddf66381>

2.  A three day effort to convert the proceedings to a new format website. Much of this work was on the website presentation, but part was on data curation. Github commit: <https://github.com/mlresearch/papersite/commit/81d7a0556948281d11d8f0652ecdca61005c4318>

3.  A three evening effort to tidy the resulting data so it could load into Python pandas via web download. Github commit: <https://github.com/mlresearch/papersite/commit/81d7a0556948281d11d8f0652ecdca61005c4318>

The result is now available via github as a short jupyter notebook: <https://github.com/sods/ods/blob/master/notebooks/pods/datasets/pmlr_data.ipynb>. Data loading is now possible with a single library call.

where the pods library is available from <https://github.com/sods/ods/>.

This data has therefore successfully transferred from the start of C (e.g. C4, "Hearsay data") to the end of C (e.g. C1, data loaded into analysis software).

The data set only contains 3,198 data points, it was all available in electronic form. There were no issues around privacy or intellectual property but work still took approximately six working days. Much of that work was laborious, but it still involved well qualified understanding of computer software, e.g. regular expressions, scripting languages, libraries for downloading from the internet, github etc.. Alongside that work a new format web page was also provided (<http://proceedings.mlr.press>). Indeed that could be argued as a consequence of the data tidy up. Naturally the hope is that future volumes will be more cleanly added to this data set.

## Data Wrangling Snafus

Some snafus that occurred in the data wrangling of the Proceedings of Machine Learning Research.

Each bibliography file is provided by the editors of the relevant volume. Because they each used different approaches to generate the bib files, there were different issues with each bib file.

-   Many of these bib files will have been produced from conference management software (e.g. CMT or Easychair). That means the original information source is often likely to be author provided. Many authors seemed to paste their abstracts from the PDFs of their papers into this software. This meant that abstracts contained ligatures. A ligature is a single typeset unit such as 'fi', which comes about when the previous letter runs into the second, the German 'Sz', ß, is an example of a ligature that in the end became a letter. Unfortunately the ligatures did not paste as unicode, but as escaped characters which the python yaml library was unable to read when presented as data.

-   Papers are mostly written in Latex, so many of the abstracts contain Latex commands. But these commands in the abstracts or title are intepreted as escaped characters in yaml files. Additional 'escaping' was needed for these commands.

-   Author names containing accents were oftentimes corrupted, perhaps due to differences in representation in the original files.

Each of the snafus above can be resolved by robust coding, but there is normally a trade off between producing the data (just getting it done quick) and producing the most robust code (ensuring the result is high quality and reusable). The right operating point for the trade off is driven by the scale of the data set. It also requires experience to judge and is dependent on the coding skills of the data engineer.

This is the work at the pit face of data mining, it's difficult to estimate the time it will take, and yet it is normally a critical dependency in any machine learning project. For each of these snafus, better planning at early stage analysis could have saved time in manual correction of the data later. But doing such planning well requires experience and an understanding of best practice. We are not associating enough value with this experience, and therefore we continue to be tripped by snafus when preparing data.

### How Data Readiness Levels could have helped

This is one of the first projects I undertook after conceiving of data readiness levels, but in reflection I think I still did not take enough of the ideas seriously enough. Because I underestimated the time to be spent on manual curation after the initial scrape of the data, I did not put enough effort into robust code for dealing with ligatures and other unusual character codes.

There were several moments where, in retrospect, I should have refactored my processing code. But since I was driven by the desire to complete the project, particularly given I'd severely underestimated how long it would take, I chose to push forward. One interesting question is how ideas from software engineering, such as agile development philosophies, could have helped in making myself more aware of these errors.

## Disease Monitoring in Uganda

The second case study refers to work in collaboration with the University of Makerere and UN Global Pulse in Kampala, Uganda in the prediction of disease outbreak \[4\]. Specifically, our original interest was understanding the spatial correlations of malarial disease in Uganda, and their interactions with other measures such as NDVI, rainfall, altitude etc. Our effort was a countrywide effort, but simultaneous to our work was an international effort in the Malaria Atlas Project.[^4]

We constructed spatial models based on Gaussian processes that were designed to detect and use the correlations between the different covariates. The work was done through two PhD students' thesis, Martin from Kampala, and Ricardo from the UK. The work in Kampala mainly drove the data collection and collation. In particular, interpretation of satellite images as NDVI, alignment of the rainfall and altitude maps. The data collation in itself took probably around 70-80% of the work on the project. Martin working on it full time, and Ricardo assisting and working on both modeling and data munging. Ricardo visited Uganda twice, initially for a scoping visit and later for a longer collaboration visit in the processing of the data during his PhD. These visits were to the University of Makerere where Martin was working.

The malaria incidence data was drawn from Health Management Information System (HMIS) data. For privacy purposes, original data was not available to us, but rather it had been aggregated across certain regions.

One key challenge in the data munching is that the administrative areas (in Uganda known as districts) change. One administrative area can divide into two. When this happens the history of the two districts needs to be divided across the two areas. This presented a missing data problem that Ricardo also spent a deal of time addressing during his thesis.

By the time of submission of the thesis, Ricardo had recovered a negative result, he couldn't show spatial correlation among the districts. This was disappointing as it had been one of the main motivations of that thesis work. Nevertheless, even without the spatial correlation there was interest in deploying the system by the Ugandan ministry of health for disease outbreak prediction.

Ricardo's expertise was such that he was able to then spend 3 months in Africa, this time at UN Global Pulse, where Martin and his supervisor were now working, rather than Makerere. This was to help with the implementation of early warning systems based on the model. These early warning systems did not require the spatial correlation that hadn't worked during the thesis.

During the period at the UN, Ricardo and Martin were able to work directly with disaggregated data. The UN had permission to see the raw health center results, whereas the University of Makerere did not. As a result, they found that aggregation we had previously been working on had been performed across data containing missing values. Ricardo and Martin reworked the model to deal with missing values and recovered the expected spatial correlation.

In other words, due to a data processing error at Band B, a negative result was obtained at Band A. A lack of documentation, or a lack of asking the right questions, led to an oversight on this error.

The models are now deployed for disease outbreak prediction across Uganda, where they are combined with knowledge of population movement from mobile phone data to trigger interventions.

The lessons learned were the following.

1.  In data where we had direct access the amount of work required to align, and the number of design decisions we made in the summarization dominated the project.

2.  There was still data that fell out of our control due to confidentiality reasons. Despite the work we'd done on our own data, which would have allowed us to infer that other design decisions would have been made by those that were aggregating the data, we failed to ask the right questions of the data providers. Indeed, those data providers may not have even known the answers. There was no substitute for directly looking at the data.

3.  Although we began the project with one set of goals in mind: understanding the correlation between different factors in malaria, our outcome: disease prediction for early intervention, was quite different. Happenstance outcomes need to be accommodated in project goals. They arise particularly around Band B of the process.

### How Data Readiness Levels could have helped

Being aware of data readiness levels would have done three things. Firstly, we would have estimated better how much time that Martin would require in data munging from satellite images etc. Secondly, we would have questioned the process by which the HMIS data was being presented to us, and what stage of readiness it was at, and how it got there. These are broadly questions that sit in Band C and B. Thirdly, awareness of the transition from Band B to Band A (data moving to its context) would have made us realize that the question may well evolve and be more responsive to that outcome. In the end this move was driven by the shift in collaboration from Makerere to the UN.

Ricardo has continued to work in this area, and one focus of his new UCSF Global Health research group has been to clean up existing data and make it available as geospatial layers for other groups to use.

# Conclusion

Machine learning researchers probably didn't enter the field to do project management, but it may be that many failings on large data projects are associated with a failure to provision resource for the challenges involved in preparing our data, rather than a failing in the algorithmics of the system. Costs of data curation are often underestimated and those who do the work in Band C and Band B are very often undervalued.

Data readiness levels highlight the different skill sets required in each stage of analysis, from software engineer, to data-munger, to data scientist to machine learning scientist.

Some consensus about such levels would help organizations (and their managers, financial controllers) quantify the value associated with data and allocate resource correctly to developing data sets that are robust and representative. A well conducted data analysis will lead to a good customer experience, but by the same token badly waste resources and give a poor customer experience.

This paper has outlined the basis for a partial solution to these problem, but to deploy in practice we would need consensus around data readiness levels and how they should be deployed. The emerging field of data science is the ideal domain to explore the utility of these ideas, evolve their specification and begin to properly account for the value of well curated data.

For convenient reference below we show examples of *technology* readiness levels from the Department of Defense (sourced from [Wikipedia](https://en.wikipedia.org/wiki/Technology_readiness_level). For *Data* Readiness we are proposing to start with the three bands described above. Technology readiness relies entirely on a numbered system.

\[1\] T. Gilovich, D. Griffin, and D. Kahneman, Eds., *Heuristics and biases: The psychology of intuitive judgment*. Cambridge University Press, 2002.

\[2\] J. Banke, "Technology readiness levels demystified," NASA, 2010.

\[3\] J. Vanschoren, J. N. Rijn, and B. Bischl, "Taking machine learning research online with OpenML," in *Proceedings of the 4th international workshop on big data, streams and heterogeneous source mining: Algorithms, systems, programming models and applications*, 2015, vol. 41, pp. 1–4.

\[4\] R. Andrade-Pacheco, M. Mubangizi, J. Quinn, and N. Lawrence, "Monitoring short term changes of infectious diseases in uganda with gaussian processes," in *Advanced analysis and learning on temporal data: First ecml pkdd workshop, aaltd 2015, porto, portugal, september 11, 2015, revised selected papers*, A. Douzal-Chouakria, J. A. Vilar, and P.-F. Marteau, Eds. Cham: Springer International Publishing, 2016, pp. 95–110.

# Exemplar Technology Readiness Levels

1.  Basic principles observed and reported

    Lowest level of technology readiness. Scientific research begins to be translated into applied research and development (R&D). Examples might include paper studies of a technology's basic properties.

2.  Technology concept and/or application formulated

    Invention begins. Once basic principles are observed, practical applications can be invented. Applications are speculative, and there may be no proof or detailed analysis to support the assumptions. Examples are limited to analytic studies.

3.  Analytical and experimental critical function and/or characteristic proof of concept

    Active R&D is initiated. This includes analytical studies and laboratory studies to physically validate the analytical predictions of separate elements of the technology. Examples include components that are not yet integrated or representative.

4.  Component and/or breadboard validation in laboratory environment

    Basic technological components are integrated to establish that they will work together. This is relatively "low fidelity" compared with the eventual system. Examples include integration of "ad hoc" hardware in the laboratory.

5.  Component and/or breadboard validation in relevant environment

    Fidelity of breadboard technology increases significantly. The basic technological components are integrated with reasonably realistic supporting elements so they can be tested in a simulated environment. Examples include "high-fidelity" laboratory integration of components.

6.  System/subsystem model or prototype demonstration in a relevant environment

    Representative model or prototype system, which is well beyond that of TRL 5, is tested in a relevant environment. Represents a major step up in a technology's demonstrated readiness. Examples include testing a prototype in a high-fidelity laboratory environment or in a simulated operational environment.

7.  System prototype demonstration in an operational environment.

    Prototype near or at planned operational system. Represents a major step up from TRL 6 by requiring demonstration of an actual system prototype in an operational environment (e.g., in an aircraft, in a vehicle, or in space).

8.  Actual system completed and qualified through test and demonstration.

    Technology has been proven to work in its final form and under expected conditions. In almost all cases, this TRL represents the end of true system development. Examples include developmental test and evaluation (DT&E) of the system in its intended weapon system to determine if it meets design specifications.

9.  Actual system proven through successful mission operations.

    Actual application of the technology in its final form and under mission conditions, such as those encountered in operational test and evaluation (OT&E). Examples include using the system under operational mission conditions.

[^1]: See appendix for examples of technology readiness level descriptions.

[^2]: The nanotechnology community has also looked at data readiness levels in this [discussion document from the nanotechnology community in 2013](http://www.nano.gov/node/1015). However, their scope doesn't seem to be general enough to deal with the challenges of data processing in domains beyond nanotechnology.

[^3]: http://proceedings.mlr.press

[^4]: http://www.map.ox.ac.uk/