An integrated brain-machine interface platform with thousands of channels

Elon Musk
Neuralink
An integrated brain-machine interface platform with thousands of channels

Brain-machine interfaces (BMIs) hold promise for the restoration of sensory and motor function and the treatment of neurological disorders, but clinical BMIs have not yet been widely adopted, in part because modest channel counts have limited their potential. In this white paper, we describe Neuralink's first steps toward a scalable high-bandwidth BMI system. We have built arrays of small and flexible electrode "threads", with as many as 3,072 electrodes per array distributed across 96 threads. We have also built a neurosurgical robot capable of inserting six threads (192 electrodes) per minute. Each thread can be individually inserted into the brain with micron precision for avoidance of surface vasculature and targeting specific brain regions. The electrode array is packaged into a small implantable device that contains custom chips for low-power on-board amplification and digitization: the package for 3,072 channels occupies less than (23 × 18.5 × 2) mm 3 . A single USB-C cable provides full-bandwidth data streaming from the device, recording from all channels simultaneously. This system has achieved a spiking yield of up to 85.5 % in chronically implanted electrodes. Neuralink's approach to BMI has unprecedented packaging density and scalability in a clinically relevant package.

Introduction

Brain-machine interfaces (BMIs) have the potential to help people with a wide range of clinical disorders. For example, researchers have demonstrated human neuroprosthetic control of computer cursors [1,2,3], robotic limbs [4,5], and speech synthesizers [6] using no more than 256 electrodes. While these successes suggest that high fidelity information transfer between brains and machines is possible, development of BMI has been critically limited by the inability to record from large numbers of neurons. Noninvasive approaches can record the average of millions of neurons through the skull, but this signal is distorted and nonspecific [7,8]. Invasive electrodes placed on the surface of the cortex can record useful signals, but they are limited in that they average the activity of thousands of neurons and cannot record signals deep in the brain [9]. Most BMI's have used invasive techniques because the most precise readout of neural representations requires recording single action potentials from neurons in distributed, functionally-linked ensembles [10].

Microelectrodes are the gold-standard technology for recording action potentials, but there has not been a clinicallytranslatable microelectrode technology for large-scale recordings [11]. This would require a system with material properties that provide high biocompatibility, safety, and longevity. Moreover, this device would also need a practical surgical approach and high-density, low-power electronics to ultimately facilitate fully-implanted wireless operation.

Most devices for long-term neural recording are arrays of electrodes made from rigid metals or semiconductors [12,13,14,15,16,17,18]. While rigid metal arrays facilitate penetrating the brain, the size, Young's modulus and bending stiffness mismatches between stiff probes and brain tissue can drive immune responses that limit the function and longevity of these devices [19,11]. Furthermore, the fixed geometry of these arrays constrains the populations of neurons that can be accessed, especially due to the presence of vasculature.

An alternative approach is to use thin, flexible multi-electrode polymer probes [20,21]. The smaller size and increased flexibility of these probes should offer greater biocompatibility. However, a drawback of this approach is that thin polymer probes are not stiff enough to directly insert into the brain; their insertion must be facilitated by stiffeners [22,21], injection [23,24] or other approaches [25], all of which are quite slow [26,27]. To satisfy the functional requirements for a high-bandwidth BMI, while taking advantage of the properties of thin-film devices, we developed a robotic approach, where large numbers of fine and flexible polymer probes are efficiently and independently inserted across multiple brain regions [28].

Here, we report Neuralink's progress towards a flexible, scalable BMI that increases channel count by an order of magnitude over prior work. Our system has three main components: ultra-fine polymer probes (section 2 of this report), a neurosurgical robot (section 3), and custom high-density electronics (section 4). We demonstrate the rapid implantation of 96 polymer threads, each thread with 32 electrodes for a total of 3,072 electrodes.

We developed miniaturized custom electronics that allow us to stream full broadband electrophysiology data simultaneously from all these electrodes (section 5). We packaged this system for long-term implantation and developed custom online spike detection software that can detect action potentials with low latency. Together, this system serves as a state-of-the-art research platform and a first prototype towards a fully implantable human BMI. We have developed a custom process to fabricate minimally displacive neural probes that employ a variety of biocompatible thin film materials. The main substrate and dielectric used in these probes is polyimide, which encapsulates a gold thin film trace. Each thin film array is composed of a "thread" area that features electrode contacts and traces and a "sensor" area where the thin film interfaces with custom chips that enable signal amplification and acquisition. A wafer-level microfabrication process enables high-throughput manufacturing of these devices. Ten thin film devices are patterned on a wafer, each with 3,072 electrode contacts. Each array has 48 or 96 threads, each of those containing 32 independent electrodes. Integrated chips are bonded to the contacts on the sensor area of the thin film using a flip-chip bonding process. One goal of this approach is to maintain a small thread cross-sectional area to minimize tissue displacement in the brain. To achieve this, while keeping channel count high, stepper lithography and other microfabrication techniques are used to form the metal film at sub-micron resolution.

Threads

We have designed and manufactured over 20 different thread and electrode types into our arrays; two example designs are shown in panels A and B of fig. 1. We have fabricated threads ranging from 5 to 50 µm in width that incorporate recording sites of several geometries ( fig. 1). Thread thickness is nominally 4 to 6 µm, which includes up to three layers of insulation and two layers of conductor. Typical thread length is approximately 20 mm. To manage these long, thin threads prior to insertion, parylene-c is deposited onto the threads to form a film on which the threads remain attached until the surgical robot pulls them off. Each thread ends in a (16 × 50) µm 2 loop to accommodate needle threading.

Since the individual gold electrode sites have small geometric surface areas ( fig. 1C), we use surface modifications to lower the impedance for electrophysiology and increase the effective charge-carrying capacity of the interface ( fig. 1D). Two such treatments that we have used are the electrically conductive polymer poly-ethylenedioxythiophene doped with polystyrene sulfonate (PEDOT:PSS) [29,30] and iridium oxide (IrOx) [31,32]. In bench-top testing we have achieved impedances of 36.97 ± 4.68 kΩ (n = 257 electrodes) and 56.46 ± 7.10 kΩ (n = 588) for PEDOT:PSS and IrOx, respectively. The lower impedance of PEDOT:PSS is promising, however the long-term stability and biocompatibility of PEDOT:PSS is less well established than for IrOx. These techniques and processes can be improved and further extended to other types of conductive electrode materials and coatings. Thin-film polymers have previously been used for electrode probes [21], but their low bending stiffness complicates insertions. Neuralink has developed a robotic insertion approach for inserting flexible probes [28], allowing rapid and reliable insertion of large numbers of polymer probes targeted to avoid vasculature and record from dispersed brain regions. The robot's insertion head is mounted on 10 µm globally accurate, 400 mm × 400 mm × 150 mm travel threeaxis stage, and holds a small, quick-swappable, "needle-pincher" assembly ( fig. 2, fig. 3A).

Robot

The needle is milled from 40 µm diameter tungsten-rhenium wire-stock electrochemically etched to 24 µm diameter along the inserted length ( fig. 2A). The tip of the needle is designed both to hook onto insertion loops-for transporting and inserting individual threads-and to penetrate the meninges and brain tissue. The needle is driven by a linear motor allowing variable insertion speeds and rapid retraction acceleration (up to 30,000 mm s −2 ) to encourage separation of the probe from the needle. The pincher is a 50 µm tungsten wire bent at the tip and driven both axially and rotationally ( fig. 2B). It serves as support for probes during transport and as a guide to ensure that threads are inserted along the needle path. Figure 4 shows a sequence of photographs of the insertion process into an agarose brain proxy.

The inserter head also holds an imaging stack (fig. 3E-G) used for guiding the needle into the thread loop, insertion targeting, live insertion viewing, and insertion verification. In addition, the inserter head contains six independent light modules, each capable of independently illuminating with 405 nm, 525 nm and 650 nm or white light ( fig. 3C). The 405 nm illumination excites fluorescence from polyimide and allows the optical stack and computer vision to reliably localize the (16 × 50) µm 2 thread loop and execute sub-micron visual servoing to guide, illuminated by 650 nm the needle through it. Stereoscopic cameras, software based monocular extended depth of field calculations, and illumination with 525 nm light allow for precise estimation of the location of the cortical surface.

The robot registers insertion sites to a common coordinate frame with landmarks on the skull, which, when combined with depth tracking, enables precise targeting of anatomically defined brain structures. An integrated custom software suite allows pre-selection of all insertion sites, enabling planning of insertion paths optimized to minimize tangling and strain on the threads. The planning feature highlights the ability to avoid vasculature during insertions, one of the key advantages of inserting electrodes individually. This is particularly important, since damage to the blood-brain barrier is thought to play a key role in the brain's inflammatory response to foreign objects [33].

The robot features an auto-insertion mode, which can insert up to 6 threads (192 electrodes) per minute. While the entire insertion procedure can be automated, the surgeon retains full control, and if desired, can make manual microadjustments to the thread position before each insertion into the cortex. The neurosurgical robot is compatible with sterile shrouding, and has features to facilitate successful and rapid insertions such as automatic sterile ultrasonic cleaning of the needle. The needle pincher cartridge (NPC; fig. 2C) is the portion of the inserter head that makes direct contact with brain tissue and is a consumable that can be replaced mid-surgery in under a minute. July 16, 2019 The inserter approaches the brain proxy with a thread. i. needle and cannula. ii. previously inserted thread.

2.

Inserter touches down on the brain proxy surface. 3. Needle penetrates tissue proxy, advancing the thread to the desired depth. iii. inserting thread. 4. Inserter pulls away, leaving the thread behind in the tissue proxy. iv. inserted thread.

With this system, we have demonstrated an average of 87.1 ± 12.6 % (mean ± s.d.) insertion success rate over 19 surgeries. In this study, precise manual adjustments were made to avoid microvasculature on the cortical surface, slowing total insertion time from the fastest possible. Even with these adjustments, the total insertion time for this study averaged ∼45 min, for an approximate insertion rate of 29.6 electrodes per minute ( fig. 6). Insertions were made in a (4 × 7) mm 2 bilateral craniotomy with >300 µm spacing between threads to maximize cortical coverage. This demonstrates that robotic insertion of thin polymer electrodes is an efficient and scalable approach for recording from large numbers of neurons in anatomically defined brain regions.

Electronics

Chronic recording from thousands of electrode sites presents significant electronics and packaging challenges. The density of recording channels necessitates placing the signal amplification and digitization stack within the array assembly, otherwise the cable and connector requirements would be prohibitive. This recording stack must amplify small neural signals (<10 µV RMS ) while rejecting out-of-band noise, sample and digitize the amplified signals, and stream out the results for real-time processing-all using minimal power and size.

The electronics are built around our custom Neuralink application specific integrated circuit (ASIC), which consists of 256 individually programmable amplifiers ("analog pixels"), on-chip analog-to-digital converters (ADCs), and peripheral control circuitry for serializing the digitized outputs. The analog pixel is highly configurable: the gains and filter July 16, 2019 properties can be calibrated to account for variability in signal-quality due to process variations and the electrophysiological environment. The on-chip ADC samples at 19.3 kHz with 10 bit resolution. Each analog pixel consumes 5.2 µW and the whole ASIC consumes ∼6 mW, including the clock drivers. Performance of the Neuralink ASIC is summarized in table 1 and a photograph of the fabricated device is shown in fig. 5A.

The Neuralink ASIC forms the core of a modular recording platform that allows for easy replacement of constitutive parts for research and development purposes (fig. 5). In the systems discussed here, a number of ASICs are integrated into a standard printed circuit board (PCB) using flip-chip integration. Each system consists of a field-programmable gate array (FPGA); real-time temperature, accelerometer, and magnetometer sensors; and a single USB-C connector for full-bandwidth data transfer. The systems are packaged in titanium cases which are coated with parylene-c, which serves as a moisture barrier to prevent fluid ingress and prolong functional lifetime.

We describe two such configurations that we have built, a 1,536 channel recording system ("System A") and a 3,072 channel recording system ("System B"), summarized in table 2. While System A employs the current-generation Neuralink ASIC, System B uses an earlier revision with comparable functionality but poorer performance specifications. System B was designed to maximize channel density and is used for applications that demand extremely high channel count. In contrast, System A was designed to facilitate faster and more reliable manufacturing; it can be built five times faster than System B with better yields.

An ethernet-connected base station converts the data streams from these systems into multicast 10G ethernet UDP packets allowing downstream users to process the data in a variety of ways, e.g. visualizing the data in real-time [34] or writing it to disk. Each base station can connect to up to three implants simultaneously. These devices are further supported by a software ecosystem that allows for plug and play usability with zero configuration: neural data begins streaming automatically when a cable is connected. July 16, 2019

Electrophysiology

We have implanted both systems A and B in male Long-Evans rats, as described in section 3. All animal procedures were performed in accordance with the National Research Council's Guide for the Care and Use of Laboratory Animals and were approved by the Neuralink Institutional Animal Care and Use Committee. Electrophysiological recordings were made as the animals freely explored an arena equipped with a commutated cable that permitted unrestricted movement. System A can record 1,344 out of 1,536 channels simultaneously, the exact channel configuration can be arbitrarily specified at the time of recording; system B can record from all 3,072 channels simultaneously. Digitized broadband signals were processed in real-time to identify action potentials (spikes) using an online detection algorithm.

Spike detection requirements for real-time BMI are different from most conventional neurophysiology. While most electrophysiologists spike-sort data offline and spend significant effort to reject false-positive spike events, BMI events must be detected in real time and spike detection parameters must maximize decoding efficacy. Using our custom online spike-detection software, we found that a permissive filter that allows an estimated false positive rate of ∼0.2 Hz performs better than setting stringent thresholds that may reject real spikes (data not shown).

Given these considerations, we set a threshold of >0. 35 Hz to quantify the number of electrodes that recorded spiking units. Since we typically do not spike sort our data, we do not report multiple units per channel. BMI decoders com- monly operate without spike sorting with minimal loss of performance [36,37]. Moreover, recent results show that spike sorting is not necessary to accurately estimate neural population dynamics [38].

Data from a recent experiment using System A are shown in fig. 7 and fig. 8. In this experiment, 40 of 44 attempted insertions were successful (90 %) for a total of 1280 implanted electrodes, of which 1,020 were recorded simultaneously. The broadband signals recorded from a representative thread show both local field and spiking activity fig. 7. A sample output of the spike detection pipeline is shown in raster form in fig. 8. In this example, two overlapping recording configurations were used to record from all 1,280 implanted channels. On this array, our spiking yield was 53.4 % of channels, with many spikes appearing on multiple neighboring channels, as has been observed in other high-density probes [16,17,21]. On other System A arrays we observed 59.10 ± 5.74 % (mean ± s.e.m.) across 19 surgeries with a maximum spiking yield of 85.5 %.

Discussion

We have described a BMI with high-channel count and single-spike resolution. It is based on flexible polymer probes, a robotic insertion system, and custom low-power electronics. This system serves two main purposes: it is a research platform for use in rodents and serves as a prototype for future human clinical implants. The ability to quickly iterate designs and testing in rodents allows for the rapid refinement of devices, manufacturing processes, and software. Because it is a research platform, the system uses a wired connection to maximize the bandwidth for raw data streaming. This is important for performance assessments and crucial for the development of signal processing and decoding algorithms. In contrast, the clinical devices that will derive from this platform will be fully implantable-which requires hermetic packaging-and have on-board signal compression, reduced power consumption, wireless power transmission, and data telemetry through the skin without percutaneous leads.

Figure 8:

Our devices allow the recording of widespread neural activity distributed across multiple brain regions and cortical layers. Left: thread insertion sites (colored circles) are indicated on rendered rodent brain. [35] Right: raster of 1,020 simultaneously recorded channels, sorted per thread (color corresponds to insertion site). Inset: enlarged raster of spikes from a single thread. This thread corresponds to the one shown in fig. 7.

Modulating neural activity will be an important part of next-generation clinical brain-machine interfaces [39], for example to provide a sense of touch or proprioception to neuroprosthetic movement control [40,41]. Therefore, we designed the Neuralink ASIC to be capable of electrical stimulation on every channel, although we have not demonstrated these capabilities here.

This BMI system has several advantages over previous approaches. The size and composition of the thin-film probes are a better match for the material properties of brain tissue than commonly used silicon probes, and therefore may exhibit enhanced biocompatibility [28,21]. Also, the ability to choose where our probes are inserted, including into sub-cortical structures, allows us to create custom array geometries for targeting specific brain regions while avoiding vasculature. This feature is significant for creating a high-performance BMI, as the distribution of electrodes can be customized depending on the task requirements. Lastly, the miniturization and design of the Neuralink ASIC affords great flexibility in system design and supports very high channel counts within practical size and power constraints.

In principle, our approach to brain-machine interfaces is highly extensible and scalable. Here we report simultaneous broadband recording from over 3,000 inserted electrodes in a freely moving rat. In a larger brain, multiple devices with this architecture could be readily implanted, and we could therefore interface with many more neurons without extensive re-engineering. Further development of surgical robotics could allow us to accomplish this without dramatically increasing surgery time.

While significant technological challenges must be addressed before a high-bandwidth device is suitable for clinical application, with such a device, it is plausible to imagine that a patient with spinal cord injury could dexterously control a digital mouse and keyboard. When combined with rapidly improving spinal stimulation techniques [42], in the future this approach could conceivably restore motor function. High-bandwidth neural interfaces should enable a variety of novel therapeutic possibilities.

Acknowledgments

We would like to acknowledge the contributions of Lawrence Livermore National Laboratory (LLNL), Berkeley Marvell Nanofabrication Laboratory, Berkeley Wireless Research Center (BWRC), Stanford Nanofabrication Facility, and former and current Neuralink employees to the work described here.

Supplementary Videos

Video 1: A series of six insertions by the neurosurgical robot into an agarose brain proxy. Thread-capture by the needle occurs off-frame. The changes background color are caused by illumination with different frequencies of light at different stages of the threading and insertion process. One thread was inserted before the start of the video.

Video 2:

A 3D rendered view of thread arrangement (same data presented in fig. 8). Thread insertion is visualized in the same order as in the actual surgery, but time has been compressed for presentation. Thread size and insertion depth are representative. The stereotaxic coordinates of each insertion are represented on the dataset provided by Calabrese and coworkers [35].

Figure 1 :
1Our novel polymer probes. A. "Linear Edge" probes, with 32 electrode contacts spaced by 50 µm. B. "Tree" probes with 32 electrode contacts spaced by 75 µm. C. Increased magnification of individual electrodes for the thread design in panel A, emphasizing their small geometric surface area. D. Distribution of electrode impedances (measured at 1 kHz) for two surface treatments: PEDOT (n = 257) and IrOx (n = 588).

Figure 2 :
2Needle pincher cartridge (NPC) compared with a penny for scale. A. Needle. B. Pincher. C. Cartridge.

Figure 3 :
3The robotic electrode inserter; enlarged view of the inserter-head shown in the inset. A. Loaded needle pincher cartridge. B. Low-force contact brain position sensor. C. Light modules with multiple independent wavelengths. D. Needle motor. E. One of four cameras focused on the needle during insertion. F. Camera with wide angle view of surgical field. G. Stereoscopic cameras.

Figure 4 : 1 .
41Figure 4: 1. The inserter approaches the brain proxy with a thread. i. needle and cannula. ii. previously inserted thread. 2. Inserter touches down on the brain proxy surface. 3. Needle penetrates tissue proxy, advancing the thread to the desired depth. iii. inserting thread. 4. Inserter pulls away, leaving the thread behind in the tissue proxy. iv. inserted thread.

Figure 5 :
5A packaged sensor device. A. individual neural processing ASIC capable of processing 256 channels of data. This particular packaged device contains 12 of these chips for a total of 3,072 channels. B. Polymer threads on parylene-c substrate. C. Titanium enclosure (lid removed). D. Digital USB-C connector for power and data.

Figure 6 :Figure 7 :
67Thread implantation and packaging. A. An example peri-operative image showing the cortical surface with implanted threads and minimal bleeding. B. Packaged sensor device ("System B") chronically implanted in a rat. Left: Broadband neural signals (unfiltered) simultaneously acquired from a single thread (32 channels) implanted in rat cerebral cortex. Each channel (row) corresponds to an electrode site on the thread (schematic at left; sites spaced by 50 µm). Spikes and local field potentials are readily apparent. Right: Putative waveforms (unsorted); numbers indicate channel location on thread. Mean waveform is shown in black.

Table 1 :
1Neuralink ASICNumber of Channels
256
Gain
42.9-59.4 dB
Bandwidth
3 Hz-27 kHz
Input-referred noise (3Hz -10KHz)
5.9 µV RMS
Maximum Differential Input Range
7.2 mV PP
ADC resolution
10 bit
Analog Pixel Power
5.2 µW

Table 2 :
2Two recording system configurationsSystem A
System B

Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Leigh R Hochberg, Nature. 442Leigh R. Hochberg et al. "Neuronal ensemble control of prosthetic devices by a human with tetraplegia". In: Nature 442 (2006), p. 164. issn: 1476-4687.

An Electrocorticographic Brain Interface in an Individual with Tetraplegia. Wei Wang, PLoS ONE. 855344Wei Wang et al. "An Electrocorticographic Brain Interface in an Individual with Tetraplegia". In: PLoS ONE 8 (2013), e55344.

Decoding motor imagery from the posterior parietal cortex of a tetraplegic human. Tyson Aflalo, Science. 348Tyson Aflalo et al. "Decoding motor imagery from the posterior parietal cortex of a tetraplegic human". In: Science 348 (2015), pp. 906-910. issn: 0036-8075.

Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Leigh R Hochberg, Nature. 485Leigh R. Hochberg et al. "Reach and grasp by people with tetraplegia using a neurally controlled robotic arm". In: Nature 485 (2012), p. 372. issn: 1476-4687.

High-performance neuroprosthetic control by an individual with tetraplegia. L Jennifer, Collinger, The Lancet. 381Jennifer L Collinger et al. "High-performance neuroprosthetic control by an individual with tetraplegia". In: The Lancet 381 (2013), pp. 557-564. issn: 0140-6736.

Speech synthesis from neural decoding of spoken sentences. K Gopala, Josh Anumanchipalli, Edward F Chartier, Chang, Nature. 568Gopala K. Anumanchipalli, Josh Chartier, and Edward F. Chang. "Speech synthesis from neural decoding of spoken sentences". In: Nature 568 (2019), pp. 493-498. issn: 0028-0836.

The origin of extracellular fields and currents -EEG, ECoG, LFP and spikes. György Buzsáki, A Costas, Christof Anastassiou, Koch, Nature Reviews Neuroscience. 13György Buzsáki, Costas A. Anastassiou, and Christof Koch. "The origin of extracellular fields and currents - EEG, ECoG, LFP and spikes". In: Nature Reviews Neuroscience 13 (2012), p. 407. issn: 1471-0048.

Investigating large-scale brain dynamics using field potential recordings: analysis and interpretation. Bijan Pesaran, Nature Neuroscience. 21Bijan Pesaran et al. "Investigating large-scale brain dynamics using field potential recordings: analysis and inter- pretation". In: Nature Neuroscience 21 (2018), pp. 903-919. issn: 1097-6256.

High Spatiotemporal Resolution ECoG Recording of Somatosensory Evoked Potentials with Flexible Micro-Electrode Arrays. Taro Kaiju, Frontiers in Neural Circuits. 1120Taro Kaiju et al. "High Spatiotemporal Resolution ECoG Recording of Somatosensory Evoked Potentials with Flexible Micro-Electrode Arrays". In: Frontiers in Neural Circuits 11 (2017), p. 20.

From the neuron doctrine to neural networks. Rafael Yuste, Nature Reviews Neuroscience. 16Rafael Yuste. "From the neuron doctrine to neural networks". In: Nature Reviews Neuroscience 16 (2015), pp. 487- 497. issn: 1471-003x.

Novel electrode technologies for neural recordings. Guosong Hong, M Charles, Lieber, Nature Reviews Neuroscience. Guosong Hong and Charles M Lieber. "Novel electrode technologies for neural recordings". In: Nature Reviews Neuroscience (2019), pp. 1-16. issn: 1471-003X.

The Utah Intracortical Electrode Array: A recording structure for potential brain-computer interfaces. Edwin M Maynard, Craig T Nordhausen, Richard A Normann, Electroencephalography and Clinical Neurophysiology. 102Edwin M. Maynard, Craig T. Nordhausen, and Richard A. Normann. "The Utah Intracortical Electrode Array: A recording structure for potential brain-computer interfaces". In: Electroencephalography and Clinical Neurophys- iology 102.3 (1997), pp. 228-239. issn: 0013-4694.

Chronic, multisite, multielectrode recordings in macaque monkeys. A L Miguel, Nicolelis, Proceedings of the National Academy of Sciences. the National Academy of Sciences100Miguel A. L. Nicolelis et al. "Chronic, multisite, multielectrode recordings in macaque monkeys". In: Proceedings of the National Academy of Sciences 100.19 (2003), pp. 11041-11046. issn: 0027-8424. eprint: https://www. pnas.org/content/100/19/11041.full.pdf.

Microelectrodes, Microelectronics, and Implantable Neural Microsystems. K D Wise, Proceedings of the IEEE 96. the IEEE 967K. D. Wise et al. "Microelectrodes, Microelectronics, and Implantable Neural Microsystems". In: Proceedings of the IEEE 96.7 (2008), pp. 1184-1202. issn: 0018-9219.

A Large-Scale Semi-Chronic Microdrive Recording System for Non-Human Primates. Nicholas M Dotson, Neuron. 96e2. issnNicholas M. Dotson et al. "A Large-Scale Semi-Chronic Microdrive Recording System for Non-Human Primates". In: Neuron 96 (2017), 769-782.e2. issn: 0896-6273.

Fully integrated silicon probes for high-density recording of neural activity. James J Jun, Nature. 551James J. Jun et al. "Fully integrated silicon probes for high-density recording of neural activity". In: Nature 551 (2017), p. 232. issn: 1476-4687.

SiNAPS: an implantable Active Pixel Sensor CMOS-probe for Simultaneous largescale Neural recordings. Gian Nicola Angotzi, Biosensors and Bioelectronics. 126Gian Nicola Angotzi et al. "SiNAPS: an implantable Active Pixel Sensor CMOS-probe for Simultaneous large- scale Neural recordings". In: Biosensors and Bioelectronics 126 (2018), pp. 355-364. issn: 0956-5663.

Amorphous silicon carbide ultramicroelectrode arrays for neural stimulation and recording. Felix Deku, Journal of Neural Engineering. 1516007Felix Deku et al. "Amorphous silicon carbide ultramicroelectrode arrays for neural stimulation and recording". In: Journal of Neural Engineering 15.1 (2018), p. 016007.

A review on mechanical considerations for chronically-implanted neural probes. Aziliz Lecomte, Emeline Descamps, Christian Bergaud, Journal of Neural Engineering. 15Aziliz Lecomte, Emeline Descamps, and Christian Bergaud. "A review on mechanical considerations for chronically-implanted neural probes". In: Journal of Neural Engineering 15 (2018), p. 031001. issn: 1741-2552.

NeuroGrid: recording action potentials from the surface of the brain. Dion Khodagholy, Nature Neuroscience. 18Dion Khodagholy et al. "NeuroGrid: recording action potentials from the surface of the brain". In: Nature Neu- roscience 18 (2014), pp. 310-315. issn: 1097-6256.

High-Density, Long-Lasting, and Multi-region Electrophysiological Recordings Using Polymer Electrode Arrays. Jason E Chung, 21-31.e5Neuron. 101Jason E. Chung et al. "High-Density, Long-Lasting, and Multi-region Electrophysiological Recordings Using Polymer Electrode Arrays". In: Neuron 101 (2019), 21-31.e5. issn: 0896-6273.

A sweet new multiple electrode for chronic single unit recording in moving animals. Stephan L Chorover, Anne-Marie Deluca, Physiology & Behavior. 9Stephan L. Chorover and Anne-Marie Deluca. "A sweet new multiple electrode for chronic single unit recording in moving animals". In: Physiology & Behavior 9 (1972), pp. 671-674. issn: 0031-9384.

Syringe-injectable electronics. Jia Liu, Nature Nanotechnology. 10Jia Liu et al. "Syringe-injectable electronics". In: Nature Nanotechnology 10 (2015), pp. 629-636. issn: 1748-3387.

Stable long-term chronic brain mapping at the single-neuron level. Tian-Ming Fu, Nature Methods. 13Tian-Ming Fu et al. "Stable long-term chronic brain mapping at the single-neuron level". In: Nature Methods 13 (2016), pp. 875-882. issn: 1548-7091.

Fluidic Microactuation of Flexible Electrodes for Neural Recording. Flavia Vitale, 10.1021/acs.nanolett.7b04184Nano Letters. 18Flavia Vitale et al. "Fluidic Microactuation of Flexible Electrodes for Neural Recording". In: Nano Letters 18.1 (2018), pp. 326-335. eprint: https://doi.org/10.1021/acs.nanolett.7b04184.

Ultraflexible nanoelectronic probes form reliable, glial scar-free neural integration. Lan Luan, e1601966. issnScience Advances. 3Lan Luan et al. "Ultraflexible nanoelectronic probes form reliable, glial scar-free neural integration". In: Science Advances 3 (2017), e1601966. issn: 2375-2548.

NeuroRoots, a bio-inspired, seamless Brain Machine Interface device for long-term recording. Marc D Ferro, In: bioRxiv. 460949Marc D. Ferro et al. "NeuroRoots, a bio-inspired, seamless Brain Machine Interface device for long-term record- ing. " In: bioRxiv (2018), p. 460949.

The "sewing machine" for minimally invasive neural recording. L Timothy, Hanson, bioRxivTimothy L Hanson et al. "The "sewing machine" for minimally invasive neural recording". In: bioRxiv (2019). eprint: https://www.biorxiv.org/content/early/2019/03/14/578542.full.pdf.

Chronic neural recordings using silicon microelectrode arrays electrochemically deposited with a poly(3,4-ethylenedioxythiophene) (PEDOT) film. A Kip, Ludwig, Journal of Neural Engineering. 3Kip A Ludwig et al. "Chronic neural recordings using silicon microelectrode arrays electrochemically deposited with a poly(3,4-ethylenedioxythiophene) (PEDOT) film". In: Journal of Neural Engineering 3 (2006), p. 59. issn: 1741-2552.

Poly(3,4-ethylenedioxythiophene) as a Micro-Neural Interface Material for Electrostimulation. Seth J Wilks, Frontiers in Neuroengineering. 2Seth J. Wilks et al. "Poly(3,4-ethylenedioxythiophene) as a Micro-Neural Interface Material for Electrostimula- tion". In: Frontiers in Neuroengineering 2 (2009), p. 7. issn: 1662-6443.

Morphology and charge capacity of sputtered iridium oxide films. S J D Klein, S F L Clauson, Cogan, Journal of Vacuum Science & Technology A: Vacuum, Surfaces, and Films. 7J D Klein, S L Clauson, and S F Cogan. "Morphology and charge capacity of sputtered iridium oxide films". In: Journal of Vacuum Science & Technology A: Vacuum, Surfaces, and Films 7 (1989), pp. 3043-3047. issn: 0734- 2101.

Sputtered Iridium Oxide Films (SIROFs) for Low-Impedance Neural Stimulation and Recording Electrodes. S F Cogan, T D Plante, J Ehrlich, The 26th Annual International Conference of the IEEE. 2S.F. Cogan, T. D. Plante, and J. Ehrlich. "Sputtered Iridium Oxide Films (SIROFs) for Low-Impedance Neural Stimulation and Recording Electrodes". In: The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society 2 (2004), pp. 4153-4156.

The impact of chronic blood-brain barrier breach on intracortical electrode function. Tarun Saxena, Biomaterials. 34Tarun Saxena et al. "The impact of chronic blood-brain barrier breach on intracortical electrode function". In: Biomaterials 34 (2013), pp. 4703-4713. issn: 0142-9612.

Open Ephys: an open-source, plugin-based platform for multichannel electrophysiology. Joshua H Siegle, J. Neural Eng. 14Joshua H. Siegle et al. "Open Ephys: an open-source, plugin-based platform for multichannel electrophysiology". In: J. Neural Eng. 14 (2017), pp. 1-13.

A quantitative magnetic resonance histology atlas of postnatal rat brain development with regional estimates of growth and variability. E Calabrese, NeuroImage. 71E. Calabrese et al. "A quantitative magnetic resonance histology atlas of postnatal rat brain development with regional estimates of growth and variability". In: NeuroImage 71 (2013), pp. 196-201.

To sort or not to sort: the impact of spike-sorting on neural decoding performance. Sonia Todorova, Journal of Neural Engineering. 11Sonia Todorova et al. "To sort or not to sort: the impact of spike-sorting on neural decoding performance". In: Journal of Neural Engineering 11 (2014), p. 056005. issn: 1741-2552.

Comparison of spike sorting and thresholding of voltage waveforms for intracortical brain-machine interface performance. P Breanne, Christie, Journal of Neural Engineering. 12Breanne P Christie et al. "Comparison of spike sorting and thresholding of voltage waveforms for intracortical brain-machine interface performance". In: Journal of Neural Engineering 12 (2015), p. 016009. issn: 1741-2552.

Accurate Estimation of Neural Population Dynamics without Spike Sorting. Eric M Trautmann, Neuron. Eric M. Trautmann et al. "Accurate Estimation of Neural Population Dynamics without Spike Sorting". In: Neuron (2019). issn: 0896-6273.

A wireless and artefact-free 128-channel neuromodulation device for closed-loop stimulation and recording in non-human primates. Andy Zhou, Nature Biomedical Engineering. 3Andy Zhou et al. "A wireless and artefact-free 128-channel neuromodulation device for closed-loop stimulation and recording in non-human primates". In: Nature Biomedical Engineering 3 (2019), pp. 15-26.

Active tactile exploration using a brain-machine-brain interface. E O' Joseph, Doherty, Nature. 479Joseph E O'Doherty et al. "Active tactile exploration using a brain-machine-brain interface". In: Nature 479 (2011). issn: 1476-4687.

Restored tactile sensation improves neuroprosthetic arm control. N Sharlene, Flesher, 653428In: bioRxivSharlene N Flesher et al. "Restored tactile sensation improves neuroprosthetic arm control". In: bioRxiv (2019), p. 653428.

Targeted neurotechnology restores walking in humans with spinal cord injury. B Fabien, Wagner, Nature. 563Fabien B. Wagner et al. "Targeted neurotechnology restores walking in humans with spinal cord injury". In: Na- ture 563 (2018), pp. 65-71. issn: 0028-0836.


Strategies for the quality assessment of the health care service providers in the treatment of Gastric Cancer in Colombia
2017

María Del Pilar Villamil
Department of Systems and Computing Engineering
University of Los Andes
Cr 1E No. 19A-10BogotáColombia

David Barrera
Department of Systems and Computing Engineering
University of Los Andes
Cr 1E No. 19A-10BogotáColombia

Nubia Velasco
Department of Systems and Computing Engineering
University of Los Andes
Cr 1E No. 19A-10BogotáColombia

Oscar Bernal
Department of Systems and Computing Engineering
University of Los Andes
Cr 1E No. 19A-10BogotáColombia

Esteban Fajardo
Department of Systems and Computing Engineering
University of Los Andes
Cr 1E No. 19A-10BogotáColombia

Carlos Urango
Department of Systems and Computing Engineering
University of Los Andes
Cr 1E No. 19A-10BogotáColombia

Sebastian Buitrago
Department of Systems and Computing Engineering
University of Los Andes
Cr 1E No. 19A-10BogotáColombia

Strategies for the quality assessment of the health care service providers in the treatment of Gastric Cancer in Colombia

Health Services Research
17654201710.1186/s12913-017-2440-8RESEARCH ARTICLE Open Access *Correspondence: Full list of author information is available at the end of the articleQuality assessmentDEAProcess miningData miningClustering
Background: While, at its inception in 1993, the health care system in Colombia was publicized as a paradigm to be copied across the developing world, numerous problems in its implementation have led to, what is now, an inefficient and crisis-ridden health system. Furthermore, as a result of inappropriate tools to measure the quality of the health service providers, several corruption scandals have arisen in the country. This study attempts to tackle this situation by proposing a strategy for the quality assessment of the health service providers (Entidades Promotoras de Salud, EPS) in the Colombian health system. In particular, as a case study, the quality of the treatment of stomach cancer is analyzed. Methods: The study uses two complementary techniques to address the problem. These techniques are applied based on data of the treatment of gastric cancer collected on a nation-wide scale by the Colombian Ministry of Health and Welfare. First, Data Envelopment Analysis (DEA) and the Malmquist Index (MI) are used to establish the most efficient EPS's within the system, according to indicators such as opportunity indicators. Second, sequential clustering algorithm, related to process mining a field of data mining, is used to determine the medical history of all patients and to construct typical care pathways of the patients belonging to efficient and inefficient EPS's. Lastly, efforts are made to identify traits and differences between efficient and inefficient EPS's. Results: Efficient and inefficient EPS were identified for the years 2010 and 2011. Additionally, a Malmquist Index was used to calculate the relative changes in the efficiency of the health providers. Using these efficiency rates, the typical treatment path of patients with gastric cancer was found for two EPSs: one efficient and another inefficient. Finally, the typical traits of the care pathways were established. Conclusions: Combining DEA and process mining proved to be a powerful approach understanding the problem and gaining valuable insight into the inner workings of the Colombian Health System, especially in terms of the treatment process performed by health care providers in critical illnesses such as cancer. However, no sufficiently compelling results were found to establish the contribution of such a combination to evaluate the quality in the delivery of health services.

Background

The Colombian Health System (CHS) is regulated by Law 100 of 1993. The CHS has evidence several problems in its implementation [1] and there is consensus across the country about the of an structural reform. Under these difficult circumstances, it is important to develop strategies and establish indicators that allow to estimate the quality of the health services in order to improve the system, which is of paramount importance for its population's well-being and quality of life. The heath care system is comprised of three independent actors. First, the institutions providing health services (IPS -Instituciones Prestadoras de Servicios de Salud) as hospitals, laboratories, health centers, among others; they are directly responsible for the treatment of patients and for providing all the necessary resources for the restoration of health and disease prevention. Second, the insurance companies known as Health Promotion Entities (EPS -Entidades Promotoras de Salud), they serve as intermediaries and administrators of the state resources. And third, the government, through the Colombian Ministry of Health and Welfare (MHW) and the Committee on Health Regulation (CHR), responsible for the regulation, control and management of the whole system.

As part of the compensation scheme the CHS, each IPS send to the EPS the minimum patient-data that the IPS are required to report in order to address, regulate and control the CHS. This data knows as Health Services Delivery Records RIPS (Registros Individuales de la Prestación de Servicios de Salud), includes, according to the law: user identification, type of consultation, type of procedures, emergency service data and medication and hospitalization data. The RIPS are issued at national-level and are consolidated and maintained by the MHW in a data warehouse.

The existence of RIPS enables the opportunity to measure the quality of the CHS, giving the possibility to identify strengths, weakness in the health system, and opportunities to improve the population's well-being. These issues, in conjunction with the necessity of measure the system quality and the need to develop unbiased measures of efficiency in the IPS to ensure an efficient system [2], motivate this work. This paper looks for evaluate the quality of the treatment for stomach cancer in Colombia, given that this type of cancer was one of the top thirteen causes of cancerrelated death in the country during the year 2015 [3], according to the Colombian epidemiological monitoring group on cancer. The proposal presents a strategy to evaluate the quality of gastric cancer treatment procedures in Colombia using DEA and process mining techniques. Firstly, quality indicators for each EPS are estimated. Secondly, a DEA model and the Malmquist Index are used to calculate the efficiency and the change in efficiency between 2010 and 2011, for a given institution. Finally, using sequential clustering, a typical algorithm in process mining, care paths for all patients are identified in order to assess the quality of a particular treatment. Furthermore, the proposed strategy can be used to identify the care paths for efficient and inefficient institutions, in order to compare them.

The paper is structured as follows: "Methods" section gives an overview of the methods used in the study and explores quality assessment in the context of gastric cancer. "Results and discussion" section presents the applicability of process mining and DEA to the treatment of gastric cancer in Colombia and presents our results. Finally, future work directions and conclusions are presented in "Conclusions" section.

Methods

The Colombian mandatory system of quality assurance (SOGCS, Sistema Obligatorio de Garantía de Calidad en Salud) includes a series of methodological elements to assess the quality of the health services. The current legislation states that the evaluation of the quality of services must be made by a comparison of observed quality and expected quality [4]. Such quality is validated based on pre-established rules, which are known to all participants in the health care system. These rules can come in different formats: manuals, practice guides, technical standards, and indicators, among others.

In this study, we chose to focus on the concept of clinical guidelines, which is defined as "a set of systematically developed recommendations to help practitioners and patients to make health care decisions and to select the most appropriate diagnostic and treatment options when addressing a health problem or specific condition" [5].

In order to evaluate the quality of gastric cancer treatments, we propose a three step strategy. First step prepares the information from RIPS to obtain those that are related to gastric cancer and with reliable information. Second step consists on applying DEA and the Malmquist Index techniques over the available information from different EPS. These techniques are applied by using a set of previously defined quality indicators related to the diseases treatment as base.

Finally, the third step selects a set of EPS with distinct efficiency characteristics and analyzed their treatment processes using the sequential clustering algorithm to construct typical care pathways. These efforts were aimed at identifying tendencies and patterns regarding the treatments applied to patients and relate them to the quantitative efficiency measures previously gathered for efficient and inefficient EPSs.

Data preparation

Data preparation consists of three basic steps: 1) Selection of patients diagnosed with gastric cancer; 2) Analysis of the quality of the available data; and 3) Grouping of the procedures performed on the patients starting from the time of the diagnosis.

Patient Identification. Table 1 shows stomach cancerrelated diseases codes used by ICD-10 standard (International Statistical Classification of Diseases and Related Health Problems 10th Revision). This standard is the one used on the RIPS an allows to identify all the patients suffering from stomach cancer in Colombia.

Analysis of the quality of the data. The patient data and their related procedures are analyzed in order to validate data quality. The two most common quality-related problems for the patients are data replication and inconsistencies between the procedures performed and the diagnosis; 9.9% of patients diagnosed with cancer are reported more than once between 2010 and 2011. Additionally, 1.2% of these patients did not have surgical procedures between 2010 and 2011, and only 39% of procedures had a patient associated to them in the analyzed data. All records with one of the aforementioned problems were excluded from the analysis.

Grouping of procedures. In order to reduce the number of different procedures to be analyzed and as suggested by an oncologist, a filter was applied based on the CUPS classification (Clasificación Única de Procedimientos en Salud). The CUPS, according to Resolution 1896 of 2001 of Colombia, are a logical, hierarchical and detailed classification of health procedures and interventions performed in Colombia, identified by a code and described by a unique nomenclature. The analyzed procedures in this work are related to the first three characters as follows: As in the precedent step, data quality-related problems were detected: (1) difficulty to validate the relationship between the procedure and the analyzed disease, (2) Data replication, and (3) reliability in the content of the attributes.

13.31% of records presented problems insofar as identifying a relationship between the procedure used and the disease analyzed. Approximately, 65% of procedures can be used for different health problems, affecting the precision in the analysis. Additionally, there are procedures related to the same patient on the same date, or a near date that are not possible. For example, complete stomach surgery followed by partial stomach surgery. Finally, the most important cases regarding reliability in the content of the attributes are related to time. In fact, the majority of records have 00:00:00.000 as their time value. All records with the aforementioned problems were identified and excluded.

Data envelopment analysis

Data Envelopment Analysis (DEA) is a non-parametric efficiency approach. It was developed by [6] and later elaborated by [7] (BCC model). DEA computes the relative efficiency of decision-making units (DMUs) with many inputs and outputs [8]. In the same way, DEA allows the quality assessment to include not only a set of performance indicators (outputs) but also the amount of consumed resources (inputs). As a result, DEA it is now considered mainstream to appraise the efficiency of health institutions [2]. Furthermore, Emrouznejad et. al in [9] concluded that DEA applications will continue to be a primary arena of research in the future. In operations management, DEA is used for benchmarking where a set of measures (indicators) is selected to estimate the performance of production and/or service operations, comparing multiple DMUs with a structure of multiple inputs and outputs. As a result, a set of DMUs that belong to a "best-practice frontier" [10], are identified. This frontier, allows us to calculate an efficient solution for every level of input or output. Any DMU not on the frontier is considered inefficient. A numerical coefficient is given to each firm, defining its relative efficiency. Where there is no actual corresponding firm, virtual efficient producers are identified to make comparisons [11].

Classical DEA models rely on the assumption that inputs have to be minimized while outputs have to be maximized [12]. However in health care, one or more outputs -called undesirable outputs-have to be minimized [13]. Even more, according to [14] considering such variables, in efficiency analysis, have paved the way for more thorough assessments. Nevertheless, modeling undesirable outputs has been object of considerable discussion in the efficiency literature, because of the lack of consensus about the most appropriated approach [3]. Even when variable transformations can be addressed to avoid this problem, [15] conclude that such transformations could generate loss of linearity. Authors also compare methods to deal with this situation.

Liu et al. [16] presented a survey of DEA applications from 1978 and 2010. According to the authors, health care is the second largest application area. Authors also state that most of the reviewed papers studied hospital performance. More recent papers studied the integration of DEA and complementary techniques to measure health care efficiency. As an example, Al-Refaie et al. [17] applied simulation and DEA to improve the emergency department of a Jordanian hospital. In this research DEA was used to identify the best possible scenario regarding nurses' workload. They concluded that using DEA to develop quality frontiers in health services is a new promising direction.

In recent years, several studies to evaluate health policies and health services has been developed. In Uganda, DEA was used to evaluate the efficiency of referral hospitals [18], as the same in Angola [19], Zambia [20], South Africa [21], among others. In Asia, recently [22] propose to evaluate the performance of maternal and child services in China hospitals using DEA, comparing poverty and non-poverty country hospitals. In Latin-America there are few studies that use DEA to evaluates performance in health institutions [23] propose DEA as new strategy to evaluate efficiencies in Chilean hospitals. In the best of our Knowledge there are not studies that allow us to evaluate the quality of a procedure in Colombia.

To evaluate the quality of gastric cancer treatments in different Colombian EPSs, a DEA model with non desirable output variables keeping the linearity between them are considered. After analyzing different models, we use the model proposed by [15], an output-oriented model with variable return to scale. The mathematical model is the following:
max ρ 0 (1) Subject to: n j=1 η j x ij + s − i = x i0 i = 1 . . . m (2) n j=1 η j y rj − s + r = ρ 0 y r0 r = 1 . . . s (3) n j=1 η j b tj − s + t = 2b t0 − ρ 0 b t0 t = 1 . . . T(4)
Let x, y and b be sets that represent input desirable variables, output desirable variables and output non desirable variables. Equation 1 is the objective function that looks to maximize efficiency considering the constrains over each set of variables, represented by Eqs. 2 to 4. This last set of equations, are used to assign weights to output and input variables and to estimate the distance of each EPS from the efficient frontier.

[24] Productivity Index was developed to measure changes in technological productivity over a period of time. This index was initially proposed by [24] and has been used in several studies with multiple variants such as [25,26]. In order to measure the changes of EPSs' efficiency over the observed period, we used the index used by [27], defined as follows:
MI = δ 1 (x 0 , y 0 ) 2 (δ 1 ((x 0 , y 0 ) 1 ) × δ 2 (x 0 , y 0 ) 2 (δ 2 ((x 0 , y 0 ) 1 ) 1 2 (5) Let δ ( t 2 )((x 0 , y 0 ) ( t 1 )
) be the efficiency of one DMU(x 0 , y 0 ) ( t 1 ) measured with respect to the technological frontier t 2 , and obtained from the results of the DEA model.

From the RIPS and based on health professionals' opinions, the following efficiency indicators were calculated for each EPS and used for the DEA analysis:

• Once the input and output variables were defined, the most (and least) efficient EPSs in the Colombian Health System were identified. Subsequently, the efficient frontier was computed using the calculated efficiencies of each institution. Finally, the Malmquist indicator for each institution was calculated to evaluate the change in efficiency in the study period.

Process mining

Process mining is a field of data mining that allows the discovery of business processes in a given domain using different types of algorithms. Examples of process mining algorithms include genetic algorithms, heuristics mining and sequential clustering, among others [28]. This work uses a ProM plugin that provide the Sequence Clustering Algorithm, a combination between sequence analysis (first-order Markov chain) and clustering. In particular, we decided to use sequential clustering, given its popularity in the process mining community, and information provided to interpret the results. In fact, sequential clustering algorithms generate a series of discrete states that are very similar internally [29]. These series, known as clusters are represented by a Markov chain consisting in states and transition probabilities between them. These probabilities depend only on the current state.

The use of process mining in this work allowed us to identify typical treatments used in patients by distinct EPSs to analyze various treatment patterns and, in turn, improve the quality of the health services. To do this, we had to carefully select the EPSs to be analyzed and the model to be used to do this, as well as prepare the data to execute the model, and evaluate the results.

This project uses two different EPSs: the most and least efficient according to DEA results. This selection allows a comparison to be made between these EPS to understand patterns affecting treatment quality. On the other hand, the design of the model consists in creating the sequential clustering model, and preparing the data used in the analysis related to procedures applied in treatments. It is important to note that a detailed granularity level of information used can increase the complexity of the results because of the number of procedures and relations between them. Finally, the evaluation, presented in the following "Evaluation" section includes the criteria of an o ncologist to validate the quality of the identified treatments.

Data models

The data model used in the study includes two entities that consolidate information from patients with gastric cancer and procedures used in the treatment of these patients. According to patients, the entity has a unique patient identifier, information about institutions related to the health service (EPS and IPS unique codes), date of the first diagnosis, main diagnosis of the consultation, and other three possible diagnoses related to the consultation. On the other hand, the entity where the procedure takes place holds information about the date of procedure, institutions related to the health service (EPS and IPS unique codes), patient identifier, procedure code (CUPS identifier), and the main diagnosis of the procedure.

The entity where the procedure takes place does not hold information on time, given the previously discussed data quality problem in this field. As a consequence, the identified paths using sequential clustering represent patients immediately operated on after the diagnosis, although, in reality, that is not the case.

Data aggregation

An aggregation strategy based on the CUPS was used in order to reduce the complexity of the models arising from the vast number of different treatments that a cancer patient can be subjected to and to facilitate analysis by the experts, according to procedure characteristics such as: type of procedure (diagnostic, radiotherapy, surgery) and affected organ (intestine, stomach). Table 2 shows the aggregation scheme proposed to characterize non-surgical and surgical procedures, presented in the CUPS section column. This schema has two levels to define the procedures, Level 1 and Level 2 columns in the Table. According to non-surgical procedures, Level 1 represents diagnosis, radiotherapy and chemotherapy procedures. Level 1 for surgical procedures represents diagnosis, pre and post operative and surgery procedures. Level 2 has more specific information about a Level 1 procedure. For example, non-surgical diagnosis procedures are classified in terms of radiography, tomography, clinical examination, derived examination and scintigraphy.

Finally, the percentage in brackets in the Table represents the percentage of procedures in each category used in this work. In this way, the information used contains 92.2% of non-surgical procedures.

Evaluation

The evaluation phase is carried out by a gastric cancer oncologist, according to whom, the quality of a given medical treatment depends on some of the features included in the patient's care pathway. Thus, the treatments administered to a patient with gastric cancer should include the following features:

• Will follow the treatment established in the clinical guide for the disease. • Will establish diagnostic procedures before and after a surgical procedure. • Will include procedures such as chemotherapy or radiation treatments. Furthermore, these procedures should be applied sequentially.

Results and discussion

The methodology described above, is tested using information from the reported RIPS in 2010 and 2011. The efficiency indicators are estimated for each EPS, for each year. Figures 1, 2   From the same figures, we can conclude that the variability of indicators decreased from 2010 to 2011, except in the case of "number of previous studies" (Fig. 2). Figure 5 shows how the treatment opportunity decreased from 2010 to 2011, reflecting the effort made for each EPS to diminish the time elapsed between the diagnoses and the beginning of treatment.

Using a mean differences hypothesis tests, with unknown variances, it is possible to say that there is no statistical evidence to affirm that there are differences in histological studies and in the number of readmissions to EPS. On the other hand, it is possible to reject the null hypothesis of average difference for treatment opportunity and previous studies indicators, which present improvements in the observation period.  Tables 3 and 4 respectively. For 2010, 27% of the EPS are considered efficient and 24% extremely inefficient (i.e., efficiencies below 60%). To become efficient, each EPS belonging to quantile 4 must improve its indicators by up to 80%. For example,  Considering the target values obtained for 2011, an overall improvement is observed: 20% of the EPS were efficient, and 37% of the EPS showed efficiencies below 60%. While in 2010, one "efficient EPS" had a treatment opportunity of 59 days and a 2.65 readmissions rate, for the same EPS, these values changed to 35 days and 1.53 readmissions. Finally, four of the efficient EPS in 2010, were also efficient during 2011. On the other hand, the Malquimst index allows for identifying the effect of EPS policies over efficiency. For the EPS 21, for example, efficiency increased from 0.67 in 2010 to 1 in 2011 (see Table 5). Target values show that, EPS21 must increase previous studies from 7 (2010) to 10.3 (2011), and decrease treatment opportunity from 445 to 233 days. However, in 2011, the EPS21 registers 55 studies and 11 days treatment opportunity. Since, the improvement in efficiency of EPS 21 is not due to changes in the frontier (given by the target values) but to its own policies, the Malquimst index is equal to 3.96. It is important to note that for one EPS with only one diagnosed patient, making changes of this magnitude in the indicators requires less investment than the one required for one EPS with 164 patients.

Process mining

This work used ProM 5.2 [30] as Process Mining tool, an open-source framework licensed under Common Public License (CPL), which allows the use of a variety of algorithms and friendly interface, in particular, sequential clustering. The sequential clustering needs to configure a number of parameters such as: minimum and maximum percentage of events that each cluster can have (assigned values were respectively 0, 100), minimum and maximum number of different events that a sequence can have (we use respectively 1 and 303), minimum and maximum number of sequences that a cluster can have (we use 1 to 102 values), and number clusters that the model should contain, we decide to use 1. It should be noted that the purpose of this work was to try to identify a general pattern of treatment for all the patients in a specific EPS. To do this, it was decided to use a single cluster for EPS to represent the entire treatment process.

According to "Methods" section, two types of EPS were chosen to study and to illustrate the methodology, using the efficiency scores obtained from the DEA process.

EPS 1:

This EPS is close to the efficient frontier and has a major positive change in the relative efficiency between the periods of study. 2. EPS 2: This EPS is well below to the efficient frontier and has a relatively low efficiency in the periods of study. Table 6 shows the number of patients, procedures and efficiency scores of the two chosen EPSs.

EPS 1 (Close to the efficient frontier)

Figures 6 and 7 demonstrate the typical treatment process for EPS 1 for each of the years studied. It is not possible to identify a general or distinctive path representing the treatment followed by patients in this EPS due to the complexity of the resulting model. This may be due to several reasons. First, a clinical guide for the treatment of gastric cancer may not be available or, this EPS may not follow it. In addition, it is also possible that the conditions during patient diagnosis are greatly varied, leading to a high number of possible treatments.

However, it is possible to discern paths in the treatment used by the EPS. For example, in the year 2010, the treatment usually started with either a non-surgical diagnostic procedure (75.9% of the time) or a surgical diagnosis procedure (18.7%). If a non-surgical diagnosis procedure was performed, usually an identical procedure was executed (82% of the time); then, the patient might have received chemotherapy treatment (6.5%) or a second operation (1.2%). Similarly, if a surgical diagnostic procedure was applied, typically a non-surgical diagnosis (74.6%) followed. Afterwards, the patient may have undergone chemotherapy treatment (2.5%) or a second operation (2.2%). Finally, when a patient underwent either surgery, chemotherapy or radiotherapy a diagnostic procedure normally followed.

Similarly, in the year 2011, the treatment usually started with either a non-surgical diagnostic procedure (71.8%) or a surgical diagnosis procedure (26.5%). If a non-surgical diagnosis procedure was performed, usually an identical procedure was executed (82%); then, the patient might have undergone chemotherapy treatment (6.3%) or entered into remission (3.4%). Likewise, if a surgical diagnostic procedure was applied, typically, a non-surgical diagnosis (66.3%) followed. Alternatively, the patient may have been subjected to chemotherapy treatment (3.3%) or a second operation (2.2%). Figures 8 and 9 exhibit the chemotherapy and/or radiotherapy cycle for each of the years studied. It should be noted that cycles of chemotherapy and radiotherapy were observed in the treatment of the patients, since 79.1% of those that received chemotherapy at least once, received chemotherapy again. The next most common procedure for the patients was radiotherapy, in 4.3% of the cases. Additionally, if a patient received radiotherapy, in 17.1% of the cases, she or he received another radiotherapy treatment and in 48.8% of the cases, the patient underwent chemotherapy.

EPS 1 diagnosed its patients in a timely, and effective manner, since its treatment traits are those identified  above as those of a high-quality procedure for gastric cancer. As a consequence, the position of the EPS on the efficient frontier is not surprising, given that the treatment of its users follows one of the most appropriate for such patients. Figures 10 and 11 show the typical treatment process for EPS 2 for each of the years studied. The resulting model is very complex as is the case for EPS 1. Hence, it is not possible to identify a distinct path representing the treatment of the patients, even though the number of patients was less than half of the patients for EPS 1.

EPS 2 (Far from the efficient frontier)

However, it is possible to discern particularities in the treatment that the EPS performed. For example, in the year 2010, 75.9% of the time the treatment started with a non-surgical diagnosis, 14.7% with a surgical diagnosis and 12% with chemotherapy or radiotherapy. If a nonsurgical diagnosis procedure was performed, usually an identical procedure was executed (87% of the times), and the patient probably ended her or his treatment (5.7%), 2% might have undergone chemotherapy or a second surgery (1.9%). Similarly, if a surgical diagnostic procedure was applied, typically a non-surgical diagnosis (64.2%) followed, after which the patient may have ended her or his treatment (15.1%), had a second surgical operation (11.3%) or received chemotherapy (5.7%). Finally, when a patient underwent either surgery, chemotherapy or radiotherapy a diagnostic procedure normally followed.

Similarly, during 2011, the treatment usually started with either a non-surgical diagnostic procedure (78.6%) or a surgical diagnosis procedure (14.3%). If a non-surgical diagnosis procedure was performed, usually an identical  procedure was executed (85.8%), then, the patient might have received chemotherapy treatment (2.2%) or a surgical diagnosis. Similarly, if a surgical diagnostic procedure was applied, typically a non-surgical diagnosis (64.2%) followed. Alternatively, the patient may have received chemotherapy treatment (5.7%) or a second surgery (11.3%).

Consequently, as we can see from the analysis of EPS 1, it is clear that EPS 2 performs more surgeries than its counterpart. Similarly, the likelihood of a patient undergoing a second surgery soars in comparison with EPS 1. However, it should be pointed out that in most cases, surgical and non-surgical diagnostics were performed before procedures such as surgeries, chemotherapy or  radiotherapy, showing that the treatment provided had at least a few desirable traits. Interestingly, unlike EPS 1, it was detected that 12% of the treatments began with a diagnosis that did not belong to the suggested clinical guidelines for the treatment of gastric cancer. Figures 12 and 13 present the chemotherapy and/or radiotherapy cycle for each of the years studied. It should be noted that cycles of chemotherapy and radiotherapy were observed in the treatment of the patients, since 49.2% of the patients that received chemotherapy at least once received chemotherapy again. The most common procedure for the patients was radiotherapy in 4.9% of the cases. Additionally, if a patient received radiotherapy, in 46.2% of the cases, the patient also received chemotherapy. It is noteworthy that none of the patients received radiotherapy in a consecutive, periodical manner.

In conclusion, the patients in EPS 2 have a higher probability of undergoing a second operation. This characteristic may be related to problems during the diagnostic step. Additionally, it does not apply radiotherapy consecutively and, in general, complies loosely with the available clinical guide. These elements can explain the position of EPS 2 in the efficient frontier.

Discussion

This paper proposes an innovative approach applying Data Envelopment Analysis (DEA), and Process Mining to evaluate efficiency and quality in the treatment of gastric cancer. The aggregate strategy proposed was very useful in reducing the complexity in the models. Data Envelopment Analysis has been used by other authors to analysis hospital efficiency in Zambia [20] and Iran [31]. Our results show variation between different indicators and treatments used by insurance companies and are similar to other studies that are more focused on clinical guidelines analysis [32]. Other studies combined DEA with Malmquist indices to measure hospital productivity [33]. There is limitation in the database used in this study and other authors recommended a level 3 data base (the most comprehensive) with information from health program enrollment files, including when eligibility begins and ends [34]. The analysis should consider factors such as complexity and current regulations and guidelines in the treatment of different types of cancer. The use of specific information about the disease analyzed such as the stage of the cancer, studies focusing on IPS, or studies related to less complex diseases are suggested for future work. Efforts should be aimed at designing objective metrics that differentiate between efficient and inefficient entities in terms of procedures. Analyses oriented to identify hospitals or doctor's behavior will be tackle using the same strategy. The analysis should consider factors such as complexity and current regulations and guidelines in the treatment of different types of cancer.

Conclusions

This paper proposes a combined strategy, applying Data Envelopment Analysis (DEA), and Process Mining to evaluate efficiency and quality in the treatment of gastric cancer undertaken by health service providers in Colombia. Results showed that the method used is a useful approach to understand and tackle the problem. The analysis of the treatment of gastric cancer allowed us to find relevant information on the factors affecting a treatment process performed efficient or inefficiently. However, the complexity of the disease studied and the diversity of the health system, make it difficult to confirm the contribution obtained using the combined strategy proposed. The use of specific information about the disease analyzed such as the stage of the cancer, studies focusing on IPS, or studies related to less complex diseases are suggested for future work. On the other hand, it is important to note that the data preparation phase is very important to understand the data and to correct data quality problems in order to improve the quality of the results. In the same way, the aggregate strategy proposed was very useful in reducing the complexity in the models. Finally, the generalized analysis of the treatment for gastric cancer (the procedures were grouped by their role in the treatment rather than being studied as individual procedures), allowed us to improve the knowledge of the clinical treatment of a typical gastric cancer patient. As future work, efforts should be aimed at designing objective metrics that measure the differences between efficient and inefficient entities in terms of procedures. Additionally, the proposed strategy must be extended so that a preliminary analysis is carried to select the disease. The analysis should take into account factors such as complexity and current regulations and guidelines in the treatment of different types of cancer.

, 3, 4 and 5 summarize those results. Each figure indicates the target value (for 2010 and 2011), the highest value (asterisk) and four quartiles.These figures show the high variability among EPS for each indicator.

Fig. 1 2012 Fig. 2 Fig. 4
1201224Number of patients. Years 2011 and Comparison between the number of previous studies and the target Fig. 3 Comparison between the number of histological studies and the target Comparison between the number of readmissions studies and the target

DEA
The proposed DEA model is implemented on Xpress-MP. The results of the efficiency and target values for each EPS in 2010 and 2011 are shown in

Fig. 5
5Comparison between the treatment opportunity and the target

Fig. 7
7Typical Care Pathways for EPS 1, year 2011

Fig. 8
8Chemotherapy / Radiotherapy Cycle for EPS 1, year 2010

Fig. 9
9Chemotherapy/Radiotherapy Cycle for EPS 1, year 2011

Fig. 10
10Typical Care Pathways for EPS 2, year 2010

Fig. 11
11Typical Care Pathways for EPS 2, year 2011

Fig. 12
12Chemotherapy/Radiotherapy Cycle for EPS 2, year 2010

Fig. 13
13Chemotherapy/Radiotherapy Cycle for EPS 2, year 2011

Abbreviations CHR: Committee on health regulation; CUPS: Clasificación Única de Procedimientos en Salud; DEA: Data envelopment analysis; DMU: Decision-making unit; EPS: Entidades Promotoras de Salud. ICD-10: International statistical classification of diseases and related health problems 10th revision; IPS: Instituciones Prestadoras de Servicios de Salud; MHW: Ministry of Health and Welfare; MI: Malmquist index; RIPS: Registros Individuales de la Prestación de Servicios de Salud; SOGCS: Sistema Obligatorio de Garantía de Calidad en Salud

Selected first-level group filter• Digestive System
• Imaging
• Consultation, Monitoring and Diagnostic Procedures
• Clinical Laboratory
• Transfusiology and Blood Bank
• Nuclear Medicine and Radiation Therapy
• Other nonsurgical procedures
• Miscellaneous Procedures

Selected second-level subgroup filters

• Transfusiology and Blood Bank
• Radiological Imaging
• Clinical Laboratory
• Nuclear Medicine and Radiation Therapy +
• Stomach Related Procedures
• Intestine Related Procedures
• Procedures in the Abdominal Wall
• Esophagus procedures
• Prophylactic, therapeutic and other miscellaneous
procedures

Table 1
1ICD-10 codes and number of patientsCode
Description

Input variables: -Number of patients: The total number of newly diagnosed patients with gastric cancer and, at least, one associated treatment procedure as classified by the EPS, for the period of study. A treatment procedure may be a surgical procedure associated with cancer, chemotherapy or radiotherapy. This indicator is taken as a desirable input variable for the model • Output variables -Treatment opportunity: The average number of days per patient between the time of diagnosis and the first procedure for the treatment of the disease, as classified by the EPS. This indicator is taken as a desirable input variable for the model. -Number of readmissions: The average number of readmissions per patient, as classified by the EPS, for patients who had at least one emergency readmission after undergoing surgery in the study period. This indicator is taken as an undesirable output variable. -Number of previous studies: The average number of diagnostic studies prior to the first diagnosis of cancer per patient as classified by the EPS. This indicator is taken as a desirable output variable. -Number of histological studies: The average number of histological studies per patient, as classified by the EPS. This indicator is taken as a desirable output variable.

Table 2
2Two-level aggregation schemeCUPS section
Level I
Level 2

Radiography (7,72%)

Tomography (5,06%)

Diagnosis (86,3%)
Clinical Examination (45,7%)

Non-surgical
Derived Examination (27,2%)

Procedures (92,2%)
Scintigraphy (0,55%)

Radiotherapy (0,5%)
Teletherapy and Therapy with radioisotope (0,5%)

Chemotherapy (5,3%)
Chemotherapy (5,3%)

Diagnosis (5,2%)
Biopsy (0,08%)

Cavity Exploration (5,1%)

Surgical
Pre-Post Operative (1,5%)
Pre-Postoperative (1,5%)

Procedures (8,2%)
Abdominal (0,08%)

Esophagus (0,17%)

Surgery (1,6%)
Stomach (1,18%)

Intestine (0,13%)

Table 3
3Results 2010 reduced from 54 days in 2010 to 22 days in 2011; the previous studies indicator changed from 15 in 2010 to 32 for 2011. These results show the need to establish expected minimum (or maximum) values for each indicator.For example, even if the treatment opportunity target value was reduced to 45 days, reflecting an improvement for the patients' health, this value is still higher than expected.ID
Efficiency
Patients
Previous studies
Histological studies
Re-admissions
Treatment opportunity

Observed
Observed
Target
Observed
Target
Observed
Target
Observed
Target

1
1
1
5
5
0
0
7
7
6
6

2
1
1
3
3
1
1
0
0
3 3
3 3

3
1
3
10,33
10,33
0
0
0
0
13,67
13,67

4
1
4
32,5
32,5
0,75
0,75
2,5
2,5
128,5
128,5

5
1
5
25
25
0
0
1,33
1,33
98,6
98,6

6
1
28
33,96
33,96
0,21
0,21
3,13
3,13
78,36
78,36

7
1
23
16,83
16,83
1,43
1,43
4
4
77,65
77,65

8
1
45
19,02
19,02
0,69
0,69
3,27
3,27
39,04
39,04

9
0,962
7
17,71
18,41
0,14
0,15
2,5
2,4
34,43
35,96

10
0,954
3
9
9,44
0,67
0,67
6
5,71
31,67
30,13

11
0,949
8
30,25
31,86
0
0
2,83
2,68
92
87,1

12
0,914
76
26,14
28,62
0,59
0,65
4,45
4,03
93,08
84,28

13
0,863
130
12,12
14,05
0,66
0,77
3,41
2,87
43,52
36,6

14
0,828
26
13,12
15,84
0,54
0,65
2,25
1,78
55,81
44,2

15
0,822
29
11,62
14,13
0,76
0,92
4,2
3,29
57,38
44,98

16
0,813
32
18,38
22,61
0,25
0,31
5,04
3,88
62,09
47,78

17
0,811
13
11,77
14,52
0,31
0,38
1
0,77
53,92
41,33

18
0,797
8
14,25
17,88
0,38
0,47
1,33
0,99
105,63
78,72

19
0,782
9
21
26,84
0,22
0,28
2,33
1,68
161,44
116,54

20
0,681
16
16,63
24,41
0,06
0,09
3,5
1,86
98,63
52,47

21
0,677
1
7
10,33
0
0
0
0
445
233,1

22
0,615
34
12,38
19,01
0,12
0,18
3,43
1,59
81,03
37,68

23
0,649
1
10
15,4
0
0
1
0,46
182
83,78

24
0,54
21
6,95
12,87
0,05
0,09
4,43
0,66
139,48
20,78

25
0,537
2
0
0
0,5
0,93
1
0,14
224
30,74

26
0,523
5
2,6
4,97
0
0
1
0,09
74,2
6,57

27
0,508
4
4
7,88
0
0
0
0
346,25
10,42

28
0,506
2
3,5
6,92
0
0
0
0
392,5
9,15

29
0,503
6
0,17
0,33
0
0
2
0,02
43,5
0,44

30
0,5
3
0
0
0
0
4,33
0
75,67
0

they have to reduce the treatment opportunity from 185
days to 11 days, and readmissions from an average rate
of 1.8 to 0.13. Finally, the results show that some EPS on
the best-practice frontier have worse indicators than the
ones that are fixed as target values for inefficient EPS. For
instance, EPS 3 has 13.67 days of treatment opportunity;
however, for 5 of the 7 inefficient EPS, target values are
below that value. This can be explained by the variability
among the indicators; while EPS 3 has a poor perfor-
mance at treatment opportunity, it has notable indicators
in readmissions and previous studies.
For 2011, the target values for all indicators were
improved: the treatment opportunity indicator was

Table 4
4Results 2010ID
Efficiency
Patients
Previous studies
Histologial studies
Readmissions
Treatment opportunity

Observed
Observed
Target
Observed
Target
Observed
Target
Observed
Target

1
1
1
9
9
1
1
1
1
3 2
3 2

21
1
1
55
55
0
0
0
0
11
11

28
1
1
36
36
1
1
1
1
18
18

6
1
7
74,86
74,86
0,29
0,29
2
2
38
38

7
1
19
24,84
24,84
1,32
1,32
2,8
2,8
73,21
73,21

8
1
32
26,25
26,25
1,22
1,22
2,4
2,4
36,56
36,56

23
0,9
2
2
2,22
0
0
1,5
1,33
0,5
0,44

17
0,888
14
42,21
47,54
0,71
0,8
2,25
1,97
37,93
33,17

3
0,846
3
13,67
16,15
0,67
0,79
2
1,64
17,33
14,18

13
0,822
164
20,26
24,65
0,84
1,02
2,81
2,2
25,52
20

18
0,803
2
47
58,51
0,5
0,62
6
4,53
82
61,92

15
0,765
15
18,47
24,16
0,84
1,13
3
2,08
42,4
29,34

30
0,705
8
42,63
60,46
0,13
0,18
1,33
0,78
48
27,92

12
0,679
112
35,82
52,74
0,47
0,7
4,37
2,31
54,95
29

9
0,64
21
23,9
37,35
0,29
0,45
2,8
1,23
28,1
12,29

5
0,634
5
42,2
66,56
0
0
4,5
1,9
63,2
30,72

20
0,633
34
15,88
25,1
0,68
1,07
3,32
1,39
97,47
40,92

14
0,604
45
26,33
43,58
0,38
0,63
1,9
0,66
54,51
18,82

26
0,6
3
13,67
22,78
0
0
0
0
13,67
4,56

19
0,594
8
27,5
46,29
0,38
0,63
4
1,27
58,88
18,66

11
0,588
3
17,33
29,5
0,33
0,57
4
1,19
40,33
12,03

29
0,567
5
10
17,65
0,2
0,35
1,5
35
34
8

4
0,563
4
8,25
14,67
0,25
0,44
2
0,44
85,5
19

27
0,563
8
10
17,78
0,13
0,22
1
0,22
64
14,22

22
0,562
29
14,38
25,57
0,14
0,25
7,44
1,65
35
7,76

16
0,557
32
24,66
44,3
0,09
0,17
13,16
2,67
52,56
10,68

10
0,533
5
0
0
0,2
0,38
3
0,38
73
9,13

25
0,529
11
6,45
12,2
0,18
0,34
5
0,55
56,18
6,19

24
0,528
4
13,25
25,08
0
0
2
0,21
46,75
5,02

2
0,521
1
9
17,26
0
0
0
0
42
3,45

Table 5
5Malmquist index    ID
Malquimst index
Efficiency

2010
2011

21
3,96
0,677
1

28
3,13
0,506
1

23
2,82
0,649
0,9

30
2,17
0,5
0,705

6
2 , 1 3
1
1

26
1,74
0,523
0,6

17
1,69
0,811
0,888

18
1,48
0,797
0,803

8
1 , 4 7
1
1

3
1,46
1
0,846

13
1,42
0,863
0,822

1
1 , 3 8
1
1

29
1,31
0,503
0,567

9
1,25
0,962
0,64

5
1,24
1
0,634

27
1,23
0,508
0,563

24
1,22
0,54
0,528

15
1,21
0,822
0,765

12
1,21
0,914
0,679

22
1,21
0,615
0,562

19
1,19
0,782
0,594

20
1,19
0,681
0,633

14
1,13
0,828
0,604

16
1,12
0,813
0,557

7
1 , 0 4
1
1

11
1,03
0,949
0,588

25
0,89
0,537
0,529

4
0,67
1
0,563

10
0,66
0,954
0,533

2
0
1
0,521

Table 6
6Number of patients, procedures and efficiency scores of the two chosen EPSNumber of patients
Number of procedures
Efficiency

2010 2011 Total
2010 2011 Total
2010 2011

EPS 1 242 350 592
7425 9253 16678
0,863 0,822

EPS 2 102 84
186
1566 1185 2751
0,651 0,562

AcknowledgementsWe thank the people of Ministry of Health and Welfare for giving access to the data to this study. We also thank the school of engineering of the Universidad de los Andes who funded this research.FundingThe school of engineering of the Universidad de los Andes funded this research.Availability of data and materialsThe database used to this research has a public access with login and password provided by the MHW (https://www.minsalud.gov.co). MHW demands previous training to access the data. This training gives the general rules to use in a proper way the data.Authors' contributionsMV was responsible for coordinating the entire Data Mining process: defining the technical problem, the Data Mining techniques, the toolkit used and the execution and evaluation of the model. DB defined the strategy used in DEA models. NV was responsible for coordinating all DEA analysis, supervising the execution of DEA models and carrying out the analysis. OB carried out an analysis of the importance of cancer from a public health perspective and provided technical assistance in the management of cancer protocols. CU was responsible for implementing and executing the data mining model. EF was responsible for the state of the art. SB was responsible for coding and executing DEA models. All authors read and approved the final manuscript.Ethics approval and consent to participateConsent to participate was not required for this research as the data was anonymised and available from a database with public access hosted by the Ministry of Health and Welfare.Consent for publicationThe published information is anonymous to the extent that the individual concerned (or anyone who knows them) cannot identify themselves from the published article. From Yoshida A, Dowa Y, Murakami H, Kosugi S. Obtaining subjects' consent to publish identifying personal information: current practices and identifying potential issues. BMC Medical Ethics. 2013 Nov 25;14:47.Competing interestsThe authors declare that they have no competing interests.Publisher's NoteSpringer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.• We accept pre-submission inquiries • Our selector tool helps you to find the most relevant journal Submit your next manuscript to BioMed Central and we will help you at every step:
Evaluating the Impact of Health Care Reform in Colombia: From Theory to Practice. A Gaviria, C Medina, C Mejía, Bogotá; Universidad de los AndesCEDE, Centro de Estudios sobre Desarollo Económico, Facultad de EconomíaGaviria A, Medina C, Mejía C. Evaluating the Impact of Health Care Reform in Colombia: From Theory to Practice. Bogotá: CEDE, Centro de Estudios sobre Desarollo Económico, Facultad de Economía, Universidad de los Andes; 2006.

The measurement of efficiency and productivity of health care delivery. B Hollingsworth, Health Econ. 17Hollingsworth B. The measurement of efficiency and productivity of health care delivery. Health Econ. 2008;17:1107-28.

Non-fetal Deaths. Vital Statistics. National Bureau of Statistics (DANEVital Statistics. National Bureau of Statistics (DANE). Non-fetal Deaths 2015. https://www.dane.gov.co/index.php/estadisticas-portema/ demografia-y-poblacion/nacimientos-y-defunciones.

Calidad en Salud en Colombia, Los Principios. Bogotá: Editorial Scripto Ltda. C A Kerguelén, Kerguelén CA. Calidad en Salud en Colombia, Los Principios. Bogotá: Editorial Scripto Ltda; 2008.

Institute of Medicine Committee to Advise the Public Health Service on Clinical Practice Guidelines. M J Field, K N Lohr, Washington DC; Washington DC National Academy PressClinical Practice Guidelines: Directions for a New ProgramField MJ, Lohr KN. Institute of Medicine Committee to Advise the Public Health Service on Clinical Practice Guidelines. Clinical Practice Guidelines: Directions for a New Program. Washington DC: Washington DC National Academy Press; 1990.

Measuring the efficiency of decision making units. A Charnes, W W Cooper, E Rhodes, Eur J Oper Res. 26Charnes A, Cooper WW, Rhodes E. Measuring the efficiency of decision making units. Eur J Oper Res. 1978;2(6):429-44.

Some models for estimating technical and scale inefficiencies in data envelopment analysis. R D Banker, A Charnes, W W Cooper, Manag Sci. 309Banker RD, Charnes A, Cooper WW. Some models for estimating technical and scale inefficiencies in data envelopment analysis. Manag Sci. 1984;30(9):1078-1092.

Handbook on Data Envelopment Analysis. W W Cooper, J Zhu, L M Seiford, Kluwer Academic PublishersNorwellCooper WW, Zhu J, Seiford LM. Handbook on Data Envelopment Analysis. Norwell: Kluwer Academic Publishers; 2004.

Evaluation of research in efficiency and productivity: A survey and analysis of the first 30 years of scholarly literature in dea. A Emrouznejad, B R Parker, G Tavares, Socio-Econ Planning Sci. 42Emrouznejad A, Parker BR, Tavares G. Evaluation of research in efficiency and productivity: A survey and analysis of the first 30 years of scholarly literature in dea. Socio-Econ Planning Sci. 2008;42:151-7.

Data envelopment analysis: Prior to choosing a model. W D Cook, K Tone, J Zhu, OMEGA. 44Cook WD, Tone K, Zhu J. Data envelopment analysis: Prior to choosing a model. OMEGA. 2014;44:1-4.

Water Utility Benchmarking: Measurement, Methodologies, and Performance Incentives. S Berg, IWA PublishingLondresBerg S. Water Utility Benchmarking: Measurement, Methodologies, and Performance Incentives. Londres: IWA Publishing; 2010.

Undesirable inputs and outputs in dea models. G R Jahanshahloo, F H Lotfi, N Shoja, G Tohidi, S Razavyan, Appl Math Comput. 1692Jahanshahloo GR, Lotfi FH, Shoja N, Tohidi G, Razavyan S. Undesirable inputs and outputs in dea models. Appl Math Comput. 2005;169(2): 917-25.

Efficiency assessment of primary care providers: A conditional nonparametric approach. J M Cordero, E Alonso-Morán, R Nuño-Solinis, J F Orueta, R S Arce, Eur J Oper Res. 2401Cordero JM, Alonso-Morán E, Nuño-Solinis R, Orueta JF, Arce RS. Efficiency assessment of primary care providers: A conditional nonparametric approach. Eur J Oper Res. 2015;240(1):235-44.

Measuring the efficiency of the Greek rural primary health care using a restricted DEA model; the case of southern and western Greece. N Oikonomou, Y Tountas, A Mariolis, K Souliotis, K Athanasakis, J Kyriopoulos, Health Care Manag Sci. 194Oikonomou N, Tountas Y, Mariolis A, Souliotis K, Athanasakis K, Kyriopoulos J. Measuring the efficiency of the Greek rural primary health care using a restricted DEA model; the case of southern and western Greece. Health Care Manag Sci. 2016;19(4):313-325.

Modeling Data Irregularities and Structural Complexities in Data Envelopment Analysis. Z Hua, Y Bian, Zhu JWDCSpringerNew YorkDEA with undesirable factorsHua Z, Bian Y. DEA with undesirable factors In: Zhu JWDC, editor. Modeling Data Irregularities and Structural Complexities in Data Envelopment Analysis. New York: Springer; 2007. p. 103-21.

A survey of dea applications. J S Liu, Lyy Lu, W M Lu, Bjy Lin, Omega. 41Liu JS, Lu LYY, Lu WM, Lin BJY. A survey of dea applications. Omega. 2013;41:893-902.

Applying simulation and {DEA} to improve performance of emergency department in a jordanian hospital. A Al-Refaie, R H Fouad, M H Li, M Shurrab, Simul Model Pract Theory. 410Al-Refaie A, Fouad RH, Li MH, Shurrab M. Applying simulation and {DEA} to improve performance of emergency department in a jordanian hospital. Simul Model Pract Theory. 2014;41(0):59-72.

How efficient are referral hospitals in Uganda? A data envelopment analysis and tobit regression approach. P N Mujasi, E Z Asbu, J Puig-Junoy, BMC Health Serv Res. 161230Mujasi PN, Asbu EZ, Puig-Junoy J. How efficient are referral hospitals in Uganda? A data envelopment analysis and tobit regression approach. BMC Health Serv Res. 2016;16(1):230.

A performance assessment method for hospitals: the case of municipal hospitals in Angola. J M Kirigia, A Emrouznejad, B Cassoma, E Z Asbu, S Barry, Journal of Medical Systems. 326Kirigia JM, Emrouznejad A, Cassoma B, Asbu EZ, Barry S. A performance assessment method for hospitals: the case of municipal hospitals in Angola. Journal of Medical Systems. 2008;32(6):509-19.

Investigating health system performance: An application of data envelopment analysis to zambian hospitals. F Masiye, BMC Health Serv Res. 71Masiye F. Investigating health system performance: An application of data envelopment analysis to zambian hospitals. BMC Health Serv Res. 2007;7(1):1-11.

Technical efficiency and productivity of public sector hospitals in three South African provinces*. E Zere, D Mcintyre, T Addison, S Afr J Econ. 692Zere E, Mcintyre D, Addison T. Technical efficiency and productivity of public sector hospitals in three South African provinces*. S Afr J Econ. 2001;69(2):336-58.

Evaluation of performance and impacts of maternal and child health hospital services using data envelopment analysis in guangxi zhuang autonomous region, china: a comparison study among poverty and non-poverty county level hospitals. X Wang, H Luo, X Qin, J Feng, H Gao, Q Feng, Int J Equity Health. 151Wang X, Luo H, Qin X, Feng J, Gao H, Feng Q. Evaluation of performance and impacts of maternal and child health hospital services using data envelopment analysis in guangxi zhuang autonomous region, china: a comparison study among poverty and non-poverty county level hospitals. Int J Equity Health. 2016;15(1):1-6.

A new strategy to evaluate technical efficiency in hospitals using homogeneous groups of casemix. M Villalobos-Cid, M Chacón, P Zitko, M Instroza-Ponta, J Med Syst. 404Villalobos-Cid M, Chacón M, Zitko P, Instroza-Ponta M. A new strategy to evaluate technical efficiency in hospitals using homogeneous groups of casemix. J Med Syst. 2016;40(4):1-12.

Index numbers and indifference surfaces. S Malmquist, Trabajos de Estadistica. 42Malmquist S. Index numbers and indifference surfaces. Trabajos de Estadistica. 1953;4(2):209-42.

Benchmarking and incentive regulation of quality of service: an application to the UK electricity distribution networks. D Giannakis, T Jamasb, M Pollitt, Energy Policy. 33Giannakis D, Jamasb T, Pollitt M. Benchmarking and incentive regulation of quality of service: an application to the UK electricity distribution networks. Energy Policy. 2005;33:2256-71.

Combining dea window analysis with the malmquist index approach in a study of the canadian banking industry. M Asmild, J Paradi, V Aggarwall, C Schaffnit, J Prod Anal. 21Asmild M, Paradi J, Aggarwall V, Schaffnit C. Combining dea window analysis with the malmquist index approach in a study of the canadian banking industry. J Prod Anal. 2004;21:67-89.

Malmquist Productivity Index: Efficiency Change Over Time. K Tone, Handbook on Data Envelopment Analysis. Cooper WW, Zhu J, Seiford LMNorwellKluwer Academic PublishersTone K. Malmquist Productivity Index: Efficiency Change Over Time In: Cooper WW, Zhu J, Seiford LM, editors. Handbook on Data Envelopment Analysis. Norwell: Kluwer Academic Publishers; 2004. p. 203-27.

Wmp Van Der Aalst, Discovery, Conformance and Enhancement of Business Processes. Berlin HeidelbergSpringervan der Aalst WMP. Discovery, Conformance and Enhancement of Business Processes. Berlin Heidelberg: Springer; 2011.

Data Mining Whit Sql Server. Z Tang, J Maclennan, B Crivat, Wiley Publishing IncIndianapolisTang Z, MacLennan J, Crivat B. Data Mining Whit Sql Server 2008. Indianapolis: Wiley Publishing Inc.; 2008.

. P Tools, ProM 5.2Tools P. ProM 5.2. http://www.promtools.org/doku.php?id=prom52. Accessed 17 June 2015.

The efficiency assessment of dental units using data envelopment analysis approach: The case of Iran. M Barouni, M Amiresmaieli, A Shahravan, S Amini, Iran J Public Health. 464Barouni M, Amiresmaieli M, Shahravan A, Amini S. The efficiency assessment of dental units using data envelopment analysis approach: The case of Iran. Iran J Public Health. 2017;46(4):552-559.

Agency for healthcare research and quality, National quality measures clearinghouse, Tutorials on quality measures: Selecting structure measures for clinical quality measurement. Agency for healthcare research and quality, National quality measures clearinghouse, Tutorials on quality measures: Selecting structure measures for clinical quality measurement. https://qualitymeasures.ahrq. gov/browse/domain.

Hospitals productivity measurement using data envelopment analysis technique. A Torabipour, M Najarzadeh, M Arab, F Farzianpour, R Ghasemzadeh, Iran J Public Health. 4311Torabipour A, Najarzadeh M, Arab M, Farzianpour F, Ghasemzadeh R. Hospitals productivity measurement using data envelopment analysis technique. Iran J Public Health. 2014;43(11):157-1581.

Medicare: A Strategy for quality assurance. Medicare: A Strategy for quality assurance, volume I. https://www.nap. edu/read/1547/chapter/15.