Hongya Wang,Caspar Daniel Adenutsi,Can Wang,Zheng Sun,Yue Zhang,Yuxin Li,Yiping Zhang,Jiahuan Wang
Construction and Application of a Big Data System for Regional Lakes in Coalbed Methane Development
With the rapid development and widespread application of big data and artificial intelligence, the upgrading of digital and intelligent industries has been rapidly popularized in the oil and gas industry. First, based on the theory of ″regional data lake″, the digital nature of the CBM governance system is analyzed, and the optimization model of CBM governance for different data types is established. Second, considering the geological characteristics and development mode of the CBM reservoir, the regional data lake expansion model is established. Third, a theoretical model of coupling ″on-site data, laboratory data, management data, and data management system″ has been established. The research shows the following: (a) The CBM governance system based on the regional data lake can be divided into four parts: basic support, data life cycle, core governance areas, and governance strategy support. (b) The coupling of the coalbed methane governance model with the BP neural network model in this article has good application results. (c) The computational efficiency of this model has been improved by 12%, which has broad application prospects.
Construction and Application of a Big Data System for Regional Lakes in Coalbed Methane Development

With the rapid development and widespread application of big data and artificial intelligence, the upgrading of digital and intelligent industries has been rapidly popularized in the oil and gas industry. First, based on the theory of ″regional data lake″, the digital nature of the CBM governance system is analyzed, and the optimization model of CBM governance for different data types is established. Second, considering the geological characteristics and development mode of the CBM reservoir, the regional data lake expansion model is established. Third, a theoretical model of coupling ″on-site data, laboratory data, management data, and data management system″ has been established. The research shows the following: (a) The CBM governance system based on the regional data lake can be divided into four parts: basic support, data life cycle, core governance areas, and governance strategy support. (b) The coupling of the coalbed methane governance model with the BP neural network model in this article has good application results. (c) The computational efficiency of this model has been improved by 12%, which has broad application prospects.

At present, various petroleum enterprises are developing toward digitization and intelligence. Among them, PetroChina has launched its independently developed digital technology platform—Dream Cloud Platform, which has enabled the rapid development of petroleum business informatization. At present, the main data lake of the headquarters has been completed. The main data lake of the headquarters is established based on the Exploration and Production Data Model of PetroChina, responsible for collecting data required by various oilfield subsidiaries. Relevant standards and management plans for upstream data are developed. To this end, each oilfield branch is deploying and building a regional data lake. The regional data lake is expanded on the basis of the main data lake of the headquarters. By building a regional data lake with a unified architecture with the main data lake of the headquarters, it provides data for the main data lake of the headquarters and various data services for local branches. The concept of data lake was first proposed by James Dixon, the founder of Pentaho. It regards data lake as a unified storage mode. Moreover, this approach only defines the data structure when applying data and does not define the data type. Based on this feature, the data lake can store all kinds of data in a unified way. Data structure definition is implemented only during data application. There are multiple ways for data storage in a data lake and data acquisition in data application. In recent years, the oil exploration industry has also begun to realize the construction and development of the data lake. Since 2018, PetroChina has completed the first phase of the ″Dream Cloud″ project. This project is mainly composed of the data source, data lake, service middle platform, and application premise and portal. Among them, the construction of data lake related to oil exploration is also carried out together with the ″Dream Cloud″ project. In 2019, all subsidiaries of PetroChina began to implement the construction of data lake. This situation provides important convenience for data exchange and integration in the oil exploration industry. On the one hand, the data information in the oil exploration of each subsidiary of PetroChina can be stored in the data lake to ensure the data interaction and integration effect and provide important data support for each subsidiary. On the other hand, the data lake built by each branch company is connected with the data lake of CNPC headquarters, and the data lake of each branch company is the source of the headquarters’ general lake, thus achieving the effect of ″converging small into large″ and further strengthening the application effect of the data lake in the oil exploration industry. Since the concept of ″data lake″ was proposed, the data integration of the data lake has never stopped developing. Many companies have proposed different data integration solutions for it. Huawei Cloud data lake solution is based on the advanced cloud system architecture and provides the data lake Query Analysis Service (DLI). As a Spark-based data integration and exploration service of the Huawei Cloud data lake, DLI solves the following difficulties when users use data: (a) Traditional use of big data requires tedious steps and has certain requirements for the objective environment. This includes the configuration of computer rooms, networks, clusters, etc., followed by the installation and application of relevant system platforms. Technical personnel should carry out timely maintenance work on the equipment during use. These have high requirements for information platform technology and heavily rely on platform technology, with a high threshold. In addition, as the amount of data increases, it is difficult to expand the existing cluster. DLI is fully hosted, allowing users to use standard SQL for data query and analysis without maintaining any infrastructure. This has saved a lot of manpower and time costs for the enterprise. (b) The original data types are diverse and have different formats, such as CSV, TXT, JSON, Parquet, ORC, and other formats. In addition, data sources are also diverse, including relational MySQL, Kafka, Hbase, and other data sources. DLI is fully compatible with the Spark ecosystem, naturally supporting multiple data sources and formats. In addition, based on Huawei Cloud’s unique extension of data sources, the joint query function between data sources allows users to directly use SQL to search for multiple data sources on the cloud. (c) Enterprises face the problem of difficulty in sharing data between different departments or users. DLI enterprise-level multitenant management allows users to choose to configure resources, including CPU and memory. Different users will choose different contents to ensure that assignments are not affected by each other. At the same time, users can perform more detailed permission classification and usage during the usage process. It can reduce options more quickly and improve efficiency. The security of data access between various departments within the company should be ensured. So that data can be better utilized and mastered by people. The control of different levels and permissions should be ensured. (d) Enterprise business systems need to face various business application requirements, such as joint queries from multiple data sources, BI analysis, machine learning, and so on. Through cross source data-federated queries, without the need for loading or relocation, standard SQL can be used to perform federated query analysis on heterogeneous data sources on the cloud. Interactive multidimensional analysis, with rich report components, supports real-time scheduled tasks and achieves terabyte-level data with second-level response. The current ″data resource catalog″ is the result of the evolution of different concepts established by companies for data documents and data provision since the early 1960s when data processing began. To this day, data documents are usually associated with metadata. Metadata is often defined as ″data about data″ and ″aimed at facilitating access, management, and sharing of large amounts of structured and unstructured data″. All of this started with field names (in the 1960s) and table definitions (in the 1970s). Before the emergence of the system-specific data dictionary in the 1980s, these data dictionaries were used for basic technical documentation of database tables. Over time, it is necessary to plan data architecture and data integration more carefully and link them to business requirements because it has become increasingly important. In order to analyze business data and facilitate decision-making, the company has established a data warehouse for centralized storage of business data. In addition, they began to record these business data in the metadata repository. Subsequently, they launched the Business Data Dictionary, which is an extension of a technology-oriented data dictionary and an answer to the increasing distribution of data storage brought about by cloud technology in the market. Defining semantics and allowing users to correctly interpret different types of data implements use case and document methods using business terminology. With the increasing interest of data resource catalogs among data managers, market research analysts have expanded their definition to include users and usage contexts. The catalog provides context that enables data analysts, data scientists, data administrators, and other data consumers to find and understand relevant application datasets to extract the business value. This definition emphasizes the discovery and understanding of application datasets by different user groups and does not reflect how to obtain data and control access. Given the continuous growth of data, access control has become increasingly important. At present, various oilfield enterprises have deployed and operated a large number of information systems. Each system is independently developed and released, forming numerous information silos. Due to the lack of unified standard descriptions and information services for data resources, it is difficult to collaborate among various systems and the efficiency of data development and application is low. It is unable to form efficient public data service products. In order to shield these information silos, various oilfields began to integrate data. Independently developed information systems are correlated to enable users to access these data transparently. And data mapping is an important step in data integration. Currently, many data integration methods are used to address heterogeneity in the data structure, but they do not effectively address semantic heterogeneity in data. This has always been a difficult point. In summary, the study on the optimization method of CBM governance for different data types is relatively limited, and there is a lack of systematic study on the regional data lake expansion model with consideration of CBM reservoir properties. In this paper, first, based on the theory of ″regional data lake″, the digital nature of the CBM governance system is analyzed, and the optimization model of CBM governance for different data types is established. Second, considering the geological characteristics and development mode of the CBM reservoir, the regional data lake expansion model is established. Third, a theoretical model of coupling ″on-site data, laboratory data, management data, and data management system″ has been established.

CBM is a regional data lake that realizes the input of professional data such as CBM single wells, analysis and testing, geologic gas reservoir, and production management to the lake. Data collection techniques based on the EPDM 2.0 storage model, the EPDMX sharing model, and the multidimensional analysis model of the China National Petroleum Headquarters allow structured, unstructured, and time-series data to be stored in a classified way, unified warehousing and unified management. Simultaneously, the EPDM 2.0 model was modified and expanded to meet the needs of the CBM industry. The self-constructed and unified databases were categorized and analyzed and compared with the new requirements. Cross-firm data were also split, merged, and consolidated to make the model structure more reasonable and responsive to the firm’s business needs. CBM master data are stored in a relational PostgreSQL (PG) database. Based on the MPP architecture, the structured data are stored in a hashed database. Unstructured data are stored in EPDMX-based document index information. Data from the document body are stored through S3 objects. Time-series data such as dynamic monitoring techniques and detection of injection wells are stored in a time-series database. The underlying environment of the database is the Hadoop technology framework. Currently, there are seven types of structured data stored in the regional lake, including drilling, logging, gas testing, downhole operations, production, and analysis and testing, among others. Of these, 3446 wells have been drilled, 2131 wells have been logged, 3530 wells have been tested, 189 wells have been tested for gas, 2981 wells have been operated, 22 wells have been produced, and 95 samples have been analyzed and tested. A total of 4215 wells and more than 3.6 million structured data were input to CBM RL. So far, the E&P structured data rate is 50%. 16.5 GBs of unstructured data have entered the Lakes such as geophysical surveys, analytical tests, and geologic gas reservoirs. The ecological basis of CBM data has been effectively established in order to provide robust data support for the transformation of the company’s digital intelligence.

Tracy Energy Technologies’ low-code data science workbench, which is based on data lake technology, deals with a range of issues such as big data storage, data governance, data mining, and the valuation of data assets. The CBM data governance scenario derives from the data management needs. Data standards, primary data, data quality, and data services form the core, based on the processes of the data life cycle. Organizational, system, process, and technological tools provide strategic support and assurance, as illustrated in Figure 1. The governance system framework consists of four levels from the bottom up.

The software side of a business data architecture system is established. Data governance was supported by the development of tools for data modeling, master data management, metadata applications, and data integration.

The data life cycle involves the following: introduction of data quality control mechanisms based on real needs, formation of a global data governance system in terms of data sources, data processing, data storage, and data application.

The development of appropriate management measures has led to the development of a data management culture with a focus on organizational management and communication management. A data governance core is formed based on data standards, master data management, data quality, and data services in order to drive the proper functioning of the overall governance system.

Governance data are supported in a strategic sense, from the promotion of the higher level organization to the implementation of the system and the development of a variety of related processes. Building on the data governance system framework above a technical data governance framework for exploring and producing CBM is established around data collection, data governance, and data services. Over the entire life cycle of data creation, acquisition, storage, maintenance, and servicing, an organizational guarantee mechanism and a data guarantee system are formed, as illustrated in Figure 2. Data organization assurance mechanism includes establishing the enterprise data strategy, improving data organization and the management structure, clarifying data management responsibilities, optimizing the data management process, improving data evaluation, data standards, and data models. Data collection is the collection of a variety of data in the lake for completeness, accuracy, and timeliness of data collection; data governance refers to control over data storage, data processing integration, and data quality. Processes to realize the framework of the data governance system for CBM exploration and production and optimization of data resources are provided. The data service provides processed and cleaned data that show the effect of data processing and is better in the application.

The BP neural network is a feed-forward multilayer neural network trained on the error back propagation algorithm. BP neural networks can learn and store various mapping relationships between inputs and outputs without the need to define the relevant data rules in advance. In the BP network model, the topology consists of one input layer, one hidden layer, and one output layer. Figure 2 shows that the hidden layer plays a very important role in the structure of the three layers. Each hidden layer is equivalent to a feature extractor, which can extract features from the data and compute weights, where the number of hidden layers is determined by the features of the input and output data vectors. Once the BP neural network structure has been determined, the data from the sample set is fed to the network for training and the rules are automatically extracted from the sample data by means of the neural network generalization function. In this study, the defined and processed feature vectors are used as training data for the BP neural network. The training data for the BP neural network must contain the corresponding outputs in advance, and the format of the output of the network must be defined. For the purposes of this paper, in BP’s corresponding neural network structure, the number of input nodes to the neural network is N = 12, and the corresponding outputs have eight categories. In other words, the number of exit nodes is M = 8. The BP neural network’s pattern matching structure is set up as shown in Figure 2. In the input layer, the number of nodes N is the number of features of the input data, and each input node represents an indicator of an attribute. Where the number of nodes in the output layer M denotes the number of categories that need to be output. The middle hidden layer is defined as (N + M)/2.

There are two types of data governance: wide data governance and narrow data governance. Broad data governance includes a systematic framework based on data standards, master data, data quality, and data services. Data governance tools bring the value of information departments, business authorities, implementation departments, and other organizations at all levels into the picture. A spiral work system is presented based on the overall evaluation and application needs of the data as governance outcomes are continually improved. The central idea is to use in order to promote governance. It is important to ensure that historical data are ″paid off quickly″ and new data are ″not owed new debts″. In this study, we propose a solution for the governance of regional lake data within energy company infrastructure. Big data governance is at the heart of the solution. Figure 3 shows the dependence of the consensus layer on kernel layer functions in this scheme. In a narrow sense, data governance refers to the unified solidification of data standards and master data in a software system through the management of master data and metadata during the building of regional lakes. The system also provides minimal data quality checks during data entry in the lake, including nonempty master data, connectivity among the data, and so on. The main regional lake building project currently being undertaken by CBM is broad data governance. At the same time, a problem list was formed for the problem data and a closed loop process was developed from sorting out problems with data entering the lake to solving problems with the operational personnel. The project of data governance in the narrow sense has been realized at the outset. With the continuous integration of regional lake data management and exploration and development, the integrated management system will become more systematic and scientific and play a more effective role.

The master data governance of CBM is a broad data governance, which is implemented and managed in four main areas: data standards, master data, data quality, and data services. The data standards work to establish a consistent data model that provides consistency and accuracy in both internal and external data usage and exchange. They also regulate the consistent definition and application of business objects in a standardized form. These include data model building methods and software tools applied to satisfy data model harmonization, i.e., data model management tools. The use of data model management tools solidifies a unified data model standard in the information system. In order to unify data standards, the various information systems are then distributed, controlled, and optimized according to certain management requirements and rules. In the real world, data models are the global abstract representation of storage management in information systems, including conceptual models, logic models, and physical models. The data model is used to communicate between CBM information systems. In order to complete the construction of the regional CBM lake, the first task is the construction of data standards. Based on existing E&P data standards at the CNPC headquarters and in combination with the unique needs of unconventional gas, the current status, needs, and recommendations for the construction and application of the system were advanced by the business department. Based on the workflow of combining business processes, collating process data tables, confirming data items and building data models, and considering the special needs of the CBM unconventional gas business, the business unit focused on 10 specialties such as physical core exploration, drilling, logging, well testing, and gas testing. This includes downhole mining, analysis and testing, production, gas recovery processes, and geologic gas reservoirs. Improvement and validation of business processes, validation of process data sheets, etc. were conducted. Full comparison, analysis, collation, and refinement were performed. A feature data model standard covering CBM activity was trained based on existing PetroChina data standards as shown in Table 1. A master data system is the data that describes an organization’s core business entities. These data have high commercial value and can be reused between systems. Master data management focuses on ensuring the uniqueness and coherence of business objects across systems and data sources. Master data link the professional activity data horizontally and the business application scenarios vertically. The basic characteristics of master data are as follows: Beyond departments: Master Data is a database that addresses the needs of cross-departmental business collaboration. These are the data required by all of the functions within a business process. This is the ‘maximum agreed-upon data’ for all functions and their operational processes rather than restricting the database to a particular function. Beyond the system: The information system is based on the master data management system. It retains a relatively independent status. The company provides services, albeit at a higher level than other business information systems. As a result, master data management should be centralized, systematic, and standardized. Apart from the process: The master data does not depend on a particular business process. These are the data needed for all of the major business processes. The basic idea is that master data reflects properties of an object’s state. They do not change with any specific process but act as an invariant part of its complete process. Over themes: In information engineering methods, the master data are different from the subject data chosen by clustering methods. This category is for basic information about business entities that are not dependent on a specific business topic but serve all business topics. Over and above technology: The master data needs to accommodate cross-departmental business collaboration and adapt to different business systems with different technical specifications. It is necessary to apply technical conditions that allow compatibility with a variety of heterogeneous systems. From this perspective, a microservice-oriented architecture provides an efficient tool for implementing master data. Master data creation includes master data entity analysis and master data attribute analysis. Based on the five steps of business domain division, entity object identification, sorting of entity–business domain relationships, confirmation of master data relationships, and master data modeling, the abstraction and clustering are performed according to the principle of ″sharing, stability, and control″ as shown in Figure 4. Sixteen categories of master data such as well, personnel, contract, geologic unit, physical exploration work area, material and equipment, production layer section, mining right, and organization have been sorted and confirmed for the construction of the CBM regional lake based on the master data construction method noted above. Priority has been given to the construction of basic master data such as organization, geological units, wells, shafts, engineering, and physical exploration areas. Authoritative CBM master data that are standardized, complete, accurate, consistent, and controllable are shared and exchanged within the data system as a service via the master data management tools. The verification policy for each control point is tailored to the basic characteristics of the operational data. New verification rules are added in a timely fashion, and verification rules with low reliability are removed in order to make the verification policy better suited to the characteristics of the data to be verified, as shown in Figure 5.

A data service case study based on the core data governance framework of standard validation, master data unification, data quality inspection and data services is done. Data governance has been carried out successively by the PetroChina Coalbed Methane Company on the basis of the construction of a regional data lake. Taking the professional data of analysis and assay as an example, based on the data standard of analysis and testing, the data governance work is carried out combined with coalbed methane business. In this paper, the performance of the proposed ontology-based data integration algorithm with a BP neural network model is analyzed. In the process of ontology matching, the BP neural network and dynamic adaptive method are used to match the experimental dataset. Furthermore, the accuracy of different algorithms is compared. According to the experimental results, the overall performance of the ontology-based matching algorithm in this paper is ideal. To check the performance of the proposed BP neural network algorithm in a more intuitive way, the proposed algorithm is compared to some existing type matching schemes. These matching systems rely on a variety of ontology matching techniques with good performance (e.g., AML, XMap) data. Figure 6 compares the accuracy tested between the proposed algorithm and the other algorithms. This indicates that the accuracy performance of the algorithm is similar to that of the other algorithms, which shows that the method is reliable. In addition, as can be seen in Figure 6, the method of ontology matching embedded in the BP neural network in this paper has the highest prediction accuracy. The BP neural network-based ontology matching algorithm can be applied to local ontology matching tasks. The use of BP neural networks for attribute recognition offers great convenience for attribute matching. Establishing attribute mappings via BP neural networks, however, is not the ultimate goal but rather the analysis of associations via local ontology matching. This work provides theoretical support to establish the correct mapping relations for local ontologies and merge them into the global ontology. Local ontology matching and attribute matching based on neural networks are designed to perform global ontology fusion. Ontology fusion is the analysis of two or more ontologies with the goal of identifying relevant concepts, attributes, and differences between them. In this paper, we use a BP neural network model to build local ontologies. Computing the concepts and attributes in different ontologies and combining features from different heterogeneous ontologies allow the merging of two or more ontologies into one unified ontology. Such ontologies can express domain knowledge globally. We also compare the execution times of different algorithms. Overall, the results show that the proposed BP neural network-based method has the highest runtime efficiency (Figure 7). Computational results show that the method has an improvement in runtime efficiency of 12% (Figure 8). The method thus has a broad developmental perspective.

Taking CBM as an example, this paper introduces the detailed application method of the data lake. The data lake still has broad application prospects in other fields of the petroleum industry, and the reasons are as follows: (a) Building a master data management module: With the deepening of the construction of the data lake, the petroleum exploration industry is gradually trying to use the advantages of the data lake to carry out data governance. The master data management module of the data lake is mainly used to manage the master data information of the data lake. In the specific data governance process, the master data management module must be constructed first. When the master data management module manages the data lake, it can effectively associate the master data in the database with the master data in the data lake by building a mapping table. The uniformity of data information in the data lake should be ensured, and the data quality should be improved. Meanwhile, to ensure data security, all data need to be audited and authenticated by the main data management module. ″Multiple sources″ of data lake should be avoided. (b) Building an ID synchronization record mechanism: When data are stored in the data lake, the data lake will generate special fields to record the source, ID, and other basic information of data. The fact that, after identifying the problem situation, the problem data can be traced in a timely manner, thereby achieving rectification at the source of the data problem, should be ensured. The accuracy of data information in the data lake should be ensured. At the same time, detailed records need to be kept during the data rectification process, and the rectification record process can only be archived and deleted after the relevant data is stably applied for a period of time. (c) Building integrated models and data maps: The main purpose of the data lake is to eliminate information silos and avoid repeated entry. However, if we want to give full play to the relevant advantages of the data lake, we need to build an integrated data model and data map. In fact, the process of building the data integration model needs to be based on the master data of the data lake. Various data models within the existing system should be reasonably organized. Furthermore, a unified storage and management model for attribute specification values that can cover all system data types is proposed. Unified and standardized management of data information should be realized. Data map construction needs to be reasonably determined and recorded according to the data flow direction and data relationship in the data lake. The fact that the data map can effectively map the flow direction of all data information, passing nodes, and the mapping relationship between the data source and the data lake should be ensured. Users should be helped to analyze data sources more quickly. Various system operation problems caused by data changes should be solved.

In this paper, first, based on the theory of ″regional data lake″, the digital nature of the CBM governance system is analyzed, and the optimization model of CBM governance for different data types is established. Second, considering the geological characteristics and development mode of the CBM reservoir, the regional data lake expansion model is established. Third, a theoretical model of coupling ″on-site data, laboratory data, management data, and data management system″ has been established. The key findings are listed below: (1) Given the differences between the coalbed methane exploration and development enterprise and the needs of the traditional oil and gas field enterprises, in the context of PetroChina’s master lake model, the regional data lake model was extended accordingly, forming a regional data lake structure model with features of coalbed methane. (2) The core framework of the CBM’s regional data lake governance system consists of four aspects: foundation support, data life cycle, core governance areas, and support for governance strategies. CBM data governance systems of this type are dominated by broad data governance, with some data implicated in narrow areas of data governance. Extended data governance encompasses the technical framework of data standards, master data, data quality, and data services. In a narrow sense, data governance refers to the modification of problem data during RL implementation. (3) This article discusses the connotations of the four aspects of the data governance technology framework of CBM. In this paper, we focus systematically on cleaning and studying various types of CBM data and practical data lake governance. Systems and processes of data governance have been applied to the field of data management for the purposes of analysis and testing. (4) When combined with the BP neural network model, the ontology-based data integration algorithm proposed in this paper has a good performance. Computational results show that the method’s operating efficiency is improved by up to 12%. The method thus has a broad developmental perspective.