CN115729998A - Large-scale processing and analyzing system for arbitrary data hybrid optimization - Google Patents
Large-scale processing and analyzing system for arbitrary data hybrid optimization Download PDFInfo
- Publication number
- CN115729998A CN115729998A CN202211478698.9A CN202211478698A CN115729998A CN 115729998 A CN115729998 A CN 115729998A CN 202211478698 A CN202211478698 A CN 202211478698A CN 115729998 A CN115729998 A CN 115729998A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- submodule
- interpretation
- parameter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Computer And Data Communications (AREA)
Abstract
The invention discloses a large-scale processing and analyzing system for arbitrary data mixing optimization, and relates to the field of data large-scale processing and analyzing systems. The method comprises a data acquisition module, a data conversion module, a data processing module, a learning module, a data management module and a data interpretation analysis module, wherein the parameter field of a source file is determined to be in a first data format and the parameter field of a target file is in a second data format by setting the corresponding relation between the parameter field of the source file and the parameter field of the target file, a format conversion protocol corresponding to each first data format is generated according to the similarity between the parameter field of the source file and the parameter field of the target file, each format conversion protocol is issued to the corresponding data conversion module, the data of the parameter field of the source file is matched to the parameter field corresponding to the target file according to the format conversion protocol, the data integration difficulty of the system facing multi-source heterogeneous mass data is reduced, and the analysis efficiency is improved.
Description
Technical Field
The invention relates to the field of data large-scale processing and analyzing systems, in particular to a large-scale processing and analyzing system for arbitrary data mixing optimization.
Background
Data is the result of facts or observations, is a logical generalization of objective things, and is raw material used to represent objective things. The data may be continuous values, such as sound, images, referred to as analog data; or discrete, such as symbols, characters, called digital data; in a computer system, data is represented in the form of binary information elements 0, 1.
A "multi-source mass data processing system and method" disclosed in publication No. "CN108427709B", said system comprising a calculation module and a task management module, wherein: the computing module is used for receiving the multi-source mass data and calling data receiving service to analyze the multi-source mass data; the multi-source mass data are open source data generated according to tasks configured in the task management module in advance; and if the computing module receives a confirmation action of the preset model selected by the outside, the analyzed multi-source mass data is input into the preset model so as to analyze an output result of the preset model. The method uses the system. According to the multi-source mass data processing system and method provided by the embodiment of the invention, the multi-source mass data is analyzed by calling the data receiving service, the analyzed multi-source mass data is input into the preset model, and the multi-source mass data is analyzed according to the output result of the preset model, so that the enterprise system is enabled to be efficiently compatible with the multi-source mass data, and the multi-source mass data can be effectively utilized for data analysis.
With the development of big data technology, the data sources are more and more extensive, and the existing processing and analyzing system is difficult to integrate data and low in analyzing efficiency when facing multi-source heterogeneous mass data; and different service scenes need to be coded based on services, when the services slightly change, a series of operations such as corresponding requirement evaluation, design, development, online and deployment need to be carried out, so that the efficiency is low, and the process is complicated.
Disclosure of Invention
The invention aims to provide a large-scale processing and analyzing system for random data hybrid optimization, which solves the problems that the existing processing and analyzing system is difficult to integrate data and low in analyzing efficiency when facing multi-source heterogeneous mass data; and different business scenes need to be coded based on business, when the business slightly changes, a series of operations such as corresponding requirement evaluation, design, development, online operation, deployment and the like need to be carried out, so that the efficiency is low, and the process is complicated:
the invention relates to a large-scale processing analysis system for arbitrary data hybrid optimization, which comprises a data acquisition module, a data conversion module, a data processing module, a learning module, a data management module and a data interpretation analysis module;
the data acquisition module is used for executing acquisition tasks in a distributed mode in parallel by utilizing the computing power of the big data platform;
the data conversion module is used for converting the massive heterogeneous data into homogeneous data and transmitting the homogeneous data to the data processing module;
the data processing module is used for analyzing the data packet, processing a result according to the parameter information configured by the user and sending the result to the data management module;
the data management module is used for receiving and storing the original data sent by the data processing module, establishing a storage structure according to the test items and finishing the real-time storage of the data; receiving a data processing result issued by the data processing module through a network in real time and displaying the data processing result in real time;
the learning module is used for integrating the existing algorithm models to form an algorithm model database, and training the accuracy of the continuously optimized model based on massive sample data;
wherein, the bottom layer of the algorithm model database is merged into open source technology components, such as Impala, YANN, spark, hbase, HDFS, hive, kafka, flink, elasticSearch, zooKeeper, and the like. Aiming at different application fields, the functional components are expanded in a plug-in mode, and the specific calculation requirements are quickly responded;
and PB-level complex query and analysis are supported, and the single cluster deployment scale exceeds 1000 nodes. And the method provides high-efficiency conversion of data files in various formats and analysis service loading in custom formats, and supports data and application separation management and seamless data translation of application. And providing an algorithm model warehouse, and training to continuously optimize the model accuracy based on massive sample data.
The data interpretation analysis module is used for automatically finishing the interpretation work of the telemetering parameters according to different test states and test flows;
a transmission submodule capable of mutually transmitting data is arranged between the data acquisition module and the data interpretation and analysis module, the data acquisition module comprises field submodules, and the data acquisition module acquires source file data, analyzes parameter fields of the source file and extracts data of each parameter field; the field submodule is used for setting the corresponding relation between the parameter field of the source file and the parameter field of the target file, determining that the parameter field of the source file is in a first data format, the parameter field of the target file is in a second data format, the field submodule generates a format conversion protocol corresponding to each first data format according to the similarity between the parameter field of the source file and the parameter field of the target file, sends each format conversion protocol to a corresponding data conversion module, matches the data of the parameter field of the source file to the parameter field corresponding to the target file according to the format conversion protocol, and matches the data of the parameter field of the source file to the parameter field corresponding to the target file;
the method comprises the steps of setting a data acquisition module, a data conversion module, a data processing module, a learning module, a data management module and a data interpretation analysis module, setting field submodules in the data acquisition module, determining that parameter fields of a source file are in a first data format and parameter fields of a target file are in a second data format by setting the corresponding relation between the parameter fields of the source file and the parameter fields of the target file, generating format conversion protocols corresponding to each first data format according to the similarity between the parameter fields of the source file and the parameter fields of the target file, sending each format conversion protocol to the corresponding data conversion module, matching the data of the parameter fields of the source file to the parameter fields corresponding to the target file according to the format conversion protocols, and generating data matched to the parameter fields corresponding to the target file after calculating and judging the data of the parameter fields of one or more source files, and integrating the data into data packets, so as to reduce the difficulty of data integration when a system faces heterogeneous mass data and improve the analysis efficiency;
moreover, when the service slightly changes, the format conversion protocol corresponding to each first data format is generated according to the similarity between the parameter field of the source file and the parameter field of the target file in the data conversion, and the similarity change between the data is small, so that a series of operations such as requirement review, design, development, online deployment and the like also change little, the efficiency is high, the process is convenient, the time consumption is greatly reduced, and the use of a user is more flexible and convenient.
The working steps of the system are as follows:
s1: the method comprises the following steps that a data acquisition module acquires source file data, analyzes parameter fields of a source file and extracts data of each parameter field; the field submodule is used for setting the corresponding relation between the parameter field of the source file and the parameter field of the target file, and determining that the parameter field of the source file is in a first data format and the parameter field of the target file is in a second data format;
s2: generating a format conversion protocol corresponding to each first data format according to the similarity between the parameter field of the source file and the parameter field of the target file, and issuing each format conversion protocol to a corresponding data conversion module;
s3: according to a format conversion protocol, matching data of parameter fields of the source file to parameter fields corresponding to the target file, and matching data of the parameter fields of the source file to parameter fields corresponding to the target file comprises calculating and judging data of the parameter fields of one or more source files, generating data matched to the parameter fields corresponding to the target file, and integrating the data into a data packet;
s4: after receiving the data, the data processing module completes data processing according to the configured parameter processing information and forwards a processing result to data management software;
s5: the data management module establishes a storage structure according to the test project, and stores the data after receiving the processing result sent by the data processing and releasing software in real time; a user can monitor data in the test process in real time through a real-time monitoring module of the data management software;
s6: after the test is finished, the data interpretation analysis module reads the data file stored on the hard disk by the data management module, calls criteria to automatically interpret the data and generates a report; the software can also call the previously stored data of the previous tests and perform transverse comparison of different tests on the data.
The data interaction of the system involves the whole process from the generation of data to the calculation processing and distribution of data and the storage and calling of data. In the whole data transmission process, uploading of data and each service unit and upper levels is involved, the system needs to be docked and data transmission is carried out according to different requirements of each level on the data, and reservation of various data interfaces is a necessary link of the whole life cycle of the data.
Preferably, the data acquisition module further comprises a microprocessor unit, a sending interface control submodule and a receiving interface control submodule; the microprocessor unit is used for realizing interface control according to use requirements; the transmission interface control submodule is used for realizing data packet transmission based on a bus protocol; and the receiving interface control submodule is used for realizing data packet receiving based on a bus protocol.
Preferably, the data processing module comprises a bus data processing submodule, an auxiliary interpretation submodule and a real-time data processing submodule; the bus data processing submodule is used for realizing data analysis based on a bus protocol; the auxiliary interpretation submodule is used for the auxiliary data interpretation analysis module to interpret and analyze the data; and the real-time data processing submodule is used for analyzing the real-time data.
Preferably, the data management module comprises a database management sub-module, a real-time detection parameter configuration sub-module, a data storage and release sub-module and a database configuration import sub-module; a real-time database is established in the data management module and used for storing real-time data transmitted by the data processing module; the real-time detection parameter configuration submodule is used for binding and importing parameter information and checking the correctness of the parameter information; the data storage and release submodule is used for releasing a data processing result through a network and displaying the data processing result in real time; the database configuration import submodule is used for migrating and backing up the test data of the previous time and supporting the import of the test data of the unit with the same format.
Preferably, the data interpretation and analysis module comprises an automatic data interpretation sub-module and a data comparison and analysis sub-module; the data automatic interpretation submodule is used for automatically finishing the interpretation work of the telemetering parameters according to different test states and test flows; the data comparison and analysis submodule is used for transversely comparing different test data, storing criteria under different tasks and different states, and creating, editing, deleting and copying the criteria.
Preferably, the data acquisition module comprises an automatic acquisition submodule for automatically receiving externally transmitted data of the source file.
Preferably, the data interpretation and analysis module further comprises a user sub-module and a report sub-module; the user submodule is used for hierarchical management of hierarchical data and criteria of users, roles and authorities; and the report sub-module is used for automatically generating an interpretation result report and completing signing and confirmation based on a network.
Preferably, the data processing module can perform dynamic compiling, packing and dynamic scheduling to realize hot deployment capability;
a large amount of research and development resources and complicated development processes are saved by flexibly configuring data processing logics, data processing processes and hot deployment.
The invention has the following beneficial effects:
1. the method comprises the steps of setting a data acquisition module, a data conversion module, a data processing module, a learning module, a data management module and a data interpretation analysis module, setting a field submodule in the data acquisition module, determining that a parameter field of a source file is in a first data format and a parameter field of a target file is in a second data format by setting a corresponding relation between the parameter field of the source file and the parameter field of the target file, generating a format conversion protocol corresponding to each first data format according to the similarity between the parameter field of the source file and the parameter field of the target file, issuing each format conversion protocol to a corresponding data conversion module, matching data of the parameter field of the source file to the parameter field corresponding to the target file according to the format conversion protocol, and generating data matched to the parameter field corresponding to the target file after calculating and judging the data of the parameter field of one or more source files, integrating the data into a data packet, reducing the difficulty of integrating heterogeneous mass data when a system faces multi-source data, and improving the analysis efficiency.
2. When the service is slightly changed, the format conversion protocol corresponding to each first data format is generated according to the similarity between the parameter fields of the source file and the parameter fields of the target file, and the similarity between the data is changed slightly, so that a series of operations such as requirement evaluation, design, development, online operation, deployment and the like are changed slightly, the efficiency is high, the process is convenient, the time consumption is greatly reduced, and the use of a user is more flexible and convenient.
Of course, it is not necessary for any product to practice the invention to achieve all of the above-described advantages at the same time.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a system diagram of an arbitrary data mixing optimized large scale process analysis system of the present invention;
FIG. 2 is a flow chart of the operation of a large-scale process analysis system for arbitrary data mixing optimization according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
In the description of the present invention, it is to be understood that the terms "upper", "middle", "outer", "inner", and the like, indicate orientations or positional relationships, are used for convenience in describing the present invention and simplifying the description, but do not indicate or imply that the referenced components or elements must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention.
As shown in fig. 1-2, the present embodiment provides an arbitrary data mixing and optimizing large-scale processing analysis system, which includes a data acquisition module, a data conversion module, a data processing module, a learning module, a data management module, and a data interpretation and analysis module;
the data acquisition module is used for executing acquisition tasks in a distributed mode in parallel by utilizing the computing power of the big data platform;
the data conversion module is used for converting the mass heterogeneous data into isomorphic data and transmitting the isomorphic data to the data processing module;
the data processing module is used for analyzing the data packet, processing a result according to the parameter information configured by the user and sending the result to the data management module;
the data management module is used for receiving and storing the original data sent by the data processing module, establishing a storage structure according to the test items and finishing the real-time storage of the data; receiving a data processing result issued by the data processing module through a network in real time and displaying the data processing result in real time;
the learning module is used for integrating the existing algorithm models to form an algorithm model database, and training the accuracy of the model to be continuously optimized based on massive sample data;
wherein, the bottom layer of the algorithm model database is merged into open source technology components, such as Impala, YANN, spark, hbase, HDFS, hive, kafka, flink, elasticSearch, zooKeeper, and the like. Aiming at different application fields, the functional components are expanded in a plug-in mode, and the specific calculation requirements are quickly responded;
and PB-level complex query and analysis are supported, and the single cluster deployment scale exceeds 1000 nodes. And the method provides high-efficiency conversion of data files in various formats and analysis service loading in custom formats, and supports data and application separation management and seamless data translation of application. And providing an algorithm model warehouse, and training to continuously optimize the model accuracy based on massive sample data.
The data interpretation analysis module is used for automatically finishing the interpretation work of the telemetering parameters according to different test states and test flows;
a transmission submodule capable of mutually transmitting data is arranged between the data acquisition module and the data interpretation analysis module, the data acquisition module comprises a field submodule, the data acquisition module acquires source file data, analyzes parameter fields of the source file and extracts data of each parameter field; the field submodule is used for setting the corresponding relation between the parameter field of the source file and the parameter field of the target file, determining that the parameter field of the source file is in a first data format, the parameter field of the target file is in a second data format, generating a format conversion protocol corresponding to each first data format according to the similarity between the parameter field of the source file and the parameter field of the target file, and sending each format conversion protocol to a corresponding data conversion module, and the field submodule is used for matching the data of the parameter field of the source file to the parameter field corresponding to the target file according to the format conversion protocol, and the step of matching the data of the parameter field of the source file to the parameter field corresponding to the target file comprises the steps of calculating and judging the data of the parameter field of one or more source files, generating the data matched to the parameter field corresponding to the target file, and integrating the data into a data packet;
the method comprises the steps of setting a data acquisition module, a data conversion module, a data processing module, a learning module, a data management module and a data interpretation analysis module, setting a field submodule in the data acquisition module, determining that a parameter field of a source file is in a first data format and a parameter field of a target file is in a second data format by setting a corresponding relation between the parameter field of the source file and the parameter field of the target file, generating a format conversion protocol corresponding to each first data format according to the similarity between the parameter field of the source file and the parameter field of the target file, issuing each format conversion protocol to a corresponding data conversion module, matching data of the parameter field of the source file to the parameter field corresponding to the target file according to the format conversion protocol, and matching data of the parameter field of the source file to the parameter field corresponding to the target file;
moreover, when the service slightly changes, the format conversion protocol corresponding to each first data format is generated according to the similarity between the parameter field of the source file and the parameter field of the target file in the data conversion, and the similarity change between the data is small, so that a series of operations such as requirement review, design, development, online deployment and the like also change little, the efficiency is high, the process is convenient, the time consumption is greatly reduced, and the use of a user is more flexible and convenient.
The working steps of the system are as follows:
s1: the data acquisition module acquires source file data, analyzes parameter fields of the source file and extracts data of each parameter field; the field submodule is used for setting the corresponding relation between the parameter field of the source file and the parameter field of the target file, and determining that the parameter field of the source file is in a first data format and the parameter field of the target file is in a second data format;
s2: generating a format conversion protocol corresponding to each first data format according to the similarity between the parameter field of the source file and the parameter field of the target file, and issuing each format conversion protocol to a corresponding data conversion module;
s3: according to a format conversion protocol, matching data of parameter fields of a source file to parameter fields corresponding to a target file, and matching the data of the parameter fields of the source file to the parameter fields corresponding to the target file comprises calculating and judging the data of the parameter fields of one or more source files, generating the data matched to the parameter fields corresponding to the target file, and integrating the data into a data packet;
s4: after receiving the data, the data processing module completes data processing according to the configured parameter processing information and forwards a processing result to data management software;
s5: the data management module establishes a storage structure according to the test project, and stores the data after receiving the processing result sent by the data processing and publishing software in real time; a user can monitor data in the test process in real time through a real-time monitoring module of the data management software;
s6: after the test is finished, the data interpretation analysis module reads the data file stored on the hard disk by the data management module, calls criteria to automatically interpret the data and generates a report; the software can also call the previously stored data of the previous tests and perform transverse comparison of different tests on the data.
The data interaction of the system relates to the whole process from the generation of data to the calculation processing and distribution of the data and the storage and calling of the data. In the whole data transmission process, uploading of data and each service unit and upper levels is involved, the system needs to be docked and data transmission is carried out according to different requirements of each level on the data, and reservation of various data interfaces is a necessary link of the whole life cycle of the data.
The data acquisition module comprises a microprocessor unit, a sending interface control submodule and a receiving interface control submodule; the microprocessor unit is used for realizing interface control according to use requirements; the transmission interface control submodule is used for realizing data packet transmission based on a bus protocol; and the receiving interface control submodule is used for realizing the data packet receiving based on the bus protocol.
The data processing module comprises a bus data processing submodule, an auxiliary interpretation submodule and a real-time data processing submodule; the bus data processing submodule is used for realizing data analysis based on a bus protocol; the auxiliary interpretation submodule is used for the auxiliary data interpretation analysis module to interpret and analyze the data; and the real-time data processing submodule is used for analyzing the real-time data.
The data management module comprises a database management sub-module, a real-time detection parameter configuration sub-module, a data storage and release sub-module and a database configuration import sub-module; a real-time database is established in the data management module and used for storing the real-time data transmitted by the data processing module; the real-time detection parameter configuration submodule is used for binding and importing the parameter information and checking the correctness of the parameter information; the data storage and release submodule is used for releasing the data processing result through a network and displaying the data processing result in real time; the database configuration import submodule is used for migrating and backing up the test data of the previous time and supporting the import of the test data of the unit with the same format.
The data interpretation and analysis module comprises an automatic data interpretation sub-module and a data comparison and analysis sub-module; the data automatic interpretation submodule is used for automatically finishing the interpretation work of the telemetering parameters according to different test states and test flows; the data comparison and analysis submodule is used for transversely comparing different test data, storing criteria under different tasks and different states, and creating, editing, deleting and copying the criteria.
The data acquisition module comprises an automatic acquisition submodule and is used for automatically receiving data of a source file transmitted from the outside.
The data interpretation and analysis module also comprises a user sub-module and a report sub-module; the user submodule is used for hierarchical management of hierarchical data and criteria of users, roles and authorities; and the reporting submodule is used for automatically generating an interpretation result report and completing signing and confirmation based on the network.
The data processing module can perform dynamic compiling, packing and dynamic scheduling to realize hot deployment capability;
a large amount of research and development resources and complicated development processes are saved by flexibly configuring data processing logics, data processing processes and hot deployment.
The preferred embodiments of the invention disclosed above are intended to be illustrative only. The preferred embodiments are not intended to be exhaustive or to limit the invention to the precise embodiments disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, to thereby enable others skilled in the art to best understand the invention for and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.
Claims (8)
1. A large-scale processing analysis system for arbitrary data mixing optimization is characterized by comprising a data acquisition module, a data conversion module, a data processing module, a learning module, a data management module and a data interpretation analysis module;
the data acquisition module is used for executing acquisition tasks in parallel in a distributed mode by utilizing the computing power of the big data platform;
the data conversion module is used for converting the massive heterogeneous data into homogeneous data and transmitting the homogeneous data to the data processing module;
the data processing module is used for analyzing the data packet, processing a result according to the parameter information configured by the user and sending the result to the data management module;
the data management module is used for receiving and storing the original data sent by the data processing module, establishing a storage structure and finishing the real-time storage of the data;
the learning module is used for integrating the existing algorithm models to form an algorithm model database and training a continuous optimization model based on massive sample data;
the data interpretation analysis module is used for automatically finishing the interpretation work of the telemetering parameters according to different test states and test flows;
a transmission submodule capable of mutually transmitting data is arranged between the data acquisition module and the data interpretation analysis module, the data acquisition module comprises field submodules, and the data acquisition module acquires source file data, analyzes parameter fields of the source file and extracts data of each parameter field; the field submodule is used for setting a corresponding relation between parameter fields of a source file and parameter fields of a target file, determining that the parameter fields of the source file are in a first data format, the parameter fields of the target file are in a second data format, the field submodule generates a format conversion protocol corresponding to each first data format according to the similarity between the parameter fields of the source file and the parameter fields of the target file, sends each format conversion protocol to a corresponding data conversion module, matches the data of the parameter fields of the source file to the parameter fields corresponding to the target file according to the format conversion protocol, and matches the data of the parameter fields of the source file to the parameter fields corresponding to the target file.
2. An arbitrary data mixing optimized large scale process analysis system according to claim 1, characterized in that: the data acquisition module also comprises a microprocessor unit, a sending interface control submodule and a receiving interface control submodule; the microprocessor unit is used for realizing interface control according to use requirements; the transmission interface control submodule is used for realizing data packet transmission based on a bus protocol; and the receiving interface control submodule is used for receiving the data packet based on the bus protocol.
3. The system of claim 1, wherein the system is configured to perform any of the following operations: the data processing module comprises a bus data processing submodule, an auxiliary interpretation submodule and a real-time data processing submodule; the bus data processing submodule is used for realizing data analysis based on a bus protocol; the auxiliary interpretation submodule is used for the auxiliary data interpretation analysis module to interpret and analyze the data; and the real-time data processing submodule is used for analyzing the real-time data.
4. The system of claim 1, wherein the system is configured to perform any of the following operations: the data management module comprises a database management sub-module, a real-time detection parameter configuration sub-module, a data storage and release sub-module and a database configuration import sub-module; a real-time database is established in the data management module and used for storing real-time data transmitted by the data processing module; the real-time detection parameter configuration submodule is used for binding and importing parameter information and checking the correctness of the parameter information; the data storage and release submodule is used for releasing a data processing result through a network and displaying the data processing result in real time; the database configuration import submodule is used for migrating and backing up the test data of the previous time and supporting the import of the test data of the unit with the same format.
5. The system of claim 1, wherein the system is configured to perform any of the following operations: the data interpretation and analysis module comprises an automatic data interpretation submodule and a data comparison and analysis submodule; the data automatic interpretation submodule is used for automatically finishing the interpretation work of the telemetering parameters according to different test states and test flows; the data comparison and analysis submodule is used for transversely comparing different test data, storing criteria under different tasks and different states, and creating, editing, deleting and copying the criteria.
6. The system of claim 1, wherein the system is configured to perform any of the following operations: the data acquisition module comprises an automatic acquisition submodule and is used for automatically receiving data of the source file transmitted from the outside.
7. The system of claim 1, wherein the system is configured to perform any of the following operations: the data interpretation and analysis module also comprises a user sub-module and a report sub-module; the user submodule is used for hierarchical management of hierarchical data and criteria of users, roles and authorities; and the report sub-module is used for automatically generating an interpretation result report and completing signing and confirmation based on a network.
8. An arbitrary data mixing optimized large scale process analysis system according to claim 1, characterized in that: the data processing module can perform dynamic compiling, packing and dynamic scheduling, and achieves hot deployment capability.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211478698.9A CN115729998A (en) | 2022-11-23 | 2022-11-23 | Large-scale processing and analyzing system for arbitrary data hybrid optimization |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211478698.9A CN115729998A (en) | 2022-11-23 | 2022-11-23 | Large-scale processing and analyzing system for arbitrary data hybrid optimization |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115729998A true CN115729998A (en) | 2023-03-03 |
Family
ID=85297824
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211478698.9A Pending CN115729998A (en) | 2022-11-23 | 2022-11-23 | Large-scale processing and analyzing system for arbitrary data hybrid optimization |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115729998A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105009078A (en) * | 2013-02-12 | 2015-10-28 | 起元科技有限公司 | Building applications for configuring processes |
| CN105786912A (en) * | 2014-12-25 | 2016-07-20 | 远光软件股份有限公司 | Data acquisition and transformation method and device |
| CN107463418A (en) * | 2017-09-12 | 2017-12-12 | 北京宝兰德软件股份有限公司 | The configuration file generation method and device of a kind of server middleware |
| CN114625371A (en) * | 2022-02-18 | 2022-06-14 | 北京理工大学 | Cross-operating-system multi-source fusion algorithm compiling method, compiler and storage medium |
-
2022
- 2022-11-23 CN CN202211478698.9A patent/CN115729998A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105009078A (en) * | 2013-02-12 | 2015-10-28 | 起元科技有限公司 | Building applications for configuring processes |
| CN105786912A (en) * | 2014-12-25 | 2016-07-20 | 远光软件股份有限公司 | Data acquisition and transformation method and device |
| CN107463418A (en) * | 2017-09-12 | 2017-12-12 | 北京宝兰德软件股份有限公司 | The configuration file generation method and device of a kind of server middleware |
| CN114625371A (en) * | 2022-02-18 | 2022-06-14 | 北京理工大学 | Cross-operating-system multi-source fusion algorithm compiling method, compiler and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105843182B (en) | A kind of power scheduling accident prediction system and method based on OMS | |
| CN109213754A (en) | A kind of data processing system and data processing method | |
| CN109710703A (en) | A kind of generation method and device of genetic connection network | |
| CA2386272A1 (en) | Collaborative design | |
| CN110908789B (en) | Visual data configuration method and system for multi-source data processing | |
| CN112347071B (en) | Power distribution network cloud platform data fusion method and power distribution network cloud platform | |
| CN112115611B (en) | Collaborative simulation management system | |
| CN111367681A (en) | Cloud computing cluster-oriented bridge design system under high load state | |
| CN113822585A (en) | Intelligent smelting factory informatization management system | |
| CN116127899A (en) | Chip design system, method, electronic device, and storage medium | |
| CN114626807A (en) | Nuclear power scene management method, system, device, computer equipment and storage medium | |
| CN115576998A (en) | Power distribution network data integration method and system based on multi-dimensional information fusion | |
| CN114969047A (en) | Data processing system and method | |
| CN112905720A (en) | Operation data processing method and device based on source data management model | |
| CN112948353A (en) | Data analysis method, system and storage medium applied to DAstudio | |
| CN115729998A (en) | Large-scale processing and analyzing system for arbitrary data hybrid optimization | |
| CN114077492A (en) | Prediction model training and prediction method and system for cloud computing infrastructure resources | |
| CN115858672A (en) | Power terminal management method and device, electronic equipment and storage medium | |
| Duan | Analysis of ERP enterprise management information system based on cloud computing mode | |
| CN106447526A (en) | Power grid load intelligent control method | |
| CN109308563A (en) | A kind of modularization industrial service platform, working method and configuration method | |
| CN114862098A (en) | Resource allocation method and device | |
| CN112988705B (en) | Data middlebox construction method for enterprise-level production | |
| CN114596046A (en) | Integrated platform based on unified digital model of business center station and data center station | |
| CN115081679A (en) | CDC-based industrial data dynamic change automatic perception processing system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |