CN110737653B - Integrated enterprise data processing system and method based on micro-service - Google Patents
Integrated enterprise data processing system and method based on micro-service Download PDFInfo
- Publication number
- CN110737653B CN110737653B CN201910987142.4A CN201910987142A CN110737653B CN 110737653 B CN110737653 B CN 110737653B CN 201910987142 A CN201910987142 A CN 201910987142A CN 110737653 B CN110737653 B CN 110737653B
- Authority
- CN
- China
- Prior art keywords
- data
- micro
- service
- pipeline
- enterprise
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 67
- 238000000034 method Methods 0.000 title claims abstract description 54
- 230000010354 integration Effects 0.000 claims abstract description 34
- 230000008569 process Effects 0.000 claims abstract description 25
- 238000007405 data analysis Methods 0.000 claims abstract description 13
- 238000005192 partition Methods 0.000 claims abstract description 11
- 238000004458 analytical method Methods 0.000 claims abstract description 8
- 230000000007 visual effect Effects 0.000 claims abstract description 8
- 230000002159 abnormal effect Effects 0.000 claims description 31
- 238000004364 calculation method Methods 0.000 claims description 27
- 238000006243 chemical reaction Methods 0.000 claims description 20
- 238000003672 processing method Methods 0.000 claims description 17
- 238000004140 cleaning Methods 0.000 claims description 15
- 238000011161 development Methods 0.000 claims description 12
- 238000012800 visualization Methods 0.000 claims description 10
- 230000008859 change Effects 0.000 claims description 5
- 238000001914 filtration Methods 0.000 claims description 4
- 238000007726 management method Methods 0.000 claims description 4
- 238000005538 encapsulation Methods 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 claims description 3
- 238000007418 data mining Methods 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 12
- 238000010586 diagram Methods 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 238000013523 data management Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000013075 data extraction Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- Entrepreneurship & Innovation (AREA)
- Human Resources & Organizations (AREA)
- Strategic Management (AREA)
- Economics (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention belongs to the technical field of data processing, and discloses an integrated enterprise data processing system and method based on micro service, wherein a data source provides data support for the system; the data pipeline converts the data into a message set divided by topics through a micro-service component corresponding to the data source; concurrent setting is carried out on the data pipeline by utilizing the partition function of the message system, so that parallel processing of data is realized; the micro-service component acquires data from the data pipeline and processes the data; the processed data are stored in a database and a plurality of bins; graphically displaying, analyzing and exporting enterprise data through a visual operation panel; and (5) interfacing with a business system of an enterprise through an SDK API to obtain a multidimensional data analysis result. The method can efficiently realize heterogeneous integration of data and processing of improving data quality with low cost, provide visual operation, realize multidimensional analysis and data mining of the data without professional staff, and find potential data value.
Description
Technical Field
The invention belongs to the technical field of data processing, and particularly relates to an integrated enterprise data processing system and method based on micro-services.
Background
Currently, the closest prior art:
the BI field product needs to be customized and developed for some complex data analysis or needs to be maintained and used by professionals, and the function set by the product is limited in the use process, so that the change of the data demand cannot be responded quickly. And when handling a large number of data sources, confusion is easy to generate or distributed deployment cannot be realized, and large-scale data access cannot be handled.
Many data processing platforms on the market are customized individually according to their own unique situation. If the business changes greatly, a great development is needed. There are also many data processing platforms that can better suit different industry data requirements, but generally will employ relatively costly solutions, over-developed. Accordingly, there is a need for an enterprise data processing system and method that utilizes a lightweight architecture that is over-designed to meet enterprise needs and solve the problems of the prior art.
In summary, the problems of the prior art are:
(1) When analyzing some complex data, the prior art needs custom development or professional personnel to maintain and use, and the function set by the limited product in the use process can not respond to the change of the data demand quickly.
(2) When a large amount of data sources are managed, the prior art is easy to be confused or can not realize distributed deployment, and can not deal with large-scale data access.
(3) The existing data processing platforms are customized individually according to the special situations. If the business changes greatly, a great development is needed.
(4) The existing data processing platform which can meet the data requirements of different industries adopts a scheme with relatively high cost, and has the problem of excessive development.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides an integrated enterprise data processing system and method based on micro service.
The invention is realized in such a way that an integrated enterprise data processing method based on micro-services comprises the following steps:
the method comprises the steps that a data source provides data support for a system, a corresponding micro-service component determines how to distribute and receive data provided by the data source according to different characteristics of the data source, and a service party actively calls a certain open access port;
the second step, link with message module of publishing and subscribing, change the data into the message set divided by the theme through the microservice assembly of the correspondent data source; concurrent setting is carried out on the data pipeline by utilizing the partition function of the message system, so that parallel processing of data is realized; the data pipe name corresponds to Topic of the corresponding message system;
thirdly, the micro-service component acquires data from the data pipeline, and integrates, cleans, calculates and converts the data through the application in the operation container; storing the processed data into a database and a plurality of bins;
fourthly, graphically displaying, analyzing and exporting enterprise data through a visual operation panel; and (5) interfacing with a business system of an enterprise through an SDK API to obtain a multidimensional data analysis result.
Further, the micro-service-based integrated enterprise data processing method determines how the data pipeline distributes and receives the data provided by the data source according to different characteristics of the data source by the corresponding micro-service component; the allocation mode comprises active pulling and opening of an access port; the open access port is actively invoked by a service party;
for data pipes, one micro-service for data access must be a publisher of data, and other micro-services may be either publishers or subscribers or both; based on the rules, combining the micro service and the data pipeline; the messaging system is transparent to the user and the data pipe can manage the correspondence with the messaging system.
Further, the data integration method of the micro-service-based integrated enterprise data processing method comprises the following steps:
step one, splitting a data integration process into sub-processes, and carrying out operation encapsulation on the sub-processes to obtain sub-operations; the child job set is a parent job;
step two, capturing abnormal data and processing the abnormal data of the sub-job;
setting the transaction of the child job and setting the transaction of the parent job; integrating the data to start operation;
monitoring the execution state of the child job, and when all the child jobs are successfully operated, successfully operating the parent job and submitting data, wherein the data integration process is completed; when the sub-job fails to run, the data is not submitted, and the data integration process is completed.
Further, the method for capturing and processing the abnormal data of the sub-job in the second step specifically includes:
1) Setting conditions of the abnormal data;
2) Capturing abnormal data conforming to the conditions, and continuously integrating the conventional data when the abnormal data is corrected to obtain the conventional data; when the abnormal data cannot be corrected to obtain the normal data, storing the abnormal data into a file, skipping over the abnormal data, and continuing data integration;
3) When other abnormal data which do not meet the conditions are captured, the data integration process is ended.
Further, the flow calculation of the micro-service based integrated enterprise data processing method includes:
step one, inputting system original data, and initializing and setting variables;
calling a stream computing engine, and processing data through the stream computing engine to obtain a stream computing operation result;
and thirdly, analyzing the stream calculation operation result obtained in the second step, sending the analysis result to a server, and displaying the analysis result through a display.
Further, in the second step, the flow calculation engine is called in a multithreading parallel mode, or different flow calculation engines are called in a serial mode;
according to different data processing requirements, different modes are selected to call a stream calculation engine; for a scene with small data volume, selecting a serial mode; for a scene with large data volume, a multithreading parallel mode is selected to process the data in parallel.
It is another object of the present invention to provide a micro-service based integrated enterprise data processing system based on the micro-service based integrated enterprise data processing method, the micro-service based integrated enterprise data processing system comprising:
the data source provides data support for the system;
the data pipeline is connected with the publishing and subscribing message module and converts the data into a message set divided by topics through a micro-service component corresponding to the data source; concurrent setting is carried out on the data pipeline by utilizing the partition function of the message system, so that parallel processing of data is realized; the data pipe name corresponds to Topic of the corresponding message system;
the micro-service component acquires data from the data pipeline and processes the data by running an application in the container;
the storage module is used for storing the processed data into a database and a plurality of bins;
the visualization engine performs diagrammatical display, data analysis and data export on the enterprise data through the visualization operation panel; and (5) interfacing with a business system of an enterprise through an SDK API to obtain a multidimensional data analysis result.
Further, the microservice component comprises:
the data integration module is used for realizing data integration through a data integration functional component built in the system;
the data cleaning module provides expression rules through a cleaning functional component built in the system and performs filtering, conversion and structural definition of data based on the rules;
the stream calculation module is used for realizing stream calculation of data through a stream calculation functional component arranged in the system;
the data conversion module realizes data conversion through a data conversion functional component built in the system;
the script uploading module is used for uploading a processing script for data with poor quality, random structure and inaccurate expression, and the user can process the data through the script;
the component development module can develop components according to access specifications aiming at users with development capability and register the components through the management platform; after registration is successful, the user may use the component to create or update a data processing procedure.
Another object of the present invention is to provide an information data processing terminal implementing the micro-service based integrated enterprise data processing method.
It is a further object of the present invention to provide a computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the micro-service based integrated enterprise data processing method.
In summary, the invention has the advantages and positive effects that:
the invention realizes integration of batch and real-time data processing based on micro-services, runs on a Kubernetes platform, automatically establishes a data pipeline after accessing data from a data source, and responds to rapid enterprise demand change better through free combination of the pipeline, thereby realizing heterogeneous integration of the data and processing for improving the data quality with high efficiency and low cost. The visual operation is provided by the invention, so that multidimensional analysis and data mining of data can be realized without professional personnel, and potential data value is found.
The integrated enterprise data processing system and method based on the micro-service provided by the invention are based on the containerized micro-service, only the enterprise data processing service is deployed, and the cost is low; meanwhile, the special scene can be easily expanded and accessed according to the enterprise situation. The invention adopts a lighter-weight architecture, and the data processing requirements of enterprises are maximally met, and the design is not excessive; the scene that the enterprise demand changes very fast only needs to make the adjustment of data pipeline and micro-service through visual panel.
Drawings
FIG. 1 is a schematic diagram of a micro-service based integrated enterprise data processing system in accordance with an embodiment of the present invention;
in the figure: 1. a data source; 2. a data pipe; 3. a publish-subscribe message module; 4. a microservice component; 5. a storage module; 6. a visualization engine.
Fig. 2 is a flowchart of a micro-service-based integrated enterprise data processing method according to an embodiment of the present invention.
FIG. 3 is a schematic diagram of an integrated micro-service based enterprise data processing system provided by an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the following examples in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
In view of the problems in the prior art, the present invention provides an integrated enterprise data processing system and method based on micro services, and the present invention is described in detail below with reference to the accompanying drawings.
As shown in fig. 1, an integrated enterprise data processing system based on micro service provided by an embodiment of the present invention, the system includes: a data source 1, a data pipeline 2, a publish-subscribe message module 3, a microservice component 4, a storage module 5, and a visualization engine 6.
A data source 1, which provides data support for the system.
The data pipeline 2 is connected with the publish-subscribe message module 3 and converts data into a message set divided by topics through a micro-service component corresponding to the data source; concurrent setting is carried out on the data pipeline by utilizing the partition function of the message system, so that parallel processing of data is realized; the data pipe name corresponds to Topic of the corresponding messaging system.
The micro service component 4 obtains data from the data pipe 2 and processes the data by running the application in the container.
And the storage module 5 is used for storing the processed data into a database and a number bin.
The visualization engine 6 performs diagrammatical display, data analysis and data export on the enterprise data through a visualization operation panel; and (5) interfacing with a business system of an enterprise through an SDK API to obtain a multidimensional data analysis result.
Further, the micro-service component 4 comprises a data integration module 4-1, a data cleaning module 4-2, a stream calculation module 4-3, a data conversion module 4-4, a script uploading module 4-5 and a component development module 4-6.
The data integration module 4-1 realizes data integration through a data integration functional component built in the system.
The data cleaning module 4-2 provides expression rules through cleaning functional components built in the system, and performs filtering, conversion and structural definition of data based on the rules.
And the stream calculation module 4-3 realizes stream calculation of data through a stream calculation functional component built in the system.
The data conversion module 4-4 realizes data conversion through a data conversion functional component built in the system.
The script uploading module 4-5 is used for uploading the processing script for the data with poor quality, random structure and inaccurate expression, and the user can process the data through the script.
The component development module 4-6 can develop components according to access specifications aiming at users with development capability and register the components through the management platform; after registration is successful, the user may use the component to create or update a data processing procedure.
Further, depending on the different characteristics of the data sources 1, it is decided by the corresponding micro service component 4 how the data pipe 2 distributes the data provided by the received data sources 1. The allocation mode comprises active pulling and opening of an access port; the open access port is actively invoked by the service party.
Further, for the data pipeline 2, one micro-service to which data is accessed must be a publisher of the data, and other micro-services may be publishers, subscribers or both (when the micro-service obtains the data from the pipeline, after the processing is finished, only one subscriber, such as synchronizing the data, is usually used in the data warehouse if the data is no longer written into the data pipeline, and if the data is written into the pipeline after the processing is finished, for use by the downstream micro-service, both the subscriber and the publisher are usually used, such as cleaning and conversion of the data). Based on the rules described above, a combination of micro services and data pipes can be performed. The message system is transparent to the user, and the data pipeline can manage the corresponding relation with the message system by itself without knowing or learning the message system. Meanwhile, the data management utilizes the partition function of the message system, and the concurrent setting of the data pipeline can be realized.
Further, the data integration component realizes integration of data by the following method:
(1) Splitting the data integration process into sub-processes, and carrying out operation encapsulation on the sub-processes to obtain sub-operations; the child job set is a parent job.
(2) And carrying out abnormal data capturing and abnormal data processing on the sub-jobs.
(3) Setting the transaction of the child job and setting the transaction of the parent job; and integrating the data to start operation.
(4) Monitoring the execution state of the child job, and when all the child jobs are successfully operated, successfully operating the parent job and submitting data, wherein the data integration process is completed; when the sub-job fails to run, the data is not submitted, and the data integration process is completed.
Further, in the step (2), performing the capturing of the abnormal data and the processing of the abnormal data on the sub-job includes:
1) And setting the condition of the abnormal data.
2) Capturing abnormal data conforming to the conditions, and continuously integrating the conventional data when the abnormal data is corrected to obtain the conventional data; when the abnormal data cannot be corrected to obtain the normal data, storing the abnormal data into a file, skipping over the abnormal data, and continuing data integration.
3) When other abnormal data which do not meet the conditions are captured, the data integration process is ended.
Further, the stream computation function component implements stream computation of data by:
step one, inputting system original data and initializing and setting variables.
And step two, calling a stream calculation engine, and processing data through the stream calculation engine to obtain a stream calculation operation result.
And thirdly, analyzing the stream calculation operation result obtained in the second step, sending the analysis result to a server, and displaying the analysis result through a display.
Further, in the second step, the stream calculation engine is invoked in a multithreaded parallel manner, or different stream calculation engines are invoked in a serial manner. According to different data processing requirements, different modes are selected to call the stream calculation engine. For the scene with small data volume, a serial mode can be selected, logic is simple, and for the scene with large data volume, a multithread parallel mode can be selected to process data in parallel.
As shown in fig. 2, the integrated enterprise data processing method based on micro service provided by the embodiment of the invention includes the following steps:
s101: the data source provides data support for the system, and according to different characteristics of the data source, the corresponding micro-service component determines how the data pipeline distributes and receives the data provided by the data source, and some data are actively pulled, some data are actively accessed to the access port, and the service party actively calls the data.
S102: the system comprises a data source, a message publishing and subscribing module and a message publishing and subscribing module, wherein the data is converted into a message set divided by topics through a micro-service component corresponding to the data source; and the data pipeline is set concurrently by utilizing the partition function of the message system, so that the parallel processing of the data is realized. The data pipe name corresponds to Topic of the corresponding messaging system.
S103: the micro-service component acquires data from the data pipeline, and processes such as data integration, cleaning, stream calculation, conversion and the like are performed through the application in the operation container; and storing the processed data into a database and a plurality of bins.
S104: and graphically displaying, analyzing and exporting the enterprise data through the visual operation panel. And (5) interfacing with a business system of an enterprise through an SDK API to obtain a multidimensional data analysis result.
The technical scheme of the invention is further described below with reference to specific embodiments.
Example 1
As shown in FIG. 3, an integrated micro-service based enterprise data processing system schematic diagram is provided in an embodiment of the present invention.
According to different characteristics of the data sources, the corresponding micro-service components determine how the data pipeline distributes and receives the data provided by the data sources, and some of the data sources are actively pulled, and other data sources are actively called by the service side. The data pipeline is implemented based on a publish-subscribe messaging system (e.g., kafka) that converts data into sets of messages divided by topics through a microservice component corresponding to the data source. The data pipe name corresponds to Topic of the corresponding messaging system.
For the data pipeline, one micro-service accessed by the data must be a data publisher, and other micro-services can be publishers, subscribers or both (when the micro-service acquires the data from the pipeline, after the processing is finished, if the data pipeline is not written, only one subscriber is usually used for synchronizing the data into the data warehouse, and if the data is written into the pipeline after the processing is finished, the data is used for the downstream micro-service, the micro-service is not the subscribers but the publishers, and the common functions are such as cleaning and conversion of the data). Based on this rule, the combination of micro services and data pipes is easy. The message system is transparent to the user, and the data pipeline can manage the corresponding related system of the message system without knowing or learning the message system. Meanwhile, the data management utilizes the partition function of the message system, so that the concurrent setting of the data pipeline is easy to realize, and the processing capacity of the system is improved.
After flowing through the pipeline, the data can be further processed by applications in the running container, such as cleaning, conversion, machine learning, stream calculation and the like, and after processing, the data can be stored in a floor mode, can flow back to the pipeline again and be further processed. Or directly presented by the visualization engine. The data pipeline is provided with the partitions, the data processing can be performed in parallel, the number of the partitions is flexibly set, and the data processing capacity is improved. The visualization engine provides a visual operation panel, so that you can quickly graphically display enterprise data, or conduct data analysis or export data. If the enterprise has the need of interfacing with its own business system, the result of multidimensional data analysis can be easily obtained through SDK API.
The operations of data extraction, cleaning, conversion and the like are individual micro-service examples, and common data access, data cleaning and conversion functions are built in the system for users to select. By cleaning the built-in cleaning function component, the expression rule is provided, and the filtering, conversion and structural definition of the data are performed based on the rule. Sometimes, in the case of poor data quality and random structure, the expression may not be accurately expressed, and as such, the user may upload a processing script through which the data is processed.
For development-capable users, the development of the component can be performed according to the access specification, and the component is registered through the management platform. After registration is successful, the new or updated set of data processing procedures may be used.
It should be noted that the embodiments of the present invention can be realized in hardware, software, or a combination of software and hardware. The hardware portion may be implemented using dedicated logic; the software portions may be stored in a memory and executed by a suitable instruction execution system, such as a microprocessor or special purpose design hardware. Those of ordinary skill in the art will appreciate that the apparatus and methods described above may be implemented using computer executable instructions and/or embodied in processor control code, such as provided on a carrier medium such as a magnetic disk, CD or DVD-ROM, a programmable memory such as read only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The device of the present invention and its modules may be implemented by hardware circuitry, such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., as well as software executed by various types of processors, or by a combination of the above hardware circuitry and software, such as firmware.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, and alternatives falling within the spirit and principles of the invention.
Claims (6)
1. The integrated enterprise data processing method based on the micro-service is characterized by comprising the following steps of:
the method comprises the steps that a data source provides data support for a system, a corresponding micro-service component determines how to distribute and receive data provided by the data source according to different characteristics of the data source, and a service party actively calls a certain open access port;
the second step, link with message module of publishing and subscribing, change the data into the message set divided by the theme through the microservice assembly of the correspondent data source; concurrent setting is carried out on the data pipeline by utilizing the partition function of the message system, so that parallel processing of data is realized; the data pipe name corresponds to Topic of the corresponding message system;
thirdly, the micro-service component acquires data from the data pipeline, and integrates, cleans, calculates and converts the data through the application in the operation container; storing the processed data into a database and a plurality of bins;
fourthly, graphically displaying, analyzing and exporting enterprise data through a visual operation panel; the method comprises the steps of interfacing a business system of an enterprise through an SDK API to obtain a multidimensional data analysis result;
according to the micro-service-based integrated enterprise data processing method, according to different characteristics of data sources, a corresponding micro-service component determines how a data pipeline distributes and receives data provided by the data sources; the allocation mode comprises active pulling and opening of an access port; the open access port is actively invoked by a service party;
for data pipes, one micro-service for data access must be a publisher of data, and other micro-services may be either publishers or subscribers or both; based on the rules, combining the micro service and the data pipeline; the message system is transparent to the user, and the data pipeline can manage the corresponding relation with the message system;
the data integration method of the integrated enterprise data processing method based on the micro service comprises the following steps:
splitting the data integration process into sub-processes, and carrying out operation encapsulation on the sub-processes to obtain sub-operations; the child job set is a parent job;
performing abnormal data capturing and abnormal data processing on the sub-jobs;
setting the transaction of the child job and setting the transaction of the parent job; integrating the data to start operation;
monitoring the execution state of the child job, and when all the child jobs are successfully operated, successfully operating the parent job and submitting data, wherein the data integration process is completed; when the sub-job fails to run, data is not submitted, and the data integration process is completed;
the flow calculation of the micro-service-based integrated enterprise data processing method comprises the following steps:
inputting system original data, and initializing variables;
invoking a stream computing engine, and processing data through the stream computing engine to obtain a stream computing operation result;
analyzing the obtained stream calculation operation result, sending the analysis result to a server, and displaying the analysis result through a display;
invoking the stream computation engine in a multithreaded parallel manner or invoking different stream computation engines in a serial manner;
according to different data processing requirements, different modes are selected to call a stream calculation engine; for a scene with small data volume, selecting a serial mode; for a scene with large data volume, selecting a multithreading parallel mode to process data in parallel;
the integrated method based on micro-service realization batch and real-time data processing runs on a Kubernetes platform, automatically establishes a data pipeline after data is accessed from a data source, and realizes heterogeneous integration of the data and processing for improving data quality through free combination of the pipeline.
2. The integrated micro-service-based enterprise data processing method of claim 1, wherein the method for capturing and processing abnormal data for sub-jobs comprises:
1) Setting conditions of the abnormal data;
2) Capturing abnormal data conforming to the conditions, and continuously integrating the conventional data when the abnormal data is corrected to obtain the conventional data; when the abnormal data cannot be corrected to obtain the normal data, storing the abnormal data into a file, skipping over the abnormal data, and continuing data integration;
3) When other abnormal data which do not meet the conditions are captured, the data integration process is ended.
3. A micro-service based integrated enterprise data processing system based on the micro-service based integrated enterprise data processing method of any one of claims 1-2, wherein the micro-service based integrated enterprise data processing system comprises:
the data source provides data support for the system;
the data pipeline is connected with the publishing and subscribing message module and converts the data into a message set divided by topics through a micro-service component corresponding to the data source; concurrent setting is carried out on the data pipeline by utilizing the partition function of the message system, so that parallel processing of data is realized; the data pipe name corresponds to Topic of the corresponding message system;
the micro-service component acquires data from the data pipeline and processes the data by running an application in the container;
the storage module is used for storing the processed data into a database and a plurality of bins;
the visualization engine performs diagrammatical display, data analysis and data export on the enterprise data through the visualization operation panel; and (5) interfacing with a business system of an enterprise through an SDK API to obtain a multidimensional data analysis result.
4. The micro-service based integrated enterprise data processing system of claim 3, wherein the micro-service component comprises:
the data integration module is used for realizing data integration through a data integration functional component built in the system;
the data cleaning module provides expression rules through a cleaning functional component built in the system and performs filtering, conversion and structural definition of data based on the rules;
the stream calculation module is used for realizing stream calculation of data through a stream calculation functional component arranged in the system;
the data conversion module realizes data conversion through a data conversion functional component built in the system;
the script uploading module is used for uploading a processing script for data with poor quality, random structure and inaccurate expression, and the user can process the data through the script;
the component development module can develop components according to access specifications aiming at users with development capability and register the components through the management platform; after registration is successful, the user may use the component to create or update a data processing procedure.
5. An information data processing terminal for implementing the micro-service-based integrated enterprise data processing method of any one of claims 1 to 2.
6. A computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the micro-service based integrated enterprise data processing method as claimed in any one of claims 1-2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910987142.4A CN110737653B (en) | 2019-10-17 | 2019-10-17 | Integrated enterprise data processing system and method based on micro-service |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910987142.4A CN110737653B (en) | 2019-10-17 | 2019-10-17 | Integrated enterprise data processing system and method based on micro-service |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110737653A CN110737653A (en) | 2020-01-31 |
| CN110737653B true CN110737653B (en) | 2023-11-24 |
Family
ID=69269182
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910987142.4A Active CN110737653B (en) | 2019-10-17 | 2019-10-17 | Integrated enterprise data processing system and method based on micro-service |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110737653B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112069021B (en) * | 2020-08-21 | 2024-02-20 | 北京五八信息技术有限公司 | Flow data storage method and device, electronic equipment and storage medium |
| CN112817711A (en) * | 2021-01-22 | 2021-05-18 | 海南大学 | Data fusion system based on micro-service |
| CN113112807A (en) * | 2021-04-19 | 2021-07-13 | 重庆交通大学 | Expressway holographic sensing method integrating 4G probe and expressway electromechanical system |
| CN113268478A (en) * | 2021-06-24 | 2021-08-17 | 中国平安人寿保险股份有限公司 | Big data analysis method and device, electronic equipment and storage medium |
| CN114422371A (en) * | 2022-01-20 | 2022-04-29 | 重庆邮电大学 | Elastic micro-service system based on distributed and container virtualization and implementation method |
| CN118210480A (en) * | 2024-05-22 | 2024-06-18 | 山东浪潮智能生产技术有限公司 | System and method for realizing general form under micro-service scene |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106897277A (en) * | 2015-12-17 | 2017-06-27 | 成都飞机工业（集团）有限责任公司 | A kind of production and operation data visualization implementation method based on data mining |
| CN107193546A (en) * | 2017-04-11 | 2017-09-22 | 国网天津市电力公司信息通信公司 | A kind of micro services business application system |
| CN107959718A (en) * | 2017-11-17 | 2018-04-24 | 西北工业大学 | The micro services framework of enterprise-level application software under a kind of cloud computing environment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10389602B2 (en) * | 2016-12-05 | 2019-08-20 | General Electric Company | Automated feature deployment for active analytics microservices |
-
2019
- 2019-10-17 CN CN201910987142.4A patent/CN110737653B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106897277A (en) * | 2015-12-17 | 2017-06-27 | 成都飞机工业（集团）有限责任公司 | A kind of production and operation data visualization implementation method based on data mining |
| CN107193546A (en) * | 2017-04-11 | 2017-09-22 | 国网天津市电力公司信息通信公司 | A kind of micro services business application system |
| CN107959718A (en) * | 2017-11-17 | 2018-04-24 | 西北工业大学 | The micro services framework of enterprise-level application software under a kind of cloud computing environment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110737653A (en) | 2020-01-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110737653B (en) | Integrated enterprise data processing system and method based on micro-service | |
| US11386893B2 (en) | Human-computer interaction processing system, method, storage medium, and electronic device | |
| US10296373B2 (en) | Generic wait service: pausing and resuming a plurality of BPEL processes arranged in correlation sets by a central generic wait server | |
| CN109634764A (en) | Work-flow control method, apparatus, equipment, storage medium and system | |
| CN114265703B (en) | Cross-region computing power scheduling method, system and equipment for cloud server | |
| CN113094125B (en) | Business process processing method, device, server and storage medium | |
| US10523508B2 (en) | Monitoring management systems and methods | |
| CN111666166B (en) | Service providing method, device, equipment and storage medium | |
| CN107479870A (en) | A kind of third party's class libraries call method, device, mobile terminal and storage medium | |
| CN114697398B (en) | Data processing method, device, electronic equipment, storage medium and product | |
| CN115760013A (en) | Operation and maintenance model construction method and device, electronic equipment and storage medium | |
| CN113485686B (en) | Information system program generation method and device, electronic equipment and storage medium | |
| CN113419921B (en) | Task monitoring method, device, equipment and storage medium | |
| CN115617480A (en) | Task scheduling method, device and system and storage medium | |
| CN114661289A (en) | Knowledge and data driving-based micro application development system and method | |
| CN114510334A (en) | Class instance calling method and device, electronic equipment and automatic driving vehicle | |
| CN109150993B (en) | Method for obtaining network request tangent plane, terminal device and storage medium | |
| CN113779026A (en) | Method and device for processing service data table | |
| US20230004322A1 (en) | Managing provenance information for data processing pipelines | |
| CN113364775B (en) | Calling method and device of microservice and server | |
| CN111625524B (en) | Data processing method, device, equipment and storage medium | |
| US11526782B2 (en) | Inferring dependencies, requirements, and productions from spreadsheet-based loosely-coupled decision tables | |
| US20240036922A1 (en) | Open distributed asynchronous dependency-based content processing | |
| CN117057411B (en) | Large language model training method, device, equipment and storage medium | |
| CN117667573A (en) | Cluster operation and maintenance method and device based on AI language model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |