CN114722075A

CN114722075A - Data stream processing method and device, server and storage medium

Info

Publication number: CN114722075A
Application number: CN202110003316.6A
Authority: CN
Inventors: 蒲承祖; 刘毅; 袁鲲; 张康; 樊昊
Original assignee: China Mobile Communications Group Co Ltd; China Mobile Group Shandong Co Ltd
Current assignee: China Mobile Communications Group Co Ltd; China Mobile Group Shandong Co Ltd
Priority date: 2021-01-04
Filing date: 2021-01-04
Publication date: 2022-07-08

Abstract

The embodiment of the application provides a data stream processing method and device, a server and a storage medium, wherein the data stream processing method comprises the steps of obtaining information of a source table and information of a final table, recording the information of the source table and the information of the final table into an initialization table, obtaining a target final table with a first flag bit in the initialization table, obtaining all paths from the source table to the target final table according to a data stream tree, respectively calculating path consumption values of all the paths, and taking the path corresponding to the minimum value in the path consumption values of all the paths as an optimal path. And a path with low resource consumption is selected as a data channel through a path optimization algorithm, so that the resource consumption is reduced, and the system operation efficiency is improved.

Description

Data stream processing method and device, server and storage medium

Technical Field

The present application relates to the field of communications technologies, and in particular, to a data stream processing method and apparatus, a server, and a storage medium.

Background

The data flow processing of the existing data analysis system of an operator at present is usually completed by adopting a manual program writing mode, the data flow programs of the same task have different quality due to personal capability, experience and other differences, so that the whole system data flow is in an unordered and disordered state, the problem is increasingly prominent along with the increase of data sources and the continuous increase of data flows, more and more data problems can be caused, when the problems occur, the positioning and the processing of the problems are more difficult, the workload is consumed, and the system risk is increased.

Disclosure of Invention

The embodiment of the application provides a data stream processing method and device, a server and a storage medium, wherein a path with low resource consumption is selected as a data channel through a path optimization algorithm by the data stream processing method, so that the resource consumption is reduced, and the system operation efficiency is improved.

In a first aspect, an embodiment of the present application provides a data stream processing method, including:

acquiring information of a source table and information of a final table, and recording the information of the source table and the information of the final table into an initialization table;

acquiring a target final table with a first zone bit in the initialization table, and acquiring all paths from the source table to the target final table according to a data flow tree; and

and respectively calculating the path consumption values of all the paths, and taking the path corresponding to the minimum value in the path consumption values of all the paths as an optimal path.

Further, the process of calculating the path consumption value of a path includes:

acquiring index information of the path, wherein the index information comprises a node number A, a path depth B and a resource consumption value C;

and calculating a path consumption value for the path according to a first formula:

L＝(A*W1)+(B*W2)+(C*W3)

wherein L represents a path consumption value of a path, W1 represents a weight value corresponding to the node number, W2 represents a weight value corresponding to the path depth, and W3 represents a weight value corresponding to the consumed resource.

Further, after the taking the path corresponding to the minimum value among the path consumption values of all paths as the optimal path, the method further includes:

and storing the blood relationship of the optimal path in a data flow relationship table to generate an executable program, and changing the first flag bit of the target final table in the initialization table into a second flag bit.

Further, the method further comprises:

in the process of program scheduling, obtaining information of a final table with failed scheduling, and obtaining all paths of the final table with failed scheduling;

calculating path consumption values of all paths of the final table with the scheduling failure, and taking the path corresponding to the minimum value in the path consumption values of all paths as the optimal path of the final table with the scheduling failure; and

and storing the blood relation of the optimal path in a data flow direction relation table to generate an executable program, and executing the executable program.

Further, after the obtaining the information of the final table with failed scheduling and obtaining all paths of the final table with failed scheduling, the method further includes:

deleting the current scheduling failure path in all paths of the final table with scheduling failure.

and setting an unexecutable label for the current scheduling failure path in all paths of the final table with scheduling failure, and generating corresponding alarm information.

In a second aspect, an embodiment of the present application further provides a data stream processing apparatus, including:

a processor and a memory for storing at least one instruction which is loaded and executed by the processor to implement the data stream processing method provided by the first aspect.

In a third aspect, an embodiment of the present application further provides a server, where the server includes the data stream processing apparatus provided in the second aspect.

In a fourth aspect, an embodiment of the present application further provides a computer storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data stream processing method provided in the first aspect.

According to the technical scheme, information of a source table and information of a final table are obtained, the information of the source table and the information of the final table are recorded into an initialization table, a target final table with a first flag bit in the initialization table is obtained, all paths from the source table to the target final table are obtained according to a data flow tree, path consumption values of all the paths are respectively calculated, and the path corresponding to the minimum value in the path consumption values of all the paths is used as an optimal path. And a path with low resource consumption is selected as a data channel through a path optimization algorithm, so that the resource consumption is reduced, and the system operation efficiency is improved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can obtain other drawings based on the drawings without inventive labor.

FIG. 1 is a schematic diagram of a process for creating a program according to an embodiment of the present application;

FIG. 2 is an architecture diagram of a data stream processing model provided in one embodiment of the present application;

FIG. 3 is a flow chart illustrating a method for data flow control according to an embodiment of the present application;

FIG. 4 is an exemplary illustration of relationship of blood vessels provided by an embodiment of the present application;

FIG. 5 is a diagram illustrating an example of a path provided by one embodiment of the present application;

FIG. 6 is a flowchart of a process for scheduling fault tolerance according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of a data stream processing apparatus according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Fig. 1 is a schematic flowchart of a program creation process provided in an embodiment of the present application, and as shown in fig. 1, the program creation process includes the following steps:

step 101: recording a final table and corresponding source table information into a data flow direction tree, and setting a first flag bit (the flag bit is 0) for the final table;

step 102: and acquiring source table information, traversing the data flow direction tree and constructing complete data flow information.

Step 103: and traversing the final table field information in a service metadata base to obtain a dimension field and an index field.

Step 104: and querying the index set to obtain an algorithm and a code corresponding to the index field.

Step 105: and creating a program according to the data flow information, the dimension field information of the final table and the index algorithm, and changing the flag bit of the final table into a second flag bit (the flag bit is 1).

The first flag bit (flag bit is 0) indicates that the current state is that the program is not created, and the second flag bit (flag bit is 1) indicates that the current state is that the program is created.

The method can improve the program operation efficiency and reduce the complexity of the data stream by continuously optimizing the whole data stream, specifically, a corresponding data stream processing model can be provided, and fig. 2 is an architecture diagram of the data stream processing model provided by an embodiment of the present application, as shown in fig. 2, the data stream processing model adopts dimension modeling and is longitudinally divided into three-layer structures, and data streams flow down and up, that is, the data streams flow from a base layer to a warehouse layer and finally flow to a market layer.

As shown in fig. 2, the whole data flow model is composed of a base layer, a warehouse layer and a market layer, and the base layer accesses the original data according to the interface specification to determine the factual entity. The warehouse layer is composed of dimension entities (recording dimension information data) and real-time entities (recording index data). The mart layer determines factual entities based on business requirements.

The data flow model avoids the problems of disorder, chaos and the like of the data flow by determining respective boundaries and responsibilities, limits the depth of the whole data flow, effectively reduces the complexity of the data flow, solves the problems that the data problem caused by the excessively complex data flow is difficult to position and process in the system evolution process, and reduces the workload of operation and maintenance personnel.

Further, the data flow self-iterative optimization method further provides a corresponding path optimization manner, and specifically, an optimal path may be selected according to the depth of the data flow and under the condition that each power consumption resource is consumed, and specifically, the present application further provides a data flow control method, fig. 3 is a schematic flow diagram of the data flow control method provided in an embodiment of the present application, and as shown in fig. 3, the data flow control method includes the following steps:

step 301: and acquiring the information of the source table and the information of the final table, and recording the information of the source table and the information of the final table into the initialization table.

The initialization table may be as shown in table one:

watch 1

Table name	Source meter	Marker bit
			Final Table 1	Source table 1	0
Final Table 1	Source table 2	0
			Final Table 1	Source table 3	0

The following is a brief description of the corresponding terms:

a source table: refers to a table of access data, belonging to the base layer.

Intermediate table: the table refers to a table for completing calculation and processing business logic and belongs to a warehouse layer.

Final table: the table means the table for storing the results, and belongs to the market layer.

Kindred relationship of data table: the data flow is a relationship constructed by taking a data source table in a data table as a starting point and taking the data table itself as an end point according to the data flow between the tables. Taking fig. 4 as an example, the data of the source tables 1 and 2 are extracted into the intermediate table 1, the data of the source table 3 is extracted into the intermediate table 2, and the data of the intermediate tables 1 and 2 are extracted into the final table 1 through processing calculation. That is, the data flow is expanded layer by layer with the source tables 1,2, and 3 as the starting points and the final table 1 as the end point, and the relationship of the blood margin in the final table 1 shown in fig. 3 is formed.

Step 302: and acquiring a target final table with a first zone bit in the initialization table, and acquiring all paths from the source table to the target final table according to a data flow tree.

Fig. 5 is an exemplary diagram of paths provided in an embodiment of the present application, and as shown in fig. 5, all paths from the source table to the destination final table include a first path and a second path, where the first path is to extract data of the source table 1 and the source table 2 into the intermediate table 1, extract data of the source table 3 into the intermediate table 2, and extract data of the intermediate table 1 and the intermediate table 2 into the final table 1 after processing and calculation. I.e. with the apparent source tables 1,2,3 as starting points and the final table 1 as end points. The second path is to extract the data in the source table 1 into the intermediate table 1, extract the data in the intermediate table 1 into the intermediate table 3 after processing calculation, extract the data in the source table 2 and the source table 3 into the intermediate table 2, and extract the data in the intermediate table 2 and the intermediate table 3 into the final table 1 after processing calculation. Again, the source tables 1,2,3 are used as starting points and the final table 1 is used as end point.

Step 303: and respectively calculating the path consumption values of all the paths, and taking the path corresponding to the minimum value in the path consumption values of all the paths as an optimal path.

Wherein the process of calculating the path consumption value of a path comprises:

acquiring index information of the target path, wherein the index information comprises the node number A, the path depth B and the resource consumption value C of the target path;

first formula of L ═ W1) + (B × W2) + (C × W3) … …

For example, all the paths of the target final table (final table 1) include the first path and the second path shown in fig. 5, and then it may be determined that the index information of the first path and the second path of the final table 1 is shown in table two:

watch 2

Table name	Route of travel	Number of nodes (A1)	Path depth (B1)	Costing resources (C1)
					Final Table 1	First path	6	2	10
Final Table 1	Second path	7	3	8

The path consumption value L1 of the first path of table 1 can be finally calculated by the first formula:

L1＝(6*W1)+(2*W2)+(10*W3)

the path consumption value L2 of the second path of table 1 is finally calculated by the first formula:

L2＝(6*W1)+(2*W2)+(10*W3)

from the calculation results, the minimum value is selected from the path consumption value L1 of the first path in table 1 and the path consumption value L2 of the second path in table 1, and if L1 < L2, the second path is set as the optimal path in table 1.

Step 304: and storing the blood relationship of the optimal path in a data flow relationship table to generate an executable program, and changing the first flag bit of the target final table in the initialization table into a second flag bit.

The data flow direction relation table for storing the blood relationship of the optimal path is shown in table three:

watch III

Classification	Table name	Source meter
			Market layer	Final Table 1	Intermediate Table 1
Market layer	Final Table 1	Middle table 2
			Warehouse layer	Intermediate Table 1	Source table 1
Warehouse layer	Intermediate Table 1	Source table 2
			Warehouse layer	Middle table 2	Source table 3
Base layer	Source table 1
			Base layer	Source table 2
Base layer	Source table 3

The data flow self-iteration is realized by combining the dimension modeling and the path optimization algorithm, the problems that the data problem caused by excessively complex data flow is difficult to position and process in the system evolution process are solved, the system complexity is reduced, the system operation efficiency is improved, and the manual investment is reduced.

Further, the fields and algorithm information construction codes of the final table 1 and the intermediate table can be obtained in the service metadata table (e.g., table four), an executable program is generated, and the flag bit in the initialization table is changed to 1.

Watch four

Generating a code:

INSERT INTO Final TABLE 1

SELECT field a, SUM (field C field D)

FROM intermediate Table 1, intermediate Table 2

GROUP BY field A

Fig. 6 is a flowchart of a scheduling fault-tolerant processing method provided in an embodiment of the present application, and as shown in fig. 6, the scheduling fault-tolerant processing method further includes the following operation steps:

step 601: in the process of program scheduling, obtaining the information of the final table with failed scheduling, and obtaining all paths of the final table with failed scheduling.

Step 602: deleting the current scheduling failure path in all paths of the final table with scheduling failure.

In this embodiment, after the information of the final table with failed scheduling is obtained and all paths of the final table with failed scheduling are obtained, an unexecutable label may be set for a current scheduling failure path in all paths of the final table with failed scheduling, and corresponding alarm information may be generated.

Step 603: and calculating path consumption values of all paths of the final table with the scheduling failure, and taking the path corresponding to the minimum value in the path consumption values of all paths as the optimal path of the final table with the scheduling failure.

Step 604: and storing the blood relationship of the optimal path in a data flow direction relationship table to generate an executable program, and executing the program.

The scheduling fault-tolerant processing is realized through the data flow direction tree, the problem of task failure caused by the problem of data flow intermediate nodes in the task scheduling process is solved, and the system fault tolerance is improved.

Fig. 7 is a schematic structural diagram of a data stream processing apparatus according to an embodiment of the present application, where as shown in fig. 7, the apparatus includes: a processor 701 and a memory 702, the memory 702 being configured to store at least one instruction that is loaded and executed by the processor 701 to perform the following:

L＝(A*W1)+(B*W2)+(C*W3)

Further, the method further comprises:

setting an inexecutable label for the current scheduling failure path in all paths of the final table of the scheduling failure, and generating corresponding alarm information.

The embodiment of the present application further provides a server, where the server may include the data stream processing apparatus provided in the embodiment shown in fig. 7.

The embodiment of the present application further provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the data stream processing method provided by the embodiment of the present application is implemented.

It should be understood that the application may be an application program (native app) installed on the terminal, or may also be a web page program (webApp) of a browser on the terminal, which is not limited in this embodiment of the present invention.

It can be clearly understood by those skilled in the art that, for convenience and simplicity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the embodiments provided in the present invention, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions in actual implementation, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.

The integrated unit implemented in the form of a software functional unit may be stored in a computer readable storage medium. The software functional unit is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) or a Processor (Processor) to execute some steps of the methods according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, an optical disk, or other various media capable of storing program codes.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing a data stream, the method comprising:

2. The method of claim 1, wherein calculating the path consumption value for a path comprises:

L＝(A*W1)+(B*W2)+(C*W3)

3. The method according to claim 1, further comprising, after taking the path corresponding to the minimum value among the path consumption values of all paths as an optimal path:

4. The method of claim 1, further comprising:

5. The method according to claim 4, further comprising, after the obtaining information of the final table with failed scheduling and obtaining all paths of the final table with failed scheduling, the steps of:

6. The method according to claim 4, further comprising, after the obtaining information of the final table with failed scheduling and obtaining all paths of the final table with failed scheduling, the steps of:

7. A data stream processing apparatus, characterized in that the apparatus comprises:

a processor and a memory for storing at least one instruction which is loaded and executed by the processor to implement the data stream processing method of any of claims 1-6.

8. A server, characterized in that the server comprises: the data stream processing apparatus of claim 7.

9. A computer storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data stream processing method according to any one of claims 1 to 6.