CN114416849A - Data processing method and device, electronic equipment and storage medium - Google Patents
Data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114416849A CN114416849A CN202210089970.8A CN202210089970A CN114416849A CN 114416849 A CN114416849 A CN 114416849A CN 202210089970 A CN202210089970 A CN 202210089970A CN 114416849 A CN114416849 A CN 114416849A
- Authority
- CN
- China
- Prior art keywords
- data
- real
- rule
- time
- physical table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of big data, and provides a data processing method, a device, electronic equipment and a storage medium, wherein the method comprises the following steps: receiving real-time data reported by a service system in real time and offline data reported by the service system offline; performing first pretreatment on real-time data to obtain a real-time physical table, and performing second pretreatment on offline data to obtain an offline physical table; converting the data processing rules into database rule statements, and writing the database rule statements, the real-time physical table and the off-line physical table into a rule table in a database management system; triggering the database rule statements at regular time, and obtaining target rule result data corresponding to the database rule statements from the real-time physical table and the off-line physical table; and extracting from the target rule result data to obtain a data processing result. The invention converts the real-time physical table and the off-line physical table into the rule result data, and the extracted data is pre-calculated, thereby improving the processing efficiency of the data.
Description
Technical Field
The invention relates to the technical field of big data, in particular to a data processing method and device, electronic equipment and a storage medium.
Background
With the internet entering the big data era, the types of data sources begin to be diversified, taking an excitation scheme of business personnel or an organization as an example, the excitation scheme needs object, index and time limitation, and needs to acquire data from a real-time data source and an offline data source.
However, when data of multiple data sources are written into one database at regular time, when a real-time data source is involved, the update frequency of the acquired data is limited by the operation frequency of a manager, and when offline data is involved, offline historical data may be refreshed at a future time to correct the data, and the manager cannot timely and accurately know such changes, which results in low data processing efficiency.
Therefore, it is necessary to provide a method for processing data rapidly and accurately.
Disclosure of Invention
In view of the above, there is a need for a data processing method, apparatus, electronic device and storage medium, which can improve the data processing efficiency by converting the real-time physical table and the off-line physical table into rule result data, and by calculating the extracted data in advance.
A first aspect of the present invention provides a data processing method, the method comprising:
analyzing the received data processing request to acquire an identification code and a data processing rule of a service system;
receiving real-time data reported by a service system corresponding to the identification code in real time and offline data reported by the service system in an offline manner;
performing first pretreatment on the real-time data to obtain a real-time physical table, and performing second pretreatment on the offline data to obtain an offline physical table;
converting the data processing rule into a database rule statement, and writing the database rule statement, the real-time physical table and the off-line physical table into a rule table in a database management system;
based on a scheduling mechanism in the rule table, regularly triggering the database rule statement, and obtaining target rule result data corresponding to the database rule statement from the real-time physical table and the off-line physical table;
and responding to the data processing request, and extracting from the target rule result data to obtain a data processing result.
Optionally, the receiving real-time data reported by the service system in real time corresponding to the identification code and offline data reported by the service system offline includes:
acquiring a first calling interface list of a plurality of preset first data sources corresponding to real-time data and a second calling interface list of a plurality of preset second data sources corresponding to offline data from a preset data source interface library according to the identification code of the service system;
and the service system corresponding to the identification code reports the real-time data corresponding to each first calling interface in the first calling interface list in real time, and the service system corresponding to the identification code reports the offline data corresponding to each second calling interface in the second calling interface list in an offline manner.
Optionally, the performing a first preprocessing on the real-time data to obtain a real-time physical table includes:
identifying a type of the real-time data;
if the type of the real-time data is static data, writing the real-time data into a message queue Kafka to form a first real-time physical table, and determining the first real-time physical table as a real-time physical table; or
And if the type of the real-time data is dynamic data, consuming the real-time data through a distributed processing engine Flink to form a second real-time physical table, and determining the second real-time physical table as a real-time physical table.
Optionally, the consuming the real-time data by the distributed processing engine Flink to form a second real-time physical table includes:
importing the real-time data into a distributed processing engine (Flink), and identifying table names and logical relations among table fields of a plurality of real-time data tables in the real-time data;
and when an idle slot is obtained in the distributed processing engine Flink, executing the logical relationship between the table names and the table fields of the plurality of real-time data tables in the idle slot to form a second real-time physical table.
Optionally, the performing a second preprocessing on the offline data to obtain an offline physical table includes:
identifying a table name and a table field of each of a plurality of offline data tables in the offline data;
and creating a Hadoop cluster according to the table name and the logic relation between the table fields of each offline data table in the plurality of offline data tables, and synchronizing the offline data into the created Hadoop cluster to form an offline physical table.
Optionally, the converting the data processing rule into a database rule statement includes:
identifying, by a rules engine, each of the data processing rules;
converting each processing rule into a corresponding sub-database rule statement according to a preset conversion mode;
and splicing all the sub-database rule statements to obtain the database rule statements.
Optionally, the periodically triggering the database rule statement based on the scheduling mechanism in the rule table, and obtaining target rule result data corresponding to the database rule statement from the real-time physical table and the offline physical table includes:
triggering each sub-database rule statement in the database rule statements at regular time, and acquiring corresponding rule result data from the real-time physical table and the off-line physical table based on each sub-database rule statement;
repeatedly executing the sub-database rule statements to acquire corresponding rule result data from the real-time physical table and the off-line physical table based on each sub-database rule statement until the rule result data corresponding to all sub-database rule statements in the database rule statements are acquired;
and determining the rule result data corresponding to all the sub-database rule statements as target rule result data corresponding to the database rule statements.
A second aspect of the present invention provides a data processing apparatus, the apparatus comprising:
the acquisition module is used for analyzing the received data processing request and acquiring the identification code and the data processing rule of the service system;
the receiving module is used for receiving real-time data reported by the service system corresponding to the identification code in real time and offline data reported by the service system in an offline manner;
the preprocessing module is used for performing first preprocessing on the real-time data to obtain a real-time physical table and performing second preprocessing on the off-line data to obtain an off-line physical table;
the conversion module is used for converting the data processing rule into a database rule statement and writing the database rule statement, the real-time physical table and the off-line physical table into a rule table in a database management system;
the triggering module is used for regularly triggering the database rule statements based on a scheduling mechanism in the rule table and obtaining target rule result data corresponding to the database rule statements from the real-time physical table and the off-line physical table;
and the extraction module is used for responding to the data processing request and extracting from the target rule result data to obtain a data processing result.
A third aspect of the invention provides an electronic device comprising a processor and a memory, the processor being configured to implement the data processing method when executing a computer program stored in the memory.
A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the data processing method.
In summary, the data processing method, the data processing apparatus, the electronic device, and the storage medium according to the present invention perform the first preprocessing on the real-time data to obtain the real-time physical table, and perform the second preprocessing on the offline data to obtain the offline physical table, so as to improve the accuracy of the offline data and the real-time data. And converting the data processing rule into a database rule statement, writing the database rule statement, the real-time physical table and the offline physical table into a rule table in a database management system, and extracting data corresponding to the database rule statement when the database rule statement is subsequently executed, so that the data extraction efficiency is improved, and the data processing efficiency is further improved. And based on a scheduling mechanism in the rule table, regularly triggering the database rule statements to obtain target rule result data corresponding to the database rule statements, and changing active query into a pre-calculation result pushing mode, so that the data processing efficiency is improved. And responding to the data processing request, extracting the target rule result data to obtain a data processing result, wherein the extracted data are pre-calculated, so that the complex query sentence is not required to be executed in real time by the database, the operation efficiency of the database is ensured, and the data processing efficiency is further improved.
Drawings
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention.
Fig. 2 is a structural diagram of a data processing apparatus according to a second embodiment of the present invention.
Fig. 3 is a schematic structural diagram of an electronic device according to a third embodiment of the present invention.
Detailed Description
In order that the above objects, features and advantages of the present invention can be more clearly understood, a detailed description of the present invention will be given below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and features of the embodiments may be combined with each other without conflict.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
Example one
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention.
In this embodiment, the data processing method may be applied to an electronic device, and for an electronic device that needs to perform data processing, the data processing function provided by the method of the present invention may be directly integrated on the electronic device, or may be run in the electronic device in the form of a Software Development Kit (SDK).
As shown in fig. 1, the data processing method specifically includes the following steps, and the order of the steps in the flowchart may be changed and some steps may be omitted according to different requirements.
S11, analyzing the received data processing request, and acquiring the identification code and the data processing rule of the service system.
In this embodiment, when a user performs data processing, a data processing request is initiated to a server through a client, specifically, the client may be a smart phone, an IPAD, or other existing intelligent device, the server may be a data processing subsystem, and in a data processing process, for example, the client may send the data processing request to the data processing subsystem, and the data processing subsystem is configured to receive the data processing request sent by the client.
In this embodiment, when the data processing subsystem receives a data processing request sent by a client, the data processing request is analyzed, and an identification code and a data processing rule of a service system are obtained.
Specifically, the identification code is used to uniquely identify the business system, and the data processing rule refers to a rule required for executing the data processing request.
In an optional embodiment, the analyzing the received data processing request and acquiring the identification code and the data processing rule of the service system includes:
analyzing the message of the data processing request to obtain message information carried by the message;
acquiring an identification code and an interface calling rule from the message information;
and determining a service system according to the identification code, and determining a data processing rule according to the interface calling rule.
In this embodiment, the data processing request includes an identification code and an interface calling rule.
And S12, receiving the real-time data reported by the service system corresponding to the identification code in real time and the off-line data reported by the service system in off-line.
In this embodiment, when data processing is performed, it may be necessary to merge real-time data and offline data, so that the real-time data reported by the service system in real time corresponding to the identification code and the offline data reported by the service system offline are received, which facilitates subsequent data processing.
In an optional embodiment, the receiving real-time data reported by the service system in real time corresponding to the identification code and offline data reported by the service system offline includes:
acquiring a first calling interface list of a plurality of preset first data sources corresponding to real-time data and a second calling interface list of a plurality of preset second data sources corresponding to offline data from a preset data source interface library according to the identification code of the service system;
and the service system corresponding to the identification code reports the real-time data corresponding to each first calling interface in the first calling interface list in real time, and the service system corresponding to the identification code reports the offline data corresponding to each second calling interface in the second calling interface list in an offline manner.
In this embodiment, the plurality of preset first data sources may be the same as the plurality of preset second data sources, or may be different from the plurality of preset second data sources, where each preset first data source corresponds to one first call interface, and each preset second data source corresponds to one second call interface.
In this embodiment, each preset first data source corresponds to one first calling interface, and each preset second data source corresponds to one second calling interface, so that the problem of data blocking caused by that all preset data sources call data through one calling interface and report the data is avoided, and the efficiency of data reporting is improved.
S13, performing first preprocessing on the real-time data to obtain a real-time physical table, and performing second preprocessing on the off-line data to obtain an off-line physical table.
In this embodiment, the physical table includes a real-time physical table and an offline physical table, where the physical table refers to a table in a specific data source.
In an optional embodiment, the performing the first preprocessing on the real-time data to obtain the real-time physical table includes:
identifying a type of the real-time data;
if the type of the real-time data is static data, writing the real-time data into a message queue Kafka to form a first real-time physical table, and determining the first real-time physical table as a real-time physical table; or
And if the type of the real-time data is dynamic data, consuming the real-time data through a distributed processing engine Flink to form a second real-time physical table, and determining the second real-time physical table as a real-time physical table.
In this embodiment, the real-time data includes two types, one type is static data, and specifically, the static data refers to data that does not need to be processed; one type is dynamic data, which refers to data that needs to be processed.
In this embodiment, the distributed processing engine Flink includes four different components, namely, a distributor, a job manager, a resource manager, and a task manager, and the four components are run on a server to process the real-time data to form a second real-time physical table, specifically, the consuming the real-time data by the distributed processing engine Flink to form the second real-time physical table includes:
importing the real-time data into a distributed processing engine (Flink), and identifying table names and logical relations among table fields of a plurality of real-time data tables in the real-time data;
and when an idle slot is obtained in the distributed processing engine Flink, executing the logical relationship between the table names and the table fields of the plurality of real-time data tables in the idle slot to form a second real-time physical table.
In this embodiment, the real-time data is imported into a distributor of a distributed processing engine Flink, and the distributor reports the real-time data to an operation manager of the distributed processing engine Flink; the operation manager identifies the table names and the table fields of a plurality of real-time data tables in the real-time data, and applies for resources from the resource manager of the distributed processing engine flight according to the logical relationship between the table names and the table fields of the plurality of real-time data tables; the resource manager starts a task manager of the distributed processing engine Flink based on the applied resources; the task manager registers information to the resource manager, and the resource manager sends a command for requesting to acquire an idle slot to the task manager of the distributed processing engine Flink according to the registration information; the task manager sends an idle slot to the job manager of the distributed processing engine Flink in response to the instruction for acquiring the idle slot, and the job manager executes a logical relationship between table names and table fields of the multiple real-time data tables in the idle slot to form a second real-time physical table, where the distributed processing engine Flink is a prior art, and this embodiment is not described in detail herein.
In this embodiment, the slot of the space refers to an idle network interface in the task manager, and may be used to execute the table names and the logical relationships between the table fields of the multiple real-time data tables to form a second real-time physical table.
In this embodiment, the distributed processing engine Flink is used for performing stateful computation on the real-time data stream, for example: the method comprises the steps that the performance of an agent M is obtained, a real-time data table 1 reported by a preset first data source A contains agent serial numbers and policy information, a real-time data table 2 reported by a preset first data source B contains agent basic information, and in order to determine the performance of the agent M, the real-time data table 1 and the real-time data table 2 need to be correlated, so that a distributed processing engine Flink consumption is adopted to correlate the real-time data table 1 with the real-time data table 2, and a real-time physical table is obtained.
In the embodiment, the distributed processing engine Flink can process streaming data with low delay and high throughput, and can process out-of-order data well to ensure the consistency of real-time data states.
In an optional embodiment, the second preprocessing the offline data to obtain an offline physical table includes:
identifying a table name and a table field of each of a plurality of offline data tables in the offline data;
and creating a Hadoop cluster according to the table name and the logic relation between the table fields of each offline data table in the plurality of offline data tables, and synchronizing the offline data into the created Hadoop cluster to form an offline physical table.
In this embodiment, the offline data includes a plurality of offline data tables, each offline data table corresponds to a table name and a table field, the logical relationship refers to a logical relationship between the plurality of offline data tables and the same table field in each offline data table, a Hadoop cluster is created based on the logical relationship, and the Hadoop cluster may execute the logical relationship between the offline data tables to form an offline physical table, for example: the offline data table 1 contains an agent code, the offline data table 2 contains an agent code and agent basic information, and the agent basic information in the offline data table 2 is inserted into the offline data table 2 through the agent code in the offline data table 1 to form an offline physical table.
In the embodiment, the Hadoop cluster has high reliability of data storage and processing capacity according to bits, data are distributed through available computer clusters to complete storage and calculation tasks, the clusters can be conveniently expanded into thousands of nodes, data can be dynamically moved among the nodes, dynamic balance of each node is guaranteed, processing speed is high, efficiency and high expansibility are achieved, in the embodiment, offline data are synchronized to the Hadoop cluster, and when tasks are added for data processing subsequently, the offline data reported by a service system can be directly synchronized to the Hadoop cluster, so that processing efficiency of the offline data is improved, and further subsequent data processing efficiency is improved.
S14, converting the data processing rule into database rule statement, and writing the database rule statement, the real-time physical table and the off-line physical table into a rule table in a database management system.
In this embodiment, the database rule statements are obtained by rule engine conversion, the rule engine is a set of metadata database management, and adds, deletes or modifies the data processing rules through an interface, so as to convert all the data processing rules into executable database rule statements.
In this embodiment, the database management system is a Clickhouse database management system, where the Clickhouse is a column-type database management system for online analysis (OLAP), and stores data in the same column physically, stores data in different columns separately, and can read only required data when subsequently reading data, thereby improving data reading efficiency.
In an optional embodiment, the converting the data processing rule into a database rule statement comprises:
identifying, by a rules engine, each of the data processing rules;
converting each processing rule into a corresponding sub-database rule statement according to a preset conversion mode;
and splicing all the sub-database rule statements to obtain the database rule statements.
In this embodiment, the data processing rule is converted into a database rule statement, for example, the data processing rule 1: acquiring the number of persons reaching a diamond grade agent in 12 months in M years; data processing rule 2: number of policies per diamond-grade agent. And converting the data processing rule 1 and the data processing rule 2 into executable sub-database rule statements according to a preset conversion mode.
In this embodiment, all sub-database rule statements may be spliced according to the data acquisition sequence in the data processing rule.
In this embodiment, the database rule statements, the real-time physical table and the offline physical table are written into a rule table in a database management system, and when the database rule statements are subsequently executed, only data corresponding to the database rule statements need to be extracted, so that the data extraction efficiency is improved, and the data processing efficiency is further improved.
And S15, based on the scheduling mechanism in the rule table, triggering the database rule statement at regular time, and obtaining target rule result data corresponding to the database rule statement from the real-time physical table and the off-line physical table.
In this embodiment, a rule table of the Clickhouse database management system includes a scheduling mechanism, where the scheduling mechanism includes execution time of the database rule statements, and the database rule statements are triggered to obtain target rule result data corresponding to the data processing rule from the real-time physical table and the offline physical table at regular time based on the execution time.
Further, the method further comprises:
and storing the target rule result data into a preset physical table.
In an optional embodiment, the periodically triggering the database rule statement based on a scheduling mechanism in the rule table, and obtaining target rule result data corresponding to the database rule statement from the real-time physical table and the offline physical table includes:
triggering each sub-database rule statement in the database rule statements at regular time, and acquiring corresponding rule result data from the real-time physical table and the off-line physical table based on each sub-database rule statement;
repeatedly executing the sub-database rule statements to acquire corresponding rule result data from the real-time physical table and the off-line physical table based on each sub-database rule statement until the rule result data corresponding to all sub-database rule statements in the database rule statements are acquired;
and determining the rule result data corresponding to all the sub-database rule statements as target rule result data corresponding to the database rule statements.
In this embodiment, a rule table in the database management system is provided with a scheduling mechanism, the scheduling mechanism is provided with an execution time, the database rule statements are triggered at regular time according to the execution time, rule result data corresponding to each sub-database rule statement in the database rule statement set are further obtained, and the rule result data are stored in a preset physical table, so that when the same sub-database rule statement occurs in the following, data extraction is directly performed from the rule result data, a mode of pushing a pre-calculated result is changed from active query, and data processing efficiency is improved.
And S16, responding to the data processing request, and extracting from the target rule result data to obtain a data processing result.
In this embodiment, the data processing request includes a result calling interface, and data extraction is performed from the target rule result data in response to the result calling interface to obtain a data processing result.
Illustratively, the data processing request: calculate the prize money of the diamond grade agent in the longevity agent in 12 months in M years. Since the method for calculating the prize money of the diamond-grade agent in the life insurance agent in month 12 of M needs to acquire the list of the diamond-grade agents in the life insurance agent in month 11 of M if the diamond-grade agent in month 12 is also a diamond-grade agent in month 11, the method for calculating the prize money in month 12 is different from the method for calculating the prize money in month one. The data processing rule corresponding to the data processing request is as follows: obtaining a list of diamond-grade agents in the longevity insurance agents in 11 months in M years; acquiring a list of diamond-grade agents in the longevity insurance agents in 12 months in M years; and acquiring performance information of the diamond-grade agent in the longevity insurance agent in 12 months in M years.
The off-line data reported by the service system comprises all off-line data of M years, and simultaneously receives real-time data of 12 months, and all off-line data of M years form an off-line physical table; and converting the real-time data of 12 months into a real-time physical table.
Converting the data processing rule corresponding to the data processing request into 3 sub-database rule statements, executing the 3 sub-database rule statements, and obtaining result data corresponding to each sub-database rule statement, for example: obtain the list of the diamond grade agents in the longevity insurance agents of 11 months in M years.
Aiming at the data processing request, directly extracting the list of the diamond-grade agents in the life insurance agents in 11 months in M years, the list of the diamond-grade agents in the life insurance agents in 12 months in M years and the performance information of the diamond-grade agents in the life insurance agents in 12 months in M years from the target rule result data.
In other optional embodiments, the real-time stream data may be checked with reference to the offline data.
The data processing method can detect the achievement condition of the incentive scheme in real time according to the performance of the agent, timely and accurately display the result to the manager, the manager does not need to repeatedly process the data to check the result, and the workload and the personal error are reduced, so that the service agent receives timely and sufficient incentive, and the service agent is promoted to improve the work performance.
In summary, in the data processing method according to this embodiment, the real-time data is subjected to the first preprocessing to obtain the real-time physical table, and the offline data is subjected to the second preprocessing to obtain the offline physical table, so that the accuracy of the offline data and the accuracy of the real-time data are improved. And converting the data processing rule into a database rule statement, writing the database rule statement, the real-time physical table and the offline physical table into a rule table in a database management system, and extracting data corresponding to the database rule statement when the database rule statement is subsequently executed, so that the data extraction efficiency is improved, and the data processing efficiency is further improved. And based on a scheduling mechanism in the rule table, regularly triggering the database rule statements to obtain target rule result data corresponding to the database rule statements, and changing active query into a pre-calculation result pushing mode, so that the data processing efficiency is improved. And responding to the data processing request, extracting the target rule result data to obtain a data processing result, wherein the extracted data are pre-calculated, so that the complex query sentence is not required to be executed in real time by the database, the operation efficiency of the database is ensured, and the data processing efficiency is further improved.
Example two
Fig. 2 is a structural diagram of a data processing apparatus according to a second embodiment of the present invention.
In some embodiments, the data processing apparatus 20 may comprise a plurality of functional modules comprised of program code segments. The program code of the various program segments in the data processing device 20 may be stored in a memory of the electronic device and executed by the at least one processor to perform the functions of data processing (described in detail in fig. 1).
In this embodiment, the data processing apparatus 20 may be divided into a plurality of functional modules according to the functions performed by the data processing apparatus. The functional module may include: the device comprises an acquisition module 201, a receiving module 202, a preprocessing module 203, a conversion module 204, a triggering module 205, a storage module 206 and an extraction module 207. The module referred to herein is a series of computer readable instruction segments stored in a memory that can be executed by at least one processor and that can perform a fixed function. In the present embodiment, the functions of the modules will be described in detail in the following embodiments.
The obtaining module 201 is configured to parse the received data processing request, and obtain an identification code and a data processing rule of the service system.
In this embodiment, when a user performs data processing, a data processing request is initiated to a server through a client, specifically, the client may be a smart phone, an IPAD, or other existing intelligent device, the server may be a data processing subsystem, and in a data processing process, for example, the client may send the data processing request to the data processing subsystem, and the data processing subsystem is configured to receive the data processing request sent by the client.
In this embodiment, when the data processing subsystem receives a data processing request sent by a client, the data processing request is analyzed, and an identification code and a data processing rule of a service system are obtained.
Specifically, the identification code is used to uniquely identify the business system, and the data processing rule refers to a rule required for executing the data processing request.
In an optional embodiment, the obtaining module 201 parses the received data processing request, and obtaining the identification code and the data processing rule of the service system includes:
analyzing the message of the data processing request to obtain message information carried by the message;
acquiring an identification code and an interface calling rule from the message information;
and determining a service system according to the identification code, and determining a data processing rule according to the interface calling rule.
In this embodiment, the data processing request includes an identification code and an interface calling rule.
A receiving module 202, configured to receive real-time data reported by the service system in real time corresponding to the identifier code and offline data reported by the service system offline.
In this embodiment, when data processing is performed, it may be necessary to merge real-time data and offline data, so that the real-time data reported by the service system in real time corresponding to the identification code and the offline data reported by the service system offline are received, which facilitates subsequent data processing.
In an optional embodiment, the receiving module 202 receives real-time data reported by a service system in real time corresponding to the identification code and offline data reported by the service system offline, where the receiving module includes:
acquiring a first calling interface list of a plurality of preset first data sources corresponding to real-time data and a second calling interface list of a plurality of preset second data sources corresponding to offline data from a preset data source interface library according to the identification code of the service system;
and the service system corresponding to the identification code reports the real-time data corresponding to each first calling interface in the first calling interface list in real time, and the service system corresponding to the identification code reports the offline data corresponding to each second calling interface in the second calling interface list in an offline manner.
In this embodiment, the plurality of preset first data sources may be the same as the plurality of preset second data sources, or may be different from the plurality of preset second data sources, where each preset first data source corresponds to one first call interface, and each preset second data source corresponds to one second call interface.
In this embodiment, each preset first data source corresponds to one first calling interface, and each preset second data source corresponds to one second calling interface, so that the problem of data blocking caused by that all preset data sources call data through one calling interface and report the data is avoided, and the efficiency of data reporting is improved.
The preprocessing module 203 is configured to perform a first preprocessing on the real-time data to obtain a real-time physical table, and perform a second preprocessing on the offline data to obtain an offline physical table.
In this embodiment, the physical table includes a real-time physical table and an offline physical table, where the physical table refers to a table in a specific data source.
In an optional embodiment, the preprocessing module 203 performs a first preprocessing on the real-time data to obtain a real-time physical table, including:
identifying a type of the real-time data;
if the type of the real-time data is static data, writing the real-time data into a message queue Kafka to form a first real-time physical table, and determining the first real-time physical table as a real-time physical table; or
And if the type of the real-time data is dynamic data, consuming the real-time data through a distributed processing engine Flink to form a second real-time physical table, and determining the second real-time physical table as a real-time physical table.
In this embodiment, the real-time data includes two types, one type is static data, and specifically, the static data refers to data that does not need to be processed; one type is dynamic data, which refers to data that needs to be processed.
In this embodiment, the distributed processing engine Flink includes four different components, namely, a distributor, a job manager, a resource manager, and a task manager, and the four components are run on a server to process the real-time data to form a second real-time physical table, specifically, the consuming the real-time data by the distributed processing engine Flink to form the second real-time physical table includes:
importing the real-time data into a distributed processing engine (Flink), and identifying table names and logical relations among table fields of a plurality of real-time data tables in the real-time data;
and when an idle slot is obtained in the distributed processing engine Flink, executing the logical relationship between the table names and the table fields of the plurality of real-time data tables in the idle slot to form a second real-time physical table.
In this embodiment, the real-time data is imported into a distributor of a distributed processing engine Flink, and the distributor reports the real-time data to an operation manager of the distributed processing engine Flink; the operation manager identifies the table names and the table fields of a plurality of real-time data tables in the real-time data, and applies for resources from the resource manager of the distributed processing engine flight according to the logical relationship between the table names and the table fields of the plurality of real-time data tables; the resource manager starts a task manager of the distributed processing engine Flink based on the applied resources; the task manager registers information to the resource manager, and the resource manager sends a command for requesting to acquire an idle slot to the task manager of the distributed processing engine Flink according to the registration information; the task manager sends an idle slot to the job manager of the distributed processing engine Flink in response to the instruction for acquiring the idle slot, and the job manager executes a logical relationship between table names and table fields of the multiple real-time data tables in the idle slot to form a second real-time physical table, where the distributed processing engine Flink is a prior art, and this embodiment is not described in detail herein.
In this embodiment, the slot of the space refers to an idle network interface in the task manager, and may be used to execute the table names and the logical relationships between the table fields of the multiple real-time data tables to form a second real-time physical table.
In this embodiment, the distributed processing engine Flink is used for performing stateful computation on the real-time data stream, for example: the method comprises the steps that the performance of an agent M is obtained, a real-time data table 1 reported by a preset first data source A contains agent serial numbers and policy information, a real-time data table 2 reported by a preset first data source B contains agent basic information, and in order to determine the performance of the agent M, the real-time data table 1 and the real-time data table 2 need to be correlated, so that a distributed processing engine Flink consumption is adopted to correlate the real-time data table 1 with the real-time data table 2, and a real-time physical table is obtained.
In the embodiment, the distributed processing engine Flink can process streaming data with low delay and high throughput, and can process out-of-order data well to ensure the consistency of real-time data states.
In an optional embodiment, the second preprocessing is performed on the offline data by the preprocessing module 203, and obtaining the offline physical table includes:
identifying a table name and a table field of each of a plurality of offline data tables in the offline data;
and creating a Hadoop cluster according to the table name and the logic relation between the table fields of each offline data table in the plurality of offline data tables, and synchronizing the offline data into the created Hadoop cluster to form an offline physical table.
In this embodiment, the offline data includes a plurality of offline data tables, each offline data table corresponds to a table name and a table field, the logical relationship refers to a logical relationship between the plurality of offline data tables and the same table field in each offline data table, a Hadoop cluster is created based on the logical relationship, and the Hadoop cluster may execute the logical relationship between the offline data tables to form an offline physical table, for example: the offline data table 1 contains an agent code, the offline data table 2 contains an agent code and agent basic information, and the agent basic information in the offline data table 2 is inserted into the offline data table 2 through the agent code in the offline data table 1 to form an offline physical table.
In the embodiment, the Hadoop cluster has high reliability of data storage and processing capacity according to bits, data are distributed through available computer clusters to complete storage and calculation tasks, the clusters can be conveniently expanded into thousands of nodes, data can be dynamically moved among the nodes, dynamic balance of each node is guaranteed, processing speed is high, efficiency and high expansibility are achieved, in the embodiment, offline data are synchronized to the Hadoop cluster, and when tasks are added for data processing subsequently, the offline data reported by a service system can be directly synchronized to the Hadoop cluster, so that processing efficiency of the offline data is improved, and further subsequent data processing efficiency is improved.
A conversion module 204, configured to convert the data processing rule into a database rule statement, and write the database rule statement, the real-time physical table, and the offline physical table into a rule table in a database management system.
In this embodiment, the database rule statements are obtained by rule engine conversion, the rule engine is a set of metadata database management, and adds, deletes or modifies the data processing rules through an interface, so as to convert all the data processing rules into executable database rule statements.
In this embodiment, the database management system is a Clickhouse database management system, where the Clickhouse is a column-type database management system for online analysis (OLAP), and stores data in the same column physically, stores data in different columns separately, and can read only required data when subsequently reading data, thereby improving data reading efficiency.
In an alternative embodiment, the converting module 204 converts the data processing rule into a database rule statement comprising:
identifying, by a rules engine, each of the data processing rules;
converting each processing rule into a corresponding sub-database rule statement according to a preset conversion mode;
and splicing all the sub-database rule statements to obtain the database rule statements.
In this embodiment, the data processing rule is converted into a database rule statement, for example, the data processing rule 1: acquiring the number of persons reaching a diamond grade agent in 12 months in M years; data processing rule 2: number of policies per diamond-grade agent. And converting the data processing rule 1 and the data processing rule 2 into executable sub-database rule statements according to a preset conversion mode.
In this embodiment, all sub-database rule statements may be spliced according to the data acquisition sequence in the data processing rule.
In this embodiment, the database rule statements, the real-time physical table and the offline physical table are written into a rule table in a database management system, and when the database rule statements are subsequently executed, only data corresponding to the database rule statements need to be extracted, so that the data extraction efficiency is improved, and the data processing efficiency is further improved.
A triggering module 205, configured to trigger the database rule statement at regular time based on a scheduling mechanism in the rule table, and obtain target rule result data corresponding to the database rule statement from the real-time physical table and the offline physical table.
In this embodiment, a rule table of the Clickhouse database management system includes a scheduling mechanism, where the scheduling mechanism includes execution time of the database rule statements, and the database rule statements are triggered to obtain target rule result data corresponding to the data processing rule from the real-time physical table and the offline physical table at regular time based on the execution time.
A storage module 206, configured to store the target rule result data in a preset physical table.
In an optional embodiment, the triggering module 205 regularly triggers the database rule statement based on a scheduling mechanism in the rule table, and obtaining target rule result data corresponding to the database rule statement from the real-time physical table and the offline physical table includes:
triggering each sub-database rule statement in the database rule statements at regular time, and acquiring corresponding rule result data from the real-time physical table and the off-line physical table based on each sub-database rule statement;
repeatedly executing the sub-database rule statements to acquire corresponding rule result data from the real-time physical table and the off-line physical table based on each sub-database rule statement until the rule result data corresponding to all sub-database rule statements in the database rule statements are acquired;
and determining the rule result data corresponding to all the sub-database rule statements as target rule result data corresponding to the database rule statements.
In this embodiment, a rule table in the database management system is provided with a scheduling mechanism, the scheduling mechanism is provided with an execution time, the database rule statements are triggered at regular time according to the execution time, rule result data corresponding to each sub-database rule statement in the database rule statement set are further obtained, and the rule result data are stored in a preset physical table, so that when the same sub-database rule statement occurs in the following, data extraction is directly performed from the rule result data, a mode of pushing a pre-calculated result is changed from active query, and data processing efficiency is improved.
And the extraction module 207 is configured to extract the target rule result data in response to the data processing request to obtain a data processing result.
In this embodiment, the data processing request includes a result calling interface, and data extraction is performed from the target rule result data in response to the result calling interface to obtain a data processing result.
Illustratively, the data processing request: calculate the prize money of the diamond grade agent in the longevity agent in 12 months in M years. Since the method for calculating the prize money of the diamond-grade agent in the life insurance agent in month 12 of M needs to acquire the list of the diamond-grade agents in the life insurance agent in month 11 of M if the diamond-grade agent in month 12 is also a diamond-grade agent in month 11, the method for calculating the prize money in month 12 is different from the method for calculating the prize money in month one. The data processing rule corresponding to the data processing request is as follows: obtaining a list of diamond-grade agents in the longevity insurance agents in 11 months in M years; acquiring a list of diamond-grade agents in the longevity insurance agents in 12 months in M years; and acquiring performance information of the diamond-grade agent in the longevity insurance agent in 12 months in M years.
The off-line data reported by the service system comprises all off-line data of M years, and simultaneously receives real-time data of 12 months, and all off-line data of M years form an off-line physical table; and converting the real-time data of 12 months into a real-time physical table.
Converting the data processing rule corresponding to the data processing request into 3 sub-database rule statements, executing the 3 sub-database rule statements, and obtaining result data corresponding to each sub-database rule statement, for example: obtain the list of the diamond grade agents in the longevity insurance agents of 11 months in M years.
Aiming at the data processing request, directly extracting the list of the diamond-grade agents in the life insurance agents in 11 months in M years, the list of the diamond-grade agents in the life insurance agents in 12 months in M years and the performance information of the diamond-grade agents in the life insurance agents in 12 months in M years from the target rule result data.
In other optional embodiments, the real-time stream data may be checked with reference to the offline data.
The data processing method can detect the achievement condition of the incentive scheme in real time according to the performance of the agent, timely and accurately display the result to the manager, the manager does not need to repeatedly process the data to check the result, and the workload and the personal error are reduced, so that the service agent receives timely and sufficient incentive, and the service agent is promoted to improve the work performance.
In summary, the data processing apparatus according to this embodiment obtains the real-time physical table by performing the first preprocessing on the real-time data, and obtains the offline physical table by performing the second preprocessing on the offline data, thereby improving the accuracy of the offline data and the real-time data. And converting the data processing rule into a database rule statement, writing the database rule statement, the real-time physical table and the offline physical table into a rule table in a database management system, and extracting data corresponding to the database rule statement when the database rule statement is subsequently executed, so that the data extraction efficiency is improved, and the data processing efficiency is further improved. And based on a scheduling mechanism in the rule table, regularly triggering the database rule statements to obtain target rule result data corresponding to the database rule statements, and changing active query into a pre-calculation result pushing mode, so that the data processing efficiency is improved. And responding to the data processing request, extracting the target rule result data to obtain a data processing result, wherein the extracted data are pre-calculated, so that the complex query sentence is not required to be executed in real time by the database, the operation efficiency of the database is ensured, and the data processing efficiency is further improved.
EXAMPLE III
Fig. 3 is a schematic structural diagram of an electronic device according to a third embodiment of the present invention. In the preferred embodiment of the present invention, the electronic device 3 comprises a memory 31, at least one processor 32, at least one communication bus 33 and a transceiver 34.
It will be appreciated by those skilled in the art that the configuration of the electronic device shown in fig. 3 does not constitute a limitation of the embodiment of the present invention, and may be a bus-type configuration or a star-type configuration, and the electronic device 3 may include more or less other hardware or software than those shown, or a different arrangement of components.
In some embodiments, the electronic device 3 is an electronic device capable of automatically performing numerical calculation and/or information processing according to instructions set or stored in advance, and the hardware thereof includes but is not limited to a microprocessor, an application specific integrated circuit, a programmable gate array, a digital processor, an embedded device, and the like. The electronic device 3 may also include a client device, which includes, but is not limited to, any electronic product that can interact with a client through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device, for example, a personal computer, a tablet computer, a smart phone, a digital camera, and the like.
It should be noted that the electronic device 3 is only an example, and other existing or future electronic products, such as those that can be adapted to the present invention, should also be included in the scope of the present invention, and are included herein by reference.
In some embodiments, the memory 31 is used for storing program codes and various data, such as the data processing device 20 installed in the electronic equipment 3, and realizes high-speed and automatic access to programs or data during the operation of the electronic equipment 3. The Memory 31 includes a Read-Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), a One-time Programmable Read-Only Memory (OTPROM), an electronically Erasable rewritable Read-Only Memory (Electrically-Erasable Programmable Read-Only Memory (EEPROM)), an optical Read-Only disk (CD-ROM) or other optical disk Memory, a magnetic disk Memory, a tape Memory, or any other medium readable by a computer capable of carrying or storing data.
In some embodiments, the at least one processor 32 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The at least one processor 32 is a Control Unit (Control Unit) of the electronic device 3, connects various components of the electronic device 3 by using various interfaces and lines, and executes various functions and processes data of the electronic device 3 by running or executing programs or modules stored in the memory 31 and calling data stored in the memory 31.
In some embodiments, the at least one communication bus 33 is arranged to enable connection communication between the memory 31 and the at least one processor 32 or the like.
Although not shown, the electronic device 3 may further include a power supply (such as a battery) for supplying power to each component, and optionally, the power supply may be logically connected to the at least one processor 32 through a power management device, so as to implement functions of managing charging, discharging, and power consumption through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device 3 may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
The integrated unit implemented in the form of a software functional module may be stored in a computer-readable storage medium. The software functional module is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, an electronic device, or a network device) or a processor (processor) to execute parts of the methods according to the embodiments of the present invention.
In a further embodiment, in conjunction with fig. 2, the at least one processor 32 may execute an operating device of the electronic device 3 and various installed application programs (such as the data processing device 20), program codes, and the like, for example, the above modules.
The memory 31 has program code stored therein, and the at least one processor 32 can call the program code stored in the memory 31 to perform related functions. For example, the modules illustrated in fig. 2 are program codes stored in the memory 31 and executed by the at least one processor 32, so as to realize the functions of the modules for the purpose of data processing.
Illustratively, the program code may be partitioned into one or more modules/units that are stored in the memory 31 and executed by the processor 32 to accomplish the present application. The one or more modules/units may be a series of computer readable instruction segments capable of performing certain functions, which are used for describing the execution process of the program code in the electronic device 3. For example, the program code may be partitioned into an acquisition module 201, a receiving module 202, a pre-processing module 203, a conversion module 204, a triggering module 205, a storage module 206, and an extraction module 207.
In one embodiment of the invention, the memory 31 stores a plurality of computer readable instructions that are executed by the at least one processor 32 to implement the functionality of data processing.
Specifically, the at least one processor 32 may refer to the description of the relevant steps in the embodiment corresponding to fig. 1, and details are not repeated here.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is obvious that the word "comprising" does not exclude other elements or that the singular does not exclude the plural. A plurality of units or means recited in the present invention may also be implemented by one unit or means through software or hardware. The terms first, second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A method of data processing, the method comprising:
analyzing the received data processing request to acquire an identification code and a data processing rule of a service system;
receiving real-time data reported by a service system corresponding to the identification code in real time and offline data reported by the service system in an offline manner;
performing first pretreatment on the real-time data to obtain a real-time physical table, and performing second pretreatment on the offline data to obtain an offline physical table;
converting the data processing rule into a database rule statement, and writing the database rule statement, the real-time physical table and the off-line physical table into a rule table in a database management system;
based on a scheduling mechanism in the rule table, regularly triggering the database rule statement, and obtaining target rule result data corresponding to the database rule statement from the real-time physical table and the off-line physical table;
and responding to the data processing request, and extracting from the target rule result data to obtain a data processing result.
2. The data processing method of claim 1, wherein the receiving of the real-time data reported by the service system corresponding to the identification code in real time and the off-line data reported by the service system in off-line comprises:
acquiring a first calling interface list of a plurality of preset first data sources corresponding to real-time data and a second calling interface list of a plurality of preset second data sources corresponding to offline data from a preset data source interface library according to the identification code of the service system;
and the service system corresponding to the identification code reports the real-time data corresponding to each first calling interface in the first calling interface list in real time, and the service system corresponding to the identification code reports the offline data corresponding to each second calling interface in the second calling interface list in an offline manner.
3. The data processing method of claim 1, wherein the performing a first preprocessing on the real-time data to obtain a real-time physical table comprises:
identifying a type of the real-time data;
if the type of the real-time data is static data, writing the real-time data into a message queue Kafka to form a first real-time physical table, and determining the first real-time physical table as a real-time physical table; or
And if the type of the real-time data is dynamic data, consuming the real-time data through a distributed processing engine Flink to form a second real-time physical table, and determining the second real-time physical table as a real-time physical table.
4. The data processing method of claim 3, wherein said consuming said real-time data by a distributed processing engine Flink to form a second real-time physical table comprises:
importing the real-time data into a distributed processing engine (Flink), and identifying table names and logical relations among table fields of a plurality of real-time data tables in the real-time data;
and when an idle slot is obtained in the distributed processing engine Flink, executing the logical relationship between the table names and the table fields of the plurality of real-time data tables in the idle slot to form a second real-time physical table.
5. The data processing method of claim 1, wherein the second preprocessing the offline data to obtain an offline physical table comprises:
identifying a table name and a table field of each of a plurality of offline data tables in the offline data;
and creating a Hadoop cluster according to the table name and the logic relation between the table fields of each offline data table in the plurality of offline data tables, and synchronizing the offline data into the created Hadoop cluster to form an offline physical table.
6. The data processing method of claim 1, wherein said converting the data processing rule into a database rule statement comprises:
identifying, by a rules engine, each of the data processing rules;
converting each processing rule into a corresponding sub-database rule statement according to a preset conversion mode;
and splicing all the sub-database rule statements to obtain the database rule statements.
7. The data processing method of claim 6, wherein the periodically triggering the database rule statement based on the scheduling mechanism in the rule table, and obtaining the target rule result data corresponding to the database rule statement from the real-time physical table and the offline physical table comprises:
triggering each sub-database rule statement in the database rule statements at regular time, and acquiring corresponding rule result data from the real-time physical table and the off-line physical table based on each sub-database rule statement;
repeatedly executing the sub-database rule statements to acquire corresponding rule result data from the real-time physical table and the off-line physical table based on each sub-database rule statement until the rule result data corresponding to all sub-database rule statements in the database rule statements are acquired;
and determining the rule result data corresponding to all the sub-database rule statements as target rule result data corresponding to the database rule statements.
8. A data processing apparatus, characterized in that the apparatus comprises:
the acquisition module is used for analyzing the received data processing request and acquiring the identification code and the data processing rule of the service system;
the receiving module is used for receiving real-time data reported by the service system corresponding to the identification code in real time and offline data reported by the service system in an offline manner;
the preprocessing module is used for performing first preprocessing on the real-time data to obtain a real-time physical table and performing second preprocessing on the off-line data to obtain an off-line physical table;
the conversion module is used for converting the data processing rule into a database rule statement and writing the database rule statement, the real-time physical table and the off-line physical table into a rule table in a database management system;
the triggering module is used for regularly triggering the database rule statements based on a scheduling mechanism in the rule table and obtaining target rule result data corresponding to the database rule statements from the real-time physical table and the off-line physical table;
and the extraction module is used for responding to the data processing request and extracting from the target rule result data to obtain a data processing result.
9. An electronic device, characterized in that the electronic device comprises a processor and a memory, the processor being configured to implement the data processing method of any one of claims 1 to 7 when executing the computer program stored in the memory.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data processing method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210089970.8A CN114416849A (en) | 2022-01-25 | 2022-01-25 | Data processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210089970.8A CN114416849A (en) | 2022-01-25 | 2022-01-25 | Data processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114416849A true CN114416849A (en) | 2022-04-29 |
Family
ID=81276975
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210089970.8A Pending CN114416849A (en) | 2022-01-25 | 2022-01-25 | Data processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114416849A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115455010A (en) * | 2022-11-09 | 2022-12-09 | 以萨技术股份有限公司 | Data processing method based on milvus database, electronic equipment and storage medium |
| CN116629805A (en) * | 2023-06-07 | 2023-08-22 | 浪潮智慧科技有限公司 | Water conservancy index service method, equipment and medium for distributed flow batch integration |
-
2022
- 2022-01-25 CN CN202210089970.8A patent/CN114416849A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115455010A (en) * | 2022-11-09 | 2022-12-09 | 以萨技术股份有限公司 | Data processing method based on milvus database, electronic equipment and storage medium |
| CN115455010B (en) * | 2022-11-09 | 2023-02-28 | 以萨技术股份有限公司 | Data processing method based on milvus database, electronic equipment and storage medium |
| CN116629805A (en) * | 2023-06-07 | 2023-08-22 | 浪潮智慧科技有限公司 | Water conservancy index service method, equipment and medium for distributed flow batch integration |
| CN116629805B (en) * | 2023-06-07 | 2023-12-01 | 浪潮智慧科技有限公司 | Water conservancy index service method, equipment and medium for distributed flow batch integration |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111625331B (en) | Task scheduling method, device, platform, server and storage medium | |
| CN114416849A (en) | Data processing method and device, electronic equipment and storage medium | |
| EP2763055B1 (en) | A telecommunication method and mobile telecommunication device for providing data to a mobile application | |
| CN111797157B (en) | Data processing method, system, electronic equipment and storage medium | |
| CN111061762A (en) | Distributed task processing method, related device, system and storage medium | |
| CN105868070A (en) | Method and apparatus for determining resources consumed by tasks | |
| CN115335821A (en) | Offloading statistics collection | |
| CN112612462A (en) | Method and device for adjusting phone configuration, electronic equipment and storage medium | |
| JP2009037369A (en) | Resource assignment method to database server | |
| CN113760491A (en) | Task scheduling system, method, equipment and storage medium | |
| CN115858499A (en) | Database partition processing method and device, computer equipment and storage medium | |
| CN115237706A (en) | Buried point data processing method and device, electronic equipment and storage medium | |
| CN113722141B (en) | Method and device for determining delay reason of data task, electronic equipment and medium | |
| CN115617480A (en) | Task scheduling method, device and system and storage medium | |
| CN115438056A (en) | Data acquisition method, device, equipment and storage medium | |
| CN113268478A (en) | Big data analysis method and device, electronic equipment and storage medium | |
| CN114881313A (en) | Behavior prediction method and device based on artificial intelligence and related equipment | |
| CN114925674A (en) | File compliance checking method and device, electronic equipment and storage medium | |
| CN114896164A (en) | Interface optimization method and device, electronic equipment and storage medium | |
| US20210312365A1 (en) | Analysis of resources utilized during execution of a process | |
| US11475017B2 (en) | Asynchronous data enrichment for an append-only data store | |
| CN113835953A (en) | Statistical method and device of job information, computer equipment and storage medium | |
| CN113220436A (en) | Universal batch operation execution method and device under distributed environment | |
| CN113282282B (en) | Business requirement form publishing method and device, electronic equipment and storage medium | |
| CN114461717A (en) | Automatic data comparison method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |