CN114490663A

CN114490663A - Data processing method and device

Info

Publication number: CN114490663A
Application number: CN202210132717.6A
Authority: CN
Inventors: 马宇申
Original assignee: Alipay Hangzhou Information Technology Co Ltd
Current assignee: Alipay Hangzhou Information Technology Co Ltd
Priority date: 2022-02-14
Filing date: 2022-02-14
Publication date: 2022-05-13

Abstract

The present specification provides a data processing method and apparatus, wherein the data processing method includes: acquiring index data and detail data corresponding to various data sources; processing the index class data of each data source according to the storage structure of the index database to obtain target index class data; processing the detail data of each data source according to a storage structure of the column-type database to obtain target detail data; writing the target index class data to the index database, and writing the target detail class data to the columnar database.

Description

Data processing method and device

Technical Field

The present disclosure relates to the field of compliance data management technologies, and in particular, to a data processing method and apparatus.

Background

With the development of the internet technology, more and more users select online consumption, and different from the traditional entity consumption scene, the users can finish shopping, reservation, stock investment, certificate transaction and the like through the internet, so that the convenience of the users is improved to a great extent. As the development of online services is deepened, more and more data are involved, and the associated data are different in different business scenes. When the service needs to be examined, data summarization and reporting are carried out on each node subordinate to the service; in the prior art, due to different data sources of each node, reported data have different data structures, and audit processing cannot be completed quickly, so an effective scheme is urgently needed to solve the above problems.

Disclosure of Invention

In view of this, the embodiments of the present specification provide a data processing method. The present specification also relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program to solve the technical problems of the prior art.

According to a first aspect of embodiments herein, there is provided a data processing method including:

acquiring index data and detail data corresponding to various data sources;

processing the index class data of each data source according to the storage structure of the index database to obtain target index class data;

processing the detail data of each data source according to a storage structure of the column-type database to obtain target detail data;

writing the target index class data to the index database, and writing the target detail class data to the columnar database.

Optionally, the obtaining of the index class data and the detail class data corresponding to the multiple data sources includes:

receiving a data reporting request submitted by aiming at a target service;

sending a data uploading request to at least two service nodes associated with a target service according to the data reporting request;

receiving service data returned by each service node according to the data reporting request;

and classifying the service data to obtain index data and detail data corresponding to various data sources.

Optionally, the processing the index class data of each data source according to the storage structure of the index database to obtain the target index class data includes:

determining a conversion rule between a storage structure of each data source and a storage structure of the index database;

and processing the index class data of each data source according to the conversion rule to obtain the target index class data with the same storage structure as the index database.

Optionally, the processing the detail class data of each data source according to the storage structure of the column database to obtain the target detail class data includes:

synchronizing the detail data corresponding to each data source to an intermediate service database;

converting the detail data in the intermediate service database according to the structure conversion rule corresponding to each data source;

and generating the target detail data with the same storage structure as the columnar database according to the conversion result.

Optionally, the converting the detail data in the intermediate service database according to the structure conversion rule corresponding to each data source includes:

analyzing the detail data in the intermediate service database to obtain initial subdata;

and screening target subdata from the initial subdata, and converting the target subdata according to a structure conversion rule corresponding to each data source.

Optionally, after the steps of writing the target index class data into the index database and writing the target detail class data into the column database are executed, the method further includes:

under the condition that the running states of the index database and the column-type database are in a stop state, extracting index class data to be reported from the index database according to a preset reporting period, and extracting detail class data to be reported from the column-type database;

integrating the index data to be reported and the detail data to be reported to obtain service data to be reported;

and sending the service data to be reported to a reporting node.

Optionally, before the step of sending the service data to be delivered to a delivery node is executed, the method further includes:

acquiring an index detection strategy corresponding to the index class data to be reported and a detail detection strategy corresponding to the detail class data to be reported;

detecting the index class data to be reported according to the index detection strategy to obtain an index class data detection result, an

Detecting the detail data to be reported according to the detail detection strategy to obtain a detail data detection result;

and sending the index data detection result and the detail data detection result to a service regulation node.

Optionally, the detecting the to-be-reported index data according to the index detection policy to obtain an index data detection result includes:

reading historical index class data according to the index detection strategy;

and comparing the historical index column data with the index class data to be reported, and determining the detection result of the index class data according to the comparison result.

Optionally, the detecting the detail data to be reported according to the detail detection policy to obtain a detail data detection result includes:

creating a structure detection condition and a rule detection condition according to the detail detection strategy;

and detecting the detail data to be reported based on the structure detection condition and the rule detection condition to obtain a detail data detection result.

According to a second aspect of embodiments herein, there is provided a data processing apparatus comprising:

the data acquisition module is configured to acquire index data and detail data corresponding to various data sources;

the first processing module is configured to process the index class data of each data source according to the storage structure of the index database to obtain target index class data;

the second processing module is configured to process the detail data of each data source according to the storage structure of the columnar database to obtain target detail data;

a data write module configured to write the target index class data to the index database and the target detail class data to the columnar database.

According to a third aspect of embodiments herein, there is provided a computing device comprising:

a memory and a processor;

the memory is used for storing computer-executable instructions, and the processor is used for realizing the steps of the data processing method when executing the computer-executable instructions.

According to a fourth aspect of embodiments herein, there is provided a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the data processing method.

According to a fifth aspect of embodiments herein, there is provided a computer program which, when executed in a computer, causes the computer to perform the steps of the data processing method.

In the data processing method provided by the present specification, after the index class data and the detail class data corresponding to the multiple data sources are obtained, the index class data of the multiple data sources can be processed according to the storage result of the index database to obtain the target index class data of the unified data structure; meanwhile, detail data of each data source is processed according to a storage structure of the column-type database to obtain target index data of a unified data structure, and finally the data are respectively written into the corresponding databases, so that service-related data are unified according to lighting details and index data, the influence efficiency of multiple data sources is avoided when the subsequent service is audited or data is reported, the nodes such as data checking or audit can be conveniently and quickly checked or audited, and the data processing efficiency is further improved.

Drawings

Fig. 1 is a schematic diagram of a data processing method provided in an embodiment of the present specification;

FIG. 2 is a flow chart of a data processing method provided in an embodiment of the present description;

FIG. 3 is a flowchart illustrating a data processing method applied in a data delivery scenario according to an embodiment of the present disclosure;

fig. 4 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present disclosure;

fig. 5 is a block diagram of a computing device according to an embodiment of the present disclosure.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. This description may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, as those skilled in the art will be able to make and use the present disclosure without departing from the spirit and scope of the present disclosure.

The terminology used in the description of the one or more embodiments is for the purpose of describing the particular embodiments only and is not intended to be limiting of the description of the one or more embodiments. As used in one or more embodiments of the present specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present specification refers to and encompasses any and all possible combinations of one or more of the associated listed items.

It will be understood that, although the terms first, second, etc. may be used herein in one or more embodiments to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first can also be referred to as a second and, similarly, a second can also be referred to as a first without departing from the scope of one or more embodiments of the present description. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.

First, the noun terms to which one or more embodiments of the present specification relate are explained.

A data source: (Data Source), the Source of the Data, is the device or original media that provides some desired Data. All information for establishing a database connection is stored in the data source. Just as files can be found in a file system by specifying the file name, the corresponding database connection can be found by providing the correct data source name.

A database: is a repository that organizes, stores, and manages data according to a data structure. Is an organized, sharable, uniformly managed collection of large amounts of data that is stored long term within a computer.

In the present specification, a data processing method is provided, and the present specification relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program, which are described in detail one by one in the following embodiments.

In practical applications, different business scenarios have different numbers of data processing nodes, and as the number of data processing nodes increases, the data processing nodes face different data sources, that is, databases of data stored in the data processing nodes may be different, such as relational databases (MySQL, maridb) or non-relational databases (Cassandra, MongoDB). When service needs to be audited or data reported, the problem of multiple data sources is faced, so that auditing or data reporting efficiency is affected to a great extent, and an effective scheme is urgently needed to solve the problem.

Referring to the schematic diagram of the data processing method shown in fig. 1, after the index class data and the detail class data corresponding to the multiple data sources are obtained, the index class data of the multiple data sources may be processed according to the storage result of the index database to obtain the target index class data of the unified data structure; meanwhile, detail data of each data source is processed according to a storage structure of the column-type database to obtain target index data of a unified data structure, and finally the data are respectively written into the corresponding databases, so that service-related data are unified according to lighting details and index data, the influence efficiency of multiple data sources is avoided when the subsequent service is audited or data is reported, the nodes such as data checking or audit can be conveniently and quickly checked or audited, and the data processing efficiency is further improved.

Fig. 2 is a flowchart illustrating a data processing method according to an embodiment of the present specification, which specifically includes the following steps:

step S202, index data and detail data corresponding to various data sources are obtained.

Specifically, the multiple data sources specifically refer to databases corresponding to data processing nodes included in the same service project; correspondingly, the index type data specifically refers to data related to quantity information in the business project, and the detail type data specifically refers to data related to business information in the business project.

For example, the business item is a payment service provided by a certain payment platform, a seller can put goods on the payment platform, and a buyer can purchase the goods provided by the seller through the payment platform. Based on this, in order to support the payment platform to provide payment services to the buyer and the seller, different nodes may be arranged for business parties of the payment platform to manage the payment services, for example, the node for managing the goods is responsible for data related to the goods, the node for managing the orders is responsible for data related to the orders, the node for managing the shipment is responsible for data related to logistics, the node for managing the information of the buyer or the seller is responsible for data related to personal information, and the like, and different nodes may manage different types of data, thereby supporting normal development of business items.

And each node usually selects a database matched with the data processed by the current node to be specially used for storing the part of data in order to facilitate data storage and backup, and when data reporting or auditing is required to be performed on a business project, each node can directly extract data from the database for reporting, so that data related to the business project corresponding to various data sources can be obtained at this time, and the part of data is divided according to index classes and detail classes, so that data unification can be performed subsequently, and auditing or reporting efficiency is improved.

If nodes related to the service project respectively correspond to the data sources DS 1-DSn (wherein each DS respectively corresponds to one database, and the types of the databases are different), data corresponding to the data sources DS 1-DSn are obtained, and then the data can be divided according to the index class and the detail class, so that index class data and detail class data corresponding to various data sources are obtained.

It should be noted that any one service project may correspond to multiple data sources, and when auditing or data reporting of the service project is required, reference may be made to the description contents related to this specification, which is not described in detail herein.

Based on this, no matter in an audit or a data submission scene, when data related to a service project needs to have statistical requirements, the problem of non-uniform data structures caused by different data sources is faced, so that the statistical efficiency is reduced to a great extent.

Further, in a data reporting scenario, when data related to a project needs to be counted, in order to improve data processing efficiency, a manner of requesting each node to actively send data may be implemented to obtain data corresponding to multiple data sources, in this embodiment, a specific implementation manner is as follows:

receiving a data submission request submitted aiming at a target service;

Specifically, the target service specifically refers to a service item which needs to be subjected to data reporting at present; correspondingly, the service node specifically refers to a data processing node supporting the target service to provide service for the user, and each service node corresponds to a data source for storing data related to the node; the service data specifically refers to all data collected or generated in the running process of the target service.

Based on this, under the condition that a data reporting request submitted by the management platform for the target service is received, it indicates that the management platform needs to perform risk prediction or data statistics on the target service at this time, where the data reporting request may include a data statistics period, so that when data is uploaded by each service node of the target service, the data reporting can be completed according to the data statistics period; at this time, the service platform may send a data upload request to each service node associated with the target service according to the data report request, and receive service data returned by each service node in response to the data upload request.

Each service node corresponds to at least one data source, so that the returned service data corresponds to multiple data sources, and in order to facilitate subsequent statistics, the service data can be classified according to the index class and the detail class, so that the index class data corresponding to the multiple data sources and the detail class data corresponding to the multiple data sources are obtained, and the subsequent data processing efficiency is improved.

In the above example, the customer number data of the corresponding data source DS1 returned by the customer data management node is received; the transaction data management node returns transaction limit data corresponding to the data source DS 2; the data of the number of active users of the corresponding DS3 returned by the user data management node; order data of the corresponding DS4 returned by the order data management node; the merchant data of the corresponding data source DS5 returned by the merchant data management node; address data of the corresponding data source DS6 returned by the logistics data management node; in order to improve the subsequent submission efficiency, the data can be classified according to the lighting details and the index class at this time, so as to obtain index class data { transaction amount data, customer quantity data and active user quantity data } corresponding to the data sources DS1, DS2 and DS 3; detail data { order data, business data, address data } corresponding to data sources DS4, DS5, and DS6 is obtained.

In conclusion, by separating the service data returned by each service node, detailed data and index data corresponding to various data sources are obtained, preliminary data statistics from two dimensions in a preprocessing mode is realized, data unification is conveniently carried out on the basis of the preliminary data statistics, and the data processing efficiency is effectively improved.

And step S204, processing the index class data of each data source according to the storage structure of the index database to obtain target index class data.

Specifically, after the index class data corresponding to the multiple data sources are obtained, because different data sources correspond to different data structures, if the index class data corresponding to the multiple data sources are directly stored, incompatibility may occur, or even data collision may be caused to cause data loss, so that in order to facilitate downstream service usage, the index class data corresponding to the different data sources may be uniformly processed according to the storage structure of the index database according to the current statistical requirements, so as to obtain the target index class data having the same data structure.

The index database is specifically a database for storing index data, and the storage structure of the database is only one, so that the data stored in the database can be unified according to the storage structure, and management and persistence are facilitated; correspondingly, the target index class data is the index class data expressed by the storage structure of the index database.

Further, before writing the index class data corresponding to the multiple data sources into the index database, the index class data of each data source needs to be converted into a data structure that is the same as the storage structure of the index database, so that the data can be persisted, and the data corresponding to different data sources needs to be implemented according to different rules when the data conversion is performed, in this embodiment, the specific implementation manner is as follows:

Specifically, the conversion rule specifically refers to a conversion mode between a storage structure of each data source and a storage structure of the index database, and the data structure of the data source corresponding to the conversion rule can be converted into an expression identical to the storage structure of the index database through the conversion rule, so that the index database can store the converted index class data conveniently.

Based on this, after the index class data corresponding to each data source is obtained, the conversion rule between the storage structure of each data source and the storage structure of the index database can be read, then the index class data corresponding to the conversion rule is processed according to the conversion rule, that is, the index class data corresponding to the conversion rule can be converted into the target index class data which is the same as the storage structure of the index database, and so on, the index class data of each data source is processed according to the conversion rule corresponding to the index class data, and when the index class data corresponding to all the data sources are converted into the target index class data, the target index class data can be stored in the index database.

Along the use example, after index class data { transaction amount data, customer quantity data and active user quantity data } corresponding to the data sources DS1, DS2 and DS3 are obtained, the storage structure of the index database is determined, and a conversion rule P1 between the storage structure of the data source DS1 and the storage structure of the index database, a conversion rule P2 between the storage structure of the data source DS2 and the storage structure of the index database, and a conversion rule P3 between the storage structure of the data source DS3 and the storage structure of the index database are read; then, converting the customer quantity data according to a conversion rule P1 to obtain target customer quantity data which accords with the storage structure of the index database; converting the transaction limit data according to a conversion rule P2 to obtain target transaction limit data which accords with a storage structure of the index database; converting the active user quantity data according to a conversion rule P3 to obtain target active user quantity data which accords with a storage structure of an index database; the target transaction limit data, the target customer quantity data and the target active user quantity data all have the same data structure, so that the target transaction limit data, the target customer quantity data and the target active user quantity data can be conveniently written into an index database and can be conveniently subjected to data transmission processing in the follow-up process.

In summary, by presetting the quasi-conversion rule between the storage structures of the different data sources and the storage structure of the index database, the target index class data can be obtained by unifying the index class data corresponding to the different data sources to facilitate the use of the downstream service.

And step S206, processing the detail data of each data source according to the storage structure of the column database to obtain target detail data.

Specifically, after the detail data corresponding to the multiple data sources are obtained, because different data sources correspond to different data structures, if the detail data corresponding to the multiple data sources are directly stored, an incompatible problem may occur, and meanwhile, because the dimensionality of the detail data is large, the related service information is also richer, so that in order to facilitate the use of downstream services, the detail data corresponding to the multiple data sources need to be unified, and meanwhile, fields or field values convenient for the use of the downstream services are extracted and stored, so that the target detail data is obtained.

The columnar database is specifically a database for storing detail data, and the storage structure of the database is only one, so that the data stored in the database can be unified according to the storage structure, and management and persistence are facilitated; correspondingly, the target detail data is detail data which is the same as the storage structure expression of the column database.

Further, when storing the detail data corresponding to multiple data sources in the column database, in order to improve data storage efficiency and facilitate use of downstream services, the detail data may be processed according to a preset data processing link, in this embodiment, the specific implementation manner is as follows:

step S2062, the detail data corresponding to each data source is synchronized to the intermediate service database.

Specifically, the intermediate service database refers to a unified data warehouse for summarizing the detail data corresponding to each data source, and the detail data stored in the intermediate service database may retain an original data structure, that is, the intermediate service database may be understood as a data warehouse for temporarily storing the detail data to be stored in the column database.

Based on this, because the detail data is complex and contains more information related to the target service, if the conversion or the processing is directly performed, an additional processing burden may be caused, so that in order to improve the subsequent storage efficiency, the detail data corresponding to each data source can be synchronized to the intermediate service database for temporary storage, so as to facilitate the subsequent processing, and the intermediate service database does not persist the data, and the storage space can be cleared after the data is processed, so as to reduce the resource consumption.

Step S2064, converting the detail data in the intermediate service database according to the structure conversion rule corresponding to each data source.

Specifically, the structure conversion rule is a rule for converting the detail data corresponding to each data source into a storage structure of the columnar database, so that the storage structure of the corresponding columnar database is realized, and the target detail data can be conveniently stored and persisted in the subsequent process.

Based on this, the detail data stored in the intermediate database corresponds to multiple data sources, and the column database corresponding to each data source may contain more information related to the service, and the data processed and used by the downstream service does not necessarily cover all the detail data completely, so when the detail data corresponding to each data source is converted, the conversion may be implemented by parsing and screening, and in this embodiment, the specific implementation manner is as follows:

Specifically, the initial sub-data specifically refers to all sub-service data constituting the detail data, and correspondingly, the target sub-data specifically refers to sub-service data required to be converted into the target detail data. Based on the method, the detail data corresponding to each data source is written into the intermediate service database according to the data processing link, after all writing is completed, the detail data can be extracted from the intermediate service database for analysis so as to obtain initial sub-service data corresponding to each detail data, then target sub-data is screened from the initial sub-service data, and then the target sub-data is converted according to the structure conversion rule corresponding to each data source, so that the target detail data with the same storage structure as that of the columnar database can be obtained.

In summary, the target sub-data capable of being converted is obtained by screening the initial sub-service data, so that useless other data can be filtered, and the data is unified, so as to meet the storage requirement of the column database, improve the efficiency of subsequently sorting out the target detailed data, and facilitate the use of downstream services.

Step S2066, generating the target detail type data having the same storage structure as the columnar database according to the conversion result.

Specifically, after the target subdata is screened out and converted according to the structure conversion rule, target detail data which is the same as the storage structure of the columnar database can be generated according to the conversion result.

According to the above example, after detail data { order data, merchant data and address data } corresponding to the data sources DS4, DS5 and DS6 are obtained, the detail data corresponding to each data source can be stored into the unified data bin according to the data processing link { original data source- > unified data bin- > row transfer- > unified data source }, and after the data are completely stored, the detail data corresponding to each data source can be extracted from the unified data bin; the storage structure of the order data corresponding to the data source DS4 is { order number, order establishment time, order transaction time and order ending time }, and the storage structure of the merchant data corresponding to the data source DS5 is { ID, merchant name, merchant address and merchant type }; the storage structure of the address data corresponding to the data source DS6 is { ID, merchant address, shipping address }, at this time, the detail data corresponding to each data source may be converted according to the structure conversion rule between the storage structure of each data source and the storage structure of the columnar database, and the target detail data corresponding to the storage structure { main body, ID, field name, field value } of the columnar database is obtained according to the conversion result, so that each piece of detail data can be recorded by storing the field value and the field name, which facilitates subsequent use, and the structure of the target detail data is shown in the following table (1):

watch (1)

Main body	ID	Name of field	Field value
				Commercial tenant	123	Type1	S1
Commercial tenant	123	Type2	S2
				Commercial tenant	145	Type1	S3
User' s	965	Type9	S4
				…	…	…	…

Different IDs represent different merchants or users, different field names represent different data contents, for example, a Type1 corresponds to a merchant address, a corresponding field value is used for representing a specific address position, or a Type2 corresponds to a merchant Type, and a corresponding field value is used for representing a specific merchant Type. In practical applications, the field name and the field value are set according to actual requirements, and the embodiment is not limited herein.

In conclusion, by converting detailed data of various data sources into a storage structure of the columnar database, data unification is realized, subsequent management and persistence are facilitated, downstream services are more conveniently used, and data processing efficiency is effectively improved.

In addition, the steps S204 and S206 are not executed in sequence, and the step S204 may be executed first, and then the step S206 is executed, or the step S206 is executed first, and then the step S204 is executed, or the step S2041 and the step S206 are executed simultaneously, which is not limited in this embodiment.

Step S208, writing the target index class data into the index database, and writing the target detail class data into the column database.

Specifically, after the data unification is completed, the target index class data and the target detail class data are obtained, at this time, the target index class data can be written into the index database, and the target detail class data can be written into the column-type database, so that the data in each database can be directly read by the downstream service when the downstream service is used, the unification processing is not needed, and the data reuse rate is improved.

Further, after the data is written into the database, in order to complete the reporting process quickly, the triggering of the data reporting process operation may be implemented by detecting the running state of the database, and in this embodiment, the specific implementation manner is as follows:

and sending the service data to be reported to a reporting node.

Specifically, the reporting period is a period of sending data to the reporting node, and may be 24 hours, 1 week or one month, and may be set according to actual requirements in actual application; correspondingly, the index class data to be reported specifically refers to the index class data in the index database in the reporting period; correspondingly, the detail data to be reported is specifically detail data in the reporting period in the columnar database, and the index data to be reported and the detail data to be reported are service data to be reported, which need to be sent to the reporting node for reporting.

Based on the method, after the target index class data and the target detail class data are continuously written into the index database and the column database, whether the report operation is triggered or not can be detected in a mode of detecting the running state of the database; under the condition that the running states of the index database and the column-type database are in a stop state, the data transmission processing can be performed at the moment, index class data to be transmitted can be extracted from the index database according to a preset transmission period, detail class data to be transmitted are extracted from the column-type database at the same time, then the index class data and the detail class data are integrated to obtain service data to be transmitted, and finally the service data to be transmitted is transmitted to a transmission node.

For example, when data related to a service of 2021 month to 2 months in 2021 year needs to be sent to the reporting node, to-be-reported index class data corresponding to the time interval may be extracted from the index database according to the reporting period, and meanwhile, to-be-reported detail data corresponding to the time interval may be extracted from the columnar database, and then, the two parts of data are integrated and sent to the reporting node for processing.

In practical applications, the submission node may be a data management node inside the service party, or may also be a data management organization, or a data management department of a parent company of the service party, and this embodiment is not limited in any way here.

In conclusion, by adopting the mode of detecting the running state of the database to trigger the submission operation, the resource consumption of the submission node can be saved, and the unified data in the submission period can be submitted, so that the submission efficiency and the processing efficiency of the submission node are effectively improved.

In addition, before the data is sent to the reporting node, the problem that the reported data has unclear problems, such as lack of key values or data loss, is avoided; and improving the compliance inside the business project, and detecting the data to be submitted, in this embodiment, the specific implementation manner is as follows:

step S2082, obtaining an index detection strategy corresponding to the index class data to be reported and a detail detection strategy corresponding to the detail class data to be reported.

Specifically, the index detection strategy is a strategy capable of detecting the index data to be reported and detecting whether the part of data is compliant; the corresponding detail detection strategy specifically refers to a strategy capable of detecting the to-be-reported detail column data and detecting whether the part of data is compliant or not.

In practical application, the index detection strategy specifically refers to detecting whether variation fluctuation of the index data to be reported is in compliance compared with that of the historical index data; the detail index strategy specifically refers to detecting whether the detail data to be reported is compliant or not and/or whether the service node is compliant or not; the detecting whether the data meets the regulation requirement of the current service scene specifically means detecting whether the data meets the regulation requirement of the current service scene.

Step S2084, detecting the index data to be reported according to the index detection strategy to obtain an index data detection result, and detecting the detail data to be reported according to the detail detection strategy to obtain a detail data detection result.

Specifically, after the index detection strategy corresponding to the index class data to be reported and the detail detection strategy corresponding to the detail class data to be reported are obtained, in order to ensure the compliance of the service scene and ensure that the service scene is more standard, each part of data can be detected according to the detection strategies, so that the compliance of the service scene is reflected through the detection result.

Further, in the process of detecting the to-be-reported index data according to the index detection strategy, in order to improve the detection accuracy, the comparison may be implemented by reading the historical index data, and in this embodiment, the specific implementation manner is as follows:

reading historical index class data according to the index detection strategy;

Specifically, the historical index class data refers to index class data corresponding to each reporting period in one or more reporting periods, and is used for comparing the index class data to be reported; correspondingly, the detection result of the index data is specifically a detection result obtained after the index data to be reported is detected, so that whether the index data is in compliance or not can be analyzed.

Based on this, after the index detection strategy is obtained, the historical index data can be read according to the index detection strategy so as to be used for detecting the index data to be reported. Furthermore, at this time, the historical index class data and the to-be-reported index class data can be compared, and the index class data detection result corresponding to the to-be-reported index class data is determined in a comparison mode, so that whether the to-be-reported index class data has a risk which can affect the business project or not is analyzed, and therefore a response is made in time, and greater loss is avoided.

In sum, the to-be-reported index class data is analyzed by comparing with the historical index class data, detection is completed by combining the characteristics of the to-be-reported index class data, detection accuracy can be improved, and compliance of a service scene can be improved.

Furthermore, in the process of detecting the to-be-reported detail data according to the detail detection policy, in order to improve the detection accuracy, the detection may be implemented in a manner of detecting a structure and a rule, in this embodiment, a specific implementation manner is as follows:

Specifically, the structure detection condition specifically refers to a condition capable of detecting the structure dimension of the detail data to be reported, that is, detecting whether the detail data to be reported is compliant or not; correspondingly, the rule detection condition specifically refers to a condition capable of detecting the service dimension of the detail data to be reported, that is, whether the service node generating or transmitting the detail data to be reported is compliant is detected.

Based on the above, after the detail detection policy is obtained, the structure detection condition and the rule detection condition can be created according to the policy, then the structure dimension detection is performed on the detail data to be reported based on the structure detection condition, and meanwhile, the service dimension detection is performed on the detail data to be reported based on the rule detection condition, so that the detail data detection result is obtained according to the detection result.

That is to say, when risk prediction of a target service is performed, reporting of data can be completed at the same time, so that data statistics time is saved, resource consumption is reduced, multiple purposes are achieved through one-time operation, compliance of a service scene is effectively guaranteed, and reporting efficiency is improved.

In conclusion, by adopting the structure detection and service detection modes to detect the detail data to be reported, not only the influence caused by the data can be avoided, but also the influence of the service level can be determined, so that the stability of the service scene is further improved, and the compliance of the service scene is improved.

Step S2086, the index data detection result and the detail data detection result are sent to a service adjustment node.

Specifically, after the index data detection result and the detail data detection structure are obtained, the detection structure can be sent to the adjustment node, so that the adjustment node adjusts the service according to the detection result, or updates the service, or corrects the data, and it is ensured that the data subsequently sent to the delivery node is real and accurate data.

For example, after acquiring the to-be-reported index class data of 2 to 3 months in 2021, the historical index class data of 1 to 2 months in 2021 may be read, and then the two parts of data are compared, so as to analyze the fluctuation condition of each index, and if the fluctuation rate is greater than a threshold value, it is indicated that the index class data has a large change and a business risk may exist, and at this time, it is determined that the index class data detection structure has a business risk; if the fluctuation rate is smaller than or equal to the threshold value, the index data change is small, the probability of the existence of the business risk is low, and the index data detection result is determined to be the absence of the business risk.

Simultaneously creating a structure detection condition and a rule detection condition, and detecting the detail data to be reported based on the two conditions, namely detecting whether each detail data to be reported has the condition of losing a field value, a field name, an ID or a main body, and detecting whether each detail data to be reported has the condition of violating the report specification or the compliance term; if the data structure is lost, the data is indicated to have problems, and the detailed data detection result is determined to be that data transmission has problems; and if the illegal data exists, the node for processing the data is proved to have illegal behaviors, and the detailed data detection result is determined to be that the service has a problem.

And finally, integrating the index data detection result and the detail data detection result and then sending the integrated result to an adjustment node, and updating or adjusting the service project through the adjustment node so as to standardize the service scene and ensure the compliance.

The following describes the data processing method further by taking an application of the data processing method provided in this specification in a data delivery scenario as an example, with reference to fig. 3. Fig. 3 shows a processing flow chart of a data processing method applied in a data delivery scenario according to an embodiment of the present specification, which specifically includes the following steps:

step S302, index data and detail data corresponding to various data sources are obtained.

Step S304, determining a conversion rule between the storage structure of each data source and the storage structure of the index database.

And step S306, processing the index class data of each data source according to the conversion rule to obtain target index class data with the same storage structure as the index database.

Step S308, the detail data corresponding to each data source is synchronized to the intermediate service database.

And step S310, converting the detail data in the intermediate service database according to the structure conversion rule corresponding to each data source.

In step S312, target detail data having the same storage structure as the columnar database is generated according to the conversion result.

In step S314, the target index class data is written into the index database and the target detail class data is written into the columnar database.

Step S316, before data is reported, index class data to be reported is extracted from the index database, and detail class data to be reported is extracted from the column database.

Step S318, obtain the index detection policy corresponding to the index class data to be reported and the detail detection policy corresponding to the detail class data to be reported.

Step S320, detecting the to-be-reported index data according to the index detection strategy to obtain the index data detection result.

Step S322, detecting the detail data to be reported according to the detail detection strategy, and obtaining the detail data detection result.

Step S324, sending the index data detection result and the detail data detection result to the service adjustment node.

Step S326, after the adjustment node returns the adjustment result, the data in the index database and the column database are sent to the delivery node.

Corresponding to the above method embodiment, this specification further provides an embodiment of a data processing apparatus, and fig. 4 shows a schematic structural diagram of a data processing apparatus provided in an embodiment of this specification. As shown in fig. 4, the apparatus includes:

an obtain data module 402 configured to obtain index class data and detail class data corresponding to a plurality of data sources;

a first processing module 404, configured to process the index class data of each data source according to the storage structure of the index database, to obtain target index class data;

a second processing module 406, configured to process the detail data of each data source according to the storage structure of the columnar database to obtain target detail data;

a data write module 408 configured to write the target metric class data to the metric database and the target detail class data to the columnar database.

In an alternative embodiment, the obtain data module 402 is further configured to:

receiving a data reporting request submitted by aiming at a target service; sending a data uploading request to at least two service nodes associated with a target service according to the data reporting request; receiving service data returned by each service node according to the data reporting request; and classifying the service data to obtain index data and detail data corresponding to various data sources.

In an optional embodiment, the first processing module 404 is further configured to:

determining a conversion rule between a storage structure of each data source and a storage structure of the index database; and processing the index class data of each data source according to the conversion rule to obtain the target index class data with the same storage structure as the index database.

In an optional embodiment, the second processing module 406 is further configured to:

synchronizing the detail data corresponding to each data source to an intermediate service database; converting the detail data in the intermediate service database according to the structure conversion rule corresponding to each data source; and generating the target detail data with the same storage structure as the columnar database according to the conversion result.

analyzing the detail data in the intermediate service database to obtain initial subdata; and screening target subdata from the initial subdata, and converting the target subdata according to a structure conversion rule corresponding to each data source.

In an optional embodiment, the data processing apparatus further includes:

the reporting module is configured to extract to-be-reported index class data in the index database according to a preset reporting period and extract to-be-reported detail class data in the columnar database when the running state of the index database and the columnar database is a stop state; integrating the index data to be reported and the detail data to be reported to obtain service data to be reported; and sending the service data to be reported to a reporting node.

In an optional embodiment, the data processing apparatus further includes:

the adjusting module is configured to acquire an index detection strategy corresponding to the index class data to be reported and a detail detection strategy corresponding to the detail class data to be reported; detecting the index data to be reported according to the index detection strategy to obtain an index data detection result, and detecting the detail data to be reported according to the detail detection strategy to obtain a detail data detection result; and sending the index data detection result and the detail data detection result to a service regulation node.

In an optional embodiment, the adjusting module is further configured to:

reading historical index class data according to the index detection strategy; and comparing the historical index column data with the index class data to be reported, and determining the detection result of the index class data according to the comparison result.

In an optional embodiment, the adjustment module is further configured to:

creating a structure detection condition and a rule detection condition according to the detail detection strategy; and detecting the detail data to be reported based on the structure detection condition and the rule detection condition to obtain a detail data detection result.

After acquiring index class data and detail class data corresponding to a plurality of data sources, the data processing apparatus provided in this specification may process the index class data of the plurality of data sources according to the storage result of the index database to obtain target index class data of a unified data structure; meanwhile, detail data of each data source is processed according to a storage structure of the column-type database to obtain target index data of a unified data structure, and finally the data are respectively written into the corresponding databases, so that service-related data are unified according to lighting details and index data, the influence efficiency of multiple data sources is avoided when the subsequent service is audited or data is reported, the nodes such as data checking or audit can be conveniently and quickly checked or audited, and the data processing efficiency is further improved.

The above is a schematic configuration of a data processing apparatus of the present embodiment. It should be noted that the technical solution of the data processing apparatus and the technical solution of the data processing method belong to the same concept, and details that are not described in detail in the technical solution of the data processing apparatus can be referred to the description of the technical solution of the data processing method.

Fig. 5 illustrates a block diagram of a computing device 500 provided according to an embodiment of the present description. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. Processor 520 is coupled to memory 510 via bus 530, and database 550 is used to store data.

Computing device 500 also includes access device 540, access device 540 enabling computing device 500 to communicate via one or more networks 560. Examples of such networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the internet. The access device 540 may include one or more of any type of network interface, e.g., a Network Interface Card (NIC), wired or wireless, such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a bluetooth interface, a Near Field Communication (NFC) interface, and so forth.

In one embodiment of the present description, the above-described components of computing device 500, as well as other components not shown in FIG. 5, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device architecture shown in FIG. 5 is for purposes of example only and is not limiting as to the scope of the present description. Other components may be added or replaced as desired by those skilled in the art.

Computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), mobile phone (e.g., smartphone), wearable computing device (e.g., smartwatch, smartglasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. Computing device 500 may also be a mobile or stationary server.

Wherein processor 520 is configured to execute the following computer-executable instructions:

acquiring index data and detail data corresponding to various data sources;

The above is an illustrative scheme of a computing device of the present embodiment. It should be noted that the technical solution of the computing device and the technical solution of the data processing method belong to the same concept, and details that are not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the data processing method.

An embodiment of the present specification also provides a computer readable storage medium storing computer instructions that, when executed by a processor, are operable to:

acquiring index data and detail data corresponding to various data sources;

The above is an illustrative scheme of a computer-readable storage medium of the present embodiment. It should be noted that the technical solution of the storage medium belongs to the same concept as the technical solution of the data processing method, and details that are not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the data processing method.

An embodiment of the present specification also provides a computer program, which, when executed in a computer, causes the computer to execute the steps of the data processing method.

The above is an illustrative scheme of a computer program of the present embodiment. It should be noted that the computer program and the technical solution of the data processing method belong to the same concept, and details that are not described in detail in the technical solution of the computer program can be referred to the description of the technical solution of the data processing method.

The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

The computer instructions comprise computer program code which may be in the form of source code, object code, an executable file or some intermediate form, or the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.

It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of acts or combinations, but those skilled in the art should understand that the present disclosure is not limited by the described order of acts, as some steps may be performed in other orders or simultaneously according to the present disclosure. Further, those skilled in the art should also appreciate that the embodiments described in this specification are preferred embodiments and that acts and modules referred to are not necessarily required for this description.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

The preferred embodiments of the present specification disclosed above are intended only to aid in the description of the specification. Alternative embodiments are not exhaustive and do not limit the invention to the precise embodiments described. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the specification and its practical application, to thereby enable others skilled in the art to best understand the specification and its practical application. The specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method of data processing, comprising:

acquiring index data and detail data corresponding to various data sources;

2. The data processing method according to claim 1, wherein the acquiring of the index class data and the detail class data corresponding to the plurality of data sources comprises:

receiving a data reporting request submitted by aiming at a target service;

3. The data processing method according to claim 1, wherein the processing the index class data of each data source according to the storage structure of the index database to obtain the target index class data comprises:

4. The data processing method according to claim 1, wherein the processing the detail class data of each data source according to the storage structure of the columnar database to obtain the target detail class data comprises:

5. The data processing method according to claim 4, wherein the converting the detail data in the intermediate service database according to the structure conversion rule corresponding to each data source comprises:

6. The data processing method of any of claims 1 to 5, further comprising, after the steps of writing the target index class data to the index database and writing the target detail class data to the columnar database are performed:

and sending the service data to be reported to a reporting node.

7. The data processing method according to claim 6, wherein before the step of sending the service data to be delivered to a delivery node is executed, the method further comprises:

8. The data processing method according to claim 7, wherein the detecting the to-be-reported index class data according to the index detection policy to obtain an index class data detection result includes:

reading historical index class data according to the index detection strategy;

9. The data processing method according to claim 7, wherein the detecting the detail data to be reported according to the detail detection policy to obtain a detail data detection result includes:

10. A data processing apparatus comprising:

11. A computing device, comprising:

a memory and a processor;

the memory is for storing computer-executable instructions, and the processor is for executing the computer-executable instructions to implement the steps of the method of any one of claims 1 to 9.

12. A computer readable storage medium storing computer instructions which, when executed by a processor, carry out the steps of the method of any one of claims 1 to 9.