WO2024103505A1 - Data processing method and apparatus - Google Patents

Data processing method and apparatus Download PDF

Info

Publication number
WO2024103505A1
WO2024103505A1 PCT/CN2022/144128 CN2022144128W WO2024103505A1 WO 2024103505 A1 WO2024103505 A1 WO 2024103505A1 CN 2022144128 W CN2022144128 W CN 2022144128W WO 2024103505 A1 WO2024103505 A1 WO 2024103505A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
processed
sent
identifier
sending
Prior art date
Application number
PCT/CN2022/144128
Other languages
French (fr)
Chinese (zh)
Inventor
魏恩军
袁瑞峰
Original Assignee
奇安信科技集团股份有限公司
奇安信网神信息技术(北京)股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 奇安信科技集团股份有限公司, 奇安信网神信息技术(北京)股份有限公司 filed Critical 奇安信科技集团股份有限公司
Publication of WO2024103505A1 publication Critical patent/WO2024103505A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/40—Transformation of program code
    • G06F8/41—Compilation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/70—Software maintenance or management
    • G06F8/71—Version control; Configuration management

Definitions

  • the present application relates to the field of data processing technology, and in particular to a data processing method and device.
  • the present application provides a data processing method and device.
  • the present application provides a data processing method, comprising:
  • the data to be processed is processed based on the data processing rule.
  • the data processing rules include at least one of the following: data parsing rules, data storage rules and data sending rules.
  • the data parsing rule is used to indicate the format identifier of the data to be processed and the first identifier of the data to be parsed;
  • the processing of the data to be processed based on the data processing rule includes:
  • the data processing rule includes the data parsing rule, determining a target parser based on a format identifier of the data to be processed;
  • the target parser parses data matching the first identifier from the data to be processed to obtain parsed data.
  • the data storage rule is used to indicate a second identifier of the data to be stored
  • the processing of the data to be processed based on the data processing rule includes:
  • the data processing rule includes the data storage rule and the data to be processed includes context data, determining the data to be stored that matches the second identifier from the parsed data; the parsed data is data parsed from the data to be processed;
  • the data processing rule includes the data storage rule, and the data to be processed includes non-context data, determining the data to be stored that matches the second identifier from the data to be processed;
  • the data to be stored is stored.
  • the data sending rule is used to indicate a third identifier of the data to be sent
  • the processing of the data to be processed based on the data processing rule includes:
  • the data processing rule includes the data sending rule and the data to be processed includes context data, determining the data to be sent that matches the third identifier from the parsed data; the parsed data is data parsed from the data to be processed;
  • the data processing rule includes the data sending rule
  • the data to be processed includes non-context data
  • the data to be sent is sent.
  • the data sending rule is also used to indicate a sending channel identifier
  • the sending of the data to be sent includes:
  • the data to be sent is sent through the target sending channel.
  • the data sending rule is also used to indicate a format identifier of the data to be sent;
  • the sending of the data to be sent includes:
  • the first target sending data is sent.
  • the data sending rule is also used to indicate the reorganization information of the data to be sent;
  • the sending of the data to be sent includes:
  • the second target sending data is sent.
  • the present application also provides a data processing device, comprising:
  • a first acquisition unit used to acquire data to be processed
  • a second acquisition unit used to acquire a data processing rule configured by a user for the data to be processed
  • a processing unit is used to process the data to be processed based on the data processing rules.
  • the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the data processing methods described above is implemented.
  • the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored.
  • a computer program is stored on which a computer program is stored.
  • the computer program is executed by a processor, any of the data processing methods described above is implemented.
  • the present application also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned data processing methods.
  • the data processing method and device provided by the present application obtain the data to be processed, obtain the data processing rules configured by the user for the data to be processed, and process the data to be processed based on the data processing rules. It can be seen that the present application is that the user configures the corresponding data processing rules based on the data to be processed, thereby realizing flexible data processing.
  • FIG1 is a schematic diagram of a data processing method according to an embodiment of the present invention.
  • FIG2 is a second flow chart of the data processing method provided in an embodiment of the present application.
  • FIG3 is a schematic diagram of a parsing process provided in an embodiment of the present application.
  • FIG4 is a third flow chart of the data processing method provided in an embodiment of the present application.
  • FIG5 is a fourth flow chart of the data processing method provided in an embodiment of the present application.
  • FIG6 is one of the data processing flow diagrams provided in an embodiment of the present application.
  • FIG7 is a second schematic diagram of a data processing flow provided in an embodiment of the present application.
  • FIG8 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application.
  • FIG. 9 is a schematic diagram of the physical structure of the electronic device provided in the present application.
  • FIG. 1 is one of the flow charts of the data processing method provided in the embodiment of the present application. As shown in FIG. 1 , the data processing method includes the following steps.
  • Step 101 Obtain data to be processed.
  • the data to be processed may be context data, or non-context data such as sample data.
  • the user may submit the data to be processed through a preset interface, for example, the user may submit the data to be processed through a clinic system interface.
  • Step 102 Obtain the data processing rules configured by the user for the data to be processed.
  • the data processing rules include at least one of the following: data parsing rules, data storage rules and data sending rules.
  • the configuration module of the registration access permission can be displayed to the user.
  • the user can configure the data processing rules for the data to be processed in the configuration module based on his own needs. For example, taking the data to be processed as user context data as an example, the user context data includes 5 fields, namely field 1, field 2, field 3, field 4 and field 5. If the user only wants to parse and store field 3, the content to be parsed can be configured in the configuration module as field 3, and the content to be stored can be the specific information of field 3. Then the content to be parsed configured by the user as field 3 and the specific information of field 3 are the data processing rules for the data to be processed.
  • Step 103 Process the data to be processed based on the data processing rules.
  • the data to be processed can be processed accordingly based on the data processing rules; for example, the obtained data processing rules are that the content to be parsed is field 3, and the content to be stored is the specific information of field 3, then field 3 of the data to be processed can be parsed to obtain the specific information of field 3, that is, the specific value of field 3, and then the specific value of field 3 can be stored, thereby realizing the specific processing process of the data to be processed.
  • the obtained data processing rules are that the content to be parsed is field 3, and the content to be stored is the specific information of field 3, then field 3 of the data to be processed can be parsed to obtain the specific information of field 3, that is, the specific value of field 3, and then the specific value of field 3 can be stored, thereby realizing the specific processing process of the data to be processed.
  • the data processing method provided by the present application obtains the data processing rules configured by the user for the data to be processed when the data to be processed is obtained, and processes the data to be processed based on the data processing rules. It can be seen that the present application is a data processing rule corresponding to the data to be processed configured by the user, thereby realizing flexible data processing.
  • the data parsing rule is used to indicate the format identifier of the data to be processed and the first identifier of the data to be parsed;
  • Figure 2 is the second flow chart of the data processing method provided in an embodiment of the present application. As shown in Figure 2, the above step 103 can be specifically implemented through the following steps.
  • Step 1031 When the data processing rule includes the data parsing rule, determine a target parser based on the format identifier of the data to be processed.
  • the parser of each data format can be pre-added to the configuration module as a component, and a parser list can be established, in which the correspondence between the format identifier and the parser is stored; when the data processing rules configured by the user include data parsing rules, the specific content of the data parsing rules is obtained, that is, the format identifier for indicating the data to be processed and the first identifier of the data to be parsed are obtained, and the format identifier includes but is not limited to a json format identifier, an xml format identifier, a csv format identifier, etc.; taking the json format identifier as an example, when the format identifier for indicating that the data to be processed is a json format identifier, the target parser corresponding to the json format identifier is searched from
  • Step 1032 parse the data matching the first identifier from the data to be processed by the target parser to obtain parsed data.
  • the target parser when the target parser is determined based on the format identifier of the data to be processed, the target parser searches for the first identifier of the data to be parsed from the data to be processed, finds the position to be parsed, and then parses the corresponding data at the position to be parsed; for example, the first identifier of the data to be parsed is field 3, then the target parser determines the position of field 3 in the data to be processed, and then parses the specific value of field 3 to obtain parsed data.
  • Figure 3 is a schematic diagram of the parsing process provided by an embodiment of the present application.
  • the data to be processed is context data.
  • the corresponding target parser is selected in the configuration module, and the format identifier of the data to be processed is configured at the position for configuring the format identifier, and the first identifier of the data to be parsed is configured at the position for configuring the identifier of the data to be parsed;
  • the context data is parsed based on the parser corresponding to the json format to obtain parsed data;
  • the context data is parsed based on the parser corresponding to the xml format to obtain parsed data;
  • the format identifier is an identifier in the csv format
  • the context data is parsed based on the parser corresponding to the csv format to obtain parsed data, wherein the parsed data may
  • the data processing method provided in the embodiment of the present application can determine the corresponding target parser based on the data parsing rules when the data processing rules include data parsing rules, and then parse the data to be processed based on the target parser, thereby realizing parsing the data required by the user in the data to be processed based on the data parsing rules configured by the user, and embodies the flexible configuration and implementation of the parsing process.
  • FIG4 is a third flow chart of the data processing method provided in an embodiment of the present application. As shown in FIG4 , the above step 103 can be specifically implemented through the following steps.
  • step 1033 and step 1035 are executed; when the data to be processed includes non-context data, step 1034 and step 1035 are executed.
  • Step 1033 When the data processing rule includes the data storage rule and the data to be processed includes context data, determine the data to be stored that matches the second identifier from the parsed data; the parsed data is data parsed from the data to be processed.
  • the data processing rule includes a data storage rule
  • the data to be processed includes context data
  • the specific content of the data storage rule is obtained, that is, the second identifier for indicating the data to be stored is obtained, and then the data corresponding to the second identifier is searched from the parsed data, and the data corresponding to the second identifier is stored as the data to be stored.
  • the data to be processed includes context data
  • the parsed data is the value of field 2 and the value of field 3. If the data storage rule configured by the user is to store the value of field 2, the value of field 2 is obtained from the parsed data, and then the value of field 2 is stored in a preset position.
  • Step 1034 When the data processing rule includes the data storage rule and the data to be processed includes non-context data, determine the data to be stored that matches the second identifier from the data to be processed.
  • the data processing rule includes a data storage rule
  • the data to be processed includes non-context data
  • the specific content of the data storage rule is obtained, that is, the second identifier for indicating the data to be stored is obtained, and then the data corresponding to the second identifier is searched from the parsed data, and the data corresponding to the second identifier is stored as the data to be stored.
  • the data to be processed includes data 1 obtained from a first Internet Protocol (IP) address and data 2 obtained from a second IP address. If the data storage rule configured by the user is to store data 1 obtained from the first IP address, data 1 obtained from the first IP address is obtained from the data to be processed, and then data 1 obtained from the first IP address is stored in a preset location.
  • IP Internet Protocol
  • Step 1035 store the data to be stored.
  • the data to be stored is stored in a preset location.
  • the user can configure the specific preset location by himself, or the electronic device can automatically select the preset location for storage based on the usage of the device memory.
  • the data to be stored is alarm data
  • the alarm data can be archived through filing. This application is not limited to this.
  • context data is data unrelated to the business and is submitted by the user
  • the context data is not required in business processing and is only stored during storage. Therefore, the context data is not parsed in business processing. Therefore, when the data to be processed is context data, the context data needs to be parsed; and non-context data is data generated in the business process, and the data parsing processing has been completed in the business. Therefore, when the data to be processed is non-context data, since the non-context data is data that has been parsed, it does not need to be parsed.
  • the data processing method provided in the embodiment of the present application can store the processed data or parsed data based on the data storage rules when the data processing rules include data storage rules, thereby realizing the storage of user-required data in the processed data based on the data storage rules configured by the user, and embodies the flexible configuration and implementation of the storage process.
  • FIG. 5 is a fourth flow chart of the data processing method provided in an embodiment of the present application. As shown in FIG. 5 , the above step 103 can be specifically implemented through the following steps.
  • step 1036 and step 1038 are executed; when the data to be processed includes non-context data, step 1037 and step 1038 are executed.
  • Step 1036 When the data processing rule includes the data sending rule and the data to be processed includes context data, determine the data to be sent that matches the third identifier from the parsed data; the parsed data is data parsed from the data to be processed.
  • the data processing rules include data sending rules
  • the data to be processed include context data
  • the specific content of the data sending rules is obtained, that is, the third identifier used to indicate the data to be sent is obtained, and then the data matching the third identifier is found in the parsed data.
  • the data matching the third identifier is found in the parsed data
  • the data corresponding to the third identifier is sent as the data to be sent.
  • the data to be processed includes context data
  • the parsed data is the value of field 2 and the value of field 3. If the data sending rule configured by the user is to send the value of field 3, the value of field 3 is obtained from the parsed data, and then the value of field 3 is sent through the default sending channel or the sending channel set by the user.
  • Step 1037 When the data processing rule includes the data sending rule and the data to be processed includes non-context data, determine the data to be sent that matches the third identifier from the data to be processed.
  • the data processing rules include data storage rules and the data to be processed include non-context data
  • the specific content of the data sending rules is obtained, that is, a third identifier for indicating the data to be stored is obtained, and then the data matching the third identifier is searched from the parsed data.
  • the data matching the third identifier is found in the data to be processed, the data corresponding to the third identifier is sent as the data to be sent.
  • the data to be processed includes data 1 obtained from a first Internet Protocol (IP) address and data 2 obtained from a second IP address.
  • IP Internet Protocol
  • data 2 obtained from the second IP address is obtained from the data to be processed, and then data 2 obtained from the second IP address is sent through the default sending channel or the sending channel set by the user.
  • inclusion relationships include inclusion (in), exclusion (not in), whether the parameter string is at the head of the original string (startWith), whether the parameter string is at the end of the original string (endWith), etc., and this application does not limit this.
  • Step 1038 Send the data to be sent.
  • the data to be sent is sent through a default sending channel or a sending channel set by the user, which is not limited in the present application.
  • the data processing method provided in the embodiment of the present application can send the processed data or parsed data based on the data sending rules when the data processing rules include data sending rules, thereby realizing the sending of user-required data in the processed data based on the data sending rules configured by the user, and embodies the flexible configuration and implementation of the sending process.
  • the data sending rule is also used to indicate a sending channel identifier; the above step 1038 can be specifically implemented in the following manner:
  • a target sending channel is determined based on the sending channel identifier; and the data to be sent is sent through the target sending channel.
  • the user can also configure the sending channel identifier of the sending channel used when sending data in the configuration module based on their own needs, wherein the sending channel includes multiple sending channels such as kafka, syslog and snmp, and the sending channel can also be expanded by implementing the sending interface. For example, if the sending channel identifier configured by the user is the sending channel identifier corresponding to kafka, then when the data to be sent is determined, the data to be sent is sent through the kafka sending channel.
  • Figure 6 is one of the data processing flow diagrams provided by an embodiment of the present application. As shown in Figure 6, taking the data to be processed as the data to be stored as an example, the second identifier of the data to be stored is configured at the position of the identifier of the data to be stored in the configuration module; taking the data to be processed as the data to be sent as an example, the sending channel identifier of the data to be sent is configured at the sending channel identifier of the data to be sent in the configuration module.
  • the data to be sent is processed accordingly based on the configured data processing rules; for example, when the second identifier of the data to be stored is filing archive, the data to be stored is stored in the storage engine, for example, the data to be stored is stored in the mongo database; when the sending channel identifier of the data to be sent is the sending channel identifier corresponding to kafka, the data to be sent is sent through the kafka sending channel; when the sending channel identifier of the data to be sent is the sending channel identifier corresponding to syslog, the data to be sent is sent through the syslog sending channel.
  • the user can configure the sending channel identifier by himself, and the data to be sent can be sent based on the sending channel identifier configured by the user, thereby realizing flexible configuration of the sending channel.
  • the data sending rule is also used to indicate a format identifier of the data to be sent; the above step 1038 can be specifically implemented in the following manner:
  • the format of the data to be sent is converted based on the format identifier of the data to be sent to obtain first target sending data; and the first target sending data is sent.
  • the user can also configure the format identifier of the data to be sent in the configuration module based on his or her own needs.
  • determining the data to be sent it is determined whether the current data format of the data to be sent is the data format corresponding to the format identifier configured by the user.
  • the format of the data to be sent is converted based on the data format corresponding to the format identifier configured by the user to obtain the data to be sent with the same data format as the user configured, that is, the first target sending data is obtained, and the first target sending data is sent.
  • the data processing method provided in the embodiment of the present application allows the user to configure the format identifier of the data to be sent by himself, and can perform format conversion on the data to be sent based on the format identifier of the data to be sent configured by the user, thereby realizing flexible configuration of the data format of the data to be sent.
  • the data transmission rule is also used to indicate reorganization information of the data to be transmitted; the above step 1038 can be specifically implemented in the following manner:
  • the content of the data to be sent is reorganized based on the reorganization information of the data to be sent to obtain second target sending data; and the second target sending data is sent.
  • the reorganization information includes defining a new data structure or defining a new field name, etc.
  • the user can also configure the reorganization information of the data to be sent in the configuration module based on his own needs.
  • the data to be sent is reorganized based on the reorganization information of the data to be sent configured by the user, and the reorganized second target data to be sent is sent.
  • the reorganization information of the data to be sent configured by the user is to modify the name of field 2 to abc.
  • field 2 is searched from the data to be sent, and then the name of field 2 is modified to abc, so that the reorganization of the data to be sent is achieved; for another example, the reorganization information of the data to be sent configured by the user is to modify the data structure of the data to be sent to data structure 1.
  • the current data structure of the data to be sent needs to be modified according to data structure 1, so that the reorganization of the data to be sent is achieved.
  • Figure 7 is a second schematic diagram of the data processing flow provided by an embodiment of the present application.
  • the reorganization information of the data to be stored is configured at the position used to configure the reorganization information of the data to be sent in the configuration module
  • the format identifier of the data to be sent is configured at the position used to configure the format identifier of the data to be sent.
  • the data to be sent is processed accordingly based on the configured data processing rules; for example, if the reorganization information of the data to be sent is to update the field name, the field name of the data to be sent is updated; when the format identifier of the data to be sent is a json format identifier, the data format of the data to be sent is converted into a json format; when the format identifier of the data to be sent is an xml format identifier, the data format of the data to be sent is converted into an xml format; when the format identifier of the data to be sent is a default csv format identifier, the data format of the data to be sent is converted into a csv format.
  • the data processing method provided in the embodiment of the present application allows the user to configure the reorganization information of the data to be sent by himself, and the data to be sent can be reorganized based on the reorganization information of the data to be sent configured by the user, thereby realizing flexible configuration of the data structure or field name of the data to be sent.
  • the data processing method provided in the embodiment of the present application can be applied to data processing scenarios of different production lines, and provide a business transit platform for data processing of different production lines.
  • the business transit platform constructs a set of data processing rules through rule templates, standardizes the uploading and calling of data between different production lines, and realizes the automatic flow of data between different production lines; the data of any production line can be used as data to be processed, and can be configurable to perform corresponding processing on the data to be processed based on user needs, thereby realizing the reuse of each module in the business transit platform and meeting the needs of different users; that is, when the data to be processed is obtained, the data processing rules configured by the user for the data to be processed are obtained, and the data to be processed is processed based on the data processing rules, thereby realizing flexible processing of different data.
  • the corresponding target parser can be determined based on the data parsing rules, and then the data to be processed can be parsed based on the target parser, thereby realizing parsing the user's required data in the data to be processed based on the user-configured data parsing rules, reflecting the flexible configuration and implementation of the parsing process.
  • the data to be processed or the parsed data can be stored based on the data storage rules, and the data required by the user in the data to be processed can be stored based on the data storage rules configured by the user, which reflects the flexible configuration and implementation of the storage process.
  • the data to be processed or the parsed data can be sent based on the data sending rules, thereby realizing the sending of user-required data in the data to be processed based on the data sending rules configured by the user, reflecting the flexible configuration and implementation of the sending process.
  • users can configure the sending channel identifier by themselves, and send the data to be sent based on the sending channel identifier configured by the user, thus realizing flexible configuration of the sending channel; users can also configure the format identifier of the data to be sent by themselves, and convert the format of the data to be sent based on the format identifier of the data to be sent configured by the user, thus realizing flexible configuration of the data format of the data to be sent; users can also configure the reorganization information of the data to be sent by themselves, and reorganize the data to be sent based on the reorganization information of the data to be sent configured by the user, thus realizing flexible configuration of the data structure or field name of the data to be sent.
  • the data processing device provided by the present application is described below.
  • the data processing device described below and the data processing method described above can be referenced to each other.
  • FIG8 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application.
  • the data processing device 800 includes a first acquisition unit 801, a second acquisition unit 802 and a processing unit 803, wherein:
  • a first acquisition unit 801 is used to acquire data to be processed
  • the second acquisition unit 802 is used to acquire the data processing rules configured by the user for the data to be processed
  • the processing unit 803 is used to process the data to be processed based on the data processing rule.
  • the data processing device obtains the data processing rules configured by the user for the data to be processed when the data to be processed is obtained, and processes the data to be processed based on the data processing rules. It can be seen that the present application is that the user configures the corresponding data processing rules based on the data to be processed, thereby realizing flexible data processing.
  • the data processing rules include at least one of the following: data parsing rules, data storage rules and data sending rules.
  • the data parsing rule is used to indicate the format identifier of the data to be processed and the first identifier of the data to be parsed; the processing unit 803 is specifically used to:
  • the data processing rule includes the data parsing rule, determining a target parser based on a format identifier of the data to be processed;
  • the target parser parses data matching the first identifier from the data to be processed to obtain parsed data.
  • the data storage rule is used to indicate the second identifier of the data to be stored; and the processing unit 803 is further specifically used to:
  • the data processing rule includes the data storage rule and the data to be processed includes context data, determining the data to be stored that matches the second identifier from the parsed data; the parsed data is data parsed from the data to be processed;
  • the data processing rule includes the data storage rule, and the data to be processed includes non-context data, determining the data to be stored that matches the second identifier from the data to be processed;
  • the data to be stored is stored.
  • the data sending rule is used to indicate a third identifier of the data to be sent; and the processing unit 803 is further specifically used to:
  • the data processing rule includes the data sending rule and the data to be processed includes context data, determining the data to be sent that matches the third identifier from the parsed data; the parsed data is data parsed from the data to be processed;
  • the data processing rule includes the data sending rule
  • the data to be processed includes non-context data
  • the data to be sent is sent.
  • the data sending rule is further used to indicate a sending channel identifier; and the processing unit 803 is further specifically used to:
  • the data to be sent is sent through the target sending channel.
  • the data sending rule is further used to indicate a format identifier of the data to be sent; and the processing unit 803 is further specifically used to:
  • the first target sending data is sent.
  • the data sending rule is further used to indicate reorganization information of the data to be sent; and the processing unit 803 is further specifically used to:
  • the second target sending data is sent.
  • FIG9 is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present application.
  • the electronic device may include: a processor 910, a communications interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communications interface 920, and the memory 930 communicate with each other through the communication bus 940.
  • the processor 910 may call the logic instructions in the memory 930 to execute a data processing method, which includes: obtaining data to be processed;
  • the data to be processed is processed based on the data processing rule.
  • the logic instructions in the above-mentioned memory 930 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.
  • the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art.
  • the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.
  • the aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., and other media that can store program codes.
  • the present application also provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the data processing method provided by the above methods, the method includes: obtaining data to be processed;
  • the data to be processed is processed based on the data processing rule.
  • the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the data processing method provided by the above methods is implemented, and the method includes: obtaining data to be processed;
  • the data to be processed is processed based on the data processing rule.
  • the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.
  • each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware.
  • the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM/RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Quality & Reliability (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The present application relates to the technical field of data processing. Provided are a data processing method and apparatus. The method comprises: acquiring data to be processed; acquiring a data processing rule, which is configured by a user for said data; and processing said data on the basis of the data processing rule. In the present application, when data to be processed is acquired, a data processing rule configured by a user for said data is acquired, and said data is processed on the basis of the data processing rule. Hence, in the present application, flexible data processing is realized by means of a user by means of configuring a data processing rule on the basis of data to be processed.

Description

数据处理方法及装置Data processing method and device
相关申请的交叉引用CROSS-REFERENCE TO RELATED APPLICATIONS
本申请要求于2022年11月18日提交的申请号为202211449670.2，名称为“数据处理方法及装置”的中国专利申请的优先权，其通过引用方式全部并入本文。This application claims priority to Chinese patent application number 202211449670.2, filed on November 18, 2022, and entitled “Data Processing Method and Device,” which is incorporated herein by reference in its entirety.
技术领域Technical Field
本申请涉及数据处理技术领域，尤其涉及一种数据处理方法及装置。The present application relates to the field of data processing technology, and in particular to a data processing method and device.
背景技术Background technique
在业务开发过程中，不同用户在相同或类似的业务场景中有不同的数据处理需求，例如，用户A需要将自己的上下文数据及所有业务数据通过kafka外发给自己，用户B需要将有告警的业务数据通过syslog外发等。During business development, different users may have different data processing requirements in the same or similar business scenarios. For example, user A needs to send his context data and all business data to himself through Kafka, and user B needs to send business data with alarms through syslog.
相关技术中，根据用户的需求，通常首先为每个用户定制一套数据处理流程，然后通过预先定制的数据处理流程处理每个用户需要处理的数据。In the related art, according to the needs of users, usually a set of data processing procedures is first customized for each user, and then the data that each user needs to process is processed through the pre-customized data processing procedure.
但上述相关技术中，若用户的需求发生改变或者增多，就需要不断对数据处理流程进行重新定制，从而降低了数据的处理灵活性。However, in the above-mentioned related technologies, if the user's needs change or increase, the data processing flow needs to be constantly re-customized, thereby reducing the flexibility of data processing.
发明内容Summary of the invention
针对现有技术中的问题，本申请提供一种数据处理方法及装置。In view of the problems in the prior art, the present application provides a data processing method and device.
本申请提供一种数据处理方法，包括：The present application provides a data processing method, comprising:
获取待处理数据；Get the data to be processed;
获取用户对所述待处理数据配置的数据处理规则；Obtaining data processing rules configured by a user for the data to be processed;
基于所述数据处理规则对所述待处理数据进行处理。The data to be processed is processed based on the data processing rule.
根据本申请提供的一种数据处理方法，所述数据处理规则包括以下至少之一：数据解析规则、数据存储规则和数据发送规则。According to a data processing method provided by the present application, the data processing rules include at least one of the following: data parsing rules, data storage rules and data sending rules.
根据本申请提供的一种数据处理方法，所述数据解析规则用于指示所述待处理数据的格式标识和待解析数据的第一标识；According to a data processing method provided by the present application, the data parsing rule is used to indicate the format identifier of the data to be processed and the first identifier of the data to be parsed;
所述基于所述数据处理规则对所述待处理数据进行处理，包括：The processing of the data to be processed based on the data processing rule includes:
在所述数据处理规则包括所述数据解析规则时，基于所述待处理数据 的格式标识确定目标解析器；When the data processing rule includes the data parsing rule, determining a target parser based on a format identifier of the data to be processed;
通过所述目标解析器从所述待处理数据中解析与所述第一标识匹配的数据，得到解析数据。The target parser parses data matching the first identifier from the data to be processed to obtain parsed data.
根据本申请提供的一种数据处理方法，所述数据存储规则用于指示待存储数据的第二标识；According to a data processing method provided by the present application, the data storage rule is used to indicate a second identifier of the data to be stored;
所述基于所述数据处理规则对所述待处理数据进行处理，包括：The processing of the data to be processed based on the data processing rule includes:
在所述数据处理规则包括所述数据存储规则、且所述待处理数据包括上下文数据时，从解析数据中确定与所述第二标识匹配的待存储数据；所述解析数据为从所述待处理数据中解析得到的数据；When the data processing rule includes the data storage rule and the data to be processed includes context data, determining the data to be stored that matches the second identifier from the parsed data; the parsed data is data parsed from the data to be processed;
在所述数据处理规则包括所述数据存储规则，且所述待处理数据包括非上下文数据时，从所述待处理数据中确定与所述第二标识匹配的待存储数据；When the data processing rule includes the data storage rule, and the data to be processed includes non-context data, determining the data to be stored that matches the second identifier from the data to be processed;
将所述待存储数据进行存储。The data to be stored is stored.
根据本申请提供的一种数据处理方法，所述数据发送规则用于指示待发送数据的第三标识；According to a data processing method provided by the present application, the data sending rule is used to indicate a third identifier of the data to be sent;
所述基于所述数据处理规则对所述待处理数据进行处理，包括：The processing of the data to be processed based on the data processing rule includes:
在所述数据处理规则包括所述数据发送规则、且所述待处理数据包括上下文数据时，从解析数据中确定与所述第三标识匹配的待发送数据；所述解析数据为从所述待处理数据中解析得到的数据；When the data processing rule includes the data sending rule and the data to be processed includes context data, determining the data to be sent that matches the third identifier from the parsed data; the parsed data is data parsed from the data to be processed;
在所述数据处理规则包括所述数据发送规则，且所述待处理数据包括非上下文数据时，从所述待处理数据中确定与所述第三标识匹配的待发送数据；When the data processing rule includes the data sending rule, and the data to be processed includes non-context data, determining the data to be sent that matches the third identifier from the data to be processed;
将所述待发送数据进行发送。The data to be sent is sent.
根据本申请提供的一种数据处理方法，所述数据发送规则还用于指示发送通道标识；According to a data processing method provided by the present application, the data sending rule is also used to indicate a sending channel identifier;
所述将所述待发送数据进行发送，包括：The sending of the data to be sent includes:
基于所述发送通道标识确定目标发送通道；Determine a target sending channel based on the sending channel identifier;
将所述待发送数据通过所述目标发送通道进行发送。The data to be sent is sent through the target sending channel.
根据本申请提供的一种数据处理方法，所述数据发送规则还用于指示待发送数据的格式标识；According to a data processing method provided by the present application, the data sending rule is also used to indicate a format identifier of the data to be sent;
所述将所述待发送数据进行发送，包括：The sending of the data to be sent includes:
基于所述待发送数据的格式标识将所述待发送数据进行格式转换，得到第一目标发送数据；Converting the format of the data to be sent based on the format identifier of the data to be sent to obtain first target sending data;
将所述第一目标发送数据进行发送。The first target sending data is sent.
根据本申请提供的一种数据处理方法，所述数据发送规则还用于指示待发送数据的重组信息；According to a data processing method provided by the present application, the data sending rule is also used to indicate the reorganization information of the data to be sent;
所述将所述待发送数据进行发送，包括：The sending of the data to be sent includes:
基于所述待发送数据的重组信息将所述待发送数据的内容进行重组，得到第二目标发送数据；reorganizing the content of the data to be sent based on the reorganization information of the data to be sent to obtain second target sending data;
将所述第二目标发送数据进行发送。The second target sending data is sent.
本申请还提供一种数据处理装置，包括：The present application also provides a data processing device, comprising:
第一获取单元，用于获取待处理数据；A first acquisition unit, used to acquire data to be processed;
第二获取单元，用于获取用户对所述待处理数据配置的数据处理规则；A second acquisition unit, used to acquire a data processing rule configured by a user for the data to be processed;
处理单元，用于基于所述数据处理规则对所述待处理数据进行处理。A processing unit is used to process the data to be processed based on the data processing rules.
本申请还提供一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现如上述任一种所述数据处理方法。The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the data processing methods described above is implemented.
本申请还提供一种非暂态计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现如上述任一种所述数据处理方法。The present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any of the data processing methods described above is implemented.
本申请还提供一种计算机程序产品，包括计算机程序，所述计算机程序被处理器执行时实现如上述任一种所述数据处理方法。The present application also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned data processing methods.
本申请提供的数据处理方法及装置，在获取到待处理数据时，获取用户对待处理数据配置的数据处理规则，基于数据处理规则对待处理数据进行处理。可知，本申请是用户基于待处理数据配置对应的数据处理规则，实现了数据的灵活处理。The data processing method and device provided by the present application obtain the data to be processed, obtain the data processing rules configured by the user for the data to be processed, and process the data to be processed based on the data processing rules. It can be seen that the present application is that the user configures the corresponding data processing rules based on the data to be processed, thereby realizing flexible data processing.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
为了更清楚地说明本申请或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作一简单地介绍，显而易见地，下面描述中的附图是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
图1是本申请实施例提供的数据处理方法的流程示意图之一；FIG1 is a schematic diagram of a data processing method according to an embodiment of the present invention;
图2是本申请实施例提供的数据处理方法的流程示意图之二；FIG2 is a second flow chart of the data processing method provided in an embodiment of the present application;
图3是本申请实施例提供的解析流程示意图；FIG3 is a schematic diagram of a parsing process provided in an embodiment of the present application;
图4是本申请实施例提供的数据处理方法的流程示意图之三；FIG4 is a third flow chart of the data processing method provided in an embodiment of the present application;
图5是本申请实施例提供的数据处理方法的流程示意图之四；FIG5 is a fourth flow chart of the data processing method provided in an embodiment of the present application;
图6是本申请实施例提供的数据处理流程示意图之一；FIG6 is one of the data processing flow diagrams provided in an embodiment of the present application;
图7是本申请实施例提供的数据处理流程示意图之二；FIG7 is a second schematic diagram of a data processing flow provided in an embodiment of the present application;
图8是本申请实施例提供的数据处理装置的结构示意图；FIG8 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application;
图9是本申请提供的电子设备的实体结构示意图。FIG. 9 is a schematic diagram of the physical structure of the electronic device provided in the present application.
具体实施方式Detailed ways
为使本申请的目的、技术方案和优点更加清楚，下面将结合本申请中的附图，对本申请中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
下面结合图1-图7描述本申请的数据处理方法。The data processing method of the present application is described below in conjunction with Figures 1 to 7.
图1是本申请实施例提供的数据处理方法的流程示意图之一，如图1所示，该数据处理方法包括以下步骤。FIG. 1 is one of the flow charts of the data processing method provided in the embodiment of the present application. As shown in FIG. 1 , the data processing method includes the following steps.
步骤101、获取待处理数据。Step 101: Obtain data to be processed.
其中，待处理数据可以为上下文数据，或者样本数据等非上下文数据。The data to be processed may be context data, or non-context data such as sample data.
示例地，用户可以通过预设接口提交待处理数据，例如，用户通过clinic***接口提交待处理数据。For example, the user may submit the data to be processed through a preset interface, for example, the user may submit the data to be processed through a clinic system interface.
步骤102、获取用户对所述待处理数据配置的数据处理规则。Step 102: Obtain the data processing rules configured by the user for the data to be processed.
其中，所述数据处理规则包括以下至少之一：数据解析规则、数据存储规则和数据发送规则。The data processing rules include at least one of the following: data parsing rules, data storage rules and data sending rules.
示例地，在获取到用户提交的待处理数据时，可以向用户显示注册准入许可的配置模块，用户可以基于自身需求在配置模块上对待处理数据的数据处理规则进行配置，例如，以待处理数据为用户上下文数据为例，用户上下文数据包括5个字段，分别为字段1、字段2、字段3、字段4和字段5，若用户只想对字段3进行解析和存储，则可以在配置模块上配置的 需要解析的内容为字段3，需要存储的内容为字段3的具体信息，则用户配置的解析的内容为字段3，需要存储的内容为字段3的具体信息即为待处理数据的数据处理规则。For example, when the data to be processed submitted by the user is obtained, the configuration module of the registration access permission can be displayed to the user. The user can configure the data processing rules for the data to be processed in the configuration module based on his own needs. For example, taking the data to be processed as user context data as an example, the user context data includes 5 fields, namely field 1, field 2, field 3, field 4 and field 5. If the user only wants to parse and store field 3, the content to be parsed can be configured in the configuration module as field 3, and the content to be stored can be the specific information of field 3. Then the content to be parsed configured by the user as field 3 and the specific information of field 3 are the data processing rules for the data to be processed.
步骤103、基于所述数据处理规则对所述待处理数据进行处理。Step 103: Process the data to be processed based on the data processing rules.
示例地，在获取到用户对待处理数据配置的数据处理规则时，可以基于数据处理规则对待处理数据进行对应处理；例如，获取的数据处理规则为需要解析的内容为字段3，需要存储的内容为字段3的具体信息，则可以对待处理数据的字段3进行解析，解析得到字段3的具体信息，也就是字段3的具体值，然后将字段3的具体值进行存储，从而实现了对待处理数据的具体处理过程。For example, when the data processing rules configured by the user for the data to be processed are obtained, the data to be processed can be processed accordingly based on the data processing rules; for example, the obtained data processing rules are that the content to be parsed is field 3, and the content to be stored is the specific information of field 3, then field 3 of the data to be processed can be parsed to obtain the specific information of field 3, that is, the specific value of field 3, and then the specific value of field 3 can be stored, thereby realizing the specific processing process of the data to be processed.
本申请提供的数据处理方法，在获取到待处理数据时，获取用户对待处理数据配置的数据处理规则，基于数据处理规则对待处理数据进行处理。可知，本申请是用户基于待处理数据配置对应的数据处理规则，实现了数据的灵活处理。The data processing method provided by the present application obtains the data processing rules configured by the user for the data to be processed when the data to be processed is obtained, and processes the data to be processed based on the data processing rules. It can be seen that the present application is a data processing rule corresponding to the data to be processed configured by the user, thereby realizing flexible data processing.
可选地，所述数据解析规则用于指示所述待处理数据的格式标识和待解析数据的第一标识；图2是本申请实施例提供的数据处理方法的流程示意图之二，如图2所示，上述步骤103具体可通过以下步骤实现。Optionally, the data parsing rule is used to indicate the format identifier of the data to be processed and the first identifier of the data to be parsed; Figure 2 is the second flow chart of the data processing method provided in an embodiment of the present application. As shown in Figure 2, the above step 103 can be specifically implemented through the following steps.
步骤1031、在所述数据处理规则包括所述数据解析规则时，基于所述待处理数据的格式标识确定目标解析器。Step 1031: When the data processing rule includes the data parsing rule, determine a target parser based on the format identifier of the data to be processed.
示例地，由于用户提交的待处理数据的数据格式和内容可能都各不相同，为了兼容多种数据格式的待处理数据，可以预先将每种数据格式的解析器以组件方式添加至配置模块中，并建立解析器列表，在解析器列表中存储有格式标识与解析器的对应关系；在用户配置的数据处理规则中包括数据解析规则时，获取数据解析规则的具体内容，即获取到用于指示待处理数据的格式标识和待解析数据的第一标识，该格式标识包括但不限于json格式的标识、xml格式的标识、csv格式的标识等；以json格式的标识为例，则在获取到用于指示待处理数据的格式标识为json格式的标识时，从解析器列表中查找与json格式的标识对应的目标解析器。For example, since the data format and content of the data to be processed submitted by the user may be different, in order to be compatible with data to be processed in multiple data formats, the parser of each data format can be pre-added to the configuration module as a component, and a parser list can be established, in which the correspondence between the format identifier and the parser is stored; when the data processing rules configured by the user include data parsing rules, the specific content of the data parsing rules is obtained, that is, the format identifier for indicating the data to be processed and the first identifier of the data to be parsed are obtained, and the format identifier includes but is not limited to a json format identifier, an xml format identifier, a csv format identifier, etc.; taking the json format identifier as an example, when the format identifier for indicating that the data to be processed is a json format identifier, the target parser corresponding to the json format identifier is searched from the parser list.
步骤1032、通过所述目标解析器从所述待处理数据中解析与所述第一标识匹配的数据，得到解析数据。Step 1032: parse the data matching the first identifier from the data to be processed by the target parser to obtain parsed data.
示例地，在基于待处理数据的格式标识确定目标解析器时，目标解析器从待处理数据中查找待解析数据的第一标识，找到待解析的位置，进而在待解析的位置处解析对应的数据；例如，待解析数据的第一标识为字段3，则目标解析器在待处理数据中确定字段3的位置，然后将字段3的具体值进行解析，得到解析数据。For example, when the target parser is determined based on the format identifier of the data to be processed, the target parser searches for the first identifier of the data to be parsed from the data to be processed, finds the position to be parsed, and then parses the corresponding data at the position to be parsed; for example, the first identifier of the data to be parsed is field 3, then the target parser determines the position of field 3 in the data to be processed, and then parses the specific value of field 3 to obtain parsed data.
图3是本申请实施例提供的解析流程示意图，如图3所示，待处理数据为上下文数据，在配置模块中选择对应的目标解析器，并在用于配置格式标识的位置处配置待处理数据的格式标识，在配置待解析数据标识的位置处配置待解析数据的第一标识；在格式标识为json格式的标识时，基于json格式对应的解析器对上下文数据进行解析，得到解析数据；在格式标识为xml格式的标识时，基于xml格式对应的解析器对上下文数据进行解析，得到解析数据；在格式标识为csv格式的标识时，基于csv格式对应的解析器对上下文数据进行解析，得到解析数据，其中，解析数据中可以包括第一标识对应的具体解析内容和具体解析内容在上下文数据中的路径信息等。Figure 3 is a schematic diagram of the parsing process provided by an embodiment of the present application. As shown in Figure 3, the data to be processed is context data. The corresponding target parser is selected in the configuration module, and the format identifier of the data to be processed is configured at the position for configuring the format identifier, and the first identifier of the data to be parsed is configured at the position for configuring the identifier of the data to be parsed; when the format identifier is an identifier in the json format, the context data is parsed based on the parser corresponding to the json format to obtain parsed data; when the format identifier is an identifier in the xml format, the context data is parsed based on the parser corresponding to the xml format to obtain parsed data; when the format identifier is an identifier in the csv format, the context data is parsed based on the parser corresponding to the csv format to obtain parsed data, wherein the parsed data may include specific parsed content corresponding to the first identifier and path information of the specific parsed content in the context data, etc.
本申请实施例提供的数据处理方法，在数据处理规则包括数据解析规则时，可以基于数据解析规则确定对应的目标解析器，进而基于目标解析器对待处理数据进行解析，实现了基于用户配置的数据解析规则对待处理数据中用户需求的数据进行解析，体现了解析过程的灵活配置和实现。The data processing method provided in the embodiment of the present application can determine the corresponding target parser based on the data parsing rules when the data processing rules include data parsing rules, and then parse the data to be processed based on the target parser, thereby realizing parsing the data required by the user in the data to be processed based on the data parsing rules configured by the user, and embodies the flexible configuration and implementation of the parsing process.
可选地，所述数据存储规则用于指示待存储数据的第二标识；图4是本申请实施例提供的数据处理方法的流程示意图之三，如图4所示，上述步骤103具体可通过以下步骤实现。Optionally, the data storage rule is used to indicate a second identifier of the data to be stored; FIG4 is a third flow chart of the data processing method provided in an embodiment of the present application. As shown in FIG4 , the above step 103 can be specifically implemented through the following steps.
需要说明的是，基于待处理数据的不同，下述执行的步骤也不同；在待处理数据包括上下文数据时，则执行步骤1033和步骤1035；在待处理数据包括非上下文数据时，则执行步骤1034和步骤1035。It should be noted that the following execution steps are different based on the different data to be processed; when the data to be processed includes context data, step 1033 and step 1035 are executed; when the data to be processed includes non-context data, step 1034 and step 1035 are executed.
步骤1033、在所述数据处理规则包括所述数据存储规则、且所述待处理数据包括上下文数据时，从解析数据中确定与所述第二标识匹配的待存储数据；所述解析数据为从所述待处理数据中解析得到的数据。Step 1033: When the data processing rule includes the data storage rule and the data to be processed includes context data, determine the data to be stored that matches the second identifier from the parsed data; the parsed data is data parsed from the data to be processed.
示例地，在数据处理规则包括数据存储规则、且待处理数据包括上下文数据时，获取数据存储规则的具体内容，即获取到用于指示待存储数据 的第二标识，然后从解析数据中查找与第二标识对应的数据，将第二标识对应的数据作为待存储数据进行存储。例如，待处理数据包括上下文数据，解析数据为字段2的值和字段3的值，若用户配置的数据存储规则为将字段2的值进行存储，则从解析数据中获取字段2的值，然后将字段2的值存储在预设位置。For example, when the data processing rule includes a data storage rule, and the data to be processed includes context data, the specific content of the data storage rule is obtained, that is, the second identifier for indicating the data to be stored is obtained, and then the data corresponding to the second identifier is searched from the parsed data, and the data corresponding to the second identifier is stored as the data to be stored. For example, the data to be processed includes context data, and the parsed data is the value of field 2 and the value of field 3. If the data storage rule configured by the user is to store the value of field 2, the value of field 2 is obtained from the parsed data, and then the value of field 2 is stored in a preset position.
步骤1034、在所述数据处理规则包括所述数据存储规则，且所述待处理数据包括非上下文数据时，从所述待处理数据中确定与所述第二标识匹配的待存储数据。Step 1034: When the data processing rule includes the data storage rule and the data to be processed includes non-context data, determine the data to be stored that matches the second identifier from the data to be processed.
示例地，在数据处理规则包括数据存储规则、且待处理数据包括非上下文数据时，获取数据存储规则的具体内容，即获取到用于指示待存储数据的第二标识，然后从解析数据中查找与第二标识对应的数据，将第二标识对应的数据作为待存储数据进行存储。例如，待处理数据包括从第一网际互连协议(Internet Protocol，IP)地址获取的数据1和从第二IP地址获取的数据2，若用户配置的数据存储规则为将从第一IP地址获取的数据1进行存储，则从待处理数据中获取从第一IP地址获取的数据1，然后将从第一IP地址获取的数据1存储在预设位置。For example, when the data processing rule includes a data storage rule, and the data to be processed includes non-context data, the specific content of the data storage rule is obtained, that is, the second identifier for indicating the data to be stored is obtained, and then the data corresponding to the second identifier is searched from the parsed data, and the data corresponding to the second identifier is stored as the data to be stored. For example, the data to be processed includes data 1 obtained from a first Internet Protocol (IP) address and data 2 obtained from a second IP address. If the data storage rule configured by the user is to store data 1 obtained from the first IP address, data 1 obtained from the first IP address is obtained from the data to be processed, and then data 1 obtained from the first IP address is stored in a preset location.
步骤1035、将所述待存储数据进行存储。Step 1035: store the data to be stored.
示例地，在确定待存储数据时，将待存储数据存储在预设位置，用户可以自行配置具体预设位置，也可以由电子设备基于设备内存的使用情况来自动选取用于存储的预设位置，例如，待存储数据为告警数据，可以通过filing存档告警数据，本申请对此不作限定。For example, when determining the data to be stored, the data to be stored is stored in a preset location. The user can configure the specific preset location by himself, or the electronic device can automatically select the preset location for storage based on the usage of the device memory. For example, if the data to be stored is alarm data, the alarm data can be archived through filing. This application is not limited to this.
需要说明的是，由于上下文数据是与业务无关的数据，由用户进行提交，业务处理中不需要上下文数据，只是在存储时进行存储处理，所以业务处理中没有对上下文数据进行解析，所以在待处理数据为上下文数据时，需要对上下文数据进行解析；而非上下文数据是业务流程中生成的数据，在业务中已经完成了数据的解析处理，因此在待处理数据为非上下文数据时，由于非上下文数据是已经解析过的数据，所以无需解析。It should be noted that since context data is data unrelated to the business and is submitted by the user, the context data is not required in business processing and is only stored during storage. Therefore, the context data is not parsed in business processing. Therefore, when the data to be processed is context data, the context data needs to be parsed; and non-context data is data generated in the business process, and the data parsing processing has been completed in the business. Therefore, when the data to be processed is non-context data, since the non-context data is data that has been parsed, it does not need to be parsed.
本申请实施例提供的数据处理方法，在数据处理规则包括数据存储规则时，可以基于数据存储规则对待处理数据或者解析数据进行存储，实现了基于用户配置的数据存储规则对待处理数据中用户需求的数据进行存 储，体现了存储过程的灵活配置和实现。The data processing method provided in the embodiment of the present application can store the processed data or parsed data based on the data storage rules when the data processing rules include data storage rules, thereby realizing the storage of user-required data in the processed data based on the data storage rules configured by the user, and embodies the flexible configuration and implementation of the storage process.
可选地，所述数据发送规则用于指示待发送数据的第三标识；图5是本申请实施例提供的数据处理方法的流程示意图之四，如图5所示，上述步骤103具体可通过以下步骤实现。Optionally, the data sending rule is used to indicate a third identifier of the data to be sent; FIG. 5 is a fourth flow chart of the data processing method provided in an embodiment of the present application. As shown in FIG. 5 , the above step 103 can be specifically implemented through the following steps.
需要说明的是，基于待处理数据的不同，下述执行的步骤也不同；在待处理数据包括上下文数据时，则执行步骤1036和步骤1038；在待处理数据包括非上下文数据时，则执行步骤1037和步骤1038。It should be noted that the following execution steps are different based on the different data to be processed; when the data to be processed includes context data, step 1036 and step 1038 are executed; when the data to be processed includes non-context data, step 1037 and step 1038 are executed.
步骤1036、在所述数据处理规则包括所述数据发送规则、且所述待处理数据包括上下文数据时，从解析数据中确定与所述第三标识匹配的待发送数据；所述解析数据为从所述待处理数据中解析得到的数据。Step 1036: When the data processing rule includes the data sending rule and the data to be processed includes context data, determine the data to be sent that matches the third identifier from the parsed data; the parsed data is data parsed from the data to be processed.
示例地，在数据处理规则包括数据发送规则、且待处理数据包括上下文数据时，获取数据发送规则的具体内容，即获取到用于指示待发送数据的第三标识，然后从解析数据中查找是否有与第三标识匹配的数据，在解析数据中查找到有与第三标识匹配的数据时，将第三标识对应的数据作为待发送数据进行发送。例如，待处理数据包括上下文数据，解析数据为字段2的值和字段3的值，若用户配置的数据发送规则为将字段3的值进行发送，则从解析数据中获取字段3的值，然后将字段3的值通过默认发送通道或者用户设定的发送通道进行发送。For example, when the data processing rules include data sending rules, and the data to be processed include context data, the specific content of the data sending rules is obtained, that is, the third identifier used to indicate the data to be sent is obtained, and then the data matching the third identifier is found in the parsed data. When the data matching the third identifier is found in the parsed data, the data corresponding to the third identifier is sent as the data to be sent. For example, the data to be processed includes context data, and the parsed data is the value of field 2 and the value of field 3. If the data sending rule configured by the user is to send the value of field 3, the value of field 3 is obtained from the parsed data, and then the value of field 3 is sent through the default sending channel or the sending channel set by the user.
步骤1037、在所述数据处理规则包括所述数据发送规则，且所述待处理数据包括非上下文数据时，从所述待处理数据中确定与所述第三标识匹配的待发送数据。Step 1037: When the data processing rule includes the data sending rule and the data to be processed includes non-context data, determine the data to be sent that matches the third identifier from the data to be processed.
示例地，在数据处理规则包括数据存储规则、且待处理数据包括非上下文数据时，获取数据发送规则的具体内容，即获取到用于指示待存储数据的第三标识，然后从解析数据中查找是否有与第三标识匹配的数据，在待处理数据中查找到有与第三标识匹配的数据时，将第三标识对应的数据作为待发送数据进行发送。例如，待处理数据包括从第一网际互连协议(Internet Protocol，IP)地址获取的数据1和从第二IP地址获取的数据2，若用户配置的数据发送规则为将从第二IP地址获取的数据2进行发送，则从待处理数据中获取从第二IP地址获取的数据2，然后将从第二IP地址获取的数据2通过默认发送通道或者用户设定的发送通道进行发送。For example, when the data processing rules include data storage rules and the data to be processed include non-context data, the specific content of the data sending rules is obtained, that is, a third identifier for indicating the data to be stored is obtained, and then the data matching the third identifier is searched from the parsed data. When data matching the third identifier is found in the data to be processed, the data corresponding to the third identifier is sent as the data to be sent. For example, the data to be processed includes data 1 obtained from a first Internet Protocol (IP) address and data 2 obtained from a second IP address. If the data sending rule configured by the user is to send data 2 obtained from the second IP address, data 2 obtained from the second IP address is obtained from the data to be processed, and then data 2 obtained from the second IP address is sent through the default sending channel or the sending channel set by the user.
需要说明的是，在从解析数据中查找是否有与第三标识匹配的数据时，可以采用逻辑运算、算术运算、包含关系、或者正则表达式等运算方式进行匹配，其中，包含关系有包括(in)、不包括(not in)、参数字符串是否在原字符串的头部(startWith)、参数字符串是否在原字符串的尾部(endWith)等，本申请对此不作限定。It should be noted that when searching for data matching the third identifier in the parsed data, logical operations, arithmetic operations, inclusion relationships, or regular expressions may be used for matching, where the inclusion relationships include inclusion (in), exclusion (not in), whether the parameter string is at the head of the original string (startWith), whether the parameter string is at the end of the original string (endWith), etc., and this application does not limit this.
步骤1038、将所述待发送数据进行发送。Step 1038: Send the data to be sent.
示例地，在确定待发送数据时，将待发送数据默认发送通道或者用户设定的发送通道进行发送，本申请对此不作限定。For example, when the data to be sent is determined, the data to be sent is sent through a default sending channel or a sending channel set by the user, which is not limited in the present application.
本申请实施例提供的数据处理方法，在数据处理规则包括数据发送规则时，可以基于数据发送规则对待处理数据或者解析数据进行发送，实现了基于用户配置的数据发送规则对待处理数据中用户需求的数据的发送，体现了发送过程的灵活配置和实现。The data processing method provided in the embodiment of the present application can send the processed data or parsed data based on the data sending rules when the data processing rules include data sending rules, thereby realizing the sending of user-required data in the processed data based on the data sending rules configured by the user, and embodies the flexible configuration and implementation of the sending process.
可选地，所述数据发送规则还用于指示发送通道标识；上述步骤1038具体可通过以下方式实现：Optionally, the data sending rule is also used to indicate a sending channel identifier; the above step 1038 can be specifically implemented in the following manner:
基于所述发送通道标识确定目标发送通道；将所述待发送数据通过所述目标发送通道进行发送。A target sending channel is determined based on the sending channel identifier; and the data to be sent is sent through the target sending channel.
示例地，用户还可以基于自身的需求，在配置模块中配置发送数据时采用的发送通道的发送通道标识，其中，发送通道包括kafka、syslog和snmp等多种发送通道，且发送通道也可通过实现发送接口进行扩展。例如，用户配置的发送通道标识为kafka对应的发送通道标识，则在确定待发送数据时，则通过kafka发送通道将待发送数据进行发送。For example, the user can also configure the sending channel identifier of the sending channel used when sending data in the configuration module based on their own needs, wherein the sending channel includes multiple sending channels such as kafka, syslog and snmp, and the sending channel can also be expanded by implementing the sending interface. For example, if the sending channel identifier configured by the user is the sending channel identifier corresponding to kafka, then when the data to be sent is determined, the data to be sent is sent through the kafka sending channel.
图6是本申请实施例提供的数据处理流程示意图之一，如图6所示，以待处理数据为待存储数据为例，在配置模块中用于配置待存储数据的标识的位置处配置待存储数据的第二标识，以待处理数据为待发送数据为例，在配置模块中用于配置待发送数据的发送通道标识处配置待发送数据的发送通道标识，在完成数据处理规则的配置后，基于配置的数据处理规则对待发送数据进行相应处理；例如，在待存储数据的第二标识为filing存档时，将待存储数据存储在存储引擎中，例如将待存储数据存储在mongo数据库中；在待发送数据的发送通道标识为kafka对应的发送通道标识时，将待发送数据通过kafka发送通道进行发送；在待发送数据的发送通道标 识为syslog对应的发送通道标识时，将待发送数据通过syslog发送通道进行发送。Figure 6 is one of the data processing flow diagrams provided by an embodiment of the present application. As shown in Figure 6, taking the data to be processed as the data to be stored as an example, the second identifier of the data to be stored is configured at the position of the identifier of the data to be stored in the configuration module; taking the data to be processed as the data to be sent as an example, the sending channel identifier of the data to be sent is configured at the sending channel identifier of the data to be sent in the configuration module. After completing the configuration of the data processing rules, the data to be sent is processed accordingly based on the configured data processing rules; for example, when the second identifier of the data to be stored is filing archive, the data to be stored is stored in the storage engine, for example, the data to be stored is stored in the mongo database; when the sending channel identifier of the data to be sent is the sending channel identifier corresponding to kafka, the data to be sent is sent through the kafka sending channel; when the sending channel identifier of the data to be sent is the sending channel identifier corresponding to syslog, the data to be sent is sent through the syslog sending channel.
本申请实施例提供的数据处理方法，用户可以自行配置发送通道标识，可基于用户配置的发送通道标识对待发送数据进行发送，实现了发送通道的灵活配置。In the data processing method provided in the embodiment of the present application, the user can configure the sending channel identifier by himself, and the data to be sent can be sent based on the sending channel identifier configured by the user, thereby realizing flexible configuration of the sending channel.
可选地，所述数据发送规则还用于指示待发送数据的格式标识；上述步骤1038具体可通过以下方式实现：Optionally, the data sending rule is also used to indicate a format identifier of the data to be sent; the above step 1038 can be specifically implemented in the following manner:
基于所述待发送数据的格式标识将所述待发送数据进行格式转换，得到第一目标发送数据；将所述第一目标发送数据进行发送。The format of the data to be sent is converted based on the format identifier of the data to be sent to obtain first target sending data; and the first target sending data is sent.
示例地，用户还可以基于自身的需求，在配置模块中配置待发送数据的格式标识，在确定待发送数据时，确定待发送数据的当前数据格式是否为用户配置的格式标识对应的数据格式，在确定待发送数据的当前数据格式不为用户配置的格式标识对应的数据格式时，则基于用户配置的格式标识对应的数据格式对待发送数据进行格式转换，得到与用户配置的数据格式相同的待发送数据，即得到第一目标发送数据，再将第一目标发送数据进行发送。For example, the user can also configure the format identifier of the data to be sent in the configuration module based on his or her own needs. When determining the data to be sent, it is determined whether the current data format of the data to be sent is the data format corresponding to the format identifier configured by the user. When it is determined that the current data format of the data to be sent is not the data format corresponding to the format identifier configured by the user, the format of the data to be sent is converted based on the data format corresponding to the format identifier configured by the user to obtain the data to be sent with the same data format as the user configured, that is, the first target sending data is obtained, and the first target sending data is sent.
本申请实施例提供的数据处理方法，用户可以自行配置待发送数据的格式标识，可基于用户配置的待发送数据的格式标识对待发送数据进行格式转换，实现了待发送数据的数据格式的灵活配置。The data processing method provided in the embodiment of the present application allows the user to configure the format identifier of the data to be sent by himself, and can perform format conversion on the data to be sent based on the format identifier of the data to be sent configured by the user, thereby realizing flexible configuration of the data format of the data to be sent.
可选地，所述数据发送规则还用于指示待发送数据的重组信息；上述步骤1038具体可通过以下方式实现：Optionally, the data transmission rule is also used to indicate reorganization information of the data to be transmitted; the above step 1038 can be specifically implemented in the following manner:
基于所述待发送数据的重组信息将所述待发送数据的内容进行重组，得到第二目标发送数据；将所述第二目标发送数据进行发送。The content of the data to be sent is reorganized based on the reorganization information of the data to be sent to obtain second target sending data; and the second target sending data is sent.
其中，重组信息包括定义新的数据结构或者定义新的字段名称等。The reorganization information includes defining a new data structure or defining a new field name, etc.
示例地，用户还可以基于自身的需求，在配置模块中配置待发送数据的重组信息，在确定待发送数据时，基于用户配置的待发送数据的重组信息对待发送数据进行重组，并将重组得到的第二目标发送数据进行发送。例如，用户配置的待发送数据的重组信息为将字段2的名称修改为abc，则在确定待发送数据时，从待发送数据中查找字段2，然后将字段2的名称修改为abc，实现了待发送数据的重组；再例如，用户配置的待发送数 据的重组信息为将待发送数据的数据结构修改为数据结构1，则在确定待发送数据时，需要将待发送数据的当前数据结构按照数据结构1进行修改，实现了待发送数据的重组。For example, the user can also configure the reorganization information of the data to be sent in the configuration module based on his own needs. When the data to be sent is determined, the data to be sent is reorganized based on the reorganization information of the data to be sent configured by the user, and the reorganized second target data to be sent is sent. For example, the reorganization information of the data to be sent configured by the user is to modify the name of field 2 to abc. When the data to be sent is determined, field 2 is searched from the data to be sent, and then the name of field 2 is modified to abc, so that the reorganization of the data to be sent is achieved; for another example, the reorganization information of the data to be sent configured by the user is to modify the data structure of the data to be sent to data structure 1. When the data to be sent is determined, the current data structure of the data to be sent needs to be modified according to data structure 1, so that the reorganization of the data to be sent is achieved.
图7是本申请实施例提供的数据处理流程示意图之二，如图7所示，以待处理数据为待发送数据为例，在配置模块中用于配置待发送数据的重组信息的位置处配置待存储数据的重组信息，在用于配置待发送数据的格式标识处配置待发送数据的格式标识，在完成数据处理规则的配置后，基于配置的数据处理规则对待发送数据进行相应处理；例如，待发送数据的重组信息为更新字段名称，则将待发送数据进行字段名称的更新；在待发送数据的格式标识为json格式标识时，将待发送数据的数据格式转化为json格式；在待发送数据的格式标识为xml格式标识时，将待发送数据的数据格式转化为xml格式；在待发送数据的格式标识为默认的csv格式标识时，将待发送数据的数据格式转化为csv格式。Figure 7 is a second schematic diagram of the data processing flow provided by an embodiment of the present application. As shown in Figure 7, taking the case where the data to be processed is the data to be sent as an example, the reorganization information of the data to be stored is configured at the position used to configure the reorganization information of the data to be sent in the configuration module, and the format identifier of the data to be sent is configured at the position used to configure the format identifier of the data to be sent. After completing the configuration of the data processing rules, the data to be sent is processed accordingly based on the configured data processing rules; for example, if the reorganization information of the data to be sent is to update the field name, the field name of the data to be sent is updated; when the format identifier of the data to be sent is a json format identifier, the data format of the data to be sent is converted into a json format; when the format identifier of the data to be sent is an xml format identifier, the data format of the data to be sent is converted into an xml format; when the format identifier of the data to be sent is a default csv format identifier, the data format of the data to be sent is converted into a csv format.
本申请实施例提供的数据处理方法，用户可以自行配置待发送数据的重组信息，可基于用户配置的待发送数据的重组信息对待发送数据进行重组，实现了待发送数据的数据结构或字段名称的灵活配置。The data processing method provided in the embodiment of the present application allows the user to configure the reorganization information of the data to be sent by himself, and the data to be sent can be reorganized based on the reorganization information of the data to be sent configured by the user, thereby realizing flexible configuration of the data structure or field name of the data to be sent.
本申请实施例提供的数据处理方法，可以应用于不同生产线的数据处理场景，为不同生产线的数据处理提供一个业务中转平台，业务中转平台通过规则模板构建了一套数据处理规则，规范了不同生产线间数据的上传和调用，实现了不同生产线间数据的自动流转；任一生产线的数据都可以作为待处理数据，能够基于用户需求，可配置化进行待处理数据的对应处理，从而实现业务中转平台中每个模块的复用，并满足不同用户的需求；即在获取到待处理数据时，获取用户对待处理数据配置的数据处理规则，基于数据处理规则对待处理数据进行处理，实现了不同数据的灵活处理。The data processing method provided in the embodiment of the present application can be applied to data processing scenarios of different production lines, and provide a business transit platform for data processing of different production lines. The business transit platform constructs a set of data processing rules through rule templates, standardizes the uploading and calling of data between different production lines, and realizes the automatic flow of data between different production lines; the data of any production line can be used as data to be processed, and can be configurable to perform corresponding processing on the data to be processed based on user needs, thereby realizing the reuse of each module in the business transit platform and meeting the needs of different users; that is, when the data to be processed is obtained, the data processing rules configured by the user for the data to be processed are obtained, and the data to be processed is processed based on the data processing rules, thereby realizing flexible processing of different data.
具体地，在数据处理规则包括数据解析规则时，可以基于数据解析规则确定对应的目标解析器，进而基于目标解析器对待处理数据进行解析，实现了基于用户配置的数据解析规则对待处理数据中用户需求的数据进行解析，体现了解析过程的灵活配置和实现。Specifically, when the data processing rules include data parsing rules, the corresponding target parser can be determined based on the data parsing rules, and then the data to be processed can be parsed based on the target parser, thereby realizing parsing the user's required data in the data to be processed based on the user-configured data parsing rules, reflecting the flexible configuration and implementation of the parsing process.
在数据处理规则包括数据存储规则时，可以基于数据存储规则对待处理数据或者解析数据进行存储，实现了基于用户配置的数据存储规则对待 处理数据中用户需求的数据进行存储，体现了存储过程的灵活配置和实现。When the data processing rules include data storage rules, the data to be processed or the parsed data can be stored based on the data storage rules, and the data required by the user in the data to be processed can be stored based on the data storage rules configured by the user, which reflects the flexible configuration and implementation of the storage process.
在数据处理规则包括数据发送规则时，可以基于数据发送规则对待处理数据或者解析数据进行发送，实现了基于用户配置的数据发送规则对待处理数据中用户需求的数据的发送，体现了发送过程的灵活配置和实现。When the data processing rules include data sending rules, the data to be processed or the parsed data can be sent based on the data sending rules, thereby realizing the sending of user-required data in the data to be processed based on the data sending rules configured by the user, reflecting the flexible configuration and implementation of the sending process.
另外，用户可以自行配置发送通道标识，可基于用户配置的发送通道标识对待发送数据进行发送，实现了发送通道的灵活配置；用户还可以自行配置待发送数据的格式标识，可基于用户配置的待发送数据的格式标识对待发送数据进行格式转换，实现了待发送数据的数据格式的灵活配置；用户还可以自行配置待发送数据的重组信息，可基于用户配置的待发送数据的重组信息对待发送数据进行重组，实现了待发送数据的数据结构或字段名称的灵活配置。In addition, users can configure the sending channel identifier by themselves, and send the data to be sent based on the sending channel identifier configured by the user, thus realizing flexible configuration of the sending channel; users can also configure the format identifier of the data to be sent by themselves, and convert the format of the data to be sent based on the format identifier of the data to be sent configured by the user, thus realizing flexible configuration of the data format of the data to be sent; users can also configure the reorganization information of the data to be sent by themselves, and reorganize the data to be sent based on the reorganization information of the data to be sent configured by the user, thus realizing flexible configuration of the data structure or field name of the data to be sent.
下面对本申请提供的数据处理装置进行描述，下文描述的数据处理装置与上文描述的数据处理方法可相互对应参照。The data processing device provided by the present application is described below. The data processing device described below and the data processing method described above can be referenced to each other.
图8是本申请实施例提供的数据处理装置的结构示意图，如图8所示，该数据处理装置800包括第一获取单元801、第二获取单元802和处理单元803，其中：FIG8 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. As shown in FIG8 , the data processing device 800 includes a first acquisition unit 801, a second acquisition unit 802 and a processing unit 803, wherein:
第一获取单元801，用于获取待处理数据；A first acquisition unit 801 is used to acquire data to be processed;
第二获取单元802，用于获取用户对所述待处理数据配置的数据处理规则；The second acquisition unit 802 is used to acquire the data processing rules configured by the user for the data to be processed;
处理单元803，用于基于所述数据处理规则对所述待处理数据进行处理。The processing unit 803 is used to process the data to be processed based on the data processing rule.
本申请提供的数据处理装置，在获取到待处理数据时，获取用户对待处理数据配置的数据处理规则，基于数据处理规则对待处理数据进行处理。可知，本申请是用户基于待处理数据配置对应的数据处理规则，实现了数据的灵活处理。The data processing device provided by the present application obtains the data processing rules configured by the user for the data to be processed when the data to be processed is obtained, and processes the data to be processed based on the data processing rules. It can be seen that the present application is that the user configures the corresponding data processing rules based on the data to be processed, thereby realizing flexible data processing.
基于上述任一实施例，所述数据处理规则包括以下至少之一：数据解析规则、数据存储规则和数据发送规则。Based on any of the above embodiments, the data processing rules include at least one of the following: data parsing rules, data storage rules and data sending rules.
基于上述任一实施例，所述数据解析规则用于指示所述待处理数据的格式标识和待解析数据的第一标识；所述处理单元803具体用于：Based on any of the above embodiments, the data parsing rule is used to indicate the format identifier of the data to be processed and the first identifier of the data to be parsed; the processing unit 803 is specifically used to:
在所述数据处理规则包括所述数据解析规则时，基于所述待处理数据 的格式标识确定目标解析器；When the data processing rule includes the data parsing rule, determining a target parser based on a format identifier of the data to be processed;
通过所述目标解析器从所述待处理数据中解析与所述第一标识匹配的数据，得到解析数据。The target parser parses data matching the first identifier from the data to be processed to obtain parsed data.
基于上述任一实施例，所述数据存储规则用于指示待存储数据的第二标识；所述处理单元803还具体用于：Based on any of the above embodiments, the data storage rule is used to indicate the second identifier of the data to be stored; and the processing unit 803 is further specifically used to:
在所述数据处理规则包括所述数据存储规则、且所述待处理数据包括上下文数据时，从解析数据中确定与所述第二标识匹配的待存储数据；所述解析数据为从所述待处理数据中解析得到的数据；When the data processing rule includes the data storage rule and the data to be processed includes context data, determining the data to be stored that matches the second identifier from the parsed data; the parsed data is data parsed from the data to be processed;
在所述数据处理规则包括所述数据存储规则，且所述待处理数据包括非上下文数据时，从所述待处理数据中确定与所述第二标识匹配的待存储数据；When the data processing rule includes the data storage rule, and the data to be processed includes non-context data, determining the data to be stored that matches the second identifier from the data to be processed;
将所述待存储数据进行存储。The data to be stored is stored.
基于上述任一实施例，所述数据发送规则用于指示待发送数据的第三标识；所述处理单元803还具体用于：Based on any of the above embodiments, the data sending rule is used to indicate a third identifier of the data to be sent; and the processing unit 803 is further specifically used to:
在所述数据处理规则包括所述数据发送规则、且所述待处理数据包括上下文数据时，从解析数据中确定与所述第三标识匹配的待发送数据；所述解析数据为从所述待处理数据中解析得到的数据；When the data processing rule includes the data sending rule and the data to be processed includes context data, determining the data to be sent that matches the third identifier from the parsed data; the parsed data is data parsed from the data to be processed;
在所述数据处理规则包括所述数据发送规则，且所述待处理数据包括非上下文数据时，从所述待处理数据中确定与所述第三标识匹配的待发送数据；When the data processing rule includes the data sending rule, and the data to be processed includes non-context data, determining the data to be sent that matches the third identifier from the data to be processed;
将所述待发送数据进行发送。The data to be sent is sent.
基于上述任一实施例，所述数据发送规则还用于指示发送通道标识；所述处理单元803还具体用于：Based on any of the above embodiments, the data sending rule is further used to indicate a sending channel identifier; and the processing unit 803 is further specifically used to:
基于所述发送通道标识确定目标发送通道；Determine a target sending channel based on the sending channel identifier;
将所述待发送数据通过所述目标发送通道进行发送。The data to be sent is sent through the target sending channel.
基于上述任一实施例，所述数据发送规则还用于指示待发送数据的格式标识；所述处理单元803还具体用于：Based on any of the above embodiments, the data sending rule is further used to indicate a format identifier of the data to be sent; and the processing unit 803 is further specifically used to:
基于所述待发送数据的格式标识将所述待发送数据进行格式转换，得到第一目标发送数据；Converting the format of the data to be sent based on the format identifier of the data to be sent to obtain first target sending data;
将所述第一目标发送数据进行发送。The first target sending data is sent.
基于上述任一实施例，所述数据发送规则还用于指示待发送数据的重组信息；所述处理单元803还具体用于：Based on any of the above embodiments, the data sending rule is further used to indicate reorganization information of the data to be sent; and the processing unit 803 is further specifically used to:
基于所述待发送数据的重组信息将所述待发送数据的内容进行重组，得到第二目标发送数据；reorganizing the content of the data to be sent based on the reorganization information of the data to be sent to obtain second target sending data;
将所述第二目标发送数据进行发送。The second target sending data is sent.
图9是本申请实施例提供的电子设备的实体结构示意图，如图9所示，该电子设备可以包括：处理器(processor)910、通信接口(Communications Interface)920、存储器(memory)930和通信总线940，其中，处理器910，通信接口920，存储器930通过通信总线940完成相互间的通信。处理器910可以调用存储器930中的逻辑指令，以执行数据处理方法，该方法包括：获取待处理数据；FIG9 is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present application. As shown in FIG9 , the electronic device may include: a processor 910, a communications interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communications interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute a data processing method, which includes: obtaining data to be processed;
获取用户对所述待处理数据配置的数据处理规则；Obtaining data processing rules configured by a user for the data to be processed;
基于所述数据处理规则对所述待处理数据进行处理。The data to be processed is processed based on the data processing rule.
此外，上述的存储器930中的逻辑指令可以通过软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., and other media that can store program codes.
另一方面，本申请还提供一种计算机程序产品，所述计算机程序产品包括计算机程序，计算机程序可存储在非暂态计算机可读存储介质上，所述计算机程序被处理器执行时，计算机能够执行上述各方法所提供的数据处理方法，该方法包括：获取待处理数据；On the other hand, the present application also provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the data processing method provided by the above methods, the method includes: obtaining data to be processed;
获取用户对所述待处理数据配置的数据处理规则；Obtaining data processing rules configured by a user for the data to be processed;
基于所述数据处理规则对所述待处理数据进行处理。The data to be processed is processed based on the data processing rule.
又一方面，本申请还提供一种非暂态计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现以执行上述各方法提供 的数据处理方法，该方法包括：获取待处理数据；On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the data processing method provided by the above methods is implemented, and the method includes: obtaining data to be processed;
获取用户对所述待处理数据配置的数据处理规则；Obtaining data processing rules configured by a user for the data to be processed;
基于所述数据处理规则对所述待处理数据进行处理。The data to be processed is processed based on the data processing rule.
以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。本领域普通技术人员在不付出创造性的劳动的情况下，即可以理解并实施。The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到各实施方式可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件。基于这样的理解，上述技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品可以存储在计算机可读存储介质中，如ROM/RAM、磁碟、光盘等，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行各个实施例或者实施例的某些部分所述的方法。Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM/RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
最后应说明的是：以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围。Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims (12)

  1. 一种数据处理方法，包括：A data processing method, comprising:
    获取待处理数据；Get the data to be processed;
    获取用户对所述待处理数据配置的数据处理规则；Obtaining data processing rules configured by a user for the data to be processed;
    基于所述数据处理规则对所述待处理数据进行处理。The data to be processed is processed based on the data processing rule.
  2. 根据权利要求1所述的数据处理方法，其中，所述数据处理规则包括以下至少之一：数据解析规则、数据存储规则和数据发送规则。The data processing method according to claim 1, wherein the data processing rules include at least one of the following: data parsing rules, data storage rules and data sending rules.
  3. 根据权利要求2所述的数据处理方法，其中，所述数据解析规则用于指示所述待处理数据的格式标识和待解析数据的第一标识；The data processing method according to claim 2, wherein the data parsing rule is used to indicate the format identifier of the data to be processed and the first identifier of the data to be parsed;
    所述基于所述数据处理规则对所述待处理数据进行处理，包括：The processing of the data to be processed based on the data processing rule includes:
    在所述数据处理规则包括所述数据解析规则时，基于所述待处理数据的格式标识确定目标解析器；When the data processing rule includes the data parsing rule, determining a target parser based on a format identifier of the data to be processed;
    通过所述目标解析器从所述待处理数据中解析与所述第一标识匹配的数据，得到解析数据。The target parser parses data matching the first identifier from the data to be processed to obtain parsed data.
  4. 根据权利要求2所述的数据处理方法，其中，所述数据存储规则用于指示待存储数据的第二标识；The data processing method according to claim 2, wherein the data storage rule is used to indicate a second identifier of the data to be stored;
    所述基于所述数据处理规则对所述待处理数据进行处理，包括：The processing of the data to be processed based on the data processing rule includes:
    在所述数据处理规则包括所述数据存储规则、且所述待处理数据包括上下文数据时，从解析数据中确定与所述第二标识匹配的待存储数据；所述解析数据为从所述待处理数据中解析得到的数据；When the data processing rule includes the data storage rule and the data to be processed includes context data, determining the data to be stored that matches the second identifier from the parsed data; the parsed data is data parsed from the data to be processed;
    在所述数据处理规则包括所述数据存储规则，且所述待处理数据包括非上下文数据时，从所述待处理数据中确定与所述第二标识匹配的待存储数据；When the data processing rule includes the data storage rule, and the data to be processed includes non-context data, determining the data to be stored that matches the second identifier from the data to be processed;
    将所述待存储数据进行存储。The data to be stored is stored.
  5. 根据权利要求2所述的数据处理方法，其中，所述数据发送规则用于指示待发送数据的第三标识；The data processing method according to claim 2, wherein the data sending rule is used to indicate a third identifier of the data to be sent;
    所述基于所述数据处理规则对所述待处理数据进行处理，包括：The processing of the data to be processed based on the data processing rule includes:
    在所述数据处理规则包括所述数据发送规则、且所述待处理数据包括上下文数据时，从解析数据中确定与所述第三标识匹配的待发送数据；所述解析数据为从所述待处理数据中解析得到的数据；When the data processing rule includes the data sending rule and the data to be processed includes context data, determining the data to be sent that matches the third identifier from the parsed data; the parsed data is data parsed from the data to be processed;
    在所述数据处理规则包括所述数据发送规则，且所述待处理数据包括非上下文数据时，从所述待处理数据中确定与所述第三标识匹配的待发送数据；When the data processing rule includes the data sending rule, and the data to be processed includes non-context data, determining the data to be sent that matches the third identifier from the data to be processed;
    将所述待发送数据进行发送。The data to be sent is sent.
  6. 根据权利要求5所述的数据处理方法，其中，所述数据发送规则还用于指示发送通道标识；The data processing method according to claim 5, wherein the data sending rule is further used to indicate a sending channel identifier;
    所述将所述待发送数据进行发送，包括：The sending of the data to be sent includes:
    基于所述发送通道标识确定目标发送通道；Determine a target sending channel based on the sending channel identifier;
    将所述待发送数据通过所述目标发送通道进行发送。The data to be sent is sent through the target sending channel.
  7. 根据权利要求5所述的数据处理方法，其中，所述数据发送规则还用于指示待发送数据的格式标识；The data processing method according to claim 5, wherein the data sending rule is further used to indicate a format identifier of the data to be sent;
    所述将所述待发送数据进行发送，包括：The sending of the data to be sent includes:
    基于所述待发送数据的格式标识将所述待发送数据进行格式转换，得到第一目标发送数据；Converting the format of the data to be sent based on the format identifier of the data to be sent to obtain first target sending data;
    将所述第一目标发送数据进行发送。The first target sending data is sent.
  8. 根据权利要求5所述的数据处理方法，其中，所述数据发送规则还用于指示待发送数据的重组信息；The data processing method according to claim 5, wherein the data sending rule is further used to indicate reorganization information of the data to be sent;
    所述将所述待发送数据进行发送，包括：The sending of the data to be sent includes:
    基于所述待发送数据的重组信息将所述待发送数据的内容进行重组，得到第二目标发送数据；reorganizing the content of the data to be sent based on the reorganization information of the data to be sent to obtain second target sending data;
    将所述第二目标发送数据进行发送。The second target sending data is sent.
  9. 一种数据处理装置，包括：A data processing device, comprising:
    第一获取单元，用于获取待处理数据；A first acquisition unit, used to acquire data to be processed;
    第二获取单元，用于获取用户对所述待处理数据配置的数据处理规则；A second acquisition unit, used to acquire a data processing rule configured by a user for the data to be processed;
    处理单元，用于基于所述数据处理规则对所述待处理数据进行处理。A processing unit is used to process the data to be processed based on the data processing rules.
  10. 一种电子设备，包括存储器、处理器及存储在所述存储器上并可在所述处理器上运行的计算机程序，其特征在于，所述处理器执行所述程序时实现如权利要求1至8任一项所述数据处理方法。An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the data processing method as claimed in any one of claims 1 to 8 when executing the program.
  11. 一种非暂态计算机可读存储介质，其上存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现如权利要求1至8任一项所述数据处理方法。A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data processing method as claimed in any one of claims 1 to 8.
  12. 一种计算机程序产品，包括计算机程序，其特征在于，所述计算机程序被处理器执行时实现如权利要求1至8任一项所述数据处理方法。A computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, the data processing method according to any one of claims 1 to 8 is implemented.
PCT/CN2022/144128 2022-11-18 2022-12-30 Data processing method and apparatus WO2024103505A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211449670.2A CN115826984A (en) 2022-11-18 2022-11-18 Data processing method and device
CN202211449670.2 2022-11-18

Publications (1)

Publication Number Publication Date
WO2024103505A1 true WO2024103505A1 (en) 2024-05-23

Family

ID=85529303

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/144128 WO2024103505A1 (en) 2022-11-18 2022-12-30 Data processing method and apparatus

Country Status (2)

Country Link
CN (1) CN115826984A (en)
WO (1) WO2024103505A1 (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106777101A (en) * 2016-12-14 2017-05-31 深圳天源迪科信息技术股份有限公司 Data processing engine
CN113886422A (en) * 2021-10-19 2022-01-04 北京锐安科技有限公司 Data extraction method, device, equipment and storage medium
US20220164423A1 (en) * 2020-11-20 2022-05-26 Wallife s.r.l. Method and apparatus for user recognition
CN114764556A (en) * 2021-01-15 2022-07-19 中国再保险(集团)股份有限公司 Data processing method, data processing device, computer equipment and storage medium
CN114860305A (en) * 2022-05-05 2022-08-05 中国农业银行股份有限公司 Data processing method and device

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106777101A (en) * 2016-12-14 2017-05-31 深圳天源迪科信息技术股份有限公司 Data processing engine
US20220164423A1 (en) * 2020-11-20 2022-05-26 Wallife s.r.l. Method and apparatus for user recognition
CN114764556A (en) * 2021-01-15 2022-07-19 中国再保险(集团)股份有限公司 Data processing method, data processing device, computer equipment and storage medium
CN113886422A (en) * 2021-10-19 2022-01-04 北京锐安科技有限公司 Data extraction method, device, equipment and storage medium
CN114860305A (en) * 2022-05-05 2022-08-05 中国农业银行股份有限公司 Data processing method and device

Also Published As

Publication number Publication date
CN115826984A (en) 2023-03-21

Similar Documents

Publication Publication Date Title
Bjorklund The YANG 1.1 data modeling language
US11516129B2 (en) Packet edit processing method and related device
EP0772318B1 (en) Filtering system and method for a high performance network management map
US10726030B2 (en) Defining event subtypes using examples
US20050265342A1 (en) System and method for transforming configuration commands
US20060259909A1 (en) Mechanism for maintaining data format synchronization between different entities
EP2835938A1 (en) Message publishing and subscribing method and apparatus
WO2007144853A2 (en) Method and apparatus for performing customized paring on a xml document based on application
WO2020062959A1 (en) Data configuration method and apparatus
WO2013063950A1 (en) Inspection method and system of multimode communication device
WO2019233616A1 (en) Method and apparatus for merging of yang configuration and state data in model-driven applications
US20120266186A1 (en) Providing inter-platform application launch in context
JP5717504B2 (en) Method and device for generating an XML expressed management transaction including an XPath expression
CN111177182A (en) Data query method and device, storage medium and electronic equipment
CN110855459B (en) Network configuration method, device and system
CN114461247A (en) Hot update method, device, electronic equipment and computer readable medium
WO2024103505A1 (en) Data processing method and apparatus
CN112883088B (en) Data processing method, device, equipment and storage medium
CN111224756B (en) Method and device for determining data transmission abnormity, storage medium and electronic equipment
US8190723B2 (en) Method and system for automatically determining commands for a network element
CN114996554A (en) Database query method and device, storage medium and electronic equipment
US20190318036A1 (en) Topic Based Publish and Parametric Subscribe Pattern
US7333971B2 (en) Helpset build system and method
CN107306202B (en) Network management method, server, client and system
KR20080083543A (en) Network management system comprising image forming apparatus and method thereof

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22965681

Country of ref document: EP

Kind code of ref document: A1