CN111090689A - Data conversion method and system based on dynamic rules - Google Patents
Data conversion method and system based on dynamic rules Download PDFInfo
- Publication number
- CN111090689A CN111090689A CN201911239287.2A CN201911239287A CN111090689A CN 111090689 A CN111090689 A CN 111090689A CN 201911239287 A CN201911239287 A CN 201911239287A CN 111090689 A CN111090689 A CN 111090689A
- Authority
- CN
- China
- Prior art keywords
- data
- source data
- rule
- drools
- working memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data conversion method and a data conversion system based on dynamic rules, wherein the data conversion method introduces a Map data structure to process source data, the source data are transmitted to a Drools rule engine in batches in a set mode, and the Drools rule engine processes the source data to obtain target data. The invention can improve the data conversion efficiency and realize the dynamic loading of the rule file.
Description
Technical Field
The present invention relates to data conversion, and more particularly, to a data conversion method and system based on dynamic rules.
Background
In the data integration process, four steps of extracting, cleaning, converting and loading data are involved, and the data are loaded from a source database (or a file) to a destination database. In the process, data with different sources, formats and characteristic properties are logically or physically organically concentrated. Data needs to be converted to form a uniform format of a destination database.
There are many conversion processes in this data integration process, and these processes are time-consuming and have a great influence on the data processing efficiency. The existing method for converting multiple processes realizes data conversion in a micro-service calling mode; this has the problem of network communication time and is difficult to accommodate for large data volume requirements. Data conversion is processed through js codes, and the design cannot process data sets, only one data set can process the data, and the problem of time consumption exists. In addition, each time the conversion logic needs to be changed, a new file needs to be generated, and file replacement needs to stop the original task first or wait for the original task to be ended and then reload the file, so that the file cannot be dynamically loaded.
Disclosure of Invention
In view of the above, the present invention provides a data conversion method and a data conversion system based on dynamic rules, which are used to solve the problems in the prior art that files cannot be dynamically loaded and the data conversion efficiency is low.
In order to achieve the purpose, the invention provides a data conversion method based on dynamic rules, which introduces a Map data structure to process source data, transmits the source data to a Drools rule engine in batches in a set mode, and the Drools rule engine processes the source data to obtain target data.
Optionally, the Drools rule engine includes a working memory and a rule file, and the working memory receives the source data in batches; the rule file matches and processes the source data in the working memory to realize the conversion from the source data to the target data.
Optionally, the rule file adopts a conditional statement structure, and when the condition is satisfied, the execution statement adds, deletes or modifies the source data in the working memory.
Optionally, the rule file adopts a conditional statement structure, and when the condition is satisfied, the execution statement sets a value of a certain attribute of the source data through the accessor, where the value may be a value in the source data or a value of a certain expression.
The invention also provides a data conversion system based on the dynamic rule, which comprises: the system comprises a Map data structure and a Drools rule engine, wherein the Map data structure transmits source data to the Drools rule engine in batches in a set mode, and the Drools rule engine processes the source data to obtain target data.
Optionally, the Drools rule engine includes a rule file and a working memory, the working memory receives the source data in batch, and the rule file matches and processes the source data in the working memory.
Compared with the prior art, the technical scheme of the invention has the following advantages: the method introduces a Map data structure to process the source data, transmits the source data to a Drools rule engine in batches in a set mode, and the Drools rule engine processes the source data to obtain target data. The invention realizes the batch processing of the data conversion of large data quantity, improves the data conversion efficiency, realizes the dynamic loading of the rule file and is completed through a user interaction interface; the data conversion file does not need to be generated firstly and then uploaded manually.
Drawings
FIG. 1 is a schematic diagram of a data conversion system of the present invention;
Detailed Description
Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but the present invention is not limited to only these embodiments. The invention is intended to cover alternatives, modifications, equivalents and alternatives which may be included within the spirit and scope of the invention.
In the following description of the preferred embodiments of the present invention, specific details are set forth in order to provide a thorough understanding of the present invention, and it will be apparent to those skilled in the art that the present invention may be practiced without these specific details.
The invention is described in more detail in the following paragraphs by way of example with reference to the accompanying drawings. It should be noted that the drawings are in simplified form and are not to precise scale, which is only used for convenience and clarity to assist in describing the embodiments of the present invention.
Referring to fig. 1, a schematic diagram of a data conversion system of the present invention is illustrated, a Map data structure is introduced instead of a java Bean data structure to deal with the problem that fields in source data are uncertain, and the source data is delivered to a Drools rule engine in batches in a set manner, wherein the delivery is referred delivery. The Drools rule engine receives source data in batches and processes the source data to obtain target data. The Drools rules engine includes a rule file and a working memory that receives source data in batches, the rule file matching and processing the source data in the working memory. The rule file adopts a conditional statement structure, and when the condition is met, the source data in the working memory is newly added, deleted or modified according to the execution statement. The rule file adopts a conditional statement structure, and when a condition is met, the execution statement sets a value of a certain attribute of the source data through the accessor, wherein the value can be a value in the source data or a value of a certain expression. An example of a rule file is as follows:
the rule is an examination condition calculation rule, and when the person is on business, the person is asked to be sick and fake, and no card punching information is available, so that the time of the sick and fake is calculated. The rule engine adopts a condition structure of the while then, and when the condition is satisfied, the statement in the then is executed, and the value of the attribute in the fact object (source data) is added or changed. Thereby completing the processing of the data.
The rule engine is used for processing, the processing efficiency is improved, when the conversion result is returned, the conversion result is written into the corresponding field according to the specified logic, and the value of a certain attribute can be set through the setter accessor, such as: set ("sickLeaveTime", "xxx"). The value may be a fact object value such as: getasint ("timelevetotal — 2"), which may also be a value for a computational expression, is supported by the rules engine. Thereby effecting a transformation of the data from the source field to the destination field.
Although the embodiments have been described and illustrated separately, it will be apparent to those skilled in the art that some common techniques may be substituted and integrated between the embodiments, and reference may be made to one of the embodiments not explicitly described, or to another embodiment described.
The above-described embodiments do not limit the scope of the present invention. Any modification, equivalent replacement, and improvement made within the spirit and principle of the above-described embodiments should be included in the protection scope of the technical solution.
Claims (6)
1. A data conversion method based on dynamic rules is characterized in that: introducing a Map data structure to process the source data, transmitting the source data to a Drools rule engine in batches in a set mode, and processing the source data by the Drools rule engine to obtain target data.
2. The dynamic rule-based data transformation method of claim 1, wherein: the Drools rule engine comprises a working memory and a rule file, wherein the working memory receives source data in batches; the rule file matches and processes the source data in the working memory to realize the conversion from the source data to the target data.
3. The dynamic rule-based data transformation method of claim 2, wherein: the rule file adopts a conditional statement structure, and when the condition is met, the execution statement adds, deletes or modifies the source data in the working memory.
4. The dynamic rule-based data transformation method of claim 2, wherein: the rule file adopts a conditional statement structure, and when a condition is met, the execution statement sets a value of a certain attribute of the source data through the accessor, wherein the value can be a value in the source data or a value of a certain expression.
5. A dynamic rule based data transformation system, comprising: the system comprises a Map data structure and a Drools rule engine, wherein the Map data structure transmits source data to the Drools rule engine in batches in a set mode, and the Drools rule engine processes the source data to obtain target data.
6. The dynamic rule based data transformation system of claim 5, wherein: the Drools rules engine includes a rule file and a working memory that receives source data in batches, the rule file matching and processing the source data in the working memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911239287.2A CN111090689A (en) | 2019-12-06 | 2019-12-06 | Data conversion method and system based on dynamic rules |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911239287.2A CN111090689A (en) | 2019-12-06 | 2019-12-06 | Data conversion method and system based on dynamic rules |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111090689A true CN111090689A (en) | 2020-05-01 |
Family
ID=70394752
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911239287.2A Pending CN111090689A (en) | 2019-12-06 | 2019-12-06 | Data conversion method and system based on dynamic rules |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111090689A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101763261A (en) * | 2009-12-28 | 2010-06-30 | 山东中创软件商用中间件股份有限公司 | Method, system and device for extracting, converting and loading data |
| CN108399176A (en) * | 2017-02-07 | 2018-08-14 | 阿里巴巴集团控股有限公司 | A kind of rule-based data processing method and regulation engine device |
| CN110297840A (en) * | 2019-05-22 | 2019-10-01 | 平安银行股份有限公司 | Data processing method, device, equipment and the storage medium of rule-based engine |
-
2019
- 2019-12-06 CN CN201911239287.2A patent/CN111090689A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101763261A (en) * | 2009-12-28 | 2010-06-30 | 山东中创软件商用中间件股份有限公司 | Method, system and device for extracting, converting and loading data |
| CN108399176A (en) * | 2017-02-07 | 2018-08-14 | 阿里巴巴集团控股有限公司 | A kind of rule-based data processing method and regulation engine device |
| CN110297840A (en) * | 2019-05-22 | 2019-10-01 | 平安银行股份有限公司 | Data processing method, device, equipment and the storage medium of rule-based engine |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110032459B (en) | Message queue configuration method and device, computer equipment and storage medium | |
| CN101246486B (en) | Method and apparatus for improved process of expressions | |
| WO2018210096A1 (en) | Rule engine-based rule configuration method, terminal and device, and storage medium | |
| US10516791B2 (en) | Information processing system, information processing apparatus, and information processing method for executing an iteration of one or more processes | |
| US20070203926A1 (en) | Scalable transformation and configuration of EDI interchanges | |
| CN110647316B (en) | Method and device for generating universal business object, computer equipment and storage medium | |
| CN110888842A (en) | File storage method, file query method, file storage device, file query device and file query equipment | |
| CN111143446A (en) | Data structure conversion processing method and device of data object and electronic equipment | |
| CN110046868A (en) | A kind of method and apparatus handling msu message | |
| CN109471893B (en) | Network data query method, equipment and computer readable storage medium | |
| CN113704307A (en) | Data query method, device, server and computer readable storage medium | |
| CN117238433B (en) | Method for automatically isolating document data based on Libreoffice | |
| CN113641700A (en) | Data processing method and device based on Spring boot frame | |
| US20070136349A1 (en) | Uniform framework for standardization and transmission of documents electronically | |
| CN113407565B (en) | Cross-database data query method, device and equipment | |
| CN102855305A (en) | Data upgrading method and device | |
| CN114328682A (en) | Data processing method, device, equipment and medium based on rule engine | |
| CN110647564A (en) | Hive table establishing method, electronic device and computer readable storage medium | |
| CN110009476A (en) | A kind of method and apparatus handling application of making loans | |
| CN111090689A (en) | Data conversion method and system based on dynamic rules | |
| CN111460021A (en) | Data export method and device | |
| CN111324645A (en) | Data processing method and device for block chain | |
| CN110764768A (en) | Method and device for mutual conversion between model object and JSON object | |
| WO2018191020A1 (en) | Message processing apparatus and methods | |
| CN114036180A (en) | Report generation method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200501 |