CN111752927A

CN111752927A - Clone-based data form generation method, device, terminal equipment and medium

Info

Publication number: CN111752927A
Application number: CN202010610508.9A
Authority: CN
Inventors: 尹小芳; 徐新丽; 李理; 江旻; 杨杨; 张晶
Original assignee: WeBank Co Ltd
Current assignee: WeBank Co Ltd
Priority date: 2020-06-30
Filing date: 2020-06-30
Publication date: 2020-10-09
Anticipated expiration: 2040-06-30
Also published as: CN111752927B

Abstract

The invention relates to the technical field of financial science and technology, and discloses a data form generation method and device based on cloning, terminal equipment and a computer storage medium. The clone-based data form generation method is based on obtaining a search key value, and calling a data clone model to capture a data form from a preset source environment according to the search key value; backing up the data form into a database, and cloning the data form into a preset target environment from the database; acquiring an insertion statement with an excessive cloned data form in the preset target environment; and carrying out batch replacement on the field value of the inserted statement, and taking the data form after replacing the field value as a target data form. The invention simplifies the process of generating the data form, improves the generation efficiency, can realize that the same data form is cloned to different target environments for different self-test joint debugging, service testing and the like, and greatly improves the reuse space of the data form.

Description

Clone-based data form generation method, device, terminal equipment and medium

Technical Field

The present invention relates to the field of financial technology (Fintech), and in particular, to a data form generation method and apparatus based on cloning, a terminal device, and a computer storage medium.

Background

With the development of computer technology, more and more technologies are applied in the financial field, and the traditional financial industry is gradually changing to financial technology (Fintech), but higher requirements are also put forward on the technologies due to the requirements of the financial industry on safety and real-time performance.

Because the whole operation of the existing loan service needs to be realized by the cooperation of an online system and an evening batch operation system, the system takes one day for loan products every time the system runs (a daily cutting and scheduling of the daily batch task are performed, wherein the daily cutting indicates that the bank system performs system date switching, namely, the daily service is processed in a centralized way, and the next day is automatically performed after the system is finished daily cutting), for example, the loan product is loaned by a user for 24 months, which indicates that the system needs to run for 24 months to calculate the loan interest of the loan product, and the system needs to consume at least 1 hour after running for one batch, so that even if batch skipping processing can be performed on the loan batch, the system needs to consume several days for completing the whole interest calculation operation. Therefore, when a financial institution in the industry performs a rest test on a loan product before distribution, it often takes a long time to run the system in advance to prepare a data form for the test (data with a complex life cycle, each stage of the data life cycle corresponds to one data form), and after the rest test is completed, a large amount of data forms prepared by construction are simultaneously invalidated and cannot be used again, so that the system needs to run again to prepare the data form when the loan product is next tested.

Disclosure of Invention

The invention mainly aims to provide a data form generation method and device based on cloning, a terminal device and a computer storage medium, and aims to solve the technical problems that the preparation of a data form through system batch needs long time, the process is complicated, the prepared form cannot be reused, and the data form generation efficiency is low.

To achieve the above object, the present invention provides a clone-based data pattern generation method, including:

acquiring a search key value, and calling a data clone model to capture a data form from a preset source environment according to the search key value;

backing up the data form into a database, and cloning the data form into a preset target environment from the database;

acquiring an insertion statement with an excessive cloned data form in the preset target environment;

and carrying out batch replacement on the field value of the inserted statement, and taking the data form after replacing the field value as a target data form.

Optionally, the step of replacing the field value of the insertion statement in batch includes:

acquiring a difference field in the insertion statement and detecting a mapping field of the difference field;

extracting respective field values of the difference field and the mapping field, and generating corresponding replacement values for the field values;

and calculating all matching results between the difference field and the mapping field and the field values, and performing batch replacement on the field values in all the matching results by using the replacement values.

Optionally, after the step of taking the data form after replacing the field value as the target data form, the method further includes:

inquiring date fields of the cloned data forms, and translating and calculating according to a preset time sequence to obtain the date time sequence of the date fields;

and calculating the real date of the date time sequence corresponding to the preset target environment, and taking the real date as the business date of the target data form.

Optionally, the clone-based data morphology generation method further includes:

obtaining data morphological characteristics based on model training to construct a data clone model, wherein the data morphological characteristics comprise: the table list, the difference field, the value logic of the difference field and the service date field;

the step of obtaining data morphological characteristics based on model training to construct a data clone model comprises the following steps:

the method comprises the steps of collecting preset data forms of a first account and a second account as training samples, wherein the data environments of the first account and the second account are the same;

inputting the training sample into a clone model to be trained for training to obtain a list, a difference field, value logic of the difference field and a service date field;

and assembling the table list, the difference field, the value logic of the difference field and the service date field to construct a data clone model.

Optionally, the step of acquiring a search key value and calling a data clone model to capture a data form from a preset source environment according to the search key value includes:

reading a search key value from a preset backup configuration;

inputting the search key value into the data clone model so that the data clone model can capture data forms in the source environment in sequence according to the query key value marked on each data table in the table list;

the step of backing up the data modality to a database comprises:

adding scene identification to the data form according to the retention date of the data form, wherein the retention date is the service date captured by the data clone model when capturing the data form;

and when detecting that the database does not contain the scene identification, backing up the data form into the database.

Optionally, after the step of backing up the data modality to the database, the method further includes:

reading characteristic values of each data form corresponding to preset characteristic parameters one by one in the database to form a characteristic value list of each data form;

traversing the characteristic values contained in each characteristic value list to detect whether a target characteristic value list with completely identical characteristic values exists;

and if so, performing deduplication processing on the data form with the target characteristic value list.

Optionally, the clone-based data morphology generation method further includes:

and reading the data form of the source card number from a preset clone configuration, and cloning the data form of the source card number to the preset target environment.

In order to achieve the above object, the present invention also provides a clone-based data pattern generation apparatus including:

the data capturing module is used for acquiring a search key value and calling a data cloning model to capture a data form from a preset source environment according to the search key value;

the data cloning module is used for backing up the data form into a database and cloning the data form into a preset target environment from the database;

the statement acquisition module is used for acquiring an inserted statement with an excessive cloned data form in the preset target environment;

and the environment adaptation module is used for carrying out batch replacement on the field value of the inserted statement so as to take the data form after the field value is replaced as the target data form.

In addition, to achieve the above object, the present invention also provides a terminal device, including: a memory, a processor, and a clone-based data modality generation program stored on the memory and executable on the processor, the clone-based data modality generation program when executed by the processor implementing the steps of the clone-based data modality generation method as described above.

In addition, to achieve the above object, the present invention also provides a computer storage medium having a clone-based data modality generation program stored thereon, which when executed by a processor implements the steps of the clone-based data modality generation method as described above.

The invention provides a data form generation method and device based on cloning, a terminal device and a computer storage medium, wherein a data cloning model is called to capture a data form from a preset source environment according to a search key value based on obtaining the search key value; backing up the data form into a database, and cloning the data form into a preset target environment from the database; acquiring an insertion statement with an excessive cloned data form in the preset target environment; and carrying out batch replacement on the field value of the inserted statement, and taking the data form after replacing the field value as a target data form.

The method comprises the steps of capturing a data form required by subsequent test application by using a data clone model obtained by model training construction, backing up and storing the captured data form into a database for subsequent repeated clone use, directly cloning the data form from the database to a target environment and then obtaining a plurality of insertion sentences when specific information test is required by using the data form, and carrying out batch replacement on field values of the insertion sentences to carry out environment adaptation on the data form so as to generate and obtain the target data form for the information test. The problem that the traditional bank system based on the prior art needs to spend a long time for running and approving the prepared data form, and the prepared data form is difficult to recycle, so that the data needs to be prepared again when a new interest-keeping test is carried out is solved, the process of generating the data form is simplified, the generation efficiency is improved, the same data form can be cloned to different target environments for self-test joint debugging, service testing and the like of development of different testers or roles, and the recycling space of the data form is greatly improved.

Drawings

FIG. 1 is a schematic diagram of an apparatus architecture of a hardware operating environment according to an embodiment of the present invention;

FIG. 2 is a schematic flow chart illustrating a first embodiment of a clone-based data pattern generation method according to the present invention;

FIG. 3 is a schematic diagram of an application scenario for constructing a data clone model according to an embodiment of the clone-based data morphology generation method of the present invention;

FIG. 4 is a schematic diagram of an application scenario of a string replacement algorithm according to an embodiment of the clone-based data shape generation method of the present invention;

FIG. 5 is a schematic diagram of an application scenario of service date time sequence translation according to an embodiment of the clone-based data form generation method of the present invention;

FIG. 6 is a schematic diagram of an application scenario of feature extraction clustering deduplication according to an embodiment of a clone-based data shape generation method of the present invention;

FIG. 7 is a functional block diagram of an embodiment of a clone-based data pattern generation apparatus according to the present invention.

The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.

Detailed Description

It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

Referring to fig. 1, fig. 1 is a schematic device structure diagram of a hardware operating environment according to an embodiment of the present invention.

The terminal device in the embodiment of the present invention may be a smart phone, or may be a terminal device such as a PC (Personal Computer), a tablet Computer, or a portable Computer.

As shown in fig. 1, the terminal device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). The memory 1005 may alternatively be a storage device separate from the processor 1001.

Those skilled in the art will appreciate that the terminal device configuration shown in fig. 1 is not intended to be limiting of the terminal device and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.

As shown in fig. 1, a memory 1005, which is a kind of computer storage medium, may include therein an operating system, a network communication module, a user interface module, and a clone-based data modality generation program.

In the terminal shown in fig. 1, the network interface 1004 is mainly used for connecting to a backend server and performing data communication with the backend server; the user interface 1003 is mainly used for connecting a client and performing data communication with the client; and the processor 1001 may be configured to call the clone-based data pattern generation program stored in the memory 1005 and perform the following operations:

Further, the processor 1001 may call the clone-based data modality generation program stored in the memory 1005, and also perform the following operations:

Further, the processor 1001 may call the clone-based data form generation program stored in the memory 1005, and after taking the data form after replacing the field value as the target data form, further perform the following operations:

obtaining data morphological characteristics based on model training to construct a data clone model, wherein the data morphological characteristics comprise: table list, difference field, value logic of difference field and service date field.

Further, the preset source environment is a source environment in a preset backup configuration, and the processor 1001 may call the clone-based data pattern generation program stored in the memory 1005, and further perform the following operations:

reading a search key value from a preset backup configuration;

and inputting the search key value into the data clone model so that the data clone model can capture data forms in the source environment in sequence according to the query key value marked on each data table in the table list.

Further, the processor 1001 may call the clone-based data modality generation program stored in the memory 1005, and after performing the backup of the data modality into the database, perform the following operations:

Based on the above hardware structure, embodiments of the clone-based data pattern generation method of the present invention are presented.

It should be noted that, in the prior art, in the execution process of interest-bearing tests for loan services, each financial institution needs to manually configure a designated table to make data snapshots when storing data forms generated by the tests, and during a regression test, if data (data forms) of a complex life cycle scene is used, manual offline maintenance numbers (a batch plan is made, different forms are constructed according to cases to meet the behavior of data packets to be tested later) are also needed, stay days (the time of slicing into a data warehouse and the service date of data form qualification) of all data forms are pressed to the same day and then data slices are made for storage.

However, the snapshot-saved data is obtained by storing the source data of the designated table without any change, the captured and stored table is not comprehensive enough and also contains information (such as an identity card, a logic card number, and the like) specific to the source card number, and due to reasons such as a key conflict and an incomplete table, a certain data form cannot be extracted immediately and inserted into a target environment for performing the interest-keeping test, so that the saved data form is difficult to be reused.

Based on this, when each financial institution performs the interest-bearing test of loan transaction, the biggest pain point is the time taken for preparing data. Because the whole operation of the existing loan service needs the mutual cooperation of an online system and a system for batch operation at night, the loan product is only interest one day when the system runs and batches once, for example, the loan product is borrowed by a user for 24 days, which means that the system needs to run for 24 months to calculate the loan interest of the loan product, and the system needs to spend at least 1 hour when the system runs one batch, so even if the batch skipping processing can be carried out on the loan batch, the system needs to complete the whole interest calculation operation and also consumes several days. Therefore, when a financial institution in the industry performs a rest test on a loan product before distribution, it often takes a long time to run the system in advance to prepare a data form for the test, and after the rest test is completed, a large amount of data forms prepared by construction are simultaneously invalidated and cannot be used again, so that the system needs to run again to prepare the data form when the loan product is tested next time.

In view of the above, the present invention provides a data pattern generation method based on cloning. Referring to fig. 2, fig. 2 is a flowchart illustrating a first embodiment of a data pattern generation method based on cloning according to the present invention.

In this embodiment, the clone-based data modality generation method includes:

step S10, acquiring a search key value, calling a data clone model to capture a data form from a preset source environment according to the search key value;

after receiving backup configuration for backing up data forms for realizing repeated use of subsequent cloning, the terminal equipment immediately acquires search key values of the original card numbers manually input in the backup configuration to form a search key value list, then calls a data cloning model constructed based on model training in advance, sequentially captures all data from a source environment manually input in the backup configuration according to the search key value list, and then performs deduplication processing on all data to obtain the data forms.

It should be noted that, in this embodiment, the data form obtained by capturing the data clone model in the source environment may specifically be a data slice of the data form at a certain time (a static set of executable data including various data forms and indexed by a batch date), the data clone model constructed based on model training includes a data form feature — a table list, and each data table recorded in the table list is labeled with a query key value, and the query key value at least exists in the obtained search key value list of the source card number, so that the data clone model may directly obtain a final data form by querying and capturing in the source environment based on the table list of the data clone model itself and the query manner corresponding to the query key value labeled in the data table recorded in the table list.

Further, in a possible embodiment, the step S10 may include:

step S201, reading a search key value from a preset backup configuration;

it should be noted that, in this embodiment, the preset backup configuration may specifically be a backup configuration of operations such as a source card number, a source environment, a database selection, and a trigger backup instruction that are autonomously input by a worker when the terminal device performs data form backup by using a data clone model constructed based on model training.

After receiving the backup configuration which is manually input and contains the operations of the source card number, the source environment, the database selection, the triggering backup instruction and the like, the terminal equipment further reads the SQL sentence of the search key value manually input in the backup configuration to obtain the search key value of the source card number from the received backup configuration manually input to form a search key value list.

And step S102, inputting the search key value into the data clone model so that the data clone model can capture data forms in the source environment in sequence according to the query key value marked on each data table in the table list.

After reading the search key value of the obtained source card number to form a search key value list, the terminal device inputs the search key value list into a data clone model which is constructed in advance based on model training, so that the data clone model records the query key value marked on the data table based on the table list contained in the data clone model, and traverses the input search key value list to sequentially capture the data form from the manually input source environment in backup configuration.

Specifically, for example, after receiving a source card number, a source environment, a database selection and a trigger backup instruction manually input by a worker, the terminal device outputs an interactive interface configured with a search key value and an sql statement for acquiring the search key value to the worker through a front-end visual interface, and receives the search key value further configured by the worker based on the interactive interface: the client number cust _ id, the account number acct _ no, or the identity card id _ no, so that the terminal device obtains a search key value list (such as cust _ id, acct _ no, or id _ no) of the manually input source card number in the backup configuration and an input parameter card _ no based on the client number cust _ id, the account number acct _ no, or the identity card id _ no.

The method comprises the steps that a terminal device loads a data clone model which is constructed in advance based on model training, based on a query key value marked on each data table recorded in a data clone model table list, a source environment which is manually input in backup configuration executes a query sql statement to capture all data forms, and in addition, the terminal device captures a service date at the moment as a stay date of the data forms while capturing the data forms.

Further, in a possible embodiment, the data morphology generation method based on cloning of the present invention may further include:

and step S50, acquiring data morphological characteristics based on model training to construct a data clone model.

The terminal device performs model training on the data form sample by using the clone model to be trained based on clone training configuration in advance, so as to obtain data form characteristics such as a list of the data form sample data table, a field value mode (logic), a date field of a service date and the like, and then constructs the data form characteristics into a data clone model required by a subsequent clone data form.

In this embodiment, the sample data of the data form is generated by the financial institution manually inputting the test configuration operation by the test staff in the interest-bearing test process for the loan product, and the data form is stored in the database, so that the terminal device can directly use the data form as the sample data for model training based on the clone training configuration. Since the data form of the loan product is stored in the database, each table of the data form at least contains the key field of the account (such as the ID card, the customer number or the account number, etc.) as the storage background. The database can be saved into a file through a database backup technology, and then Linux (GNU/Linux, which is a set of UNIX-like operating systems free to use and propagate freely) is used for self-bring grep (Global search hierarchical retrieval and Print, which is a powerful text search tool, and can search texts by using specific pattern matching (including Regular expressions) and output matching lines by default), and the data tables generated by the data form can be fetched out in a full amount by specifying limited search key values (such as identity cards, client numbers and account numbers).

Further, in one possible embodiment, the data morphology features include, but are not limited to: the table list, the difference field, the value logic of the difference field, and the service date field, where the step S40 may include:

step S501, collecting respective data forms of a preset first account and a preset second account as training samples, wherein the respective data environments of the first account and the second account are the same;

it should be noted that, in this embodiment, since in the process of generating the clone-based data form, two data fields with the same form are presented in the database table in the same manner except for the personalized difference field. Therefore, by using the characteristic, two accounts with the same form are trained through a model, and an enumeration value comparison method is carried out on data form samples, so that a difference field set of the table can be obtained, and the truest difference field set of each table can be obtained by using a plurality of groups of samples to train and summing up the difference fields. It is to be understood that the data form feature of the present invention is based on a clone data form generation method, and the type, number, and the like of the data form feature are not specifically limited.

After receiving the clone training configuration, the terminal device identifies two accounts in the clone training configuration, wherein the two accounts are manually input, and the scene forms are the same: the data form of each of the first account and the second account is backed up to be a corresponding database file to be used as a training sample.

Step S502, inputting the training sample into a clone model to be trained for training to obtain a list, a difference field, the value logic of the difference field and a service date field;

step S503, assembling the table list, the difference field, the value logic of the difference field and the service date field to construct a data clone model.

The terminal equipment inputs a training sample into a pre-designated clone model to be trained, the clone model to be trained executes the cloning of sample data to a manually input empty account in the received clone training configuration, so that data morphology characteristics of the training sample, such as a list, a difference field, value logic of the difference field, a service date field and the like, are sequentially acquired based on model training in the process, and finally the clone model to be trained, which is converged in training, is used as a data clone model required by the subsequent clone data morphology.

It should be noted that, in this embodiment, the clone model to be trained may be specifically specified when clone training configuration is manually input, and based on different design requirements of practical applications, the clone model to be trained may be any network model based on machine learning, such as a neural network model or a convolutional neural network model.

Specifically, for example, in the application scenario shown in fig. 3, the terminal device itself runs in linux, and the terminal device backs up the database by naming the data storing the data form of the loan product as a file db0.txt, then receives the search key value configured by the operator and obtains the sql statement of the search key (such as the client number cust _ id, the account number acc _ no, or the id _ no), and adds two accounts (card 1 and card 2) having the same scene form and the empty account (card a) into the model training request participating in triggering the cloning of the data form, for which the terminal device first executes the sql statement received and configured by the operator, so as to obtain the search key values of the card 1, the card 2, and the card a, and respectively forms a respective search key value list-keyList for the card 1, the card 2, and the card a, then, the terminal device may obtain the table list of all data tables of the card 1 or the card 2 and the query key value (the search key value hit by the grep lookup file db0.txt) by using the grep lookup file db0.txt (e.g., grep keyList (a relationship of "or" is used for multiple search key values)) carried by its system, linux.

It should be noted that, in this embodiment, when the terminal device searches for the file db0.txt by using its own system, i.e., the native grep of linux, two layers of searches need to be performed, where the first layer of search is: the field names without the search key values are searched only by using the search key values, because the first-layer search can obtain search results with different field meanings but the same value, the second-layer search is introduced, namely, the search is carried out on the basis of the first-layer results by carrying the field names and the values of the search key values (wherein the field names of the search key values comprise mapping fields of the field names), after the field names of the search key values are obtained after model training is finished, the difference between the field names of the search key values and the mapping fields of the field names of the search key values is checked, the mapping fields of the search key values are further perfected, and the search results of the second-layer search are accurate data as long as the mapping fields of the search key values are complete.

In addition, the terminal equipment performs table-by-table and field-by-field content comparison for the card 1 and the card 2, if the contents of the corresponding fields of the current data table of the card 1 and the card 2 are the same, the corresponding fields are classified as the same fields, otherwise, the corresponding fields are classified as difference (diff) fields.

It should be noted that, in this embodiment, based on that the card 1 and the card 2 are data of the same form, when the number of a certain field recorded in some tables of the card 1 and the card 2 is more than one, the terminal device will press all enumerated values appearing in this field into a dictionary, as long as the dictionary contents of the enumerated values corresponding to this field are the same (no order is distinguished), it is determined that this field is the same field, otherwise, this field is classified as a difference field.

Further, the value taking logic of the difference field comprises automatic value taking logic and value taking logic based on manual configuration.

And the terminal equipment preferentially acquires the automatic value-taking logic when acquiring the value-taking logic of the difference field of the data form sample based on model training. That is, the entire data modality of the card 1 is cloned to the card A, and each assignable difference field is assigned a value, that is, when the card 1, the card 2, and the card a all record the field contents and the number of the fields is the same, only the same fields of the card 1, the card 2, and the card a are updated, and the difference fields are not processed, or, when the card 1, the card 2 and the card a all record the field contents but have different numbers, only the same field is updated for the same record, and directly copies the record of the card 1 for the newly complemented record and updates the difference field, or, when the card 1 and the card 2 record the contents of the fields but the card a does not, the record of the card 1 is also copied directly, and the difference field is updated, when the difference field is updated, the self-increment field is firstly identified, the value of the target environment card A is used for self-increment, and then the date field of the terminal equipment system is identified to be automatically ignored: e.g., 2020-05-2612: 13:14, then identifies the account search key value field and its mapping field, automatically replaces it with the card a result, and finally uses manually configured field value logic (initially the manual is not configured and the list is empty).

And then, the terminal equipment acquires the value taking logic based on manual configuration aiming at the residual difference fields which are successfully subjected to value taking according to the automatic value taking logic. That is, the terminal device configures mapping fields (fields are the same as field values of existing value logic, but field names are different, so field mapping can be performed, for example, defilt _ logic _ CARD _ NO mapping fields include CARD _ NO, BOOK _ available _ NO, and logcal _ CARD _ NO), configures field value logic (a value method for configuring some fields, such as a random number value or a configure sql statement), configures exception sql statements (i.e., configures sql to be executed after the entire cloning is completed to take values to deal with special fields), and configures ignored fields (i.e., data of CARD 1 is used by DEFAULT without performing special processing when cloning configured fields).

Further, after the terminal device obtains a list of all data tables of the card 1 or the card 2 by using a grep lookup file db0.txt of a self system-linux, the terminal device performs date formatting on each field value in the list based on a date formatting function conversion principle, so as to obtain a service date field.

Further, the terminal device can obtain a data clone model containing a table list, a difference field, value logic of the difference field and a service date field after model training is completed, and then the terminal device performs union set with a previous data clone model aiming at the data clone model obtained based on model training to store the data clone model.

In this embodiment, in the process of performing model training, the data samples in different forms can be selected and used to perform model training for multiple times, so that the finally obtained data clone model is more accurate. In addition, the time consumed for model training once is about 5 seconds, so that quick follow-up processing can be realized when the structure of a new iteration data table is changed, and the overall efficiency of loan test can be improved in an auxiliary manner.

Step S20, backing up the data form into a database, and cloning the data form into a preset target environment from the database;

the terminal equipment captures the obtained data forms from the source environment of the received backup configuration, and adds scene identifiers to the data forms one by one, so that the data form backup is stored in a database based on the scene identifiers for subsequent cloning to a target environment for application. After adding scene identification to the captured data form and storing the data form backup into a database, the terminal equipment reads the data form from the database according to the received clone configuration prepared by the loan test data and clones the data form to the empty account of the designated target environment in the clone configuration.

Further, in a possible embodiment, the step of "backing up the data modality in the database" in the step S20 may include:

step S201, adding scene identification to the data form according to the retention date of the data form, wherein the retention date is the service date captured by the data clone model when capturing the data form;

it should be noted that, in this embodiment, because the terminal device executes the query sql statement in the source environment manually input in the backup configuration to capture the data form and also captures the current service date as the stay date of the data form, the terminal device may add a unique scene identifier to the captured data form based on the stay date.

Specifically, for example, the terminal device uniquely determines a scene identifier from the manually input source card number in the backup configuration, the retention date (specifically, the service date) of the captured data modality, and the scene name, and then adds the scene identifier to the currently captured data modality.

Step S202, when detecting that the database does not contain the scene mark, backing up the data form into the database.

The method comprises the steps that after a unique scene mark is added to a captured data form, the terminal device detects whether the scene mark exists in a database to be stored in the data form, the terminal device stores the data form into the database only when the scene mark does not exist in the database, on the contrary, if the scene mark exists in the database, the terminal device outputs a prompt of 'data form backup repetition' to a worker through a front-end visual interface, and does not store the data form into the database any more, or replaces the data form with the same scene mark in the database.

In this embodiment, the query sql statement is executed by the source environment manually input by the terminal device in the backup configuration to capture the data form, and meanwhile, the current service date is captured as the stay date of the data form, so that the terminal device can add a unique scene identifier to the captured data form based on the stay date; and when detecting that the database does not contain the scene identification, backing up the captured data form into the database. Therefore, the data form can be directly cloned from the database to be used in a target environment needing loan testing, the data form can be reused to the maximum space, namely the same data form can be developed by a plurality of testers or roles for self-testing joint debugging, business testing and the like, and the loan testing efficiency is greatly improved.

Further, in another possible embodiment, the method for generating data morphology based on cloning of the present invention may further include:

step S60, reading the data format of the measured environment source card number from the preset cloning configuration, and cloning the data format of the measured environment source card number to the target environment.

It should be noted that, in this embodiment, the preset clone configuration is the clone configuration for preparing the loan test data received by the terminal device, and the main-test environment source card number may specifically be a source card number synchronously input by the worker when inputting the clone configuration. The terminal device can clone the data form of the pre-backup from the data to the designated target environment empty account in the clone configuration for preparing the loan test data, and can also directly clone the data form of the main test environment source card number input by the staff from the received clone configuration to the empty account of the target environment.

Step S30, obtaining an insertion statement with an excessive cloned data form in the preset target environment;

specifically, for example, after receiving a clone configuration including a source card number and a source environment, which are manually input by a worker, and a target card number and a target environment, the terminal device performs a data backup operation to obtain all data tables of data forms of the source card number, then generates a clone sql statement, and processes the data tables one by one, and the generation logic is as follows:

firstly, the method comprises the following steps: if the source card number and the target card number have records and are the same in number, reading the difference field of the current data table in the data clone model and removing the difference field, and if the difference field has the residual field, updating the data of the source card number to the target card number to finish the cloning of the data form; alternatively, the first and second electrodes may be,

secondly, the method comprises the following steps: if the source card number and the target card number are recorded but the numbers are different, the inserting sentences which are added on the source card number are directly copied to finish the cloning of the data form; in the alternative to this, either,

thirdly, the method comprises the following steps: if the source card has record and the target card number has no record, the insertion statement of the source card is directly copied to complete the cloning of the data form.

And step S40, carrying out batch replacement on the field dereferencing of the insertion statement, and taking the data form after replacing the field dereferencing as a target data form.

After cloning the data form to the empty account of the target environment, the terminal device acquires the input and output statements of the data form in the empty account of the target environment according to the preset generation logic, and then replaces all field values of the inserted statements by adopting a character string batch replacement method, so that the data form after replacing the field values is used as the target data form capable of performing test application on the empty account of the target environment.

It should be noted that, in this embodiment, after cloning a data form to an empty account of a target environment, a terminal device performs environment adaptation on the cloned data form to generate a target data form that conforms to the target environment for test application, based on different environment types of the target environment or other adaptation factors, so as to perform test application on the empty account of the target environment by using the target data form.

In another possible embodiment, after the terminal device processes to clone the data form to the target card number, based on the difference between the source environment and the target environment, field value replacement is performed on the cloned data form, and further environment adaptation operation may be performed on the data form after the field value replacement synchronously or asynchronously according to adaptation processing of data such as date field time sequence translation, so as to generate a target data form that can be applied to the target card number in the target environment for test application, and finally, the terminal device performs a test application process on the target card number by using the target data form.

The embodiment of the invention provides a clone-based data form generation method, which is characterized in that a terminal device is configured on the basis of clone training in advance, a clone model to be trained is utilized to carry out model training on a data form sample, so that data form characteristics such as a list of the data form sample data table, a field value mode (logic), a date field of a service date and the like are obtained, and then the data form characteristics are constructed into a data clone model required by the subsequent clone data form. After receiving backup configuration for backing up data forms for realizing repeated use of subsequent cloning, the terminal equipment immediately acquires search key values of the original card numbers manually input in the backup configuration to form a search key value list, then calls a data cloning model constructed based on model training in advance, sequentially captures all data from a source environment manually input in the backup configuration according to the search key value list, and then performs deduplication processing on all data to obtain the data forms. The terminal equipment captures the obtained data forms from the source environment of the received backup configuration, and adds scene identifiers to the data forms one by one, so that the data form backup is stored in a database based on the scene identifiers for subsequent cloning to a target environment for application. After adding scene identification to the captured data form and storing the data form backup into a database, the terminal equipment reads the data form from the database according to the received clone configuration prepared by the loan test data and clones the data form to the empty account of the designated target environment in the clone configuration. After cloning the data form to the empty account of the target environment, the terminal device performs environment adaptation of data according to the cloned data form based on different environment types of the target environment or other adaptation factors to generate a target data form which conforms to the target environment and performs test application, so as to perform test application on the empty account of the target environment by using the target data form.

The invention avoids the problems that the traditional bank system requires long time for running and approving the prepared data form, and the prepared data form is difficult to be reused, so that the data needs to be prepared again when a new interest-bearing test is carried out, simplifies the process of generating the data form, improves the generation efficiency, can realize that the same data form is cloned to different target environments for self-test joint debugging, service testing and the like of development of different testers or roles, and greatly improves the reuse space of the data form.

Further, based on the first embodiment, a second embodiment of the data shape generating method based on cloning according to the present invention is provided, in this embodiment, the step of "performing batch replacement on the field value of the insertion statement" in the step S40 may include:

step S401, obtaining a difference field in the insertion statement and detecting a mapping field of the difference field;

step S402, extracting respective field values of the difference field and the mapping field, and generating corresponding replacement values for the field values;

step S403, calculating all matching results between the difference field and the mapping field and the field values, and performing batch replacement on the field values in all the matching results by using the replacement values.

After cloning the data form to the empty account of the target environment, the terminal device acquires the insertion statement of the data form which is added out of the empty account of the target environment according to a preset generation logic, acquires the difference field of the insertion statement and the mapping field corresponding to the difference field, extracts the field values corresponding to the difference field and the mapping field from the database, correspondingly generates a replacement value aiming at the field values, and replaces the field values in all the matching results by adopting a character string batch replacement method to be the corresponding generated replacement values after calculating all the matching results existing before the difference field and the mapping field and the field values, thereby taking the data form after replacing the field values as the target data form capable of performing test application on the empty account of the target environment.

It should be noted that, in this embodiment, the preset generation logic is the generation logic adopted when the clone sql statement is generated for each data table, that is, the generation logic is as follows:

firstly, the method comprises the following steps: if the source card number and the target card number have records and are the same in number, reading the difference field of the current data table in the data clone model and removing the difference field, and if the difference field has the residual field, updating the data of the source card number to the target card number;

secondly, the method comprises the following steps: if the source card number and the target card number are recorded but the numbers are different, the insertion sentences which are more than the source card number are directly copied;

thirdly, the method comprises the following steps: if the source card has record and the target card number has no record, the insertion statement of the source card is directly copied.

Specifically, for example, referring to the corresponding scenario shown in fig. 4, in the target environment empty account, if a field a in any inserted statement is a difference field and the field a is configured with mapping fields such as a field B and a field C (that is, the field a, the field B, and the field C will be treated as one field by leveling), the terminal device reads all values existing in the database for the field a, the field B, and the field C, generates a new value for each value in turn according to field value logic included in the data clone model, performs cartesian product operation on the field a, the field B, and the field C to obtain all possible matching results between the field a, the field B, and the field C and the value, and finally, the terminal device brings the field a, the field B, and the field C to a batch replacement method based on the existing mature string replacement method And carrying out batch replacement on the field names of the field B and the field C and the generated new values together on the values in all the matching results, so as to obtain the field A, the field B and the field C after the values of the field A, the field B and the field C are replaced respectively, and the terminal equipment takes the data form after the values of the field A, the field B and the field C are replaced respectively as the target data form for testing the empty account of the target environment.

In the embodiment, based on that the data form is directly cloned to the target card number to be subjected to the loan test, and then the target card number is integrally replaced aiming at the difference field of the data form, the relationship between the data form table and the relationship between different libraries are kept intact, and the relationship does not need to be manually sorted, so that the test efficiency of the loan test is further assisted to be improved.

Further, a third embodiment of the data pattern generation method based on cloning according to the present invention is proposed based on the first embodiment, and in this embodiment, after the step of "setting the data pattern after replacing the field value as the target data pattern" in the step S40, the data pattern generation method based on cloning according to the present invention may further include:

step S70, inquiring date field of cloned data form, and obtaining date time sequence of the date field according to preset time sequence translation reference calculation;

it should be noted that, in this embodiment, the preset timing shift reference may specifically be a first billing date found from the data format of the source card number after the clone configuration including the source card number, the source environment, the target card number and the target environment is inputted by the received staff and the data format of the source card number is directly cloned to the target card number (all systems in the loan core may have a date in a unified manner, and this date is the date on which the system itself rolls forward, and all transactions will be based on this date; the production environment business date is daily cut at 00:00:00 every night, so the business date is the same as the system date; but the test environment may be daily cut multiple times every day, or may skip lots, so the business date is different from the system date), and considering that the month exists, the timing shift must ensure that the billing date is a certain day fixed every month, therefore, in this example, the first billing day is selected as the time-series translation reference; in addition, the date sequence is specifically the relative days of the business date of each transaction and the bill day, and is in the format of NT +/-M (representing the Nth bill day +/-M days), and if the first bill day represents 1T +0, the date sequence of the day before the bill day is 1T-1.

Step S80, calculating a real date of the preset target environment corresponding to the date time sequence, and using the real date as a service date of the target data form.

It should be noted that, in this embodiment, when each date field extracted by the data clone model is copied to a target environment for performing a loan test, it is necessary to calculate a correct service date in the target environment, so that the clone data form meets the application requirement of cross-environment and cross-service dates.

After replacing the difference field and taking the replaced data form as a target data form for testing application on the empty account of the target environment, the terminal device further inquires and acquires a date field of the data clone model, synchronously captures the data form, translates and refers to the date time sequence of the date field according to a preset time sequence, and finally calculates the real date of the date time sequence in the empty account of the target environment to be used as the service date of the target data form when the loan test is carried out on the empty account of the target environment.

Specifically, for example, referring to the application scenario shown in fig. 5, the terminal device first extracts each date field and a staying day of the source data (or data of the source card number) stored in the database based on the date field synchronously captured by the data clone model when capturing the data form, then, the terminal device continues to query the first billing day business date of the source data based on the billing day sql statement configured in the data clone model (knowing that the first billing day time sequence is 1T +0), performs calculation according to the first billing day business date to obtain the date time sequence of each date field of the source data, and based on the principle that the data form of the clone does not change the staying day of the data form, if the date time sequence of the staying day of the source data is 4T +1 (i.e. stays on 4T +1 day), after cloning the source data to the target card number of the target environment, the time sequence of the target environment corresponding to the current business date and the stay date is also 4T +1 (i.e. the data form of the target environment stays at 4T + 1), so that the terminal device calculates all the date time sequences of the data form after the target card number cloned to the target environment corresponds to the real date-2021-10-16 of the target environment.

In the embodiment, by adopting the service date time sequence translation method, the relative date of the source card number is cloned to the target environment without changing, so that the test application of the cloned data form across services and dates is realized, and the recycling efficiency of the cloned data form is improved.

Further, a fourth embodiment of the data pattern generation method based on cloning according to the present invention is proposed based on the first embodiment, and in this embodiment, after "backup the data pattern in the database" in the step S20, the data pattern generation method based on cloning according to the present invention may further include:

step S203, reading characteristic values of each data form corresponding to preset characteristic parameters one by one in the database to form a characteristic value list of each data form;

step S204, traversing the characteristic values contained in each characteristic value list to detect whether a target characteristic value list with completely identical characteristic values exists;

in step S205, deduplication processing is performed for the data modality having the target feature value list.

It should be noted that, in this embodiment, as the terminal device continuously backs up the data form in the database, the data form in the database will be more and more, and as the backed-up data increases, the situation of repeatedly backing up the same data form in the database is inevitable to exist, so, for the benign management of the data form of the database, deduplication processing needs to be performed on the data form in the database, that is, a worker manually selects a feature parameter and a corresponding feature extraction mode, and performs clustering deduplication based on a feature value of the data form obtained by the feature parameter, and in addition, the preset feature parameter is the manually selected feature parameter, which includes but is not limited to: the action sequence, whether the action sequence is overdue and locking code identification, wherein the action sequence characteristic parameters are based on a data clone model produced by model training, and the action sequence is formed by extracting the date sequence + of each data form by using a date field in the data clone model and a date sequence calculated during multiplexing cloning.

It should be understood that, based on different design requirements of practical applications, in other embodiments, different characteristic parameters from those listed herein may be adopted, and the present invention is based on a clone data form generation method, and the type and number of the preset characteristic parameters are not specifically limited.

The terminal equipment performs characteristic value extraction on all data forms backed up and stored in a database one by one based on each manually selected characteristic parameter to form a characteristic value list of each data form, performs clustering operation on the sequence of the characteristic values based on taking the whole sequence of the characteristic values in the characteristic value list as a unique identifier to detect whether the characteristic value list has the same sequence, and thus performs deduplication processing on the data forms corresponding to the characteristic value list having the same sequence, or does not perform deduplication processing on the data forms if no characteristic value list having the same sequence is detected based on the operation.

Specifically, referring to the application scenario shown in fig. 6, the terminal device extracts the feature values of the data forms stored in the database (shown in the figure) one by one based on the features of the action sequence, expiration, lock code identification, and the like, that is, because each type of transaction action in the loan product has a unique transaction code, and each account transaction from the account is accumulated and stored in a fixed table, i.e., a transaction flow meter, the terminal device sequentially extracts the transaction actions of each data form in the database by using the transaction flow meter, except for the action, the terminal device needs to extract the time point (i.e., the service date, which is represented by the service date time sequence) at which the transaction action of each data form occurs, and the terminal device can perform equivalent classification for the date time sequence according to the life cycle performance of the loan product (so that the action date is not different but the action date is not represented by the service date sequence) Sequence does not converge) to obtain the following characteristic values of the date sequence:

day before the first bill: 0T, 0T +1, [0T +2, 1T-2], 1T-1,

the first billing day later: 1T, 1T +1, 1T +2, [1T +3, 2T-2], 2T-1,

the second billing day later: 2T, 2T +1, 2T +2, [2T +3, 3T-2], 3T-1,

the third billing day later: 3T, [3T +1, 4T-2], 4T-1,

fourth billing day later: 4T, [4T +1, 5T-1],

and (4) at the end stage: [ 5T ], [ to ],

the terminal device calculates a date time sequence of each service date of the transaction flow meter based on an algorithm that the current service date of the target environment is the same through calculation, so that the terminal device can extract an action sequence of the form from the transaction flow meter, and simultaneously define a staying day as an action, so that the extracted action sequence of the data form can be [0T day for borrowing, 1T +1 overdue for repayment, and stay in 1T +1 day ].

In addition, the terminal device may further extract characteristic values from the sql statement received from the configuration input by the staff, that is, whether the loan product is most concerned about overdue and the lock code identifier contained in the client, and the list of characteristic values of each data form extracted from the single table or the multiple tables by the terminal device based on the sql statement may be represented as: [ "eigenvalue 1": borrowing on 0T day, paying overdue for 1T +1, staying on 1T +1 day, and 'characteristic value 2': "is overdue", "eigenvalue 3": "W lock code" ].

The terminal equipment performs clustering operation on all data forms in the data warehouse by the characteristic value list of each data form to obtain the same type of data form of the same characteristic value list, so that the duplicate removal processing is performed on the same type of data form.

In the embodiment, the data form clustering deduplication algorithm performs deduplication processing on data forms backed up and stored in the database, so that virtuous circle and management of the database are facilitated, efficient clone data forms from the database are facilitated when the clone data forms are used for testing, and the efficiency of recycling the data forms is further improved.

Furthermore, the invention also provides a data form generation device based on cloning.

Referring to fig. 7, fig. 7 is a functional module diagram of an embodiment of a data pattern generation apparatus based on cloning according to the invention. As shown in fig. 7, the clone-based data pattern generation apparatus of the present invention includes:

the data capturing module 10 is configured to acquire a search key value, and call a data cloning model to capture a data form from a preset source environment according to the search key value;

a data cloning module 20, configured to backup the data form into a database, and clone the data form from the database to a preset target environment;

a statement acquiring module 30, configured to acquire an insertion statement in a cloned data form in the preset target environment;

and the environment adaptation module 40 is configured to perform batch replacement on the field dereferencing of the insertion statement, so that a data form after the field dereferencing is replaced is used as a target data form.

Further, the environment adaptation module 40 includes:

an obtaining unit, configured to obtain a difference field in the insertion statement and detect a mapping field of the difference field;

an extracting unit, configured to extract respective field values of the difference field and the mapping field, and generate a corresponding replacement value for each field value;

and the replacing unit is used for calculating all matching results between the difference field and the mapping field and each field value and performing batch replacement on the field values in all the matching results by using the replacing values.

Further, the environment adaptation module 40 further includes:

the query unit is used for querying the date field in the cloned data form and obtaining the date time sequence of the date field according to the preset time sequence translation reference calculation;

and the calculating unit is used for calculating the real date of the preset target environment corresponding to the date time sequence so as to take the real date as the business date of the target data form.

Further, the data modality generation apparatus based on cloning of the present invention further includes:

the model training module is used for acquiring data morphological characteristics based on model training to construct a data clone model, wherein the data morphological characteristics comprise: table list, difference field, value logic of difference field and service date field.

Further, the model training module comprises:

the system comprises a collecting unit, a processing unit and a processing unit, wherein the collecting unit is used for collecting the respective data forms of a preset first account and a preset second account as training samples, and the respective data environments of the first account and the second account are the same;

the training unit is used for inputting the training sample into a clone model to be trained for training so as to obtain a list, a difference field, the value logic of the difference field and a service date field;

and the model construction unit is used for assembling the table list, the difference field, the value logic of the difference field and the service date field to construct a data clone model.

Further, the preset source environment is a source environment in a preset backup configuration, and the data capture module 10 includes:

the reading unit is used for reading a search key value from a preset backup configuration;

and the capturing unit is used for inputting the search key value into the data clone model so that the data clone model sequentially captures data forms in the source environment according to the query key value marked on each data table in the table list.

Further, the data cloning module 20 includes:

the adding unit is used for adding scene identification to the data form according to the retention date of the data form, wherein the retention date is the service date captured by the data clone model when capturing the data form;

and the backup unit is used for backing up the data form into the database when detecting that the database does not contain the scene identification.

the data deduplication module is used for reading characteristic values of each data form corresponding to preset characteristic parameters one by one in the database to form a characteristic value list of each data form; and traversing the characteristic values contained in each characteristic value list to detect whether a target characteristic value list with completely identical characteristic values exists or not, and performing deduplication processing on the data form with the target characteristic value list.

Further, the data cloning module 20 of the cloning-based data modality generation apparatus of the present invention is further configured to:

The function implementation of each module in the clone-based data form generation apparatus corresponds to each step in the clone-based data form generation method embodiment, and the function and implementation process thereof are not described in detail herein.

The present invention also provides a computer storage medium having stored thereon a clone-based data modality generation program that, when executed by a processor, implements the steps of a clone-based data modality generation method according to any one of the above embodiments.

The specific embodiment of the computer storage medium of the present invention is substantially the same as the embodiments of the above-mentioned clone-based data pattern generation method, and is not described herein again.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.

The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.

Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) as described above and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.

The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.

Claims

1. A clone-based data modality generation method, characterized in that the clone-based data modality generation method comprises:

2. The clone-based data modality generation method according to claim 1, wherein the step of performing a batch replacement for the field values of the insert statements comprises:

3. The clone-based data modality generation method of claim 1, wherein after the step of taking the data modality after replacing the field value as a target data modality, further comprising:

4. The clone-based data modality generation method of claim 1, wherein the clone-based data modality generation method further comprises:

5. The clone-based data modality generation method of claim 1, wherein the preset source environment is a source environment in a preset backup configuration, the step of obtaining a search key value, invoking a data clone model to grab a data modality from the preset source environment according to the search key value comprises:

reading a search key value from a preset backup configuration;

the step of backing up the data modality to a database comprises:

6. The clone-based data modality generation method of claim 1, wherein after said step of backing up the data modality to a database, further comprising:

7. The clone-based data modality generation method of claim 1, wherein the clone-based data modality generation method further comprises:

8. A clone-based data modality generation apparatus, characterized in that the clone-based data modality generation apparatus comprises:

9. A terminal device, characterized in that the terminal device comprises: a memory, a processor, and a clone-based data modality generation program stored on the memory and executable on the processor, the clone-based data modality generation program, when executed by the processor, implementing the steps of the clone-based data modality generation method according to any one of claims 1 to 7.

10. A computer storage medium having stored thereon a clone-based data modality generation program that, when executed by a processor, implements the steps of a clone-based data modality generation method according to any one of claims 1 to 7.