CN113901118A

CN113901118A - Data export method, terminal equipment and storage medium

Info

Publication number: CN113901118A
Application number: CN202111158401.6A
Authority: CN
Inventors: 李政军; 陈娅芳
Original assignee: Hunan New Cloudnet Technology Co ltd
Current assignee: Hunan New Cloudnet Technology Co ltd
Priority date: 2021-09-28
Filing date: 2021-09-28
Publication date: 2022-01-07

Abstract

The application is applicable to the technical field of data processing, and provides a data export method, terminal equipment and a storage medium. The method comprises the steps of obtaining a data characteristic statement set of a first database, wherein the data characteristic statement set is used for representing data characteristics of data to be exported in the form of codes; splitting the data characteristic statement set to obtain at least two subdata characteristic statement sets; executing at least one sub data characteristic statement set one by one; when a subdata characteristic statement set is executed, a corresponding data characteristic set is obtained, and the data characteristic set is used for representing the data characteristics of the data to be exported in a plaintext form; the data to be exported stored in the first database is exported to the second database through the data feature set, the data sentences of the first database are split into at least two sub-data sentences, the load during data extraction can be reduced, the data extraction speed is increased in a parallel data extraction mode, and therefore the data export efficiency is improved.

Description

Data export method, terminal equipment and storage medium

Technical Field

The present application belongs to the field of data processing technologies, and in particular, to a data export method, a terminal device, and a storage medium.

Background

At present, enterprises and public institutions which have data processing services or generate a large amount of data in the process of processing the services usually choose to import the data into a database for unified storage and management, the operation database needs to have certain professional knowledge, data export cannot be realized according to requirements for users lacking the professional knowledge of the database, and in order to meet the data export requirements of most users, a table export script with data capture capacity and export capacity can be adopted to realize automatic data export.

The data export speed and the data volume of the existing table export script are in positive correlation, so that when the data volume is large, the time required for exporting data by the table export script is synchronously improved, the data export efficiency is low, and therefore how to improve the data export efficiency of the table export script becomes a problem which needs to be solved urgently at present.

Disclosure of Invention

In view of this, embodiments of the present application provide a data export method, a terminal device, and a storage medium, so as to solve the problems that when the data volume is large, the time required for exporting data by a table export script is also increased synchronously, and the data export efficiency is low, because the export speed of data of an existing table export script is positively correlated with the data volume.

A first aspect of an embodiment of the present application provides a data export method, where the method includes:

acquiring a data characteristic statement set of a first database, wherein the data characteristic statement set is used for representing data characteristics of data to be exported in a code form;

splitting the data characteristic statement set to obtain at least two subdata characteristic statement sets;

executing at least one sub data characteristic statement set one by one;

when the sub-data characteristic statement set is executed, a corresponding data characteristic set is obtained, and the data characteristic set is used for representing the data characteristics of the data to be exported in a plaintext form;

and exporting the data to be exported stored in the first database to a second database through the data feature set.

In one embodiment, the method further comprises:

and when the data characteristic statement set is split, adding 1 to the numerical value of the counter every time one subdata characteristic statement set is obtained, wherein the numerical value of the counter is used for representing the number of the current subdata characteristic statement sets.

In one embodiment, the method further comprises:

and when the sub-data characteristic statement sets are executed, subtracting 1 from the value of the counter every time one sub-data characteristic statement set is executed, wherein the value of the counter is used for representing the number of the currently unexecuted sub-database statement sets.

In one embodiment, the method further comprises:

and storing the obtained data characteristic set to a export queue every time one sub-data characteristic statement set is executed.

In an embodiment, the above causing the data to be exported, which is stored in the first database, to the second database through the data feature set includes:

reading a plurality of data feature sets stored in the export queue when the value of the counter is reduced to 0;

and exporting the data to be exported stored in the first database to a second database according to the plurality of data feature sets.

In an embodiment, the exporting the data to be exported stored in the first database to the second database according to the plurality of data feature sets includes:

exporting the data to be exported stored in the first database to any table of the second database according to the plurality of data feature sets;

when the data volume of the data to be exported exceeds the preset data volume of the table, controlling the second database to newly build the table;

exporting the un-exported data to be exported to a newly-built table.

In one embodiment, the method further comprises:

when the data to be exported are not exported and the table number of the second database exceeds a preset table number, a second database is newly established;

exporting the un-exported data to be exported to a newly-built second database.

In one embodiment, the method further comprises:

the database type of the second database is workbook Excel.

A second aspect of the embodiments of the present application provides a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the data export method provided by the first aspect of the embodiments of the present application when executing the computer program.

A third aspect of the embodiments of the present application provides a computer-readable storage medium, where a computer program is stored, and the computer program, when executed by a processor, implements the data export method provided by the first aspect of the embodiments of the present application.

A first aspect of the embodiments of the present application provides a data export method, in which a data statement of a first database is split into at least two sub-data statements, so that a load during data extraction can be reduced, and a speed of data extraction is increased by parallel data extraction, thereby improving data export efficiency.

It is understood that, the beneficial effects of the second aspect and the third aspect can be referred to the related description of the first aspect, and are not described herein again.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.

Fig. 1 is a schematic flowchart of a data export method provided in an embodiment of the present application;

fig. 2 is a schematic flowchart of step S104 in the data export method provided in the embodiment of the present application;

fig. 3 is a schematic flowchart of step S105 in the data export method provided in the embodiment of the present application;

fig. 4 is a module schematic diagram of a terminal device provided in an embodiment of the present application.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It should also be understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to" determining "or" in response to detecting ". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".

Furthermore, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used for distinguishing between descriptions and not necessarily for describing or implying relative importance.

Reference throughout this specification to "one embodiment" or "some embodiments," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," or the like, in various places throughout this specification are not necessarily all referring to the same embodiment, but rather "one or more but not all embodiments" unless specifically stated otherwise. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless expressly specified otherwise.

In application, the data export speed and the data volume of the existing table export script are in positive correlation, so that when the data volume is large, the time required for exporting the data by the table export script is synchronously improved, and the data export efficiency is low.

In view of the above technical problems, embodiments of the present application provide a data export method, which splits a data statement of a first database into at least two sub-data statements, so as to reduce a load when data is extracted, and increase a speed of extracting data by a parallel data extraction manner, thereby improving data export efficiency.

The data export method provided by the embodiment of the application can be applied to any terminal equipment with a data processing function. The terminal device may be a mobile phone, a tablet computer, a wearable device, an in-vehicle device, an Augmented Reality (AR)/Virtual Reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a Personal Digital Assistant (PDA), and the like, and the specific type of the terminal device is not limited in this embodiment.

As shown in fig. 1, the data export method provided in the embodiment of the present application includes the following steps S101 to S105:

step S101, a data characteristic statement set of a first database is obtained, and the data characteristic statement set is used for representing data characteristics of data to be exported in the form of codes.

In application, the data feature statement set represents the data feature of the data to be exported in the form of a code, where the code may be an SQL (Structured Query Language) code applied to a database, and the specific type of the code is different according to different database types, for example, when the database type of the first database is Oracle, the specific type of the code is Oracle SQL. The database types of the first database may further include MySQL, MongoDB, SQL Server, IBM Db2, and dameng database, and the database types of the first database are not limited in this embodiment of the application. The data to be exported may be multi-column data of the first database, data of multiple tables, or complete data, and the data range of the data to be exported may be set according to the actual needs of the user.

In application, the data feature of the data to be derived may include a subject of a title (Class), that is, one data feature statement set may characterize a subject of one or more titles, the data feature statement set may include a plurality of data feature statements, and the plurality of data feature statements in one data feature statement set may characterize, in the form of codes, fields included in the corresponding titles of the data feature statement set.

In one embodiment, step S101 includes:

and acquiring a data characteristic statement set of the first database according to a preset time period.

In application, when a user continuously has a demand for data export, the data feature statement set of the first database may be obtained according to a preset time period, for example, assuming that the preset time period is 1 month, the data feature statement set generated by the first database in the 1 st month is obtained at the end of the 1 st month, the data feature statement set generated by the first database in the 2 nd month is obtained at the end of the 2 nd month, and so on, and the data feature statement set generated by the first database in the corresponding month is obtained month by month. The preset time period can be set according to actual needs, and the specific duration of the preset time period is not limited in any way in the embodiment of the application.

And S102, splitting the data characteristic statement set to obtain at least two subdata characteristic statement sets.

In application, the sub-data feature statement set is a subset of the data feature statement set. At least two subdata characteristic statement sets can be obtained by splitting the data characteristic statement sets, and the data volume of a single subdata characteristic statement set is smaller than that of the data characteristic statement sets.

In application, the data feature statement set may be split according to the number of preset data feature statements, for example, assuming that the number of preset data feature statements is 10, splitting every 10 preset data feature statements in the data feature statement set into one sub data feature statement set. The specific size of the preset data feature statement quantity can be set according to actual needs.

In one embodiment, step S102 includes:

and splitting the data characteristic statement sets according to a preset splitting period to obtain at least two subdata characteristic statement sets.

In application, when the data feature statement set of the first database is obtained according to a preset time period, the data feature statement set may be split according to a preset split period, for example, assuming that the preset time period is 1 month and the preset split period is 1 week, when the 1 st month ends, the data feature statement set generated by the first database in the 1 st month is obtained and split, so as to obtain the sub-data feature statement set of the 1 st week in the 1 st month, the sub-data feature statement set of the 2 nd week in the 1 st month, the sub-data feature statement set of the 3 rd week in the 1 st month, and the sub-data feature statement set of the 4 th week in the 1 st month, respectively. It is easily understood that the splitting times of the data feature statement set is equal to the preset splitting period divided by the preset time period.

Step S103, executing at least one subdata characteristic statement set successively;

and S104, executing each subdata characteristic statement set to obtain a corresponding data characteristic set, wherein the data characteristic set is used for representing the data characteristics of the data to be exported in a plaintext form.

In the application, a data feature set for parsing the sub-data feature statement set in the form of code into a plaintext form is executed, and changing the form of the characterizing data feature does not change the content of the characterizing data feature. When a subdata characteristic statement set is executed, the subdata characteristic statement set needs to be sent into a memory, a corresponding data characteristic set is obtained through processing of a processor, and by executing the subdata characteristic statement set one by one, compared with the data characteristic statement set with large data volume, the load of the memory and the processor can be greatly reduced, and the memory overflow caused by the fact that the data volume input to the memory in a short time is overlarge is avoided. Wherein the set of data features is used to characterize the data features of the data to be derived in the clear.

And S105, exporting the data to be exported stored in the first database to a second database through the data feature set.

In application, the data feature set can be input into the second database, and data to be exported stored in the first database is extracted and exported to the second database by using a Java reflection mechanism (the Java reflection mechanism means that in a running state, for any object, all attributes and methods included in the object can be acquired), so that data export is realized. Specifically, the object of the title corresponding to the data feature set may be obtained, and the fields included in the title may be extracted according to the object of the title through a reflection mechanism of Java. In the process of data export, the data export speed can be increased through multi-thread parallel processing on the premise of not exceeding the load capacity of the processor.

Firstly, acquiring a parent class of an object through a getSuperclass function, and acquiring attributes of the object and the parent class thereof through getDeclearFields function reflection; and obtaining the annotation value of the attribute of the object and the parent class thereof through a getAnnoutation function. And the three functions are encapsulated into a method, so that the method is conveniently and directly called subsequently to realize data export.

In one embodiment, the database type of the second database is workbook Excel.

In application, the database model of the second database is consistent with the database model of the first database, and may be specifically a workbook Excel.

In application, the data statement of the first database is split into at least two sub-data statements, so that the load during data extraction can be reduced, the data extraction speed is increased in a parallel data extraction mode, and the data export efficiency is improved.

As shown in fig. 2, in an embodiment, the step S104 further includes the following steps:

step S1041, when executing the subdata characteristic statement sets, subtracting 1 from the value of the counter when executing one subdata characteristic statement set, wherein the value of the counter is used for representing the number of the currently unexecuted subdata database statement sets;

step S1042, when a subdata characteristic statement set is executed, storing the obtained data characteristic set to a export queue;

in step S1043, when the value of the counter is decreased to 0, the plurality of data feature sets stored in the export queue are read.

In application, regarding step S1041, the number of the sub-data feature statement sets may be recorded by a counter, and specifically, each time a sub-data feature statement set is obtained, the value of the counter is incremented by 1. When the sub-data feature statement set is executed, the sub-data feature statement set can be executed in parallel in multiple threads, and specifically, one sub-data feature statement set can be executed in one thread. And after each sub data characteristic statement set is executed, subtracting 1 from the value of the counter, so as to accurately obtain the execution condition of the sub data characteristic statement set. The counter may be a Java-based thread counter (CountDownLatch), and the thread counter may obtain an execution condition of each thread and count when each thread executes a corresponding task, so that each thread may automatically count after executing one sub-data feature statement set, thereby improving accuracy and convenience of counting.

In application, for each execution of the sub data feature statement set in step S1042 and step S1043, the obtained data feature set may be stored in the export queue, and the export queue may temporarily store the data feature set in the memory. When the value of the counter is reduced to 0, the data feature sets stored in the export queue can be read, so that the processor and the memory can extract data in a first time period and read data in a second time period, the data are prevented from being extracted and read in the same time period, the load of the processor and the memory is reduced, the data export speed is increased, the resource occupation is reduced, and the data export stability is improved. The export queue may be specifically implemented based on a blocking queue (blocking queue) of Java.

As shown in fig. 3, in one embodiment, step S105 further includes the steps of:

step S1051, exporting the data to be exported stored in the first database to any table of the second database according to a plurality of data feature sets;

step 1052, controlling a second database to newly build a table when the data volume of the data to be exported exceeds the preset data volume of the table;

step S1053, exporting the data to be exported which is not exported to a newly-built table;

step S1054, when the data to be exported is not exported, and the table number of the second database exceeds the preset table number, a second database is newly built;

and step S1055, exporting the un-exported data to be exported to a newly-built second database.

In the application, regarding steps S1051 to S1053, the second database may be provided with a preset data amount of the table, where the preset data amount may specifically be a preset number of cells (e.g., one thousand cells or two thousand cells, etc.), or may also be a preset storage capacity (e.g., 1 megabyte or 2 megabytes, etc.), when the data amount of the data to be exported exceeds the preset data amount of the table, a new table is created in the second database, and the remaining data to be exported that is not exported is exported to the new table, where the specific size of the preset data amount may be set according to actual needs.

In the application, regarding to step S1054 and step S1055, when the data to be exported is not completed, and the second database continues to build a new table, and the number of tables in the second database exceeds the preset number of tables, a second database is built, and the remaining data to be exported, which is not exported, is exported to the newly built second database. For example, when the second database is the workbook Excel, when the number of tables (Sheet) of one workbook Excel exceeds the preset number of tables, a new workbook Excel is created to continuously store the imported data.

In application, by limiting the data volume of each table and the data volume of each database, the situation that a single table or a single database stores multiple data can be avoided, the data processing pressure of a memory is reduced, and the flexibility of data storage is improved by storing the data in different tables or different databases.

According to the data export method provided by the embodiment of the application, the data sentences of the first database are split into at least two sub-data sentences, so that the load during data extraction can be reduced, the data extraction speed is increased in a parallel data extraction mode, and the data export efficiency is improved.

It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

As shown in fig. 4, the present embodiment further provides a terminal device 10, which includes a memory 110, a processor 111, and a computer program 112 stored in the memory 110 and executable on the processor 111, where the processor 111 implements the steps in the data exporting method embodiments when executing the computer program 112.

In an application, the terminal device may include, but is not limited to, a processor, a memory. Those skilled in the art will appreciate that fig. 4 is merely an example of a terminal device, and does not constitute a limitation of the terminal device, and may include more or less components than those shown, or combine some components, or different components, such as an input-output device, a network access device, etc.

In an Application, the Processor may be a Central Processing Unit (CPU), and the Processor may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

In some embodiments, the storage may be an internal storage unit of the terminal device, such as a hard disk or a memory of the terminal device. The memory may also be an external storage device of the terminal device in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the terminal device. Further, the memory may also include both an internal storage unit of the terminal device and an external storage device. The memory is used for storing an operating system, application programs, a BootLoader (BootLoader), data, and other programs, such as program codes of computer programs. The memory may also be used to temporarily store data that has been output or is to be output.

It should be noted that, because the contents of information interaction, execution process, and the like between the modules are based on the same concept as that of the embodiment of the method of the present application, specific functions and technical effects thereof may be specifically referred to a part of the embodiment of the method, and details are not described here.

It will be clear to those skilled in the art that, for convenience and simplicity of description, the foregoing division of the functional modules is merely illustrated, and in practical applications, the above function distribution may be performed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to perform all or part of the above described functions. Each functional module in the embodiments may be integrated into one processing module, or each module may exist alone physically, or two or more modules are integrated into one module, and the integrated module may be implemented in a form of hardware, or in a form of software functional module. In addition, specific names of the functional modules are only used for distinguishing one functional module from another, and are not used for limiting the protection scope of the application. The specific working process of the modules in the system may refer to the corresponding process in the foregoing method embodiment, and is not described herein again.

The embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the computer program can implement the steps in the data exporting method embodiments.

The integrated module may be stored in a computer-readable storage medium if it is implemented in the form of a software functional module and sold or used as a separate product. Based on such understanding, all or part of the processes in the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium and can implement the steps of the embodiments of the methods described above when the computer program is executed by a processor. The computer program includes computer program code, and the computer program code may be in a source code form, an object code form, an executable file or some intermediate form. The computer-readable medium may include at least: any entity or apparatus capable of carrying computer program code to a photographing terminal device, recording medium, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, and software distribution medium. Such as a usb-disk, a removable hard disk, a magnetic or optical disk, etc.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.

Those of ordinary skill in the art will appreciate that the various illustrative modules and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiments provided in the present application, it should be understood that the disclosed terminal device and method may be implemented in other ways. For example, the above-described terminal device embodiment is only illustrative, and for example, the division of the above modules is only one logical function division, and there may be other division ways in actual implementation, for example, a plurality of modules or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.

The above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.

Claims

1. A method of data derivation, the method comprising:

acquiring a data characteristic statement set of a first database, wherein the data characteristic statement set is used for representing data characteristics of data to be exported in the form of codes;

executing at least one sub data characteristic statement set in sequence;

when one subdata feature statement set is executed, a corresponding data feature set is obtained, and the data feature set is used for representing the data features of the data to be exported in a plaintext form;

2. The data derivation method of claim 1, wherein the method further comprises:

and when the data characteristic statement set is split, adding 1 to the numerical value of a counter every time one subdata characteristic statement set is obtained, wherein the numerical value of the counter is used for representing the number of the current subdata characteristic statement sets.

3. The data derivation method of claim 2, wherein the method further comprises:

and when the sub-data characteristic statement sets are executed, subtracting 1 from the value of the counter when one sub-data characteristic statement set is executed, wherein the value of the counter is used for representing the number of the currently unexecuted sub-database statement sets.

4. The data derivation method of claim 1, wherein the method further comprises:

and storing the obtained data feature set to a export queue every time one sub-data feature statement set is executed.

5. The data export method of claim 1, wherein the causing of the data to be exported stored in the first database to the second database via the data feature set comprises:

6. The data export method of claim 5, wherein exporting the data to be exported stored in the first database to the second database according to the plurality of data feature sets comprises:

exporting the data to be exported which are not exported to a newly-built table.

7. The data derivation method of claim 6, wherein the method further comprises:

when the data to be exported is not exported, and the table number of the second database exceeds the preset table number, a second database is newly established;

exporting the un-exported data to be exported to a newly-built second database.

8. The data export method of any of claims 1 to 7, wherein the database type of the second database is workbook Excel.

9. A terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the data exporting method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, carries out a data derivation method according to any one of claims 1 to 8.