CN112860729A - Operation method, operation device, electronic equipment and readable storage medium - Google Patents
Operation method, operation device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN112860729A CN112860729A CN202110333963.3A CN202110333963A CN112860729A CN 112860729 A CN112860729 A CN 112860729A CN 202110333963 A CN202110333963 A CN 202110333963A CN 112860729 A CN112860729 A CN 112860729A
- Authority
- CN
- China
- Prior art keywords
- data
- source data
- module
- data table
- mth
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 73
- 230000002776 aggregation Effects 0.000 claims abstract description 36
- 238000004220 aggregation Methods 0.000 claims abstract description 36
- 230000008569 process Effects 0.000 claims abstract description 17
- 238000004590 computer program Methods 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000012935 Averaging Methods 0.000 claims description 2
- 238000012545 processing Methods 0.000 abstract description 21
- 239000012634 fragment Substances 0.000 description 11
- 230000006870 function Effects 0.000 description 10
- 230000009286 beneficial effect Effects 0.000 description 8
- 230000003287 optical effect Effects 0.000 description 4
- 229920001621 AMOLED Polymers 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000005034 decoration Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 238000011022 operating instruction Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000002096 quantum dot Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
- G06F16/2438—Embedded query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24558—Binary matching operations
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of data processing, in particular to an operation method, an operation device, electronic equipment and a readable storage medium. The method comprises the following steps: acquiring source data; sequentially loading the source data into N data tables; wherein N is a positive integer greater than 1; in the source data loading process, executing target aggregation operation on the Mth data table meeting preset conditions; wherein M is a positive integer less than or equal to N. The application provides an operation scheme, and the operation efficiency of source data with large data volume is improved.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to an operation method, an operation device, electronic equipment and a readable storage medium.
Background
The analysis and calculation are carried out after batch processing is carried out on the account entry, account checking and statistical report data which are required to be carried out at the end of each transaction day in the financial industry, but the current business batch processing process in the financial industry has the characteristics of large data processing amount, long time consumption, automatic execution, periodic execution and the like. Particularly, when the system needs to load external source data and then perform aggregation operation processing, the problem caused by the batch processing is more serious because the current processing mode is to load the external source data into the existing database and perform aggregation operation on all the data after the data is loaded, so that the aggregation operation on all the data needs to be performed after all the data is loaded, if the data size is large, the loading and running time is long, and the efficiency of the business batch processing and analysis operation is low.
Disclosure of Invention
The present application aims to solve at least one of the above technical drawbacks. The technical scheme adopted by the application is as follows:
in a first aspect, an embodiment of the present application discloses an operation method, where the method includes:
acquiring source data;
sequentially loading the source data into N data tables; wherein N is a positive integer greater than 1;
in the source data loading process, executing target aggregation operation on the Mth data table meeting preset conditions; wherein M is a positive integer less than or equal to N.
Optionally, the mth data table satisfying the preset condition includes:
the data quantity loaded in the Mth data table meets a preset threshold value; or when M is equal to N, the source data is completely loaded.
Optionally, while acquiring the source data, the method further includes:
acquiring the storage capacity of the source data;
and creating N data tables in a database according to the storage capacity of the source data.
Optionally, when the amount of data loaded in the mth data table meets a preset threshold or the source data is completely loaded, the method further includes:
and marking the Mth data table as a data table which is completely loaded.
Optionally, the target aggregation operation comprises at least one of:
summing, counting, maximizing, minimizing, averaging.
Optionally, after performing the target aggregation operation on the mth data table satisfying the preset condition, the method further includes:
and storing the result of the target aggregation operation executed by the Mth data table in the intermediate table.
Further, the method further comprises:
and after the result of executing the target aggregation operation on the Nth data table is stored in the intermediate table, executing a unified target aggregation operation on the intermediate table.
In a second aspect, an embodiment of the present application provides an arithmetic device, including: an interface module, a creation module, a loading module, a storage module and an operation module, wherein,
the interface module is used for acquiring source data;
the creating module is used for creating N data tables in a database;
the loading module is used for sequentially loading the source data into N data tables; wherein N is a positive integer greater than 1;
the storage module is used for judging preset conditions of the Mth database;
the operation module is used for executing target aggregation operation on the Mth data table meeting preset conditions in the source data loading process; wherein M is a positive integer less than or equal to N.
In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory;
the memory is used for storing operation instructions;
the processor is configured to execute the method in any of the embodiments by calling the operation instruction.
In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the computer program implements the method of any one of the above embodiments.
The embodiment of the application provides an operation scheme, by carrying out fragment (data table) loading on source data and executing operation on a loaded fragment data table in a fragment loading process, that is, a scheme of synchronizing loading and operation is adopted, and the consistency of data operation logic is ensured through final summary operation. The method has the advantages that the operation efficiency of the source data with large data quantity is at least improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings used in the description of the embodiments of the present application will be briefly described below.
Fig. 1 is a schematic flowchart of an operation method according to an embodiment of the present disclosure;
fig. 2 is a schematic diagram of an arithmetic device according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary only for the purpose of explaining the present application and are not to be construed as limiting the present invention.
It will be understood by those skilled in the art that, unless otherwise specified, the singular forms "a", "an", "the" and "the" may include the plural forms, and the plural forms "a", "an", "a", and "the" are merely intended to illustrate the object definition for clarity and do not limit the object itself, and certainly, the object definition for "a" and "an" may be the same terminal, device, user, etc., and may also be the same terminal, device, user, etc. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.
In addition, it is to be understood that "at least one" in the embodiments of the present application means one or more, "a plurality" means two or more. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a alone, both A and B, and B alone, where A, B may be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. "at least one of the following" or similar expressions refer to any combination of these items, including any combination of the singular or plural items. For example, at least one (one) of a, b, or c, may represent: a, b, c, a and b, a and c, b and c, or a, b and c, wherein a, b and c can be single or multiple.
To more clearly describe the technical solutions of the present application, some concepts, terms or devices that the following embodiments may relate to are described below to help understand the operation schemes disclosed in the present application:
batch processing of data refers to a calculation process in which a system periodically and automatically performs business processing and analysis on large batches of data.
The aggregation operation of data mainly refers to a grouping aggregation function in the SQL language. The grouping aggregation function mainly includes sum, count, max, min, avg, and these functions can calculate the corresponding value of each grouping according to the grouping usage.
To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
The following describes the technical solutions of the present application and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
The following describes the technical solutions of the present application and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
Fig. 1 shows a schematic flowchart of an operation provided in an embodiment of the present application, and as shown in fig. 1, the method mainly includes:
s101, acquiring source data;
s102, sequentially loading the source data into N data tables; wherein N is a positive integer greater than 1;
in an optional embodiment of the present application, while obtaining the source data, the method further includes:
and acquiring the storage capacity of the source data, namely the size of a source data file, and creating N (fragment) data tables in a database according to the storage capacity of the source data. For example, 1000 pieces of source data, and if one data table segment is configured to store 200 pieces of data, 5 data tables need to be created. Creating a data table in the SQl query-capable database to complete the loading of source data as described herein.
S103, in the source data loading process, executing target aggregation operation on the Mth data table meeting preset conditions; wherein M is a positive integer less than or equal to N.
Optionally, the mth data table satisfying the preset condition includes: and marking the Mth data table as a loaded data table when the data quantity loaded in the Mth data table meets a preset threshold value. Or when M is equal to N, the Nth table can be marked with the loaded data table as long as the source data is loaded completely.
In this embodiment of the present application, in the loading process, when a certain data table, for example, the mth loaded data reaches a preset data amount threshold (for example, 200 ten thousand pieces of data mentioned in the above example), the current mth fragmented data table is stopped from being loaded and marked as having been loaded, and then the (M + 1) th data table is created again to place the data being loaded and the loading process is continued. The source data loading step is repeated until all the source data are loaded. Optionally, the target aggregation operation comprises at least one of: sum, count, max, min, avg, which are used in conjunction with group by to calculate the corresponding value for each table. Further, the result of the aggregation operation on the mth data table is placed into the aggregation operation intermediate table. According to the requirements of the original aggregation operation in the prior art, such as sum (), count (), avg (), max (), and min (), the same operator is used for the mth data table as the original aggregation operation, and if sum () operation is required to be used for data loaded in the mth database during the original overall operation, sum () operation is also used for the data of the mth data table in the present application. Therefore, when the data loading of a new data fragment table is completed, the aggregation operation is continuously performed on the data fragment table which is just loaded while the data is loaded.
In a specific embodiment, after performing the target aggregation operation on the mth data table satisfying the preset condition, the method further includes:
the result of the target aggregation operation performed by the mth data table is stored in the intermediate table, that is, after each data table is loaded, the operation is performed, and then the operation result of each data table is stored in an independent intermediate table, so that N operation results are stored in the intermediate table. Further, a unified target gather operation is performed again on the intermediate table at the end. In an alternative embodiment, if each loaded data table is subjected to aggregation operation according to a determined dimension, the data loading sequence is loaded according to the dimension and placed in the same data table, for example, if people gather, the data loading sequence is sorted according to people, that is, data of the same person is only placed in the same data table, then a second aggregation operation for the intermediate table may not be needed, and the result of the first operation stored in the intermediate table is correct.
To verify that the operation is correct for the N sliced data tables. The specific implementation process is as follows: according to the original requirement of aggregation operation, in the subsequent data processing, the summary operation needs to be performed on the existing aggregation operation result table again to complete the original logic. If the original aggregation operation is sum (), avg (), max (), and min (), the same summary operation is performed again; if the original aggregation operation is a count, the subsequent operation uses sum (), because the first count is equivalent to that the count is performed once for each fragment of the original large table data, and finally, the count result of each fragment needs to be sum in order to obtain the count after the last group by.
According to the embodiment of the application, the data are loaded in a fragmentation mode, the data loading and the data processing are processed in parallel in the whole data loading process, the consistency of data operation logic is guaranteed through final summary operation, and the whole operation efficiency of the whole data operation is improved.
Based on the operation method shown in fig. 1, another aspect of the present application provides an operation apparatus, as shown in fig. 2, the apparatus may include: 201 interface module, 202 creation module, 203 load module, 204 store module and 205 operation module, wherein,
the 201 interface module is used for acquiring source data;
the 202 creating module is configured to create N data tables in a database;
the 203 loading module is configured to sequentially load the source data into N data tables; wherein N is a positive integer greater than 1;
the 204 storage module is used for judging preset conditions of the Mth database;
the 205 operation module is configured to, in the source data loading process, perform a target aggregation operation on an mth data table that meets a preset condition; wherein M is a positive integer less than or equal to N.
It is to be understood that the above-described respective constituent devices of the arithmetic apparatus in the present embodiment have functions of realizing the respective steps of the method in the embodiment shown in fig. 1. The function can be realized by hardware, and can also be realized by executing corresponding software by hardware. The hardware or software includes one or more modules or systems corresponding to the above-described functions. The modules and systems can be software and/or hardware, and the modules and systems can be realized independently or integrated by a plurality of modules and systems. For the functional description of each module and system, reference may be specifically made to the corresponding description of the method in the embodiment shown in fig. 1, and therefore, the beneficial effects that can be achieved by the method may refer to the beneficial effects in the corresponding method provided above, which are not described again here.
It should be understood that the illustrated configuration of the embodiment of the present invention does not specifically limit the specific configuration of the arithmetic device. In other embodiments of the present application, the computing device may include more or fewer components than shown, or some components may be combined, some components may be split, or a different arrangement of components may be used. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
The embodiment of the application provides an electronic device, which comprises a processor and a memory;
a memory for storing operating instructions;
and the processor is used for executing the operation method provided by any embodiment of the application by calling the operation instruction.
As an example, fig. 3 shows a schematic structural diagram of an electronic device to which the embodiment of the present application is applied, and as shown in fig. 3, the electronic device 300 includes: a processor 301 and a memory 303. Wherein processor 301 is coupled to memory 303, such as via bus 302. Optionally, the electronic device 300 may further include a transceiver 304. It should be noted that the practical application of the transceiver 304 is not limited to one. It is to be understood that the illustrated structure of the embodiment of the present invention does not constitute a specific limitation to the specific structure of the electronic device 300. In other embodiments of the present application, electronic device 300 may include more or fewer components than shown, or some components may be combined, some components may be split, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. Optionally, the electronic device may further include a display screen 305 for displaying images or receiving operation instructions of a user as needed.
The processor 301 is applied to the embodiment of the present application, and is configured to implement the method shown in the foregoing method embodiment. The transceiver 304 may include a receiver and a transmitter, and the transceiver 304 is applied in the embodiment of the present application and is used for implementing the function of the electronic device of the embodiment of the present application to communicate with other devices when executed.
The Processor 301 may be a CPU (Central Processing Unit), a general-purpose Processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other Programmable logic device, a transistor logic device, a hardware component, or any combination thereof. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure. The processor 301 may also be a combination of computing functions, e.g., comprising one or more microprocessors, a combination of a DSP and a microprocessor, or the like.
The processor 301 may run the operation method provided in the embodiment of the present application, so as to reduce the operation complexity of the user, improve the intelligent degree of the terminal device, and improve the user experience. The processor 301 may include different devices, for example, when the CPU and the GPU are integrated, the CPU and the GPU may cooperate to execute the operation method provided in the embodiment of the present application, for example, part of the algorithm in the operation method is executed by the CPU, and another part of the algorithm is executed by the GPU, so as to obtain faster processing efficiency.
The Memory 303 may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact disk), a high speed Random Access Memory, a non-volatile Memory such as at least one magnetic disk storage device, a flash Memory device, a universal flash Memory (UFS), or other optical disk storage, optical disk storage (including Compact disk, laser disk, optical disk, digital versatile disk, blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, a magnetic disk storage medium, or other magnetic storage device, Or any other medium which can be used to carry or store desired program code in the form of instructions or data structures and which can be accessed by a computer, but is not limited to such.
Optionally, the memory 303 is used for storing application program codes for executing the scheme of the present application, and is controlled by the processor 301 to execute. The processor 301 is configured to execute the application program code stored in the memory 303 to implement the operation method provided in any embodiment of the present application.
The memory 303 may be used to store computer-executable program code, which includes instructions. The processor 301 executes various functional applications of the electronic device 300 and data processing by executing instructions stored in the memory 303. The memory 303 may include a program storage area and a data storage area. Wherein, the storage program area can store the codes of the operating system and the application program, etc. The storage data area may store data created during use of the electronic device 300 (e.g., images, video, etc. captured by a camera application), and the like.
The memory 303 may further store one or more computer programs corresponding to the operation methods provided in the embodiments of the present application. The one or more computer programs stored in the memory 303 and configured to be executed by the one or more processors 301 include instructions that may be used to perform the various steps in the respective embodiments described above.
Of course, the codes of the operation methods provided in the embodiments of the present application may also be stored in the external memory. In this case, the processor 301 may execute the code of the arithmetic method stored in the external memory through the external memory interface, and the processor 301 may control the operation flow.
The display screen 305 includes a display panel. The display panel may be a Liquid Crystal Display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (active-matrix organic light-emitting diode, AMOLED), a flexible light-emitting diode (FLED), a miniature, a Micro-oeld, a quantum dot light-emitting diode (QLED), or the like. In some embodiments, the electronic device 300 may include 1 or N display screens 305, N being a positive integer greater than 1. The display screen 305 may be used to display information input by or provided to the user as well as various Graphical User Interfaces (GUIs). For example, the display screen 305 may display a photograph, video, web page, or file, etc.
The electronic device provided by the embodiment of the present application is applicable to any embodiment of the above method, and therefore, the beneficial effects that can be achieved by the electronic device can refer to the beneficial effects in the corresponding method provided above, and are not described again here.
The embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the computer program implements the operation method shown in the above method embodiment.
The computer-readable storage medium provided in the embodiments of the present application is applicable to any embodiment of the foregoing method, and therefore, the beneficial effects that can be achieved by the computer-readable storage medium can refer to the beneficial effects in the corresponding method provided above, and are not described herein again.
The embodiment of the present application further provides a computer program product, which when running on a computer, causes the computer to execute the above related steps to implement the method in the above embodiment. The computer program product provided in the embodiments of the present application is applicable to any of the embodiments of the method described above, and therefore, the beneficial effects that can be achieved by the computer program product can refer to the beneficial effects in the corresponding method provided above, and are not described herein again.
The embodiment of the application provides an operation scheme, by carrying out fragment (data table) loading on source data and executing operation on a loaded fragment data table in a fragment loading process, that is, a scheme of synchronizing loading and operation is adopted, and the consistency of data operation logic is ensured through final summary operation. The method has the advantages that the operation efficiency of the source data with large data quantity is at least improved.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, a module or a unit may be divided into only one logical function, and may be implemented in other ways, for example, a plurality of units or components may be combined or integrated into another apparatus, or some features may be discarded or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
Units described as separate parts may or may not be physically separate, and parts displayed as units may be one physical unit or a plurality of physical units, may be located in one place, or may be distributed to a plurality of different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially or partially contributed to by the prior art, or all or part of the technical solutions may be embodied in the form of a software product, where the software product is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the methods of the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of changes or substitutions within the technical scope of the present application, and can make several modifications and decorations, and these changes, substitutions, improvements and decorations should also be considered to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A method of operation, the method comprising:
acquiring source data;
sequentially loading the source data into N data tables; wherein N is a positive integer greater than 1;
in the source data loading process, executing target aggregation operation on the Mth data table meeting preset conditions; wherein M is a positive integer less than or equal to N.
2. The calculation method according to claim 1, wherein the mth data table satisfying the preset condition comprises:
the data quantity loaded in the Mth data table meets a preset threshold value; or the like, or, alternatively,
and when M is equal to N, finishing the loading of the source data.
3. The operational method of claim 2, wherein while acquiring the source data, the method further comprises:
acquiring the storage capacity of the source data;
and creating N data tables in a database according to the storage capacity of the source data.
4. The calculation method according to claim 3, wherein when the amount of data loaded in the Mth data table satisfies a preset threshold or the source data is loaded completely, the method further comprises:
and marking the Mth data table as a data table which is completely loaded.
5. The method of claim 1 or 4, wherein the target gather operation comprises at least one of:
summing, counting, maximizing, minimizing, averaging.
6. The method according to claim 5, wherein after performing the target aggregation operation on the mth data table satisfying the preset condition, the method further comprises:
and storing the result of the target aggregation operation executed by the Mth data table in the intermediate table.
7. The method of claim 6, further comprising:
and after the result of executing the target aggregation operation on the Nth data table is stored in the intermediate table, executing a unified target aggregation operation on the intermediate table.
8. An arithmetic device, the device comprising: an interface module, a creation module, a loading module, a storage module and an operation module, wherein,
the interface module is used for acquiring source data;
the creating module is used for creating N data tables in a database;
the loading module is used for sequentially loading the source data into N data tables; wherein N is a positive integer greater than 1;
the storage module is used for judging preset conditions of the Mth data table;
the operation module is used for executing target aggregation operation on the Mth data table meeting preset conditions in the source data loading process; wherein M is a positive integer less than or equal to N.
9. An electronic device comprising a processor and a memory;
the memory is used for storing operation instructions;
the processor is used for executing the method of any one of claims 1-7 by calling the operation instruction.
10. A computer-readable storage medium, characterized in that the storage medium has stored thereon a computer program which, when being executed by a processor, carries out the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110333963.3A CN112860729A (en) | 2021-03-29 | 2021-03-29 | Operation method, operation device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110333963.3A CN112860729A (en) | 2021-03-29 | 2021-03-29 | Operation method, operation device, electronic equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112860729A true CN112860729A (en) | 2021-05-28 |
Family
ID=75993119
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110333963.3A Pending CN112860729A (en) | 2021-03-29 | 2021-03-29 | Operation method, operation device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112860729A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130238596A1 (en) * | 2012-03-08 | 2013-09-12 | International Business Machines Corporation | Method to detect reference data tables in etl processes |
| CN103731451A (en) * | 2012-10-12 | 2014-04-16 | 腾讯科技（深圳）有限公司 | Method and system for uploading file |
| CN104063376A (en) * | 2013-03-18 | 2014-09-24 | 阿里巴巴集团控股有限公司 | Multi-dimensional grouping operation method and system |
| CN108874738A (en) * | 2018-06-05 | 2018-11-23 | 中国平安人寿保险股份有限公司 | Distributed parallel operation method, device, computer equipment and storage medium |
| CN111160865A (en) * | 2019-12-31 | 2020-05-15 | 新奥数能科技有限公司 | Workflow management method and device |
| CN112256218A (en) * | 2020-09-30 | 2021-01-22 | 神思旭辉医疗信息技术有限责任公司 | Method and device for drawing massive data graphs |
-
2021
- 2021-03-29 CN CN202110333963.3A patent/CN112860729A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130238596A1 (en) * | 2012-03-08 | 2013-09-12 | International Business Machines Corporation | Method to detect reference data tables in etl processes |
| CN103731451A (en) * | 2012-10-12 | 2014-04-16 | 腾讯科技（深圳）有限公司 | Method and system for uploading file |
| CN104063376A (en) * | 2013-03-18 | 2014-09-24 | 阿里巴巴集团控股有限公司 | Multi-dimensional grouping operation method and system |
| CN108874738A (en) * | 2018-06-05 | 2018-11-23 | 中国平安人寿保险股份有限公司 | Distributed parallel operation method, device, computer equipment and storage medium |
| CN111160865A (en) * | 2019-12-31 | 2020-05-15 | 新奥数能科技有限公司 | Workflow management method and device |
| CN112256218A (en) * | 2020-09-30 | 2021-01-22 | 神思旭辉医疗信息技术有限责任公司 | Method and device for drawing massive data graphs |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10943324B2 (en) | Data processing method, apparatus, and electronic device | |
| CN112487784B (en) | Technical document management method, device, electronic equipment and readable storage medium | |
| CN112764911A (en) | Task scheduling method and device, electronic equipment and readable storage medium | |
| US20190122421A1 (en) | Batch rendering method, device, and apparatus | |
| US11734007B2 (en) | Address generation method, related apparatus, and storage medium | |
| CN112486806A (en) | Test case execution method and device, electronic equipment and readable storage medium | |
| US20190318461A1 (en) | Histogram Statistics Circuit and Multimedia Processing System | |
| CN112187530A (en) | Log analysis method and device, electronic equipment and readable storage medium | |
| WO2020248365A1 (en) | Intelligent model training memory allocation method and apparatus, and computer-readable storage medium | |
| CN114529658A (en) | Graph rendering method and related equipment thereof | |
| US20220351490A1 (en) | Convolution calculation method, convolution calculation apparatus, and terminal device | |
| CN116662038B (en) | Industrial information detection method, device, equipment and medium based on shared memory | |
| US11635904B2 (en) | Matrix storage method, matrix access method, apparatus and electronic device | |
| CN105243399A (en) | Method of realizing image convolution and device, and method of realizing caching and device | |
| CN111932595A (en) | Image registration method and device, electronic equipment and storage medium | |
| US20220343512A1 (en) | Method and apparatus of processing image, electronic device, and storage medium | |
| CN112487071A (en) | Transaction query method and device, electronic equipment and readable storage medium | |
| CN112487013A (en) | Database operation code generation method and device | |
| CN112860729A (en) | Operation method, operation device, electronic equipment and readable storage medium | |
| CN116451174A (en) | Task execution device, method, electronic device, and storage medium | |
| CN111382831A (en) | Method and device for accelerating forward reasoning of convolutional neural network model | |
| CN113657408B (en) | Method and device for determining image characteristics, electronic equipment and storage medium | |
| CN112817972A (en) | Data storage method, data query method, data storage device, data query device and electronic equipment | |
| CN108491546A (en) | A kind of page switching method and electronic equipment | |
| JP2014099848A (en) | Image division system and image division method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |