CN117827949A - Method and device for batch writing of data into database based on memory queue - Google Patents
Method and device for batch writing of data into database based on memory queue Download PDFInfo
- Publication number
- CN117827949A CN117827949A CN202311862320.3A CN202311862320A CN117827949A CN 117827949 A CN117827949 A CN 117827949A CN 202311862320 A CN202311862320 A CN 202311862320A CN 117827949 A CN117827949 A CN 117827949A
- Authority
- CN
- China
- Prior art keywords
- data
- batch
- queue
- piece
- reaches
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 33
- 238000000605 extraction Methods 0.000 claims abstract description 19
- 238000013480 data collection Methods 0.000 claims description 4
- 230000007246 mechanism Effects 0.000 abstract description 11
- 230000010354 integration Effects 0.000 abstract description 4
- 238000013075 data extraction Methods 0.000 abstract description 3
- 238000003780 insertion Methods 0.000 abstract description 3
- 230000037431 insertion Effects 0.000 abstract description 3
- 238000012545 processing Methods 0.000 description 12
- 238000004891 communication Methods 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000007547 defect Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a method and a device for writing data into a database in batches based on a memory queue, and relates to the technical field of computers, wherein the method comprises the following steps: acquiring a large amount of single data reported by a data source at high frequency, collecting the single data piece by piece, and then collecting the single data piece by piece into a memory queue; judging whether the number of data buffered in the queue in the memory queue reaches a set number value or not, and judging whether the difference between the current moment and the last batch extraction moment reaches a set time or not; and if the data quantity of the queue cache reaches a quantity set value or the difference between the current time and the last batch drawing time reaches a set time, carrying out batch drawing on the queue cache and writing the batch into a database. Based on the scheme provided by the application, for high-frequency data with small data packets, batch integration is carried out on piece-by-piece data before warehousing, the data is inserted piece by piece and converted into batch insertion, and data extraction is carried out through a timing mechanism and a self-defining mechanism, so that the batch and real-time performance of data warehousing are ensured.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a method and an apparatus for batch writing of data into a database based on a memory queue.
Background
At present, in the software development process (particularly in the Internet of things and big data scenes), a single high-frequency data processing condition is usually encountered. Due to network transmission and database storage mechanisms, the situation that the content is less in each writing and the writing frequency is high is faced, and the pressure on database processing is caused.
In the prior art, a set is usually maintained in a memory and is used for accumulating data, piling the data to a certain amount, and then warehousing the data in batches, or the data are acquired in batches and directly warehousing the data in batches, so that the following defects exist:
(1) The data batch collection is not flexible enough, and the custom collection is complicated to process during reading, so that the complexity of the system is improved, and the reliability is reduced;
(2) The data real-time requirement is lacking, batch data is simply formed according to the quantity requirement, and once the data is relatively small, the batch data can wait for a long time. Leading to the previous data waiting for a long time to be written into the database;
(3) The lack of a multithreaded concurrent processing mechanism makes the data inefficient in processing.
Disclosure of Invention
The present application aims to solve, at least to some extent, one of the technical problems in the related art.
Therefore, a first object of the present application is to provide a method for writing data into a database in batches based on a memory queue, so as to solve the performance problem of a large-scale data source when continuously reporting data and warehousing the data.
A second object of the present application is to provide a device for batch writing of data into a database based on a memory queue.
A third object of the present application is to propose an electronic device.
A fourth object of the present application is to propose a computer readable storage medium.
To achieve the above objective, an embodiment of a first aspect of the present application provides a method for batch writing data into a database based on a memory queue, including:
acquiring a large amount of single data reported by a data source at high frequency, collecting the single data piece by piece, and then collecting the single data into a memory queue;
judging whether the data quantity cached in the memory queue reaches a quantity set value or not, and judging whether the difference value between the current moment and the last batch extraction moment reaches a set time or not;
and if the data quantity of the queue cache reaches a quantity set value or the difference between the current time and the last batch extraction time reaches a set time, batch extraction is carried out on the queue cache and batch writing is carried out on the queue cache into a database.
To achieve the above object, a second aspect of the present invention provides a device for writing data into a database in batch based on a memory queue, including
The data collection module is used for obtaining a large amount of single data reported by a data source at high frequency, collecting the single data piece by piece and then collecting the single data into the memory queue;
the judging module is used for judging whether the data quantity cached in the memory queue reaches a quantity set value or not and judging whether the difference value between the current moment and the last batch extraction moment reaches set time or not;
and the batch extracting and writing module is used for carrying out batch extraction on the queue cache and writing the data into the database in batches if the data quantity of the queue cache reaches a quantity set value or the difference between the current time and the last batch extracting time reaches a set time.
To achieve the above object, an embodiment of a third aspect of the present application provides an electronic device, including: a processor, and a memory communicatively coupled to the processor;
the memory stores computer-executable instructions;
the processor executes computer-executable instructions stored by the memory to implement the method of any one of the first aspects above.
To achieve the above object, an embodiment of a fourth aspect of the present application proposes a computer-readable storage medium having stored therein computer-executable instructions for implementing the method according to any of the above first aspects when being executed by a processor.
The technical scheme provided by the embodiment of the application at least brings the following beneficial effects:
based on the batch writing performance advantage of the database, for high-frequency data with small data packets, batch integration is carried out on piece by piece data before warehousing, the data is inserted piece by piece and converted into batch insertion, and data extraction is carried out based on a timing mechanism and a self-defining mechanism, so that the integration of two dimensions is realized, and the batch and real-time performance of data warehousing are ensured.
Additional aspects and advantages of the application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the application.
Drawings
The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:
FIG. 1 is a flow chart illustrating a method for batch writing of data to a database based on a memory queue according to an embodiment of the present application;
FIG. 2 is a flow chart illustrating another method for batch writing of data to a database based on a memory queue according to an embodiment of the present application;
FIG. 3 is a flow diagram illustrating another method for batch writing of data to a database based on a memory queue according to an embodiment of the present application;
FIG. 4 is a block diagram illustrating a memory queue based data bulk write database apparatus according to an embodiment of the present application;
fig. 5 is a block diagram of an electronic device.
Detailed Description
Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and intended for the purpose of explaining the present application and are not to be construed as limiting the present application.
A method and apparatus for batch writing of data into a database based on a memory queue according to embodiments of the present application are described below with reference to the accompanying drawings.
FIG. 1 is a flowchart of a method for batch writing of data to a database based on a memory queue according to an embodiment of the present application, as shown in FIG. 1, the method includes the steps of:
step 101, obtaining a large amount of single data reported by a data source at high frequency, collecting the single data piece by piece, and then collecting the single data to a memory queue.
As shown in fig. 2 and 3, for a large amount of single data reported at high frequency by a data source, the single data is collected piece by piece through a data collection program, and the single data is collected into a memory queue to form a specified amount of batch data.
Step 102, determining whether the number of data buffered in the queue in the memory queue reaches a set number value, and determining whether the difference between the current time and the last batch drawing time reaches a set time.
In order to solve the problem that the prior art lacks a multithreading concurrent processing mechanism and the efficiency of data is low in the processing process, the application provides two dimension mechanisms for respectively judging whether the data quantity of the queue cache reaches a quantity set value and judging whether the difference value between the current moment and the last batch extraction moment reaches a set time.
It is to be understood that the number setting value and the setting time are set according to the actual scenario, and the present application is not limited in particular herein.
And step 103, if the data quantity of the queue cache reaches a quantity set value or the difference between the current time and the last batch drawing time reaches a set time, batch drawing is carried out on the queue cache and batch writing is carried out on the queue cache into a database.
As shown in fig. 2 and fig. 3, the batch extraction process is performed from the memory queue as long as a condition of one dimension is satisfied, that is, when the number of data buffered in the queue reaches a set number value or when the difference between the current time and the last batch extraction time reaches a set time.
When the number of the data buffered in the queue reaches a set value or the difference between the current time and the last batch of data buffered in the queue reaches a set time, all the data buffered in the queue, i.e. how many data are buffered in the queue, and how many data are buffered in the queue are buffered.
It can be understood that the timeliness of data extraction can be realized through a timing mechanism, namely, when a specified amount of data cannot be collected in a short time, the extraction of batch data can be triggered after a set time, and the concurrent execution of a single data writing queue and a batch data reading queue can be realized through the ready of a number of threads and time, so that the operation efficiency of the data is improved.
According to the embodiment of the application, for the data with high frequency and small data packets, batch integration is carried out on the piece-by-piece data before warehousing, the data are inserted piece by piece and converted into batch insertion, and the data are extracted through a timing mechanism and a self-defining mechanism, so that the batch and real-time performance of data warehousing are ensured.
Fig. 4 is a block diagram of a memory queue based data batch writing database apparatus 10 according to an embodiment of the present application, including a data collection module 100, a determination module 200, and a batch extraction and writing module.
The data collection module 100 is configured to obtain a large amount of single data reported by the data source at high frequency, collect the single data piece by piece, and then collect the single data piece by piece into the memory queue;
the judging module 200 is configured to judge whether the number of data buffered in the queue in the memory queue reaches a set number value, and whether a difference between a current time and a last batch extraction time reaches a set time;
the batch extracting and writing module 300 is configured to perform batch extraction on the queue cache and batch writing the data into the database if the number of data cached in the queue reaches a set number value or if a difference between a current time and a last batch extraction time reaches a set time.
The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.
Fig. 5 shows a schematic block diagram of an example electronic device 700 that may be used to implement embodiments of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the application described and/or claimed herein.
As shown in fig. 5, the apparatus 700 includes a computing unit 701 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 702 or a computer program loaded from a storage unit 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 may also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
Various components in device 700 are connected to I/O interface 705, including: an input unit 706 such as a keyboard, a mouse, etc.; an output unit 707 such as various types of displays, speakers, and the like; a storage unit 708 such as a magnetic disk, an optical disk, or the like; and a communication unit 709 such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The computing unit 701 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 701 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the respective methods and processes described above, such as a voice instruction response method. For example, in some embodiments, the voice instruction response method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 700 via ROM 702 and/or communication unit 709. When the computer program is loaded into RAM 703 and executed by computing unit 701, one or more steps of the voice instruction response method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the voice instruction response method by any other suitable means (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present application may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this application, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), the internet, and blockchain networks.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service ("Virtual Private Server" or simply "VPS") are overcome. The server may also be a server of a distributed system or a server that incorporates a blockchain.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present application may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solutions of the present application are achieved, and the present application is not limited herein.
The above embodiments do not limit the scope of the application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application are intended to be included within the scope of the present application.
Claims (4)
1. A method for batch writing of data into a database based on a memory queue, comprising:
acquiring a large amount of single data reported by a data source at high frequency, collecting the single data piece by piece, and then collecting the single data into a memory queue;
judging whether the data quantity cached in the memory queue reaches a quantity set value or not, and judging whether the difference value between the current moment and the last batch extraction moment reaches a set time or not;
and if the data quantity of the queue cache reaches a quantity set value or the difference between the current time and the last batch extraction time reaches a set time, batch extraction is carried out on the queue cache and batch writing is carried out on the queue cache into a database.
2. A data batch writing database device based on a memory queue is characterized by comprising
The data collection module is used for obtaining a large amount of single data reported by a data source at high frequency, collecting the single data piece by piece and then collecting the single data into the memory queue;
the judging module is used for judging whether the data quantity cached in the memory queue reaches a quantity set value or not and judging whether the difference value between the current moment and the last batch extraction moment reaches set time or not;
and the batch extracting and writing module is used for carrying out batch extraction on the queue cache and writing the data into the database in batches if the data quantity of the queue cache reaches a quantity set value or the difference between the current time and the last batch extracting time reaches a set time.
3. An electronic device, comprising: a processor, and a memory communicatively coupled to the processor;
the memory stores computer-executable instructions;
the processor executes computer-executable instructions stored in the memory to implement the method of claim 1.
4. A computer readable storage medium having stored therein computer executable instructions which when executed by a processor are adapted to implement the method of claim 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311862320.3A CN117827949A (en) | 2023-12-29 | 2023-12-29 | Method and device for batch writing of data into database based on memory queue |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311862320.3A CN117827949A (en) | 2023-12-29 | 2023-12-29 | Method and device for batch writing of data into database based on memory queue |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117827949A true CN117827949A (en) | 2024-04-05 |
Family
ID=90512961
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311862320.3A Pending CN117827949A (en) | 2023-12-29 | 2023-12-29 | Method and device for batch writing of data into database based on memory queue |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117827949A (en) |
-
2023
- 2023-12-29 CN CN202311862320.3A patent/CN117827949A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112540806B (en) | Method and device for rendering small program page, electronic equipment and storage medium | |
| CN113778644B (en) | Task processing method, device, equipment and storage medium | |
| CN112883041B (en) | Data updating method and device, electronic equipment and storage medium | |
| CN117009283A (en) | Multi-core multi-chip data processing method, device, chip and storage medium | |
| CN116578502A (en) | Access request processing device, processing method, equipment and storage medium | |
| CN115103024B (en) | Sequence number generation method and device, electronic equipment and storage medium | |
| CN114564149B (en) | Data storage method, device, equipment and storage medium | |
| CN115904240A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN117827949A (en) | Method and device for batch writing of data into database based on memory queue | |
| CN115543416A (en) | Configuration updating method and device, electronic equipment and storage medium | |
| CN114553894B (en) | Data synchronization method, device, system and storage medium | |
| CN117271840B (en) | Data query method and device of graph database and electronic equipment | |
| CN116579914B (en) | Execution method and device of graphic processor engine, electronic equipment and storage medium | |
| CN116860826A (en) | Data processing method, device, equipment and medium based on time sequence database | |
| CN116107763B (en) | Data transmission method, device, equipment and storage medium | |
| CN116431561B (en) | Data synchronization method, device, equipment and medium based on heterogeneous many-core accelerator card | |
| US12026541B2 (en) | Method for applet page rendering, electronic device and storage medium | |
| CN117312095A (en) | Java virtual machine heap memory set object monitoring method and device | |
| CN117573331A (en) | Memory control method, device, equipment and medium | |
| CN116226251A (en) | Data export method and device, electronic equipment and storage medium | |
| CN116126249A (en) | Data reading method and device, electronic equipment and storage medium | |
| CN117806764A (en) | Data collection method, device, equipment and storage medium of user interface | |
| CN117156398A (en) | Message processing method, device, electronic equipment and storage medium | |
| CN118250175A (en) | Distributed computing implementation method and device, network card chip and medium | |
| CN116028228A (en) | Bitmap memory occupation detection method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |