CN112231288B - Log storage method, device and medium - Google Patents
Log storage method, device and medium Download PDFInfo
- Publication number
- CN112231288B CN112231288B CN202011062297.6A CN202011062297A CN112231288B CN 112231288 B CN112231288 B CN 112231288B CN 202011062297 A CN202011062297 A CN 202011062297A CN 112231288 B CN112231288 B CN 112231288B
- Authority
- CN
- China
- Prior art keywords
- storage file
- log
- class
- type
- condition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003860 storage Methods 0.000 title claims abstract description 371
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000010586 diagram Methods 0.000 description 12
- 238000012545 processing Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 7
- 238000007689 inspection Methods 0.000 description 6
- 238000004590 computer program Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 238000010926 purge Methods 0.000 description 4
- 230000010076 replication Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000001902 propagating effect Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000010200 validation analysis Methods 0.000 description 2
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000000739 chaotic effect Effects 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 229910052802 copper Inorganic materials 0.000 description 1
- 239000010949 copper Substances 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/113—Details of archiving
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Embodiments of the present disclosure relate to log storage methods and apparatuses, and computer-readable storage media. The method comprises the following steps: writing a log into a current first-class storage file; under the condition that the current first-type storage file is fully written, determining the next first-type storage file from a plurality of first-type storage files according to a preset first-type storage file circulation sequence; performing a preset operation, the preset operation comprising: checking whether the next first-class storage file meets an overlay condition or not; and under the condition that the next first-class storage file accords with the coverage condition, the log is subjected to coverage writing into the next first-class storage file.
Description
Technical Field
The present disclosure relates to the field of computer technology, and more particularly, to a log storage method and apparatus, and a medium.
Background
During the daily operation of a computer system, a log may be generated to record the operational status of the system. At present, a mode of continuously increasing storage files is mainly adopted to store logs, and then early storage files are cleaned regularly to release storage space. However, the addition of new storage files and the deletion of early storage files increases the number of I/O (Input/Output) times of the disk, which may cause a decrease in disk performance. Therefore, it is necessary to provide a new log storage scheme.
Disclosure of Invention
An object of the present disclosure is to provide a log storage method and apparatus, and a medium.
According to a first aspect of embodiments of the present disclosure, there is provided a log storage method, including
Writing a log into a current first-class storage file;
Under the condition that the current first-type storage file is fully written, determining the next first-type storage file from a plurality of first-type storage files according to a preset first-type storage file circulation sequence;
performing a preset operation, the preset operation comprising:
checking whether the next first-class storage file meets an overlay condition or not;
and under the condition that the next first-class storage file accords with the coverage condition, the log is subjected to coverage writing into the next first-class storage file.
Optionally, the preset operation further includes:
And if the next first-type storage file does not meet the coverage condition, creating a second-type storage file for writing into the log.
Optionally, the preset operation further includes:
Under the condition that the next first-class storage file does not accord with the coverage condition, obtaining the usage amount of the disk space;
If the usage amount of the disk space exceeds a preset threshold value and service data corresponding to the log in the next first-class storage file completely completes the disk-dropping operation, overwriting the log in the next first-class storage file;
If the usage amount of the disk space exceeds a preset threshold value and the service data corresponding to the log in the next first-class storage file does not completely complete the disk-dropping operation, creating a second-class storage file for writing the log;
if the disk space usage does not exceed the preset threshold, a second type of storage file is created for writing to the log.
Optionally, the method further comprises:
and executing the preset operation under the condition that the second type of storage files are fully written.
Optionally, the checking whether the next first-class storage file meets the coverage condition includes:
checking whether the service data corresponding to the log in the next first-class storage file completely completes the disc-dropping operation or not;
And under the condition that the service data corresponding to the log in the next first-class storage file completely completes the disc-dropping operation, confirming that the next first-class storage file meets the coverage condition.
Optionally, the checking whether the next first-class storage file meets the coverage condition includes:
checking whether the service data corresponding to the log in the next first-class storage file completely completes the disc-dropping operation or not;
checking whether the next first-class storage file has completed a master-slave copy operation;
And under the condition that the service data corresponding to the log in the next first-class storage file has all completed the disc-dropping operation and the next first-class storage file has completed the master-slave copy operation, confirming that the next first-class storage file meets the coverage condition.
Optionally, the method further comprises:
after the second type storage file is fully written, checking whether service data corresponding to the log in the second type storage file completely completes the disc-dropping operation;
and deleting the second-class storage file under the condition that the service data corresponding to the log in the second-class storage file has all completed the disc-dropping operation.
Optionally, the method further comprises:
Storing the writing sequence of the storage file in a management file, and reading a log from the storage file according to the writing sequence; the storage files comprise a first type storage file sequence and a second type storage file sequence.
According to a second aspect of embodiments of the present disclosure, there is provided a log storage device comprising:
The writing module is used for writing a log into the current first-class storage file;
the determining module is used for determining a next first-type storage file from a plurality of first-type storage files according to a preset first-type storage file circulation sequence under the condition that a current first-type storage file is fully written;
The execution module is used for executing preset operations, and the preset operations comprise: checking whether the next first-class storage file meets an overlay condition or not; and under the condition that the next first-class storage file accords with the coverage condition, the log is subjected to coverage writing into the next first-class storage file.
According to a third aspect of embodiments of the present disclosure, there is provided a log storage device comprising a processor and a memory storing computer readable instructions which, when executed by the processor, implement a method of any one of the preceding claims.
According to a fourth aspect of embodiments of the present disclosure, there is provided a computer readable storage medium having stored thereon computer readable instructions which, when executed by a processor, implement a method of any of the preceding claims.
According to the log storage method, the log storage device and the log storage medium, the log is written into the first type of storage files in an overlapping mode, and the first type of storage files do not need to be deleted, so that the I/O times of a disk can be reduced.
Other features of the present disclosure and its advantages will become apparent from the following detailed description of exemplary embodiments of the disclosure, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description, serve to explain the principles of the disclosure.
FIG. 1 is a schematic diagram of a log storage device that may be used to implement embodiments of the present disclosure.
Fig. 2 is a flowchart of a log storage method provided by an embodiment of the present disclosure.
FIG. 3 is a block diagram of a log storage device that may be used to implement embodiments of the present disclosure.
Detailed Description
Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless it is specifically stated otherwise.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses.
Techniques, methods, and apparatus known to one of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the techniques, methods, and apparatus should be considered part of the specification.
In all examples shown and discussed herein, any specific values should be construed as merely illustrative, and not a limitation. Thus, other examples of exemplary embodiments may have different values.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further discussion thereof is necessary in subsequent figures.
< Hardware configuration >
Fig. 1 illustrates a hardware configuration of a log storage device that may be used to implement embodiments of the present disclosure.
Referring to fig. 1, the log storage device 1000 includes a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, and an input device 1600.
The processor 1100 may be, for example, a central processing unit CPU, a micro control unit MCU, or the like. The memory 1200 includes, for example, ROM (read only memory), RAM (random access memory), nonvolatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, a USB interface, a serial interface, and the like. The communication device 1400 is, for example, a wired network card or a wireless network card. The display device 1500 is, for example, a liquid crystal display or a touch panel. The input device 1600 includes, for example, a touch screen, keyboard, mouse, microphone, etc.
In an embodiment applied to the present disclosure, the memory 1200 of the log storage 1000 is used to store instructions for controlling the processor 1100 to operate in support of implementing the methods of any embodiment of the present disclosure. The skilled artisan can design instructions in accordance with the disclosed aspects of the present disclosure. How the instructions control the processor to operate is well known in the art and will not be described in detail here.
It will be appreciated by those skilled in the art that although a plurality of devices of log storage device 1000 are shown in fig. 1, log storage device 1000 of embodiments of the present disclosure may refer to only some of the devices thereof, for example, only processor 1100, memory 1200, and communication device 1400.
In this embodiment, the log storage device 1000 is, for example, a log server of a database.
The hardware configuration shown in fig. 1 is merely illustrative and is in no way intended to limit the disclosure, its application, or uses.
< Journal storage method embodiment >
< First embodiment >
Referring to fig. 2, a log storage method provided by an embodiment of the present disclosure is described. The log storage method includes steps S102-S106.
S102, writing a log into a current first-class storage file.
In the embodiment of the disclosure, a plurality of first type storage files are created in advance, that is, a preset number of first type storage files are created in advance, the preset number is N, N is an integer, N is greater than or equal to 2, and the N first type storage files are W 1,W2,…,WN in sequence.
S104, determining a next first-type storage file from the plurality of first-type storage files according to a preset first-type storage file circulation sequence under the condition that the current first-type storage file is fully written.
In the embodiment of the disclosure, for the first type of storage file, the log is written in an overlay mode. That is, the existing log in the first storage file is not required to be cleared first and then written into the new log, but the new log is directly written into the first storage file to cover the existing log. This way, operations to purge the log may be reduced, thereby reducing the number of I/Os on the disk.
In the embodiment of the present disclosure, the cycle order of the first type storage files is from W 1 to W N, that is, the cycle order is ：W1,W2,…,WN,W1,W2,…,WN,W1,W2,…,WN,..... if the current first storage file is W i,Wi, according to the cycle order of the first type storage files, it may be determined that the next first type storage file is W i+1, where 1.ltoreq.i.ltoreq.n—1. If the current first type storage file is W N, after W N is full, the next first type storage file may be determined to be W 1 according to the first type storage file cycle order.
S106, executing a preset operation.
The preset operation includes steps S1062-S1064.
S1062, checking whether the next first-class storage file meets the coverage condition.
S1064, when the next first-class storage file accords with the coverage condition, the log is subjected to coverage writing into the next first-class storage file.
That is, after the current first-type storage file is fully written, if the next first-type storage file meets the coverage condition, the log starts to be written into the next first-type storage file in a coverage mode, and the next first-type storage file becomes the current first-type storage file.
In one example, the preset operation may further include step S1066.
S1066, if the next first-class storage file does not meet the coverage condition, creating a second-class storage file for writing into the log.
That is, after the current first-type storage file is filled, if the next first-type storage file does not meet the coverage condition, a second-type storage file is created, and the log starts to be written into the second-type storage file.
In one example, the preset operation is performed again in case the second type of storage file is full. That is, when the second-type storage file is full, it is checked again whether the next first-type storage file meets the overwrite condition.
And if the next first-class storage file is determined to meet the coverage condition through the second check, the log is subjected to the coverage writing in the next first-class storage file.
If the next first type storage file is determined to be out of coverage condition after the second check, a new second type storage file is created for writing the log, that is, the log is started to be written into the newly created second type storage file.
In the embodiment of the present disclosure, after the second type of storage file created at this time is written full, it is checked again whether the next first type of storage file covers a condition. If the next first-class storage file is found to meet the coverage condition after the second inspection, the log is written into the next first-class storage file in a coverage mode, and the next first-class storage file becomes the current first-class storage file. Otherwise, if the next first-class storage file is found to be out of coverage condition after the second inspection, a new second-class storage file is created for storing the log.
In one example, checking whether a first type of storage file meets the overlay condition may be: and checking whether the service data corresponding to the log in the first type of storage file has completed the disc-dropping operation.
The disk-dropping operation refers to refreshing data in a memory to a disk to persist the data. In the disclosed embodiments, a landing operation may be implemented by checkpointing (checkpoint).
For example, when a service data is written into the database, a redo log (redo log) is generated to record the relevant content of the writing operation, the service data is temporarily stored in the memory, and then the service data is refreshed into the disk through a disk-dropping operation. If downtime occurs before the disc-landing operation is completed, the redo log can be read for data recovery. Therefore, the redo log needs to be maintained before the landing operation of the service data is completed.
In the embodiment of the disclosure, the progress of the landing operation can be determined by reading the variable corresponding to the checkpointing operation to determine which logs correspond to the business data that have been landed. Or a management file related to the checkpointing operation can be used to record the progress of the checkpointing operation, and the content in the management file is read to determine which business data corresponding to the log has been dropped.
In this example, when the service data corresponding to each log in the first type storage file has completed the disc-dropping operation, the first type storage file meets the coverage condition. Otherwise, the first type storage file does not accord with the coverage condition, and the log in the first type storage file is reserved.
In the embodiment of the disclosure, the log storage method may be applied to a database system, that is, the log may be a log of the database system. For example, the log may be a redo log (redo log), an archive log (binlog log), a rollback log (undolog).
The database system can adopt a master-slave backup mode. In order to relieve the pressure of the main library, read-write separation can be performed on the application level of the system, the main library is moved by write operation, and the sub-library is moved by read operation, so that the access pressure of the database is dispersed, the performance and the availability of the whole system are improved, and the failure rate of database downtime caused by large access quantity is reduced. When unexpected situations such as network flashover occur, the master-slave replication can have problems such as faults and delay, so that the logs in the first-class storage files cannot be synchronized to the slave library in time, and in this case, the logs in the first-class storage files are reserved.
In one example, checking whether the first type of storage file meets the overlay condition may be: checking whether the business data corresponding to the log in the first type storage file has all completed the disc-dropping operation, and checking whether the first type storage file has completed the master-slave copy operation. The first type storage file completes the master-slave copy operation, which means that all the logs in the first type storage file are synchronized into the slave library.
In this example, if the service data corresponding to the log in the first type storage file has all completed the disc-dropping operation and the first type storage file completes the master-slave copy operation, the first type storage file meets the overlay condition. Otherwise, the first type of storage file does not conform to the coverage condition.
In one example, the method further includes the step of deleting the second type of storage file, including steps S502-S504.
S502, after the second type storage file is fully written, checking whether service data corresponding to the log in the second type storage file completely completes the disc-dropping operation.
S504, deleting the second-class storage file under the condition that the service data corresponding to the log in the second-class storage file has all completed the disc-dropping operation.
In one example, after the second type storage file is fully written, checking is performed at a preset period to check whether the service data corresponding to the log in the second type storage file has all completed the disc-dropping operation. And if the service data corresponding to the log in the second-class storage file has all completed the disc-dropping operation, deleting the second-class storage file.
That is, the second type of storage file is used as an extended storage space for a short period of time, as compared to the first type of storage file. And creating a second type storage file under the condition that the next first type storage file does not meet the coverage condition, and deleting the whole second type storage file to release the disk space when the service data corresponding to the log in the second type storage file completely completes the disk dropping operation after the second type storage file is fully written.
In one example, an end-of-log may be inserted at the end of a log when writing the log to a first type of storage file or a second type of storage file. For example, 5 spaces are inserted at the end of each log as an end symbol. When the log is required to be read later, the end of one log can be judged according to the ending symbol so as to completely read one log.
In an embodiment of the present disclosure, a plurality of management files are provided to record storage state information.
The first management file is used for recording the data length of the dropped service data corresponding to each storage file, and the format of the first management file can be 'file name-data length'. The unit of data length may be bytes. For example: after the previous first type storage file is written up for W 3,W3, it needs to check whether the service data corresponding to the log in the next first type storage file W 4 has all completed the disc-dropping operation. Assuming that the size of the service data contained in the log stored in W 4 is 1G (equal to 1073741824 bytes), at this time, "W 4, 56741824" is read from the first management file, that is, the length of the dropped service data corresponding to W 4 is 56741824 bytes, which means that the dropped service data corresponding to the log in W 4 is not completely dropped.
By reading the content in the first management file, the landing progress of the service data corresponding to the log in any one storage file can be obtained, so that whether the landing operation of the service data corresponding to the log in the storage file is completed or not is determined.
The second management file is used for recording the data length of the log which has undergone master-slave copying in each storage file, and the format of the second management file can be 'file name-data length'. The unit of data length may be bytes. For example: after the previous first type storage file is written to W 3,W3, it is necessary to check whether the next first type storage file W 4 completes the master-slave copy operation. Assuming that the size of W 4 is 2G (equal to 214748648 bytes), at this time, "W 4, 214748648 bytes" are read from the second management file, that is, the data length of the log indicating that W 4 has performed master-slave replication is 214748648 bytes, it is known that W 4 has completed master-slave replication.
By reading the content in the second management file, the master-slave replication progress of any one of the storage files can be determined.
The third management file is used to record the storage file currently used for writing the log, and may be in the format of "file name". For example, when the current storage file for writing the log is W 3 and the content in the second management file is "W 3".W3 is full, if it is checked that the next first storage file W 4 meets the overlay condition, writing the log to W 4 is started and the content in the second management file is updated to" W 4 ". If the next first-class storage file W 4 is found to be non-compliant with the overwrite condition, a second-class storage file Q 1 is created, writing of the log to Q 1 is started, and the contents in the second management file are updated to "Q 1".
By reading the content in the third management file, the storage file currently used for writing the log can be determined so as to be convenient for management of the storage file.
The fourth management file is used for recording the writing sequence of the storage files, and the storage files comprise a first storage file type and a second storage file type. Assuming N is 5, the first type of storage file includes W 1,W2,W3,W4,W5, and the second type of storage file currently existing in the first type of storage file cycle order ：W1,W2,W3,W4,W5,W1,W2,W3,W4,W5 W1,W2,W3,W4,W5..... includes Q 1 and Q 2. The fourth management file record has a content of "… W 5,W1,W2,W3,Q1,W4,Q2", which indicates that at the current time, the log in W 5 is the earliest for each storage file, and then W 1,W2,W3,Q1,W4,Q2 in turn.
When the log is read, the operation can be performed according to the record of the fourth management file, namely, the log can be read from the storage file according to the writing sequence, so that the log is read and copied according to the generation time sequence of the log, the chaotic conditions such as log loss and the like are avoided, and the subsequent log analysis is facilitated.
< Second embodiment >
The log storage method provided by the embodiment of the disclosure comprises steps S202-S206.
S202, writing a log into a current first-class storage file.
In the embodiment of the disclosure, a plurality of first type storage files are created in advance, that is, a preset number of first type storage files are created in advance, the preset number is N, N is an integer, N is greater than or equal to 2, and the N first type storage files are W 1,W2,…,WN in sequence.
S204, under the condition that the current first-type storage file is fully written, determining the next first-type storage file from the plurality of first-type storage files according to a preset first-type storage file circulation sequence.
In the embodiment of the disclosure, for the first type of storage file, the log is written in an overlay mode. That is, the existing log in the first storage file is not required to be cleared first and then written into the new log, but the new log is directly written into the first storage file to cover the existing log. This way, operations to purge the log may be reduced, thereby reducing the number of I/Os on the disk.
In the embodiment of the present disclosure, the cycle order of the first type storage files is from W 1 to W N, that is, the cycle order is ：W1,W2,…,WN,W1,W2,…,WN,W1,W2,…,WN,..... if the current first storage file is W i,Wi, according to the cycle order of the first type storage files, it may be determined that the next first type storage file is W i+1, where 1.ltoreq.i.ltoreq.n—1. If the current first type storage file is W N, after W N is full, the next first type storage file may be determined to be W 1 according to the first type storage file cycle order.
S206, executing a preset operation.
The preset operation includes steps S2062 to S2064.
S2062, checking whether the next first-type storage file meets the coverage condition.
And S2064, overwriting the log in the next first-type storage file under the condition that the next first-type storage file accords with the overwriting condition.
That is, after the current first-type storage file is fully written, if the next first-type storage file meets the coverage condition, the log starts to be written into the next first-type storage file in a coverage mode, and the next first-type storage file becomes the current first-type storage file.
In one example, the preset operation may further include steps S2066-S2068.
S2066, obtaining the disk space usage amount under the condition that the next first-class storage file does not meet the coverage condition.
S2068, if the disk space usage exceeds a preset threshold and the business data corresponding to the log in the next first-class storage file has completed the disk-dropping operation, overwriting the log in the next first-class storage file. If the usage amount of the disk space exceeds a preset threshold value and the service data corresponding to the log in the next first-class storage file does not completely complete the disk-dropping operation, a second-class storage file is created for writing the log. If the disk space usage does not exceed the preset threshold, a second type of storage file is created for writing to the log.
In one example, the preset operation is performed again in case the second type of storage file is full. That is, when the second-type storage file is full, it is checked again whether the next first-type storage file meets the overwrite condition.
And if the next first-class storage file is determined to meet the coverage condition through the second check, the log is subjected to the coverage writing in the next first-class storage file.
And if the next first-class storage file is determined to be not in accordance with the coverage condition after the second check, acquiring the usage amount of the disk space. And if the usage amount of the disk space exceeds a preset threshold value and the service data corresponding to the log in the next first-class storage file completely completes the disk-dropping operation, overwriting the log in the next first-class storage file. If the usage amount of the disk space exceeds a preset threshold value and the service data corresponding to the log in the next first-class storage file does not completely complete the disk-dropping operation, a second-class storage file is created for writing the log. If the disk space usage does not exceed the preset threshold, a second type of storage file is created for writing to the log.
It can be seen that the second embodiment differs from the first embodiment in that: when the next first-class storage file is detected to be out of compliance with the coverage condition, if the current first-class storage file is detected to be out of compliance with the coverage condition, and service data corresponding to the log in the next first-class storage file is completely subjected to the disc-dropping operation, the log can be written into the next first-class storage file in a coverage mode instead of creating a new second-class storage file, so that the influence of the overhigh usage of the disc space on the disc reading speed is prevented.
The threshold may be set according to the size of the disk space and the availability, which is not limited by the embodiments of the present disclosure.
According to the log storage method provided by the embodiment of the disclosure, for the first type of storage files, the log is written in an overlay mode. That is, the existing log in the first type of storage file does not need to be cleared first, and then a new log is written, but the new log is directly written to cover the existing log. This way, operations to purge the log may be reduced, thereby reducing the number of I/Os on the disk.
According to the log storage method provided by the embodiment of the disclosure, the first type storage files can be recycled, the first type storage files do not need to be deleted, and the I/O times of the disk can be reduced.
According to the log storage method provided by the embodiment of the disclosure, the second type of storage files can be created according to the requirements, so that the storage space is conveniently expanded.
< Log storage device >
Referring to fig. 3, an embodiment of the present disclosure further provides a log storage device. The log storage device 10 includes:
A writing module 11, configured to write a log to a current first type storage file.
The determining module 12 is configured to determine, when a previous first type storage file is full, a next first type storage file from the plurality of first type storage files according to a preset first type storage file cycle order.
A first execution module 13, configured to execute a preset operation. The preset operation includes: checking whether the next first-class storage file meets an overlay condition or not; and under the condition that the next first-class storage file accords with the coverage condition, the log is subjected to coverage writing into the next first-class storage file.
In one example, the preset operation further includes: and if the next first-type storage file does not meet the coverage condition, creating a second-type storage file for writing into the log.
In one example, the preset operation further includes: under the condition that the next first-class storage file does not accord with the coverage condition, obtaining the usage amount of the disk space; if the usage amount of the disk space exceeds a preset threshold value and service data corresponding to the log in the next first-class storage file completely completes the disk-dropping operation, overwriting the log in the next first-class storage file; if the usage amount of the disk space exceeds a preset threshold value and the service data corresponding to the log in the next first-class storage file does not completely complete the disk-dropping operation, creating a second-class storage file for writing the log; if the disk space usage does not exceed the preset threshold, a second type of storage file is created for writing to the log.
In one example, the log storage device further includes a second execution module.
The second execution module is configured to execute the preset operation when the second type of storage file is full.
In one example, the first execution module 13 includes a first inspection module and a first validation module.
And the first checking module is used for checking whether the service data corresponding to the log in the next first-type storage file completely completes the disc-dropping operation.
The first confirming module is configured to confirm that the next first type storage file meets the coverage condition when the service data corresponding to the log in the next first type storage file has all completed the disc-dropping operation.
In one example, the first execution module 13 includes a first inspection module, a second inspection module, and a second validation module.
And the first checking module is used for checking whether the service data corresponding to the log in the next first-type storage file completely completes the disc-dropping operation.
The first checking module is configured to check whether the next first-class storage file has completed a master-slave copy operation.
The second confirmation module is configured to confirm that the next first type storage file meets an overlay condition when the service data corresponding to the log in the next first type storage file has all completed the disc-dropping operation and the next first type storage file has completed the master-slave copy operation.
In one example, the log store includes a third inspection module and a delete module.
The third checking module is configured to check whether the service data corresponding to the log in the second type storage file has completely completed the disc-dropping operation after the second type storage file is full;
The deleting module is configured to delete the second type storage file when the service data corresponding to the log in the second type storage file has all completed the disc-dropping operation.
In one example, the log storage device further comprises a logging module.
The recording module is used for storing the writing sequence of the storage file in the management file and reading the log from the storage file according to the writing sequence; the storage files comprise a first type storage file sequence and a second type storage file sequence.
The log storage device 10 is, for example, a log server of a database.
Embodiments of the present disclosure also provide a log storage device comprising a processor and a memory storing computer readable instructions that, when executed by the processor, implement the method disclosed in any of the previous embodiments.
The log storage means is for example a log server of a database.
According to the log storage device provided by the embodiment of the disclosure, for the first type of storage files, the log is written in an overlay mode. That is, the existing log in the first type of storage file does not need to be cleared first, and then a new log is written, but the new log is directly written to cover the existing log. This way, operations to purge the log may be reduced, thereby reducing the number of I/Os on the disk.
The log storage device provided by the embodiment of the disclosure can recycle the first-class storage files without deleting the first-class storage files, and can reduce the I/O times of the disk.
The log storage device provided by the embodiment of the disclosure can create the second type of storage files according to the requirements, and is convenient for expanding the storage space.
< Computer-readable storage Medium >
Embodiments of the present disclosure also provide a computer readable storage medium having stored thereon computer readable instructions which, when executed by a processor, implement the method disclosed in any of the previous embodiments.
The present disclosure may be a method, apparatus, device, system, and/or computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions embodied thereon for causing a processor to implement aspects of the present disclosure.
The computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: portable computer disks, hard disks, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static Random Access Memory (SRAM), portable compact disk read-only memory (CD-ROM), digital Versatile Disks (DVD), memory sticks, floppy disks, mechanical coding devices, punch cards or in-groove structures such as punch cards or grooves having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, are not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted through wires.
The computer readable program instructions described herein may be downloaded from a computer readable storage medium to a respective computing/processing device or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network interface card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing/processing device.
The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as SMALLTALK, C ++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present disclosure are implemented by personalizing electronic circuitry, such as programmable logic circuitry, field Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), with state information of computer readable program instructions, which can execute the computer readable program instructions.
Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable medium having the instructions stored therein includes an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.
The foregoing description of the embodiments of the present disclosure has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the various embodiments described. The terminology used herein was chosen in order to best explain the principles of the embodiments, the practical application, or the technical improvements in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present disclosure is defined by the appended claims.
Claims (9)
1. A log storage method, comprising:
writing a log into a current first-class storage file, wherein the log is a log of a database system;
Under the condition that the current first-type storage file is fully written, determining the next first-type storage file from a plurality of first-type storage files according to a preset first-type storage file circulation sequence;
performing a preset operation, the preset operation comprising:
checking whether the next first-class storage file meets an overlay condition or not;
In case that the next first-class storage file meets the coverage condition, the log is covered and written into the next first-class storage file,
Wherein the checking whether the next first-class storage file meets the coverage condition includes:
checking whether the service data corresponding to the log in the next first-class storage file completely completes the disc-dropping operation or not;
checking whether the next first-class storage file has completed a master-slave copy operation;
And under the condition that the service data corresponding to the log in the next first-class storage file has all completed the disc-dropping operation and the next first-class storage file has completed the master-slave copy operation, confirming that the next first-class storage file meets the coverage condition.
2. The method of claim 1, wherein the preset operation further comprises:
And if the next first-type storage file does not meet the coverage condition, creating a second-type storage file for writing into the log.
3. The method of claim 1, wherein the preset operation further comprises:
Under the condition that the next first-class storage file does not accord with the coverage condition, obtaining the usage amount of the disk space;
If the usage amount of the disk space exceeds a preset threshold value and service data corresponding to the log in the next first-class storage file completely completes the disk-dropping operation, overwriting the log in the next first-class storage file;
If the usage amount of the disk space exceeds a preset threshold value and the service data corresponding to the log in the next first-class storage file does not completely complete the disk-dropping operation, creating a second-class storage file for writing the log;
if the disk space usage does not exceed the preset threshold, a second type of storage file is created for writing to the log.
4. A method according to claim 2 or 3, characterized in that the method further comprises:
and executing the preset operation under the condition that the second type of storage files are fully written.
5. A method according to claim 2 or 3, characterized in that the method further comprises:
after the second type storage file is fully written, checking whether service data corresponding to the log in the second type storage file completely completes the disc-dropping operation;
and deleting the second-class storage file under the condition that the service data corresponding to the log in the second-class storage file has all completed the disc-dropping operation.
6. A method according to claim 2 or 3, characterized in that the method further comprises:
Storing the writing sequence of the storage file in a management file, and reading a log from the storage file according to the writing sequence; the storage files comprise a first type storage file sequence and a second type storage file sequence.
7. A log storage device, comprising:
the writing module is used for writing a log into a current first-class storage file, wherein the log is a log of a database system;
the determining module is used for determining a next first-type storage file from a plurality of first-type storage files according to a preset first-type storage file circulation sequence under the condition that a current first-type storage file is fully written;
The execution module is used for executing preset operations, and the preset operations comprise: checking whether the next first-class storage file meets an overlay condition or not; under the condition that the next first-class storage file accords with an overlay condition, overwriting a log into the next first-class storage file;
the first checking module is used for checking whether the service data corresponding to the log in the next first-class storage file completely completes the disc-dropping operation or not; checking whether the next first-class storage file has completed master-slave copy operation;
And the second confirming module is used for confirming that the next first-type storage file accords with the coverage condition under the condition that the service data corresponding to the log in the next first-type storage file has all completed the disc-dropping operation and the next first-type storage file has completed the master-slave copy operation.
8. A log storage device comprising a processor and a memory, the memory storing computer readable instructions which, when executed by the processor, implement the method of any of claims 1-6.
9. A computer readable storage medium having stored thereon computer readable instructions which, when executed by a processor, implement the method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011062297.6A CN112231288B (en) | 2020-09-30 | 2020-09-30 | Log storage method, device and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011062297.6A CN112231288B (en) | 2020-09-30 | 2020-09-30 | Log storage method, device and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112231288A CN112231288A (en) | 2021-01-15 |
| CN112231288B true CN112231288B (en) | 2024-07-02 |
Family
ID=74119948
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011062297.6A Active CN112231288B (en) | 2020-09-30 | 2020-09-30 | Log storage method, device and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112231288B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113849126A (en) * | 2021-09-01 | 2021-12-28 | 北京腾凌科技有限公司 | Log storage method and device |
| CN115129680B (en) * | 2022-06-29 | 2024-01-16 | 苏州浪潮智能科技有限公司 | Method, device, equipment and readable medium for improving database performance |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111581171A (en) * | 2020-04-30 | 2020-08-25 | 中国工商银行股份有限公司 | Log processing method and device, computing equipment and medium |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7953937B2 (en) * | 2005-09-30 | 2011-05-31 | Cleversafe, Inc. | Systems, methods, and apparatus for subdividing data for storage in a dispersed data storage grid |
| CN103927251B (en) * | 2014-03-18 | 2017-02-01 | 烽火通信科技股份有限公司 | Log management method for embedded system |
| US10193696B2 (en) * | 2015-06-02 | 2019-01-29 | ALTR Solutions, Inc. | Using a tree structure to segment and distribute records across one or more decentralized, acylic graphs of cryptographic hash pointers |
| US10613941B1 (en) * | 2015-09-30 | 2020-04-07 | EMC IP Holding Company LLC | Hybrid NVRAM logging in filesystem namespace |
| CN106993147B (en) * | 2017-03-21 | 2019-08-23 | 深圳英飞拓科技股份有限公司 | A kind of video recording covering method, device and network hard disk video recorder |
| CN108399186A (en) * | 2018-01-12 | 2018-08-14 | 联动优势科技有限公司 | A kind of collecting method and device |
| CN110569177B (en) * | 2019-09-26 | 2022-08-19 | 苏州浪潮智能科技有限公司 | Method and device for testing log coverage mechanism |
| CN111026728A (en) * | 2019-11-19 | 2020-04-17 | 山东中创软件商用中间件股份有限公司 | Log data processing method and related device |
| CN111339044A (en) * | 2020-02-20 | 2020-06-26 | 苏州浪潮智能科技有限公司 | Distributed file system log submitting method, device, equipment and storage medium |
-
2020
- 2020-09-30 CN CN202011062297.6A patent/CN112231288B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111581171A (en) * | 2020-04-30 | 2020-08-25 | 中国工商银行股份有限公司 | Log processing method and device, computing equipment and medium |
Non-Patent Citations (2)
| Title |
|---|
| python用logging模块写循环日志;zebra_zzh;《CSDN博客》;第2-3页 * |
| zebra_zzh.python用logging模块写循环日志.《CSDN博客》.2020,第2-3页. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112231288A (en) | 2021-01-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| RU2646334C2 (en) | File management using placeholders | |
| JP5991699B2 (en) | Information processing apparatus, information processing system, backup method, and program | |
| US10705917B2 (en) | Consolidated full backup of a restored virtual machine | |
| CN107203574B (en) | Aggregation of data management and data analysis | |
| CN106844102B (en) | Data recovery method and device | |
| US20110093437A1 (en) | Method and system for generating a space-efficient snapshot or snapclone of logical disks | |
| CN112231288B (en) | Log storage method, device and medium | |
| US10394661B2 (en) | Policy driven data updates | |
| CN112948340A (en) | Data synchronization method and device, electronic equipment and readable storage medium | |
| US10656860B2 (en) | Tape drive library integrated memory deduplication | |
| US10795579B2 (en) | Methods, apparatuses, system and computer program products for reclaiming storage units | |
| EP3474143B1 (en) | Method and apparatus for incremental recovery of data | |
| CN111125021B (en) | Method and system for efficiently restoring consistent views of file system images from an asynchronous remote system | |
| CN110119386B (en) | Data processing method, data processing apparatus, medium, and computing device | |
| CN110109891B (en) | Method, apparatus and storage medium for data migration | |
| CN110674084A (en) | Method, apparatus, and computer-readable storage medium for data protection | |
| US10877992B2 (en) | Updating a database | |
| CN111488117B (en) | Method, electronic device, and computer-readable medium for managing metadata | |
| US10248335B2 (en) | Reducing a size of backup data in storage | |
| CN111104787B (en) | Method, apparatus and computer program product for comparing files | |
| US20210373881A1 (en) | Memory efficient software patching for updating applications on computing devices | |
| US9733922B2 (en) | Smarter operating systems: file system events | |
| CN111858145B (en) | Method, apparatus and computer program product for data backup and restore | |
| US11335364B1 (en) | Secure data erase for tape storage | |
| JP5585565B2 (en) | File management apparatus, file management apparatus control method, and program thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |