CN112231288A - Log storage method and device and medium - Google Patents
Log storage method and device and medium Download PDFInfo
- Publication number
- CN112231288A CN112231288A CN202011062297.6A CN202011062297A CN112231288A CN 112231288 A CN112231288 A CN 112231288A CN 202011062297 A CN202011062297 A CN 202011062297A CN 112231288 A CN112231288 A CN 112231288A
- Authority
- CN
- China
- Prior art keywords
- storage file
- log
- type storage
- type
- condition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/113—Details of archiving
Abstract
Embodiments of the present disclosure relate to a log storage method and apparatus, and a computer-readable storage medium. The method comprises the following steps: writing a log into a current first type storage file; determining a next first type storage file from the plurality of first type storage files according to a preset first type storage file circulation sequence under the condition that the current first type storage file is fully written; executing preset operation, wherein the preset operation comprises the following steps: checking whether the next first type storage file meets a coverage condition; and under the condition that the next first-type storage file meets the coverage condition, writing the log into the next first-type storage file in a coverage mode.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a log storage method, apparatus, and medium.
Background
During the daily operation of a computer system, a log is generated to record the operating state of the system. At present, logs are mainly stored in a mode of continuously increasing storage files, and then early storage files are cleaned regularly to release storage space. However, adding storage files and deleting earlier storage files increase the number of I/O (Input/Output) times of the disk, which may result in a decrease in disk performance. Therefore, there is a need to provide a new log storage scheme.
Disclosure of Invention
An object of the present disclosure is to provide a log storage method and apparatus, and a medium.
According to a first aspect of embodiments of the present disclosure, there is provided a log storage method, including
Writing a log into a current first type storage file;
determining a next first type storage file from the plurality of first type storage files according to a preset first type storage file circulation sequence under the condition that the current first type storage file is fully written;
executing preset operation, wherein the preset operation comprises the following steps:
checking whether the next first type storage file meets a coverage condition;
and under the condition that the next first-type storage file meets the coverage condition, writing the log into the next first-type storage file in a coverage mode.
Optionally, the preset operation further includes:
and creating a second type storage file for writing into the log under the condition that the next first type storage file does not meet the coverage condition.
Optionally, the preset operation further includes:
acquiring the space usage amount of the disk under the condition that the next first-type storage file does not meet the coverage condition;
if the disk space usage exceeds a preset threshold and the service data corresponding to the log in the next first type storage file has all completed the disk-dropping operation, overwriting the log into the next first type storage file;
if the disk space usage exceeds a preset threshold and the service data corresponding to the log in the next first type storage file does not completely complete the disk-dropping operation, creating a second type storage file for writing the log;
and if the disk space usage does not exceed a preset threshold, creating a second type storage file for writing in the log.
Optionally, the method further comprises:
and executing the preset operation under the condition that the second type storage file is fully written.
Optionally, the checking whether the next first type storage file meets the coverage condition includes:
checking whether the service data corresponding to the log in the next first-class storage file completely completes the disk-dropping operation or not;
and under the condition that the service data corresponding to the log in the next first-class storage file completely completes the disk-dropping operation, confirming that the next first-class storage file meets the coverage condition.
Optionally, the checking whether the next first type storage file meets the coverage condition includes:
checking whether the service data corresponding to the log in the next first-class storage file completely completes the disk-dropping operation or not;
checking whether the next first-type storage file completes master-slave copy operation;
and under the condition that the service data corresponding to the log in the next first-class storage file has completely finished the disk-dropping operation and the next first-class storage file has finished the master-slave copying operation, confirming that the next first-class storage file meets the coverage condition.
Optionally, the method further comprises:
after the second type storage file is fully written, checking whether the service data corresponding to the logs in the second type storage file completely complete the disk-dropping operation;
and deleting the second type storage file under the condition that the service data corresponding to the log in the second type storage file completely completes the disk-dropping operation.
Optionally, the method further comprises:
storing the writing sequence of the storage files in a management file, and reading logs from the storage files according to the writing sequence; the storage files comprise a first type of storage files and a second type of storage files.
According to a second aspect of embodiments of the present disclosure, there is provided a log storage device including:
the writing module is used for writing a log into a current first type storage file;
the determining module is used for determining the next first type storage file from the plurality of first type storage files according to the preset first type storage file circulation sequence under the condition that the current first type storage file is fully written;
an execution module, configured to execute a preset operation, where the preset operation includes: checking whether the next first type storage file meets a coverage condition; and under the condition that the next first-type storage file meets the coverage condition, writing the log into the next first-type storage file in a coverage mode.
According to a third aspect of embodiments of the present disclosure, there is provided a log storage device comprising a processor and a memory, the memory storing computer readable instructions which, when executed by the processor, implement the method of any one of the preceding claims.
According to a fourth aspect of embodiments of the present disclosure, there is provided a computer readable storage medium having stored thereon computer readable instructions which, when executed by a processor, implement the method of any one of the preceding claims.
According to the log storage method, device and medium, the log is written into the first type of storage file in a covering mode, the first type of storage file does not need to be deleted, and therefore the I/O frequency of a disk can be reduced.
Other features of the present disclosure and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the disclosure and together with the description, serve to explain the principles of the disclosure.
FIG. 1 is a schematic diagram of a log storage that can be used to implement an embodiment of the present disclosure.
Fig. 2 is a flowchart of a log storage method provided by an embodiment of the present disclosure.
FIG. 3 is a block diagram of a log storage device that can be used to implement embodiments of the present disclosure.
Detailed Description
Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, the numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
< hardware configuration >
Fig. 1 illustrates a hardware configuration of a log storage apparatus that can be used to implement an embodiment of the present disclosure.
Referring to fig. 1, the log storage apparatus 1000 includes a processor 1100, a memory 1200, an interface apparatus 1300, a communication apparatus 1400, a display apparatus 1500, and an input apparatus 1600.
The processor 1100 may be, for example, a central processing unit CPU, a micro control unit MCU, or the like. The memory 1200 includes, for example, a ROM (read only memory), a RAM (random access memory), a nonvolatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, a USB interface, a serial interface, and the like. The communication device 1400 is, for example, a wired network card or a wireless network card. The display device 1500 is, for example, a liquid crystal display panel or a touch panel. The input device 1600 includes, for example, a touch screen, a keyboard, a mouse, a microphone, and the like.
Memory 1200 of log storage 1000 is used to store instructions that control processor 1100 to operate in support of methods that implement any of the embodiments of the present disclosure, as applied to embodiments of the present disclosure. The skilled person can design the instructions according to the disclosed solution of the present disclosure. How the instructions control the operation of the processor is well known in the art and will not be described in detail herein.
It should be understood by those skilled in the art that although a plurality of devices of the log storage device 1000 are illustrated in fig. 1, the log storage device 1000 of the embodiment of the present disclosure may refer to only some of the devices, for example, only the processor 1100, the memory 1200 and the communication device 1400.
In the present embodiment, the log storage device 1000 is, for example, a log server of a database.
The hardware configuration shown in fig. 1 is merely illustrative and is in no way intended to limit the present disclosure, its application, or uses.
< journal storage method embodiment >
< first embodiment >
Referring to fig. 2, a method for storing a log provided by an embodiment of the present disclosure is described. The log storing method includes steps S102-S106.
And S102, writing a log into the current first-type storage file.
In the embodiment of the present disclosure, a plurality of first-type storage files are created in advance, that is, a preset number of first-type storage files are created in advance, where the preset number is N, where N is an integer and N is greater than or equal to 2, and the N first-type storage files are sequentially W1,W2,…,WN。
S104, under the condition that the current first-type storage file is fully written, determining the next first-type storage file from the plurality of first-type storage files according to the preset first-type storage file circulation sequence.
In the embodiment of the disclosure, for the first type of storage files, the logs are written in an overlay mode. That is, the existing log in the first type storage file does not need to be cleared first and then the new log is written, but the new log is directly written to overwrite the existing log. This approach may reduce the operations to flush the log, thereby reducing the number of I/O times to the disk.
In the embodiment of the disclosure, the first type of storage files are circularly ordered from W1To WNThe cycle of (a), i.e. the cycle sequence, is: w1,W2,…,WN,W1,W2,…,WN,W1,W2,…,WN,..... If the current first storage file is Wi，WiAfter the write is full, according to the circulation sequence of the first type storage files, the next first type storage file can be determined to be Wi+1I is more than or equal to 1 and less than or equal to N < -1 >. If the current first type storage file is WNIn WNAfter the write is full, according to the circulation sequence of the first type storage files, the next first type storage file can be determined to be W1。
And S106, executing preset operation.
The preset operation includes steps S1062-S1064.
S1062, checking whether the next first-class storage file meets the coverage condition.
And S1064, under the condition that the next first-class storage file meets the coverage condition, overwriting the log into the next first-class storage file.
That is, after the current first-type storage file is fully written, if the next first-type storage file meets the overwriting condition, the log is written into the next first-type storage file in an overwriting manner, and the next first-type storage file becomes the current first-type storage file.
In one example, the preset operation may further include step S1066.
S1066, under the condition that the next first-class storage file does not meet the coverage condition, creating a second-class storage file for writing in the log.
That is, after the current first-type storage file is fully written, if the next first-type storage file does not meet the coverage condition, a second-type storage file is created, and the log is written into the second-type storage file.
In one example, the preset operation is performed again in case the second type storage file is fully written. That is, when the second type of storage file is fully written, it is checked again whether the next first type of storage file meets the overwrite condition.
And if the next first-type storage file is checked again, determining that the next first-type storage file meets the coverage condition, and writing the log coverage into the next first-type storage file.
If the next first type storage file is determined not to meet the coverage condition after the check again, a new second type storage file is created for writing the journal, namely, the journal is written into the newly created second type storage file.
In the embodiment of the present disclosure, after the second type storage file created this time is fully written, it is checked again whether the next first type storage file covers the condition. And if the next first-type storage file is checked again, the next first-type storage file is found to meet the coverage condition, and the log is written into the next first-type storage file in a coverage mode, wherein the next first-type storage file becomes the current first-type storage file. Otherwise, if the next first-type storage file is found to be not in accordance with the coverage condition after the check again, a new second-type storage file is created for storing the log.
In one example, the step of checking whether a first type of storage file meets the coverage condition may be: and checking whether the service data corresponding to the logs in the first-class storage file completely complete the disk-dropping operation.
The disk-dropping operation refers to refreshing data in the memory to a magnetic disk to persistently store the data. In the embodiments of the present disclosure, the disk-down operation may be implemented by a checkpoint operation (checkpoint).
For example, when writing a service data into a database, a redo log (redo log) is generated to record the relevant content of the write operation, the service data is temporarily stored in a memory, and then the service data is refreshed into a magnetic disk through a disk-down operation. If the crash occurs before the completion of the disk-dropping operation, the redo log can be read for data recovery. Therefore, the redo log needs to be retained before the completion of the disk-down operation of the business data.
In the embodiment of the disclosure, the progress of the disk-dropping operation can be determined by reading the variable corresponding to the checkpoint operation to determine which logs correspond to the business data that have been dropped. Alternatively, a management file related to the checkpointing can be used to record the progress of the checkpointing, and the contents in the management file are read to determine which logs correspond to the business data that has been landed.
In this example, when all the service data corresponding to each log in the first storage file has completed the disk-dropping operation, the first storage file meets the coverage condition. Otherwise, the first type storage file does not accord with the coverage condition, and the log in the first type storage file is reserved.
In the embodiment of the present disclosure, the log storage method may be applied to a database system, that is, the log may be a log of the database system. For example, the log may be a redo log (redo log), an archive log (binlog log), a rollback log (undo log).
The database system may employ a master-slave backup approach. In order to reduce the pressure of the main library, read-write separation can be performed on the system application level, write operation is performed to move the main library, and read operation is performed to move the slave library, so that the access pressure of the database is dispersed, the performance and the usability of the whole system are improved, and the fault rate of database downtime caused by large access is reduced. When network flash occurs and other accidents occur, the problems of failure, delay and the like can occur in master-slave copy, so that the logs in the first type of storage files cannot be synchronized to the slave library in time, and in this case, the logs in the first type of storage files are reserved.
In one example, the checking whether the first type of storage file meets the coverage condition may be: and checking whether the service data corresponding to the logs in the first type of storage file completely complete the disk-dropping operation, and checking whether the first type of storage file completes the master-slave copy operation. The first type storage file completes master-slave copy operation, which means that the logs in the first type storage file are synchronized to the slave library.
In this example, if all the service data corresponding to the logs in the first type storage file have completed the disk-dropping operation and the first type storage file completes the master-slave copy operation, the first type storage file meets the coverage condition. Otherwise, the first type storage file does not accord with the coverage condition.
In one example, the method further comprises the step of deleting the second type of storage file, including steps S502-S504.
S502, after the second type storage file is fully written, whether the service data corresponding to the logs in the second type storage file completely complete the disk-dropping operation is checked.
S504, under the condition that the service data corresponding to the logs in the second type storage file are all subjected to the disk-dropping operation, deleting the second type storage file.
In one example, after the second type storage file is fully written, whether service data corresponding to logs in the second type storage file have all completed the disk-dropping operation is checked at preset cycle timing. And if the service data corresponding to the logs in the second type storage file completely complete the disk-dropping operation, deleting the second type storage file.
That is, the second type of storage file is used as an extended storage space of a short term as compared to the first type of storage file. And creating a second type storage file under the condition that the next first type storage file does not meet the coverage condition, and after the second type storage file is fully written, deleting the whole second type storage file when the service data corresponding to the log in the second type storage file completely complete the disk-dropping operation so as to release the disk space.
In one example, when writing a log to a first type of storage file or a second type of storage file, a terminator may be inserted at the end of the log. For example, 5 spaces are inserted at the end of each log as endings. When the logs need to be read subsequently, the end of one log can be judged according to the ending character so as to completely read one log.
In the disclosed embodiment, a plurality of management files are provided to record storage status information.
The first management file is used to record the data length of the dropped service data corresponding to each storage file, and may be in the format of "file name-data length". The unit of the data length may be bytes. For example: the current first type storage file is W3，W3After being fully written, the next first type storage file W needs to be checked4Whether the service data corresponding to the log in the log is completely finished with the disk-dropping operation or not. Suppose W4The size of the service data contained in the log stored in (1) is 1G (equal to 1073741824 bytes), and at this time, "W" is read from the first management file456741824 ", i.e. indicating a AND W4The length of the corresponding dropped service data is 56741824 bytes, and W is known4The service data corresponding to the log in the log is not all completed with the operation of dropping the disk.
By reading the content in the first management file, the tray falling progress of the service data corresponding to the log in any storage file can be known, so that whether the service data corresponding to the log in the storage file completely complete the tray falling operation is determined.
The second management file is used for recording the data length of the log which is subjected to master-slave copy in each storage file, and the format of the second management file can be 'file name-data length'. The unit of the data length may be bytes. For example: the current first type storage file is W3，W3After being fully written, the next first type storage file W needs to be checked4Whether the master-slave copy operation is complete. Suppose W4Is 2G (equal to 2147483648 bytes), at this time, the "W" is read from the second management file42147483648 bytes ", i.e. indicating W4The data length of the log subjected to master-slave copy is 2147483648 bytes, and W is known4Master-slave replication has been completed.
By reading the contents in the second management file, the master-slave copy progress of any one of the storage files can be determined.
The third management file is used for recording the storage file currently used for writing the log, and the format of the third management file can be 'file name'. For example, the current storage file for write logs is W3The content in the second management file is "W3”。W3After being fully written, if the check finds the next first type storage file W4If the coverage condition is met, starting to move to W4Writing into the log, updating the content in the second management file to' W4". If the check finds the next first type storage file W4If the coverage condition is not met, a second type storage file Q is created1Start to Q1Writing into the log, updating the content in the second management file to be' Q1”。
By reading the content in the third management file, the storage file currently used for writing the log can be determined so as to facilitate the management of the storage file.
The fourth management file is used for recording the writing sequence of the storage files, and the storage files comprise a first type of storage files and a second type of storage files. Assuming that N is 5, the first type of storage file includes W1,W2,W3,W4,W5The first type of storage file cycle sequence is: w1,W2,W3,W4,W5,W1,W2,W3,W4,W5 W1,W2,W3,W4,W5..... The second class of currently existing storage files includes Q1And Q2. The content of the fourth management file record is "… W5，W1，W2，W3，Q1，W4，Q2", illustrates that at the present time, for each stored file, W5Is the earliest of the logs in, then in turn is W1，W2，W3，Q1，W4，Q2。
When the log is read, the operation can be performed according to the record of the fourth management file, that is, the log can be read from the storage file according to the writing sequence, so that the log can be read and copied according to the generation time sequence of the log, the chaotic situations of log loss and the like are avoided, and the subsequent log analysis is facilitated.
< second embodiment >
The log storage method provided by the embodiment of the disclosure includes steps S202-S206.
S202, writing a log into the current first-type storage file.
In the embodiment of the present disclosure, a plurality of first-type storage files are created in advance, that is, a preset number of first-type storage files are created in advance, where the preset number is N, where N is an integer and N is greater than or equal to 2, and the N first-type storage files are sequentially W1,W2,…,WN。
S204, under the condition that the current first-class storage file is fully written, determining the next first-class storage file from the plurality of first-class storage files according to the preset first-class storage file circulation sequence.
In the embodiment of the disclosure, for the first type of storage files, the logs are written in an overlay mode. That is, the existing log in the first type storage file does not need to be cleared first and then the new log is written, but the new log is directly written to overwrite the existing log. This approach may reduce the operations to flush the log, thereby reducing the number of I/O times to the disk.
In the embodiment of the disclosure, the first type of storage files are circularly ordered from W1To WNThe cycle of (a), i.e. the cycle sequence, is: w1,W2,…,WN,W1,W2,…,WN,W1,W2,…,WN,..... If the current first storage file is Wi，WiAfter the write is full, according to the circulation sequence of the first type storage files, the next first type storage file can be determined to be Wi+1I is more than or equal to 1 and less than or equal to N < -1 >. If the current first type storage file is WNIn WNAfter the file is fully written, the next first type of storage can be determined according to the circulation sequence of the first type of storage filesStore the file as W1。
And S206, executing preset operation.
The preset operation includes steps S2062 to S2064.
S2062, checking whether the next first-type storage file meets the coverage condition.
S2064, under the condition that the next first-type storage file meets the covering condition, the log is written into the next first-type storage file in a covering mode.
That is, after the current first-type storage file is fully written, if the next first-type storage file meets the overwriting condition, the log is written into the next first-type storage file in an overwriting manner, and the next first-type storage file becomes the current first-type storage file.
In one example, the preset operation may further include steps S2066 to S2068.
S2066, under the condition that the next first-type storage file does not meet the covering condition, the disk space usage amount is obtained.
S2068, if the disk space usage exceeds a preset threshold and the service data corresponding to the log in the next first-class storage file completely completes the disk-dropping operation, overwriting the log in the next first-class storage file. And if the disk space usage exceeds a preset threshold and the service data corresponding to the log in the next first type storage file does not complete the disk-dropping operation, creating a second type storage file for writing the log. And if the disk space usage does not exceed a preset threshold, creating a second type storage file for writing in the log.
In one example, the preset operation is performed again in case the second type storage file is fully written. That is, when the second type of storage file is fully written, it is checked again whether the next first type of storage file meets the overwrite condition.
And if the next first-type storage file is checked again, determining that the next first-type storage file meets the coverage condition, and writing the log coverage into the next first-type storage file.
And if the next first-type storage file is determined to be not in accordance with the coverage condition after the check again, acquiring the disk space usage. And if the disk space usage exceeds a preset threshold and the service data corresponding to the log in the next first-class storage file completely completes the disk-dropping operation, overwriting the log into the next first-class storage file. And if the disk space usage exceeds a preset threshold and the service data corresponding to the log in the next first type storage file does not complete the disk-dropping operation, creating a second type storage file for writing the log. And if the disk space usage does not exceed a preset threshold, creating a second type storage file for writing in the log.
It can be seen that the second embodiment differs from the first embodiment in that: the method comprises the steps of presetting a threshold value of the usage amount of the disk space, and when the next first-class storage file is checked and found to be not in accordance with the coverage condition, if the usage amount of the disk space at the moment exceeds the preset threshold value and service data corresponding to a log in the next first-class storage file completely complete a disk-dropping operation, writing the log into the next first-class storage file in a coverage mode instead of creating a new second-class storage file so as to prevent the disk reading speed from being influenced by overhigh usage amount of the disk space.
The threshold may be set according to the size and availability of the disk space, which is not limited in this disclosure.
According to the log storage method provided by the embodiment of the disclosure, the log is written in the first type of storage file in a covering mode. That is, the existing log in the first type storage file does not need to be cleared first and then the new log is written, but the new log is directly written to overwrite the existing log. This approach may reduce the operations to flush the log, thereby reducing the number of I/O times to the disk.
The log storage method provided by the embodiment of the disclosure can recycle the first type of storage file without deleting the first type of storage file, and can reduce the I/O frequency of a disk.
According to the log storage method provided by the embodiment of the disclosure, the second type of storage file can be created according to requirements, and the storage space is conveniently expanded.
< journal storage means >
Referring to fig. 3, an embodiment of the present disclosure further provides a log storage device. The log storage device 10 includes:
and the writing module 11 is used for writing the log into the current first-type storage file.
The determining module 12 is configured to determine, when a current first-type storage file is fully written, a next first-type storage file from the plurality of first-type storage files according to a preset first-type storage file circulation sequence.
The first executing module 13 is configured to execute a preset operation. The preset operation comprises the following steps: checking whether the next first type storage file meets a coverage condition; and under the condition that the next first-type storage file meets the coverage condition, writing the log into the next first-type storage file in a coverage mode.
In one example, the presetting operation further includes: and creating a second type storage file for writing into the log under the condition that the next first type storage file does not meet the coverage condition.
In one example, the presetting operation further includes: acquiring the space usage amount of the disk under the condition that the next first-type storage file does not meet the coverage condition; if the disk space usage exceeds a preset threshold and the service data corresponding to the log in the next first type storage file has all completed the disk-dropping operation, overwriting the log into the next first type storage file; if the disk space usage exceeds a preset threshold and the service data corresponding to the log in the next first type storage file does not completely complete the disk-dropping operation, creating a second type storage file for writing the log; and if the disk space usage does not exceed a preset threshold, creating a second type storage file for writing in the log.
In one example, the log storage further comprises a second execution module.
The second execution module is configured to execute the preset operation when the second type of storage file is fully written.
In one example, the first execution module 13 includes a first checking module and a first confirming module.
The first checking module is configured to check whether all the service data corresponding to the log in the next first-class storage file has completed the disk-dropping operation.
The first confirming module is configured to confirm that the next first-class storage file meets the coverage condition when all the service data corresponding to the log in the next first-class storage file has completed the disk-dropping operation.
In one example, the first execution module 13 includes a first checking module, a second checking module, and a second confirming module.
The first checking module is configured to check whether all the service data corresponding to the log in the next first-class storage file has completed the disk-dropping operation.
The first checking module is configured to check whether the next first type storage file has completed a master-slave copy operation.
The second confirming module is configured to confirm that the next first-type storage file meets the coverage condition when all the service data corresponding to the log in the next first-type storage file has completed the disk-dropping operation and the next first-type storage file has completed the master-slave copy operation.
In one example, the log storage includes a third checking module and a deleting module.
The third checking module is configured to check whether all the service data corresponding to the logs in the second type storage file have completed a disk-dropping operation after the second type storage file is fully written;
and the deleting module is used for deleting the second type storage file under the condition that the service data corresponding to the log in the second type storage file completely completes the disk dropping operation.
In one example, the log storage device further comprises a logging module.
The recording module is used for storing the writing sequence of the storage file in a management file and reading the log from the storage file according to the writing sequence; the storage files comprise a first type of storage files and a second type of storage files.
The log storage device 10 is, for example, a log server of a database.
Embodiments of the present disclosure also provide a log storage device, which includes a processor and a memory, where the memory stores computer readable instructions, and the computer readable instructions, when executed by the processor, implement the method disclosed in any of the foregoing embodiments.
The log storage device is, for example, a log server of a database.
According to the log storage device provided by the embodiment of the disclosure, for the first type of storage file, a log is written in a covering mode. That is, the existing log in the first type storage file does not need to be cleared first and then the new log is written, but the new log is directly written to overwrite the existing log. This approach may reduce the operations to flush the log, thereby reducing the number of I/O times to the disk.
The log storage device provided by the embodiment of the disclosure can recycle the first type of storage file without deleting the first type of storage file, and can reduce the I/O frequency of a disk.
The log storage device provided by the embodiment of the disclosure can create the second type of storage file according to requirements, so that the storage space is conveniently expanded.
< computer-readable storage Medium >
Embodiments of the present disclosure also provide a computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by a processor, implement a method disclosed in any of the foregoing embodiments.
The present disclosure may be methods, apparatus, devices, systems, and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied thereon for causing a processor to implement various aspects of the present disclosure.
The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.
The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.
The computer program instructions for carrying out operations of the present disclosure may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, the electronic circuitry that can execute the computer-readable program instructions implements aspects of the present disclosure by utilizing the state information of the computer-readable program instructions to personalize the electronic circuitry, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA).
Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, by software, and by a combination of software and hardware are equivalent.
Having described embodiments of the present disclosure, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present disclosure is defined by the appended claims.
Claims (11)
1. A method of log storage, comprising:
writing a log into a current first type storage file;
determining a next first type storage file from the plurality of first type storage files according to a preset first type storage file circulation sequence under the condition that the current first type storage file is fully written;
executing preset operation, wherein the preset operation comprises the following steps:
checking whether the next first type storage file meets a coverage condition;
and under the condition that the next first-type storage file meets the coverage condition, writing the log into the next first-type storage file in a coverage mode.
2. The method of claim 1, wherein the pre-setting operation further comprises:
and creating a second type storage file for writing into the log under the condition that the next first type storage file does not meet the coverage condition.
3. The method of claim 1, wherein the pre-setting operation further comprises:
acquiring the space usage amount of the disk under the condition that the next first-type storage file does not meet the coverage condition;
if the disk space usage exceeds a preset threshold and the service data corresponding to the log in the next first type storage file has all completed the disk-dropping operation, overwriting the log into the next first type storage file;
if the disk space usage exceeds a preset threshold and the service data corresponding to the log in the next first type storage file does not completely complete the disk-dropping operation, creating a second type storage file for writing the log;
and if the disk space usage does not exceed a preset threshold, creating a second type storage file for writing in the log.
4. A method according to claim 2 or 3, characterized in that the method further comprises:
and executing the preset operation under the condition that the second type storage file is fully written.
5. The method according to any of claims 1-4, wherein said checking whether said next first type of storage file meets an override condition comprises:
checking whether the service data corresponding to the log in the next first-class storage file completely completes the disk-dropping operation or not;
and under the condition that the service data corresponding to the log in the next first-class storage file completely completes the disk-dropping operation, confirming that the next first-class storage file meets the coverage condition.
6. The method according to any of claims 1-4, wherein said checking whether said next first type of storage file meets an override condition comprises:
checking whether the service data corresponding to the log in the next first-class storage file completely completes the disk-dropping operation or not;
checking whether the next first-type storage file completes master-slave copy operation;
and under the condition that the service data corresponding to the log in the next first-class storage file has completely finished the disk-dropping operation and the next first-class storage file has finished the master-slave copying operation, confirming that the next first-class storage file meets the coverage condition.
7. The method according to any one of claims 2-6, further comprising:
after the second type storage file is fully written, checking whether the service data corresponding to the logs in the second type storage file completely complete the disk-dropping operation;
and deleting the second type storage file under the condition that the service data corresponding to the log in the second type storage file completely completes the disk-dropping operation.
8. A method according to claim 2 or 3, characterized in that the method further comprises:
storing the writing sequence of the storage files in a management file, and reading logs from the storage files according to the writing sequence; the storage files comprise a first type of storage files and a second type of storage files.
9. A log storage device, comprising:
the writing module is used for writing a log into a current first type storage file;
the determining module is used for determining the next first type storage file from the plurality of first type storage files according to the preset first type storage file circulation sequence under the condition that the current first type storage file is fully written;
an execution module, configured to execute a preset operation, where the preset operation includes: checking whether the next first type storage file meets a coverage condition; and under the condition that the next first-type storage file meets the coverage condition, writing the log into the next first-type storage file in a coverage mode.
10. A log storage device comprising a processor and a memory, the memory storing computer readable instructions which, when executed by the processor, implement the method of any one of claims 1 to 8.
11. A computer-readable storage medium having computer-readable instructions stored thereon which, when executed by a processor, implement the method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011062297.6A CN112231288A (en) | 2020-09-30 | 2020-09-30 | Log storage method and device and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011062297.6A CN112231288A (en) | 2020-09-30 | 2020-09-30 | Log storage method and device and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112231288A true CN112231288A (en) | 2021-01-15 |
Family
ID=74119948
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011062297.6A Pending CN112231288A (en) | 2020-09-30 | 2020-09-30 | Log storage method and device and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112231288A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113849126A (en) * | 2021-09-01 | 2021-12-28 | 北京腾凌科技有限公司 | Log storage method and device |
| CN115129680A (en) * | 2022-06-29 | 2022-09-30 | 苏州浪潮智能科技有限公司 | Method, device, equipment and readable medium for improving database performance |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103927251A (en) * | 2014-03-18 | 2014-07-16 | 烽火通信科技股份有限公司 | Log management method for embedded system |
| CN106993147A (en) * | 2017-03-21 | 2017-07-28 | 深圳英飞拓科技股份有限公司 | One kind video recording covering method, device and network hard disk video recorder |
| US20180205552A1 (en) * | 2015-06-02 | 2018-07-19 | ALTR Solutions, Inc. | Utilizing a tree-structure to segment and distribute files across a series of blockchains |
| CN110569177A (en) * | 2019-09-26 | 2019-12-13 | 苏州浪潮智能科技有限公司 | Method and device for testing log coverage mechanism |
| CN111026728A (en) * | 2019-11-19 | 2020-04-17 | 山东中创软件商用中间件股份有限公司 | Log data processing method and related device |
| CN111339044A (en) * | 2020-02-20 | 2020-06-26 | 苏州浪潮智能科技有限公司 | Distributed file system log submitting method, device, equipment and storage medium |
| CN111581171A (en) * | 2020-04-30 | 2020-08-25 | 中国工商银行股份有限公司 | Log processing method and device, computing equipment and medium |
-
2020
- 2020-09-30 CN CN202011062297.6A patent/CN112231288A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103927251A (en) * | 2014-03-18 | 2014-07-16 | 烽火通信科技股份有限公司 | Log management method for embedded system |
| US20180205552A1 (en) * | 2015-06-02 | 2018-07-19 | ALTR Solutions, Inc. | Utilizing a tree-structure to segment and distribute files across a series of blockchains |
| CN106993147A (en) * | 2017-03-21 | 2017-07-28 | 深圳英飞拓科技股份有限公司 | One kind video recording covering method, device and network hard disk video recorder |
| CN110569177A (en) * | 2019-09-26 | 2019-12-13 | 苏州浪潮智能科技有限公司 | Method and device for testing log coverage mechanism |
| CN111026728A (en) * | 2019-11-19 | 2020-04-17 | 山东中创软件商用中间件股份有限公司 | Log data processing method and related device |
| CN111339044A (en) * | 2020-02-20 | 2020-06-26 | 苏州浪潮智能科技有限公司 | Distributed file system log submitting method, device, equipment and storage medium |
| CN111581171A (en) * | 2020-04-30 | 2020-08-25 | 中国工商银行股份有限公司 | Log processing method and device, computing equipment and medium |
Non-Patent Citations (1)
| Title |
|---|
| ZEBRA_ZZH: "python用logging模块写循环日志", 《CSDN博客》, pages 2 - 3 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113849126A (en) * | 2021-09-01 | 2021-12-28 | 北京腾凌科技有限公司 | Log storage method and device |
| CN115129680A (en) * | 2022-06-29 | 2022-09-30 | 苏州浪潮智能科技有限公司 | Method, device, equipment and readable medium for improving database performance |
| CN115129680B (en) * | 2022-06-29 | 2024-01-16 | 苏州浪潮智能科技有限公司 | Method, device, equipment and readable medium for improving database performance |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11741046B2 (en) | Method and apparatus for creating system disk snapshot of virtual machine | |
| US9460184B2 (en) | Application of a differential dataset to a data store using sequential change sets | |
| JP6463393B2 (en) | Tenant data recovery across tenant migration | |
| RU2646334C2 (en) | File management using placeholders | |
| US9417972B2 (en) | Cascade ordering | |
| JP5646639B2 (en) | Backup using metadata virtual hard drive and differential virtual hard drive | |
| AU2012347883B2 (en) | System and method for restoring application data | |
| CN112114999A (en) | Data backup method, data recovery method, data backup device, data recovery device and electronic equipment | |
| US8396833B2 (en) | Inode management in redirect-on-write file system | |
| CN106844102B (en) | Data recovery method and device | |
| US20110093437A1 (en) | Method and system for generating a space-efficient snapshot or snapclone of logical disks | |
| EP3474143B1 (en) | Method and apparatus for incremental recovery of data | |
| US10656860B2 (en) | Tape drive library integrated memory deduplication | |
| US10795579B2 (en) | Methods, apparatuses, system and computer program products for reclaiming storage units | |
| CN112231288A (en) | Log storage method and device and medium | |
| US10223206B1 (en) | Method and system to detect and delete uncommitted save sets of a backup | |
| CN106502830B (en) | A kind of method for restoring system backup based on Btrfs file system | |
| US11093348B2 (en) | Method, device and computer program product for recovering metadata | |
| US10228879B1 (en) | System and method for backup and restore of offline disks in mainframe computers | |
| US10877992B2 (en) | Updating a database | |
| WO2014111773A1 (en) | Data backup recovery | |
| CN105760456A (en) | Method and device for maintaining data consistency | |
| CN114281798A (en) | Data migration method and device for heterogeneous database and computer readable medium | |
| US20180143883A1 (en) | Version-Controlling Storage Volume Snapshots | |
| JP2009163334A (en) | Information processor, program, and backup method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |