CN109634930B - Method and device for cleaning logs, storage medium and electronic device - Google Patents
Method and device for cleaning logs, storage medium and electronic device Download PDFInfo
- Publication number
- CN109634930B CN109634930B CN201811261221.9A CN201811261221A CN109634930B CN 109634930 B CN109634930 B CN 109634930B CN 201811261221 A CN201811261221 A CN 201811261221A CN 109634930 B CN109634930 B CN 109634930B
- Authority
- CN
- China
- Prior art keywords
- postgresql
- xlog
- cleaning
- preset period
- wal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000004140 cleaning Methods 0.000 title claims abstract description 62
- 238000000034 method Methods 0.000 title claims abstract description 55
- 239000000872 buffer Substances 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 18
- 238000012545 processing Methods 0.000 claims description 8
- 230000010076 replication Effects 0.000 claims description 8
- 238000013515 script Methods 0.000 claims description 7
- 230000008859 change Effects 0.000 claims description 6
- 230000002159 abnormal effect Effects 0.000 abstract description 9
- 238000005516 engineering process Methods 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 8
- 230000005540 biological transmission Effects 0.000 description 6
- 230000001960 triggered effect Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method and a device for clearing logs, a storage medium and an electronic device, wherein the method comprises the following steps: triggering a cleaning program in Postgresql according to a preset period; and cleaning the xlog in the Postgresql by using the cleaning program, wherein the xlog is used for recording the prewritten wal information in the Postgresql. The invention solves the technical problem that the Postgresql can not automatically clear the xlog when the system is abnormal in the related technology, and improves the stability and the safety of the Postgresql system.
Description
Technical Field
The invention relates to the field of computers, in particular to a method and a device for clearing logs, a storage medium and an electronic device.
Background
The Postgresql in the related technology is a powerful open-source object relational database system and is widely applied.
In the related art, Postgresql can delete xlog using official supply parameters under normal copying conditions. Such as: the parameter setting is reserved for 15-day xlog, the database can reserve the xlog within 15 days from the current time according to the parameters, the xlog is automatically cleared for more than 15 days, and the xlog of the timestamp before 15 days is deleted by reading the xlog timestamp. However, when the log cannot be cleared normally by the official configuration parameters after the copying is stopped or abnormally interrupted, the system log with busy service suddenly increases, which causes the database to be down due to insufficient disk space of the database.
In view of the above problems in the related art, no effective solution has been found at present.
Disclosure of Invention
The embodiment of the invention provides a method and a device for clearing logs, a storage medium and an electronic device.
According to an embodiment of the present invention, there is provided a method of cleaning up a log, including: triggering a cleaning program in Postgresql according to a preset period; and cleaning the xlog in the Postgresql by using the cleaning program, wherein the xlog is used for recording the prewritten wal information in the Postgresql.
Optionally, before triggering the cleaning procedure in Postgresql according to a preset period, the method further includes: and determining that the replication between the main database and the standby database is stopped or interrupted in the Postgresql.
Optionally, before triggering the cleaning procedure in Postgresql according to a preset period, the method includes: setting the preset period and editing the cleaning program; and adding a backup script to the scheduled task Crontab according to the cleaning program and the preset period.
Optionally, editing the cleaning program includes: creating a task table in the Postgresql; and adding a cleaning task in the task table, wherein the cleaning task is used for operating the data manipulation language DML in the task table once every preset period.
Optionally, the cleaning the xlog in the Postgresql by using the cleaning program includes: creating a checkpoint in the Postgresql using the clean-up program; and updating the last system change number LSCN of the check point, and using the check point to clean the xlog in the memory buffer area in the Postgresql to a disk.
Optionally, the clearing, to the disk, xlog in the memory buffer in the Postgresql using the check point includes: using the check point to clean the xlog into a disk according to the queue sequence of the xlog in the memory buffer area in the Postgresql; using the check point to clean the xlog into a disk according to the priority order of the xlog in the Postgresql; using the checkpoint, cleaning xlog into a disk according to attribute information of xlog in the Postgresql, wherein the attribute information includes at least one of the following: whether it is operated on, file size, whether it includes specified content.
Optionally, before triggering the cleaning procedure in Postgresql according to a preset period, the method further includes: configuring a WAL parameter of the Postgresql, wherein the WAL parameter comprises: wal _ level, wal _ sync _ method, fsync ═ on, wal _ keep _ segments; setting the storage path of the xlog after the wal parameter configuration is completed.
Optionally, before triggering the cleaning procedure in Postgresql according to a preset period, the method further includes at least one of: determining that the available storage space of the Postgresql is below a first threshold value; determining that the occupied storage amount of xlog in the Postgresql is higher than a second threshold value.
According to another embodiment of the present invention, there is provided an apparatus for clearing a log, including: the trigger module is used for triggering a cleaning program in Postgresql according to a preset period; and the cleaning module is used for cleaning the xlog in the Postgresql by using the cleaning program, wherein the xlog is used for recording the information of the prewritten wal in the Postgresql.
Optionally, the apparatus further comprises: the first determining module is used for determining that the copying between the main database and the standby database in the Postgresql is stopped or interrupted before the triggering module triggers the cleaning program in the Postgresql according to a preset period.
Optionally, the apparatus includes: the setting module is used for setting a preset period and editing the cleaning program before the trigger module triggers the cleaning program in Postgresql according to the preset period; and the adding module is used for adding the backup script to the scheduled task Crontab according to the cleaning program and the preset period.
Optionally, the setting module includes: a creating unit, configured to create a task table in the Postgresql; and the cleaning unit is used for adding a cleaning task in the task table, wherein the cleaning task is used for operating the data manipulation language DML in the task table once every preset period.
Optionally, the cleaning module includes: a creating unit for creating a checkpoint in the Postgresql using the cleaning program; and the processing unit is used for updating the last system change number LSCN of the check point and cleaning the xlog in the memory buffer area in the Postgresql to a disk by using the check point.
Optionally, the processing unit is further configured to: using the check point to clean the xlog into a disk according to the queue sequence of the xlog in the memory buffer area in the Postgresql; using the check point to clean the xlog into a disk according to the priority order of the xlog in the Postgresql; using the checkpoint, cleaning xlog into a disk according to attribute information of xlog in the Postgresql, wherein the attribute information includes at least one of the following: whether it is operated on, file size, whether it includes specified content.
Optionally, the apparatus further comprises: a configuration module, configured to configure a WAL parameter of Postgresql before the cleaning module triggers a cleaning program in the Postgresql according to a preset period, where the WAL parameter includes: wal _ level, wal _ sync _ method, fsync ═ on, wal _ keep _ segments; and the setting module is used for setting the storage path of the xlog after the wal parameter configuration is completed.
Optionally, the apparatus further comprises at least one of: the second determining module is used for determining that the available storage space of the Postgresql is lower than a first threshold value before the triggering module triggers the cleaning program in the Postgresql according to a preset period; and the third determining module is used for determining that the occupied storage amount of the xlog in the Postgresql is higher than a second threshold value before the triggering module triggers the cleaning program in the Postgresql according to a preset period.
According to a further embodiment of the present invention, there is also provided a storage medium having a computer program stored therein, wherein the computer program is arranged to perform the steps of any of the above method embodiments when executed.
According to yet another embodiment of the present invention, there is also provided an electronic device, including a memory in which a computer program is stored and a processor configured to execute the computer program to perform the steps in any of the above method embodiments.
According to the invention, the cleaning program in the Postgresql is triggered according to the preset period to clean the xlog in the Postgresql, and the xlog in the Postgresql is automatically cleaned by using the preset period, so that the abnormal condition of shutdown of a database caused by the fact that the xlog cannot be automatically cleaned when the Postgresql is abnormally copied by the official parameter of the Postgresql is avoided, the technical problem that the xlog cannot be automatically cleaned when the system is abnormal in the Postgresql in the related technology is solved, and the stability and the safety of the Postgresql system are improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
fig. 1 is a block diagram of a hardware configuration of a server for clearing logs according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method of cleaning up logs according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating an xlog log record structure according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a clean up program clean up xlog according to an embodiment of the present invention;
fig. 5 is a block diagram of an apparatus for clearing a log according to an embodiment of the present invention.
Detailed Description
The invention will be described in detail hereinafter with reference to the accompanying drawings in conjunction with embodiments. It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
Example 1
The method provided by the first embodiment of the present application may be executed in a server, a computer terminal, or a similar computing device. Taking an example of the present invention running on a server, fig. 1 is a block diagram of a hardware structure of a server for clearing a log according to an embodiment of the present invention. As shown in fig. 1, the server 10 may include one or more (only one shown in fig. 1) processors 102 (the processors 102 may include, but are not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, and optionally may also include a transmission device 106 for communication functions and an input-output device 108. It will be understood by those skilled in the art that the structure shown in fig. 1 is only an illustration, and is not intended to limit the structure of the server. For example, the server 10 may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
The memory 104 may be used for storing a computer program, for example, a software program and a module of an application software, such as a computer program corresponding to a method for clearing a log according to an embodiment of the present invention, and the processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, so as to implement the method described above. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, memory 104 may further include memory located remotely from processor 102, which may be connected to server 10 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used for receiving or transmitting data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the server 10. In one example, the transmission device 106 includes a Network adapter (NIC), which can be connected to other Network devices through a base station so as to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
In this embodiment, a method for clearing a log is provided, and fig. 2 is a flowchart of a method for clearing a log according to an embodiment of the present invention, as shown in fig. 2, the flowchart includes the following steps:
step S202, triggering a cleaning program in Postgresql according to a preset period;
in step S204, the cleanout program is used to clean xlog in Postgresql, where xlog is used to record information of the prewritten formula ((Write-Ahead Logging, wal)) in Postgresql.
Through the steps, the cleaning program in the Postgresql is triggered according to the preset period to clean the xlog in the Postgresql, and the xlog in the Postgresql is automatically cleaned by using the preset period, so that the problem that the xlog cannot be automatically cleaned when the Postgresql is abnormally copied by the official parameters of the Postgresql to cause the abnormal crash of the database is avoided, the technical problem that the xlog cannot be automatically cleaned when the system is abnormal in the related technology is solved, and the stability and the safety of the Postgresql system are improved.
Alternatively, the executing agent of the above steps may be a server, a device for running or managing Postgresql, a storage device for storing data, and the like, but is not limited thereto.
Postgresql in this embodiment supports most Structured Query Language (SQL) standards and provides many other modern features: complex queries, foreign keys, triggers, views, transaction integrity, Multi-Version Concurrency Control (MVCC). Likewise, PostgreSQL can be extended in many ways, such as by adding new data types, functions, operators, aggregation functions, indexes. PostgreSQL is used, modified, and distributed free of charge. Xlog in this embodiment, also called Pg _ xlog, is recorded wal information of Postgresql, i.e. some transaction log information (transaction log), by default a single size of 16M.
Fig. 3 is a schematic diagram of an XLOG log record structure according to an embodiment of the present invention, which includes a header, record 1, record 2, record 3, and so on, where XLogRecord records related control information of XLOG, and when XLOG records data information, rmgr data is written by XLogInsert () function, and there are one or more XLOG rcdata data structures, and there are two cases when there are multiple XLOG rcdata structures: 1. the source data is not physically adjacent on the memory; 2. data is specified in a plurality of buffers.
In this embodiment, before triggering the cleaning procedure in Postgresql according to a preset period, the method further includes: and determining that the replication between the main database and the standby database is stopped or interrupted in Postgresql.
Because Postgresql is under the abnormal copying environment, pg _ xlog cannot be automatically cleaned, the rapid increase of storage space can cause the risk of database downtime. Therefore, under an abnormal environment such as a replication stop or a replication interruption between the primary database and the secondary database, a backup xlog cleaning scheme, that is, the log cleaning scheme of the embodiment is triggered. The problem that the parameters provided by the default of the official parties do not automatically clear the log in the abnormal situation of the copy can be avoided.
Optionally, before triggering the cleaning procedure in Postgresql according to a preset period, the method includes:
s11, setting a preset period and editing a cleaning program;
in an optional implementation manner of this embodiment, the edit cleaning program includes: creating a task table in Postgresql; adding a cleaning task (JOB) in the task table, wherein the cleaning task is used for operating the data manipulation language DML in the task table once every preset period. When the preset period is one minute, the specific operation flow and codes include:
the DML is an instruction set in the SQL language responsible for operating data access work on database objects, takes three instructions, namely INSERT, UPDATE and DELETE, as cores, and is an instruction that must be used for developing data-centric applications.
S12, adding a cleaning program and a preset period to a backup script to a planning task Crontab of Postgresql, wherein the executed program codes comprise:
1/*****/home/postgres/keep_rule.sh
edit keep_rule.sh
find/pgdata/pg_xlog/-cmin+80-name"00000001*"-exec rm-rf{}\。
the instruction list is periodically executed by crond on linux, the executed task is called cron joband a plurality of tasks are called crontab.
The process of starting the Crontab service comprises the following steps: checking whether crontab-l is installed; installation of yum install crontabs; looking up the service status, service connected status; and starting the service, and starting service connected start.
Optionally, the cleaning the xlog in Postgresql using a cleaning program includes:
s21, creating a check point in Postgresql by using a cleaning program;
s22, updating the Last System Change Number (LSCN) of the check point, and using the check point to clean the xlog in the memory buffer area in Postgresql to the disk. The created check points are used to perform timing insertion and deletion, i.e., a previously set preset period, such as a timing period of 1 minute.
In an optional implementation manner of this embodiment, the using the checkpoint to flush xlog in the memory buffer in Postgresql to the disk includes: using a check point to clean the xlog into a disk according to the queue sequence of the xlog in a memory buffer area in Postgresql; using a check point to clean the xlog into a disk according to the priority order of the xlog in Postgresql; using the checkpoint, cleaning the xlog into a disk according to the attribute information of the xlog in Postgresql, wherein the attribute information comprises at least one of the following: whether it is operated on, file size, whether it includes specified content. The specified content may be an error field, alarm content, etc.
Fig. 4 is a schematic diagram of a cleaning program cleaning xlog according to an embodiment of the present invention, and Postgresql includes three events: checkpoint, wal receiver process, write fsync, where checkpoint is an internal event, and after the event is activated, it will trigger a database write process (DBWR) to write out a dirty data block in a data buffer (DATABUFFER CACHE) into a data file, that is, clean xlog to a disk; the wal receiver process is a receiving process of wal information, and a log generated by a receiving device; the write fsync is write synchronization, all modified file data (log) in the memory are synchronized to the storage device, and when the write synchronization is abnormal, the system triggers a cleaning program to automatically clean the xlog in the memory to avoid overflow and downtime.
Optionally, before triggering the cleaning procedure in Postgresql according to a preset period, the method further includes: configuring a WAL parameter of Postgresql, wherein the WAL parameter comprises: wal _ level, wal _ sync _ method, fsync ═ on, wal _ keep _ segments; the xlog's storage path is set. The configured WAL parameters and the cleaning procedures triggered by the configuration period have no time sequence relationship, and may be configured simultaneously, or the cleaning procedures triggered by the configuration period may be configured after the WAL parameters are configured, or before the WAL parameters are configured.
In one example, after configuring the WAL parameters, a cleaning procedure (a procedure for updating the LSCN) is configured, and the complete scheme applied in this example includes:
step 1: configuring relevant parameters WAL _ level, WAL _ sync _ method, fsync ═ on, and WAL _ keep _ segments of the database WAL;
step 2: setting a pg _ xlog log path;
pg_xlog＝/db/postgres/pg_xlog
and step 3: restarting the database to enable the parameters to take effect;
pg_ctl restart
and 4, step 4: writing a program to update the LSCN, further comprising:
and 5: add backup scripts to the planning task Crontab.
1/*****/home/postgres/keep_rule.sh
edit keep_rule.sh
find/pgdata/pg_xlog/-cmin+80-name"00000001*"-exec rm-rf{}\；
In an optional implementation manner of this embodiment, before triggering the cleaning procedure in Postgresql according to a preset period, the method further includes at least one of the following: determining that the available storage space of Postgresql is below a first threshold value; it is determined that the occupied storage capacity of xlog in Postgresql is above a second threshold value. The available storage space refers to an available storage space of a memory buffer, and the occupied storage amount of xlog also refers to an occupied space of the memory buffer, which may be used in combination with the above-mentioned replication exception triggering mechanism as a triggering condition of this embodiment, or may be used alone, for example, even if there is no replication stop or replication interruption between the primary database and the backup database, the available storage space of Postgresql is lower than a first threshold, or when the occupied storage amount of xlog in Postgresql is higher than a second threshold, a cleaning program in Postgresql is triggered according to a preset period, and the xlog in Postgresql is cleaned by the cleaning program. The first threshold and the second threshold in this embodiment are determined by the total storage space, and may be a storage amount, for example, the first threshold is 1G, or may be a proportional value, for example, the proportion of occupied storage amount of which the second threshold is xlog to the total storage amount is 90%.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
Example 2
In this embodiment, a device for clearing a log is further provided, and the device is used to implement the foregoing embodiments and preferred embodiments, and details are not repeated after the description is given. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 5 is a block diagram of an apparatus for clearing a log according to an embodiment of the present invention, which can be applied to a server, a terminal, and the like, as shown in fig. 5, and includes:
the trigger module 50 is used for triggering a cleaning program in Postgresql according to a preset period;
and a cleaning module 52, configured to clean xlog in the Postgresql by using the cleaning program, where the xlog is used to record the prewritten wal information in the Postgresql.
In an optional implementation manner of this embodiment, the apparatus further includes: the first determining module is used for determining that the copying between the main database and the standby database in the Postgresql is stopped or interrupted before the triggering module triggers the cleaning program in the Postgresql according to a preset period.
Optionally, the apparatus further comprises: the setting module is used for setting a preset period and editing the cleaning program before the trigger module triggers the cleaning program in Postgresql according to the preset period; and the adding module is used for adding the backup script to the scheduled task Crontab according to the cleaning program and the preset period.
Optionally, the setting module includes: a creating unit, configured to create a task table in the Postgresql; and the cleaning unit is used for adding a cleaning task in the task table, wherein the cleaning task is used for operating the data manipulation language DML in the task table once every preset period.
In an optional implementation manner of this embodiment, the cleaning module includes: a creating unit for creating a checkpoint in the Postgresql using the cleaning program; and the processing unit is used for updating the last system change number LSCN of the check point and cleaning the xlog in the memory buffer area in the Postgresql to a disk by using the check point.
Optionally, the processing unit is further configured to: using a check point to clean the xlog into a disk according to the queue sequence of the xlog in a memory buffer area in Postgresql; using a check point to clean the xlog into a disk according to the priority order of the xlog in Postgresql; using the checkpoint, cleaning the xlog into a disk according to the attribute information of the xlog in Postgresql, wherein the attribute information comprises at least one of the following: whether it is operated on, file size, whether it includes specified content. The specified content may be an error field, alarm content, etc.
Optionally, the apparatus further comprises: a configuration module, configured to configure a WAL parameter of Postgresql before the cleaning module triggers a cleaning program in the Postgresql according to a preset period, where the WAL parameter includes: wal _ level, wal _ sync _ method, fsync ═ on, wal _ keep _ segments; and the setting module is used for setting the storage path of the xlog.
In an optional implementation manner of this embodiment, the apparatus further includes at least one of: the second determining module is used for determining that the available storage space of the Postgresql is lower than a first threshold value before the triggering module triggers the cleaning program in the Postgresql according to a preset period; and the third determining module is used for determining that the occupied storage amount of the xlog in the Postgresql is higher than a second threshold value before the triggering module triggers the cleaning program in the Postgresql according to a preset period.
It should be noted that, the above modules may be implemented by software or hardware, and for the latter, the following may be implemented, but not limited to: the modules are all positioned in the same processor; alternatively, the modules are respectively located in different processors in any combination.
Example 3
Embodiments of the present invention also provide a storage medium having a computer program stored therein, wherein the computer program is arranged to perform the steps of any of the above method embodiments when executed.
Alternatively, in the present embodiment, the storage medium may be configured to store a computer program for executing the steps of:
s1, triggering a cleaning program in Postgresql according to a preset period;
s2, cleaning the xlog in the Postgresql by using the cleaning program, wherein the xlog is used for recording the information of the prewritten wal in the Postgresql.
Optionally, in this embodiment, the storage medium may include, but is not limited to: various media capable of storing computer programs, such as a usb disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.
Embodiments of the present invention also provide an electronic device comprising a memory having a computer program stored therein and a processor arranged to run the computer program to perform the steps of any of the above method embodiments.
Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:
s1, triggering a cleaning program in Postgresql according to a preset period;
s2, cleaning the xlog in the Postgresql by using the cleaning program, wherein the xlog is used for recording the information of the prewritten wal in the Postgresql.
Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation manners, and this embodiment is not described herein again.
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the principle of the present invention should be included in the protection scope of the present invention.
Claims (16)
1. A method of cleaning logs, comprising:
responding to Postgresql copy abnormity and not automatically cleaning the xlog, and triggering a cleaning program in the Postgresql according to a preset period;
cleaning an xlog in the Postgresql by using the cleaning program, wherein the xlog is used for recording the prewritten wal information in the Postgresql;
before triggering a cleaning program in Postgresql according to a preset period, setting the preset period, and editing the cleaning program; and adding a backup script to the planned task Crontab of the Postgresql according to the cleaning program and the preset period.
2. The method of claim 1, wherein before triggering the cleanup procedure in Postgresql at a preset period, the method further comprises:
and determining that the replication between the main database and the standby database is stopped or interrupted in the Postgresql.
3. The method of claim 1, wherein editing the cleaning program comprises:
creating a task table in the Postgresql;
and adding a cleaning task in the task table, wherein the cleaning task is used for operating the data manipulation language DML in the task table once every preset period.
4. The method of claim 1, wherein cleaning xlog in the Postgresql using the cleaning program comprises:
creating a checkpoint in the Postgresql using the clean-up program;
and updating the last system change number LSCN of the check point, and using the check point to clean the xlog in the memory buffer area in the Postgresql to a disk.
5. The method of claim 4, wherein using the checkpoint to flush xlog in the in-memory buffer in the Postgresql to disk comprises one of:
using the check point to clean the xlog into a disk according to the queue sequence of the xlog in the memory buffer area in the Postgresql;
using the check point to clean the xlog into a disk according to the priority order of the xlog in the Postgresql;
using the checkpoint, cleaning xlog into a disk according to attribute information of xlog in the Postgresql, wherein the attribute information includes at least one of the following: whether it is operated on, file size, whether it includes specified content.
6. The method of claim 1, wherein before triggering the cleanup procedure in Postgresql at a preset period, the method further comprises:
configuring wal parameters of the Postgresql, wherein the wal parameters include: wal _ level, wal _ sync _ method, fsync, and wal _ keep _ segments, where the parameter fsync = on;
setting the storage path of the xlog after the wal parameter configuration is completed.
7. The method of claim 1, wherein before triggering the cleanup procedure in Postgresql at a preset period, the method further comprises at least one of:
determining that the available storage space of the Postgresql is below a first threshold value;
determining that the occupied storage amount of xlog in the Postgresql is higher than a second threshold value.
8. An apparatus for clearing a log, comprising:
the trigger module is used for responding to Postgresql copy abnormity and not automatically cleaning the xlog and triggering a cleaning program in the Postgresql according to a preset period;
a cleaning module, configured to clean xlog in the Postgresql by using the cleaning program, where the xlog is used to record information of prewritten wal in the Postgresql;
the setting module is used for setting a preset period and editing the cleaning program before the trigger module triggers the cleaning program in Postgresql according to the preset period;
and the adding module is used for adding the cleaning program and the preset period into a backup script to the Crontab of the Postgresql plan task.
9. The apparatus of claim 8, further comprising:
the first determining module is used for determining that the copying between the main database and the standby database in the Postgresql is stopped or interrupted before the triggering module triggers the cleaning program in the Postgresql according to a preset period.
10. The apparatus of claim 8, wherein the setup module comprises:
a creating unit, configured to create a task table in the Postgresql;
and the cleaning unit is used for adding a cleaning task in the task table, wherein the cleaning task is used for operating the data manipulation language DML in the task table once every preset period.
11. The apparatus of claim 8, wherein the cleaning module comprises:
a creating unit for creating a checkpoint in the Postgresql using the cleaning program;
and the processing unit is used for updating the last system change number LSCN of the check point and cleaning the xlog in the memory buffer area in the Postgresql to a disk by using the check point.
12. The apparatus of claim 11, wherein the processing unit is further configured to one of:
using the check point to clean the xlog into a disk according to the queue sequence of the xlog in the memory buffer area in the Postgresql;
using the check point to clean the xlog into a disk according to the priority order of the xlog in the Postgresql;
using the checkpoint, cleaning xlog into a disk according to attribute information of xlog in the Postgresql, wherein the attribute information includes at least one of the following: whether it is operated on, file size, whether it includes specified content.
13. The apparatus of claim 8, further comprising:
a configuration module, configured to configure wal parameters of Postgresql before the cleaning module triggers a cleaning program in the Postgresql according to a preset period, where the wal parameters include: wal _ level, wal _ sync _ method, fsync, and wal _ keep _ segments, where the parameter fsync = on;
and the setting module is used for setting the storage path of the xlog after the wal parameter configuration is completed.
14. The apparatus of claim 8, further comprising at least one of:
the second determining module is used for determining that the available storage space of the Postgresql is lower than a first threshold value before the triggering module triggers the cleaning program in the Postgresql according to a preset period;
and the third determining module is used for determining that the occupied storage amount of the xlog in the Postgresql is higher than a second threshold value before the triggering module triggers the cleaning program in the Postgresql according to a preset period.
15. A storage medium, in which a computer program is stored, wherein the computer program is arranged to perform the method of any of claims 1 to 7 when executed.
16. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and wherein the processor is arranged to execute the computer program to perform the method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811261221.9A CN109634930B (en) | 2018-10-26 | 2018-10-26 | Method and device for cleaning logs, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811261221.9A CN109634930B (en) | 2018-10-26 | 2018-10-26 | Method and device for cleaning logs, storage medium and electronic device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109634930A CN109634930A (en) | 2019-04-16 |
| CN109634930B true CN109634930B (en) | 2020-12-04 |
Family
ID=66066790
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811261221.9A Expired - Fee Related CN109634930B (en) | 2018-10-26 | 2018-10-26 | Method and device for cleaning logs, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109634930B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110727446A (en) * | 2019-09-25 | 2020-01-24 | 光通天下网络科技股份有限公司 | Automatic deployment method, device, equipment and medium |
| CN114036000A (en) * | 2021-10-19 | 2022-02-11 | 航天信息股份有限公司 | File backup and cleaning method and system suitable for PostgreSQL master and standby environment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103793479A (en) * | 2014-01-14 | 2014-05-14 | 上海上讯信息技术股份有限公司 | Log management method and log management system |
| CN105373625A (en) * | 2015-12-09 | 2016-03-02 | 浪潮电子信息产业股份有限公司 | ORACLE data backup method |
| CN107145499A (en) * | 2016-03-01 | 2017-09-08 | 阿里巴巴集团控股有限公司 | The method and apparatus filed for ahead daily record |
| CN107622120A (en) * | 2017-09-22 | 2018-01-23 | 新华三技术有限公司 | System journal method for cleaning and device |
-
2018
- 2018-10-26 CN CN201811261221.9A patent/CN109634930B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103793479A (en) * | 2014-01-14 | 2014-05-14 | 上海上讯信息技术股份有限公司 | Log management method and log management system |
| CN105373625A (en) * | 2015-12-09 | 2016-03-02 | 浪潮电子信息产业股份有限公司 | ORACLE data backup method |
| CN107145499A (en) * | 2016-03-01 | 2017-09-08 | 阿里巴巴集团控股有限公司 | The method and apparatus filed for ahead daily record |
| CN107622120A (en) * | 2017-09-22 | 2018-01-23 | 新华三技术有限公司 | System journal method for cleaning and device |
Non-Patent Citations (3)
| Title |
|---|
| ode.PostgreSQL Plus Advanced Server配合crontab实现定时维护工作.《bbsmax》.2013, * |
| PostgreSQL Plus Advanced Server配合crontab实现定时维护工作;ode;《bbsmax》;20130717;全文 * |
| postgresql 的三类日志;LJ你是唯一LT;《iteye》;20160114;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109634930A (en) | 2019-04-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11023448B2 (en) | Data scrubbing method and apparatus, and computer readable storage medium | |
| JP4321705B2 (en) | Apparatus and storage system for controlling acquisition of snapshot | |
| CN103761165B (en) | Log backup method and device | |
| US20150213100A1 (en) | Data synchronization method and system | |
| US9317374B2 (en) | Performing a background copy process during a backup operation | |
| CN108509462B (en) | Method and device for synchronizing activity transaction table | |
| CA2839014A1 (en) | Managing replicated virtual storage at recovery sites | |
| KR20170130448A (en) | System and method for automatic cloud-based full data backup and restore on mobile devices | |
| CN107423112B (en) | Docker container state real-time synchronization method | |
| US10809922B2 (en) | Providing data protection to destination storage objects on remote arrays in response to assignment of data protection to corresponding source storage objects on local arrays | |
| CN105573859A (en) | Data recovery method and device of database | |
| CN110990432A (en) | Device and method for synchronizing distributed cache clusters across machine rooms | |
| US10976942B2 (en) | Versioning a configuration of data storage equipment | |
| CN106777126B (en) | Data online migration method supporting heterogeneous time sequence database | |
| CN109634930B (en) | Method and device for cleaning logs, storage medium and electronic device | |
| CN104516796A (en) | Command set based network element backup and recovery method and device | |
| CN114490677A (en) | Data synchronization in a data analysis system | |
| CN114968966A (en) | Distributed metadata remote asynchronous replication method, device and equipment | |
| US9639701B1 (en) | Scheduling data protection operations based on data activity | |
| CN104216802A (en) | Memory database recovery method and device | |
| US9619506B2 (en) | Method and system to avoid deadlocks during a log recovery | |
| CN114791901A (en) | Data processing method, device, equipment and storage medium | |
| CN111078418A (en) | Operation synchronization method and device, electronic equipment and computer readable storage medium | |
| US8352440B2 (en) | Using repeated incremental background consistency checking to detect problems with content closer in time to when a failure occurs | |
| WO2023111910A1 (en) | Rolling back database transaction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20201204 |