CN117194344A - File management method and device and electronic equipment - Google Patents
File management method and device and electronic equipment Download PDFInfo
- Publication number
- CN117194344A CN117194344A CN202311158466.XA CN202311158466A CN117194344A CN 117194344 A CN117194344 A CN 117194344A CN 202311158466 A CN202311158466 A CN 202311158466A CN 117194344 A CN117194344 A CN 117194344A
- Authority
- CN
- China
- Prior art keywords
- management table
- files
- file
- management
- folder
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007726 management method Methods 0.000 title claims abstract description 295
- 238000000034 method Methods 0.000 claims description 42
- 238000013508 migration Methods 0.000 claims description 9
- 230000005012 migration Effects 0.000 claims description 9
- 238000005192 partition Methods 0.000 claims description 9
- 238000012545 processing Methods 0.000 abstract description 6
- 230000006870 function Effects 0.000 description 8
- 208000000044 Amnesia Diseases 0.000 description 6
- 208000026139 Memory disease Diseases 0.000 description 6
- 238000004140 cleaning Methods 0.000 description 6
- 230000006984 memory degeneration Effects 0.000 description 6
- 208000023060 memory loss Diseases 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 230000005856 abnormality Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000013515 script Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000003190 augmentative effect Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000000246 remedial effect Effects 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a file management method, a file management device and electronic equipment, and belongs to the technical field of data processing. The file management method comprises the following steps: creating a second management table under the condition that the first management table is a management table of the first type of files; reading a file in a first management table; combining the files of the first management table by adjusting the number of the written target files, and writing the files of the first management table into the second management table; and updating the files in the first management table according to the files in the second management table.
Description
Technical Field
The application belongs to the technical field of data processing, and particularly relates to a file management method, a file management device and electronic equipment.
Background
In order to cope with and handle the ever-increasing mass of data, distributed file systems should nevertheless be in existence. The distributed file system is divided into a metadata storage part and a physical storage part, wherein the metadata is responsible for managing the name space (namely, the namespace) of the file system and the access of the client to the file, and the physical storage part is responsible for storing the actual file. During the reading and writing process of the data, the metadata is accessed to acquire the actual reading and writing address, and then the corresponding operation is performed based on the address.
Specifically, the distributed file system takes Hadoop Distributed File System, HDFS as an example, where the NameNode service is responsible for metadata management, and the DataNode service is responsible for actual storage, where a file is generally divided into several blocks for storage, each block has a size of 128 megabytes, and metadata information of each block has 150 bytes stored in a memory.
A large number of small files can cause NameNode service to frequently recycle garbage, so that the memory is consumed and down.
Under the scenes of dynamic partitioning, incremental data writing and the like, a large number of small files can be generated, and the small files cause the influence of NameNode service overload, read-write performance reduction and the like.
Disclosure of Invention
The embodiment of the application aims to provide a file management method, a file management device and electronic equipment, which can solve the problems that a large number of small files are generated under the scenes of dynamic partition, incremental data writing and the like, and the NameNode service is overloaded and the read-write performance is reduced due to the large number of small files.
In a first aspect, an embodiment of the present application provides a file management method, including: creating a second management table under the condition that the first management table is a management table of the first type of files; reading a file in a first management table; combining the files of the first management table by adjusting the number of the written target files, and writing the files of the first management table into the second management table; and updating the files in the first management table according to the files in the second management table.
In a second aspect, an embodiment of the present application provides a file management apparatus, including: the creating module is used for creating a second management table under the condition that the first management table is the management table of the first type of files; the reading module is used for reading the file in the first management table; the writing module is used for merging the files of the first management table by adjusting the number of the written target files and writing the files of the first management table into the second management table; and the updating module is used for updating the files in the first management table according to the files in the second management table.
In a third aspect, embodiments of the present application provide an electronic device comprising a processor and a memory storing a program or instructions executable on the processor, which when executed by the processor, implement the steps of the method as in the first aspect.
In a fourth aspect, embodiments of the present application provide a readable storage medium having stored thereon a program or instructions which when executed by a processor perform the steps of the method as in the first aspect.
In a fifth aspect, embodiments of the present application provide a chip comprising a processor and a communication interface, the communication interface being coupled to the processor, the processor being configured to execute programs or instructions to implement a method as in the first aspect.
In a sixth aspect, embodiments of the present application provide a computer program product stored in a storage medium, the program product being executable by at least one processor to implement a method as in the first aspect.
In the embodiment of the application, the first type of files are small files, wherein the small files are files with occupied storage space values lower than a threshold value, and correspondingly, the large files are files with occupied storage space values larger than or equal to the threshold value.
The files of the first management table are combined by adjusting the number of the written target files, and a large number of files of the first type in the first management table are combined into large files so as to reduce the number of small files, further reduce the number of metadata information in the memory, reduce the overall amount of data stored in the memory, reduce the memory loss and reduce the probability of downtime of the distributed file system, thereby improving the reliability of the distributed file system.
Because the number of small files is reduced, in the distributed file system taking HDFS as an example, the possibility of NameNode service overload is reduced, and the read-write performance of the distributed file system is ensured.
Drawings
FIG. 1 is a flow chart of a file management method according to an embodiment of the present application;
FIG. 2 is a second flowchart of a file management method according to an embodiment of the application;
FIG. 3 is a third flow chart of a file management method according to an embodiment of the application;
FIG. 4 is a schematic block diagram of a file management apparatus in an embodiment of the application;
FIG. 5 is a schematic block diagram of an electronic device in an embodiment of the application;
fig. 6 is a schematic diagram of a hardware structure of an electronic device in an embodiment of the present application.
Detailed Description
Embodiments of the present application will be described more fully hereinafter with reference to the accompanying drawings, in which it is shown, however, in which some, but not all embodiments of the application are shown. All other embodiments, which are obtained by a person skilled in the art based on the embodiments of the present application, fall within the scope of protection of the present application.
The terms first, second and the like in the description and in the claims, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged, as appropriate, such that embodiments of the present application may be implemented in sequences other than those illustrated or described herein, and that the objects identified by "first," "second," etc. are generally of a type, and are not limited to the number of objects, such as the first object may be one or more. Furthermore, in the description and claims, "and/or" means at least one of the connected objects, and the character "/", generally means that the associated object is an "or" relationship.
The file management method, the device and the electronic equipment provided by the embodiment of the application are described in detail through specific embodiments and application scenes thereof with reference to the accompanying drawings.
In one embodiment, as shown in fig. 1, a file management method is proposed, including:
102, creating a second management table under the condition that the first management table is a management table of the first type of files;
104, reading a file in a first management table;
step 106, merging the files of the first management table by adjusting the written target file number, and writing the files of the first management table into the second management table;
and step 108, updating the files in the first management table according to the files in the second management table.
In the embodiment of the application, the first type of files are small files, wherein the small files are files with occupied storage space values lower than a threshold value, and correspondingly, the large files are files with occupied storage space values larger than or equal to the threshold value.
The files of the first management table are combined by adjusting the number of the written target files, and a large number of files of the first type in the first management table are combined into large files so as to reduce the number of small files, further reduce the number of metadata information in the memory, reduce the overall amount of data stored in the memory, reduce the memory loss and reduce the probability of downtime of the distributed file system, thereby improving the reliability of the distributed file system.
Because the number of small files is reduced, in the distributed file system taking HDFS as an example, the possibility of NameNode service overload is reduced, and the read-write performance of the distributed file system is ensured.
In some embodiments of the present application, the file management method further includes: reading mirror image files of a storage system, and obtaining the number of files of a first management table and the storage space value occupied by each file when storing; determining a first memory value based on the number of files and the memory value; and determining the first management table as the management table of the first type file under the condition that the first storage space value is smaller than or equal to a first threshold value.
In this embodiment, the number of files of the first management table and the storage space value occupied by each file when stored are acquired, so that the total value of the storage space occupied by the files of the first management table is calculated based on the number of files of the first management table and the storage space value occupied by each file when stored, and the first storage space value is calculated based on the ratio of the total value to the number of files.
If the first storage space value is large, the first management table is considered to be not a management table of the first type file, whereas if the first storage space value is small, the first management table is considered to be a management table of the first type file.
Specifically, the first threshold is set so as to measure the size of the first storage space value based on the first threshold, thereby realizing the judgment of whether the first management table is the management table of the first type file.
In some embodiments of the application, the first threshold is 80 megabits.
In some embodiments of the present application, the number of files in the first management table and the storage space value occupied by each file during storage are obtained by limiting the reading of the image files in the storage system, so that the image files are analyzed by utilizing the characteristic that the image files are snapshots of the distributed file system during operation and the content of the image files is unchanged, and in the process, the influence on the distributed file system during the process of obtaining the number of files in the first management table and the storage space value occupied by each file during storage can be reduced.
In some embodiments of the present application, where the distributed file system is HDFS, the image file of the storage system may be an fsimage file.
In some embodiments of the present application, merging the files of the first management table by adjusting the number of target files written, writing the files of the first management table into the second management table, includes: adjusting the number of the written target files by adjusting the partition number; merging the files of the first management table, merging the files of the first management table and writing the merged files into the second management table; wherein the number of files in the second management table is the target number of files.
In this embodiment, the number of partitions is adjusted in advance so as to adjust the number of target files according to the number of partitions, and then, in the process of writing the files of the first management table into the second management table, the plurality of small files in the first management table are combined, so that the files of the target number of files are finally formed after all the files in the first management table are written into the second management table.
In the process of merging and writing a plurality of small files to obtain the files of the target file number, the storage space value occupied by the merged files is larger, if the storage space value is larger than a first threshold value.
In the process, a plurality of small files in the first management table can be combined and converted into large files with the target file number, so that the quantity of metadata information in the memory is reduced, the integral quantity of stored data in the memory is reduced, the memory loss is reduced, the downtime probability of the distributed file system is reduced, and the reliability of the distributed file system is improved.
In some embodiments of the present application, updating the file in the first management table based on the file in the second management table includes: comparing the files in the second management table with the files in the first management table; and storing the file in the second management table to the path in the first management table under the condition that the comparison is consistent.
In this embodiment, the files in the second management table are compared with the files in the first management table, so as to verify whether the written files in the second management table are consistent with the files in the first management table, and then find the difference in time under the condition that the written files are inconsistent with the files in the first management table, so that corresponding remedial measures are executed, the probability of abnormality of data stored in the distributed file system due to association management is reduced, and further the reliability of the distributed file system is ensured.
In some embodiments of the present application, updating of files in the first management table is accomplished by storing files in the second management table to paths in the first management table.
In this embodiment, after the files in the first management table are updated, the probability of occurrence of small files in the first management table is smaller, so that the overall amount of data stored in the memory is reduced, the memory loss is reduced, the probability of occurrence of downtime of the distributed file system is reduced, and the reliability of the distributed file system is improved.
In the above embodiment, the path of storing the file in the second management table in the first management table is adopted to update the first management table, so that the probability of occurrence of data abnormality in the process of replacing the management table is reduced compared with the scheme of directly replacing the first management table with the second management table, and the reliability of the distributed file system is improved.
In some embodiments of the present application, the alert information is output in the event of inconsistent alignment.
In this embodiment, the reminding information is output to remind the user by using the reminding information, or the reminding information is stored as a log, so that the distributed file system can be maintained according to the reminding information.
In some embodiments of the present application, in the event of inconsistent alignment, the files of the first management table are re-merged and written to the second management table.
In this embodiment, under the condition of inconsistent comparison, the files of the first management table are merged and written into the second management table again, so as to implement self-processing of the exception, and in this process, the influence of sporadic read-write exceptions on file management can be eliminated.
In some embodiments of the present application, updating the file in the first management table based on the file in the second management table further comprises performing at least one of: and migrating the file of the first management table to a target position and deleting the second management table.
In this embodiment, the file in the second management table is stored in the path in the first management table by migrating the file of the first management table to the target location. In this process, while providing a writing space for the files in the second management table, the target location may be used as a temporary storage location of the files in the first management table, and further, in the case that an abnormality occurs in the distributed file system after updating the files in the first management table, the distributed file system may be restored according to the files in the first management table stored in the target location, thereby improving the reliability of the distributed file system.
In the above embodiment, by deleting the second management table, the occupation of the storage space by the second management table is reduced, the redundancy of data is reduced, and the storage performance of the distributed file system is utilized to the greatest extent.
In some embodiments of the present application, the migration of the file of the first management table to the target location corresponding to the first folder includes: files of the first management table migrated in the first time period are archived to the first folder.
In this embodiment, the first duration is defined so that the migration time of the file stored in the first folder is within the first duration, so that the file can be managed conveniently.
In some embodiments of the present application, the value of the first duration may be 1 hour, 1 day, or one week, or may be selected according to actual design needs, which is not described herein.
In the above embodiment, archiving may be understood as storing in the first folder.
In some embodiments of the present application, as the distributed file system works, files of the first management table migrated in the next first period are archived to the first folder, the original first folder is changed to the second folder, and so on until the distributed file system runs continuously, and the files stored in the first folder are archived.
In some embodiments of the present application, the naming of the first folder is associated with a migration time of a file of the first management table, and the file management method further includes: acquiring the archived time length of the first folder based on the naming of the first folder; and in the case that the archived time length is greater than or equal to the second time length, clearing the first folder.
In this embodiment, by defining that the first folder name is associated with the migration time of the file of the first management table, the migration time of the file stored therein can be known according to the name of the first folder, so that the archived time period can be conveniently known according to the name of the first folder, thereby managing the first folder according to the archived time period.
In the process, the files with overlong filing time can be cleaned, so that the occupation of the storage space is reduced, and the utilization rate of the storage space is further improved to the greatest extent.
In some embodiments of the present application, where the first management table is a management table of a first type of file, creating the second management table includes: acquiring a table structure of a first management table under the condition that the first management table is a management table of a first type of files; creating a second management table based on the table structure of the first management table; wherein, the table structure of the first management table is the same as the table structure of the second management table.
In this embodiment, by creating the second management table having the same table structure as the first management table, so that in the process of writing the file of the first management table into the second management table, the file content and structure in the second management table can be kept identical to those in the first management table, thereby reducing the occurrence probability of data abnormality and the like caused by file management, and thus ensuring the reliability of data processing.
In some embodiments of the present application, in the case that the first folder is a recycle bin, the first management table is an original table, and the second management table is a temporary table, as shown in fig. 2, the file management method includes:
step 202, analyzing the fsimage file, and judging whether the average file size of the table is smaller than 80M or not based on the number of files and the file size;
step 204, if the file is a small file, executing step 206 if the result is yes, and ending if the result is no;
step 206, obtaining information of a table of whether the average file size is smaller than 80M, and scheduling 1000 small file merging scripts per hour;
step 208, the script reads the whole content of the table through the sql statement, writes the read content into the temporary table, and controls the written file number through controlling the partition number in the writing process;
step 210, if the original table and the temporary table data are consistent, executing step 212 if the judgment result is yes, and ending if the judgment result is no;
step 212, the original table file is placed in the recycle bin, the temporary table file is placed under the original table path, and the temporary table is deleted.
In some embodiments of the present application, after the original table file is placed in the recycle bin, a cleaning process is performed, as shown in fig. 3, where the cleaning process includes:
step 302, cleaning files in a recycle bin once per hour, and archiving the files put in the recycle bin in approximately 1 hour into a folder named by time based on cleaning time;
step 304, the archived folder in the recycle bin is accessed once an hour, how many days the folder has been archived is calculated based on the folder name, and if more than 7 days, the archived file in the recycle bin is completely purged.
From the above, the file management method is mainly divided into five steps:
1. and (3) identifying a small file table:
a) Analyzing the fsimage file to obtain the information of the number of files, the size of the files and the like of each table;
b) Calculating an average file size for each table based on the number of files and the file size for that table;
c) If the average file size is less than 80M, the table is considered a small file table.
2. Small file merging:
a) Scheduling 1000 times of small file merging scripts per hour based on a timing task to merge small files;
b) Creating a temporary table with the same table structure based on the original table sentence in the small file merging script;
c) And reading the whole content of the original table by adopting the sql statement, writing the read content into the temporary table, and controlling the number of finally written files by controlling the number of partitions in the writing process.
3. And (3) data inspection:
a) And reading the data of the original table and the temporary table based on the sql statement, and comparing whether the data of the original table and the temporary table are consistent or not.
4. Data migration
a) Placing the files in the original table into a recycle bin;
b) The file of the temporary table which is successfully combined is put into the original table path;
c) The temporary table is deleted.
5. Cleaning:
step 1: archiving
a) Files in the recycle bin are cleaned once per hour, and files placed in the recycle bin within approximately 1 hour are archived in a time ordered folder based on the cleaning time.
Step 2: clearing
b) The archived folders in the recycle bin are accessed once an hour, how many days the folders have been archived are calculated based on the folder names, and if more than 7 days, the archived files in the recycle bin are thoroughly purged.
In the embodiment, the method is simple, and small file management can be performed based on sql sentences and timing scheduling tasks; meanwhile, the small file problem generated by various calculation engines can be treated by only selecting one calculation engine without adapting to various calculation engines, so that the method has better universality, and meanwhile, files with different sizes are adapted by adjusting the first threshold.
According to the file management method provided by the embodiment of the application, the execution main body can be a file management device. In the embodiment of the present application, a method for executing file management by a file management device is taken as an example, and the file management device provided by the embodiment of the present application is described.
In one embodiment, as shown in fig. 4, there is provided a file management apparatus 400 including: a creating module 402, configured to create a second management table if the first management table is a management table of a first type of file; the reading module 404 reads the file in the first management table; a writing module 406, configured to combine the files of the first management table by adjusting the number of the written target files, and write the files of the first management table into the second management table; and the updating module 408 is configured to update the file in the first management table according to the file in the second management table.
In the embodiment of the application, the first type of files are small files, wherein the small files are files with occupied storage space values lower than a threshold value, and correspondingly, the large files are files with occupied storage space values larger than or equal to the threshold value.
The files of the first management table are combined by adjusting the number of the written target files, and a large number of files of the first type in the first management table are combined into large files so as to reduce the number of small files, further reduce the number of metadata information in the memory, reduce the overall amount of data stored in the memory, reduce the memory loss and reduce the probability of downtime of the distributed file system, thereby improving the reliability of the distributed file system.
Because the number of small files is reduced, in the distributed file system taking HDFS as an example, the possibility of NameNode service overload is reduced, and the read-write performance of the distributed file system is ensured.
In some embodiments of the present application, the creation module 402 is further configured to: reading mirror image files of a storage system, and obtaining the number of files of a first management table and the storage space value occupied by each file when storing; determining a first memory value based on the number of files and the memory value; and determining the first management table as the management table of the first type file under the condition that the first storage space value is smaller than or equal to a first threshold value.
In some embodiments of the present application, the writing module 406 is specifically configured to: adjusting the number of the written target files by adjusting the partition number; and merging the files of the first management table and writing the merged files into the second management table until the number of the files in the second management table reaches the target number of the files.
In some embodiments of the present application, the update module 408 is specifically configured to: comparing the files in the second management table with the files in the first management table; and storing the file in the second management table to the path in the first management table under the condition that the comparison is consistent.
In some embodiments of the present application, the update module 408 is further configured to perform at least one of: and migrating the file of the first management table to a target position and deleting the second management table.
In some embodiments of the present application, the target location corresponds to the first folder, and the update module 408 is specifically configured to: files of the first management table migrated in the first time period are archived to the first folder.
In some embodiments of the present application, the naming of the first folder is associated with a migration time of the file of the first management table, and the updating module 408 is further configured to: acquiring the archived time length of the first folder based on the naming of the first folder; and in the case that the archived time length is greater than or equal to the second time length, clearing the first folder.
In some embodiments of the present application, a module 402 is created for: acquiring a table structure of a first management table under the condition that the first management table is a management table of a first type of files; creating a second management table based on the table structure of the first management table; wherein, the table structure of the first management table is the same as the table structure of the second management table.
The file management apparatus 400 in the embodiment of the present application may be an electronic device, or may be a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal, or may be other devices than a terminal. By way of example, the electronic device may be a mobile phone, tablet computer, notebook computer, palm computer, vehicle-mounted electronic device, mobile internet appliance (Mobile Internet Device, MID), augmented reality (augmented reality, AR)/Virtual Reality (VR) device, robot, wearable device, ultra-mobile personal computer, UMPC, netbook or personal digital assistant (personal digital assistant, PDA), etc., but may also be a server, network attached storage (Network Attached Storage, NAS), personal computer (personal computer, PC), television (TV), teller machine or self-service machine, etc., and the embodiments of the present application are not limited in particular.
The file management apparatus 400 in the embodiment of the present application may be an apparatus having an operating system. The operating system may be an Android operating system, an ios operating system, or other possible operating systems, and the embodiment of the present application is not limited specifically.
The file management apparatus 400 provided in the embodiment of the present application can implement each process implemented by the embodiment of the file management method in fig. 1, and can achieve the same technical effects, so that repetition is avoided, and no further description is provided here.
As shown in fig. 5, the embodiment of the present application further provides an electronic device 500, which includes a processor 502 and a memory 504, where the memory 504 stores a program or an instruction that can be executed on the processor 502, and the program or the instruction implements each step of the above-mentioned file management method embodiment when executed by the processor 502, and the steps can achieve the same technical effects, so that repetition is avoided, and details are not repeated here.
The electronic device in the embodiment of the application includes the mobile electronic device and the non-mobile electronic device.
Fig. 6 is a schematic diagram of a hardware structure of an electronic device implementing an embodiment of the present application.
As shown in fig. 6, the electronic device 600 includes, but is not limited to: radio frequency unit 601, network module 602, audio output unit 603, input unit 604, sensor 605, display unit 606, user input unit 607, interface unit 608, memory 609, and processor 610.
Those skilled in the art will appreciate that the electronic device 600 may further include a power source (e.g., a battery) for powering the various components, which may be logically connected to the processor 610 by a power management system to perform functions such as managing charge, discharge, and power consumption by the power management system. The electronic device structure shown in fig. 6 does not constitute a limitation of the electronic device, and the electronic device may include more or less components than shown, or may combine certain components, or may be arranged in different components, which are not described in detail herein.
In one embodiment, the processor 610 is configured to: creating a second management table under the condition that the first management table is a management table of the first type of files; reading a file in a first management table; combining the files of the first management table by adjusting the number of the written target files, and writing the files of the first management table into the second management table; and updating the files in the first management table according to the files in the second management table.
In one embodiment, the processor 610 is configured to: reading mirror image files of a storage system, and obtaining the number of files of a first management table and the storage space value occupied by each file when storing; determining a first memory value based on the number of files and the memory value; and determining the first management table as the management table of the first type file under the condition that the first storage space value is smaller than or equal to a first threshold value.
In one embodiment, the processor 610 is configured to: adjusting the number of the written target files by adjusting the partition number; merging and writing the files of the first management table into a second management table; wherein the number of files in the second management table is the target number of files.
In one embodiment, the processor 610 is configured to: comparing the files in the second management table with the files in the first management table; and storing the file in the second management table to the path in the first management table under the condition that the comparison is consistent.
In one embodiment, the processor 610 is further configured to perform at least one of: and migrating the file of the first management table to a target position and deleting the second management table.
In one embodiment, the target location corresponds to a first folder, and the processor 610 is further configured to: files of the first management table migrated in the first time period are archived to the first folder.
In one embodiment, the naming of the first folder is associated with a migration time of the file of the first management table, and the processor 610 is further configured to: acquiring the archived time length of the first folder based on the naming of the first folder; and in the case that the archived time length is greater than or equal to the second time length, clearing the first folder.
In one embodiment, the processor 610 is configured to: acquiring a table structure of a first management table under the condition that the first management table is a management table of a first type of files; creating a second management table based on the table structure of the first management table; wherein, the table structure of the first management table is the same as the table structure of the second management table.
In the embodiment of the application, the files of the first management table are combined by adjusting the number of the written target files, and a large number of first files in the first management table are combined into a large file, so that the number of small files is reduced, the number of metadata information in the memory is further reduced, the integral quantity of data stored in the memory is reduced, the memory loss is reduced, the probability of downtime of the distributed file system is reduced, and the reliability of the distributed file system is improved.
Because the number of small files is reduced, in the distributed file system taking HDFS as an example, the possibility of NameNode service overload is reduced, and the read-write performance of the distributed file system is ensured.
It should be understood that in an embodiment of the present application, the input unit 604 may include a graphics processor (Graphics Processing Unit, GPU) 6041 and a microphone 6042, and the graphics processor 6041 processes image data of still pictures or video obtained by an image capturing apparatus (e.g., a camera) in a video capturing mode or an image capturing mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 607 includes at least one of a touch panel 6071 and other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts of a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and so forth, which are not described in detail herein.
The memory 609 may be used to store software programs as well as various data. The memory 609 may mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area may store an operating system, application programs or instructions (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like. Further, the memory 609 may include volatile memory or nonvolatile memory, or the memory 609 may include both volatile and nonvolatile memory. The nonvolatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable EPROM (EEPROM), or a flash Memory. The volatile memory may be random access memory (Random Access Memory, RAM), static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (ddr SDRAM), enhanced SDRAM (Enhanced SDRAM), synchronous DRAM (SLDRAM), and Direct RAM (DRRAM). Memory 609 in embodiments of the present application includes, but is not limited to, these and any other suitable types of memory.
The processor 610 may include one or more processing units; optionally, the processor 610 integrates an application processor that primarily processes operations involving an operating system, user interface, application programs, etc., and a modem processor that primarily processes wireless communication signals, such as a baseband processor. It will be appreciated that the modem processor described above may not be integrated into the processor 610.
The embodiment of the application also provides a readable storage medium, and the readable storage medium stores a program or an instruction, which when executed by a processor, implements each process of the above embodiment of the file management method, and can achieve the same technical effects, so that repetition is avoided, and no further description is given here.
The processor is a processor in the electronic device in the above embodiment. Readable storage media include computer readable storage media such as computer readable memory ROM, random access memory RAM, magnetic or optical disks, and the like.
The embodiment of the application further provides a chip, the chip comprises a processor and a communication interface, the communication interface is coupled with the processor, the processor is used for running programs or instructions, the processes of the embodiment of the file management method can be realized, the same technical effects can be achieved, and the repetition is avoided, and the description is omitted here.
It should be understood that the chips referred to in the embodiments of the present application may also be referred to as system-on-chip chips, chip systems, or system-on-chip chips, etc.
Embodiments of the present application provide a computer program product stored in a storage medium, where the program product is executed by at least one processor to implement the respective processes of the above-described embodiments of the file management method, and achieve the same technical effects, and for avoiding repetition, a detailed description is omitted herein.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in an opposite order depending on the functions involved, e.g., the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the embodiments of the present application may be embodied in essence or contributing to the prior art in the form of a computer software product stored on a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal (which may be a cell phone, computer, server, or network device, etc.) to perform the methods of the various embodiments of the present application.
The embodiments of the present application have been described above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those having ordinary skill in the art without departing from the spirit of the present application and the scope of the claims, which are to be protected by the present application.
Claims (10)
1. A method of file management comprising:
creating a second management table under the condition that the first management table is a management table of the first type of files;
reading a file in the first management table;
merging the files of the first management table by adjusting the number of the written target files, and writing the files of the first management table into the second management table;
and updating the files in the first management table according to the files in the second management table.
2. The file management method according to claim 1, further comprising:
reading mirror image files of a storage system, and obtaining the number of files of the first management table and the storage space value occupied by each file when storing;
determining a first memory value based on the file number and the memory value;
and under the condition that the first storage space value is smaller than or equal to a first threshold value, determining the first management table as the management table of the first type of files.
3. The file management method according to claim 1, wherein the merging the files of the first management table by adjusting the number of the written target files, writing the files of the first management table into the second management table, comprises:
adjusting the written target file number by adjusting the partition number;
merging the files of the first management table, merging the files of the first management table and writing the merged files into the second management table;
wherein the number of files in the second management table is the target number of files.
4. A file management method according to any one of claims 1 to 3, wherein updating the file in the first management table based on the file in the second management table comprises:
comparing the files in the second management table with the files in the first management table;
and storing the file in the second management table to the path in the first management table under the condition that the comparison is consistent.
5. A file management method according to any one of claims 1 to 3, wherein said updating the file in the first management table from the file in the second management table further comprises performing at least one of:
and migrating the file of the first management table to a target position and deleting the second management table.
6. The method according to claim 5, wherein the target location corresponds to a first folder, and the migrating the file of the first management table to the target location includes:
and archiving the files of the first management table migrated in the first time period to the first folder.
7. The file management method according to claim 6, wherein the naming of the first folder is associated with migration time of the files of the first management table, the file management method further comprising:
acquiring the archived duration of the first folder based on the naming of the first folder;
and clearing the first folder under the condition that the archived time length is greater than or equal to a second time length.
8. A file management method according to any one of claims 1 to 3, wherein, in the case where the first management table is a management table of a first type of file, creating the second management table includes:
acquiring a table structure of a first management table under the condition that the first management table is a management table of a first type of files;
creating the second management table based on a table structure of the first management table;
the table structure of the first management table is the same as that of the second management table.
9. A document management apparatus, comprising:
the creating module is used for creating a second management table under the condition that the first management table is the management table of the first type of files;
the reading module is used for reading the file in the first management table;
the writing module is used for merging the files of the first management table by adjusting the number of the written target files and writing the files of the first management table into the second management table;
and the updating module is used for updating the files in the first management table according to the files in the second management table.
10. An electronic device comprising a processor and a memory storing a program or instructions executable on the processor, which when executed by the processor, implement the steps of the method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311158466.XA CN117194344A (en) | 2023-09-08 | 2023-09-08 | File management method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311158466.XA CN117194344A (en) | 2023-09-08 | 2023-09-08 | File management method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117194344A true CN117194344A (en) | 2023-12-08 |
Family
ID=88986449
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311158466.XA Pending CN117194344A (en) | 2023-09-08 | 2023-09-08 | File management method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117194344A (en) |
-
2023
- 2023-09-08 CN CN202311158466.XA patent/CN117194344A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3098715B1 (en) | System and method for object-based continuous data protection | |
| US9547560B1 (en) | Amortized snapshots | |
| EP3101545B1 (en) | System and method for generating backups of a protected system from a recovery system | |
| US8965850B2 (en) | Method of and system for merging, storing and retrieving incremental backup data | |
| US9811577B2 (en) | Asynchronous data replication using an external buffer table | |
| US8548948B2 (en) | Methods and apparatus for a fine grained file data storage system | |
| EP3159796A1 (en) | System and method for generating backups of a protected system from a recovery system | |
| US10489289B1 (en) | Physical media aware spacially coupled journaling and trim | |
| CN110825694A (en) | Data processing method, device, equipment and storage medium | |
| CN114528127A (en) | Data processing method and device, storage medium and electronic equipment | |
| US10430115B2 (en) | System and method for optimizing multiple packaging operations in a storage system | |
| CN109947712A (en) | Automatically merge method, system, equipment and the medium of file in Computational frame | |
| CN111090701B (en) | Service request processing method, device, readable storage medium and computer equipment | |
| EP3264254B1 (en) | System and method for a simulation of a block storage system on an object storage system | |
| CN114924911B (en) | Method, device, equipment and storage medium for backing up effective data of Windows operating system | |
| US20230409235A1 (en) | File system improvements for zoned storage device operations | |
| CN117194344A (en) | File management method and device and electronic equipment | |
| CN108021562B (en) | Disk storage method and device applied to distributed file system and distributed file system | |
| CN111221817B (en) | Service information data storage method, device, computer equipment and storage medium | |
| WO2019001272A1 (en) | Data migration system and method | |
| CN114385368A (en) | Method and device for flashing pre-written log | |
| US9646014B1 (en) | Systems and methods for selective defragmentation | |
| CN107609038A (en) | Data clearing method and device | |
| US9256626B2 (en) | Maintaining multiple copy versions of component values in a system | |
| US8977814B1 (en) | Information lifecycle management for binding content |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |