WO2019219021A1 - 一种流式文件***中的索引数据更新方法、装置 - Google Patents

一种流式文件***中的索引数据更新方法、装置 Download PDF

Info

Publication number
WO2019219021A1
WO2019219021A1 PCT/CN2019/087002 CN2019087002W WO2019219021A1 WO 2019219021 A1 WO2019219021 A1 WO 2019219021A1 CN 2019087002 W CN2019087002 W CN 2019087002W WO 2019219021 A1 WO2019219021 A1 WO 2019219021A1
Authority
WO
WIPO (PCT)
Prior art keywords
index
time
data
block
update
Prior art date
Application number
PCT/CN2019/087002
Other languages
English (en)
French (fr)
Inventor
叶敏
林鹏
林起芊
汪渭春
Original Assignee
杭州海康威视***技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 杭州海康威视***技术有限公司 filed Critical 杭州海康威视***技术有限公司
Publication of WO2019219021A1 publication Critical patent/WO2019219021A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/13—File access structures, e.g. distributed indices

Definitions

  • the present application relates to the field of data management technologies, and in particular, to a method and an apparatus for updating index data in a streaming file system.
  • the streaming file system is often used to manage streaming data.
  • the streaming file system is mainly composed of a main boot block, a boot boot block, an object block group and a reserved block.
  • the object block group includes a main index block, a backup index block and an object block ( The contents of the primary index block and the standby index block are exactly the same, and each is a backup.
  • the specific structure is shown in Figure 1.
  • the object block is used to store streaming data
  • the index block is used to store index data of streaming data in the object block. (such as data write time, data size, etc.), and the index block has a corresponding relationship with the object block.
  • the index data in the index block corresponding to the object block needs to be updated correspondingly, so that the index data in the index block corresponds to the streaming data in the object block.
  • the index data whose data writing time is within the preset time range is to be updated, since the number of index blocks is not unique, the index block where the index data to be updated is directly cannot be determined. Therefore, the data writing time needs to be found first.
  • the object block stored in the streaming data in the preset time range is then searched for the index block to be processed by the correspondence between the object block and the index block, and then the index data stored in the to-be-processed index block is performed.
  • Update When it is necessary to update a plurality of index data, the above update operation is performed once for each index data separately. Since the update operation of the index data is complicated, the update efficiency is low when updating the plurality of index data.
  • the present application provides an index data update method and apparatus in a streaming file system to solve the problem of low index data update efficiency in a streaming file system.
  • the specific technical solutions are as follows:
  • an embodiment of the present application provides a method for updating a streaming file system index data, where the method includes:
  • index time is a time range determined based on a data write time of the streaming data stored in the object block corresponding to the index block;
  • index data stored in the to-be-processed index block, where the index data includes a data writing time of the streaming data stored in the object block corresponding to the to-be-processed index block;
  • the index data in which the data write time is within the pre-update time range is updated.
  • the pre-update time range is: a time range less than or equal to the acquired pre-update time point.
  • the method further includes:
  • the updating the index data of the data write time within the pre-update time range includes:
  • index time of the to-be-processed index block is completely included in the pre-update time range, all index data in the to-be-processed index block is updated.
  • the updating the index data of the data write time within the pre-update time range includes:
  • index time of the to-be-processed index block is not completely included in the pre-update time range, updating an index time of the to-be-processed index block in the to-be-processed index block and the pre-update time Index data within the overlapping time range of the range.
  • the obtaining the index time of the index block includes:
  • a time range greater than or equal to the earliest write time and less than or equal to the latest write time is determined as an index time of the index block.
  • the embodiment of the present application provides an index data update apparatus in a streaming file system, where the apparatus includes:
  • An index time obtaining module configured to acquire an index time of the index block, where the index time is a time range determined according to a data writing time of the streaming data stored in the object block corresponding to the index block;
  • a first determining module configured to determine whether there is an intersection between the index time and a pre-update time range
  • the to-be-processed index block determining module determines the index block as a to-be-processed index block when there is an intersection between the index time and the pre-update time range;
  • a data reading module configured to read each index data stored in the to-be-processed index block, where the index data includes a data writing time of the streaming data stored in the object block corresponding to the to-be-processed index block;
  • a data update module configured to update index data whose data write time is within the pre-update time range.
  • the pre-update time range is: a time range less than or equal to the acquired pre-update time point.
  • the device further includes:
  • a second determining module configured to determine whether an index time of the to-be-processed index block is completely included in the pre-update time range
  • the data update module is configured to: if the index time of the to-be-processed index block is completely included in the pre-update time range, update all index data in the to-be-processed index block.
  • the data update module is further configured to: if the index time of the to-be-processed index block is not completely included in the pre-update time range, update a data write time in the to-be-processed index block in the The index data within the overlapping time range of the index time of the index block and the pre-update time range is processed.
  • the index time acquiring module is specifically configured to:
  • a time range greater than or equal to the earliest write time and less than or equal to the latest write time is determined as an index time of the index block.
  • an embodiment of the present application provides an electronic device, including a processor and a memory;
  • a memory for storing a computer program
  • the processor is configured to implement the step of updating the index data in any of the above streaming file systems when executing the program stored on the memory.
  • the embodiment of the present application further provides a computer readable storage medium, where the computer readable storage medium stores instructions, when executed on a computer, causing the computer to execute any of the above-described streaming file systems. Index data update method.
  • the index block to be processed is determined by judging whether there is an intersection between the index time and the pre-update time range, and then the index data whose data writing time is within the pre-update time range is updated, and the above data is
  • the index data with the write time within the pre-update time range includes multiple, that is, the synchronization of multiple index data synchronization is implemented, and the time required for the index data update is reduced, thereby improving the index data in the streaming file system. Update efficiency.
  • Figure 1 is a structural diagram of a streaming file system
  • FIG. 2 is a schematic flowchart of a method for updating index data in a streaming file system according to an embodiment of the present disclosure
  • FIG. 3 is a schematic flowchart of a method for updating index data in a streaming file system according to another embodiment of the present disclosure
  • FIG. 4 is a schematic structural diagram of an index data updating apparatus in a streaming file system according to an embodiment of the present disclosure
  • FIG. 5 is a schematic structural diagram of an index data updating apparatus in a streaming file system according to another embodiment of the present disclosure
  • FIG. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
  • the embodiment of the present application provides a method, an apparatus, and an electronic device for updating index data in a streaming file system.
  • the streaming file system consists of a main boot block, a standby boot block, an object block group, and a reserved block.
  • the startup block is used to save key startup information of the file system;
  • the object block group includes a primary index block, a backup index block, and an object block (the contents of the primary index block and the backup index block are exactly the same, and each is a backup), wherein the object block is used.
  • the index block is used to store the index data of the streaming data in the object block, and the index block has a correspondence relationship with the object block.
  • an index block can store index data of multiple object blocks, for example, When the index data size of each object block is 4 KB, and each index block size is 1 MB, each index block can store index data of 256 object blocks; the reserved block is in the startup block or the object block group. Used to replace a damaged boot block or index block when the index block is corrupted.
  • the index data update method in a streaming file system provided by the embodiment of the present application is introduced below.
  • the index data update in the embodiment of the present application may include deletion of the index data, and may also include overlay update of the index data.
  • the method for updating the index data in the streaming file system may specifically include the following steps:
  • Step 101 Obtain an index time of an index block.
  • the index time is a time range determined based on a data write time of the streaming data stored in the object block corresponding to the index block.
  • an index block can store index data of multiple object blocks. For example, in the streaming file system shown in FIG. 1, the index data size of each object block is 4 KB, and each index block size is 1 MB, and each Index data of 256 object blocks can be stored in the index block.
  • the index time of each index block may be a data write time based on the streaming data stored in the object block corresponding to the index block, and a time range established and stored in advance, and the method may be directly obtained during execution.
  • the index time of the index block is a time range indicating a data write time of the stream data stored in the object block corresponding to the index block, and the data write time of the stream data stored in the object block is within the index time.
  • the index data of the streaming data in the object block No. 1 to the object block No. 256 is stored, and the data writing time range of the above streaming data is from 8:00 am to 9:00 am on December 31, 2017.
  • the index time of the index block may be the same as the above time range, that is, from 8:00 am to 9:00 am on December 31, 2017, or may be any time range including the above time range, such as 7 am, December 31, 2017. No. 10:00 am, 7:00 am to 9:00 am on December 31, 2017, or 8:00 am to 11:00 am on December 31, 2017. There is no restriction on this.
  • the time range of the data writing time of the streaming data stored in the object block corresponding to each index block can be obtained, and the index of the writing time within the preset time range can also be directly determined.
  • the index block in which the data is located without searching for the object block stored in the streaming data whose data is written within the above preset time range, and then according to the correspondence between the object block and the index block, the write time is found.
  • the index block where the index data in the preset time range is located improve the efficiency of index block lookups.
  • the step of obtaining an index time of the index block may include:
  • a time range greater than or equal to the earliest write time and less than or equal to the latest write time is determined as the index time of the index block.
  • the index block of the index block is set to a time range greater than or equal to the earliest write time and less than or equal to the latest write time, which can avoid adjacent index blocks. There is an intersection between the corresponding index times.
  • the index block storing the index data can be determined according to the index time of each index block, and the index block is further improved. Index block lookup efficiency.
  • Step 102 Determine whether there is an intersection between the time index and the pre-update time range.
  • the time index here refers to the index time of the index block obtained in step 101.
  • the step may be determining whether there is an intersection between the acquired index time and the pre-update time range.
  • the pre-update time range may be a specific time range that the user needs to perform index data update according to requirements, for example, when the user needs to write the data at the time between 8:00 am and 9:00 am on December 31, 2017.
  • the pre-update time range is: 8:00 am to 9:00 am on December 31, 2017; the pre-update time range may also be all time periods except a specific time range specified by the user.
  • the pre-updating time range is: as early as December 31, 2017
  • the time range of 8:00 am and the time range later than 9:00 am on December 31, 2017; the pre-update time range may also be the time range before a specific time point specified by the user.
  • the pre-update time range may be: a time range less than or equal to the acquired pre-update time point.
  • the pre-update time point obtained is 8:00 am on December 31, 2017.
  • the pre-update time range may be a time range earlier than or equal to 8:00 am on December 31, 2017.
  • Step 103 If there is an intersection, determine the index block as the index block to be processed.
  • the index time and the pre-update time range may include the following three cases: the index time of the to-be-processed index block is completely included in the pre-update time range; the pre-update time range is completely included in the index time of the index block to be processed. Within; there is only a partial overlap time range between the index time of the index block to be processed and the pre-update time range.
  • Step 104 Read each index data stored in the index block to be processed.
  • the index data includes a data write time of the streaming data stored in the object block corresponding to the index block to be processed. While the streaming data is written into the object block, the data writing time of the streaming data is stored in the index block corresponding to the object block, and the data size of the streaming data may also be stored in the index block. For the verification information and the like, the data writing time, the data size, and the verification information may be included in the index data.
  • Step 105 Update the index data whose data writing time is within the pre-update time range.
  • step 103 The case where there is an intersection between the index time and the pre-update time range in step 103 can be further summarized into two types: the index time of the index block to be processed is completely included in the pre-update time range; the index time of the index block to be processed is not Fully included in the pre-update time range.
  • the method before updating the index data whose data writing time is within the pre-update time range, the method further includes:
  • the index data of the update data write time within the pre-update time range may include:
  • the index block to be processed is index block 1, where the index time of index block 1 is from 7:00 am to 8:00 am on December 31, 2017, if the pre-update time range is from 0:00 am to 5:00 am on December 31, 2017. 11: At this time, the index time of the index block to be processed is completely included in the pre-update time range, that is, the data writing time in the index data stored in the index block 1 is within the pre-update time range, All index data in the index block (ie index block 1) to be processed can be updated.
  • the index block to be processed is index block 2, where the index time of index block 2 is from 9:00 am to 11:00 am on December 31, 2017, if the pre-update time range is from 10:00 am to 5:00 am on December 31, 2017.
  • the index time of the index block to be processed is not completely included in the pre-update time range, that is, only a part of the overlap time range between the index time and the pre-update time range in the index block 2 (2017 12 On the 31st of the month, from 10:00 am to 11:00 am, the index data within the overlapping time range can be updated.
  • the index data stored in the to-be-processed index block is read, when the index data whose data write time is within the pre-update time range is updated, it is determined whether the index time of the to-be-processed index block is completely included in the pre-update time range.
  • the data write time of all index data in the index block to be processed must be within the pre-update time range. In this case, the index block may be directly processed. All the index data is updated, which can further shorten the update time of the index data and improve the update efficiency.
  • the steps 101 to 105 may be sequentially performed for each index block in the file system according to a preset order; or the index may be completed as follows.
  • Data update After obtaining the index time of all the index blocks in the file system, it is determined whether there is an intersection between all the index times and the pre-update time range; and the index blocks corresponding to the index times of the intersections are respectively determined as the to-be-processed index blocks. And performing the step 104 and the step 105 in parallel on the index block to be processed, or selecting an index block from the to-be-processed index block each time in the selection order set by the user, and then performing the index block on the selected index block. Step 104 and step 105, until all the above-mentioned index blocks to be processed are updated.
  • the index time of the index block determined based on the data writing time of the streaming data stored in the object block corresponding to the index block is acquired. Determining an index block having an intersection between the index time and the pre-update time range as a to-be-processed index block, reading each index data stored in the to-be-processed index block; updating an index in which the data write time is within a pre-update time range data.
  • the index block having the intersection between the index time and the pre-update time range is determined as the index block to be processed, and the index data whose data writing time is within the pre-update time range is updated, and the above data writing time is pre-
  • the index data within the update time range includes multiple, that is, the synchronization of multiple index data synchronization is implemented, and the time required for the index data update is reduced, thereby improving the update efficiency of the index data in the streaming file system.
  • another method for updating the index file data of the streaming file system includes the following steps:
  • Step 201 Acquire an index time of an index block, where the index time is a time range determined according to a data writing time of the streaming data stored in the object block corresponding to the index block.
  • Step 202 Determine whether there is an intersection between the index time and the pre-update time range. If there is an intersection, step 203 is performed; if there is no intersection, step 206 is performed.
  • Step 203 If there is an intersection, determine the index block as the index block to be processed.
  • Step 204 Read each index data stored in the to-be-processed index block, where the index data includes data writing time of the streaming data stored in the object block corresponding to the index block to be processed.
  • Step 205 Update index data whose data write time is within a pre-update time range.
  • Step 201, step 202, step 203, step 204, and step 205 respectively correspond to the specific contents in step 101, step 102, step 103, step 104, and step 105, and details are not described herein again.
  • Step 206 If there is no intersection, end the intersection judgment process between the pre-update time range and the index time, and enter an intersection judgment process for the next pre-update time range and the index time.
  • the pre-update time When there is no intersection between the index time and the pre-update time range, it indicates that the index data that needs to be updated within the preset update time range is not stored in the streaming file system, and the pre-update time may be ended.
  • the intersection judgment process between the range and the index time thereby ending the index data update process.
  • the process of determining the intersection between the next pre-update time range and the index time may be entered, thereby completing the next index data update process.
  • the index time of the index block determined based on the data writing time of the streaming data stored in the object block corresponding to the index block is acquired. Determining an index block having an intersection between the index time and the pre-update time range as a to-be-processed index block, reading each index data stored in the to-be-processed index block; updating an index in which the data write time is within a pre-update time range Data; when there is no intersection between the index time and the pre-update time range, the intersection judgment process between the pre-update time range and the index time is ended, and the intersection judgment for the next pre-update time range and the index time is entered. Process.
  • the index block having the intersection between the index time and the pre-update time range is determined as the index block to be processed, and the index data whose data writing time is within the pre-update time range is updated, and the above data writing time is pre-
  • the index data within the update time range includes multiple, that is, the synchronization of multiple index data synchronization is implemented, and the time required for the index data update is reduced, thereby improving the update efficiency of the index data in the streaming file system.
  • the embodiment of the present application provides an index data updating device in the streaming file system, and the structure thereof
  • the schematic diagram is shown in Figure 4 and includes:
  • the index time obtaining module 301 is configured to obtain an index time of the index block, where the index time is a time range determined according to a data writing time of the streaming data stored in the object block corresponding to the index block;
  • the first determining module 302 is configured to determine whether there is an intersection between the index time and the pre-update time range;
  • the to-be-processed index block determining module 303 determines the index block as the to-be-processed index block when there is an intersection between the index time and the pre-update time range;
  • the data reading module 304 is configured to read each index data stored in the index block to be processed, where the index data includes data writing time of the streaming data stored in the object block corresponding to the index block to be processed;
  • the data update module 305 is configured to update index data whose data write time is within a pre-update time range.
  • the pre-update time range is: a time range less than or equal to the acquired pre-update time point.
  • the embodiment of the present application further provides an index data updating apparatus in a streaming file system, and the apparatus may include:
  • the index time obtaining module 501 is configured to acquire an index time of the index block, where the index time is a time range determined according to a data writing time of the streaming data stored in the object block corresponding to the index block;
  • the first determining module 502 is configured to determine whether there is an intersection between the index time and the pre-update time range;
  • the to-be-processed index block determining module 503 determines the index block as the to-be-processed index block when there is an intersection between the index time and the pre-update time range;
  • the data reading module 504 is configured to read each index data stored in the index block to be processed, where the index data includes data writing time of the streaming data stored in the object block corresponding to the index block to be processed;
  • a data update module 505, configured to update index data whose data write time is within a pre-update time range;
  • the second determining module 506 is configured to determine whether the index time of the index block to be processed is completely included in the pre-update time range.
  • the data update module 505 is specifically configured to update all index data in the to-be-processed index block if the index time of the index block to be processed is completely included in the pre-update time range.
  • the data update module 505 is further configured to: if the index time of the to-be-processed index block is not completely included in the pre-update time range, update the index time and the pre-update of the data write time in the to-be-processed index block in the to-be-processed index block. Index data within the overlapping time range of the time range.
  • the index time obtaining module 501 is specifically configured to: obtain an earliest write time and a latest write time in the data write time in the index data of the index block; and be greater than or equal to the earliest write time, and less than or equal to The time range of the latest write time is determined as the index time of the index block.
  • the index time obtaining module 501 acquires the index time of the index block; the first determining module 502 determines whether there is an intersection between the index time and the pre-update time range; When there is an intersection between the index time and the pre-update time range, the to-be-processed index block determining module 503 determines the index block as the to-be-processed index block; then the data reading module 504 reads the index data stored in the to-be-processed index block; The data update module 505 updates the index data whose data write time is within the pre-update time range.
  • the index data in the range includes multiple, that is, the synchronization of multiple index data synchronization is implemented, and the time required for the index data update is reduced, thereby improving the update efficiency of the index data in the streaming file system.
  • the embodiment of the present application further provides an electronic device, as shown in FIG. 6, including the processor 601 and Memory 602, wherein
  • a memory 602 configured to store a computer program
  • the processor 601 is configured to implement an index data update method in the streaming file system provided by the embodiment of the present application when the program stored in the memory 602 is executed.
  • index time is a time range determined according to a data writing time of the streaming data stored in the object block corresponding to the index block;
  • the index block is determined as a to-be-processed index block
  • Update index data whose data write time is within the pre-update time range.
  • the memory may include a random access memory (English: Random Access Memory, RAM for short), and may also include a non-volatile memory (Non-volatile Memory, NVM for short), such as at least one disk storage. Further, the memory may also be at least one storage device located away from the aforementioned processor.
  • RAM Random Access Memory
  • NVM Non-volatile Memory
  • the above processor may be a general-purpose processor, including a central processing unit (English: Central Processing Unit, CPU for short), a network processor (English: Network Processor, NP for short), or a digital signal processor. :Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices , discrete gates or transistor logic devices, discrete hardware components.
  • a central processing unit English: Central Processing Unit, CPU for short
  • a network processor English: Network Processor, NP for short
  • DSP Digital Signal Processor
  • ASIC Application Specific Integrated Circuit
  • FPGA Field-Programmable Gate Array
  • the processor 601 and the memory 602 may be connected by a communication bus such as an address bus, a data bus, a control bus, etc.
  • the communication bus may be a Peripheral Component Interconnect (PCI) bus or an extended industry standard structure ( English: Extended Industry Standard Architecture, referred to as EISA) bus.
  • PCI Peripheral Component Interconnect
  • EISA Extended Industry Standard Architecture
  • the electronic device can communicate with other external devices through a communication interface.
  • communication between the processor 601 and the memory 602, and between the electronic device and other external devices can also be performed through the wireless connection mode of the wireless module.
  • the method is: acquiring an index time of an index block determined based on a data writing time of the streaming data stored in the object block corresponding to the index block; and setting the index time and the pre-updating time An index block having an intersection between the ranges is determined as a to-be-processed index block, and each index data stored in the to-be-processed index block is read; and index data whose data write time is within a pre-update time range is updated.
  • the index block having the intersection between the index time and the pre-update time range is determined as the index block to be processed, and the index data whose data writing time is within the pre-update time range is updated, and the above data writing time is pre-
  • the index data within the update time range includes multiple, that is, the synchronization of multiple index data synchronization is implemented, and the time required for the index data update is reduced, thereby improving the update efficiency of the index data in the streaming file system.
  • a computer readable storage medium having stored therein instructions that, when run on a computer, cause the computer to perform any of the above embodiments
  • the index data update method in the above streaming file system is provided.
  • the computer readable storage medium provided by the embodiment of the present application adopts a method of: acquiring an index time of an index block determined based on a data writing time of the streaming data stored in the object block corresponding to the index block; The index block having an intersection between the pre-update time ranges is determined as the to-be-processed index block, and each index data stored in the to-be-processed index block is read; and the index data whose data writing time is within the pre-update time range is updated.
  • the index block having the intersection between the index time and the pre-update time range is determined as the index block to be processed, and the index data whose data writing time is within the pre-update time range is updated, and the above data writing time is pre-
  • the index data within the update time range includes multiple, that is, the synchronization of multiple index data synchronization is implemented, and the time required for the index data update is reduced, thereby improving the update efficiency of the index data in the streaming file system.
  • a computer program product comprising instructions which, when run on a computer, cause the computer to execute index data in any of the above-described streaming file systems of the above embodiments Update method.
  • the computer program product including the instruction provided by the embodiment of the present application adopts a method of: acquiring an index time of an index block determined based on a data writing time of the streaming data stored in the object block corresponding to the index block; The index block having an intersection with the pre-update time range is determined as the to-be-processed index block, and each index data stored in the to-be-processed index block is read; and the index data whose data writing time is within the pre-update time range is updated.
  • the index block having the intersection between the index time and the pre-update time range is determined as the index block to be processed, and the index data whose data writing time is within the pre-update time range is updated, and the above data writing time is pre-
  • the index data within the update time range includes multiple, that is, the synchronization of multiple index data synchronization is implemented, and the time required for the index data update is reduced, thereby improving the update efficiency of the index data in the streaming file system.
  • a computer program product includes one or more computer instructions.
  • the computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable device.
  • the computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, computer instructions can be wired from a website site, computer, server or data center (eg Coaxial cable, optical fiber, digital subscriber line (English: Digital Subscriber Line, DSL) or wireless (such as infrared, wireless, microwave, etc.) to another website, computer, server or data center.
  • the computer readable storage medium described above can be any available media that can be accessed by a computer or a data storage device such as a server, data center, or the like that includes one or more available media.
  • the above usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a digital video disc (DVD), or a semiconductor medium (for example, a solid state hard disk (English: Solid). State Disk, referred to as: SSD)).
  • a magnetic medium for example, a floppy disk, a hard disk, a magnetic tape
  • an optical medium for example, a digital video disc (DVD)
  • DVD digital video disc
  • semiconductor medium for example, a solid state hard disk (English: Solid). State Disk, referred to as: SSD)

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

一种流式文件***中的索引数据更新方法、装置，包括：获取索引块的索引时间；判断索引时间与预更新时间范围之间是否存在交集；若存在交集，则将索引块确定为待处理索引块；读取待处理索引块中存储的各索引数据；更新数据写入时间在预更新时间范围之内的索引数据。通过本方案可以提高流式文件***中索引数据的更新效率。

Description

一种流式文件***中的索引数据更新方法、装置
本申请要求于2018年05月16日提交中国专利局、申请号为201810470680.1、发明名称为“流式文件***中的索引数据更新方法、装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及数据管理技术领域，特别是涉及一种流式文件***中的索引数据更新方法、装置。
背景技术
近年来，随着视频监控技术的不断发展，所需要处理的视频数据的数据量越来越大，例如在安防行业，视频数据是实时采集的，所采集的视频数据在时间轴上具有连续性，并且随着时间的延续，视频数据会无限增长，通常将这一类数据称为流式数据。
常采用流式文件***对流式数据进行管理，流式文件***主要由主启动块、备启动块、对象块组及保留块组成，对象块组中包括主索引块、备索引块和对象块(主索引块和备索引块的内容完全一样，互为备份)，具体结构图如图1所示，其中，对象块用于存储流式数据，索引块用于存储对象块中流式数据的索引数据(如数据写入时间、数据大小等)，且索引块与对象块之间具有对应关系。
当对象块中存储的流式数据发生改变时，需要对与该对象块对应的索引块中的索引数据进行相应更新，以达到索引块中的索引数据与对象块中的流式数据相对应的目的。具体地，若要更新数据写入时间在预设时间范围内的索引数据，由于索引块数量不唯一，无法直接确定需要更新的索引数据所在的索引块，因此，需先找到数据写入时间在上述预设时间范围内的流式数据所存储的对象块，然后通过该对象块与索引块间的对应关系，查找到待处理的索引块，再对该待处理索引块中存储的索引数据进行更新。当需要对多个索引数据进行更新时，需分别针对每一个索引数据进行一次上述更新操作，由于上述索引数据的更新操作实现复杂，导致对多个索引数据进行更新时，更新效率低。
发明内容
本申请提供了一种流式文件***中的索引数据更新方法、装置，以解决流式文件***中的索引数据更新效率低的问题。具体技术方案如下：
第一方面，本申请实施例提供了一种流式文件***索引数据更新方法，所述方法包括：
获取索引块的索引时间，所述索引时间为基于所述索引块对应的对象块中存储的流式数据的数据写入时间确定的时间范围；
判断所述索引时间与预更新时间范围之间是否存在交集；
若存在交集，则将所述索引块确定为待处理索引块；
读取所述待处理索引块中存储的各索引数据，所述索引数据包括所述待处理索引块对应的对象块中存储的流式数据的数据写入时间；
更新数据写入时间在所述预更新时间范围之内的索引数据。
进一步的，所述预更新时间范围为：小于或等于获取到的预更新时间点的时间范围。
进一步的，在所述更新数据写入时间在所述预更新时间范围之内的索引数据之前，所述方法还包括：
判断所述待处理索引块的索引时间是否完全包含于所述预更新时间范围；
所述更新所述数据写入时间在所述预更新时间范围之内的索引数据，包括：
若所述待处理索引块的索引时间完全包含于所述预更新时间范围，则更新所述待处理索引块中的所有索引数据。
进一步的，所述更新所述数据写入时间在所述预更新时间范围之内的索引数据，包括：
若所述待处理索引块的索引时间不完全包含于所述预更新时间范围，则更新所述待处理索引块中数据写入时间在所述待处理索引块的索引时间与所 述预更新时间范围的重叠时间范围之内的索引数据。
进一步的，所述获取索引块的索引时间，包括：
获取所述索引块的索引数据中数据写入时间中的最早写入时间及最晚写入时间；
将大于或等于所述最早写入时间、且小于或等于所述最晚写入时间的时间范围，确定为所述索引块的索引时间。
第二方面，本申请实施例提供了一种流式文件***中的索引数据更新装置，装置包括：
索引时间获取模块，用于获取索引块的索引时间，所述索引时间为基于所述索引块对应的对象块中存储的流式数据的数据写入时间确定的时间范围；
第一判断模块，用于判断所述索引时间与预更新时间范围之间是否存在交集；
待处理索引块确定模块，当所述索引时间与预更新时间范围之间存在交集时，将所述索引块确定为待处理索引块；
数据读取模块，用于读取所述待处理索引块中存储的各索引数据，所述索引数据包括所述待处理索引块对应的对象块中存储的流式数据的数据写入时间；
数据更新模块，用于更新数据写入时间在所述预更新时间范围之内的索引数据。
进一步的，所述预更新时间范围为：小于或等于获取到的预更新时间点的时间范围。
进一步的，所述装置还包括：
第二判断模块，用于判断所述待处理索引块的索引时间是否完全包含于所述预更新时间范围；
所述数据更新模块，具体用于若所述待处理索引块的索引时间完全包含 于所述预更新时间范围，则更新所述待处理索引块中的所有索引数据。
进一步的，所述数据更新模块，还用于若所述待处理索引块的索引时间不完全包含于所述预更新时间范围，则更新所述待处理索引块中数据写入时间在所述待处理索引块的索引时间与所述预更新时间范围的重叠时间范围之内的索引数据。
进一步的，所述索引时间获取模块，具体用于：
获取所述索引块的索引数据中数据写入时间中的最早写入时间及最晚写入时间；
将大于或等于所述最早写入时间、且小于或等于所述最晚写入时间的时间范围，确定为所述索引块的索引时间。
第三方面，本申请实施例提供了一种电子设备，包括处理器和存储器；
存储器，用于存放计算机程序；
处理器，用于执行存储器上所存放的程序时，实现上述任一流式文件***中的索引数据更新方法的步骤。
第四方面，本申请实施例还提供了一种计算机可读存储介质，所述计算机可读存储介质中存储有指令，当其在计算机上运行时，使得计算机执行上述任一流式文件***中的索引数据更新方法。
由上述的技术方案可见，通过判断索引时间与预更新时间范围间是否存在交集，确定出待处理索引块，再将数据写入时间在预更新时间范围之内的索引数据进行更新，而上述数据写入时间在预更新时间范围之内的索引数据包括多个，即实现了对多个索引数据同步进行更新，减小了索引数据更新所需的时间，从而提高了流式文件***中索引数据的更新效率。
附图说明
为了更清楚地说明本申请实施例和现有技术的技术方案，下面对实施例和现有技术中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为流式文件***的结构图；
图2为本申请一个实施例提供的流式文件***中的索引数据更新方法的流程示意图；
图3为本申请另一个实施例提供的流式文件***中的索引数据更新方法的流程示意图；
图4为本申请一个实施例提供的流式文件***中的索引数据更新装置的结构示意图；
图5为本申请另一个实施例提供的流式文件***中的索引数据更新装置的结构示意图；
图6为本申请一个实施例提供的电子设备的结构示意图。
具体实施方式
为使本申请的目的、技术方案、及优点更加清楚明白，以下参照附图并举实施例，对本申请进一步详细说明。显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
为了提高流式文件***中的索引数据更新效率，本申请实施例提供了一种流式文件***中的索引数据更新方法、装置及电子设备。
针对具有时间连续性的流式数据，常采用流式文件***对该类数据进行管理，首先，结合图1对流式文件***的结构组成进行介绍。流式文件***由主启动块、备启动块、对象块组及保留块组成。启动块用于保存文件***的关键启动信息；对象块组中包括主索引块、备索引块和对象块(主索引块和备索引块的内容完全一样，互为备份)，其中，对象块用于存储流式数据，索引块则用于存放对象块中流式数据的索引数据，且索引块与对象块间具有对应关系，一般情况下，一个索引块可以存储多个对象块的索引数据，例如，当每个对象块的索引数据大小为4KB，每个索引块大小为1MB时，则每个索引块可以存储256个对象块的索引数据；预留块是在启动块或者对象块组中的索引块损坏时，用于替换损坏的启动块或索引块。
下面对本申请实施例所提供的一种流式文件***中的索引数据更新方法进行介绍。
本申请实施例中的索引数据更新可以包括对索引数据的删除，也可以包括对索引数据的覆盖更新。
如图2所示，本申请实施例所提供的一种流式文件***中的索引数据更新方法，具体可以包括如下步骤：
步骤101，获取索引块的索引时间。
其中，索引时间为基于索引块对应的对象块中存储的流式数据的数据写入时间确定的时间范围。一般情况下，一个索引块可以存储多个对象块的索引数据，例如图1所示的流式文件***中，每个对象块的索引数据大小为4KB，每个索引块大小为1MB，则每个索引块中可以存储256个对象块的索引数据。
各索引块的索引时间可以为基于该索引块对应的对象块中存储的流式数据的数据写入时间，预先建立并存储的时间范围，本方法在执行过程中，可以直接获取。
索引块的索引时间为表征该索引块对应的对象块中存储的流式数据的数据写入时间的时间范围，上述对象块中存储的流式数据的数据写入时间均在该索引时间内。例如，1号索引块中存储有1号对象块至256号对象块中流式数据的索引数据，上述流式数据的数据写入时间范围为2017年12月31日上午8点至上午9点，则索引块的索引时间可以与上述时间范围相同，即2017年12月31日上午8点至上午9点，也可以为包含上述时间范围的任一时间范围，如2017年12月31日上午7点至上午10点、2017年12月31日上午7点至上午9点或者2017年12月31日上午8点至上午11点等，对此不作限制。
通过获取各索引块的索引时间，可以得到各索引块对应的对象块中存储的流式数据的数据写入时间所属的时间范围，同时也可以直接确定写入时间在预设时间范围内的索引数据所在的索引块，而无需通过查找数据写入时间在上述预设时间范围内的流式数据所存储的对象块，再根据该对象块与索引块间的对应关系，查找到写入时间在预设时间范围内的索引数据所在的索引 块。提高了索引块查找的效率。
进一步的，获取索引块的索引时间的步骤，可以包括：
获取索引块的索引数据中数据写入时间中的最早写入时间及最晚写入时间；
将大于或等于最早写入时间、且小于或等于最晚写入时间的时间范围，确定为索引块的索引时间。
由于流式数据在时间轴上具有连续性，因此，将索引块的索引时间设定为大于或等于最早写入时间、且小于或等于最晚写入时间的时间范围，可以避免相邻索引块对应的索引时间之间存在交集的情况，同时，当已知某一索引数据中的数据写入时间时，可以根据各索引块的索引时间确定出唯一的存储该索引数据的索引块，进一步提高了索引块查找效率。
步骤102，判断时间索引与预更新时间范围之间是否存在交集。
这里的时间索引就是指步骤101中所获取的索引块的索引时间。具体的，本步骤可以是判断所获取的索引时间与预更新时间范围之间是否存在交集。
预更新时间范围可以是用户根据需要预先给定的需要进行索引数据更新的某一具体时间范围，例如，当用户需要对数据写入时间处于2017年12月31日上午8点至上午9点间的索引数据进行更新时，预更新时间范围即为：2017年12月31日上午8点至上午9点；预更新时间范围也可以是出用户指定的某一具体时间范围之外的所有时间段，例如，当用户需要对数据写入时间处于2017年12月31日上午8点至上午9点之外的所有索引数据进行更新时，预更新时间范围即为：早于2017年12月31日上午8点的时间范围，及晚于2017年12月31日上午9点的时间范围；预更新时间范围还可以是用户指定的某一具体时间点之前的时间范围。
进一步的，预更新时间范围可以为：小于或等于获取到的预更新时间点的时间范围。
例如，获取到的预更新时间点为2017年12月31日上午8点，此时，预更新时间范围即可以为早于或者等于2017年12月31日上午8点的时间范围。
步骤103，若存在交集，则将索引块确定为待处理索引块。
索引时间与预更新时间范围之间存在交集，可以包括下面3种情况：待处理索引块的索引时间完全包含于预更新时间范围之内；预更新时间范围完全包含于待处理索引块的索引时间之内；待处理索引块的索引时间与预更新时间范围之间仅有部分重叠时间范围。
步骤104，读取待处理索引块中存储的各索引数据。
其中，索引数据包括待处理索引块对应的对象块中存储的流式数据的数据写入时间。在将流式数据写入对象块的同时，会将该流式数据的数据写入时间存储在与对象块对应的索引块中，同时存入索引块的还可以有流式数据的数据大小以及校验信息等，上述数据写入时间、数据大小及校验信息等均可以包含于索引数据之内。
步骤105，更新数据写入时间在预更新时间范围之内的索引数据。
可以将步骤103中索引时间与预更新时间范围之间存在交集的情况进一步的概括为两种：待处理索引块的索引时间完全包含于预更新时间范围之内；待处理索引块的索引时间不完全包含于预更新时间范围之内。
进一步的，在更新数据写入时间在预更新时间范围之内的索引数据之前，方法还包括：
判断待处理索引块的索引时间是否完全包含于预更新时间范围；
更新数据写入时间在预更新时间范围之内的索引数据，可以包括：
若待处理索引块的索引时间完全包含于预更新时间范围，则更新待处理索引块中的所有索引数据。例如，待处理索引块为索引块1，其中索引块1的索引时间为2017年12月31日上午7点至上午8点，若预更新时间范围为2017年12月31日上午0点至上午11点，此时，待处理索引块的索引时间完全包含于预更新时间范围内，也就是说，上述索引块1中存储的索引数据中的数据写入时间均在预更新时间范围内，则可以对待处理索引块(即索引块1)中的所有索引数据进行更新。
进一步的，若待处理索引块的索引时间不完全包含于预更新时间范围， 则更新待处理索引块中数据写入时间在待处理索引块的索引时间与预更新时间范围的重叠时间范围之内的索引数据。例如，待处理索引块为索引块2，其中索引块2的索引时间为2017年12月31日上午9点至上午11点，若预更新时间范围为2017年12月31日上午10点至上午11点，此时，待处理索引块的索引时间不完全包含于预更新时间范围，也就是说，上述索引块2中的索引时间与预更新时间范围之间只有一部分重叠时间范围(2017年12月31日上午10点至上午11点)，则可以更新该重叠时间范围之内的索引数据。
在读取待处理索引块中存储的索引数据之后，对数据写入时间在预更新时间范围之内的索引数据进行更新时，判断待处理索引块的索引时间是否完全包含于预更新时间范围，当待处理索引块的索引时间完全包含于预更新时间范围时，待处理索引块中的所有索引数据的数据写入时间必然在预更新时间范围之内，此时，可以直接对处理索引块中的所有索引数据进行更新，可以进一步缩短索引数据的更新时间，提高更新效率。
本申请实施例提供的流式文件***中的索引数据更新方法中，可以按照预设顺序，针对文件***中的每一个索引块依次执行上述101至105的步骤；也可以按照下述方式完成索引数据的更新：在获取文件***中所有索引块的索引时间后，判断上述所有索引时间与预更新时间范围之间是否存在交集；将存在交集的索引时间分别对应的索引块确定为待处理索引块，再对上述待处理索引块，同时并行地执行步骤104和步骤105，或者按照用户设定的选择顺序每次从上述待处理索引块中选择一个索引块，然后对该选择到的索引块执行步骤104和步骤105，直至将上述待处理索引块全部更新完毕。
在本申请实施例提供的图1所示的流式文件***中的索引数据更新方法中，获取基于索引块对应的对象块中存储的流式数据的数据写入时间确定的索引块的索引时间；将索引时间与预更新时间范围之间存在交集的索引块确定为待处理索引块，读取待处理索引块中存储的各索引数据；更新数据写入时间在预更新时间范围之内的索引数据。通过将索引时间与预更新时间范围之间存在交集的索引块确定为待处理索引块，再将数据写入时间在预更新时间范围之内的索引数据进行更新，而上述数据写入时间在预更新时间范围之内的索引数据包括多个，即实现了对多个索引数据同步进行更新，减小了索引数据更新所需的时间，从而提高了流式文件***中索引数据的更新效率。
如附图3所示，本申请实施例提供的另一种流式文件***索引数据更新方法，具体包括如下步骤：
步骤201，获取索引块的索引时间，索引时间为基于索引块对应的对象块中存储的流式数据的数据写入时间确定的时间范围。
步骤202，判断索引时间与预更新时间范围之间是否存在交集。若存在交集，则执行步骤203；若不存在交集，则执行步骤206。
步骤203，若存在交集，将索引块确定为待处理索引块。
步骤204，读取待处理索引块中存储的各索引数据，索引数据包括待处理索引块对应的对象块中存储的流式数据的数据写入时间。
步骤205，更新数据写入时间在预更新时间范围之内的索引数据。
步骤201、步骤202、步骤203、步骤204及步骤205分别对应于步骤101、步骤102、步骤103、步骤104及步骤105中的具体内容，在此不再赘述。
步骤206，若不存在交集，结束该预更新时间范围与索引时间之间的交集判断流程，并进入针对下一预更新时间范围与索引时间之间的交集判断流程。
当索引时间与预更新时间范围之间不存在交集时，表明流式文件***中未存储有在本次预设更新时间范围内的需要进行更新的索引数据，此时，可以结束该预更新时间范围与索引时间之间的交集判断流程，进而结束索引数据更新流程。用户设定了下一预设更新时间范围时，可以进入针对下一预更新时间范围与索引时间之间的交集判断流程，进而完成下一次的索引数据更新过程。
在本申请实施例提供的图2所示的流式文件***中的索引数据更新方法中，获取基于索引块对应的对象块中存储的流式数据的数据写入时间确定的索引块的索引时间；将索引时间与预更新时间范围之间存在交集的索引块确定为待处理索引块，读取待处理索引块中存储的各索引数据；更新数据写入时间在预更新时间范围之内的索引数据；当索引时间与预更新时间范围之间不存在交集时，结束该预更新时间范围与索引时间之间的交集判断流程，并进入针对下一预更新时间范围与索引时间之间的交集判断流程。通过将索引 时间与预更新时间范围之间存在交集的索引块确定为待处理索引块，再将数据写入时间在预更新时间范围之内的索引数据进行更新，而上述数据写入时间在预更新时间范围之内的索引数据包括多个，即实现了对多个索引数据同步进行更新，减小了索引数据更新所需的时间，从而提高了流式文件***中索引数据的更新效率。
基于同一发明构思，根据本申请上述实施例提供的一种流式文件***中的索引数据更新方法，相应地，本申请实施例提供了一种流式文件***中的索引数据更新装置，其结构示意图如图4所示，包括：
索引时间获取模块301，用于获取索引块的索引时间，索引时间为基于索引块对应的对象块中存储的流式数据的数据写入时间确定的时间范围；
第一判断模块302，用于判断索引时间与预更新时间范围之间是否存在交集；
待处理索引块确定模块303，当索引时间与预更新时间范围之间存在交集时，将索引块确定为待处理索引块；
数据读取模块304，用于读取待处理索引块中存储的各索引数据，索引数据包括待处理索引块对应的对象块中存储的流式数据的数据写入时间；
数据更新模块305，用于更新数据写入时间在预更新时间范围之内的索引数据。
进一步的，预更新时间范围为：小于或等于获取到的预更新时间点的时间范围。
进一步的，如图5所示，本申请实施例还提供了一种流式文件***中的索引数据更新装置，该装置可以包括：
索引时间获取模块501，用于获取索引块的索引时间，索引时间为基于索引块对应的对象块中存储的流式数据的数据写入时间确定的时间范围；
第一判断模块502，用于判断索引时间与预更新时间范围之间是否存在交集；
待处理索引块确定模块503，当索引时间与预更新时间范围之间存在交集 时，将索引块确定为待处理索引块；
数据读取模块504，用于读取待处理索引块中存储的各索引数据，索引数据包括待处理索引块对应的对象块中存储的流式数据的数据写入时间；
数据更新模块505，用于更新数据写入时间在预更新时间范围之内的索引数据；
第二判断模块506，用于判断待处理索引块的索引时间是否完全包含于预更新时间范围。
数据更新模块505，具体用于若待处理索引块的索引时间完全包含于预更新时间范围，则更新待处理索引块中的所有索引数据。
进一步的，数据更新模块505，还用于若待处理索引块的索引时间不完全包含于预更新时间范围，则更新待处理索引块中数据写入时间在待处理索引块的索引时间与预更新时间范围的重叠时间范围之内的索引数据。
进一步的，索引时间获取模块501，具体用于：获取索引块的索引数据中数据写入时间中的最早写入时间及最晚写入时间；将大于或等于最早写入时间、且小于或等于最晚写入时间的时间范围，确定为索引块的索引时间。
本申请实施例提供的流式文件***中的索引数据更新装置中，索引时间获取模块501获取索引块的索引时间；第一判断模块502判断索引时间与预更新时间范围之间是否存在交集；当索引时间与预更新时间范围之间存在交集时，待处理索引块确定模块503将上述索引块确定为待处理索引块；然后数据读取模块504读取待处理索引块中存储的各索引数据；数据更新模块505更新数据写入时间在预更新时间范围之内的索引数据。通过判断索引时间与预更新时间范围间是否存在交集，确定出待处理索引块，再将数据写入时间在预更新时间范围之内的索引数据进行更新，而上述数据写入时间在预更新时间范围之内的索引数据包括多个，即实现了对多个索引数据同步进行更新，减小了索引数据更新所需的时间，从而提高了流式文件***中索引数据的更新效率。
基于同一发明构思，根据本申请上述实施例提供的流式文件***中的索引数据更新方法，相应地，本申请实施例还提供了一种电子设备，如图6所示， 包括处理器601和存储器602，其中，
存储器602，用于存放计算机程序；
处理器601，用于执行存储器602上所存放的程序时，实现本申请实施例提供的流式文件***中的索引数据更新方法。
例如，可以包括如下步骤：
获取索引块的索引时间，索引时间为基于索引块对应的对象块中存储的流式数据的数据写入时间确定的时间范围；
判断索引时间与预更新时间范围之间是否存在交集；
若存在交集，则将索引块确定为待处理索引块；
读取待处理索引块中存储的各索引数据，索引数据包括待处理索引块对应的对象块中存储的流式数据的数据写入时间；
更新数据写入时间在预更新时间范围之内的索引数据。
进一步的，还可以包括本申请实施例提供的上述流式文件***中的索引数据更新方法中的其他处理流程，在此不再进行详细描述。存储器可以包括随机存取存储器(英文：Random Access Memory，简称：RAM)，也可以包括非易失性存储器(英文：Non-volatile Memory，简称：NVM)，例如至少一个磁盘存储器。进一步的，存储器还可以是至少一个位于远离前述处理器的存储装置。
上述的处理器可以是通用处理器，包括中央处理器(英文：Central Processing Unit，简称：CPU)、网络处理器(英文：Network Processor，简称：NP)等；还可以是数字信号处理器(英文：Digital Signal Processor，简称：DSP)、专用集成电路(英文：Application Specific Integrated Circuit，简称：ASIC)、现场可编程门阵列(英文：Field-Programmable Gate Array，简称：FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。
上述处理器601和存储器602可以通过地址总线、数据总线、控制总线等通信总线连接，通信总线可以是外设部件互连标准(英文：Peripheral Component Interconnect，简称：PCI)总线或扩展工业标准结构(英文：Extended  Industry Standard Architecture，简称：EISA)总线等。电子设备可以通过通信接口与其他外部设备进行通信。
当然，上述处理器601和存储器602之间、电子设备和其他外部设备之间还可以通过无线模块的无线连接方式进行通信。
本申请实施例提供的电子设备中，其采用的方法为：获取基于索引块对应的对象块中存储的流式数据的数据写入时间确定的索引块的索引时间；将索引时间与预更新时间范围之间存在交集的索引块确定为待处理索引块，读取待处理索引块中存储的各索引数据；更新数据写入时间在预更新时间范围之内的索引数据。通过将索引时间与预更新时间范围之间存在交集的索引块确定为待处理索引块，再将数据写入时间在预更新时间范围之内的索引数据进行更新，而上述数据写入时间在预更新时间范围之内的索引数据包括多个，即实现了对多个索引数据同步进行更新，减小了索引数据更新所需的时间，从而提高了流式文件***中索引数据的更新效率。
在本申请提供的又一实施例中，还提供了一种计算机可读存储介质，该计算机可读存储介质中存储有指令，当其在计算机上运行时，使得计算机执行上述实施例中任一上述的流式文件***中的索引数据更新方法。
本申请实施例提供的计算机可读存储介质中，其采用的方法为：获取基于索引块对应的对象块中存储的流式数据的数据写入时间确定的索引块的索引时间；将索引时间与预更新时间范围之间存在交集的索引块确定为待处理索引块，读取待处理索引块中存储的各索引数据；更新数据写入时间在预更新时间范围之内的索引数据。通过将索引时间与预更新时间范围之间存在交集的索引块确定为待处理索引块，再将数据写入时间在预更新时间范围之内的索引数据进行更新，而上述数据写入时间在预更新时间范围之内的索引数据包括多个，即实现了对多个索引数据同步进行更新，减小了索引数据更新所需的时间，从而提高了流式文件***中索引数据的更新效率。
在本申请提供的又一实施例中，还提供了一种包含指令的计算机程序产品，当其在计算机上运行时，使得计算机执行上述实施例中任一上述的流式文件***中的索引数据更新方法。
本申请实施例提供的包含指令的计算机程序产品中，其采用的方法为：获取基于索引块对应的对象块中存储的流式数据的数据写入时间确定的索引块的索引时间；将索引时间与预更新时间范围之间存在交集的索引块确定为待处理索引块，读取待处理索引块中存储的各索引数据；更新数据写入时间在预更新时间范围之内的索引数据。通过将索引时间与预更新时间范围之间存在交集的索引块确定为待处理索引块，再将数据写入时间在预更新时间范围之内的索引数据进行更新，而上述数据写入时间在预更新时间范围之内的索引数据包括多个，即实现了对多个索引数据同步进行更新，减小了索引数据更新所需的时间，从而提高了流式文件***中索引数据的更新效率。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行上述计算机程序指令时，全部或部分地产生按照本申请实施例上述的流程或功能。计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(英文：Digital Subscriber Line，简称：DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。上述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。上述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，数字视频光盘(英文：Digital Video Disc，简称：DVD))、或者半导体介质(例如固态硬盘(英文：Solid State Disk，简称：SSD))等。
需要说明的是，在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备 所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
本说明书中的各个实施例均采用相关的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于装置、电子设备以及存储介质实施例而言，由于其基本相似于方法实施例，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
以上所述仅为本申请的较佳实施例而已，并不用以限制本申请，凡在本申请的精神和原则之内，所做的任何修改、等同替换、改进等，均应包含在本申请保护的范围之内。

Claims (10)

  1. 一种流式文件***中的索引数据更新方法，其特征在于，所述方法包括：
    获取索引块的索引时间，所述索引时间为基于所述索引块对应的对象块中存储的流式数据的数据写入时间确定的时间范围；
    判断所述索引时间与预更新时间范围之间是否存在交集；
    若存在交集，则将所述索引块确定为待处理索引块；
    读取所述待处理索引块中存储的各索引数据，所述索引数据包括所述待处理索引块对应的对象块中存储的流式数据的数据写入时间；
    更新数据写入时间在所述预更新时间范围之内的索引数据。
  2. 根据权利要求1所述的方法，其特征在于，所述预更新时间范围为：小于或等于获取到的预更新时间点的时间范围。
  3. 根据权利要求1所述的方法，其特征在于，在所述更新数据写入时间在所述预更新时间范围之内的索引数据之前，所述方法还包括：
    判断所述待处理索引块的索引时间是否完全包含于所述预更新时间范围；
    所述更新所述数据写入时间在所述预更新时间范围之内的索引数据，包括：
    若所述待处理索引块的索引时间完全包含于所述预更新时间范围，则更新所述待处理索引块中的所有索引数据。
  4. 根据权利要求3所述的方法，其特征在于，所述更新所述数据写入时间在所述预更新时间范围之内的索引数据，包括：
    若所述待处理索引块的索引时间不完全包含于所述预更新时间范围，则更新所述待处理索引块中数据写入时间在所述待处理索引块的索引时间与所述预更新时间范围的重叠时间范围之内的索引数据。
  5. 根据权利要求1-4任一所述的方法，其特征在于，所述获取索引块的索 引时间，包括：
    获取所述索引块的索引数据中数据写入时间中的最早写入时间及最晚写入时间；
    将大于或等于所述最早写入时间、且小于或等于所述最晚写入时间的时间范围，确定为所述索引块的索引时间。
  6. 一种流式文件***中的索引数据更新装置，其特征在于，包括：
    索引时间获取模块，用于获取索引块的索引时间，所述索引时间为基于所述索引块对应的对象块中存储的流式数据的数据写入时间确定的时间范围；
    第一判断模块，用于判断所述索引时间与预更新时间范围之间是否存在交集；
    待处理索引块确定模块，当所述索引时间与预更新时间范围之间存在交集时，将所述索引块确定为待处理索引块；
    数据读取模块，用于读取所述待处理索引块中存储的各索引数据，所述索引数据包括所述待处理索引块对应的对象块中存储的流式数据的数据写入时间；
    数据更新模块，用于更新数据写入时间在所述预更新时间范围之内的索引数据。
  7. 根据权利要求6所述的装置，其特征在于，所述预更新时间范围为：小于或等于获取到的预更新时间点的时间范围。
  8. 根据权利要求6所述的装置，其特征在于，所述装置还包括：
    第二判断模块，用于判断所述待处理索引块的索引时间是否完全包含于所述预更新时间范围；
    所述数据更新模块，具体用于若所述待处理索引块的索引时间完全包含于所述预更新时间范围，则更新所述待处理索引块中的所有索引数据。
  9. 根据权利要求8所述的装置，其特征在于，所述数据更新模块，还用 于若所述待处理索引块的索引时间不完全包含于所述预更新时间范围，则更新所述待处理索引块中数据写入时间在所述待处理索引块的索引时间与所述预更新时间范围的重叠时间范围之内的索引数据。
  10. 根据权利要求6-8任一所述的装置，其特征在于，所述索引时间获取模块，具体用于：
    获取所述索引块的索引数据中数据写入时间中的最早写入时间及最晚写入时间；
    将大于或等于所述最早写入时间、且小于或等于所述最晚写入时间的时间范围，确定为所述索引块的索引时间。
PCT/CN2019/087002 2018-05-16 2019-05-15 一种流式文件***中的索引数据更新方法、装置 WO2019219021A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810470680.1A CN110569217A (zh) 2018-05-16 2018-05-16 流式文件***中的索引数据更新方法、装置
CN201810470680.1 2018-05-16

Publications (1)

Publication Number Publication Date
WO2019219021A1 true WO2019219021A1 (zh) 2019-11-21

Family

ID=68539541

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/087002 WO2019219021A1 (zh) 2018-05-16 2019-05-15 一种流式文件***中的索引数据更新方法、装置

Country Status (2)

Country Link
CN (1) CN110569217A (zh)
WO (1) WO2019219021A1 (zh)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090094186A1 (en) * 2007-10-05 2009-04-09 Nec Corporation Information Retrieval System, Registration Apparatus for Indexes for Information Retrieval, Information Retrieval Method and Program
CN101989301A (zh) * 2010-10-22 2011-03-23 复旦大学 一种支持多数据源的索引维护方法
CN103177116A (zh) * 2013-04-08 2013-06-26 国电南瑞科技股份有限公司 一种基于两级索引的分布式日志处理和查询方法

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030101183A1 (en) * 2001-11-26 2003-05-29 Navin Kabra Information retrieval index allowing updating while in use
CN104794123B (zh) * 2014-01-20 2018-07-27 阿里巴巴集团控股有限公司 一种为半结构化数据构建NoSQL数据库索引的方法及装置
CN107315825B (zh) * 2017-07-05 2020-02-28 北京奇艺世纪科技有限公司 一种索引更新***、方法及装置

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090094186A1 (en) * 2007-10-05 2009-04-09 Nec Corporation Information Retrieval System, Registration Apparatus for Indexes for Information Retrieval, Information Retrieval Method and Program
CN101989301A (zh) * 2010-10-22 2011-03-23 复旦大学 一种支持多数据源的索引维护方法
CN103177116A (zh) * 2013-04-08 2013-06-26 国电南瑞科技股份有限公司 一种基于两级索引的分布式日志处理和查询方法

Also Published As

Publication number Publication date
CN110569217A (zh) 2019-12-13

Similar Documents

Publication Publication Date Title
US11853549B2 (en) Index storage in shingled magnetic recording (SMR) storage system with non-shingled region
US9152474B2 (en) Context aware synchronization using context and input parameter objects associated with a mutual exclusion lock
BR112016022388B1 (pt) Sistemas e métodos para otimizar suporte de várias versões em índices
US20070156778A1 (en) File indexer
KR20160003682A (ko) 플레이스홀더에 의한 하이드레이션 및 디하이드레이션 기법
CN108573019B (zh) 一种数据迁移方法、装置、电子设备及可读存储介质
CN109284066B (zh) 一种数据处理方法、装置、设备及***
US10838944B2 (en) System and method for maintaining a multi-level data structure
CN107577775B (zh) 一种读取数据方法、装置、电子设备及可读存储介质
US20140040207A1 (en) Method for a cloning process to enable cloning a larger System drive to a smaller system
CN110633046A (zh) 一种分布式***的存储方法、装置、存储设备及存储介质
US11853284B2 (en) In-place updates with concurrent reads in a decomposed state
WO2018049883A1 (zh) 文件操作方法及装置
CN109062602B (zh) 一种针对应用程序的数据处理方法、装置及电子设备
US9588884B2 (en) Systems and methods for in-place reorganization of device storage
US9817837B2 (en) Method and system for file storage and access
US20150067192A1 (en) System and method for adjusting sas addresses of sas expanders
WO2021042542A1 (zh) 目录存储方法、装置、计算机设备及存储介质
WO2019219021A1 (zh) 一种流式文件***中的索引数据更新方法、装置
WO2019228009A1 (zh) 一种lsm树的优化方法、装置及计算机设备
US20160062841A1 (en) Database and data accessing method thereof
CN111767339B (zh) 一种数据同步方法、装置、电子设备及存储介质
US10706012B2 (en) File creation
CN111506250B (zh) 一种数据删除方法、装置、电子设备及存储介质
CN114282550B (zh) 一种实现qt多语言翻译***的方法、装置及相关组件

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19803213

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19803213

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 01/06/2021)

122 Ep: pct application non-entry in european phase

Ref document number: 19803213

Country of ref document: EP

Kind code of ref document: A1