CN115237819A - Data storage method, data reading method, data storage device and data reading device - Google Patents

Data storage method, data reading method, data storage device and data reading device Download PDF

Info

Publication number
CN115237819A
CN115237819A CN202110450349.5A CN202110450349A CN115237819A CN 115237819 A CN115237819 A CN 115237819A CN 202110450349 A CN202110450349 A CN 202110450349A CN 115237819 A CN115237819 A CN 115237819A
Authority
CN
China
Prior art keywords
data
target
memory
fragment
sub
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110450349.5A
Other languages
Chinese (zh)
Inventor
刘建文
李照辉
王伟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Hikvision System Technology Co Ltd
Original Assignee
Hangzhou Hikvision System Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Hikvision System Technology Co Ltd filed Critical Hangzhou Hikvision System Technology Co Ltd
Priority to CN202110450349.5A priority Critical patent/CN115237819A/en
Publication of CN115237819A publication Critical patent/CN115237819A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02Addressing or allocation; Relocation
    • G06F12/0223User address space allocation, e.g. contiguous or non contiguous base addressing
    • G06F12/023Free address space management
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638Organizing or formatting or addressing of data
    • G06F3/0644Management of space entities, e.g. partitions, extents, pools
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0662Virtualisation aspects
    • G06F3/0665Virtualisation aspects at area level, e.g. provisioning of virtual or logical volumes
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/54Interprogram communication
    • G06F9/544Buffers; Shared memory; Pipes

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the invention provides a data storage method, a data reading method and a data storage device, and belongs to the technical field of data storage. The storage method comprises the following steps: acquiring data to be stored; determining a first target storage medium and a target address range; determining a target memory fragment in a first target subregion based on a target address range; and storing the data to be stored to the target memory fragment. The reading method comprises the following steps: determining a second target sub-region; searching data to be read in a second target subregion; if all the data are found, writing the data to be read into the virtual cache; if the partial data are found, writing the partial data and the missing data into the virtual cache; and if the data to be read is not found, writing the data to be read from the second target storage medium into the virtual cache. Compared with the prior art, the scheme provided by the embodiment of the invention can ensure the consistency of the data accessed by a plurality of processes when the plurality of processes need to access the same data.

Description

Data storage method, data reading method, data storage device and data reading device
Technical Field
The present invention relates to the field of data storage technologies, and in particular, to a method and an apparatus for data storage and data reading.
Background
In general, various types of storage media such as a magnetic Disk and an SSD (Solid State Disk) may be provided in the electronic device, and these storage media are used to store data generated by each process executed by the electronic device and data required by the electronic device when executing each process.
In the related art, when each process writes data to a storage medium, the data to be stored is usually written into a virtual buffer area accessible only by itself, and then written from the virtual buffer area to the storage medium.
However, in the above related art, since data in the virtual cache area of each process is visible only to the process, when a plurality of processes need to access the same address range of the same storage medium, consistency of data accessed by the plurality of processes cannot be guaranteed.
For example, process 1 is to write data a to be stored into address range A0 of storage medium a, and the data a to be stored is data required for process 2 to run, and process 2 needs to read data required for its own running from address range A0 of storage medium a. Since the process 1 needs to write the data a to be stored into the virtual cache area accessible only by itself, and the process 2 cannot access the virtual cache area of the process 1, when the data a to be stored is not written into the address range A0 in the storage medium a, the data read by the process 2 from the address range A0 in the storage medium a is inconsistent with the data a to be stored.
Disclosure of Invention
Embodiments of the present invention provide a method and an apparatus for storing and reading data, so as to ensure consistency of data accessed by multiple processes when the multiple processes need to access the same segment of data. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present invention provides a data storage method, which is applied to an electronic device, where sub-regions corresponding to respective storage media of the electronic device are set in a shared memory of the electronic device, and each sub-region includes a plurality of memory segments; the method comprises the following steps:
acquiring data to be stored;
determining a first target storage medium for storing the data to be stored and a target address range of the data to be stored in the first target storage medium;
determining a target memory fragment for storing the data to be stored in a first target subregion corresponding to the first target storage medium based on the target address range;
and storing the data to be stored to the target memory fragment.
Optionally, in a specific implementation manner, the step of determining, in a first target sub-area corresponding to the first target storage medium based on the target address range, a target memory slice for storing the to-be-stored data includes:
judging whether a first fragment exists in a first target subarea corresponding to the first target storage medium; wherein the first segment is: establishing a corresponding relation with an address range in the first target storage medium, wherein the corresponding address range in the first target storage medium and the target address range have overlapped memory fragments;
if yes, judging whether the target address range is located in the address range corresponding to the first fragment in the first target storage medium;
if yes, determining the first fragment as a target memory fragment;
otherwise, determining a second fragment for storing the residual data from the first target sub-area, and determining the first fragment and the second fragment as target memory fragments; wherein the remaining data is: and the data which cannot be stored in the first fragment in the data to be stored.
Optionally, in a specific implementation manner, the step of determining, from the first target sub-area, a second fragment used for storing remaining data includes:
judging whether the storage space of the memory fragment in the available state in the first target subregion meets the data volume of the residual data or not;
if yes, selecting a second fragment for storing the residual data from the memory fragments in the available state;
otherwise, changing the state of at least one memory fragment in the unavailable state in the first target sub-region into an available state, so that after the change is completed, the storage space of the memory fragment in the available state in the first target sub-region meets the data volume of the residual data;
and after the change is completed, selecting a second fragment for storing the remaining data from the memory fragments in the available state in the first target sub-region.
Optionally, in a specific implementation manner, before the step of changing the state of at least one memory slice in the first target sub-area, which is in the unavailable state, to the available state, the method further includes:
determining memory fragments to be refreshed according to the relationship between the updating times and the disk refreshing times of each memory fragment in an unavailable state in the first target sub-area, and writing data stored in the memory fragments to be refreshed into the first target storage medium; wherein, the brush dish is: writing the stored data to the first target storage medium;
after the writing is completed, the step of changing the state of at least one memory slice in the first target sub-area which is in the unavailable state into the available state is executed.
Optionally, in a specific implementation manner, before the step of determining the first partition as the target memory partition, the method further includes:
judging whether the first fragment meets any one of preset writing conditions; wherein the preset writing condition comprises: the first fragment does not store data, the first fragment stores data, and a neutral zone does not exist between an address range of the stored data in the first fragment and an address range of the target address range corresponding to the first fragment;
if yes, determining the first fragment as a target memory fragment;
otherwise, writing the data to be stored into the target address range of the first target storage medium.
Optionally, in a specific implementation manner, the method further includes:
and if the first fragment does not exist in a first target sub-area corresponding to the first target storage medium, determining a third fragment for storing the data to be stored from the first target sub-area.
Optionally, in a specific implementation manner, the step of storing the data to be stored in the target memory slice includes:
if the target memory fragment is the first fragment, determining a target starting offset address in the first fragment, and writing the data to be stored into the first fragment from the target starting offset address; wherein, the storage address of the target start offset address in the first target storage medium is the start address of the target address range;
if the target memory fragment is the first fragment and the second fragment, determining a specified offset address range in the first fragment, writing target data in the data to be stored into the specified offset address range, and writing residual data in the data to be stored into the second fragment; wherein an address range corresponding to the specified offset address range in the first target storage medium is an address range of the first segment that coincides with the target address range in the address range corresponding to the first target storage medium, and the target data is: the storage address in the first target storage medium is the data of the storage address corresponding to the specified offset address range; the remaining data is: and the data to be stored cannot be stored in the first fragment.
Optionally, in a specific implementation manner, each sub-region further includes: sharing a memory index area; the shared memory index area comprises: sharing memory header information and an index of each memory slice in the sub-area; the shared memory header information includes: counting IO, wherein the index of each memory fragment comprises: updating the memory, counting by brushing the disk and eliminating the sequence; the memory updating count is used for representing the updating times of the data stored in the memory fragment, the flash disk count is used for representing the times of writing the data stored in the memory fragment into the corresponding storage medium, and the elimination sequence represents the sequencing for changing the unavailable state of the memory fragment into the available state; the method further comprises the following steps:
after data is written into each memory fragment, adding 1 to the IO count in the shared memory header information of the sub-region where the memory fragment is located, updating the elimination sequence in the index information of the memory fragment to the IO count added with 1, and adding 1 to the memory update count in the index information of the memory fragment;
after the data stored in each memory slice to be refreshed is written into the first target storage medium, adding 1 to the disk refreshing count in the index information of the memory slice to be refreshed.
Optionally, in a specific implementation manner, the shared memory header information further includes: marking initialization of a memory; before the step of obtaining data to be stored, the method further comprises:
for each storage medium, determining whether the sub-region corresponding to the storage medium is initialized or not according to the memory initialization mark in the shared memory header information of the sub-region corresponding to the storage medium;
if so, determining the to-be-cleaned memory fragments with data stored in the sub-area corresponding to the storage medium, writing the data stored in the to-be-cleaned memory fragments into the storage medium, and marking the state of the to-be-cleaned memory fragments as an available state;
otherwise, initializing the sub-area corresponding to the storage medium.
Optionally, in a specific implementation manner, the method further includes:
for each sub-region, the following steps are performed:
detecting whether the memory updating count and the flash count in the index of each memory fragment in the unavailable state in the sub-area are the same or not;
if the memory fragments are the same, judging whether the elimination sequence of the memory fragments is in a preset range;
if the memory is in the available state, the state of the memory fragment is changed into the available state;
if not, judging whether the storage allocation meets any one of preset elimination conditions; wherein the preset elimination conditions comprise: the ratio of the storage space occupied by the data stored in the memory fragment to the storage space of the memory fragment is greater than a preset ratio, and the ratio of the storage space occupied by the data stored in the memory fragment to the storage space of the memory fragment is not greater than the preset ratio, and the eliminated sequence of the memory fragment is located in the preset range;
and when the condition is met, writing the data stored in the memory fragment into the storage medium corresponding to the sub-area, adding 1 to the disk refreshing count in the index of the memory fragment, and changing the state of the memory fragment into an available state.
Optionally, in a specific implementation manner, the method further includes:
for each sub-area, when the sub-area meets a preset disk refreshing condition, writing data stored in each memory sub-slice in the sub-area into a storage medium corresponding to the sub-area;
wherein, the preset brushing condition comprises: the data volume of the stored data exceeds the preset data volume, or the time length from the last disk brushing reaches the preset time length.
Optionally, in a specific implementation manner, the method further includes:
for each sub-region, when the number of the memory fragments in the usable state in the sub-region is smaller than a preset number, changing the state of at least one memory fragment in the unusable state in the sub-region into the usable state, so that after the change is completed, the number of the memory fragments in the usable state in the sub-region is not smaller than the preset number.
In a second aspect, an embodiment of the present invention provides a data reading method, which is applied to an electronic device, where sub-regions corresponding to respective storage media of the electronic device are set in a shared memory of the electronic device, and each sub-region includes a plurality of memory fragments; the method comprises the following steps:
acquiring a data reading instruction generated by a target process, and determining a second target sub-area corresponding to a second target storage medium for storing data to be read;
searching the data to be read in each memory fragment of the second target sub-area, in which the data is stored;
if all the data of the data to be read are found, writing the data to be read into the virtual cache of the target process;
if partial data of the data to be read is found, writing the partial data and missing data read from the second target storage medium into a virtual cache of the target process; wherein the missing data is: the data to be read is not stored in the second target subarea;
and if the data to be read is not found, writing the data to be read from the second target storage medium into the virtual cache of the target process.
In a third aspect, an embodiment of the present invention provides a data storage apparatus, which is applied to an electronic device, where sub-areas corresponding to respective storage media of the electronic device are set in a shared memory of the electronic device, and each sub-area includes a plurality of memory segments; the device comprises:
the data acquisition module is used for acquiring data to be stored;
the information determining module is used for determining a first target storage medium for storing the data to be stored and a target address range of the data to be stored in the first target storage medium;
the fragment determining module is used for determining a target memory fragment for storing the data to be stored in a first target subregion corresponding to the first target storage medium based on the target address range;
and the data storage module is used for storing the data to be stored to the target memory fragment.
Optionally, in a specific implementation manner, the fragment determining module includes:
a first fragmentation judgment submodule for judging whether a first fragmentation exists in a first target sub-area corresponding to the first target storage medium; if yes, triggering an address range judgment submodule; wherein the first segment is: establishing a corresponding relation with an address range in the first target storage medium, wherein the corresponding address range in the first target storage medium and the target address range have overlapped memory fragments;
the address range judging submodule is used for judging whether the target address ranges are all located in the address ranges corresponding to the first fragments in the first target storage medium; if yes, triggering a first target fragment determining submodule; otherwise, triggering a second fragment determining submodule;
the first target fragment determining submodule is used for determining the first fragment as a target memory fragment;
the second fragmentation determination submodule is configured to determine, from the first target sub-area, a second fragmentation used for storing remaining data, and determine the first fragmentation and the second fragmentation as a target memory fragmentation; wherein the remaining data is: and the data which cannot be stored in the first fragment in the data to be stored.
Optionally, in a specific implementation manner, the second fragment determining sub-module includes:
a space judgment unit, configured to judge whether a storage space of the memory partition in an available state in the first target sub-area satisfies a data amount of remaining data; if yes, triggering a second fragmentation determination unit; otherwise, triggering the state change unit;
the second fragmentation determining unit is configured to select a second fragmentation used for storing remaining data from the memory fragmentation in the available state;
the state changing unit is configured to change a state of at least one memory slice in the first target sub-area that is in an unavailable state to an available state, so that after the change is completed, a storage space of the memory slice in the available state in the first target sub-area satisfies a data amount of the remaining data;
the second partition determining unit is further configured to select, after the change is completed, a second partition for storing remaining data from the memory partitions in the available state in the first target sub-area.
Optionally, in a specific implementation manner, the apparatus further includes:
before the state of at least one memory fragment in the unavailable state in the first target sub-area is changed into the available state, determining a memory fragment to be refreshed according to the relationship between the update times and the disk refreshing times of each memory fragment in the unavailable state in the first target sub-area, and writing data stored in the memory fragment to be refreshed into the first target storage medium; after the writing is finished, triggering the state changing unit;
wherein, the brush dish is: writing the stored data to the first target storage medium.
Optionally, in a specific implementation manner, the apparatus further includes:
a write condition determining module, configured to determine whether the first partition meets any condition of preset write conditions before determining the first partition as a target memory partition; if yes, triggering the first target fragment determining submodule, otherwise, triggering a data writing module; wherein the preset writing condition comprises: the first slice does not store data and the first slice stores data, and a neutral zone does not exist between the address range of the stored data in the first slice and the corresponding address range of the target address range in the first slice;
the data writing module is configured to write the data to be stored into the target address range of the first target storage medium.
Optionally, in a specific implementation manner, the apparatus further includes:
a third partition determining submodule, configured to determine, if the first partition does not exist in a first target sub-area corresponding to the first target storage medium, a third partition for storing the data to be stored from the first target sub-area.
Optionally, in a specific implementation manner, the data storage module is specifically configured to:
if the target memory fragment is the first fragment, determining a target starting offset address in the first fragment, and writing the data to be stored into the first fragment from the target starting offset address; wherein, the storage address of the target start offset address in the first target storage medium is the start address of the target address range;
if the target memory fragment is the first fragment and the second fragment, determining a specified offset address range in the first fragment, writing target data in the data to be stored into the specified offset address range, and writing residual data in the data to be stored into the second fragment; wherein an address range corresponding to the specified offset address range in the first target storage medium is an address range of the first segment that coincides with the target address range in the address range corresponding to the first target storage medium, and the target data is: the storage address in the first target storage medium is the data of the storage address corresponding to the specified offset address range; the remaining data are: and the data to be stored cannot be stored in the first fragment.
Optionally, in a specific implementation manner, each sub-region further includes: sharing a memory index area; the shared memory index area comprises: sharing memory header information and the index of each memory slice in the sub-area; the shared memory header information includes: counting IO, wherein the index of each memory fragment comprises: updating the memory, counting by brushing the disk and eliminating the sequence; the memory updating count is used for indicating the updating times of the data stored in the memory fragment, the disk refreshing count is used for indicating the times of writing the data stored in the memory fragment into the corresponding storage medium, and the eliminating sequence is used for indicating the sequence of changing the memory fragment from the unavailable state to the available state; the device further comprises:
a first index updating module, configured to add 1 to an IO count in shared memory header information of a sub-region where the memory segment is located after data is written into each memory segment, update an obsolete sequence in index information of the memory segment to the IO count added with 1, and add 1 to a memory update count in index information of the memory segment;
and the second index updating module is used for adding 1 to the disk refreshing count in the index information of the memory fragments to be refreshed after the data stored in each memory fragment to be refreshed is written into the first target storage medium.
Optionally, in a specific implementation manner, the shared memory header information further includes: marking initialization of a memory; the device further comprises:
an initialization judgment module, configured to determine, for each storage medium, whether a sub-region corresponding to the storage medium is initialized according to a memory initialization flag in shared memory header information of the sub-region corresponding to the storage medium before the data to be stored is acquired; if yes, triggering a data cleaning module; otherwise, triggering the initialization module;
the data cleaning module is used for determining the to-be-cleaned memory fragments with data stored in the sub-area corresponding to the storage medium, writing the data stored in the to-be-cleaned memory fragments into the storage medium, and marking the states of the to-be-cleaned memory fragments as available states;
the initialization module is used for initializing the sub-area corresponding to the storage medium.
Optionally, in a specific implementation manner, the apparatus further includes a culling module, configured to execute, for each sub-region, the following steps:
detecting whether the memory updating count and the flash count in the index of each memory fragment in the sub-area are the same or not aiming at each memory fragment in the unavailable state;
if the memory fragments are the same, judging whether the elimination sequence of the memory fragments is in a preset range;
if so, changing the state of the memory fragment into an available state;
if not, judging whether the storage allocation meets any one of preset elimination conditions; wherein, the preset elimination condition comprises the following steps: the ratio of the storage space occupied by the data stored in the memory fragment to the storage space of the memory fragment is greater than a preset ratio, and the ratio of the storage space occupied by the data stored in the memory fragment to the storage space of the memory fragment is not greater than the preset ratio, and the eliminated sequence of the memory fragment is located in the preset range;
and when the condition is met, writing the data stored in the memory fragment into the storage medium corresponding to the sub-area, adding 1 to the flash count in the index of the memory fragment, and changing the state of the memory fragment into an available state.
Optionally, in a specific implementation manner, the apparatus further includes:
the data disk refreshing module is used for writing the data stored in each memory fragment in each sub-area into a storage medium corresponding to the sub-area when the sub-area meets a preset disk refreshing condition;
wherein, the preset brushing condition comprises: the data volume of the stored data exceeds the preset data volume, or the time length from the last disk brushing reaches the preset time length.
Optionally, in a specific implementation manner, the apparatus further includes:
and the state changing module is used for changing the state of at least one memory fragment in the unavailable state in the sub-region into the available state when the number of the memory fragments in the available state in the sub-region is less than the preset number aiming at each sub-region, so that the number of the memory fragments in the available state in the sub-region is not less than the preset number after the change is finished.
In a fourth aspect, an embodiment of the present invention provides a data reading apparatus, which is applied to an electronic device, where sub-regions corresponding to respective storage media of the electronic device are set in a shared memory of the electronic device, and each sub-region includes a plurality of memory fragments; the device comprises:
the instruction acquisition module is used for acquiring a data reading instruction generated by the target process and determining a second target sub-area corresponding to a second target storage medium for storing data to be read;
the data searching module is used for searching the data to be read in each memory fragment of the second target sub-area, which stores the data;
the first data reading module is used for writing the data to be read into the virtual cache of the target process if all the data of the data to be read are found;
a second data reading module, configured to, if partial data of the data to be read is found, write the partial data and missing data read from the second target storage medium into the virtual cache of the target process; wherein the missing data is: the data to be read is not stored in the second target subarea;
and the third data reading module is used for writing the data to be read from the second target storage medium into the virtual cache of the target process if the data to be read is not found.
In a fifth aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor and the communication interface complete communication between the processor and the memory through the communication bus;
a memory for storing a computer program;
a processor, configured to implement the steps of any data storage method provided in the first aspect and/or the steps of any data reading method provided in the second aspect when executing a program stored in a memory.
In a sixth aspect, the present invention provides a computer-readable storage medium, in which a computer program is stored, and the computer program, when executed by a processor, implements the steps of any data storage method provided in the first aspect and/or the steps of any data reading method provided in the second aspect.
In a seventh aspect, an embodiment of the present invention provides a computer program product containing instructions, which when run on a computer, causes the computer to perform the steps of any of the data storage methods provided in the above first aspect and/or the steps of any of the data reading methods provided in the above second aspect.
The embodiment of the invention has the following beneficial effects:
as can be seen from the above, by applying the solution provided in the embodiment of the present invention, the sub-areas respectively corresponding to the storage media of the electronic device are set in the shared memory of the electronic device, so that when the process in the electronic device stores the generated data to be stored in the storage media, the data to be stored can be written into the sub-area corresponding to the storage media first; similarly, when a process in the electronic device needs to read data stored in the storage medium during operation, it may also be checked first whether the sub-area corresponding to the storage medium stores the required data.
Therefore, when a plurality of processes need to access the same address range of the same storage medium, the data stored in the same address range of the same storage medium can be stored in the sub-region of the shared memory corresponding to the storage medium, so that the plurality of processes can access the data from the sub-region, and consistency of the data accessed by the plurality of processes is ensured.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a schematic diagram of the structure of each sub-region disposed in an embodiment;
fig. 2 is a schematic flowchart of a data storage method according to an embodiment of the present invention;
fig. 3 is a flowchart illustrating a specific implementation manner of S203 in fig. 2;
FIG. 4 is a schematic flow chart of another specific implementation of S203 in FIG. 2;
fig. 5 is a schematic flow chart of a specific implementation of S2034 in fig. 3 and 4;
fig. 6 is a schematic flow chart of another specific implementation of S2034 in fig. 3 and 4;
FIG. 7 is a flowchart illustrating another specific implementation manner of S203 in FIG. 2;
FIG. 8 (a) is a schematic diagram of a first tile having no data stored therein;
8 (b) -8 (d) are schematic diagrams illustrating various storage situations of data to be stored in the first slice when the data to be stored is stored in the first slice;
8 (e) -f) are diagrams illustrating the relationship between the address range of the stored data and the corresponding address range of the target address range in the first slice shown in FIG. 8 (d);
8 (g) -8 (h) are diagrams illustrating the relationship between the address range of the stored data and the corresponding address range of the target address range in the first slice shown in FIG. 8 (c);
8 (i) -8 (m) are diagrams illustrating the relationship between the address range of the stored data and the corresponding address range of the target address range in the first slice shown in FIG. 8 (b);
FIG. 9 is a schematic flowchart of another data storage method according to an embodiment of the present invention;
fig. 10 is a flowchart illustrating a data reading method according to an embodiment of the present invention;
FIG. 11 is a schematic structural diagram of a data storage device according to an embodiment of the present invention;
fig. 12 is a schematic structural diagram of a data reading apparatus according to an embodiment of the present invention;
fig. 13 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the related art, when each process writes data to a storage medium, the data to be stored is usually written into a virtual buffer area accessible only by itself, and then written from the virtual buffer area to the storage medium. However, in the above related art, since the data in the virtual cache area of each process is visible only to the process, when a plurality of processes need to access the same address range of the same storage medium, consistency of data accessed by the plurality of processes cannot be guaranteed.
In order to solve the above technical problem, embodiments of the present invention provide a data storage method and a data reading method.
The data storage method provided by the embodiment of the invention can be suitable for any application scene in which data generated by a process needs to be stored in a storage medium; correspondingly, the data reading method provided by the embodiment of the invention can be suitable for any application scene needing to read data required by process running from the storage medium. In addition, both the data storage method and the data reading method provided by the embodiment of the present invention may be applied to various types of electronic devices that are provided with a shared memory and a storage medium, and the electronic devices may be devices in various systems or may be separate devices.
Further, since the same electronic device may have a need to store data generated by a process in a storage medium and a need to read data required by the process to run from the storage medium, a data storage method and a data reading method provided by the embodiments of the present invention may be applied to the same electronic device. Of course, it is reasonable that the data storage method and the data reading method provided by the embodiments of the present invention can also be applied to different electronic devices, respectively.
In an embodiment of the present invention, an electronic device for executing a data storage method and/or a data reading method provided by the embodiment of the present invention is provided with a shared memory, and sub-regions respectively corresponding to storage media of the electronic device are provided in the shared memory. Furthermore, each sub-region comprises a plurality of memory fragments.
When a plurality of storage media exist in the electronic equipment, a plurality of sub-areas are arranged in a shared memory of the electronic equipment, and each sub-area corresponds to one storage medium; when a storage medium exists in the electronic device, a sub-region is set in the shared memory of the electronic device, and the sub-region corresponds to the storage medium, and the sub-region may be the entire shared memory or a partial region in the shared memory, which is all reasonable.
Furthermore, for each memory slice, the memory slice may have two states, an unavailable state and an available state, wherein when a certain memory slice is in the available state, data can be written into the memory slice regardless of whether the data is stored in the memory slice, and thus, the newly written data can cover the data originally stored in the memory slice. In addition, when a certain memory slice is in an unavailable state, the memory slice can be changed from the unavailable state to an available state according to the data storage requirement in practical application, wherein the changing process can be called to eliminate the data stored in the memory slice.
Optionally, when the state of the memory slice in the unavailable state is changed into the available state, the data stored in the memory slice may be cleared.
For each electronic device, the structure of the sub-region set for each storage medium in the electronic device may be determined according to the number of storage media in the electronic device, the size of the shared memory, and the actual situation of the running process, for example, the data amount of data generated by the running process, the data amount of data required by the running process, and the like.
For example, fig. 1 is a schematic structural diagram of each sub-region disposed in an embodiment.
As shown in fig. 1, each sub-region includes a shared memory index region and n memory slices. The total size of the n memory slices is 1GB, the size of each memory slice is 1MB, and a region formed by the n memory slices can be a shared memory data region of the sub-region. Furthermore, the shared memory index area includes shared memory header information and an index of each memory slice.
Thus, the size of each sub-region = the size of the shared memory header information + the size of the shared memory data area + the size of the index of each memory slice n.
Further, the data structure of the shared memory header includes: a device UUID (universal Unique Identifier), a shared memory size, a total number of fragments, a single fragment size, a memory initialization flag, an idle number of fragments, an IO count, and a reserved field; the data structure of the index of each memory slice comprises: an Identification (ID) of a segment, a memory start address, a memory segment size, a valid data offset, a valid data size, a disk start offset, a memory update count, a flash count, an idle flag, and a kill sequence.
As shown in table 1, for the description of the data structure of the shared memory header:
TABLE 1
Figure BDA0003038455470000111
As shown in table 2, for the description of the data structure of the index of each memory slice:
Figure BDA0003038455470000112
Figure BDA0003038455470000121
optionally, for the memory initialization flag, whether the sub-region has been initialized or not may be indicated by numbers "0" and "1", for example, when the memory initialization flag is 1, it may indicate that the sub-region has been initialized, and when the memory initialization flag is 0, it may indicate that the sub-region has not been initialized.
In addition, optionally, for the IO count, after data update is performed on each memory fragment, the IO count is incremented by 1, and the IO count obtained after the increment by 1 is used as an obsolete sequence of the memory fragment and is updated to the index information of the memory fragment. Further, the smaller the obsolete sequence in the index of each memory slice, the earlier the memory slice changes from an unavailable state to an available state.
For example, if the obsolete sequence of the memory slice 1 is 1, the obsolete sequence of the memory slice 2 is 2, the obsolete sequence of the memory slice 3 is 3, and the memory slices 1 to 3 are all in an unavailable state, the memory slice 1 needs to be changed from the unavailable state to the available state when the state of the memory slices in 1 unavailable state needs to be changed into the available state; when the state of the memory slice with 2 unavailable states needs to be changed into an available state, the memory slices 1 and 2 need to be changed from the unavailable state into the available state.
Optionally, for each memory slice, after data is written into the memory slice each time, adding 1 to a memory update count in an index of the memory slice; after the data stored in the memory fragments are written into the corresponding storage medium each time, the disk-refreshing count in the index of the memory fragments is increased by 1; that is, the process of writing the data stored in the memory slice to the corresponding storage medium may be referred to as a disk-flushing.
Further, optionally, in order to reduce the number of IO times in the data storage and data reading processes more in practical applications, the memory slices in each sub-area may be aligned.
On the basis of the above description, a data storage method provided in an embodiment of the present invention may include the following steps:
acquiring data to be stored;
determining a first target storage medium for storing the data to be stored and a target address range of the data to be stored in the first target storage medium;
determining a target memory fragment for storing the data to be stored in a first target sub-area corresponding to the first target storage medium based on the target address range;
and storing the data to be stored to the target memory fragment.
Correspondingly, the data reading method provided by the embodiment of the invention can comprise the following steps:
acquiring a data reading instruction generated by a target process, and determining a first target sub-area corresponding to a first target storage medium for storing data to be read;
searching the data to be read in each memory fragment of the first target sub-area, which stores the data;
if all the data of the data to be read are found, writing the data to be read into the virtual cache of the target process;
if partial data of the data to be read is found, writing the partial data and the missing data read from the first target storage medium into a virtual cache of the target process; wherein the missing data is: in the data to be read, the data in the first target subarea is not stored;
and if the data to be read is not found, writing the data to be read from the first target storage medium into a virtual cache of the target process.
As can be seen from the above, by applying the solution provided in the embodiment of the present invention, the sub-areas respectively corresponding to the storage media of the electronic device are set in the shared memory of the electronic device, so that when the process in the electronic device stores the generated data to be stored in the storage media, the data to be stored can be written into the sub-area corresponding to the storage media first; similarly, when a process in the electronic device needs to read data stored in the storage medium during running, it may also be checked first whether the sub-area corresponding to the storage medium stores the required data.
Therefore, when a plurality of processes need to access the same address range of the same storage medium, the data stored in the same address range of the same storage medium can be stored in the sub-region of the shared memory corresponding to the storage medium, so that the plurality of processes can access the data from the sub-region, and consistency of the data accessed by the plurality of processes is ensured.
First, a data storage method provided in an embodiment of the present invention is specifically described with reference to the accompanying drawings.
Fig. 2 is a schematic flow chart of a data storage method according to an embodiment of the present invention, and as shown in fig. 2, the data storage method may include the following steps:
s201: acquiring data to be stored;
when storing data, the electronic device may first obtain data to be stored.
The data to be stored may be data generated by a process running in the electronic device and needing to be stored in a storage medium in the electronic device; or may be data required for a process running in the electronic device read from a storage medium of the electronic device. Of course, it is reasonable that the data to be stored may also be other data related to processes running in the electronic device.
S202: determining a first target storage medium for storing data to be stored and a target address range of the data to be stored in the first target storage medium;
when the data to be stored is acquired, the electronic device may determine a first target storage medium for storing the data to be stored, and further determine a target address range for storing the data to be stored in the first target storage medium.
When the data to be stored is data which is generated by a process running in the electronic device and needs to be stored in a storage medium in the electronic device, the process can simultaneously determine which address range in which storage medium the data needs to be stored in when generating and obtaining the data, so that when the electronic device obtains the data, a first target storage medium used for storing the data to be stored and a target address range of the data to be stored in the first target storage medium can be determined accordingly;
when the data to be stored is data required by a process running in the electronic device read from a storage medium of the electronic device, a first target storage medium for storing the data to be stored and a target address range of the data to be stored in the first target storage medium may be directly determined when the data is read.
S203: determining a target memory fragment for storing data to be stored in a first target subregion corresponding to a first target storage medium based on the target address range;
since different storage media in the electronic device correspond to different sub-areas, after determining a first target storage medium for storing data to be stored and a target address range of the data to be stored in the first target storage medium, a first target sub-area corresponding to the first target storage medium may be determined among a plurality of sub-areas set in the shared memory.
Furthermore, since the corresponding address ranges of the different memory slices in the first target sub-area in the first target storage medium are different, and the states of the different memory slices in the first target sub-area are different, in order to make the address range of the data to be stored in the first target sub-area correspond to the target address range in the first target storage medium, the target memory slice for storing the data to be stored in the first target sub-area may be determined based on the target address range.
The determined target memory fragment for storing the data to be stored is in an available state, and at this time, if the stored data exists in the target memory fragment, the data to be stored may cover the stored data and be stored in the target memory fragment.
S204: and storing the data to be stored to the target memory fragment.
After the target memory fragment for storing the data to be stored is determined, the acquired data to be stored can be stored in the target memory fragment.
Optionally, after the data to be stored is stored in the target memory segment, the index of the target memory segment may be updated, and the corresponding relationship between the target memory segment and the target address range is established.
As can be seen from the above, since data in the shared memory is visible to all processes, when multiple processes need to access the same address range of the same storage medium, the data stored in the same address range of the same storage medium may be stored in a sub-region of the shared memory corresponding to the storage medium, so that the multiple processes can access the data from the sub-region, and consistency of the data accessed by the multiple processes is ensured.
Optionally, in a specific implementation manner, as shown in fig. 3, the step S203 of determining, based on the target address range, a target memory partition used for storing data to be stored in a first target sub-area corresponding to a first target storage medium may include the following steps:
s2031: judging whether a first fragment exists in a first target subregion corresponding to a first target storage medium; if yes, go to step S2032;
wherein, the first fragmentation is: establishing a corresponding relation with an address range in a first target storage medium, wherein the corresponding address range in the first target storage medium and the target address range have overlapped memory fragments;
in this specific implementation, after determining the first target storage medium and the target address range, the electronic device may determine a first target sub-area corresponding to the first target storage medium. Furthermore, the electronic device may determine whether the first segment exists in the plurality of memory segments included in the first target sub-region.
When data is stored in a certain memory slice in the first target sub-area, the address range occupied by the data stored in the first target storage medium can be determined, and further, the corresponding relation between the memory slice and the address range in the first target storage medium can be determined according to the size of the memory slice, the size of the data, and the address of the initial storage position of the data in the memory slice in the shared memory.
The address range in the first target storage medium corresponding to the memory slice refers to: when the memory slice is full of data, the data occupies an address range in the first target storage medium.
Thus, when a certain memory slice in the first target sub-area has established a corresponding relationship with an address range in the first target storage medium, and the corresponding address range in the first target storage medium and the target address range overlap, the memory slice is the first slice.
For example, on the basis of the structural diagram of the sub-region shown in fig. 1, for each memory slice, the address range of the valid data in the corresponding storage medium may be determined according to the disk start offset and the valid data size in the index information of the memory slice, and further, the corresponding relationship between the memory slice and the address range in the storage medium may be determined according to the memory start address, the memory slice size, and the valid data offset of the memory slice.
Optionally, the step S2031 may include the following step 1:
step 1: and judging whether data with the address range occupied in the first target storage medium and the target address range overlapped are stored in a first target subarea corresponding to the first target storage medium or not.
After determining the first target storage medium and the target address range, the electronic device may first determine a first target sub-region corresponding to the first target storage medium, and then search, in each memory slice included in the first target sub-region, whether data in which an address range occupied in the first target storage medium and the target address range overlap each other is stored. The data in which the address range occupied in the first target storage medium coincides with the target address range may be referred to as data conflicting with the data to be stored.
That is to say, the electronic device may determine each data stored in each memory fragment included in the first target sub-region, and determine, according to the correspondence between the memory fragment in which the data is stored and the address range in the first target storage medium, the address range occupied by each data in the first target storage medium, so as to determine whether there is an address range that overlaps with the target address range in each determined address range.
And if the judgment result is yes, the memory fragment where the data is located is the first fragment.
S2032: judging whether the target address ranges are all located in the address ranges corresponding to the first fragments in the first target storage medium; if yes, go to step S2033; otherwise, executing step S2034;
s2033: determining the first fragment as a target memory fragment;
s2034: determining a second fragment for storing the remaining data from the first target subregion, and determining the first fragment and the second fragment as target memory fragments;
wherein, the remaining data is: and data which cannot be stored in the first fragment in the data to be stored.
After determining that the first fragment exists in the first target sub-area, considering whether the first fragment can store all the data to be stored, it needs to further determine whether the target address range is all located in the address range corresponding to the first fragment in the first target storage medium.
When the target address range is judged to be located in the address range corresponding to the first segment in the first target storage medium, it is indicated that if the data to be stored is completely stored in the first segment, the data to be stored may be stored in the target address range in the first target storage medium according to the corresponding relationship between the first segment and the address range in the first target storage medium. Therefore, when the target address range is judged to be located in the corresponding address range of the first fragment in the first target storage medium, it is indicated that the data to be stored can be completely stored in the first fragment, and thus, the first fragment can be directly determined as the target memory fragment.
Correspondingly, when it is determined that the target address range is not all located in the address range corresponding to the first segment in the first target storage medium, it may be stated that the storage range occupied by the partial data in the data to be stored in the first target storage medium is not in the address range corresponding to the first segment in the first target storage medium, and thus, the partial data may not be stored in the first segment, and if the partial data is stored in the first segment, the partial data may not be stored in the correct address range in the first target storage medium. Therefore, when it is determined that the target address range is not located in the address range corresponding to the first partition in the first target storage medium, while the first partition is determined as the target memory partition, a second partition for storing the remaining data needs to be determined from the first target sub-area, and thus, the determined second partition may also be determined as the target storage allocation. And the determined second fragment is a memory fragment in an available state.
Thus, the data to be stored can be stored in the first fragment and the second fragment. The first fragment is used for storing data in an address range corresponding to the first fragment in the first target storage medium, and the second fragment is used for the rest of the data to be stored, wherein the rest of the data cannot be stored in the first fragment.
In step S2031, it may be determined that the first segment does not exist in the first target sub area corresponding to the first target storage medium.
Based on this, optionally, in a specific implementation manner, as shown in fig. 4, on the basis of the specific implementation manner shown in fig. 3, the step S203 may further include the following steps:
s2035: if the first partition does not exist in a first target sub-area corresponding to the first target storage medium, determining a third partition for storing data to be stored from the first target sub-area.
When it is determined that the first partition does not exist in the first target sub-area, it may be stated that the correspondence between the memory partition and the target address range is not yet established, and thus, the electronic device may determine, from the first target sub-area, a third partition used for storing the data to be stored.
And the determined third partition is a memory partition in an available state.
Optionally, in a specific implementation manner, as shown in fig. 5, in the step S2034, determining the second segment for storing the remaining data from the first target sub-area may include the following steps:
s2034a: judging whether the storage space of the memory fragment in the available state in the first target subregion meets the data volume of the residual data or not; if so, go to S2034b; otherwise, go to step S2034c;
s2034b: selecting a second fragment for storing the residual data from the memory fragments in the available state;
s2034c: changing the state of at least one memory fragment in the unavailable state in the first target sub-area into an available state, so that after the change is completed, the storage space of the memory fragment in the available state in the first target sub-area meets the data volume of the residual data;
s2034d: and after the change is completed, selecting a second fragment for storing the residual data from the memory fragments in the available state in the first target sub-area.
When the target address range is judged not to be completely located in the address range corresponding to the first fragment in the first target storage medium, according to the coincidence condition of the target address range and the address range corresponding to the first fragment in the first target storage medium, the remaining data which cannot be stored in the data to be stored in the first fragment can be determined, and further, the data volume of the remaining data is obtained.
In this way, it may be determined whether the storage space of the memory slice in the available state in the first target sub-area satisfies the data amount of the remaining data. The data amount satisfying the remaining data is: not less than the data amount of the remaining data.
When the storage space of the memory fragments in the available state in the first target sub-area is judged to meet the data volume of the remaining data, the electronic device can directly select the second fragment for storing the remaining data from the memory fragments in the available state. The storage space of the selected second fragment is not less than the data volume of the residual data, and the second fragment is in a usable state;
when it is determined that the storage space of the memory partition in the available state in the first target sub-area does not satisfy the data amount of the remaining data, that is, all the remaining data cannot be stored in the first target sub-area only with the currently existing memory partition in the available state, so that the states of some memory partitions in the unavailable state may be updated to the available state for storing the remaining data.
In this way, when it is determined that the storage space of the memory slice in the available state in the first target sub-area does not satisfy the data amount of the remaining data, the state of at least one memory slice in the unavailable state in the first target sub-area may be changed to the available state, so that after the change is completed, the storage space of the memory slice in the available state in the first target sub-area satisfies the data amount of the remaining data.
Furthermore, after the change is completed, a second fragment for storing the remaining data is selected from the memory fragments in the available state in the first target sub-area. And the storage space of the selected second fragment is not less than the data size of the residual data, and the second fragment is in an available state.
Optionally, the number of the memory slices for performing the state change may be: determining according to the data volume of the remaining data and the size of the memory fragments, so that after the change is completed, the storage space of the memory fragments in the available state in the first target sub-region meets the data volume of the remaining data and the minimum number; it can also be: specifying the quantity, judging whether the storage space of the memory fragments in the available state in the first target sub-area meets the data quantity of the residual data after one-time change is finished, if so, continuing to execute the subsequent step S2034d, otherwise, executing one-time change again; this is all reasonable.
In addition, optionally, the order of the memory fragments in the unavailable state may be determined according to the descending order of the eliminated sequences in the index of each memory fragment in the unavailable state, and the state of at least one memory fragment in the unavailable state is changed to the available state from the memory fragment arranged at the first position;
optionally, the order of the memory slices in the unavailable state may be determined according to the order of the importance of the data stored in each memory slice in the unavailable state from low to high, and the state of at least one memory slice in the unavailable state is changed to the available state from the memory slice arranged at the first bit;
of course, at least one memory partition in an unavailable state, which needs to be subjected to state change, may also be determined in other manners, which is not specifically limited in the embodiment of the present invention.
When the data to be stored is data generated by a process running in the electronic device and needing to be stored in a storage medium in the electronic device, then the data stored in each memory slice needs to be written into the corresponding storage medium. Thus, for each memory slice, the update times of the memory slice may indicate the update times of the data stored in the memory slice, that is, the times of writing data into the memory slice, and the flash times of the memory slice may indicate the times of writing the data stored in the memory slice into the corresponding storage medium, so that when the data currently stored in the memory slice is written into the storage medium, the update times and the flash times of the memory slice are the same.
However, in many cases, there may be memory slices in the first target sub-area, where the stored data is not written to the corresponding storage medium, and then the data stored in these memory slices may be lost when the states of these memory slices are changed into usable states.
Based on this, optionally, in a specific implementation manner, as shown in fig. 6, in step S2034, determining the second segment for storing the remaining data from the first target sub-area may further include the following steps:
s2034e: determining the memory fragments to be refreshed according to the relationship between the updating times and the disk refreshing times of each memory fragment in an unavailable state in the first target sub-area, and writing the data stored in the memory fragments to be refreshed into a first target storage medium;
wherein, the brush dish is: writing the stored data to a first target storage medium;
after the writing is completed, step S2034c is executed to change the state of at least one memory slice in the unavailable state in the first target sub-area to the available state.
In this specific implementation manner, before changing the state of at least one memory slice in the unavailable state in the first target sub-area to the available state, a relationship between the update times and the disk-flushing times of each memory slice in the unavailable state in the first target sub-area may be first determined, so that the memory slice to be refreshed is determined in the memory slices in the unavailable state in the first target sub-area according to the determined relationship. In this way, the data stored in the determined memory slice to be refreshed can be written to the first target storage medium. Furthermore, after the writing is completed, the step of changing the state of at least one memory slice in the unavailable state in the first target sub-area to the available state may be continuously performed.
In addition, in step S2035 of the specific implementation manner shown in fig. 4, the specific implementation manner of determining the third segment for storing the data to be stored from the first target sub-area is similar to the specific implementation manner of determining the second segment for storing the remaining data from the first target sub-area in step S2034, and details are not repeated here.
Optionally, as shown in fig. 7, in a specific implementation manner, on the basis of the specific implementation manner shown in fig. 3, after determining that all of the target address ranges are located in the address range corresponding to the first segment in the first target storage medium and before determining the first segment as the target memory segment, step S203 may further include, based on the target address range, determining the target memory segment for storing the data to be stored in the first target sub-region corresponding to the first target storage medium, and further include:
s2036: judging whether the first fragment meets any one of preset writing conditions or not; if yes, go to step S2033; otherwise, step S2037 is executed;
wherein the preset writing condition comprises: the first fragment does not store data, the first fragment stores data, and a neutral zone does not exist between the address range of the stored data in the first fragment and the corresponding address range of the target address range in the first fragment;
s2037: and writing the data to be stored into the target address range of the first target storage medium.
After determining that the target address ranges are all located in the address ranges corresponding to the first fragments in the first target storage medium, the electronic device may further determine whether the first fragments satisfy any of the preset writing conditions.
When the determination result is yes, the electronic device may determine the first partition as a target memory partition, so that the data to be stored may be stored in the first partition;
that is, when the first segment does not store data, the first segment may be determined as the target memory segment;
for example, as shown in fig. 8 (a), it is a schematic diagram of a first slice in which data is not stored, and as shown in fig. 8 (b) -8 (d), it is a schematic diagram of various storage situations of data to be stored in the first slice when the data to be stored is stored in the first slice; the idle data is an idle area, which indicates that no data is stored in the area, and the valid data is data to be stored in the first segment.
When the first segment stores data, it may be further determined whether a neutral area exists between an address range of the stored data in the first segment and an address range of the target address range corresponding to the first segment. That is, when data is stored in the first segment, it is necessary to determine whether an intersection of the stored data and an address range occupied by the data to be stored in the first target storage medium is a continuous address range.
When it is determined that the first segment stores data, the address range of the stored data in the first segment and the address range of the target address range corresponding to the first segment may be determined, so as to determine whether the intersection of the two address ranges is a continuous address range. Therefore, when the intersection of the two address ranges is a continuous address range, it can be said that there is no neutral zone between the address range of the stored data in the first fragment and the corresponding address range of the target address range in the first fragment; otherwise, it may be stated that a neutral zone exists between the address range of the stored data in the first segment and the address range of the target address range corresponding to the first segment.
When it is determined that the neutral area does not exist, the first segment may be determined as the target memory segment, and thus, the data to be stored is written into the first segment.
Correspondingly, when the judgment result is that the first partition does not meet any of the preset writing conditions, the data to be stored may not be stored in the memory partition of the first target sub-area, but may be directly written into the target address range of the first target storage medium.
For example, as shown in fig. 8 (d), when data is stored in the beginning portion of the first segment, the first segment is schematically illustrated; as shown in fig. 8 (e) -fig. (f), which are schematic diagrams of the relationship between the address range of the stored data and the corresponding address range of the target address range in the first slice shown in fig. 8 (d); the idle data is an idle area, which indicates that no data is stored in the area, the valid data is the data stored in the first segment, and the new data is the data to be stored;
as shown in fig. 8 (c), when data is stored at the tail of the first segment, the schematic diagram of the first segment is shown; as shown in fig. 8 (g) -8 (h), which are schematic diagrams of the relationship between the address range of the stored data and the corresponding address range of the target address range in the first slice shown in fig. 8 (c); the idle data is an idle area, which indicates that no data is stored in the area, the valid data is the data stored in the first segment, and the new data is the data to be stored;
as shown in fig. 8 (b), when data is stored in the middle portion of the first segment, the first segment is schematically illustrated; as shown in fig. 8 (i) -8 (m), which are schematic diagrams of the relationship between the address range of the stored data and the corresponding address range of the target address range in the first slice shown in fig. 8 (b); the idle data is an idle area, which indicates that no data is stored in the area, the valid data is data stored in the first segment, and the new data is data to be stored.
Optionally, in a specific implementation manner, on the basis of the specific implementation manners shown in fig. 3 and fig. 7, after determining that the target address range is not all located in the address range corresponding to the first segment in the first target storage medium, before determining the second segment for storing the remaining data from the first target sub-area, the step S203 of determining the target memory segment for storing the data to be stored in the first target sub-area corresponding to the first target storage medium based on the target address range may further include the following steps 2-3:
step 2: judging whether the first fragment meets any one of preset storage conditions or not; if yes, go to step S2033; otherwise, executing step 3;
wherein, the preset storage condition comprises: the first fragment does not store data, the first fragment stores data, and the stored data does not have a neutral area between the address range in the first fragment and the designated address range; the specified address range is: the address range occupied by the data capable of being stored in the first fragment in the data to be stored in the first target storage medium is in the corresponding address range in the first fragment;
and step 3: and writing the data to be stored into the target address range of the first target storage medium.
Optionally, in a specific implementation manner, the step S204 of storing the data to be stored in the target memory segment may include the following steps 4-5:
and 4, step 4: if the target memory fragment is a first fragment, determining a target starting offset address in the first fragment, and writing data to be stored into the first fragment from the target starting offset address;
wherein, the storage address corresponding to the target initial offset address in the first target storage medium is the initial address of the target address range;
in this specific implementation manner, if the target memory partition is a first partition, and the data to be stored may be completely written into the first partition, then according to the corresponding relationship between the first partition and the address range in the first target storage medium, a target start offset address corresponding to the start address of the target address range in the first partition may be determined, that is, a storage address corresponding to the target start offset address in the first target storage medium is a start address of the target address range. In this way, the data to be stored can be written to the first slice starting from the target starting offset address.
And 5: if the target memory fragment is a first fragment and a second fragment, determining a specified offset address range in the first fragment, writing target data in the data to be stored into the specified offset address range, and writing residual data in the data to be stored into the second fragment;
wherein, the address range corresponding to the specified offset address range in the first target storage medium is an address range of the first fragment in the address range corresponding to the first target storage medium and coinciding with the target address range, and the target data is: the storage address in the first target storage medium is data of a storage address corresponding to the specified offset address range; the remaining data are: and the data which cannot be stored in the first fragment in the data to be stored.
In this specific implementation manner, if the target memory partition is a first partition and a second partition, part of data in the data to be stored is written into the first partition, and the other part of data is written into the second partition.
Then, according to the corresponding relationship between the first segment and the address range in the first target storage medium and the target address range, an address range where the address range corresponding to the first segment in the first target storage medium coincides with the target address range is determined, and thus, an address range where the coinciding address range corresponds in the first segment is determined, and the specified offset address range in the first segment is obtained. And then, determining target data in the data to be stored according to the specified offset address range, wherein the storage address of the target data in the first target storage medium is the storage address corresponding to the specified offset address range in the first target storage medium.
Thus, target data in the data to be stored can be written into the specified offset address range, and the rest of the data to be stored can be written into the second fragment.
The remaining data may be written to any position in the second fragment, which is not limited in this embodiment of the present invention.
It is to be emphasized that: when the data to be stored is written into the target memory slice, if the position where the data to be stored is to be written has the previously stored data, the data to be stored can be stored at the position in an overwriting manner, that is, the previously stored data is overwritten by the data to be stored.
Further, optionally, in a specific implementation manner, in the step S204, storing the data to be stored in the target memory segment, the following step 6 may also be included:
and 6: if the target memory fragment is a third fragment, writing the data to be stored into the third fragment;
in this specific implementation manner, the data to be stored may be written to any position in the third segment, which is not specifically limited in the embodiment of the present invention.
Optionally, in a specific implementation manner, each sub-region further includes: sharing a memory index area; the shared memory index area comprises: sharing memory header information and index information of each memory slice in the sub-area; the shared memory header information includes: counting IO, wherein the index information of each memory fragment comprises: updating the memory, counting by brushing the disk and eliminating the sequence;
the memory updating count is used for representing the updating times of the data stored in the memory fragment, the flash disk count is used for representing the times of writing the data stored in the memory fragment into the corresponding storage medium, and the elimination sequence represents the sequencing for changing the unavailable state of the memory fragment into the available state;
for example, as shown in fig. 1, each sub-region includes a shared memory index region and n memory slices. Furthermore, the shared memory index area includes shared memory header information and an index of each memory slice, and the data structure of the shared memory header includes: IO counting; the data structure of the index of each memory slice comprises: memory updating counting, disk refreshing counting and elimination sequence.
Furthermore, in this specific implementation manner, the data storage method provided in the embodiment of the present invention may further include the following steps 7 to 8:
and 7: after data are written into each memory fragment, adding 1 to the IO count in the shared memory header information of the sub-region where the memory fragment is located, updating the elimination sequence in the index information of the memory fragment to the IO count added with 1, and adding 1 to the memory update count in the index information of the memory fragment;
and step 8: after the data stored in each memory slice to be refreshed is written into the first target storage medium, the disk refreshing count in the index information of the memory slice to be refreshed is increased by 1.
In this specific implementation manner, after data is written into each memory fragment, 1 may be added to the IO count in the shared memory header information of the sub-region where the memory fragment is located, and the IO count obtained after adding 1 is used as the obsolete sequence of the memory fragment and updated into the index information of the memory fragment, that is, the obsolete sequence in the index information of the memory fragment is updated to the IO count after adding 1, and the memory update count in the index information of the memory fragment may also be added with 1.
Furthermore, after the data stored in each memory slice to be refreshed is written into the first target storage medium, the disk refreshing count in the index information of the memory slice to be refreshed can be increased by 1.
In addition, optionally, after data is written into each memory slice, other information in the shared memory header information of the sub-region where the memory slice is located may be further updated; for example, the number of idle slices in the shared memory header information shown in fig. 1;
of course, after data is written into each memory slice, and after the data stored in each memory slice to be refreshed is written into the first target storage medium, other information in the index of the memory slice may be further updated; such as valid data offset, disk start time, valid data size, free flag, etc., as shown in fig. 1.
Optionally, in a specific implementation manner, the shared memory header information further includes: initializing a mark of a memory; in this specific implementation manner, as shown in fig. 9, the data storage method provided in the embodiment of the present invention may further include the following steps:
s201a: for each storage medium, determining whether the sub-region corresponding to the storage medium is initialized or not according to the memory initialization mark in the shared memory header information of the sub-region corresponding to the storage medium; if yes, go to step S201b; otherwise, step S201c is executed;
s201b: determining memory fragments to be cleaned, which are stored with data in the sub-area corresponding to the storage medium, writing the data stored in the memory fragments to be cleaned into the storage medium, and marking the state of the memory fragments to be cleaned as a usable state;
s201c: initializing the sub-area corresponding to the storage medium.
Before executing any data storage method provided by the above embodiments of the present invention, it is necessary to first set sub-areas corresponding to respective storage media of the electronic device in the shared memory of the electronic device.
Optionally, the electronic device may obtain the identifier of each storage medium, so that, according to the identifier and a requirement in practical application, based on a user instruction, in the shared memory, sub-regions corresponding to each storage medium of the electronic device are determined.
Furthermore, before acquiring the data to be stored, the electronic device needs to first determine whether the initialization of the sub-area corresponding to each set storage medium has been completed, that is, for each storage medium, the electronic device may acquire the memory initialization flag in the shared memory header information of the sub-area corresponding to the storage medium, so as to determine whether the sub-area corresponding to the storage medium is initialized according to the content of the acquired memory initialization flag.
For example, whether the sub-region is initialized or not may be indicated by numbers "0" and "1", and for example, when the memory initialization flag is 1, the sub-region may be initialized, and when the memory initialization flag is 0, the sub-region may be uninitialized.
Furthermore, for each storage medium, when it is determined that the sub-area corresponding to the storage medium is not initialized, the electronic device may first initialize the sub-area. The initializing the sub-region means: and initially setting the shared memory information head of the sub-area and the index of each memory fragment of the sub-area.
Correspondingly, for each storage medium, when it is determined that the sub-area corresponding to the storage medium is initialized, the to-be-cleaned memory slice in which the data is stored in the sub-area can be determined, the data stored in the to-be-cleaned memory slice is written into the storage medium, and then the state of the to-be-cleaned memory slice can be marked as the available state by the electronic device.
Optionally, in a specific implementation manner, each sub-region includes: sharing a memory index area; the shared memory index area comprises an index of each memory slice in the sub-area; the index of each memory slice includes: on the premise of memory update counting, disk refreshing counting and elimination sequence, in the data storage method provided by the embodiment of the invention, the electronic device can also execute the following steps for each sub-area:
step 11: detecting whether the memory updating count and the flash count in the index of each memory fragment in the sub-area are the same or not aiming at each memory fragment in the unavailable state; if the two are the same, executing step 12; otherwise, executing step 13;
step 12; judging whether the elimination sequence of the memory fragment is within a preset range; if so, go to step 14;
step 14: changing the state of the memory fragment into an available state;
step 13: judging whether the storage allocation meets any one of preset elimination conditions; if yes, go to step 15;
wherein the preset elimination conditions comprise: the ratio of the storage space occupied by the data stored in the memory fragment to the storage space of the memory fragment is greater than a preset ratio, and the ratio of the storage space occupied by the data stored in the memory fragment to the storage space of the memory fragment is not greater than the preset ratio, and the elimination sequence of the memory fragment is within a preset range;
step 15: writing the data stored in the memory fragment into the storage medium corresponding to the sub-region, adding 1 to the disk-refreshing count in the index of the memory fragment, and changing the state of the memory fragment into an available state.
In this specific implementation manner, in order to enable each sub-region to include more memory slices in an available state, the electronic device may determine, for each sub-region, states of some memory slices in the sub-region that are in an unavailable state as an available state.
For each sub-region, the electronic device may first determine the memory slice in the unavailable state in the sub-region, and then, for each determined memory slice in the unavailable state, the electronic device may detect whether a memory update count and a flash count in an index of the memory slice are the same.
If the memory update count in the index of the memory fragment is the same as the disk refreshing count, it indicates that the data currently stored in the memory fragment has been written into the corresponding storage medium, and further, the electronic device can further determine whether the elimination sequence of the memory fragment is within the preset range;
furthermore, when the obsolete sequence of the memory fragment is judged to be located within the preset range, the state of the memory fragment can be determined as the available state.
Wherein, the preset range may be: any integer value which is not larger than the total number of the memory fragments included in the sub-region; determining the arrangement sequence of each memory fragment in an unavailable state according to the sequence of values of the obsolete sequences of the memory fragments in the unavailable state of the subarea from small to large, wherein the obsolete sequences of the memory fragments in the unavailable state before the integer value are positioned in a preset range;
the preset range may be: any percentage not greater than 1; after the arrangement sequence of the memory fragments in the unavailable state is determined according to the sequence of the values of the elimination sequences of the memory fragments in the unavailable state of the sub-region from small to large, the ratio of the arrangement digit to the total number of the memory fragments in the unavailable state of the sub-region is smaller than that of the elimination sequences of the memory fragments in the percentage and is within a preset range.
Correspondingly, if the memory update count and the flash count in the index of the memory fragment are different, it indicates that the data currently stored in the memory fragment is not written into the storage medium corresponding to the sub-region, and further, the electronic device may further determine whether the memory fragment meets any one of the preset elimination conditions;
furthermore, when the memory fragment is judged to meet any one of the preset elimination conditions, the data stored in the memory fragment can be written into the storage medium corresponding to the sub-region, the disk-refreshing count in the index of the memory fragment is increased by 1, and the state of the memory fragment is determined as the available state.
When detecting that the memory update count and the flash count in the index of the memory fragment are different, the electronic device may further determine whether a ratio of a storage space occupied by data stored in the memory fragment to a storage space of the memory fragment is greater than a preset ratio;
if the number of the data in the memory fragment is greater than the number of the data in the memory fragment, writing the data stored in the memory fragment into a storage medium corresponding to the sub-region, adding 1 to the flash count in the index of the memory fragment, and changing the state of the memory fragment into an available state;
if not, further judging whether the elimination sequence of the memory fragment is located in the preset range;
if the current status is in the available status, the data stored in the memory segment may be written into the storage medium corresponding to the sub-area, the disk-flushing count in the index of the memory segment is incremented by 1, and the status of the memory segment is changed to the available status.
Optionally, in a specific implementation manner, the data storage method provided in the embodiment of the present invention may further include the following step 9:
and step 9: for each sub-area, when the sub-area meets a preset disk refreshing condition, writing data stored in each memory sub-slice in the sub-area into a storage medium corresponding to the sub-area;
wherein, the preset brush plate condition comprises: the data volume of the stored data exceeds the preset data volume, or the time length from the last disk brushing reaches the preset time length.
In this specific implementation manner, for each sub-region, the electronic device may detect whether the sub-region meets a preset scrub condition, and write data stored in each memory tile in the sub-region into a storage medium corresponding to the sub-region when it is detected that the sub-region meets the preset scrub condition.
For each sub-region, the electronic device may detect whether the data amount of the data stored in the sub-region exceeds a preset data amount, so that when the data amount of the data stored in the sub-region exceeds the preset data amount, the data stored in each memory slice in the sub-region may be written to the storage medium corresponding to the sub-region;
for each sub-region, the electronic device may also periodically write the data stored in each memory slice in the sub-region to the storage medium corresponding to the sub-region every preset time period.
Optionally, after writing the data stored in each memory slice in the sub-area to the storage medium corresponding to the sub-area, the electronic device may also empty each memory slice in the sub-area in which the data is stored.
Optionally, in a specific implementation manner, the data storage method provided in the embodiment of the present invention may further include the following step 10:
step 10: for each sub-region, when the number of the memory fragments in the usable state in the sub-region is smaller than a preset number, changing the state of at least one memory fragment in the unusable state in the sub-region into the usable state, so that after the change is completed, the number of the memory fragments in the usable state in the sub-region is not smaller than the preset number.
In this specific implementation manner, for each sub-region, the electronic device may detect the number of the memory fragments in the available state included in the sub-region, so that when it is detected that the number of the memory fragments in the available state in the sub-region is smaller than the preset number, the electronic device may change the state of at least one memory fragment in the unavailable state in the sub-region to the available state, so that after the change is completed, the number of the memory fragments in the available state in the sub-region is not smaller than the preset number.
The number of the memory fragments to be subjected to state change may be: after the change is completed, the number of the memory fragments in the available state in the sub-area is not less than the minimum number of the preset number; the following steps can be also included: the number of memory slices in an unavailable state; the number of the memory fragments in the usable state included in the sub-area may also be specified, and after the state is changed once, it is detected whether the number of the memory fragments in the usable state included in the sub-area is smaller than a preset value, and if the number of the memory fragments in the usable state is smaller than the preset value, the states of the memory fragments in the unusable state in the specified number may be changed to the usable state again.
Furthermore, when the number of the memory slices to be subjected to the state change is a specified number and the number of the included memory slices in the unavailable state is less than the specified number, the states of all the memory slices in the unavailable state can be changed into the available state.
Next, a data reading method provided in an embodiment of the present invention is described in detail with reference to the accompanying drawings.
Fig. 10 is a schematic flowchart of a data reading method according to an embodiment of the present invention, and as shown in fig. 10, the reading method may include the following steps:
s1001: acquiring a data reading instruction generated by a target process, and determining a second target sub-area corresponding to a second target storage medium for storing data to be read;
when a target process needs to obtain some data from a storage medium of an electronic device in operation, the target process may generate a data reading instruction about the data to be read, where the data reading instruction carries an identifier of the storage medium in which the data to be read is stored, and a target address range of the data to be read in the storage medium.
In this way, the electronic device may obtain a data reading instruction generated by the target process, and after obtaining the data reading instruction, determine a second target sub-region corresponding to a second target storage medium for storing data to be read, and a target address range of the data to be read in the storage medium.
S1002: searching data to be read in each memory fragment of the second target sub-area, wherein the memory fragment is stored with the data;
furthermore, after a second target sub-area corresponding to a second target storage medium for storing data to be read is determined, the data to be read can be searched in each memory slice of the second target sub-area, in which data is stored.
In the second target sub-area, each memory slice storing data may establish a corresponding relationship with an address range in the second target storage medium, so that the electronic device may search the target address range in the corresponding relationship, and in the corresponding relationship, the memory slice corresponding to the target address range may store data to be read.
Furthermore, the electronic device may determine an offset address corresponding to the target address range in the memory slice corresponding to the target address range, so as to check whether data is stored in the offset address and whether the stored data is all data of the data to be read.
S1003: if all the data of the data to be read are found, writing the data to be read into the virtual cache of the target process;
if all data of the data to be read can be found in each memory partition of the second target sub-area, in which the data is stored, the electronic device can directly read the found data, and thus, the read data to be read is written into the virtual cache of the target process for use by the target process.
S1004: if partial data of the data to be read is found, writing the partial data and the missing data read from the second target storage medium into the virtual cache of the target process;
wherein, the missing data is: the data to be read is not stored in the second target subarea;
if partial data of the data to be read can be found in each memory fragment of the second target sub-region, where the data is stored, the electronic device may determine missing data in the data to be read and a storage address of the missing data in the second target storage medium according to an offset address of the partial data in the memory fragment, a correspondence between the memory fragment and a storage range in the second target storage medium, and a target address range. Therefore, the electronic device can read the missing data from the second target storage medium according to the determined storage address. Furthermore, the electronic device may write the partial data read from the memory slice of the second target sub-region and the missing data read from the second target storage medium into the virtual cache of the target process for use by the target process.
Optionally, the electronic device may write part of the data read from the memory partition of the second target sub-region into the virtual cache of the target process, then read the missing data from the second target storage medium, and write the missing data directly into the virtual cache of the target process;
optionally, the electronic device may write part of the data read from the memory partition of the second target sub-region into the virtual cache of the target process, then read the missing data from the second target storage medium, and write the missing data into the memory partition of the second target sub-region, so that the missing data is read from the memory partition of the second target sub-region, and the missing data is written into the virtual cache of the target process;
optionally, the electronic device may write part of the data read from the memory partition of the second target sub-region into the virtual cache of the target process, then read the missing data from the second target storage medium, write the missing data directly into the virtual cache of the target process, and write the missing data into the memory partition of the second target sub-region;
optionally, the electronic device may read missing data from the second target storage medium, and write the missing data into the memory partition of the second target sub-region, so that complete data to be read is read from the memory partition of the second target sub-region, and the read data to be read is written into the virtual cache of the target process.
S1005: and if the data to be read is not found, writing the data to be read from the second target storage medium into the virtual cache of the target process.
If any data in the data to be read cannot be found in each memory fragment of the second target sub-area, the electronic device may directly read the data to be read from the second target storage medium according to the target address range, and write the data to be read from the second target storage medium into the virtual cache of the target process for the target process to use.
Optionally, the electronic device may read data to be read from the second target storage medium, and directly write the data to be read into the virtual cache of the target process;
optionally, the electronic device may read data to be read from the second target storage medium, and write the data to be read into the memory partition of the second target sub-region, so that the data to be read is read from the memory partition of the second target sub-region, and the data to be read is written into the virtual cache of the target process;
optionally, the electronic device may read data to be read from the second target storage medium, and directly write the data to be read into the virtual cache of the target process, and at the same time, write the data to be read into the memory partition of the second target sub-region.
In the data reading method provided in the embodiment of the present invention, a manner of writing missing data read from the second target storage medium or data to be stored into the memory partition in the second target sub-area is similar to that of the data storage method provided in the embodiment of the present invention, and is not described again here.
Corresponding to the data storage method and the data reading method provided by the embodiment of the invention, the embodiment of the invention also provides a data storage device and a data reading device.
Fig. 11 is a schematic structural diagram of a data storage device according to an embodiment of the present invention, where the data storage device is applied to an electronic device, and sub-regions respectively corresponding to storage media of the electronic device are set in a shared memory of the electronic device, where each sub-region includes multiple memory segments; as shown in fig. 11, the storage device may include the following modules:
a data obtaining module 1110, configured to obtain data to be stored;
an information determining module 1120, configured to determine a first target storage medium for storing the data to be stored, and a target address range of the data to be stored in the first target storage medium;
a fragment determining module 1130, configured to determine, based on the target address range, a target memory fragment for storing the to-be-stored data in a first target sub-region corresponding to the first target storage medium;
a data storage module 1140, configured to store the data to be stored to the target memory slice.
As can be seen from the above, since data in the shared memory is visible to all processes, when multiple processes need to access the same address range of the same storage medium, the data stored in the same address range of the same storage medium may be stored in a sub-region of the shared memory corresponding to the storage medium, so that the multiple processes can access the data from the sub-region, and consistency of the data accessed by the multiple processes is ensured.
Optionally, in a specific implementation manner, the fragment determining module 1130 includes:
a first fragmentation judgment submodule for judging whether a first fragmentation exists in a first target sub-area corresponding to the first target storage medium; if yes, triggering an address range judgment submodule; wherein the first segment is: establishing a corresponding relation with an address range in the first target storage medium, wherein the corresponding address range in the first target storage medium and the target address range have overlapped memory fragments;
the address range judging submodule is used for judging whether the target address ranges are all located in the address ranges corresponding to the first fragments in the first target storage medium; if yes, triggering a first target fragment determining submodule; otherwise, triggering a second fragment determining submodule;
the first target fragment determining submodule is used for determining the first fragment as a target memory fragment;
the second fragmentation determination submodule is configured to determine, from the first target sub-area, a second fragmentation used for storing remaining data, and determine the first fragmentation and the second fragmentation as a target memory fragmentation; wherein the remaining data is: and the data which cannot be stored in the first fragment in the data to be stored.
Optionally, in a specific implementation manner, the second fragment determining submodule includes:
a space judgment unit, configured to judge whether a storage space of the memory partition in an available state in the first target sub-area satisfies a data amount of remaining data; if yes, triggering a second fragment determining unit; otherwise, triggering the state change unit;
the second fragmentation determining unit is configured to select a second fragmentation used for storing remaining data from the memory fragmentation in the available state;
the state changing unit is configured to change a state of at least one memory slice in the first target sub-area that is in an unavailable state to an available state, so that after the change is completed, a storage space of the memory slice in the available state in the first target sub-area satisfies a data amount of the remaining data;
the second partition determining unit is further configured to select, after the change is completed, a second partition for storing remaining data from the memory partitions in the available state in the first target sub-area.
Optionally, in a specific implementation manner, the apparatus further includes:
before the state of at least one memory fragment in the unavailable state in the first target sub-area is changed into the available state, determining a memory fragment to be refreshed according to the relationship between the update times and the disk refreshing times of each memory fragment in the unavailable state in the first target sub-area, and writing data stored in the memory fragment to be refreshed into the first target storage medium; after the writing is finished, triggering the state changing unit;
wherein, the brush dish is: writing the stored data to the first target storage medium.
Optionally, in a specific implementation manner, the apparatus further includes:
a write condition determining module, configured to determine whether the first partition meets any condition of preset write conditions before determining the first partition as a target memory partition; if yes, triggering the first target fragment determining submodule, otherwise, triggering a data writing module; wherein the preset writing condition comprises: the first fragment does not store data, the first fragment stores data, and a neutral zone does not exist between an address range of the stored data in the first fragment and an address range of the target address range corresponding to the first fragment;
the data writing module is configured to write the data to be stored into the target address range of the first target storage medium.
Optionally, in a specific implementation manner, the apparatus further includes:
a third partition determining submodule, configured to determine, if the first partition does not exist in a first target sub-area corresponding to the first target storage medium, a third partition for storing the data to be stored from the first target sub-area.
Optionally, in a specific implementation manner, the data storage module 1140 is specifically configured to:
if the target memory fragment is the first fragment, determining a target starting offset address in the first fragment, and writing the data to be stored into the first fragment from the target starting offset address; wherein, the storage address of the target start offset address in the first target storage medium is the start address of the target address range;
if the target memory fragment is the first fragment and the second fragment, determining a specified offset address range in the first fragment, writing target data in the data to be stored into the specified offset address range, and writing residual data in the data to be stored into the second fragment; wherein the address range corresponding to the specified offset address range in the first target storage medium is an address range of the first segment that coincides with the target address range in the address range corresponding to the first target storage medium, and the target data is: the storage address in the first target storage medium is the data of the storage address corresponding to the specified offset address range; the remaining data are: and the data to be stored cannot be stored in the first fragment.
Optionally, in a specific implementation manner, each sub-region further includes: sharing a memory index area; the shared memory index area comprises: sharing memory header information and the index of each memory slice in the sub-area; the shared memory header information includes: counting IO, wherein the index of each memory fragment comprises: updating a memory, counting by brushing a disk, and eliminating a sequence; the memory updating count is used for indicating the updating times of the data stored in the memory fragment, the disk refreshing count is used for indicating the times of writing the data stored in the memory fragment into the corresponding storage medium, and the eliminating sequence is used for indicating the sequence of changing the memory fragment from the unavailable state to the available state; the device further comprises:
a first index updating module, configured to add 1 to an IO count in shared memory header information of a sub-region where the memory segment is located after data is written into each memory segment, update a deselected sequence in index information of the memory segment to the IO count added with 1, and add 1 to a memory update count in the index information of the memory segment;
and the second index updating module is used for adding 1 to the disk refreshing count in the index information of the memory slices to be refreshed after the data stored in each memory slice to be refreshed is written into the first target storage medium.
Optionally, in a specific implementation manner, the shared memory header information further includes: marking initialization of a memory; the device further comprises:
an initialization judgment module, configured to determine, for each storage medium, before the data to be stored is acquired, whether a sub-region corresponding to the storage medium is initialized according to a memory initialization flag in shared memory header information of the sub-region corresponding to the storage medium; if yes, triggering a data cleaning module; otherwise, triggering the initialization module;
the data cleaning module is used for determining the to-be-cleaned memory fragments with data stored in the sub-area corresponding to the storage medium, writing the data stored in the to-be-cleaned memory fragments into the storage medium, and marking the states of the to-be-cleaned memory fragments as available states;
the initialization module is configured to initialize the sub-area corresponding to the storage medium.
Optionally, in a specific implementation manner, the apparatus further includes a culling module, configured to execute, for each sub-region, the following steps:
detecting whether the memory updating count and the flash count in the index of each memory fragment in the unavailable state in the sub-area are the same or not;
if the memory fragments are the same, judging whether the eliminated sequence of the memory fragments is in a preset range;
if so, changing the state of the memory fragment into an available state;
if not, judging whether the storage allocation meets any one of preset elimination conditions; wherein the preset elimination conditions comprise: the ratio of the storage space occupied by the data stored in the memory fragment to the storage space of the memory fragment is greater than a preset ratio, and the ratio of the storage space occupied by the data stored in the memory fragment to the storage space of the memory fragment is not greater than the preset ratio, and the elimination sequence of the memory fragment is located in the preset range;
and when the condition is met, writing the data stored in the memory fragment into the storage medium corresponding to the sub-area, adding 1 to the disk refreshing count in the index of the memory fragment, and changing the state of the memory fragment into an available state.
Optionally, in a specific implementation manner, the apparatus further includes:
the data disk refreshing module is used for writing the data stored in each memory fragment in each sub-area into a storage medium corresponding to the sub-area when the sub-area meets a preset disk refreshing condition;
wherein, the preset brushing condition comprises: the data volume of the stored data exceeds the preset data volume, or the time length from the last disk brushing reaches the preset time length.
Optionally, in a specific implementation manner, the apparatus further includes:
and the state changing module is used for changing the state of at least one memory fragment in the unavailable state in the sub-region into the available state when the number of the memory fragments in the available state in the sub-region is less than the preset number aiming at each sub-region, so that the number of the memory fragments in the available state in the sub-region is not less than the preset number after the change is finished.
Fig. 12 is a schematic structural diagram of a data reading apparatus according to an embodiment of the present invention, where the data reading apparatus is applied to an electronic device, and sub-regions respectively corresponding to storage media of the electronic device are set in a shared memory of the electronic device, where each sub-region includes a plurality of memory segments; as shown in fig. 12, the reading apparatus may include the following modules:
an instruction obtaining module 1210, configured to obtain a data reading instruction generated by a target process, and determine a second target sub-area corresponding to a second target storage medium for storing data to be read;
the data searching module 1220 is configured to search the to-be-read data in each memory fragment, in which data is stored, of the second target sub-region;
the first data reading module 1230 is configured to, if all data of the data to be read is found, write the data to be read into the virtual cache of the target process;
a second data reading module 1240, configured to, if a part of the data to be read is found, write the part of the data and missing data read from the second target storage medium into a virtual cache of the target process; wherein the missing data is: the data to be read is not stored in the second target subarea;
a third data reading module 1250, configured to, if the data to be read is not found, write the data to be read, which is read from the second target storage medium, into the virtual cache of the target process.
As can be seen from the above, since data in the shared memory is visible to all processes, when multiple processes need to access the same address range of the same storage medium, the data stored in the same address range of the same storage medium may be stored in a sub-region of the shared memory corresponding to the storage medium, so that the multiple processes can access the data from the sub-region, and consistency of the data accessed by the multiple processes is ensured.
Corresponding to the data storage method and the data reading method provided in the above embodiments of the present invention, an embodiment of the present invention further provides an electronic device, as shown in fig. 13, including a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, where the processor 1301, the communication interface 1302 and the memory 1303 complete communication with each other through the communication bus 1304,
a memory 1303 for storing a computer program;
the processor 1301 is configured to, when executing the program stored in the memory 1303, implement the steps of any data storage method provided in the foregoing embodiment of the present invention and/or the steps of any data reading method provided in the foregoing embodiment of the present invention.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Alternatively, the memory may be at least one storage medium located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
In another embodiment of the present invention, a computer-readable storage medium is further provided, in which a computer program is stored, and the computer program, when executed by a processor, implements the steps of any data storage method provided in the above embodiment of the present invention and/or the steps of any data reading method provided in the above embodiment of the present invention.
In a further embodiment provided by the present invention, a computer program product containing instructions is also provided, which when run on a computer, causes the computer to perform the steps of any of the data storage methods provided in the above-described embodiments of the present invention and/or the steps of any of the data reading methods provided in the above-described embodiments of the present invention.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage medium, such as a server, a data center, etc., that includes one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), among others.
It should be noted that, in this document, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on differences from other embodiments. In particular, apparatus embodiments, electronic device embodiments, computer-readable storage medium examples, and computer program product embodiments are described for simplicity as they are substantially similar to method embodiments, where relevant, reference may be made to some descriptions of method embodiments.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims (15)

1. A data storage method is characterized in that the method is applied to electronic equipment, sub-regions corresponding to storage media of the electronic equipment are arranged in a shared memory of the electronic equipment, and each sub-region comprises a plurality of memory fragments; the method comprises the following steps:
acquiring data to be stored;
determining a first target storage medium for storing the data to be stored and a target address range of the data to be stored in the first target storage medium;
determining a target memory fragment for storing the data to be stored in a first target sub-area corresponding to the first target storage medium based on the target address range;
and storing the data to be stored to the target memory fragment.
2. The method according to claim 1, wherein the step of determining, in a first target sub-area corresponding to the first target storage medium, a target memory slice for storing the data to be stored based on the target address range comprises:
judging whether a first fragment exists in a first target subregion corresponding to the first target storage medium; wherein the first segment is: establishing a corresponding relation with an address range in the first target storage medium, wherein the corresponding address range in the first target storage medium and the target address range have overlapped memory fragments;
if yes, judging whether the target address range is located in the address range corresponding to the first fragment in the first target storage medium;
if so, determining the first fragment as a target memory fragment;
otherwise, determining a second fragment for storing the residual data from the first target sub-region, and determining the first fragment and the second fragment as target memory fragments; wherein the remaining data is: and the data which cannot be stored in the first fragment in the data to be stored.
3. The method according to claim 2, wherein the step of determining the second fragment for storing the remaining data from the first target sub-area comprises:
judging whether the storage space of the memory fragment in the available state in the first target subregion meets the data volume of the residual data or not;
if yes, selecting a second fragment for storing the residual data from the memory fragments in the available state;
otherwise, changing the state of at least one memory fragment in the unavailable state in the first target sub-region into an available state, so that after the change is completed, the storage space of the memory fragment in the available state in the first target sub-region meets the data volume of the residual data;
and after the change is completed, selecting a second fragment for storing the remaining data from the memory fragments in the available state in the first target sub-area.
4. The method according to claim 3, wherein before the step of changing the state of at least one memory slice in an unavailable state in the first target sub-area to an available state, the method further comprises:
determining memory fragments to be refreshed according to the relationship between the updating times and the disk refreshing times of each memory fragment in an unavailable state in the first target sub-area, and writing data stored in the memory fragments to be refreshed into the first target storage medium; wherein, the brush dish is: writing the stored data to the first target storage medium;
after the writing is completed, the step of changing the state of at least one memory slice in the first target sub-area which is in the unavailable state into the available state is executed.
5. The method of claim 2, wherein prior to the step of determining the first partition as a target memory partition, the method further comprises:
judging whether the first fragment meets any one of preset writing conditions; wherein the preset writing condition comprises: the first slice does not store data and the first slice stores data, and a neutral zone does not exist between the address range of the stored data in the first slice and the corresponding address range of the target address range in the first slice;
if yes, determining the first fragment as a target memory fragment;
otherwise, writing the data to be stored into the target address range of the first target storage medium.
6. The method of claim 2, further comprising:
and if the first fragment does not exist in a first target sub-area corresponding to the first target storage medium, determining a third fragment for storing the data to be stored from the first target sub-area.
7. The method according to claim 3, wherein the step of storing the data to be stored in the target memory slice comprises:
if the target memory fragment is the first fragment, determining a target starting offset address in the first fragment, and writing the data to be stored into the first fragment from the target starting offset address; wherein, the storage address of the target start offset address in the first target storage medium is the start address of the target address range;
if the target memory fragment is the first fragment and the second fragment, determining a specified offset address range in the first fragment, writing target data in the data to be stored into the specified offset address range, and writing residual data in the data to be stored into the second fragment; wherein the address range corresponding to the specified offset address range in the first target storage medium is an address range of the first segment that coincides with the target address range in the address range corresponding to the first target storage medium, and the target data is: the storage address in the first target storage medium is the data of the storage address corresponding to the specified offset address range; the remaining data is: and the data to be stored cannot be stored in the first fragment.
8. The method of claim 4, wherein each sub-region further comprises: sharing a memory index area; the shared memory index area comprises: sharing memory header information and an index of each memory slice in the sub-area; the shared memory header information includes: counting IO, wherein the index of each memory fragment comprises: updating the memory, counting by brushing the disk and eliminating the sequence; the memory updating count is used for indicating the updating times of the data stored in the memory fragment, the disk refreshing count is used for indicating the times of writing the data stored in the memory fragment into the corresponding storage medium, and the eliminating sequence is used for indicating the sequence of changing the memory fragment from the unavailable state to the available state; the method further comprises the following steps:
after data is written into each memory fragment, adding 1 to the IO count in the shared memory header information of the sub-region where the memory fragment is located, updating the eliminated sequence in the index information of the memory fragment into the IO count added with 1, and adding 1 to the memory update count in the index information of the memory fragment;
after the data stored in each memory slice to be refreshed is written into the first target storage medium, adding 1 to the disk refreshing count in the index information of the memory slice to be refreshed.
9. The method of claim 8, wherein the shared memory header information further comprises: initializing a mark of a memory; before the step of obtaining data to be stored, the method further comprises:
for each storage medium, determining whether the sub-region corresponding to the storage medium is initialized or not according to the memory initialization mark in the shared memory header information of the sub-region corresponding to the storage medium;
if yes, determining the memory fragments to be cleaned, which are stored with data, in the sub-area corresponding to the storage medium, writing the data stored in the memory fragments to be cleaned into the storage medium, and marking the state of the memory fragments to be cleaned as a usable state;
otherwise, initializing the sub-area corresponding to the storage medium.
10. The method of claim 9, further comprising:
for each sub-region, the following steps are performed:
detecting whether the memory updating count and the flash count in the index of each memory fragment in the unavailable state in the sub-area are the same or not;
if the memory fragments are the same, judging whether the eliminated sequence of the memory fragments is in a preset range;
if the memory is in the available state, the state of the memory fragment is changed into the available state;
if not, judging whether the storage allocation meets any one of preset elimination conditions; wherein the preset elimination conditions comprise: the ratio of the storage space occupied by the data stored in the memory fragment to the storage space of the memory fragment is greater than a preset ratio, and the ratio of the storage space occupied by the data stored in the memory fragment to the storage space of the memory fragment is not greater than the preset ratio, and the elimination sequence of the memory fragment is located in the preset range;
and when the condition is met, writing the data stored in the memory fragment into the storage medium corresponding to the sub-area, adding 1 to the flash count in the index of the memory fragment, and changing the state of the memory fragment into an available state.
11. The method according to any one of claims 1-9, further comprising:
for each sub-area, when the sub-area meets a preset disk refreshing condition, writing data stored in each memory sub-slice in the sub-area into a storage medium corresponding to the sub-area;
wherein, the preset brushing condition comprises: the data volume of the stored data exceeds the preset data volume, or the time length from the last disk brushing reaches the preset time length.
12. The method according to any one of claims 1-9, further comprising:
for each sub-region, when the number of the memory fragments in the usable state in the sub-region is smaller than a preset number, changing the state of at least one memory fragment in the unusable state in the sub-region into the usable state, so that after the change is completed, the number of the memory fragments in the usable state in the sub-region is not smaller than the preset number.
13. A data reading method is characterized in that the method is applied to electronic equipment, sub-areas corresponding to storage media of the electronic equipment are arranged in a shared memory of the electronic equipment, and each sub-area comprises a plurality of memory fragments; the method comprises the following steps:
acquiring a data reading instruction generated by a target process, and determining a second target sub-area corresponding to a second target storage medium for storing data to be read;
searching the data to be read in each memory fragment of the second target sub-area, in which the data is stored;
if all the data of the data to be read are found, writing the data to be read into the virtual cache of the target process;
if partial data of the data to be read is found, writing the partial data and missing data read from the second target storage medium into a virtual cache of the target process; wherein the missing data is: the data to be read is not stored in the second target subarea;
and if the data to be read is not found, writing the data to be read from the second target storage medium into the virtual cache of the target process.
14. A data storage device is characterized in that the data storage device is applied to electronic equipment, sub-areas respectively corresponding to storage media of the electronic equipment are arranged in a shared memory of the electronic equipment, and each sub-area comprises a plurality of memory fragments; the device comprises:
the data acquisition module is used for acquiring data to be stored;
the information determining module is used for determining a first target storage medium for storing the data to be stored and a target address range of the data to be stored in the first target storage medium;
the fragment determining module is used for determining a target memory fragment for storing the data to be stored in a first target subregion corresponding to the first target storage medium based on the target address range;
and the data storage module is used for storing the data to be stored to the target memory fragment.
15. A data reading device is characterized in that the data reading device is applied to electronic equipment, sub-areas corresponding to storage media of the electronic equipment are arranged in a shared memory of the electronic equipment, and each sub-area comprises a plurality of memory fragments; the device comprises:
the instruction acquisition module is used for acquiring a data reading instruction generated by the target process and determining a second target sub-area corresponding to a second target storage medium for storing data to be read;
the data searching module is used for searching the data to be read in each memory fragment of the second target sub-area, which stores the data;
the first data reading module is used for writing the data to be read into the virtual cache of the target process if all the data of the data to be read are found;
the second data reading module is used for writing the partial data and the missing data read from the second target storage medium into the virtual cache of the target process if the partial data of the data to be read is found; wherein the missing data is: in the data to be read, the data in the second target subarea is not stored;
and the third data reading module is used for writing the data to be read from the second target storage medium into the virtual cache of the target process if the data to be read is not found.
CN202110450349.5A 2021-04-25 2021-04-25 Data storage method, data reading method, data storage device and data reading device Pending CN115237819A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110450349.5A CN115237819A (en) 2021-04-25 2021-04-25 Data storage method, data reading method, data storage device and data reading device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110450349.5A CN115237819A (en) 2021-04-25 2021-04-25 Data storage method, data reading method, data storage device and data reading device

Publications (1)

Publication Number Publication Date
CN115237819A true CN115237819A (en) 2022-10-25

Family

ID=83666874

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110450349.5A Pending CN115237819A (en) 2021-04-25 2021-04-25 Data storage method, data reading method, data storage device and data reading device

Country Status (1)

Country Link
CN (1) CN115237819A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115905219A (en) * 2022-11-04 2023-04-04 上海威固信息技术股份有限公司 Data storage driving system for multi-channel data storage cloud library and transmission method thereof

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115905219A (en) * 2022-11-04 2023-04-04 上海威固信息技术股份有限公司 Data storage driving system for multi-channel data storage cloud library and transmission method thereof
CN115905219B (en) * 2022-11-04 2023-09-19 上海威固信息技术股份有限公司 Data storage driving system for multichannel data cloud storage library and transmission method thereof

Similar Documents

Publication Publication Date Title
US11853549B2 (en) Index storage in shingled magnetic recording (SMR) storage system with non-shingled region
CN110096227B (en) Data storage method, data processing device, electronic equipment and computer readable medium
US20170090794A1 (en) Method and Apparatus for Providing a Shared Nonvolatile Memory System Using a Distributed FTL Scheme
CN108874298B (en) Data storage method and device
KR101933766B1 (en) Methods and systems for improving flash memory flushing
CN109976669B (en) Edge storage method, device and storage medium
US7536536B1 (en) Method, system, and computer readable medium for updating and utilizing the contents of a non-essential region of a memory device
CN111090663A (en) Transaction concurrency control method, device, terminal equipment and medium
CN113625973B (en) Data writing method, device, electronic equipment and computer readable storage medium
CN112204515A (en) Logical to physical data structure
CN112835528A (en) Dirty page refreshing method and device, electronic equipment and storage medium
CN115857801A (en) Data migration method and device, electronic equipment and storage medium
CN114020711B (en) Storage space processing method and device, electronic equipment and readable storage medium
CN115237819A (en) Data storage method, data reading method, data storage device and data reading device
CN112416860B (en) Data rollback updating method, device, computer equipment and storage medium
CN116661690A (en) Method, device, computer equipment and storage medium for recording memory state
CN117331498A (en) Method, device, equipment and medium for constructing mapping table of solid state disk
CN116301614A (en) Memory data access method, system, device and storage medium
CN108959517B (en) File management method and device and electronic equipment
CN111104435B (en) Metadata organization method, device and equipment and computer readable storage medium
CN112015672A (en) Data processing method, device, equipment and storage medium in storage system
CN113253939A (en) Data processing method and device, electronic equipment and storage medium
CN117708000B (en) Random writing method and device of data, electronic equipment and storage medium
CN109144399B (en) Data storage method and device and electronic equipment
CN103514953A (en) Emulated electrically erasable memory having an address RAM for data stored in flash memory

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination