WO2017028514A1 - 一种数据存储、读取方法及装置 - Google Patents
一种数据存储、读取方法及装置 Download PDFInfo
- Publication number
- WO2017028514A1 WO2017028514A1 PCT/CN2016/073811 CN2016073811W WO2017028514A1 WO 2017028514 A1 WO2017028514 A1 WO 2017028514A1 CN 2016073811 W CN2016073811 W CN 2016073811W WO 2017028514 A1 WO2017028514 A1 WO 2017028514A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- format
- mapping relationship
- module
- value
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/221—Column-oriented storage; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2372—Updates performed during offline database operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0661—Format or protocol conversion arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
Definitions
- the present invention relates to the field of storage technologies, and in particular, to a data storage and reading method and apparatus.
- KeyValue In a distributed database, data is typically stored in the form of KeyValue (key-value).
- the Key part is used to uniquely identify a row of data, and is also used to quickly retrieve data. For example, by searching for a Key of a data, the data can be searched.
- the Key part is usually a specific structure that is customized by the user.
- the Value section is used to store the actual data. It can be considered that Key is an identification information for distinguishing different data, and value part is for real data content.
- a data When storing data in KeyValue format, there may be different storage methods. For example, if only one Value is stored in one row, if a data includes multiple Values, it may be divided into multiple rows for storage. This storage method may be called a single row. In the single Value mode, or for example, storing multiple Values in one row, one data may be stored in one row in total. This storage method may be referred to as a single row multiple value method.
- the application layer needs to specify the storage mode of the data each time the data is stored in the KeyValue format. For example, it is necessary to specify how many Values are stored in each row.
- the application layer needs to read the KeyValue format. The data is self-spliced into the original data.
- the storage strategy requires the application layer to participate in the designation or implementation, and the experience is not good for the user, and the burden on the application layer is relatively large.
- the application layer needs to specify the storage mode of the data separately, and the storage process is complicated.
- the embodiment of the invention provides a data storage and reading method and device, which are used to solve the technical problem that the data storage process is relatively complicated.
- a data storage method including:
- the second format is a KeyValue format, where the first mapping relationship is used to indicate data conversion according to the first format.
- the second data is stored.
- the method further includes:
- the second mapping relationship is used to indicate a row attribute and a column attribute corresponding to each Value obtained by data conversion according to the first format, and the a row attribute corresponding to each value in a mapping relationship is different from a row attribute corresponding to each value in the second mapping relationship, and/or a column attribute corresponding to each value in the first mapping relationship
- the column attributes corresponding to each value in the second mapping relationship are different;
- the stored second data is updated according to the third data.
- the method further includes:
- the fourth data is outputted to the application layer.
- a data reading method including:
- the first mapping relationship is used to indicate each Value corresponding to the KeyValue format obtained by converting the first data according to the first format Row attributes and column attributes;
- the output format converted data is fed back to the application layer.
- the method before the storing the second data, the method further includes:
- the second format is a KeyValue format
- the second data is stored.
- the method further includes:
- the second mapping relationship is used to indicate a row attribute and a column attribute corresponding to each Value obtained by data conversion according to the first format, and the a row attribute corresponding to each value in a mapping relationship is different from a row attribute corresponding to each value in the second mapping relationship, and/or a column attribute corresponding to each value in the first mapping relationship
- the column attributes corresponding to each value in the second mapping relationship are different;
- the stored second data is updated according to the third data.
- a data storage device including:
- a receiving module configured to receive first data in a first format to be written by an application layer
- An acquiring module configured to acquire a first mapping relationship between the data in the first format and the data in the second format;
- the second format is a KeyValue format, where the first mapping relationship is used to indicate that the first mapping
- a conversion module configured to convert the format of the first data into the second format according to the first mapping relationship, to obtain second data
- An operation module configured to store the second data.
- the device further includes a recovery module, a first update module, and a second update module;
- the recovery module is configured to: after the operating module stores the second data, restore the second data to the first data in the first format according to a first mapping relationship before updating;
- the first update module is configured to: update the stored first mapping relationship by using a second mapping relationship, where the second mapping relationship is used to indicate a row corresponding to each value obtained by data conversion according to the first format Attributes and column attributes, and the line attributes corresponding to each value in the first mapping relationship are different from the line attributes corresponding to each value in the second mapping relationship, and/or, in the first mapping relationship
- the column attribute corresponding to each value of the value is different from the column attribute corresponding to each value in the second mapping relationship;
- the converting module is further configured to: convert the format of the first data into a third format according to the updated first mapping relationship, to obtain third data;
- the second update module is configured to: update the stored second data according to the third data.
- the device further includes a reading module and an output module;
- the reading module is configured to: after the operation module stores the second data, if the instruction for reading the first data sent by the application layer is received, the stored second data is read ;
- the converting module is further configured to: format the second data according to the first mapping relationship Converting to the first format to obtain fourth data;
- the output module is configured to: feed back the fourth data to an application layer.
- a data reading apparatus including:
- a reading module configured to read the stored second data if receiving an instruction to read the first data sent by the application layer
- a conversion module configured to convert a KeyValue format of the second data into a first format according to a first mapping relationship, where the first mapping relationship is used to indicate a KeyValue format obtained by converting the first data according to the first format
- An output module for feeding back output format converted data to the application layer.
- the device further includes a receiving module, an acquiring module, and an operating module;
- the receiving module is configured to: before the reading module reads the stored second data, receive the first data in a first format to be written by an application layer;
- the acquiring module is configured to: acquire the first mapping relationship between the data in the first format and the data in the second format; the second format is a KeyValue format;
- the converting module is further configured to: convert the format of the first data into the second format according to the first mapping relationship, to obtain second data;
- the operation module is configured to: store the second data.
- the device further includes a recovery module, a first update module, and a second update module;
- the recovery module is configured to: after the operating module stores the second data, restore the second data to the first data in the first format according to a first mapping relationship before updating;
- the first update module is configured to: update the stored first mapping relationship by using a second mapping relationship, where the second mapping relationship is used to indicate a row corresponding to each value obtained by data conversion according to the first format An attribute and a column attribute, and the row attribute corresponding to each value in the first mapping relationship is different from the row attribute corresponding to each value in the second mapping relationship, and/or the a column attribute corresponding to each value in a mapping relationship is different from a column attribute corresponding to each value in the second mapping relationship;
- the converting module is further configured to: convert the format of the first data into a third format according to the updated first mapping relationship, to obtain third data;
- the second update module is configured to: update the stored second data according to the third data.
- a data storage device including:
- a receiver configured to receive first data in a first format to be written by an application layer
- a processor configured to acquire a first mapping relationship between the data in the first format and the data in the second format, and convert the format of the first data into the second according to the first mapping relationship Format, the second data is obtained;
- the second format is a KeyValue format, and the first mapping relationship is used to indicate a row attribute and a column attribute corresponding to each Value obtained by data conversion according to the first format;
- a memory for storing the second data.
- the processor is further configured to:
- the second mapping relationship is used to indicate a row attribute and a column attribute corresponding to each Value obtained by data conversion according to the first format, and the a row attribute corresponding to each value in a mapping relationship is different from a row attribute corresponding to each value in the second mapping relationship, and/or a column attribute corresponding to each value in the first mapping relationship
- the column attributes corresponding to each value in the second mapping relationship are different;
- the stored second data is updated according to the third data.
- the device is further included Transmitter
- the processor is further configured to: after receiving the second data by using the memory, if receiving an instruction sent by the application layer to read the first data, reading the stored second data Converting the format of the second data into the first format according to the first mapping relationship, to obtain fourth data;
- the transmitter is configured to: feed back the fourth data to an application layer.
- a data reading apparatus including:
- a processor configured to: read an instruction to read the first data sent by the application layer, and read the stored second data; and, according to the first mapping relationship, the key value of the second data stored in the memory Converting the KeyValue format to the first format; the first mapping relationship is used to indicate a row attribute and a column attribute corresponding to each Value of the KeyValue format obtained according to the first data conversion of the first format;
- the transmitter is configured to feed back the output format converted data to the application layer.
- the device further includes a receiver
- the receiver is configured to receive, before the processor reads the stored second data, the first data that is sent by the application layer in a first format to be written;
- the processor is further configured to: acquire the first mapping relationship between the data of the first format and the data of the second format; the second format is a KeyValue format; according to the first mapping relationship, Converting the format of the first data to the second format to obtain the second data;
- the memory is further configured to: store the second data.
- the processor is further configured to:
- the stored second data is updated according to the third data.
- a data storage device includes a processor and a memory, and the processor is connected to the memory through a bus;
- the memory is configured to store computer execution instructions that, when the data storage device is in operation, read the computer execution instructions stored by the memory to cause the data storage device to perform the method of any of the preceding aspects Data storage method.
- a data reading apparatus comprising a processor and a memory, the processor being connected to the memory via a bus;
- the memory is configured to store a computer execution instruction, when the data reading device is in operation, the processor reads the computer execution instruction stored in the memory, so that the data reading device performs any of the foregoing aspects The data reading method.
- the mapping between the format before the data storage and the KeyValue format (that is, the first mapping relationship) is set.
- the device automatically stores the data according to the first mapping relationship, without requiring too many users. Participation, and whether it is to set mapping, storage mapping or format conversion, no need to complete the application layer, the user environment is more friendly, improve user experience, reduce the burden of the application layer.
- data conversion and storage can be directly performed according to the mapping relationship, without having to specify the storage mode of the data every time, which reduces the complexity of the data processing process and is beneficial to popularization and application.
- FIG. 1 is a flowchart of a data storage method according to an embodiment of the present invention.
- FIG. 2 is a flowchart of a data reading method according to an embodiment of the present invention.
- 3A is a schematic diagram of a data storage process according to an embodiment of the present invention.
- 3B is a schematic diagram of a data reading process according to an embodiment of the present invention.
- FIG. 4 is a structural block diagram of a data storage device according to an embodiment of the present invention.
- FIG. 5 is a structural block diagram of a data reading apparatus according to an embodiment of the present invention.
- 6A-6B are schematic structural diagrams of a data storage device according to an embodiment of the present invention.
- FIGS. 7A-7B are schematic structural diagrams of a data reading apparatus according to an embodiment of the present invention.
- FIG. 8 is another schematic structural diagram of a data storage device according to an embodiment of the present invention.
- FIG. 9 is another schematic structural diagram of a data reading apparatus according to an embodiment of the present invention.
- the technical solution of the embodiment of the present invention can be applied to a distributed database of the KeyValue type.
- the data exists in the form of KeyValue.
- the Key part of the data is used to uniquely identify a row of data records, and is also used to retrieve index information of the data.
- the Key part of the data may be a specific structure defined by the user.
- the Value part of the data is used to store the actual data content.
- the KeyValue distributed database can be used to store web page information.
- a Key may be used to store a URL of a webpage (Uniform Resoure Locator), and the actual content of the webpage indicated by the URL may be stored in the Value section, as follows:
- the KeyValue distributed database can be used to store some basic information of the user.
- the ID can be used as the content of the Key part, and the others are the contents of the Value part.
- the basic information includes a plurality of Values, that is, Name, Address, Age, Gender, Phone, and Education can be regarded as a Value, respectively.
- a row of data records in the KeyValue distributed database may include only one Value, or may include multiple Values. If a row of data records includes only one Value, then such information may be divided into multiple rows for storage. If a row of data records includes multiple Values, such information can be stored as a row of records.
- a row of data records includes multiple Values
- a basic information can also be stored as multiple rows, depending on whether a row of data records includes several Values.
- the basic information is stored in the KeyValue distributed database, and the specific storage manner has been described previously.
- the new content has no effect on the existing data records.
- each row of data records of the KeyValue distributed database (in the embodiment of the present invention, a row of data records, which can also be understood as a data record) usually has a specific structure definition, and the following information may be included in the structure definition.
- a row of data records which can also be understood as a data record
- Key can be used to identify the uniqueness of each row of data records, and can be used to retrieve data
- Timestamp used to record the generation time of a row of data records
- Type The type used to record a row of data records
- Column name information Similar to the concept of column names in a normal relational database, it can reflect which part of the data is recorded in a row of data records.
- the first format may be any type of data format, for example, the first format may be the format of the basic information of the user as described above, or may be the format of the webpage data, and the like.
- the process of setting the mapping relationship, the storage mapping relationship, and performing the format conversion does not need to be completed by the application layer, but may be completed by other modules under the application layer, and is invisible to the application layer.
- the user experience is better and the application layer is less burdensome.
- a specific data platform running under the application layer can be provided, the data platform can correspond to a middleware, and the middleware can be understood as a function interface capable of providing some operations for completion through the data platform, by calling the middleware
- the function interface can complete the corresponding operations in the data platform.
- the processes of setting a mapping relationship, a storage mapping relationship, and performing format conversion may all be completed in the data platform by calling middleware.
- system and “network” are used interchangeably herein.
- the term “and/or” in this context is merely an association describing the associated object, indicating that there may be three relationships, for example, A and / or B, which may indicate that A exists separately, and both A and B exist, respectively. B these three situations.
- the character "/" in this article unless otherwise specified, generally indicates that the contextual object is an "or" relationship.
- an embodiment of the present invention provides a data storage method, and a process of the method is described as follows.
- Step 101 Receive first data of a first format to be written delivered by an application layer.
- the user writes the first data to the KeyValue distributed database.
- the format of the first data is the first format.
- the user interacts with the application layer, that is, the application layer receives the first data, and sends the first data to be written to other modules.
- the other module may be the data platform including the middleware, and the middleware may be in the middle.
- Step 101 correspondingly acquires the first data. In this way, subsequent operations such as entire format conversion, storage, and the like on the first data are invisible to the application layer.
- Step 102 Acquire a first mapping relationship between data in the first format and data in the second format.
- the second format is a KeyValue format, and the mapping relationship is used to indicate a row attribute and a column attribute corresponding to each Value obtained by data conversion according to the first format.
- a possible mapping relationship between the data of the first format and the data of the second format is referred to as a first mapping relationship.
- the first mapping relationship between the data of the first format and the data of the KeyValue format is pre-stored in the system, for example, may be stored in a database, and the stored location may be invisible to the application layer.
- a key and at least one Value may be included.
- the row attribute and the column attribute of each Value may be known, that is, where each Value belongs One column and which row, so you can determine how each Value is stored.
- the mapping relationship may be preset by the system, for example, the system default mapping relationship may be used, or may be preset by the user, which is not limited by the present invention.
- the method before receiving the first data in the first format that is written, the method further includes:
- the first mapping relationship is received and stored.
- the system may first set the first mapping relationship by itself, or the first mapping relationship may also be preset by the user.
- the system may store the first mapping relationship in a database, for example, The database in the specific data platform as described above, or can also be stored in the corresponding file, no matter where it is stored, is invisible to the application layer, and the present invention does not limit the specific storage location.
- mapping relationship after the mapping relationship is stored, the mapping relationship can also be changed according to the situation.
- the first mapping relationship between the data of the first format and the data of the KeyValue format is first stored. If it is necessary to store the mapping relationship between the data of the other first format and the data of the KeyValue format, the first mapping relationship is involved. A mapping relationship is changed, or the first mapping relationship is deleted first, and then another mapping relationship obtained is stored. Whether it is directly obtained by modifying the original mapping relationship to obtain a new mapping relationship, or deleting the original mapping relationship and then storing the new mapping relationship, it can be regarded as a new mapping relationship by changing the original mapping relationship.
- the method before receiving the first data in the first format that is written, the method further includes:
- the third mapping relationship is used to indicate a row attribute and a column attribute corresponding to each Value obtained by data conversion according to the first format.
- the method further includes:
- the row attribute corresponding to each value in the first mapping relationship is different from the row attribute corresponding to each value in the third mapping relationship, and/or, in the first mapping relationship
- the column attribute corresponding to each value is different from the column attribute corresponding to each value in the third mapping relationship;
- the updated third mapping relationship is stored.
- the updated third mapping relationship is the first mapping relationship. Updating the third mapping relationship with the first mapping relationship may be understood as replacing the third mapping relationship with the first mapping relationship.
- mapping relationship is changed once, that is, the mapping relationship started to be stored is the third mapping relationship, and later changed to the first mapping relationship.
- mapping relationship needs to be changed, if the data has not been stored according to the mapping relationship that needs to be changed, it can be directly changed, as described in this embodiment, and if the data has been stored according to the mapping relationship that needs to be changed, It involves the re-storing of data, which will be described later.
- Step 103 Convert the format of the first data into the second format according to the first mapping relationship, to obtain the second data.
- the specific data platform may invoke the first mapping relationship, and the format of the first data may be converted according to the first mapping relationship.
- the converted first data is referred to as second data.
- the first data is:
- the second data converted according to the first mapping relationship is:
- the first mapping relationship defines that the row attribute of the first value (ie, ColA) in the first format is the row attribute of the first row, and the column attribute is the second column (the column in which the ID is located, for example, the first column) Column attribute, the second value (ie ColB) and the third value (ie ColC) line attribute are the second line of the row attribute, the second value of the column attribute is the second column of the column attribute, the third The column attribute of value is the column attribute of the third column, the row attribute of the fourth value (ie, ColD) and the fifth value (ie, ColE) is the row attribute of the third line, and the column attribute of the fourth value is the second.
- the column attribute of the column, the column attribute of the fifth value is the column attribute of the third column.
- Step 104 Store the second data.
- the second data can be stored.
- the second data can be stored, for example, in the KeyValue database
- the KeyValue database may include, for example, a plurality of data tables. For example, if one mapping relationship corresponds to one data table, the second data may be stored in a data table corresponding to the first mapping relationship.
- the second data can also be stored in other possible locations, and the invention is not limited.
- mapping relationship can be changed before the data is stored.
- mapping relationship can be changed before the data is stored.
- mapping relationship can be changed before the data is stored. The following describes how to change the mapping relationship after storing the data.
- the method further includes:
- the second mapping relationship is used to indicate the row attribute and the column attribute corresponding to each Value obtained by the data conversion according to the first format, and each value in the first mapping relationship
- the corresponding row attribute is different from the row attribute corresponding to each value in the second mapping relationship, and/or the column attribute corresponding to each value in the first mapping relationship and the column corresponding to each value in the second mapping relationship Different attributes;
- the stored second data is updated according to the third data.
- the updated first mapping relationship is the second mapping relationship. Updating the first mapping relationship with the second mapping relationship may be understood as replacing the first mapping relationship with the second mapping relationship.
- the data corresponding to the first mapping relationship may be prior to the update.
- the format is restored to the original format corresponding to the data according to the first mapping relationship.
- the format corresponding to the original data is the first format.
- the first format data is re-mapped according to the updated first mapping.
- the relationship is mapped, and the data obtained after the mapping is stored.
- the data obtained by re-mapping the first data according to the updated first mapping relationship in the embodiment of the present invention is referred to as third data. Updating the stored second data according to the third data can be understood as replacing the second data with the third data.
- the two data formats corresponding to the second mapping relationship are respectively the first format and The third format, the third format may also be the KeyValue format.
- the row attribute corresponding to each value in the third mapping relationship may be the same as or different from the row attribute corresponding to each value in the second mapping relationship, and/or each value in the third mapping relationship corresponds to
- the column attributes corresponding to each value in the second mapping relationship may be the same or different.
- the first mapping relationship, the second mapping relationship, and the third mapping relationship are mappings between data in the first format and data in the KeyValue format. Therefore, the data in the first format is in accordance with the first mapping.
- the relationship between the second mapping relationship and the third mapping relationship is obtained, and the data in the second format is obtained, that is, the data in the KeyValue format, but only the row attribute or the column attribute corresponding to each value in the data of the obtained KeyValue format. May be different.
- the first data is:
- the third data converted according to the third mapping relationship is:
- the second mapping relationship defines that the row attribute of the first value (ie, ColA) in the first format is the row attribute of the first row, and the column attribute is the second column (the column in which the ID is located, for example, the first column)
- the column attribute, the second value (ie ColB), the third value (ie ColC) and the fourth value (ie ColD) have the row attribute of the second row and the second value of the column attribute.
- the column attribute of the second column, the column attribute of the third value is the column attribute of the third column, the column attribute of the fourth value is the column attribute of the fourth column, and the row attribute of the fifth value (ie ColE) is the third attribute.
- the row attribute of the row, and the column property is the column property of the second column.
- the method further includes:
- the fourth data is outputted to the application layer.
- the application layer may send the instruction to another module, for example, to a specific data platform as described above, and the other module receives the read the first data.
- the first mapping relationship may be invoked again (in this embodiment, the first mapping relationship is not updated according to the second mapping relationship), and the format of the stored second data corresponding to the first data is re-created according to the first mapping relationship. Converting to the first format, obtaining the fourth data, and feeding back the fourth data to the application layer.
- the method further includes:
- the fifth data is outputted to the application layer.
- the application layer still reads the first data. Then, after receiving the instruction for reading the first data, the application layer may send the instruction to another module, for example, to a specific data platform as described above, and the other module receives the read the first data.
- the second mapping relationship may be invoked (in this embodiment, the first mapping relationship is updated according to the second mapping relationship), and the format of the stored third data corresponding to the first data is re-converted according to the second mapping relationship. For the first format, the fifth data is obtained and the fifth data is fed back to the application layer.
- the application layer needs to specify the storage mode of the data each time the data is stored in the KeyValue format. For example, it is necessary to specify how many Values are stored in each row.
- the application layer needs to read the KeyValue format.
- the data is self-spliced into the original data. It can be seen that the storage strategy requires the application layer to participate in the designation or implementation. The experience is not good for the user, and the burden of the application layer is relatively heavy. And, When storing data, the application layer needs to specify how the data is stored, and the process is complicated.
- mapping relationship can be directly performed according to whether the storage relationship or the reading is performed, without excessive participation of the user, and regardless of Whether the mapping relationship, the storage mapping relationship, or the format conversion is performed, the application layer is not required to be completed, and the experience is better for the user, and the burden of the application layer is reduced.
- mapping relationship is set, data conversion and storage can be directly performed according to the mapping relationship, without having to specify the storage mode of the data every time, the complexity of the data processing process is reduced, and the data storage process and the reading process are simplified.
- the mapping relationship and the data may be stored in a specific data platform as described above, if the mapping relationship is to be set, the mapping relationship is updated, the mapping relationship is deleted, the mapping relationship is invoked, Operations such as reading data, writing data, or data conversion can call the corresponding function interface through the middleware to operate the mapping relationship or data stored in the data platform.
- an embodiment of the present invention provides a data reading method, and the process of the method is described as follows.
- Step 201 If the instruction for reading the first data sent by the application layer is received, the stored second data is read;
- Step 202 Convert the KeyValue format of the second data into the first format according to the first mapping relationship.
- the first mapping relationship is used to indicate a row attribute and a column attribute corresponding to each Value of the KeyValue format obtained by the data conversion according to the first format;
- Step 203 Feed back the output format converted data to the application layer.
- the format converted data may be the fourth data as described in the previous embodiment.
- the method before the storing the stored second data, the method further includes:
- the second format is a KeyValue format
- the method further includes:
- the second mapping relationship is used to indicate the row attribute and the column attribute corresponding to each Value obtained by the data conversion according to the first format, and each value in the first mapping relationship
- the corresponding row attribute is different from the row attribute corresponding to each value in the second mapping relationship, and/or the column attribute corresponding to each value in the first mapping relationship and the column corresponding to each value in the second mapping relationship Different attributes;
- the stored first data is updated according to the third data.
- the method introduced in the flow of Fig. 2 is a method of data reading
- the method introduced in the flow of Fig. 1 is a method of data storage.
- the two methods are mutually corresponding, and some implementation processes can refer to each other, so no introduction is made to duplicate content. .
- the application layer processes only the data in the first format, and whether the data in the first format is subsequently converted into data in other formats (for example, the second format or the third format, etc.), and the application layer It is invisible, that is, the subsequent processing is transparent to the application layer, effectively reducing the workload of the application layer.
- the user needs to store the basic information of the user in the KeyValue distributed database, and the corresponding storage table, that is, the data table for storing the basic information of the user, has been created in the KeyValue distributed database, for example, the name of the storage table is UserInfo.
- the basic information of the user includes, for example, the following fields:
- the user can determine the number of Values included in a row of data records, that is, determine how the data is stored, that is, determine Mapping relationship Such as the first mapping relationship).
- determine Mapping relationship Such as the first mapping relationship.
- the user needs to invoke the metadata definition interface provided by the embodiment of the present invention, and defines the first mapping relationship as follows:
- KeyValue2 Key(ID)-> ⁇ Address,Age,Gender,Phone, Education ⁇
- the first column of a row of data records is uniformly determined as the content of the record Key, and the content of the Value is recorded from the second column.
- KeyValue1 corresponds to the first row in the storage table
- KeyValue1 corresponds to the first in the storage table.
- the first mapping relationship may indicate that each of the Values to be stored is: the column attribute of the Name is the column attribute of the second column, the row attribute is the row attribute of the first row, and the column attribute of the Address is the column of the second column.
- the row attribute is the row attribute of the second row
- the column attribute of Age is the column attribute of the third column
- the row attribute is the row attribute of the second row
- the column attribute of Gender is the column attribute of the fourth column
- the row attribute is the The row attribute of the second line
- the column attribute of Phone is the column attribute of the fifth column
- the row attribute is the row attribute of the second row
- the column attribute of Education is the column attribute of the sixth column
- the row attribute is the row attribute of the second line.
- the first mapping relationship here is only an example, and the specific mapping relationship may be set by the system or the user according to the situation. In addition, in order to save storage space, it is recommended to store as few Values as possible in one row of data records.
- the user can directly use it later without having to define it each time.
- the first mapping relationship can be obtained in the system provided by the embodiment of the present invention.
- the first mapping relationship may be obtained in the system provided by the embodiment of the present invention, and the first mapping relationship is adopted.
- the conversion between the data input by the user and the data in the KeyValue format can be completed, and the converted first data (ie, the second data) is obtained, and the layer conversion is transparent to the application.
- the second data can be stored in UserInfo.
- the user When the user reads the data from the UserInfo, the user can read the data through the read interface provided by the embodiment of the present invention.
- the user can read the second data, or it can be other data stored in UserInfo.
- the user wants to read the second data.
- two KeyValues are first read, that is, KeyValue1 and KeyValue2 as described above, and then, according to the first mapping relationship, the system re-converts KeyValue1 and KeyValue2 into data required by the user.
- the format of the converted data is as follows:
- the user does not perceive the KeyValue information in the process of reading and writing data, and does not need to know how the data is stored, which is very simple for the user and improves the operation efficiency.
- the embodiment of the present invention provides a metadata table, which can be used to store a mapping relationship.
- the system automatically searches for the mapping relationship stored in the metadata table by referring to the mapping relationship stored in the metadata layer of the application layer.
- the conversion of the data format is completed, the format of the original data is converted into a corresponding KeyValue format, and then the converted content is stored in a corresponding data table (referred to as a user data table in FIG. 3A) of the KeyValue distributed database.
- the user data table in FIG. 3B when the user reads data from the data table of the KeyValue distributed database (referred to as the user data table in FIG. 3B), the corresponding KeyValue is first read out in the module that is not visible in the application layer. The formatted data is then automatically restored to the format of the raw data visible to the user based on the mapping stored in the metadata table. Similarly, the data format conversion process is not visible to the application layer.
- one or more data tables may be included in the KeyValue distributed database.
- data tables may be included in the KeyValue distributed database.
- an embodiment of the present invention provides a data storage device, where the device includes a receiving module 401, an obtaining module 402, a converting module 403, and an operating module 404.
- the receiving module 401 is configured to receive first data that is sent by the application layer and is in a first format to be written.
- the obtaining module 402 is configured to obtain a first mapping relationship between data in the first format and data in the second format.
- the second format is a KeyValue format, where the first mapping relationship is used to indicate that each data conversion according to the first format is obtained. Row attributes and column attributes corresponding to Value;
- the converting module 403 is configured to convert the format of the first data into the second format according to the first mapping relationship to obtain the second data.
- the operation module 404 is configured to store the second data.
- the apparatus further includes a recovery module, a first update module, and a second update module;
- the recovery module is configured to: after the operation module 404 stores the second data, restore the second data to the first data in the first format according to the first mapping relationship before the update;
- the first update module is configured to: update the stored first mapping relationship by using the second mapping relationship;
- the mapping relationship is used to indicate the row attribute and the column attribute corresponding to each value obtained by the data conversion according to the first format, and the row attribute corresponding to each value in the first mapping relationship corresponds to each value in the second mapping relationship.
- the row attributes are different, and/or, the column attributes corresponding to each value in the first mapping relationship are different from the column attributes corresponding to each value in the second mapping relationship;
- the converting module 403 is further configured to: convert the format of the first data into the third format according to the updated first mapping relationship, to obtain the third data;
- the second update module is configured to: update the stored second data according to the third data.
- the device further includes a reading module and an output module;
- the reading module is configured to: after the operating module 404 stores the second data, if receiving an instruction sent by the application layer to read the first data, reading the stored first data;
- the converting module 403 is further configured to: convert the format of the second data into the first format according to the first mapping relationship, to obtain the fourth data;
- the output module is configured to: feed back fourth data to the application layer.
- an embodiment of the present invention provides a data reading apparatus, which may include a reading module 501, a conversion module 502, and an output module 503.
- the reading module 501 is configured to: if the instruction for reading the first data sent by the application layer is received, read the stored second data;
- the conversion module 502 is configured to convert the KeyValue format of the second data into the first format according to the first mapping relationship, where the first mapping relationship is used to indicate each Value corresponding to the KeyValue format obtained by converting the first data according to the first format.
- the output module 503 is configured to feed back the output format converted data to the application layer.
- the apparatus further includes a receiving module, an obtaining module, and an operating module;
- the receiving module is configured to: before the reading module 501 reads the stored second data, receive the first data in the first format that is to be written by the application layer;
- the obtaining module is configured to: obtain a first mapping relationship between the data in the first format and the data in the second format; the second format is a KeyValue format;
- the converting module 502 is further configured to: convert the format of the first data into the second format according to the first mapping relationship, to obtain the second data;
- the operation module is configured to: store the second data.
- the apparatus further includes a recovery module, a first update module, and a second update module;
- the recovery module is configured to: after the operation module stores the second data, restore the second data to the first data in the first format according to the first mapping relationship before the update;
- the first update module is configured to: update the stored first mapping relationship by using a second mapping relationship, where the second mapping relationship is used to indicate a row attribute and a column attribute corresponding to each Value obtained by data conversion according to the first format, and The row attribute corresponding to each value in the first mapping relationship is different from the row attribute corresponding to each value in the second mapping relationship, and/or the column attribute and the second mapping corresponding to each value in the first mapping relationship Each column in the relationship corresponds to a different column attribute;
- the converting module 502 is further configured to: convert the format of the first data into the third format according to the updated first mapping relationship, to obtain the third data;
- the second update module is configured to: update the stored first data according to the third data.
- an embodiment of the present invention further provides a data storage device, which may include a memory 601, a processor 602, and a receiver 603.
- the data storage device may further include a transmitter 604.
- the processor 602 may be a CPU (Central Processing Unit) or an ASIC (Application Specific Integrated Circuit), and may be one or more integrated circuits for controlling program execution, and may be an FPGA (Field Programmable Gate Array).
- the hardware circuit developed by the field programmable gate array can be a baseband chip.
- the number of memories 601 may be one or more.
- the memory 601 may include a ROM (Read Only Memory), RAM (Random Access Memory) and disk storage.
- the receiver 603 and the transmitter 604 may belong to a radio frequency system, and the receiver 603 and the transmitter 604 are configured to perform network communication with an external device, and specifically, may communicate with an external device through a network such as an Ethernet, a radio access network, or a wireless local area network.
- Receiver 603 and transmitter 604 may be physically separate components or may be physically identical components.
- These memories 601, receivers 603 and transmitters 604 may be coupled to the processor 602 via a bus or may be coupled to the processor 602 via dedicated connection lines, respectively.
- the code corresponding to the method shown above is solidified into the chip, thereby enabling the chip to perform the method shown in the previous embodiment while it is running.
- How to design and program the processor 602 is a technique well known to those skilled in the art, and details are not described herein.
- the data storage device may be located in other modules (such as a specific data platform) as described above, and thus may not directly interact with the user.
- the receiver 603 is configured to receive first data that is sent by the application layer and is in a first format to be written.
- the processor 602 is configured to obtain a first mapping relationship between the data in the first format and the data in the second format, and convert the format of the first data into the second format according to the first mapping relationship to obtain the second data.
- the second format is a KeyValue format, and the first mapping relationship is used to indicate a row attribute and a column attribute corresponding to each Value obtained by data conversion according to the first format;
- the memory 601 is configured to store the second data.
- the processor 602 is further configured to:
- the second data is restored to the first data in the first format according to the first mapping relationship before the update;
- the second mapping relationship is used to indicate the row attribute and the column attribute corresponding to each Value obtained by the data conversion according to the first format, and each value in the first mapping relationship
- the corresponding row attribute is different from the row attribute corresponding to each value in the second mapping relationship, and/or the column attribute corresponding to each value in the first mapping relationship and the column corresponding to each value in the second mapping relationship Different attributes;
- the stored second data is updated according to the third data.
- the processor 602 is further configured to:
- the stored second data After storing the second data by the memory 601, if the instruction for reading the first data sent by the application layer is received, the stored second data is read; according to the first mapping relationship, the format of the first data is Converting to the first format, obtaining the fourth data;
- the transmitter 604 is configured to: feed back fourth data to the application layer.
- an embodiment of the present invention further provides a data reading apparatus, which may include a memory 701, a processor 702, and a transmitter 703.
- a data reading apparatus may further include a receiver 704.
- the processor 702 may be a CPU or an ASIC, and may be one or more integrated circuits for controlling program execution, may be hardware circuits developed using an FPGA, and may be a baseband chip.
- the number of memories 701 may be one or more.
- the memory 701 may include a ROM, a RAM, and a disk storage.
- the transmitter 703 and the receiver 704 may belong to a radio frequency system, and the transmitter 703 and the receiver 704 are configured to perform network communication with an external device, and specifically, may communicate with an external device through a network such as an Ethernet, a radio access network, or a wireless local area network.
- Transmitter 703 and receiver 704 may be physically separate components or may be physically identical components.
- These memories 701, transmitters 703 and receivers 704 may be coupled to the processor 702 via a bus or may be coupled to the processor 702 via dedicated connection lines, respectively.
- the code corresponding to the method shown above is solidified into the chip, thereby enabling the chip to perform the method shown in the previous embodiment while it is running.
- How to design and program the processor 702 is a technique well known to those skilled in the art, and details are not described herein.
- the processor 702 is configured to: if receiving an instruction sent by the application layer to read the first data, to the storage The second data is read; according to the first mapping relationship, the KeyValue format of the second data stored in the memory 701 is converted into a first format; the first mapping relationship is used to indicate the first data conversion according to the first format.
- the transmitter 703 is configured to feed back the format converted data to the application layer.
- the receiver 704 is configured to: before the processor 702 reads the stored second data, receive the first data in the first format that is sent by the application layer to be written;
- the processor 702 is further configured to: obtain a first mapping relationship between the data in the first format and the data in the second format; the second format is a KeyValue format; and convert the format of the first data into the second according to the first mapping relationship Format, get the second data;
- the memory 701 is further configured to: store the second data.
- the processor 702 is further configured to:
- the second data is stored in the memory 701, the second data is restored to the first data in the first format according to the first mapping relationship before the update;
- the second mapping relationship is used to indicate the row attribute and the column attribute corresponding to each Value obtained by the data conversion according to the first format, and each value in the first mapping relationship
- the corresponding row attribute is different from the row attribute corresponding to each value in the second mapping relationship, and/or the column attribute corresponding to each value in the first mapping relationship and the column corresponding to each value in the second mapping relationship Different attributes;
- the stored first data is updated according to the third data.
- an embodiment of the present invention further provides a data storage device 800 , which may include a memory 802 and a processor 801 .
- the data storage device 800 may further include an input/output interface (I/O interface) 805 and a communication interface 804.
- I/O interface input/output interface
- the processor 801 may specifically be a CPU or an ASIC, and may be one or more integrated circuits for controlling program execution, may be hardware circuits developed using an FPGA, and may be baseband chips.
- the number of memories 802 can be one or more.
- Memory 802 can include ROM, RAM, and disk storage.
- Input/output interface 805 includes a number of special purpose registers and corresponding control logic circuitry that are the medium and bridge for exchanging information between processor 801 (e.g., CPU) and I/O devices.
- Communication Interface 804 may refer to an interface between a central processing unit and a standard communication subsystem, such as an RS232 interface.
- the code corresponding to the method shown above is solidified into the chip, thereby enabling the chip to perform the method shown in the previous embodiment while it is running.
- How to design and program the processor 801 is a technique well known to those skilled in the art, and details are not described herein again.
- the memory 802 is configured to store a computer execution instruction.
- the processor 801 reads the computer execution instruction stored in the memory 802, so that the data storage device 800 executes as shown in FIG. 1. The data storage method described in the process.
- an embodiment of the present invention further provides a data reading device 900, which may include a memory 902 and a processor 901.
- the data reading device 900 may further include an input/output interface 905 and a communication interface 904.
- the processor 901 may specifically be a CPU or an ASIC, and may be one or more integrated circuits for controlling program execution, may be hardware circuits developed using an FPGA, and may be baseband chips.
- the number of memories 902 can be one or more.
- the memory 902 can include a ROM, a RAM, and a disk storage.
- Input/output interface 905 includes a number of special purpose registers and corresponding control logic circuitry that are the medium and bridge for exchanging information between processor 901 (e.g., CPU) and I/O devices.
- Communication interface 904 may refer to an interface between a central processing unit and a standard communication subsystem, such as an RS232 interface.
- These memories 902, input/output interface 905, and communication interface 904 can be coupled to processor 901 via bus 903.
- the memory 902 is configured to store a computer execution instruction.
- the processor 901 reads the computer execution instruction stored in the memory 902, so that the data reading device 900 performs the process shown in FIG. Introduced data reading method.
- the devices in the embodiments of the present invention are respectively corresponding to the methods described above. Therefore, the working process of each module in the device may refer to the description of the method portion.
- the data storage device and the data reading device in the above embodiments may be the same device, or may be different devices, and the invention is not limited thereto.
- the mapping between the format before the data storage and the KeyValue format (that is, the first mapping relationship) is set.
- the device automatically stores the data according to the first mapping relationship, without requiring too many users. Participation, and whether it is to set mapping, storage mapping or format conversion, no need to complete the application layer, the user environment is more friendly, improve user experience.
- data conversion and storage can be directly performed according to the mapping relationship, without having to specify the storage mode of the data every time, which reduces the complexity of the data processing process and is beneficial to popularization and application.
- the disclosed apparatus and method may be implemented in other manners.
- the device embodiments described above are merely illustrative,
- the division of the unit or unit is only a logical function division, and the actual implementation may have another division manner, for example, multiple units or components may be combined or may be integrated into another system, or some features may be ignored. Or not.
- the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.
- the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
- each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
- the above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.
- the integrated unit if implemented in the form of a software functional unit and sold or used as a standalone product, may be stored in a computer readable storage medium.
- a computer readable storage medium A number of instructions are included to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present application.
- the foregoing storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明公开了一种数据存储、读取方法及装置,用于解决数据存储过程较为复杂的技术问题。所述数据存储方法包括:接收应用层下发的待写入的第一格式的第一数据;获取所述第一格式的数据与第二格式的数据之间的第一映射关系;所述第二格式为KeyValue格式,所述第一映射关系用于表明根据所述第一格式的数据转换得到的每个Value对应的行属性及列属性;根据所述第一映射关系,将所述第一数据的格式转换为所述第二格式,得到第二数据;存储所述第二数据。
Description
本发明涉及存储技术领域,特别涉及一种数据存储、读取方法及装置。
在分布式数据库中,数据一般是以KeyValue(键-值)的形式进行存储的。其中,Key部分,用来唯一识别一行数据,也是用来进行快速检索数据的信息,例如,通过搜索一个数据的Key,就可以搜索到该数据。Key部分,通常是由用户自定义的特定结构。Value部分,用来存储实际的数据。可以认为,Key是一个标识性信息,用于区分不同的数据,value部分存储的是真正的数据内容。
在使用KeyValue格式存储数据时,可以有不同的存储方式,例如,一行只存储一个Value,那么一个数据如果包括多个Value,则可能要分为多行进行存储,这种存储方式可以称为单行单Value方式,或者例如,一行存储多个Value,则一个数据可能总共存储为一行,这种存储方式可以称为单行多Value方式。
现有技术中,应用层每次在将数据存储为KeyValue格式时,需要指定数据的存储方式,例如需要指定每行究竟存储多少个Value,在读取时,应用层需要将读取的KeyValue格式的数据自行拼接为原本的数据。
可见,现有技术中的存储方式,存储策略需要应用层参与指定或实施,对用户来说体验不太好,对应用层来说负担也比较大。而且,在存储不同应用的数据时,应用层需要分别指定数据的存储方式,存储过程都较为复杂。
发明内容
本发明实施例提供一种数据存储、读取方法及装置,用于解决数据存储过程较为复杂的技术问题。
第一方面,提供一种数据存储方法,包括:
接收应用层下发的待写入的第一格式的第一数据;
获取所述第一格式的数据与第二格式的数据之间的第一映射关系;所述第二格式为KeyValue格式,所述第一映射关系用于表明根据所述第一格式的数据转换得到的每个Value对应的行属性及列属性;
根据所述第一映射关系,将所述第一数据的格式转换为所述第二格式,得到第二数据;
存储所述第二数据。
结合第一方面,在第一方面的第一种可能的实现方式中,在存储所述第二数据之后,还包括:
根据更新前的第一映射关系将所述第二数据恢复为所述第一格式的所述第一数据;
使用第二映射关系更新存储的所述第一映射关系;所述第二映射关系用于表明根据所述第一格式的数据转换得到的每个Value对应的行属性及列属性,且所述第一映射关系中的每个value对应的行属性与所述第二映射关系中的每个value对应的行属性不同,和/或,所述第一映射关系中的每个value对应的列属性与所述第二映射关系中的每个value对应的列属性不同;
根据更新后的第一映射关系将所述第一数据的格式转换为第三格式,得到第三数据;
根据所述第三数据更新存储的所述第二数据。
结合第一方面,在第一方面的第二种可能的实现方式中,在存储所述第二数据之后,还包括:
若接收到应用层下发的读取所述第一数据的指令,对存储的所述第二数据进行读取;
根据所述第一映射关系,将所述第二数据的格式转换为所述第一格式,得到第四数据;
向应用层反馈输出所述第四数据。
第二方面,提供一种数据读取方法,包括:
若接收到应用层下发的读取第一数据的指令,对存储的第二数据进行读取;
根据第一映射关系,将所述第二数据的KeyValue格式转换为第一格式;所述第一映射关系用于表明根据所述第一格式的第一数据转换得到的KeyValue格式的每个Value对应的行属性及列属性;
向应用层反馈输出格式转换后的数据。
结合第二方面,在第二方面的第一种可能的实现方式中,在对存储的第二数据进行读取之前,还包括:
接收应用层下发的待写入的所述第一格式的所述第一数据;
获取所述第一格式的数据与第二格式的数据之间的所述第一映射关系;所述第二格式为KeyValue格式;
根据所述第一映射关系,将所述第一数据的格式转换为所述第二格式,得到所述第二数据;
存储所述第二数据。
结合第二方面,在第二方面的第二种可能的实现方式中,在存储所述第二数据之后,还包括:
根据更新前的第一映射关系将所述第二数据恢复为所述第一格式的所述第一数据;
使用第二映射关系更新存储的所述第一映射关系;所述第二映射关系用于表明根据所述第一格式的数据转换得到的每个Value对应的行属性及列属性,且所述第一映射关系中的每个value对应的行属性与所述第二映射关系中的每个value对应的行属性不同,和/或,所述第一映射关系中的每个value对应的列属性与所述第二映射关系中的每个value对应的列属性不同;
根据更新后的第一映射关系将所述第一数据的格式转换为第三格式,得到第三数据;
根据所述第三数据更新存储的所述第二数据。
第三方面,提供一种数据存储装置,包括:
接收模块,用于接收应用层下发的待写入的第一格式的第一数据;
获取模块,用于获取所述第一格式的数据与第二格式的数据之间的第一映射关系;所述第二格式为KeyValue格式,所述第一映射关系用于表明根据所述第一格式的数据转换得到的每个Value对应的行属性及列属性;
转换模块,用于根据所述第一映射关系,将所述第一数据的格式转换为所述第二格式,得到第二数据;
操作模块,用于存储所述第二数据。
结合第三方面,在第三方面的第一种可能的实现方式中,所述装置还包括恢复模块、第一更新模块和第二更新模块;
所述恢复模块用于:在所述操作模块存储所述第二数据之后,根据更新前的第一映射关系将所述第二数据恢复为所述第一格式的所述第一数据;
所述第一更新模块用于:使用第二映射关系更新存储的所述第一映射关系;所述第二映射关系用于表明根据所述第一格式的数据转换得到的每个Value对应的行属性及列属性,且所述第一映射关系中的每个value对应的行属性与所述第二映射关系中的每个value对应的行属性不同,和/或,所述第一映射关系中的每个value对应的列属性与所述第二映射关系中的每个value对应的列属性不同;
所述转换模块还用于:根据更新后的第一映射关系将所述第一数据的格式转换为第三格式,得到第三数据;
所述第二更新模块用于:根据所述第三数据更新存储的所述第二数据。
结合第三方面,在第三方面的第二种可能的实现方式中,所述装置还包括读取模块和输出模块;
所述读取模块用于:在所述操作模块存储所述第二数据之后,若接收到应用层下发的读取所述第一数据的指令,对存储的所述第二数据进行读取;
所述转换模块还用于:根据所述第一映射关系,将所述第二数据的格式
转换为所述第一格式,得到第四数据;
所述输出模块用于:向应用层反馈输出所述第四数据。
第四方面,提供一种数据读取装置,包括:
读取模块,用于若接收到应用层下发的读取第一数据的指令,对存储的第二数据进行读取;
转换模块,用于根据第一映射关系,将所述第二数据的KeyValue格式转换为第一格式;所述第一映射关系用于表明根据所述第一格式的第一数据转换得到的KeyValue格式的每个Value对应的行属性及列属性;
输出模块,用于向应用层反馈输出格式转换后的数据。
结合第四方面,在第四方面的第一种可能的实现方式中,所述装置还包括接收模块、获取模块和操作模块;
所述接收模块用于:在所述读取模块对存储的第二数据进行读取之前,接收应用层下发的待写入的第一格式的所述第一数据;
所述获取模块用于:获取所述第一格式的数据与第二格式的数据之间的所述第一映射关系;所述第二格式为KeyValue格式;
所述转换模块还用于:根据所述第一映射关系,将所述第一数据的格式转换为所述第二格式,得到第二数据;
所述操作模块用于:存储所述第二数据。
结合第四方面,在第四方面的第二种可能的实现方式中,所述装置还包括恢复模块、第一更新模块和第二更新模块;
所述恢复模块用于:在所述操作模块存储第二数据之后,根据更新前的第一映射关系将所述第二数据恢复为所述第一格式的所述第一数据;
所述第一更新模块用于:使用第二映射关系更新存储的所述第一映射关系;所述第二映射关系用于表明根据所述第一格式的数据转换得到的每个Value对应的行属性及列属性,且所述第一映射关系中的每个value对应的行属性与所述第二映射关系中的每个value对应的行属性不同,和/或,所述第
一映射关系中的每个value对应的列属性与所述第二映射关系中的每个value对应的列属性不同;
所述转换模块还用于:根据更新后的第一映射关系将所述第一数据的格式转换为第三格式,得到第三数据;
所述第二更新模块用于:根据所述第三数据更新存储的所述第二数据。
第五方面,提供一种数据存储装置,包括:
接收器,用于接收应用层下发的待写入的第一格式的第一数据;
处理器,用于获取所述第一格式的数据与第二格式的数据之间的第一映射关系,并根据所述第一映射关系,将所述第一数据的格式转换为所述第二格式,得到第二数据;所述第二格式为KeyValue格式,所述第一映射关系用于表明根据所述第一格式的数据转换得到的每个Value对应的行属性及列属性;
存储器,用于存储所述第二数据。
结合第五方面,在第五方面的第一种可能的实现方式中,所述处理器还用于:
在通过所述存储器存储所述第二数据之后,根据更新前的第一映射关系将所述第二数据恢复为所述第一格式的所述第一数据;
使用第二映射关系更新存储的所述第一映射关系;所述第二映射关系用于表明根据所述第一格式的数据转换得到的每个Value对应的行属性及列属性,且所述第一映射关系中的每个value对应的行属性与所述第二映射关系中的每个value对应的行属性不同,和/或,所述第一映射关系中的每个value对应的列属性与所述第二映射关系中的每个value对应的列属性不同;
根据更新后的第一映射关系将所述第一数据的格式转换为第三格式,得到第三数据;
根据所述第三数据更新存储的所述第二数据。
结合第五方面,在第五方面的第二种可能的实现方式中,所述装置还包
括发送器;
所述处理器还用于:在通过所述存储器存储所述第二数据之后,若接收到应用层下发的读取所述第一数据的指令,对存储的所述第二数据进行读取;根据所述第一映射关系,将所述第二数据的格式转换为所述第一格式,得到第四数据;
所述发送器用于:向应用层反馈输出所述第四数据。
第六方面,提供一种数据读取装置,包括:
处理器,用于若接收到应用层下发的读取第一数据的指令,对存储的第二数据进行读取;根据第一映射关系,将存储器中存储的所述第二数据的键值KeyValue格式转换为第一格式;所述第一映射关系用于表明根据所述第一格式的第一数据转换得到的KeyValue格式的每个Value对应的行属性及列属性;
发送器,用于向应用层反馈输出格式转换后的数据。
结合第六方面,在第六方面的第一种可能的实现方式中,所述装置还包括接收器;
所述接收器用于:在所述处理器对存储的第二数据进行读取之前,接收应用层下发的待写入的第一格式的所述第一数据;
所述处理器还用于:获取所述第一格式的数据与第二格式的数据之间的所述第一映射关系;所述第二格式为KeyValue格式;根据所述第一映射关系,将所述第一数据的格式转换为所述第二格式,得到所述第二数据;
所述存储器还用于:存储所述第二数据。
结合第六方面,在第六方面的第二种可能的实现方式中,所述处理器还用于:
在所述存储器存储所述第二数据之后,根据更新前的第一映射关系将所述第二数据恢复为所述第一格式的所述第一数据;
使用第二映射关系更新存储的所述第一映射关系;所述第二映射关系用
于表明根据所述第一格式的数据转换得到的每个Value对应的行属性及列属性,且所述第一映射关系中的每个value对应的行属性与所述第二映射关系中的每个value对应的行属性不同,和/或,所述第一映射关系中的每个value对应的列属性与所述第二映射关系中的每个value对应的列属性不同;
根据更新后的第一映射关系将所述第一数据的格式转换为第三格式,得到第三数据;
根据所述第三数据更新存储的所述第二数据。
第七方面,提供一种数据存储装置,所述数据存储装置包括处理器和存储器,所述处理器与所述存储器通过总线连接;
所述存储器用于存储计算机执行指令,当所述数据存储装置运行时,所述处理器读取所述存储器存储的所述计算机执行指令,以使所述数据存储装置执行如前任一方面所述的数据存储方法。
第七方面,提供一种数据读取装置,所述数据读取装置包括处理器和存储器,所述处理器与所述存储器通过总线连接;
所述存储器用于存储计算机执行指令,当所述数据读取装置运行时,所述处理器读取所述存储器存储的所述计算机执行指令,以使所述数据读取装置执行如前任一方面所述的数据读取方法。
本发明实施例中设置了数据存储前的格式以及KeyValue格式之间的映射关系(即第一映射关系),在进行数据存储时,设备会自动根据第一映射关系进行存储,无需用户过多的参与,并且无论是设置映射关系、存储映射关系还是进行格式转换,都无需应用层完成,对用户来说使用环境较为友好,提高用户体验,减轻应用层的负担。在设置映射关系后,就可以直接根据映射关系进行数据转换和存储,无需每次都指定数据的存储方式,降低了数据处理过程的复杂度,有利于推广应用。
图1为本发明实施例中数据存储方法的流程图;
图2为本发明实施例中数据读取方法的流程图;
图3A为本发明实施例中数据存储过程的示意图;
图3B为本发明实施例中数据读取过程的示意图;
图4为本发明实施例中数据存储装置的结构框图;
图5为本发明实施例中数据读取装置的结构框图;
图6A-图6B为本发明实施例中数据存储装置的结构示意图;
图7A-图7B为本发明实施例中数据读取装置的结构示意图;
图8为本发明实施例中数据存储装置的另一种结构示意图;
图9为本发明实施例中数据读取装置的另一种结构示意图。
为使本发明实施例的目的、技术方案和优点更加清楚,下面将结合本发明实施例中的附图,对本发明实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例是本发明一部分实施例,而不是全部的实施例。基于本发明中的实施例,本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例,都属于本发明保护的范围。
本发明实施例的技术方案可以应用于KeyValue类型的分布式数据库。
在KeyValue分布式数据库中,数据都是以KeyValue形式存在的。其中,数据的Key部分,用来唯一识别一行数据记录,也是用来检索数据的索引信息,数据的Key部分可以是由用户自定义的特定结构。数据的Value部分用来存储真正的数据内容。
举例来说明一下KeyValue分布式数据库的特点:
例1:
KeyValue分布式数据库可以用于存储网页信息。例如,可以使用Key来存储网页的URL(Uniform Resoure Locator,统一资源定位符),在Value部分可以存储该URL指示的网页的实际内容,如下:
Key(URL)->Value(Page Content(页面内容))
例2:
KeyValue分布式数据库可以用于存储用户的一些基础信息。
假设用户的基础信息包含如下字段:
{ID(Identity,身份标识号),Name(姓名),Address(地址),Age(年龄),Gender(性别),Phone(电话),Education(教育)}
在使用KeyValue分布式数据库存储该基础信息时,可以将其中的ID作为Key部分的内容,其他均作为Value部分的内容。并且可以认为,该基础信息中包含多个Value,即,Name、Address、Age、Gender、Phone、以及Education分别可以看作一个Value。在存储这类信息时,在KeyValue分布式数据库的一行数据记录中,可以只包括一个Value,或者也可以包括多个Value。如果一行数据记录只包括一个Value,那么这类信息可能要分为多行进行存储,如果一行数据记录中包括多个Value,那么这类信息可以存储为一行记录。例如:
一行数据记录包括多个Value的情况如下:
Key(ID)->Value(Name,Address,Age,Gender,Phone,Education)
一行数据记录包括一个Value的情况如下:
Key(ID)->Value(Name)
Key(ID)->Value(Address)
Key(ID)->Value(Age)
Key(ID)->Value(Gender)
Key(ID)->Value(Phone)
Key(ID)->Value(Education)
当然,如果一行数据记录包括多个Value,那么一个基础信息也可以存储为多行,具体取决于一行数据记录究竟包括几个Value。
可以看出来,使用KeyValue分布式数据库进行数据存储时,方式比较灵活,与传统的关系型数据库的使用方式有所不同。比如,在传统的关系型
数据库中,需要预先定义存储的数据表有哪些列(属性),一旦定义好了,通常是不允许任意更改的,除非要付出较大的代价。而在KeyValue分布式数据库中,用户可以无需预先定义一个数据表中有哪些列(KeyValue),可以在使用的时候临时定义,对于一行数据记录中究竟包括多少个Value,并未做任何限定。
例3:
假设初始时一个用户的基础信息包括如下字段:
{ID,Name,Address,Age,Gender,Phone,Education}
按照一行数据记录包括一个Value的格式,将该基础信息存储到了KeyValue分布式数据库中,具体存储方式如前已有描述。
过了一段时间之后,用户又为该基础信息增加了一个属性,即增加了一个字段,为Company(公司)。此时,该基础信息变为:
{ID,Name,Address,Age,Gender,Phone,Education,Company}
此时,只需要在KeyValue分布式数据库中为该基础信息新增一行数据记录即可,增加的一行数据记录如下:
Key(ID)->Value(Company)
新增的内容对于已经存在的数据记录没有任何影响。
一般来说,KeyValue分布式数据库的每行数据记录(本发明实施例中,一行数据记录,也可以理解为一条数据记录)通常拥有特定的结构定义,在这个结构定义中,可以包含了如下信息中的一种或任意几种:
Key:可用于标识每一行数据记录的唯一性,并且可用来检索数据;
Timestamp(时间戳):用于记录一行数据记录的生成时间;
Type(类型):用于记录一行数据记录的类型;
Value:用于记录数据内容;
列名信息:与普通的关系型数据库中的列名的概念类似,可反映一行数据记录所记录的是数据的哪部分信息。
本发明实施例中,第一格式可以是任意类型的数据格式,例如第一格式可以是如前所述的用户的基础信息的格式,或者也可以是网页数据的格式,等等。
尽管在KeyValue分布式数据库中,单行多Value是其灵活性的一处体现,但在实际应用中,这会导致数据有一定的膨胀。例如,在上面的例2中,使用单行单Value的形式进行存储就会比使用单行多Value的形式进行存储占用更少的存储空间。原因就在于KeyValue本身拥有特定的结构定义,而这种结构定义将会导致一定的数据膨胀。
因此,在一行数据记录中建议放置尽量少的KeyValue,这样更为节省存储空间。
且,本发明实施例中,设置映射关系、存储映射关系、及进行格式转换等过程都无需应用层完成,而可以由应用层之下的其他模块完成,对应用层来说是不可见的,用户体验较好,应用层负担较小。例如,可以提供一个运行在应用层之下的特定的数据平台,该数据平台可以对应有中间件,中间件可以理解为能够提供用于通过数据平台完成的一些操作的函数接口,通过调用中间件的函数接口,就能在数据平台中完成相应的操作。则,本发明实施例中,设置映射关系、存储映射关系、及进行格式转换等过程,都可以通过调用中间件的方式在数据平台中完成。
另外,本文中术语“***”和“网络”在本文中常被可互换使用。本文中术语“和/或”,仅仅是一种描述关联对象的关联关系,表示可以存在三种关系,例如,A和/或B,可以表示:单独存在A,同时存在A和B,单独存在B这三种情况。另外,本文中字符“/”,如无特殊说明,一般表示前后关联对象是一种“或”的关系。
下面结合说明书附图对本发明实施例作进一步详细描述。
请参见图1,本发明实施例提供一种数据存储方法,所述方法的流程描述如下。
步骤101:接收应用层下发的待写入的第一格式的第一数据。
用户向KeyValue分布式数据库中写入第一数据,用户在写入时,第一数据的格式为第一格式。其中,用户是与应用层进行交互,即应用层接收第一数据,并将待写入的第一数据下发至其他模块,例如其他模块可以是上述包含中间件的数据平台,中间件可在步骤101对应获取该第一数据。这样,后续对第一数据进行的整个格式转换、存储等操作,对应用层来说都是不可见的。
步骤102:获取第一格式的数据与第二格式的数据之间的第一映射关系。
其中,第二格式为KeyValue格式,映射关系用于表明根据第一格式的数据转换得到的每个Value对应的行属性及列属性。本发明实施例中,将第一格式的数据和第二格式的数据之间的一种可能的映射关系称为第一映射关系。
即,***中预先存储了第一格式的数据与KeyValue格式的数据之间的第一映射关系,例如可以存储在数据库中,存储的位置对应用层来说可以是不可见的。在将第一格式的数据转换为KeyValue格式的数据后,可以包括一个Key及至少一个Value,通过第一映射关系,可以知道每个Value的行属性及列属性,也就是知道每个Value属于哪一列和哪一行,这样就可以确定将每个Value如何存储。
映射关系可以由***预先设定,例如可以使用***默认的映射关系,或者也可以由用户预先设定,本发明对此不作限制。
可选的,在本发明另一实施例中,在接收写入的第一格式的第一数据之前,还包括:
根据第一格式和第二格式,设置并存储第一映射关系;或
接收并存储第一映射关系。
即,***可以先自行设定第一映射关系,或者第一映射关系也可以由用户预先设定。例如,***可以将第一映射关系存储在数据库中,例如可以是
如前所述的特定的数据平台中的数据库,或者也可以存储在相应的文件中,无论存储在哪里,对应用层来说都是不可见的,本发明对具体存储位置不作限制。
另外,在存储映射关系后,也可以根据情况更改映射关系。一般来说,对于一种数据格式来说,其与KeyValue之间的映射关系只能存在一种,否则可能出现数据转换混乱的情况。例如先存储了第一格式的数据和KeyValue格式的数据之间的第一映射关系,如果需要存储另一种第一格式的数据和KeyValue格式的数据之间的映射关系,就会涉及到对第一映射关系进行更改,或者先删除第一映射关系,之后再存储得到的另一种映射关系。无论是直接通过对原映射关系进行修改得到新的映射关系,还是先删除原映射关系之后再存储新的映射关系,都可以看作是对原映射关系进行更改得到了新的映射关系。
可选的,在本发明另一实施例中,在接收写入的第一格式的第一数据之前,还包括:
根据第一格式和第二格式,设置并存储第三映射关系;或
接收并存储第三映射关系;
其中,第三映射关系用于表明根据第一格式的数据转换得到的每个Value对应的行属性及列属性。
可选的,在本发明另一实施例中,在存储第三映射关系之后,还包括:
根据第一映射关系更新第三映射关系;第一映射关系中的每个value对应的行属性与第三映射关系中的每个value对应的行属性不同,和/或,第一映射关系中的每个value对应的列属性与第三映射关系中的每个value对应的列属性不同;
存储更新后的第三映射关系。
更新后的第三映射关系,即为第一映射关系。用第一映射关系更新第三映射关系,可以理解为用第一映射关系替换第三映射关系。
即,在本实施例中,在未存储数据之前,映射关系就进行了一次更改,即开始存储的映射关系是第三映射关系,后来更改为第一映射关系。其中,在需要更改映射关系时,如果还未根据需要更改的映射关系存储数据,则可以直接更改,如本实施例中的介绍,而,如果已经根据需要更改的映射关系存储了数据,则还涉及到对数据进行重新存储,这部分内容将在后面进行介绍。
步骤103:根据第一映射关系,将第一数据的格式转换为第二格式,得到第二数据。
比如,若是如前所述的特定的数据平台接收第一数据,则在接收第一数据后,该特定的数据平台可以调用第一映射关系,根据第一映射关系可以将第一数据的格式转换为第二格式,例如将转换后的第一数据称为第二数据。
例如,第一数据为:
{ColA,ColB,ColC,ColD,ColE}
根据第一映射关系转换得到的第二数据为:
Key(ID)->Value(ColA)
Key(ID)->Value(ColB,ColC)
Key(ID)->Value(ColD,ColE)
即,第一映射关系定义了,第一格式中的第一个value(即ColA)的行属性为第一行的行属性,列属性为第二列(ID所在的列例如为第一列)的列属性,第二个value(即ColB)和第三个value(即ColC)的行属性为第二行的行属性,第二个value的列属性为第二列的列属性,第三个value的列属性为第三列的列属性,第四个value(即ColD)和第五个value(即ColE)的行属性为第三行的行属性,第四个value的列属性为第二列的列属性,第五个value的列属性为第三列的列属性。
步骤104:存储第二数据。
即,在得到第二数据后,可以存储第二数据。第二数据例如可以存储在
KeyValue数据库中,KeyValue数据库中例如可以包括多个数据表,例如一个映射关系对应一个数据表,则第二数据可以存储在与第一映射关系对应的数据表中。当然第二数据也可以存储在其他可能的位置,本发明不作限制。
如前介绍了在未存储数据之前可以对映射关系进行更改,下面介绍在存储数据之后更改映射关系的情况。
可选的,在本发明另一实施例中,在存储第二数据之后,还包括:
根据更新前的第一映射关系将第二数据恢复为第一格式的第一数据;
使用第二映射关系更新存储的第一映射关系;第二映射关系用于表明根据第一格式的数据转换得到的每个Value对应的行属性及列属性,且第一映射关系中的每个value对应的行属性与第二映射关系中的每个value对应的行属性不同,和/或,第一映射关系中的每个value对应的列属性与第二映射关系中的每个value对应的列属性不同;
根据更新后的第一映射关系将第一数据的格式转换为第三格式,得到第三数据;
根据第三数据更新存储的第二数据。
更新后的第一映射关系,即为第二映射关系。用第二映射关系更新第一映射关系,可以理解为用第二映射关系替换第一映射关系。
在该实施例中,在更新第一映射关系时,如果已经存储了与第一映射关系对应的数据,例如已经存储了第一数据,那么,可以在更新之前先将第一映射关系对应的数据的格式,按照第一映射关系恢复为数据原来对应的格式,例如数据原来对应的格式为第一格式,在第一映射关系更新之后,再重新将第一格式的数据按照更新后的第一映射关系进行映射,并存储映射后得到的数据,例如,本发明实施例中将第一数据按照更新后的第一映射关系重新映射后得到的数据称为第三数据。根据第三数据更新存储的第二数据,可以理解为用第三数据替换第二数据。
在该实施例中,第二映射关系所对应的两种数据格式分别为第一格式和
第三格式,第三格式也可以是KeyValue格式。其中,第三映射关系中的每个value对应的行属性与第二映射关系中的每个value对应的行属性可以相同也可以不同,和/或,第三映射关系中的每个value对应的列属性与第二映射关系中的每个value对应的列属性可以相同也可以不同。
需要说明的是,第一映射关系、第二映射关系和第三映射关系都是第一格式的数据和KeyValue格式的数据之间的映射关系,因此,将第一格式的数据无论按照第一映射关系、第二映射关系还是第三映射关系进行映射,得到的都是第二格式的数据,即KeyValue格式的数据,只是得到的KeyValue格式的数据中,每个value对应的行属性或列属性等可能会不同。
例如,第一数据为:
{ColA,ColB,ColC,ColD,ColE}
根据第三映射关系转换得到的第三数据为:
Key(ID)->Value(ColA)
Key(ID)->Value(ColB,ColC,ColD)
Key(ID)->Value(ColE)
即,第二映射关系定义了,第一格式中的第一个value(即ColA)的行属性为第一行的行属性,列属性为第二列(ID所在的列例如为第一列)的列属性,第二个value(即ColB)、第三个value(即ColC)和第四个value(即ColD)的行属性为第二行的行属性,第二个value的列属性为第二列的列属性,第三个value的列属性为第三列的列属性,第四个value的列属性为第四列的列属性,第五个value(即ColE)的行属性为第三行的行属性,列属性为第二列的列属性。结合前例,可以看到,第二映射关系相对于第一映射关系来说,ColD的行属性和列属性发生了变化,及ColE的列属性发生了变化。
可选的,在本发明另一实施例中,在存储第二数据之后,还包括:
若接收到应用层下发的读取所述第一数据的指令,对存储的第二数据进行读取;
根据第一映射关系,将第二数据的格式转换为第一格式,得到第四数据;
向应用层反馈输出第四数据。
即,因为整个处理过程对应用层来说不可见,所以应用层要读取的还是第一数据。那么,应用层在接收到用于读取第一数据的指令后,可以将指令发送给其他模块,比如发送给如前所述的特定的数据平台,则其他模块接收该读取第一数据的指令后,可以再次调用第一映射关系(该实施例中,第一映射关系未根据第二映射关系进行更新),根据第一映射关系将存储的与第一数据对应的第二数据的格式重新转换为第一格式,得到第四数据,并向应用层反馈第四数据。
在本发明另一实施例中,如果第一映射关系已经根据第二映射关系进行了更新,那么,在根据第三数据更新存储的第二数据之后,还包括:
若接收到应用层下发的读取第一数据的指令,对存储的第三数据进行读取;
根据第二映射关系,将第三数据的格式转换为第一格式,得到第五数据;
向应用层反馈输出第五数据。
同样的,因为整个处理过程对应用层来说不可见,所以应用层要读取的还是第一数据。那么,应用层在接收到用于读取第一数据的指令后,可以将指令发送给其他模块,比如发送给如前所述的特定的数据平台,则其他模块接收该读取第一数据的指令后,可以调用第二映射关系(该实施例中,第一映射关系根据第二映射关系进行了更新),根据第二映射关系将存储的与第一数据对应的第三数据的格式重新转换为第一格式,得到第五数据,并向应用层反馈第五数据。
现有技术中,应用层每次在将数据存储为KeyValue格式时,需要指定数据的存储方式,例如需要指定每行究竟存储多少个Value,在读取时,应用层需要将读取的KeyValue格式的数据自行拼接为原本的数据。可见,存储策略需要应用层参与指定或实施,对用户来说体验不太好,应用层的负担也比较重。而且,
在存储数据时,应用层都需要指定数据的存储方式,过程都较为复杂。而采用本发明实施例中的存储和读取方式后,由于预先设置有映射关系,从而无论在存储时还是在读取时都可以直接根据该映射关系进行,无需用户过多的参与,并且无论是设置映射关系、存储映射关系还是进行格式转换,都无需应用层完成,对用户来说体验较好,减轻应用层的负担。在设置映射关系后,就可以直接根据映射关系进行数据转换和存储,无需每次都指定数据的存储方式,降低了数据处理过程的复杂度,简化了数据的存储过程和读取过程。
例如,在本发明另一实施例中,映射关系和数据可以都存储在如前所述的特定的数据平台中,如果要想进行设置映射关系、更新映射关系、删除映射关系、调用映射关系、读数据、写数据、或数据转换等操作,可以通过中间件调用相应的函数接口,从而对数据平台中存储的映射关系或数据等信息进行操作。
请参见图2,基于同一发明构思及上述各实施例,本发明实施例提供一种数据读取方法,所述方法的流程描述如下。
步骤201:若接收到应用层下发的读取第一数据的指令,对存储的第二数据进行读取;
步骤202:根据第一映射关系,将第二数据的KeyValue格式转换为第一格式。其中,第一映射关系用于表明根据第一格式的数据转换得到的KeyValue格式的每个Value对应的行属性及列属性;
步骤203:向应用层反馈输出格式转换后的数据。
本发明实施例中,格式转换后的数据即可以是如前实施例所述的第四数据。
可选的,在本发明另一实施例中,在对存储的第二数据进行读取之前,还包括:
接收应用层下发的待写入的第一格式的第一数据;
获取第一格式的数据与第二格式的数据之间的第一映射关系;第二格式为KeyValue格式;
根据第一映射关系,将第一数据的格式转换为第二格式,得到第二数据;
存储第二数据。
可选的,在本发明另一实施例中,在存储第二数据之后,还包括:
根据更新前的第一映射关系将第二数据恢复为第一格式的第一数据;
使用第二映射关系更新存储的第一映射关系;第二映射关系用于表明根据第一格式的数据转换得到的每个Value对应的行属性及列属性,且第一映射关系中的每个value对应的行属性与第二映射关系中的每个value对应的行属性不同,和/或,第一映射关系中的每个value对应的列属性与第二映射关系中的每个value对应的列属性不同;
根据更新后的第一映射关系将第一数据的格式转换为第三格式,得到第三数据;
根据第三数据更新存储的第一数据。
图2流程所介绍的方法是数据读取的方法,图1流程所介绍的方法是数据存储的方法,两个方法是相互对应的,一些实施过程可相互参考,因此对于重复内容不多做介绍。
本发明实施例中,可以认为,应用层处理的只是第一格式的数据,至于后续是否将第一格式的数据转换为了其他格式(例如第二格式或第三格式等)的数据,对应用层来说都是不可见的,即后续的处理过程都对应用层透明,有效减少了应用层的工作量。
以下举例介绍如前提供的技术方案。
假设用户需要在KeyValue分布式数据库中存储用户的基础信息,并且在该KeyValue分布式数据库中已经创建了对应的存储表,即用于存储用户的基础信息的数据表,例如该存储表的名称为UserInfo。
用户的基础信息例如包含如下字段:
{ID,Name,Address,Age,Gender,Phone,Education}
如果用户是首次使用该KeyValue分布式数据库,则,用户在理解了KeyValue分布式数据库如何设计KeyValue的思路之后,可以确定一行数据记录中包括的Value的数目,即确定数据的存储方式,也就是确定映射关系(例
如为第一映射关系)。假设用户确定使用2行数据记录存储一个用户的基础信息。
此时,用户需要调用本发明实施例所提供的元数据定义接口,将第一映射关系定义如下:
KeyValue1:Key(ID)->{Name}
KeyValue2:Key(ID)->{Address,Age,Gender,Phone,Education}
例如,将一行数据记录的第一列统一确定为记录Key的内容,从第二列开始记录Value的内容,例如KeyValue1对应的是存储表中的第一行,KeyValue1对应的是存储表中的第二行,则第一映射关系可以表明,待存储的各Value中:Name的列属性为第二列的列属性,行属性为第一行的行属性,Address的列属性为第二列的列属性,行属性为第二行的行属性,Age的列属性为第三列的列属性,行属性为第二行的行属性,Gender的列属性为第四列的列属性,行属性为第二行的行属性,Phone的列属性为第五列的列属性,行属性为第二行的行属性,Education的列属性为第六列的列属性,行属性为第二行的行属性。
这里的第一映射关系只是一种举例,具体的映射关系可以由***或用户根据情况设定。另外,为了更好地节省存储空间,建议在一行数据记录中存储尽量少的Value。
当然,用户如果已经预先定义了第一映射关系,则后续可以直接使用,无需每次都进行定义。
在本发明实施例所提供的***内部,能够获取第一映射关系。用户在通过本发明实施例所提供的写入接口写入数据(例如写入第一数据)后,在本发明实施例所提供的***内部,可以获取第一映射关系,通过第一映射关系,可以完成用户输入的数据到KeyValue格式的数据之间的转换,得到转换后的第一数据(即第二数据),且这层转换对应用透明。在得到第二数据后,可以将第二数据存储在UserInfo中。
用户在从UserInfo中读取数据时,可以通过本发明实施例所提供的读取接口进行读取。用户读取的可以是第二数据,或者也可以是UserInfo中存储的其他数据。
例如,用户要读取第二数据。在本发明实施例所提供的***内部,首先读取到两个KeyValue,即如前所述的KeyValue1和KeyValue2,然后,***根据第一映射关系,将KeyValue1和KeyValue2重新转换为用户所需的数据的格式,转换后的数据的格式如下:
{ID,Name,Address,Age,Gender,Phone,Education}
这样,用户在读写数据的过程中,并不会感知到KeyValue的信息,也无需知道数据具体是如何存储的,对于用户来说操作十分简便,也提高了操作效率。
例如,本发明实施例提供一个元数据表,可以用于存储映射关系。
请参见图3A,用户通过本发明实施例提供的写入接口写入数据时,***在一个应用层不可见的模块内部,通过参考元数据表中所存储的映射关系,自动
完成数据格式的转换,将原始数据的格式转换成对应的KeyValue格式,然后,将转换后的存储到KeyValue分布式数据库的对应的数据表(图3A中称为用户数据表)中。
请继续参见图3B,用户在从KeyValue分布式数据库的数据表(图3B中称为用户数据表)中读取数据时,同样在这个应用层不可见的模块内部,首先读取出对应的KeyValue格式的数据,然后根据元数据表中所存储的映射关系,自动将KeyValue格式的数据的格式还原回用户可见的原始数据的格式。同样的,数据格式的转换过程对应用层不可见。
需要说明的是,KeyValue分布式数据库中可以包括一个或多个数据表,具体可参考现有技术中KeyValue分布式数据库的结构。
以下结合附图介绍本发明实施例中的装置。
请参见图4,基于同一发明构思及上述各实施例,本发明实施例提供一种数据存储装置,该装置包括接收模块401、获取模块402、转换模块403和操作模块404。
接收模块401,用于接收应用层下发的待写入的第一格式的第一数据;
获取模块402,用于获取第一格式的数据与第二格式的数据之间的第一映射关系;第二格式为KeyValue格式,第一映射关系用于表明根据第一格式的数据转换得到的每个Value对应的行属性及列属性;
转换模块403,用于根据第一映射关系,将第一数据的格式转换为第二格式,得到第二数据;
操作模块404,用于存储第二数据。
可选的,在本发明另一实施例中,所述装置还包括恢复模块、第一更新模块和第二更新模块;
恢复模块用于:在操作模块404存储第二数据之后,根据更新前的第一映射关系将第二数据恢复为第一格式的第一数据;
第一更新模块用于:使用第二映射关系更新存储的第一映射关系;第二
映射关系用于表明根据第一格式的数据转换得到的每个Value对应的行属性及列属性,且第一映射关系中的每个value对应的行属性与第二映射关系中的每个value对应的行属性不同,和/或,第一映射关系中的每个value对应的列属性与第二映射关系中的每个value对应的列属性不同;
转换模块403还用于:根据更新后的第一映射关系将第一数据的格式转换为第三格式,得到第三数据;
第二更新模块用于:根据第三数据更新存储的第二数据。
可选的,在本发明另一实施例中,所述装置还包括读取模块和输出模块;
读取模块用于:在操作模块404存储第二数据之后,若接收到应用层下发的读取所述第一数据的指令,对存储的第一数据进行读取;
转换模块403还用于:根据第一映射关系,将第二数据的格式转换为第一格式,得到第四数据;
输出模块用于:向应用层反馈输出第四数据。
请参见图5,基于同一发明构思及上述各实施例,本发明实施例提供一种数据读取装置,该装置可以包括读取模块501、转换模块502和输出模块503。
读取模块501,用于若接收到应用层下发的读取第一数据的指令,对存储的第二数据进行读取;
转换模块502,用于根据第一映射关系,将第二数据的KeyValue格式转换为第一格式;第一映射关系用于表明根据第一格式的第一数据转换得到的KeyValue格式的每个Value对应的行属性及列属性;
输出模块503,用于向应用层反馈输出格式转换后的数据。
可选的,在本发明另一实施例中,该装置还包括接收模块、获取模块和操作模块;
接收模块用于:在读取模块501对存储的第二数据进行读取之前,接收应用层下发的待写入的第一格式的第一数据;
获取模块用于:获取第一格式的数据与第二格式的数据之间的第一映射关系;第二格式为KeyValue格式;
转换模块502还用于:根据第一映射关系,将第一数据的格式转换为第二格式,得到第二数据;
操作模块用于:存储第二数据。
可选的,在本发明另一实施例中,该装置还包括恢复模块、第一更新模块和第二更新模块;
恢复模块用于:在操作模块存储第二数据之后,根据更新前的第一映射关系将第二数据恢复为第一格式的第一数据;
第一更新模块用于:使用第二映射关系更新存储的所述第一映射关系;第二映射关系用于表明根据第一格式的数据转换得到的每个Value对应的行属性及列属性,且第一映射关系中的每个value对应的行属性与第二映射关系中的每个value对应的行属性不同,和/或,第一映射关系中的每个value对应的列属性与第二映射关系中的每个value对应的列属性不同;
转换模块502还用于:根据更新后的第一映射关系将第一数据的格式转换为第三格式,得到第三数据;
第二更新模块用于:根据第三数据更新存储的第一数据。
请参见图6A,基于同一发明构思以及上述各实施例,本发明实施例还提供一种数据存储装置,该数据存储装置可以包括存储器601、处理器602和接收器603。可选的,请参见图6B,在本发明另一实施例中,该数据存储装置还可以包括发送器604。
处理器602具体可以是CPU(中央处理器)或ASIC(Application Specific Integrated Circuit,特定应用集成电路),可以是一个或多个用于控制程序执行的集成电路,可以是使用FPGA(Field Programmable Gate Array,现场可编程门阵列)开发的硬件电路,可以是基带芯片。存储器601的数量可以是一个或多个。存储器601可以包括ROM(Read Only Memory,只读存储器)、
RAM(Random Access Memory,随机存取存储器)和磁盘存储器。接收器603和发送器604可以属于射频***,接收器603和发送器604用于与外部设备进行网络通信,具体可以通过以太网、无线接入网、无线局域网等网络与外部设备进行通信。接收器603和发送器604可以是物理上相互独立的两个元件,也可以是物理上的同一个元件。
这些存储器601、接收器603和发送器604可以通过总线与处理器602相连接,或者也可以通过专门的连接线分别与处理器602连接。
通过对处理器602进行设计编程,将前述所示的方法所对应的代码固化到芯片内,从而使芯片在运行时能够执行前述实施例中的所示的方法。如何对处理器602进行设计编程为本领域技术人员所公知的技术,这里不再赘述。
本发明实施例中,数据存储装置可以位于如前所述的其他模块(比如特定的数据平台)中,因此可以不直接跟用户进行交互。
接收器603,用于接收应用层下发的待写入的第一格式的第一数据;
处理器602,用于获取第一格式的数据与第二格式的数据之间的第一映射关系,并根据第一映射关系,将第一数据的格式转换为第二格式,得到第二数据;第二格式为KeyValue格式,第一映射关系用于表明根据第一格式的数据转换得到的每个Value对应的行属性及列属性;
存储器601,用于存储第二数据。
可选的,在本发明另一实施例中,处理器602还用于:
在通过存储器601存储第二数据之后,根据更新前的第一映射关系将第二数据恢复为第一格式的第一数据;
使用第二映射关系更新存储的第一映射关系;第二映射关系用于表明根据第一格式的数据转换得到的每个Value对应的行属性及列属性,且第一映射关系中的每个value对应的行属性与第二映射关系中的每个value对应的行属性不同,和/或,第一映射关系中的每个value对应的列属性与第二映射关系中的每个value对应的列属性不同;
根据更新后的第一映射关系将第一数据的格式转换为第三格式,得到第三数据;
根据第三数据更新存储的第二数据。
可选的,在本发明另一实施例中,处理器602还用于:
在通过存储器601存储第二数据之后,若接收到应用层下发的读取所述第一数据的指令,对存储的第二数据进行读取;根据第一映射关系,将第一数据的格式转换为第一格式,得到第四数据;
发送器604用于:向应用层反馈输出第四数据。
请参见图7A,基于同一发明构思以及上述各实施例,本发明一个实施例还提供一种数据读取装置,该数据读取装置可以包括存储器701、处理器702和发送器703。可选的,请参见图7B,在本发明另一实施例中,该数据读取装置还可以包括接收器704。
处理器702具体可以是CPU或ASIC,可以是一个或多个用于控制程序执行的集成电路,可以是使用FPGA开发的硬件电路,可以是基带芯片。存储器701的数量可以是一个或多个。存储器701可以包括ROM、RAM和磁盘存储器。发送器703和接收器704可以属于射频***,发送器703和接收器704用于与外部设备进行网络通信,具体可以通过以太网、无线接入网、无线局域网等网络与外部设备进行通信。发送器703和接收器704可以是物理上相互独立的两个元件,也可以是物理上的同一个元件。
这些存储器701、发送器703和接收器704可以通过总线与处理器702相连接,或者也可以通过专门的连接线分别与处理器702连接。
通过对处理器702进行设计编程,将前述所示的方法所对应的代码固化到芯片内,从而使芯片在运行时能够执行前述实施例中的所示的方法。如何对处理器702进行设计编程为本领域技术人员所公知的技术,这里不再赘述。
本发明实施例中,
处理器702,用于若接收到应用层下发的读取第一数据的指令,对存储
的第二数据进行读取;根据第一映射关系,将存储器701中存储的第二数据的KeyValue格式转换为第一格式;第一映射关系用于表明根据第一格式的第一数据转换得到的KeyValue格式的每个Value对应的行属性及列属性;
发送器703,用于向应用层反馈输出格式转换后的数据。
可选的,在本发明另一实施例中,
接收器704用于:在处理器702对存储的第二数据进行读取之前,接收应用层下发的待写入的第一格式的第一数据;
处理器702还用于:获取第一格式的数据与第二格式的数据之间的第一映射关系;第二格式为KeyValue格式;根据第一映射关系,将第一数据的格式转换为第二格式,得到第二数据;
存储器701还用于:存储第二数据。
可选的,在本发明另一实施例中,处理器702还用于:
在存储器701存储第二数据之后,根据更新前的第一映射关系将第二数据恢复为第一格式的第一数据;
使用第二映射关系更新存储的第一映射关系;第二映射关系用于表明根据第一格式的数据转换得到的每个Value对应的行属性及列属性,且第一映射关系中的每个value对应的行属性与第二映射关系中的每个value对应的行属性不同,和/或,第一映射关系中的每个value对应的列属性与第二映射关系中的每个value对应的列属性不同;
根据更新后的第一映射关系将第一数据的格式转换为第三格式,得到第三数据;
根据第三数据更新存储的第一数据。
请参见图8,基于同一发明构思以及上述各实施例,本发明一个实施例还提供一种数据存储装置800,数据存储装置800可以包括存储器802和处理器801。可选的,请继续参见图8,在本发明另一实施例中,数据存储装置800还可以包括输入/输出接口(I/O接口)805和通信接口804。
处理器801具体可以是CPU或ASIC,可以是一个或多个用于控制程序执行的集成电路,可以是使用FPGA开发的硬件电路,可以是基带芯片。存储器802的数量可以是一个或多个。存储器802可以包括ROM、RAM和磁盘存储器。输入/输出接口805内包括若干专用寄存器和相应的控制逻辑电路,它是处理器801(例如CPU)和I/O设备之间交换信息的媒介和桥梁。通信接口(Communication Interface)804可以是指中央处理器和标准的通信子***之间的接口,例如RS232接口。
这些存储器802、输入/输出接口805和通信接口804可以通过总线803与处理器801相连接。
通过对处理器801进行设计编程,将前述所示的方法所对应的代码固化到芯片内,从而使芯片在运行时能够执行前述实施例中的所示的方法。如何对处理器801进行设计编程为本领域技术人员所公知的技术,这里不再赘述。
本发明实施例中,存储器802用于存储计算机执行指令,当数据存储装置800运行时,处理器801读取所述存储器802存储的所述计算机执行指令,以使数据存储装置800执行如图1流程所介绍的数据存储方法。
请参见图9,基于同一发明构思以及上述各实施例,本发明一个实施例还提供一种数据读取装置900,数据读取装置900可以包括存储器902和处理器901。可选的,请继续参见图9,在本发明另一实施例中,数据读取装置900还可以包括输入/输出接口905和通信接口904。
处理器901具体可以是CPU或ASIC,可以是一个或多个用于控制程序执行的集成电路,可以是使用FPGA开发的硬件电路,可以是基带芯片。存储器902的数量可以是一个或多个。存储器902可以包括ROM、RAM和磁盘存储器。输入/输出接口905内包括若干专用寄存器和相应的控制逻辑电路,它是处理器901(例如CPU)和I/O设备之间交换信息的媒介和桥梁。通信接口904可以是指中央处理器和标准的通信子***之间的接口,例如RS232接口。
这些存储器902、输入/输出接口905和通信接口904可以通过总线903与处理器901相连接。
通过对处理器901进行设计编程,将前述所示的方法所对应的代码固化到芯片内,从而使芯片在运行时能够执行前述实施例中的所示的方法。如何对处理器901进行设计编程为本领域技术人员所公知的技术,这里不再赘述。
本发明实施例中,存储器902用于存储计算机执行指令,当数据读取装置900运行时,处理器901读取存储器902存储的计算机执行指令,以使数据读取装置900执行如图2流程所介绍的数据读取方法。
本发明实施例中的装置均是与如前所述的方法分别对应的装置,因此,装置中各个模块的工作过程可参考方法部分的描述。
上述各实施例中的数据存储装置和数据读取装置,可以是同一设备,或者也可以是不同的设备,本发明不作限制。
本发明实施例中设置了数据存储前的格式以及KeyValue格式之间的映射关系(即第一映射关系),在进行数据存储时,设备会自动根据第一映射关系进行存储,无需用户过多的参与,并且无论是设置映射关系、存储映射关系还是进行格式转换,都无需应用层完成,对用户来说使用环境较为友好,提高用户体验。在设置映射关系后,就可以直接根据映射关系进行数据转换和存储,无需每次都指定数据的存储方式,降低了数据处理过程的复杂度,有利于推广应用。
所属领域的技术人员可以清楚地了解到,为描述的方便和简洁,仅以上述各功能单元的划分进行举例说明,实际应用中,可以根据需要而将上述功能分配由不同的功能单元完成,即将装置的内部结构划分成不同的功能单元,以完成以上描述的全部或者部分功能。上述描述的***,装置和单元的具体工作过程,可以参考前述方法实施例中的对应过程,在此不再赘述。
在本申请所提供的几个实施例中,应该理解到,所揭露的装置和方法,可以通过其它的方式实现。例如,以上所描述的装置实施例仅仅是示意性的,
例如,所述单元或单元的划分,仅仅为一种逻辑功能划分,实际实现时可以有另外的划分方式,例如多个单元或组件可以结合或者可以集成到另一个***,或一些特征可以忽略,或不执行。另一点,所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口,装置或单元的间接耦合或通信连接,可以是电性,机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的,作为单元显示的部件可以是或者也可以不是物理单元,即可以位于一个地方,或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外,在本申请各个实施例中的各功能单元可以集成在一个处理单元中,也可以是各个单元单独物理存在,也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现,也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时,可以存储在一个计算机可读取存储介质中。基于这样的理解,本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来,该计算机软件产品存储在一个存储介质中,包括若干指令用以使得一台计算机设备(可以是个人计算机,服务器,或者网络设备等)或processor(处理器)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括:U盘、移动硬盘、ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述,以上实施例仅用以对本申请的技术方案进行了详细介绍,但以上实施例的说明只是用于帮助理解本发明的方法及其核心思想,不应理解为对本发明的限制。本技术领域的技术人员在本发明揭露的技术范围内,可轻易想到的变化或替换,都应涵盖在本发明的保护范围之内。
Claims (14)
- 一种数据存储方法,其特征在于,包括:接收应用层下发的待写入的第一格式的第一数据;获取所述第一格式的数据与第二格式的数据之间的第一映射关系;所述第二格式为键值KeyValue格式,所述第一映射关系用于表明根据所述第一格式的数据转换得到的每个Value对应的行属性及列属性;根据所述第一映射关系,将所述第一数据的格式转换为所述第二格式,得到第二数据;存储所述第二数据。
- 如权利要求1所述的方法,其特征在于,在存储所述第二数据之后,还包括:根据更新前的第一映射关系将所述第二数据恢复为所述第一格式的所述第一数据;使用第二映射关系更新存储的所述第一映射关系;所述第二映射关系用于表明根据所述第一格式的数据转换得到的每个Value对应的行属性及列属性,且所述第一映射关系中的每个value对应的行属性与所述第二映射关系中的每个value对应的行属性不同,和/或,所述第一映射关系中的每个value对应的列属性与所述第二映射关系中的每个value对应的列属性不同;根据更新后的第一映射关系将所述第一数据的格式转换为第三格式,得到第三数据;根据所述第三数据更新存储的所述第二数据。
- 如权利要求1所述的方法,其特征在于,在存储所述第二数据之后,还包括:若接收到应用层下发的读取所述第一数据的指令,对存储的所述第二数据进行读取;根据所述第一映射关系,将所述第二数据的格式转换为所述第一格式, 得到第四数据;向应用层反馈输出所述第四数据。
- 一种数据读取方法,其特征在于,包括:若接收到应用层下发的读取第一数据的指令,对存储的第二数据进行读取;根据第一映射关系,将所述第二数据的键值KeyValue格式转换为第一格式;所述第一映射关系用于表明根据所述第一格式的第一数据转换得到的KeyValue格式的每个Value对应的行属性及列属性;向应用层反馈输出格式转换后的数据。
- 如权利要求4所述的方法,其特征在于,在对存储的第二数据进行读取之前,还包括:接收应用层下发的待写入的所述第一格式的所述第一数据;获取所述第一格式的数据与第二格式的数据之间的所述第一映射关系;所述第二格式为KeyValue格式;根据所述第一映射关系,将所述第一数据的格式转换为所述第二格式,得到所述第二数据;存储所述第二数据。
- 如权利要求4所述的方法,其特征在于,在存储所述第二数据之后,还包括:根据更新前的第一映射关系将所述第二数据恢复为所述第一格式的所述第一数据;使用第二映射关系更新存储的所述第一映射关系;所述第二映射关系用于表明根据所述第一格式的数据转换得到的每个Value对应的行属性及列属性,且所述第一映射关系中的每个value对应的行属性与所述第二映射关系中的每个value对应的行属性不同,和/或,所述第一映射关系中的每个value对应的列属性与所述第二映射关系中的每个value对应的列属性不同;根据更新后的第一映射关系将所述第一数据的格式转换为第三格式,得到第三数据;根据所述第三数据更新存储的所述第二数据。
- 一种数据存储装置,其特征在于,包括:接收模块,用于接收应用层下发的待写入的第一格式的第一数据;获取模块,用于获取所述第一格式的数据与第二格式的数据之间的第一映射关系;所述第二格式为键值KeyValue格式,所述第一映射关系用于表明根据所述第一格式的数据转换得到的每个Value对应的行属性及列属性;转换模块,用于根据所述第一映射关系,将所述第一数据的格式转换为所述第二格式,得到第二数据;操作模块,用于存储所述第二数据。
- 如权利要求7所述的装置,其特征在于,所述装置还包括恢复模块、第一更新模块和第二更新模块;所述恢复模块用于:在所述操作模块存储所述第二数据之后,根据更新前的第一映射关系将所述第二数据恢复为所述第一格式的所述第一数据;所述第一更新模块用于:使用第二映射关系更新存储的所述第一映射关系;所述第二映射关系用于表明根据所述第一格式的数据转换得到的每个Value对应的行属性及列属性,且所述第一映射关系中的每个value对应的行属性与所述第二映射关系中的每个value对应的行属性不同,和/或,所述第一映射关系中的每个value对应的列属性与所述第二映射关系中的每个value对应的列属性不同;所述转换模块还用于:根据更新后的第一映射关系将所述第一数据的格式转换为第三格式,得到第三数据;所述第二更新模块用于:根据所述第三数据更新存储的所述第二数据。
- 如权利要求7所述的装置,其特征在于,所述装置还包括读取模块和输出模块;所述读取模块用于:在所述操作模块存储所述第二数据之后,若接收到应用层下发的读取所述第一数据的指令,对存储的所述第二数据进行读取;所述转换模块还用于:根据所述第一映射关系,将所述第二数据的格式转换为所述第一格式,得到第四数据;所述输出模块用于:向应用层反馈输出所述第四数据。
- 一种数据读取装置,其特征在于,包括:读取模块,用于若接收到应用层下发的读取第一数据的指令,对存储的第二数据进行读取;转换模块,用于根据第一映射关系,将所述第二数据的键值KeyValue格式转换为第一格式;所述第一映射关系用于表明根据所述第一格式的第一数据转换得到的KeyValue格式的每个Value对应的行属性及列属性;输出模块,用于向应用层反馈输出格式转换后的数据。
- 如权利要求10所述的装置,其特征在于,所述装置还包括接收模块、获取模块和操作模块;所述接收模块用于:在所述读取模块对存储的第二数据进行读取之前,接收应用层下发的待写入的第一格式的所述第一数据;所述获取模块用于:获取所述第一格式的数据与第二格式的数据之间的所述第一映射关系;所述第二格式为KeyValue格式;所述转换模块还用于:根据所述第一映射关系,将所述第一数据的格式转换为所述第二格式,得到第二数据;所述操作模块用于:存储所述第二数据。
- 如权利要求10所述的装置,其特征在于,所述装置还包括恢复模块、第一更新模块和第二更新模块;所述恢复模块用于:在所述操作模块存储第二数据之后,根据更新前的第一映射关系将所述第二数据恢复为所述第一格式的所述第一数据;所述第一更新模块用于:使用第二映射关系更新存储的所述第一映射关 系;所述第二映射关系用于表明根据所述第一格式的数据转换得到的每个Value对应的行属性及列属性,且所述第一映射关系中的每个value对应的行属性与所述第二映射关系中的每个value对应的行属性不同,和/或,所述第一映射关系中的每个value对应的列属性与所述第二映射关系中的每个value对应的列属性不同;所述转换模块还用于:根据更新后的第一映射关系将所述第一数据的格式转换为第三格式,得到第三数据;所述第二更新模块用于:根据所述第三数据更新存储的所述第二数据。
- 一种数据存储装置,其特征在于,所述数据存储装置包括处理器和存储器,所述处理器与所述存储器通过总线连接;所述存储器用于存储计算机执行指令,当所述数据存储装置运行时,所述处理器读取所述存储器存储的所述计算机执行指令,以使所述数据存储装置执行权利要求1至3任一项所述的数据存储方法。
- 一种数据读取装置,其特征在于,所述数据读取装置包括处理器和存储器,所述处理器与所述存储器通过总线连接;所述存储器用于存储计算机执行指令,当所述数据读取装置运行时,所述处理器读取所述存储器存储的所述计算机执行指令,以使所述数据读取装置执行权利要求4至6任一项所述的数据读取方法。
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP16836379.4A EP3252592A4 (en) | 2015-08-18 | 2016-02-15 | Method and device for storing and reading data |
US15/845,665 US20180107725A1 (en) | 2015-08-18 | 2017-12-18 | Data Storage Method and Apparatus, and Data Read Method and Apparatus |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201510508092.9A CN105094707B (zh) | 2015-08-18 | 2015-08-18 | 一种数据存储、读取方法及装置 |
CN201510508092.9 | 2015-08-18 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/845,665 Continuation US20180107725A1 (en) | 2015-08-18 | 2017-12-18 | Data Storage Method and Apparatus, and Data Read Method and Apparatus |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2017028514A1 true WO2017028514A1 (zh) | 2017-02-23 |
Family
ID=54575267
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2016/073811 WO2017028514A1 (zh) | 2015-08-18 | 2016-02-15 | 一种数据存储、读取方法及装置 |
Country Status (4)
Country | Link |
---|---|
US (1) | US20180107725A1 (zh) |
EP (1) | EP3252592A4 (zh) |
CN (1) | CN105094707B (zh) |
WO (1) | WO2017028514A1 (zh) |
Families Citing this family (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105094707B (zh) * | 2015-08-18 | 2018-03-13 | 华为技术有限公司 | 一种数据存储、读取方法及装置 |
EP3279810B1 (en) | 2016-06-14 | 2019-09-04 | Huawei Technologies Co., Ltd. | Data access method and related apparatus and system |
CN106202438A (zh) * | 2016-07-13 | 2016-12-07 | 乐视控股(北京)有限公司 | 存储关联数据的方法和*** |
EP3336717A1 (en) * | 2016-12-14 | 2018-06-20 | Huawei Technologies Co., Ltd. | System, device and method for data update notification |
CN108268586B (zh) * | 2017-09-22 | 2020-06-16 | 阿里巴巴(中国)有限公司 | 跨多数据表的数据处理方法、装置、介质和计算设备 |
CN107818161A (zh) * | 2017-10-30 | 2018-03-20 | 山东浪潮通软信息科技有限公司 | 一种数据获取方法及装置 |
US10990571B1 (en) * | 2018-01-19 | 2021-04-27 | Amazon Technologies, Inc. | Online reordering of database table columns |
CN108681456A (zh) * | 2018-04-02 | 2018-10-19 | 长沙八思量信息技术有限公司 | 版本兼容的数据结构存取方法、计算机装置及存储介质 |
US10579606B2 (en) | 2018-05-03 | 2020-03-03 | Samsung Electronics Co., Ltd | Apparatus and method of data analytics in key-value solid state device (KVSSD) including data and analytics containers |
CN108664609B (zh) * | 2018-05-10 | 2021-04-30 | 中国银行股份有限公司 | 一种数据共享的方法、网络设备及终端 |
US11106629B2 (en) * | 2018-05-18 | 2021-08-31 | Futurewei Technologies, Inc. | Method and system for a distributing computing platform |
CN108828488B (zh) * | 2018-06-12 | 2020-10-27 | 福建星云电子股份有限公司 | 一种兼容两种因子平台的实现方法 |
CN110569329B (zh) * | 2019-10-28 | 2022-08-02 | 深圳市商汤科技有限公司 | 数据处理方法及装置、电子设备和存储介质 |
CN111405035A (zh) * | 2020-03-13 | 2020-07-10 | 北京旷视科技有限公司 | 数据的传输方法及数据通道*** |
WO2021218280A1 (en) * | 2020-04-28 | 2021-11-04 | Huawei Technologies Co., Ltd. | Data storage and data retrieval methods and devices |
US11747996B2 (en) | 2020-08-05 | 2023-09-05 | Dropbox, Inc. | System and methods for implementing a key-value data store |
JP2022059247A (ja) * | 2020-10-01 | 2022-04-13 | 富士フイルムビジネスイノベーション株式会社 | 情報処理装置及びプログラム |
CN113961558B (zh) * | 2021-10-08 | 2022-06-14 | 上海信宝博通电子商务有限公司 | 前端数据存储方法、装置及存储介质 |
CN114327256A (zh) * | 2021-11-22 | 2022-04-12 | 南京风兴科技有限公司 | 一种用于神经网络处理器的数据格式在线转换架构及方法 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102332030A (zh) * | 2011-10-17 | 2012-01-25 | 中国科学院计算技术研究所 | 用于分布式键-值存储***的数据存储、管理和查询方法及*** |
CN103077183A (zh) * | 2012-12-14 | 2013-05-01 | 北京普泽天玑数据技术有限公司 | 一种分布式顺序表的数据导入方法及其*** |
CN103678716A (zh) * | 2013-12-31 | 2014-03-26 | 中国科学院深圳先进技术研究院 | 一种基于格式数据集的分布式数据存储与计算方法 |
CN105094707A (zh) * | 2015-08-18 | 2015-11-25 | 华为技术有限公司 | 一种数据存储、读取方法及装置 |
Family Cites Families (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6999956B2 (en) * | 2000-11-16 | 2006-02-14 | Ward Mullins | Dynamic object-driven database manipulation and mapping system |
TWI240861B (en) * | 2002-01-11 | 2005-10-01 | Integrated Circuit Solution In | Data access method and architecture of flash memory |
US7149730B2 (en) * | 2002-05-03 | 2006-12-12 | Ward Mullins | Dynamic class inheritance and distributed caching with object relational mapping and cartesian model support in a database manipulation and mapping system |
JP4456909B2 (ja) * | 2004-03-29 | 2010-04-28 | 株式会社日立製作所 | バックアップ方法、ストレージシステム及びそのプログラム |
US8949294B2 (en) * | 2011-03-23 | 2015-02-03 | Red Hat, Inc. | Data grid supporting multiple protocols |
EP2765510A4 (en) * | 2011-10-06 | 2016-07-06 | Fujitsu Ltd | DATA PROCESSING, DISTRIBUTED PROCESSING SYSTEM AND PROGRAM |
CN102982130A (zh) * | 2012-11-16 | 2013-03-20 | 深圳市融创天下科技股份有限公司 | 一种nosql与rdbms的数据库同步方法和*** |
US9104762B1 (en) * | 2013-01-14 | 2015-08-11 | Amazon Technologies, Inc. | Universal database management |
JP6361199B2 (ja) * | 2014-03-20 | 2018-07-25 | 日本電気株式会社 | 情報記憶システム |
CN104090730B (zh) * | 2014-07-08 | 2017-02-22 | 飞天诚信科技股份有限公司 | 一种对存储设备进行数据读写的方法及装置 |
CN104182184B (zh) * | 2014-08-27 | 2017-08-25 | 浪潮电子信息产业股份有限公司 | 一种分布式块存储克隆方法 |
-
2015
- 2015-08-18 CN CN201510508092.9A patent/CN105094707B/zh active Active
-
2016
- 2016-02-15 WO PCT/CN2016/073811 patent/WO2017028514A1/zh active Application Filing
- 2016-02-15 EP EP16836379.4A patent/EP3252592A4/en not_active Ceased
-
2017
- 2017-12-18 US US15/845,665 patent/US20180107725A1/en not_active Abandoned
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102332030A (zh) * | 2011-10-17 | 2012-01-25 | 中国科学院计算技术研究所 | 用于分布式键-值存储***的数据存储、管理和查询方法及*** |
CN103077183A (zh) * | 2012-12-14 | 2013-05-01 | 北京普泽天玑数据技术有限公司 | 一种分布式顺序表的数据导入方法及其*** |
CN103678716A (zh) * | 2013-12-31 | 2014-03-26 | 中国科学院深圳先进技术研究院 | 一种基于格式数据集的分布式数据存储与计算方法 |
CN105094707A (zh) * | 2015-08-18 | 2015-11-25 | 华为技术有限公司 | 一种数据存储、读取方法及装置 |
Non-Patent Citations (1)
Title |
---|
See also references of EP3252592A4 * |
Also Published As
Publication number | Publication date |
---|---|
US20180107725A1 (en) | 2018-04-19 |
EP3252592A1 (en) | 2017-12-06 |
CN105094707A (zh) | 2015-11-25 |
CN105094707B (zh) | 2018-03-13 |
EP3252592A4 (en) | 2018-05-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2017028514A1 (zh) | 一种数据存储、读取方法及装置 | |
EP3113043B1 (en) | Method, device and host for updating metadata stored in columns in distributed file system | |
US10467192B2 (en) | Method and apparatus for updating data table in keyvalue database | |
CN102272751B (zh) | 在数据库环境通过背景同步的数据完整性 | |
WO2021077912A1 (zh) | 一种数据同步的方法及设备 | |
WO2023231665A1 (zh) | 分布式事务处理方法、***、设备及可读存储介质 | |
CN110851474A (zh) | 数据查询方法、数据库中间件、数据查询设备及存储介质 | |
CN114722119A (zh) | 数据同步方法及*** | |
CN113190534A (zh) | 数据库数据迁移方法及装置 | |
CN115470156A (zh) | 基于rdma的内存使用方法、***、电子设备和存储介质 | |
CN112328592A (zh) | 数据存储方法、电子设备及计算机可读存储介质 | |
CN113051102A (zh) | 文件备份方法、装置、***、存储介质和计算机设备 | |
CN103246729A (zh) | 一种处理安卓移动端的多媒体文件的方法和*** | |
CN107977344A (zh) | 数据存储方法、获取方法及服务器 | |
CN113297267A (zh) | 数据缓存和任务处理方法、装置、设备以及存储介质 | |
CN104572638A (zh) | 数据读写方法及装置 | |
WO2023098593A1 (zh) | 数据处理方法及装置 | |
CN110837499B (zh) | 数据访问处理方法、装置、电子设备和存储介质 | |
WO2022223038A1 (zh) | 一种键名称生成方法、装置和计算机可读存储介质 | |
CN114928620B (zh) | 用户信息同步方法、装置、设备、存储介质和程序产品 | |
CN115098588A (zh) | 跨集群的元数据同步方法、设备及介质 | |
WO2021237704A1 (zh) | 数据同步方法及相关装置 | |
CN114116717A (zh) | 一种数据库信息查询方法、装置、设备及可读存储介质 | |
US10114864B1 (en) | List element query support and processing | |
CN112052234A (zh) | 业务数据的处理方法和装置、存储介质、电子装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 16836379 Country of ref document: EP Kind code of ref document: A1 |
|
REEP | Request for entry into the european phase |
Ref document number: 2016836379 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |