KR101785166B1 - Selective DB Configuration Method in accordance with Data Type and System applying the same - Google Patents
Selective DB Configuration Method in accordance with Data Type and System applying the same Download PDFInfo
- Publication number
- KR101785166B1 KR101785166B1 KR1020150167311A KR20150167311A KR101785166B1 KR 101785166 B1 KR101785166 B1 KR 101785166B1 KR 1020150167311 A KR1020150167311 A KR 1020150167311A KR 20150167311 A KR20150167311 A KR 20150167311A KR 101785166 B1 KR101785166 B1 KR 101785166B1
- Authority
- KR
- South Korea
- Prior art keywords
- data
- dbms
- stored
- determined
- determining
- Prior art date
Links
Images
Classifications
-
- G06F17/30557—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
-
- G06F17/30289—
-
- G06F17/30309—
-
- G06F17/30312—
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
An optional DB configuration method and system according to data type is provided. A DB configuration method according to an embodiment of the present invention determines one of a plurality of DBMSs based on characteristics of data and stores data. Accordingly, an optimal DB can be selectively configured according to the data type, and the data can be more efficiently managed, which leads to an increase in the speed of providing data retrieval results.
Description
The present invention relates to a data storage technology, and more particularly, to a method and system for storing and managing data in an optimal DB (Data Base).
Efficient data management includes storing and safely storing large amounts of data, as well as providing data quickly for retrieval requests of data.
At present, in Internet of Things (IoT), various and many data are generated by various devices, so data management is very difficult.
Effective data management approaches should be sought, taking into account a wide variety of data.
Also, the most generated data from devices constituting IoT is measurement data. Since measurement time is important for this measurement data, it is important to match and manage the measurement data.
However, in the case of a device such as a simple sensor that can not generate a measurement time, it is difficult to manage the measurement data in a time-series manner, and it is also necessary to search for an effective management method for the measurement data generated in such devices Do.
It is an object of the present invention to provide a method and system for selectively configuring an optimal DB according to a data type.
It is another object of the present invention to provide a method and system for accurately managing time intervals of received time series data.
According to an aspect of the present invention, there is provided a method of configuring a DB, comprising: determining one of a plurality of DBMSs as a DBMS for storing the data based on characteristics of data; And storing the data in a determined DBMS.
The determining step may determine a DBMS to store the data based on the profile of the data.
The determining step determines that the general data or metadata is stored in the first DBMS, the time-series data is determined to be stored in the second DBMS, the image data is determined to be stored in the third DBMS, The moving image data is determined to be stored in the fourth DBMS, and the fast sampling data is determined to be stored in the fifth DBMS.
The storing step may store the time series data together with the time stamp calculated from the sequence number or the timer value received together with the time series data.
In addition, if the capacity of the stored data exceeds a predetermined capacity, the holding step can be performed.
The determining step may determine a DBMS to store the data based on at least one of a storage period of the data, a type of the data, and a size of the data.
In addition, the determining step may determine the type of the data based on a query processing history of the data.
According to another embodiment of the present invention, a data storage system includes a plurality of DBMSs (Data Base Management Systems); And a processor for determining one of the plurality of DBMSs as a DBMS for storing the data based on the characteristics of the data and storing the data in the determined DBMS.
INDUSTRIAL APPLICABILITY As described above, according to the embodiments of the present invention, an optimum DB can be selectively configured according to a data type, and data can be managed more efficiently, which leads to an increase in the speed of providing data search results .
Further, according to the embodiments of the present invention, time intervals for received time series data can be more accurately managed and used.
BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram illustrating a conceptual description of a method of configuring an optional DB according to a data type according to an embodiment of the present invention;
2 is a diagram schematically illustrating a concept of a method for determining an optimal DBMS from a storage type of data,
3 is a diagram schematically illustrating a concept of a method for determining an optimal DBMS from a query processing history of data,
Figures 4-7 illustrate the message sequence charts provided in the description of the process by which the platform collects time series data from devices that do not have RTCs,
8 is a block diagram of a data storage system according to another embodiment of the present invention,
FIG. 9 is a flowchart provided in an explanation of a method of configuring a selective DB according to a data type according to another embodiment of the present invention.
Hereinafter, the present invention will be described in detail with reference to the drawings.
1 is a diagram provided in a conceptual description of a method of configuring a selective DB according to a data type according to an embodiment of the present invention.
The selective DB configuration method according to the embodiment of the present invention is applicable to an IoT / CoT platform / server that stores data collected from IoT (Internet of Things) / CoT (Cloud of Things) devices.
However, this is merely an application example, and it goes without saying that the present invention may be applied to other kinds of platforms / servers that collect data from other kinds of devices.
As shown in FIG. 1, data to be stored / queried through an API (I / F) is data virtualized for management and retrieval purposes. Also, the data are stored in different DBMSs (DBMS-1, DBMS-2, DBMS-3, ..., DBMS-n).
The DBMS in which the data is to be stored is determined by the data profile. That is, when the data profile is 1) a profile of a first type (Profile-1), it is stored in a DBMS of a first type (DBMS-1) 3) stored in the DBMS (DBMS-3) of the third type in the case of the third type profile (Profile-) 3, and stored in the DBMS (DBMS-2) of the second type, and is stored in the n-th type DBMS (DBMS-n) when the profile is n-type profile (Profile-n).
The data profile can be classified into general data or metadata, time series data, image data, moving image data, and high-speed sampling data. The time-series data is data consisting of a pair of time-measurement values.
DBMSs can also be implemented in various types. For example, it is configured to include an RDB (Relational DB) for storing general data or metadata, a TSDB (Time Series DB) for storing time-series data, and other DBs.
On the other hand, when the DBMS can not be determined by the data profile, the DBMS-1 can temporarily store the data, and determine the optimal DBMS by determining the storage type of the data later.
Alternatively, if the DBMS can not be determined by the data profile, the data can be temporarily stored in the DBMS-1, and the optimal DBMS can be determined by analyzing the query processing history of the data later.
FIG. 2 schematically shows a concept of a method for determining an optimal DBMS from a storage type of data. In FIG. 2, the data requested to be stored is temporarily stored in the DBMS-1.
Then, if the capacity of the stored data exceeds the predetermined capacity, the storage type of the data is determined based on the storage period, type, and size of the data. Then, by inferring the profile of the data from the storage type of the data, the optimal DBMS is determined for the data.
In FIG. 2, a curve arrow indicates a process of determining an optimal DBMS for data temporarily stored in DBMS-1 as DBMS-n and moving data temporarily stored in DBMS-1 to DBMS-n.
If the optimal DBMS for the data temporarily stored in the DBMS-1 is DBMS-1, no data movement is performed.
FIG. 3 schematically shows a concept of a method for determining an optimal DBMS from a query processing history of data. In FIG. 3, the data requested to be stored temporarily is temporarily stored in the DBMS-1.
Then, based on the query processing history of the stored data, the optimum DBMS is determined by deducing the profile of the data with reference to the time required for the query processing of the data.
Data must be stored in an optimal DBMS to reflect the fastest query processing speed (query result return rate). If the data is stored in an inappropriate DBMS, the query processing speed (query result returning speed) is slow. The optimal DBMS is derived by analyzing the processing time for the requested amount of queries to the current DBMS.
In FIG. 3, a curve arrow indicates a process of determining an optimal DBMS for data temporarily stored in the DBMS-1 as DBMS-n and moving data temporarily stored in the DBMS-1 to DBMS-n.
If the optimal DBMS for the data temporarily stored in the DBMS-1 is DBMS-1, no data movement is performed.
It will be appreciated that the method of determining the optimal DBMS shown in FIG. 2 and FIG. 3 may be further applied for updating even when the optimal DBMS is determined by referring to the data profile at the time of initial storage.
For example, when an optimal DBMS is determined as DBMS-2 by referring to the profile of data at the time of initial storage, DBMS-2 is determined as an optimal DBMS and stored. Update the DBMS. In this case, the data stored / stored in DBMS-2 is moved to DBMS-3.
Hereinafter, the collection / preprocessing process before storing the above-described time series data in a DBMS (for example, TSDB) will be described in detail.
FIGS. 4 to 7 are diagrams provided for explaining a process of collecting time series data from an IoT / CoT device in which the IoT / CoT platform / server / gateway does not have a RTC (Real Time Clock).
In the embodiment of the present invention, the measurement interval of the measurement data having no accurate measurement time is restored to the maximum accuracy and the measurement time is recovered as accurately as possible.
4 is a message sequence chart illustrating a method of collecting measurement data from a device having a constant sampling rate (period) and storing the collected measurement data in a DBMS.
In FIG. 4, the device first registers time information in the platform and requests time initialization. The registered time information is the sampling rate (period) of the device. Then, by the device's time initialization request, the platform generates a time stamp at the initial time or at the current time of the device.
The device then periodically transmits the measurement data to the platform along with the sequence number. Then, the platform sequentially stores the measurement data while increasing the initially generated time stamp by a sampling period (delta T).
5 is a message sequence chart illustrating another method of collecting measurement data from a device having a constant sampling rate and storing the same in a DBMS.
In Fig. 5, there is a difference from the method shown in Fig. 4 in that a plurality of measurement data is transmitted as a set from the device to the platform. The transmitted data set may be configured such that 1) measurement data is stored in order and 2) measurement data to which sequence numbers are added are stored.
The platform maps the measurement data included in the received data set to the corresponding DBMS by mapping it with increasing the time stamp based on the sampling period.
6 is a message sequence chart illustrating a method of collecting measurement data generated by aperiodic sampling in a device and storing the collected measurement data in a DBMS.
As in Fig. 4, in Fig. 6, the device first registers time information in the platform and requests time initialization. The registered time information is information indicating that the sampling mode of the device is an aperiodic mode and timer basic information.
By the time initialization request of the device, the platform generates a timestamp at the initial time or at its current time.
The device then transmits its timer value to the platform along with measurement data generated aperiodically. Here, the timer of the device is a general timer, not an RTC.
The platform sequentially stores the initial generated timestamps along with the measurement data, increasing the timer based on the timer value.
7 is a message sequence chart illustrating another method of collecting and storing measurement data generated by aperiodic sampling in a device in a DBMS.
In Fig. 7, there is a difference from the method shown in Fig. 6 in that a plurality of measurement data is transmitted as a set from the device to the platform. The transmitted data set is configured by storing measurement data to which timer values are added.
The platform maps the measurement data included in the received data set to the corresponding DBMS by increasing the time stamp based on the timer value.
8 is a block diagram of a data storage system in accordance with another embodiment of the present invention. 8, the data storage system according to the embodiment of the present invention includes an
The
The DBMSs 130-1, 130-2, ..., and 130-n are different types of DBMSs that store data collected through the
The
The optimal DBMS selection is made by reference to data profile, storage type, and data query processing history. Hereinafter, this will be described in detail with reference to FIG. FIG. 9 is a flowchart provided in an explanation of a method of configuring a selective DB according to a data type according to another embodiment of the present invention.
9, if there is a data storage request from the IoT / CoT device through the interface 110 (S210-Y), the
If the optimal DBMS for the requested data can be determined from the profile identified in step S220 (S230-Y), the data is stored in the optimal DBMS (S240).
On the other hand, if the optimal DBMS for the requested data can not be determined (S230-N), the data is temporarily stored in the designated DBMS (e.g., DBMS-1 130-1) by default (S250).
Thereafter, when the capacity of the temporarily stored data exceeds the predetermined capacity (S260), the storage type of the data is determined based on the storage period, type, and size of the data, and the profile of the data is deduced from the storage type of the data. (Step S270).
If the optimal DBMS determined in step S270 is different from the DBMS in which the data is stored in step S240 or step S250, data movement to the determined optimal DBMS is additionally performed in step S270.
On the other hand, if the optimal DBMS determined in step S270 is identical to the DBMS in which the data is stored in step S240 or step S250, no data movement is performed in step S270.
Thereafter, when the cumulative query processing history for the stored data exceeds a predetermined amount (S280-Y), the profile of the data is inferred with reference to the time required for the query processing of the data from the accumulated query processing history, And determines an optimal DBMS (S290).
If the optimal DBMS determined in step S290 is different from the DBMS in which the current data is stored, data movement to the optimal DBMS is additionally performed. However, if there is no change in the optimal DBMS, no data movement is performed in step S270.
While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present invention.
110: Interface
120: Processor
130-1, 130-2, ..., 130-n: DBMS
Claims (8)
And storing the data in a determined DBMS,
Wherein,
Determining a DBMS to store the data based on the profile of the data requested to be saved,
If it can not be determined, it is temporarily stored in the designated DBMS by default,
Determining a DBMS to store the data based on the storage period of the data if the capacity of the stored data exceeds a predetermined capacity,
Determining a DBMS in which to store the data based on a time required for query processing of the data obtained from the cumulative query processing history if the query processing history for the data exceeds a predetermined amount.
Wherein,
The general data or meta data is determined to be stored in the first DBMS,
It is determined that the time series data is stored in the second DBMS,
The image data is determined to be stored in the third DBMS,
The moving picture data is determined to be stored in the fourth DBMS,
And the fast sampling data is determined to be stored in the fifth DBMS.
Wherein,
Wherein the time series data is stored together with a time stamp calculated from a sequence number or a timer value received together with the time series data.
And a processor for determining one of the plurality of DBMSs as a DBMS for storing data and storing the data in a determined DBMS,
The processor comprising:
Determining a DBMS to store the data based on the profile of the data requested to be saved,
If it can not be determined, it is temporarily stored in the designated DBMS by default,
Determining a DBMS to store the data based on the storage period of the data if the capacity of the stored data exceeds a predetermined capacity,
Wherein the DBMS to store the data is determined based on a time required for query processing of the data obtained from the cumulative query processing history when the query processing history for the data exceeds a predetermined amount.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| KR1020150167311A KR101785166B1 (en) | 2015-11-27 | 2015-11-27 | Selective DB Configuration Method in accordance with Data Type and System applying the same |
| PCT/KR2015/012851 WO2017090799A1 (en) | 2015-11-27 | 2015-11-27 | Method and system for selectively configuring db according to data type |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| KR1020150167311A KR101785166B1 (en) | 2015-11-27 | 2015-11-27 | Selective DB Configuration Method in accordance with Data Type and System applying the same |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| KR20170062010A KR20170062010A (en) | 2017-06-07 |
| KR101785166B1 true KR101785166B1 (en) | 2017-10-12 |
Family
ID=58763285
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| KR1020150167311A KR101785166B1 (en) | 2015-11-27 | 2015-11-27 | Selective DB Configuration Method in accordance with Data Type and System applying the same |
Country Status (2)
| Country | Link |
|---|---|
| KR (1) | KR101785166B1 (en) |
| WO (1) | WO2017090799A1 (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR102297612B1 (en) * | 2019-11-25 | 2021-09-03 | 한국전자기술연구원 | method for location data service by real time streaming analysis of large-scale location data |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013064815A1 (en) * | 2011-11-04 | 2013-05-10 | Performance Horizon Group | Method and database system for manipulating data |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070192323A1 (en) * | 2006-02-10 | 2007-08-16 | Vertical Systems, Inc. | System and method of access and control management between multiple databases |
| US7937232B1 (en) * | 2006-07-06 | 2011-05-03 | Pivotal Systems Corporation | Data timestamp management |
| US20110153603A1 (en) * | 2009-12-17 | 2011-06-23 | Yahoo! Inc. | Time series storage for large-scale monitoring system |
| US8438573B2 (en) * | 2010-01-15 | 2013-05-07 | Oracle International Corporation | Dependency on a resource type |
-
2015
- 2015-11-27 KR KR1020150167311A patent/KR101785166B1/en active IP Right Grant
- 2015-11-27 WO PCT/KR2015/012851 patent/WO2017090799A1/en active Application Filing
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013064815A1 (en) * | 2011-11-04 | 2013-05-10 | Performance Horizon Group | Method and database system for manipulating data |
Also Published As
| Publication number | Publication date |
|---|---|
| KR20170062010A (en) | 2017-06-07 |
| WO2017090799A1 (en) | 2017-06-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102133951B1 (en) | Short link handling methods, devices, and servers | |
| CN108874803B (en) | Data storage method, device and storage medium | |
| CN103559300B (en) | The querying method and inquiry unit of data | |
| CN104978324B (en) | Data processing method and device | |
| CN111221469B (en) | Method, device and system for synchronizing cache data | |
| US20160328475A1 (en) | Method and system for scheduling web crawlers according to keyword search | |
| US9990396B2 (en) | Forecasting query access plan obsolescence | |
| CN108694188B (en) | Index data updating method and related device | |
| US20160292233A1 (en) | Discarding data points in a time series | |
| CN108509453B (en) | Information processing method and device | |
| CA3167981A1 (en) | Offloading statistics collection | |
| WO2020159397A1 (en) | Method and computerized device for processing numeric time series data | |
| US10990570B2 (en) | Methods, systems and apparatus to improve spatial-temporal data management | |
| KR101785166B1 (en) | Selective DB Configuration Method in accordance with Data Type and System applying the same | |
| EP2778921A2 (en) | A method and a system for distributed processing of a datasheet | |
| JP2013161342A (en) | Method, system and program for generating file list, and file list generation device | |
| CN111124650B (en) | Stream data processing method and device | |
| JP6084700B2 (en) | Search system and search method | |
| CN110647542A (en) | Data acquisition method and device | |
| CN108989383A (en) | Data processing method and client | |
| CN109753505B (en) | Method and system for creating temporary storage unit in big data storage system | |
| CN107070987B (en) | Data acquisition method and system for distributed object storage system | |
| US20210124718A1 (en) | Database device, computer program product, and data processing method | |
| US10855743B1 (en) | Efficient video transfer system | |
| KR102157591B1 (en) | Apparatus for Spatial Query in Big Data Environment and Computer-Readable Recording Medium with Program therefor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| A201 | Request for examination | ||
| E902 | Notification of reason for refusal | ||
| AMND | Amendment | ||
| AMND | Amendment | ||
| X701 | Decision to grant (after re-examination) | ||
| GRNT | Written decision to grant |