CN105786918A - Data loading storage space-based data query method and device - Google Patents
Data loading storage space-based data query method and device Download PDFInfo
- Publication number
- CN105786918A CN105786918A CN201410828712.2A CN201410828712A CN105786918A CN 105786918 A CN105786918 A CN 105786918A CN 201410828712 A CN201410828712 A CN 201410828712A CN 105786918 A CN105786918 A CN 105786918A
- Authority
- CN
- China
- Prior art keywords
- data
- memory block
- task
- storage
- data acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data loading storage space-based data query method. The method comprises the following steps: dividing a storage space into storage blocks, configuring data acquisition tasks and configuring a corresponding relationship between the data acquisition tasks and the storage blocks; creating data storage arrays in the storage blocks corresponding to the data acquisition tasks according to the configured data acquisition tasks; executing the configured data acquisition tasks to acquire data, loading the acquired data into the corresponding storage arrays in the corresponding storage blocks according to a configured style in the data acquisition tasks, and updating the data loading state; and directly executing a data query task according to the task execution state, or carrying out query to obtain the data in the data storage arrays in the storage blocks of the storage space. Through the method disclosed in the invention, high-speed and efficient data query can be realized. The invention furthermore discloses a data loading storage space-based data query device.
Description
Technical field
The present invention relates to computer realm, be specifically related to the technology of data query.
Background technology
Along with the development of IT field, the various money of software system management that the application of increasing field is large-scale
Source data, needs to obtain abundant substantial amounts of resource letter in conventional management system during processing certain data
Breath, such as geographical location information in application-specific scene, other as standardized information, engineering subscription information,
Grouping information etc..
Obtaining the conventional means of these information is to go to search in data base by SQL, and the process of a data needs
Wanting ten several times to the inquiry not waited for tens times, these inquiries relate to a lot of database table and further relate to huge
Data volume, can have a strong impact on the efficiency of inquiry, and a set of software system data storehouse is whole based on list structure inquiry
Vital ingredient in individual system, due to the increase of data processing amount, causes data base to look into frequently
Ask, occur that cpu busy percentage crosses the problems such as height, data base is produced the heaviest pressure, reduce data base
Disposed of in its entirety ability, once data base goes wrong will affect the operation of whole system.Therefore SQL is passed through
The mode efficiency of inquiry data is the lowest, and easily affects the operation of whole system.
The mode also having in prior art is by caching, is positioned over data cached in Java heap, this side
Formula is also problematic, and JVM (Java Virtual Machine Java Virtual Machine) the heap size of increase exceedes
Certain quantity, it will usually cause GC (GarbageCollection) long for time delay, have a strong impact on Java
Application performance, and then affect the performance of whole system.
The data query technique of a kind of efficient raising systematic function urgently occurs.
Summary of the invention
The present invention provides the data query method being loaded into memory space based on data, and described method includes:
Memory space is divided into memory block, configuration data acquisition task and configure described data acquisition task with
The corresponding relation of described memory block；
Data acquisition task according to described configuration creates in memory block corresponding with described data acquisition task
Datum number storage group；
The data acquisition task performing described configuration obtains data, by the data of described acquisition according to described data
In acquisition task, the pattern of configuration is loaded into and described corresponding with its corresponding memory block stores array, more new data
Loading state；
It is loaded into state according to described data and directly performs data query task, or, inquiry obtains described storage sky
Between the data in datum number storage group in memory block.
Preferably, described method also includes:
Data in described each memory block are carried out persistent storage, by described persistence when being again started up storing
The data of storage are disposably loaded into storage.
Preferred:
Divide memory block according to service attribute, identify the class of service of this memory block storage data；
Described memory block is divided into three layers by the needs according to application, specially caching, out-pile caching and magnetic in heap
Disk storage, according to the use temperature of described acquisition data, loads data into different grades of memory block.
Detailed, described configuration data acquisition task particularly as follows:
Configure described data acquisition task title, storage dope vector, perform command information and described data
Obtain the cycle of tasks carrying, number of times, execution pattern, the data volume of execution operation；
According to the relatedness of described data acquisition task, configure described data acquisition task group, described with group
The corresponding same memory block of data acquisition task.
More detailed:
According to the described storage dope vector configured and data acquisition task and the corresponding relation of described memory block
Datum number storage group is created in corresponding memory block；
Execution command information according to described configuration, described data acquisition task perform cycle, number of times, hold
Row mode, the data volume performing to operate perform described data acquisition task, obtain target data；
The pattern that the target data of described acquisition is configured according to described data acquisition task be loaded into create with institute
State and data acquisition task corresponding memory block stores array accordingly.
Further, described according to described data be loaded into state directly perform data query task, or, inquiry
The method of the data obtained in described memory space memory block in datum number storage group particularly as follows:
When described data are loaded into state for being not fully complete, directly target database is performed query task, and will
The data that described execution query task obtains are loaded into memory block；
When described data are loaded into state for completing, inquiry obtains data storage in described memory space memory block
Data in array.
Invention additionally discloses the data query arrangement being loaded into memory space based on data, described device includes:
Memory space, comprises some memory blocks, is used for storing data；
Storage space management unit, is used for managing described memory space, memory space is divided into memory block；
Task dispensing unit, is used for configuring data acquisition task and configuring described data acquisition task depositing with described
The corresponding relation of storage space management dividing elements memory block；
Task executing units, for according to described task dispensing unit configuration data acquisition task, with institute
State establishment datum number storage group in the corresponding memory block of data acquisition task；Perform described data acquisition task to obtain
Fetch data, by the data of described acquisition according to the pattern of configuration in described data acquisition task be loaded into described and its
Corresponding memory block stores array accordingly；Update described data and be loaded into state；
Data query unit, is used for inquiring about described data and is loaded into state, be loaded into state according to described data direct
Perform data query task, or, inquiry obtains the number in described memory space memory block in datum number storage group
According to.
Preferably, described device also includes:
Data persistence unit, for by the data in each memory block of described storage space management dividing elements
Carry out persistent storage, be again started up during storage the data of described persistent storage are disposably loaded into storage.
Detailed, described storage space management unit farther includes:
Memory block divides module, for dividing memory block according to service attribute, identifies this memory block storage data
Class of service；
Storage partition of the level module, divides depositing of Module Division for the needs according to application by described memory block
Storage block is divided into three layers, specially caching, out-pile caching and disk storage in heap；
Storage space management module, for according to described task dispensing unit configuration data acquisition task with deposit
Memory block described in the relation management of storage block divides the memory block of Module Division.
Detailed, the method for described task dispensing unit configuration data acquisition task particularly as follows:
Configure described data acquisition task title, storage dope vector, perform command information and described data
Obtain the cycle of tasks carrying, number of times, the data volume of execution operation；
According to the relatedness of described data acquisition task, configure described data acquisition task group, described with group
The corresponding same memory block of data acquisition task.
Detailed, described task executing units farther includes:
Storage array creation module, for according to described task dispensing unit configuration storage dope vector and
Data acquisition task creates in corresponding memory block with the corresponding relation of memory block in described memory space
Datum number storage group；
Data acquisition module, for the execution command information according to the configuration of described task dispensing unit, described number
Perform described data obtain according to the acquisition cycle of tasks carrying, number of times, execution pattern, the data volume of execution operation
Take task, obtain target data；
Data insmod, single according to the configuration of described task for the data obtained by described data acquisition module
The pattern of unit's configuration is loaded into the storing corresponding with its corresponding memory block of described storage array creation module establishment
Array；
Described data insmod the use temperature according to described acquisition data, load data into described storage level
Do not divide the different grades of memory block of Module Division.
Detailed, described data query unit farther includes:
Data are loaded into state-query module, are used for inquiring about data and are loaded into state；
Query task performs module, for being loaded into, according to described data, the data that state-query module inquiry obtains
It is loaded into unfinished state and directly performs data query task to the target database of described query task；
Data inquiry module, for being loaded into, according to described data, the data loading that state-query module inquiry obtains
The data that completion status inquiry obtains in described memory space memory block in datum number storage group.
Memory space is divided into memory block by the present invention, creates storage storage of array and look into accordingly in memory block
Asking target data, data consumer only need to inquire about array can obtain the target data of inquiry, improves inquiry
Efficiency；It is pre-configured with the corresponding relation of data acquisition task and memory block so that storage configuration can be simple
Extending transversely, add memory block storage data motility；The present invention is also by the data in each memory block
Carry out persistent storage so that be again started up during storage data can disposably be loaded into storage, solve weight
New startup reacquires the problem that data reload storage；Further storage is divided into three storage ranks,
Load data into different grades of memory block according to the temperature that data use, data frequently will be used, put into
Caching in the heap that rank is high, obtains the efficiency of data when further increasing inquiry；In order to ensure the suitable of inquiry
Profit is carried out, and when data have not been loaded into, data base directly can perform data query task, then will
The data obtained are loaded into memory block for inquiring about later；The invention provides a kind of based on data in sum
It is loaded into the efficient data querying method of storage.
Accompanying drawing explanation
The data query method flow being loaded into memory space based on data that Fig. 1 provides for the embodiment of the present invention one
Schematic diagram；
The schematic diagram that the memory block that Fig. 1-1 provides for the embodiment of the present invention one is corresponding with data acquisition task packet；
The method flow schematic diagram that Fig. 2 provides for the embodiment of the present invention two；
The memory space hierarchical management schematic diagram that Fig. 2-1 provides for the embodiment of the present invention two；
The method flow schematic diagram that Fig. 3 provides for the embodiment of the present invention three；
The table data that Fig. 3-1 provides for the embodiment of the present invention three are to data cached schematic diagram；
The method flow schematic diagram that Fig. 4 provides for the embodiment of the present invention four；
The data query arrangement structure being loaded into memory space based on data that Fig. 5 provides for the embodiment of the present invention five
Schematic diagram；
The apparatus structure schematic diagram that Fig. 6 provides for the embodiment of the present invention six.
Detailed description of the invention
Embodiments of the present invention are described in detail, thereby to the present invention such as below in conjunction with graphic and embodiment
What application technology means solves technical problem and reaches the process that realizes of technology effect and can fully understand and according to this
Implement.
Below as it is shown in figure 1, provide embodiments of the invention one to illustrate the number being loaded into memory space based on data
Include according to querying method, such as described method:
Step S101: memory space is divided into memory block, configuration data acquisition task also configures described data
Acquisition task and the corresponding relation of described memory block.
Memory space referred to herein is temporary file exchange area, computer the most frequently used file from memorizer
In put forward to be placed in caching temporarily, the same just as instrument is brought up to workbench with material, so can compare the used time
Warehouse is now gone to take more convenient.Because temporary file exchange area often uses the impermanent storage that power-off is i.e. fallen,
Its preservation form is Key-Value form.
Memory block can be divided according to type of service.One memory block performs depositing of one or more types of service
Storage.
Data acquisition task is grouped also dependent on the relatedness of data, and the data such as same table can be drawn
It is divided into same data acquisition task group；The table data being associated can be divided into same data acquisition task
Group.Identical data obtains the corresponding identical memory block of data acquisition task of task groups.As Figure 1-1.
DB is database purchase, Table1 ... n represents database table, and taskl is data acquisition task, group
For data acquisition task group, cache1 ... n represents memory block.
Step S102: deposit corresponding with described data acquisition task according to the data acquisition task of described configuration
Storage block creates datum number storage group.
Configure described data acquisition task title, storage dope vector, perform command information and described data
Obtain the cycle of tasks carrying, number of times, execution pattern, the data volume of execution operation.
Each memory block of the i.e. memory space in temporary file exchange area preserves data with Key-Value form, therefore
This simple mode of array is used in memory block, data to be stored, can during configuration data acquisition task
The Key of configuration array, and the order of configuration acquisition data.
Step S103: the data acquisition task performing described configuration obtains data, the data of described acquisition is pressed
Array is stored according to corresponding with its corresponding memory block described in the pattern loading of configuration in described data acquisition task,
More new data is loaded into state.
Step S104: be loaded into state according to described data and directly perform data query task, or, inquiry obtains
Data in datum number storage group in described memory space memory block.
When receiving data query task, if data to be inquired about are being loaded into memory block, represent that data obtain
Take task and be not fully complete, then can not find this data in memory block, now can be directly to target database table
Assign data query task, perform data query order, thus obtain target data to be inquired about, inquired about
Become after obtaining target data, Query Result can be generated data acquisition task in the lump together with data query task,
It is loaded into the memory block specified, uses for inquiry next time.If data acquisition task has performed, i.e.
Directly the storage array from memory block can obtain data.
Data in order to solve to restart recover problem, recover generally to have two kinds of situations: one is directly from data base
Reloading such as memory space, the time depends on the performance of data base, and recovering data when data volume is big needs several
Ten minutes, speed was slower.
Preferably, the present invention recovers, into memory space, to present invention additionally comprises step S105 by persistence means.
Step S105: the data in described each memory block are carried out persistent storage, will when being again started up storing
The data of described persistent storage are disposably loaded into storage.
Data persistence, i.e. data (object as in internal memory) being saved in can the storage device of persistence
In (such as disk).The main application of persistence is to be stored in the data base of relationship type by the object in internal memory,
Can certainly be stored in disk file, in XML data file etc..Persistence is to be existed by routine data
The mechanism of conversion between permanent state and instantaneous state.
Owing to memory space is interim impermanent storage, the mode of data persistence therefore can be used to preserve this
A little data, can disposably be loaded into storage when being again powered up.
When scope of the enterprise expands, effective way originally is to increase management level, and present effective way
It is to increase the span of control.When managing level minimizing and the span of control increases, the type of organization of pyramid shape is just
It is ' compressed ' into the type of organization of flat.In this article memory space is divided into memory block one by one,
Storing corresponding target data in each memory block, the data that will be originally stored in table are transferred to
In memory block, fast processing speeds based on memory block, quick data query can be realized.And memory block carries
Enter what the extension of datamation and memory block was planar as, significantly more efficient storage management can be realized.To sum up institute
State the data query realizing efficient high-speed.
In order to preferably illustrate the present invention, embodiments of the invention two are given below, as shown in Figure 2:
Step S201: memory space is divided into memory block according to service attribute.
Divide memory block according to service attribute, identify the class of service of this memory block storage data.
Service attribute such as transmits business, data service etc., and memory block size sets according to practical situations
Fixed.
Described memory block can also be divided into three layers, as shown in Fig. 2-1, specially in heap according to the needs of application
Caching, out-pile caching and disk storage, according to the use temperature of described acquisition data, load data into difference
The memory block of grade.
Simple memory space layered approach is used to manage data, needs when running according to application, automatically
Carry out data transfer between the different layers.
Ground floor is caching (Heap) in heap, can be the heap memory of JVM, it is possible to reach the data of nanosecond
Access performance.
The second layer is that out-pile caches (Off-Heap), can be the region of memory of process embedding, also belong to this locality
Hardware RAM, it is provided that the data access performance to hundreds of GB of Microsecond grade, avoids JAVA GC simultaneously
The problem of the operating lag caused.
Third layer disk areas, it is provided that the data access performance to hundreds of GB to hundreds of TB of Microsecond grade.
Caching can be according to the practical situation during operation applied, and according to the temperature of data, (i.e. data use
Frequency, the highest then temperature of frequency is the highest, it is desirable to can complete to access with the shorter time), scheduling data arrive
Different layerings, most-often used data are configurable on ground floor and store, with the fastest during convenient use
Speed obtains, the like, provide optimum handling capacity for application program.
The storage size of every layer can be set according to practical situation, and the storage of data is also dependent on reality
Situation configures.Such as storage size can set:
MaxElementsInMemory=" 10000 "
It it is caching preservation data amount check 1W in the heap of configuration；
MaxMemoryOffHeap=" 4G "
Configuration out-pile cache size is 4G.
Step S202: configure described data acquisition task title, storage dope vector, perform command information with
And described data acquisition task perform cycle, number of times, execution pattern, perform the data volume of operation.
According to the relatedness of described data acquisition task, configure described data acquisition task group, described with group
The corresponding same memory block of data acquisition task.
Storage dope vector predominantly specifies the key of array, and performing command information is acquisition data from object table
SQL statement, configuration the cycle of execution task, number of times, execution model, perform operation data volume, can
Performing task according to the parameter of configuration, these parameters can increase according to practical situation, to reach configuration
The data acquisition task of laminating user's request.
Step S203: according to the described storage dope vector configured and data acquisition task and described memory block
Corresponding relation in corresponding memory block, create datum number storage group.
Storage array creates in memory block, according to the memory block that data acquisition task in above-mentioned steps is corresponding,
Creating storage array in this memory block, the key of storage array is the key configured in data acquisition task.
Array type is usually object array.
Step S204: according to the execution command information of described configuration, described data acquisition task perform cycle,
Number of times, execution pattern, the data volume of execution operation perform described data acquisition task, obtain target data.
Step S205: the pattern that the target data of described acquisition configures according to described data acquisition task is loaded into
The corresponding with described data acquisition task corresponding memory block of establishment stores array, and more new data is loaded into state.
Step S206: receive data query task, obtains data and is loaded into state, if data are loaded into state has been
Become and then enter step S207, if data loading state is for being not fully complete, enter step S208.
Step S207: inquiry obtains the data in described memory space memory block in datum number storage group.
Step S208: directly target database is performed the data query order in data query task.
Step S209: the data obtained by described execution querying command are loaded into memory block.
Step S210: the data in described each memory block are carried out persistent storage, will when being again started up storing
The data of described persistent storage are disposably loaded into storage.
400000 times/s, low latency can be brought up to improving data query speed by 100 times/s by the present invention
High-throughput, performance obtains linear expansion, and follow-up processing speed to data is greatly improved.Use off-heap
Technology, is stored in out-pile (Off heap) by data cached, solves the garbage reclamation mechanism of Java for a long time
Performance bottleneck, the size solving internal memory limits, and solves Java process Full GC and causes Business Processing to occur stopping
Phenomenon.Data storage of hierarchically, can realize hot spot data high speed access problem.It is adapted to all main flows JVM
With general low cost hardware.
For the method better illustrating the present invention, provide embodiments of the invention three below in conjunction with example, say
Bright configuration data acquisition task obtains data and is loaded into the process of caching as shown in Figure 3:
Memory space and memory block divide previously according to practical situation, and the caching that this example relates to is entitled
“resourceCache”。
Step S301: configuration data acquisition task.
By the form of configuration file, data acquisition task is configured, as follows:
Configuration instruction:
Group: task groups title, for gcity
Cache: caching title, for resourceCache
Wherein it is configured with two Key and illustrates that same SQL has two kinds of different Key and selects.One be with
Region_city_local:{0} is key, i.e. with create array under be designated as the content in 0 position as key；
One is with region_city_local:{5} as Key, i.e. with create array under be designated as in 5 positions in
Hold for Key.
PageSize: the number of page of data when data base querying.
PageSum: how many page datas batch is submitted to caching.
The Data Source of SQL: caching, obtains data by tables of data region_city_local is performed SQL.
Scheduler: periodic scheduling time " 0305**？" represent that 5:30 every day starts to load.
Warmup:1 represents Asynchronous loading, and 2 tables are sought common ground to walk and loaded.
Repeatinterval: load time interval when unsuccessfully retrying, is 60 seconds.
Repeatcount: load the number of times unsuccessfully retried, is 2 times.
Configuration Context resolution is two caching loading tasks by loading procedure, and content stores slow with the form of array
In depositing, one of them is with region_city_local:{0} as Key, and another is region_city_local:{5}.
Wherein { 0} is No. 0 index of array in SQL result, can be with city_id and city_name field when i.e. running
Value be replaced such as: region_city_local:1 and region_city_local: Beijing, be saved in caching name
It is referred to as in the caching of resourceCache, within 5: 30, carries out data renewal in every day, if failure,
Can rerun twice, every minor tick 60 seconds.
Step S302: create data in corresponding caching according to the storage dope vector in data acquisition task
Storage array.
Here select to create object array, i.e. Object [] array, it is not necessary to logarithm assembles to be put, and can maintain former
Beginning database field type.Convenient caching user is developed, it is not necessary to quote other object, only needs
The index finding correspondence is the most permissible.
Here according to the configuration in data acquisition task, in resourceCache caches, create two arrays.
One key is region_city_local:{0}, and a key is region_city_local:{5}.
Step S303: perform the execution command information in data acquisition task, obtains the data in object table.
Execution command information in data acquisition task is: select
city_id,province_name,province_id,region_name,region_id,city_name from
region_city_local
Performing this order, obtaining city_id from region_city_local table is 13, and province_name is
Beijing, province_id, region_name are 1, and region_id is Haidian District, and city_name is the letters such as Beijing
Breath.
Step S304: the data of acquisition are inserted establishment datum number storage group in caching.
The data of acquisition are stored in respectively:
Key is the array of region_city_local:{13}
Table 1Key is the array of region_city_local:{13}
Be region_city_local:{ Beijing with key } array.
Table 2Key is region_city_local:{ Beijing } array
By table data to data cached transformation process as shown in figure 3-1.
When caching loading data in the event of the request of inquiry data, then can be by Asynchronous loading mechanism
Solve this problem.
When there being data request notification, first pass through data and be loaded into whether data corresponding to status poll are loaded into task
Complete, this refers to individual task, be not required to all tasks and all loaded, but which has loaded
Which will use.The data loading state obtained when not loaded is for being not fully complete, and this time will
Cache inquiry is converted to Sql inquiry, and the result set obtaining data base returns.After caching loading completes then certainly
Dynamic will request returns data from caching.Thus solve start-up course to be no longer necessary to wait that caching loading completes
The problem that just can carry out Business Processing.Can be obtained by the data of above-mentioned configuration according to practical application request
Take whether the execution mechanism state modulator in task carries out the Asynchronous loading of data.
For more detailed explanation said process, provide embodiments of the invention four, as shown in Figure 4.
Step S401: data acquisition task obtains data, and is loaded into the caching corresponding with data acquisition task, more
New data is loaded into state.
Step S402: receive the inquiry data task of user, and inquire about described data loading state, if data
Loading state, for being not fully complete then entrance step S403, if data loading state is for completing, then enters step S405
Step S403: inquiry data task is converted to SQL request, performs SQL to target database.
Step S404: return target database SQL to user and perform result, will perform result and be loaded into caching.
Step S405: obtained the data of user's inquiry by Key from the storage array of described caching.
The present invention also provides for being loaded into the data query arrangement of memory space in order to realize based on data load based on data
Enter the data query method of memory space, embodiments of the invention five are given below in order to described device to be described
Concrete structure, as shown in Figure 5.
The data query arrangement being loaded into memory space based on data includes:
Memory space 1, comprises some memory blocks, is used for storing data.
Memory space referred to herein is temporary file exchange area, computer the most frequently used file from memorizer
In put forward to be placed in caching temporarily, the same just as instrument is brought up to workbench with material, so can compare the used time
Warehouse is now gone to take more convenient.Because temporary file exchange area often uses the impermanent storage that power-off is i.e. fallen,
Its preservation form is Key-Value form.Storage space management unit 2, deposits for being divided into by memory space
Storage block.
Memory block can be divided according to type of service.One memory block performs depositing of one or more types of service
Storage.
Data acquisition task is grouped also dependent on the relatedness of data, and the data such as same table can be drawn
It is divided into same data acquisition task group；The table data being associated can be divided into same data acquisition task
Group.Identical data obtains the corresponding identical memory block of data acquisition task of task groups.
Task dispensing unit 3, is used for configuring data acquisition task and configuring described data acquisition task depositing with described
The corresponding relation of storage space management dividing elements memory block.
Task dispensing unit configuration data acquisition task method particularly as follows:
Configure described data acquisition task title, storage dope vector, perform command information and described data
Obtain the cycle of tasks carrying, number of times, the data volume of execution operation；
According to the relatedness of described data acquisition task, configure described data acquisition task group, described with group
The corresponding same memory block of data acquisition task.
Each memory block of the i.e. memory space in temporary file exchange area preserves data with Key-Value form, therefore
This simple mode of array is used in memory block, data to be stored, can during configuration data acquisition task
The Key of configuration array, and the order of configuration acquisition data.
Task executing units 4, for according to described task dispensing unit configuration data acquisition task, with institute
State establishment datum number storage group in the corresponding memory block of data acquisition task；Perform described data acquisition task to obtain
Fetch data, by the data of described acquisition according to the pattern of configuration in described data acquisition task be loaded into described and its
Corresponding memory block stores array accordingly；Update described data and be loaded into state.
Data query unit 5, is used for inquiring about described data and is loaded into state, be loaded into state according to described data direct
Perform data query task, or, inquiry obtains the number in described memory space memory block in datum number storage group
According to.
When receiving data query task, if data to be inquired about are being loaded into memory block, represent that data obtain
Take task and be not fully complete, then can not find this data in memory block, now can be directly to target database table
Assign data query task, perform data query order, thus obtain target data to be inquired about, inquired about
Become after obtaining target data, Query Result can be generated data acquisition task in the lump together with data query task,
It is loaded into the memory block specified, uses for inquiry next time.If data acquisition task has performed, i.e.
Directly the storage array from memory block can obtain data.This pattern whether is used to carry out data query
Can be configured by " execution pattern " item in above-mentioned data acquisition task.
Data persistence unit 6, for by the data in each memory block of described storage space management dividing elements
Carry out persistent storage, be again started up during storage the data of described persistent storage are disposably loaded into storage.
The operation principle of the present invention is: memory space is first drawn by storage space management unit 2 according to service attribute
Dividing good memory block and the rank of memory block, task dispensing unit 3 has configured data acquisition task, tasks carrying
Unit 4 performs these data acquisition task, creates storage array and by the data of acquisition according to situation about configuring
Being loaded into the storage array in memory block, data consumer can directly inquire about quick obtaining institute by the key of array
The data needed, it is achieved that the data query of high-speed and high-efficiency.
In order to describe the structure of the data query arrangement each several part being loaded into memory space based on data in detail, below
Provide embodiments of the invention six, as shown in Figure 6.
Memory space 1, comprises some memory blocks, is used for storing data.
Storage space management unit 2 farther includes:
Memory block divides module 21, for dividing memory block according to service attribute, identifies this memory block storage number
According to class of service.
Storage partition of the level module 22, for needing described memory block division Module Division according to application
Memory block is divided into three layers, and specially caching, out-pile caching and disk storage in heap, according to described acquisition data
Use temperature, load data into different grades of memory block.
Task dispensing unit 3, is used for configuring data acquisition task and configuring described data acquisition task depositing with described
The corresponding relation of storage space management dividing elements memory block
Task executing units 4 farther includes:
Storage array creation module 41, for according to described task dispensing unit configuration storage dope vector with
And the corresponding relation of memory block is created in corresponding memory block in data acquisition task and described memory space
Build datum number storage group；
Data acquisition module 42, for the execution command information, described according to the configuration of described task dispensing unit
The cycle of data acquisition task execution, number of times, execution pattern, the data volume of execution operation perform described data
Acquisition task, obtains target data；
Data insmod 43, configure according to described task for the data obtained by described data acquisition module
The pattern of cell location is loaded into the depositing corresponding with its corresponding memory block of described storage array creation module establishment
Storage array.
Data query unit 5 farther includes:
Data are loaded into state-query module 51, are used for inquiring about data and are loaded into state.
Query task performs module 52, for being loaded into, according to described data, the number that state-query module inquiry obtains
Directly data query task is performed to the target database of described query task according to being loaded into unfinished state.
Data inquiry module 53, for being loaded into, according to described data, the data load that state-query module inquiry obtains
Enter the data that completion status inquiry obtains in described memory space memory block in datum number storage group.
Data persistence unit 6, for by the data in each memory block of described storage space management dividing elements
Carry out persistent storage, be again started up during storage the data of described persistent storage are disposably loaded into storage.
Assembly of the invention is in order to realize being loaded into the querying method in space, the therefore work of each several part based on data
Principle is similar with method part, can refer to the description of said method part, and application example can also refer to method portion
The description divided, does not repeats them here.
Although the embodiment that disclosed herein is as above, but described content being not used to directly limits this
The protection domain of invention.Any the technical staff in the technical field of the invention, is being taken off without departing from the present invention
On the premise of the spirit and scope of dew, in form and a little change can be made in details implement.This
Bright protection domain, still must be defined in the range of standard with appending claims.
Claims (12)
1. it is loaded into the data query method of memory space based on data, it is characterised in that described method includes:
Memory space is divided into memory block, and configuration data acquisition task also configures described data acquisition task
Corresponding relation with described memory block；
Data acquisition task according to described configuration is created in memory block corresponding with described data acquisition task
Build datum number storage group；
The data acquisition task performing described configuration obtains data, by the data of described acquisition according to described number
It is loaded into described store array corresponding with its corresponding memory block according to the pattern of configuration in acquisition task, updates
Data are loaded into state；
It is loaded into state according to described data and directly performs data query task, or, inquiry obtains described storage
Data in datum number storage group in the memory block of space.
Method the most according to claim 1, it is characterised in that described method also includes:
Data in described each memory block are carried out persistent storage, be again started up storage time by described persistently
The data changing storage are disposably loaded into storage.
Method the most according to claim 2, it is characterised in that:
Divide memory block according to service attribute, identify the class of service of this memory block storage data；
Described memory block is divided into three layers by the needs according to application, specially caching in heap, out-pile caching and
Disk storage, according to the use temperature of described acquisition data, loads data into different grades of memory block.
Method the most according to claim 3, it is characterised in that described configuration data acquisition task particularly as follows:
Configure described data acquisition task title, storage dope vector, perform command information and described number
According to obtaining the cycle of tasks carrying, number of times, execution pattern, the data volume of execution operation；
According to the relatedness of described data acquisition task, configure described data acquisition task group, with organizing institute
State the corresponding same memory block of data acquisition task.
Method the most according to claim 4, it is characterised in that:
Described storage dope vector according to configuration and data acquisition task pass corresponding with described memory block
Tie up to corresponding memory block creates datum number storage group；
Execution command information according to described configuration, described data acquisition task perform cycle, number of times,
Execution pattern, the data volume performing to operate perform described data acquisition task, obtain target data；
The pattern that the target data of described acquisition is configured according to described data acquisition task be loaded into create with
Described data acquisition task corresponding memory block stores array accordingly.
Method the most according to claim 5, it is characterised in that described straight according to described data loading state
Connect execution data query task, or, in the inquiry described memory space memory block of acquisition in datum number storage group
Data method particularly as follows:
When described data are loaded into state for being not fully complete, directly target database is performed query task, and
The data obtained by described execution query task are loaded into memory block；
When described data are loaded into state for completing, inquiry obtains data in described memory space memory block and deposits
Data in storage array.
7. it is loaded into the data query arrangement of memory space based on data, it is characterised in that described device includes:
Memory space, comprises some memory blocks, is used for storing data；
Storage space management unit, is used for managing described memory space, memory space is divided into memory block；
Task dispensing unit, is used for configuring data acquisition task and configuring described data acquisition task with described
The corresponding relation of storage space management dividing elements memory block；
Task executing units, for according to described task dispensing unit configuration data acquisition task, with
The corresponding memory block of described data acquisition task creates datum number storage group；Perform described data acquisition to appoint
Business obtains data, and according to the pattern of configuration in described data acquisition task, the data of described acquisition are loaded into institute
State and corresponding with its corresponding memory block store array；Update described data and be loaded into state；
Data query unit, is used for inquiring about described data and is loaded into state, be loaded into state according to described data straight
Connect execution data query task, or, in the inquiry described memory space memory block of acquisition in datum number storage group
Data.
Device the most according to claim 7, it is characterised in that described device also includes:
Data persistence unit, for by the number in each memory block of described storage space management dividing elements
According to carrying out persistent storage, when being again started up storage, the data of described persistent storage are disposably loaded into and deposit
Storage.
Device the most according to claim 8, it is characterised in that described storage space management unit is further
Including:
Memory block divides module, for dividing memory block according to service attribute, identifies this memory block storage number
According to class of service；
Storage partition of the level module, for needing described memory block division Module Division according to application
Memory block is divided into three layers, specially caching, out-pile caching and disk storage in heap；
Storage space management module, for according to described task dispensing unit configuration data acquisition task with
Memory block described in the relation management of memory block divides the memory block of Module Division.
Device the most according to claim 9, it is characterised in that described task dispensing unit configuration number
According to obtain task method particularly as follows:
Configure described data acquisition task title, storage dope vector, perform command information and described number
According to obtaining the cycle of tasks carrying, number of times, the data volume of execution operation；
According to the relatedness of described data acquisition task, configure described data acquisition task group, with organizing institute
State the corresponding same memory block of data acquisition task.
11. devices according to claim 10, it is characterised in that described task executing units is further
Including:
Storage array creation module, for according to described task dispensing unit configuration storage dope vector with
And in data acquisition task and described memory space the corresponding relation of memory block in corresponding memory block
Create datum number storage group；
Data acquisition module, for the execution command information, described according to the configuration of described task dispensing unit
The cycle of data acquisition task execution, number of times, execution pattern, the data volume of execution operation perform described number
According to the task of acquisition, obtain target data；
Data insmod, and configure according to described task for the data obtained by described data acquisition module
The pattern of cell location is loaded into the corresponding with its corresponding memory block of described storage array creation module establishment
Storage array；
Described data insmod the use temperature according to described acquisition data, load data into described storage
The different grades of memory block of partition of the level Module Division.
12. devices according to claim 11, it is characterised in that described data query unit wraps further
Include:
Data are loaded into state-query module, are used for inquiring about data and are loaded into state；
Query task performs module, for being loaded into, according to described data, the number that state-query module inquiry obtains
Directly data query task is performed to the target database of described query task according to being loaded into unfinished state；
Data inquiry module, for being loaded into, according to described data, the data load that state-query module inquiry obtains
Enter the data that completion status inquiry obtains in described memory space memory block in datum number storage group.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410828712.2A CN105786918B (en) | 2014-12-26 | 2014-12-26 | Data query method and device based on data loading storage space |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410828712.2A CN105786918B (en) | 2014-12-26 | 2014-12-26 | Data query method and device based on data loading storage space |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105786918A true CN105786918A (en) | 2016-07-20 |
| CN105786918B CN105786918B (en) | 2020-08-04 |
Family
ID=56388624
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410828712.2A Active CN105786918B (en) | 2014-12-26 | 2014-12-26 | Data query method and device based on data loading storage space |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105786918B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106844721A (en) * | 2017-02-09 | 2017-06-13 | 济南浪潮高新科技投资发展有限公司 | Date storage method, device, system, computer-readable recording medium and storage control |
| CN107844262A (en) * | 2016-09-21 | 2018-03-27 | 东软集团股份有限公司 | Data buffer storage, sending method and device |
| CN109408689A (en) * | 2018-10-24 | 2019-03-01 | 北京金山云网络技术有限公司 | Data capture method, device, system and electronic equipment |
| CN111831699A (en) * | 2020-09-21 | 2020-10-27 | 北京新唐思创教育科技有限公司 | Data caching method, electronic equipment and computer readable medium |
| CN113204564A (en) * | 2021-05-20 | 2021-08-03 | 山东英信计算机技术有限公司 | Database high-frequency SQL query method, system and storage medium |
| CN113641711A (en) * | 2021-08-17 | 2021-11-12 | 天津卓盛云科技有限公司 | Data caching processing method, device and medium for SAAS tenant |
| CN114840498A (en) * | 2022-07-05 | 2022-08-02 | 北京优合融宜科技有限公司 | Method and device for realizing memory key value data management based on Java technology |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101408852A (en) * | 2008-11-26 | 2009-04-15 | 阿里巴巴集团控股有限公司 | Method, apparatus and system for scheduling task |
| CN101710983A (en) * | 2009-11-24 | 2010-05-19 | 中兴通讯股份有限公司 | Method for playing real-time streaming files and device thereof |
| CN102103545A (en) * | 2009-12-16 | 2011-06-22 | 中兴通讯股份有限公司 | Method, device and system for caching data |
| US8015358B2 (en) * | 2005-02-10 | 2011-09-06 | International Business Machines Corporation | System bus structure for large L2 cache array topology with different latency domains |
| CN103023800A (en) * | 2012-11-29 | 2013-04-03 | 北京航空航天大学 | Method for scheduling traffic under multi-core network processor by traffic chart mapping scheduling strategy |
| CN103390061A (en) * | 2013-07-31 | 2013-11-13 | 浙江大学 | Multi-level cache-based sea spatio-temporal data accessing method taking user as center |
| CN103455284A (en) * | 2013-09-18 | 2013-12-18 | 北京华胜天成科技股份有限公司 | Method and device for reading and writing data |
-
2014
- 2014-12-26 CN CN201410828712.2A patent/CN105786918B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8015358B2 (en) * | 2005-02-10 | 2011-09-06 | International Business Machines Corporation | System bus structure for large L2 cache array topology with different latency domains |
| CN101408852A (en) * | 2008-11-26 | 2009-04-15 | 阿里巴巴集团控股有限公司 | Method, apparatus and system for scheduling task |
| CN101710983A (en) * | 2009-11-24 | 2010-05-19 | 中兴通讯股份有限公司 | Method for playing real-time streaming files and device thereof |
| CN102103545A (en) * | 2009-12-16 | 2011-06-22 | 中兴通讯股份有限公司 | Method, device and system for caching data |
| CN103023800A (en) * | 2012-11-29 | 2013-04-03 | 北京航空航天大学 | Method for scheduling traffic under multi-core network processor by traffic chart mapping scheduling strategy |
| CN103390061A (en) * | 2013-07-31 | 2013-11-13 | 浙江大学 | Multi-level cache-based sea spatio-temporal data accessing method taking user as center |
| CN103455284A (en) * | 2013-09-18 | 2013-12-18 | 北京华胜天成科技股份有限公司 | Method and device for reading and writing data |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107844262A (en) * | 2016-09-21 | 2018-03-27 | 东软集团股份有限公司 | Data buffer storage, sending method and device |
| CN107844262B (en) * | 2016-09-21 | 2021-06-25 | 东软集团股份有限公司 | Data caching and sending method and device |
| CN106844721A (en) * | 2017-02-09 | 2017-06-13 | 济南浪潮高新科技投资发展有限公司 | Date storage method, device, system, computer-readable recording medium and storage control |
| CN109408689A (en) * | 2018-10-24 | 2019-03-01 | 北京金山云网络技术有限公司 | Data capture method, device, system and electronic equipment |
| CN111831699A (en) * | 2020-09-21 | 2020-10-27 | 北京新唐思创教育科技有限公司 | Data caching method, electronic equipment and computer readable medium |
| CN111831699B (en) * | 2020-09-21 | 2021-01-08 | 北京新唐思创教育科技有限公司 | Data caching method, electronic equipment and computer readable medium |
| CN113204564A (en) * | 2021-05-20 | 2021-08-03 | 山东英信计算机技术有限公司 | Database high-frequency SQL query method, system and storage medium |
| CN113204564B (en) * | 2021-05-20 | 2023-02-28 | 山东英信计算机技术有限公司 | Database high-frequency SQL query method, system and storage medium |
| CN113641711A (en) * | 2021-08-17 | 2021-11-12 | 天津卓盛云科技有限公司 | Data caching processing method, device and medium for SAAS tenant |
| CN113641711B (en) * | 2021-08-17 | 2024-05-31 | 天津卓盛云科技有限公司 | Data caching processing method, device and medium for SAAS tenant |
| CN114840498A (en) * | 2022-07-05 | 2022-08-02 | 北京优合融宜科技有限公司 | Method and device for realizing memory key value data management based on Java technology |
| CN114840498B (en) * | 2022-07-05 | 2022-09-13 | 北京优合融宜科技有限公司 | Method and device for realizing memory key value data management based on Java technology |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105786918B (en) | 2020-08-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105786918A (en) | Data loading storage space-based data query method and device | |
| Marcu et al. | Spark versus flink: Understanding performance in big data analytics frameworks | |
| US9767131B2 (en) | Hierarchical tablespace space management | |
| CN102831120B (en) | A kind of data processing method and system | |
| CN101354726B (en) | Method for managing memory metadata of cluster file system | |
| CN105487818B (en) | For the efficient De-weight method of repeated and redundant data in cloud storage system | |
| CN102591970B (en) | Distributed key-value query method and query engine system | |
| CN101930472A (en) | Parallel query method for distributed database | |
| US8214356B1 (en) | Apparatus for elastic database processing with heterogeneous data | |
| KR20180027326A (en) | Efficient data caching management in scalable multi-stage data processing systems | |
| CN103455526A (en) | ETL (extract-transform-load) data processing method, device and system | |
| US9176867B2 (en) | Hybrid DRAM-SSD memory system for a distributed database node | |
| CN102236580B (en) | Method for distributing node to ETL (Extraction-Transformation-Loading) task and dispatching system | |
| CN104050250A (en) | Distributed key-value query method and query engine system | |
| CN109918450B (en) | Distributed parallel database based on analysis type scene and storage method | |
| CN104111936A (en) | Method and system for querying data | |
| Xiong et al. | Data vitalization: a new paradigm for large-scale dataset analysis | |
| CN103198097A (en) | Massive geoscientific data parallel processing method based on distributed file system | |
| CN105975345A (en) | Video frame data dynamic equilibrium memory management method based on distributed memory | |
| CN115391427A (en) | System, method, medium, and apparatus for automatic scaling of Impala | |
| Zhao et al. | Toward efficient and flexible metadata indexing of big data systems | |
| US20190243581A1 (en) | Flash registry with write leveling | |
| CN106156049A (en) | A kind of method and system of digital independent | |
| JP5464017B2 (en) | Distributed memory database system, database server, data processing method and program thereof | |
| Le et al. | Namenode and datanode coupling for a power-proportional hadoop distributed file system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |