CN101136029A - Main ems memory data centre data processing method - Google Patents
Main ems memory data centre data processing method Download PDFInfo
- Publication number
- CN101136029A CN101136029A CNA2007101309065A CN200710130906A CN101136029A CN 101136029 A CN101136029 A CN 101136029A CN A2007101309065 A CNA2007101309065 A CN A2007101309065A CN 200710130906 A CN200710130906 A CN 200710130906A CN 101136029 A CN101136029 A CN 101136029A
- Authority
- CN
- China
- Prior art keywords
- data
- dataset
- mmdc
- sql
- udbi
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention comprises a universal data base access interface platform (UDBI FRAMEWORK) and a main memory data center (MMDC); the data base uses the UDBI FRAMEWORK to access the data of MMDC; the component of MMDC is released by using C/C++ language; using UDBI FRAMEWORK interface is namely to make the abstract package for the UDBI FRAMEWORK interface; if needing to make modification, adding SQL, and using PATCH mechanism to update the corresponding SQL version; the data set returned from SQL is received by DATASET component; using one or more Dataset and MMDC head used for saving the dataset to compose the MMDC; the DATASET mode has a LOCAL mode used in single progress and a SHARE mode shared by progresses.
Description
Technical field
The present invention relates to the data processing method of calculator memory data center, especially carrier class backstage charging account big data quantity disposal route, the particularly technology of the data processing method of main memory data center and constituent components.
Background technology
(the MAIN MEMORY DATA CENTER of main memory data center, be called for short MMDC) assembly, be by general Sybase (Oracle, MS SQL Server, DB2, MySQL, Sybase, Informix) access interface platform (UDBIFRAMEWORK) and main memory data center form, and be similar to memory database, but it are different from memory database again.
Access interface platform UDBI FRAMEWORK: be unified abstract package to the mainstream data bank interface, form unified calling interface, the data base query language SQL that interface is carried out need not to write complex interface by the configuration file configuration and calls, and also has the buffer memory effect simultaneously.SQL to configuration also has heat to dispose (upgrading rollback) function.
MMDC is the center of storage data, is divided into LOCAL MEMORY (local internal memory), SHARED MEMORY (shared drive), FILE MAP (File mapping storage).MMDC is made up of one or more DATASET data capsules, and corresponding D ATASET also has 3 types: LOCAL, SHARED, FILEMAP.DATASET is similar to the TABLE in the database, can the combination of any row and row be set up and be searched index.
In the prior art, the MMDC assembly is that language such as JAVA or C/C++ are realized.Compare, the C/C++ language has very big advantage on execution efficient, but some drawbacks are also arranged:
●UDBI?FRAMEWORK
1) relational database is of a great variety in the market, and the interface of visit has nothing in common with each other, if set up a unified access mode, and must an abstract unified access interface platform.
2) handle the data set (resultset) that returns.Because so the limitation (not supporting reflex etc.) of C/C++ language itself is can not the direct return results collection of formation object.
3) in the practical application because the variation of demand, with the requirement meeting of access data database being carried out SQL revises or increases newly, and normally ran without interruption in 7*24 hour for OLTP system (online transaction processing system/towards the disposal system of transaction), impossible frequent start-stop system disposes (not update system under the situation of start stop system) so will there be a kind of mechanism to guarantee heat.
MMDC structurally also has following requirement:
1) DATASET, the memory module of data is similar to the table in the database, needs to support the dynamic creation list structure, increases the literary name section.
2) data sharing mode (LOCAL, SHARED, FILEMAP).
3) internal memory uses, and the C++ object uses internal memory normally to create by key word NEW mode, and if this mode is destroyed object for the establishment of big data quantity and may be produced memory fragmentation, influence the service efficiency of internal memory.
4) access mode, column index is not supported in corresponding big data quantity visit.
Summary of the invention
In order to solve the aforementioned problems in the prior, the present invention proposes the data processing method and the assembly of a kind of main memory data center (MMDC).
Concrete technical scheme is: the data processing method of a kind of main memory data center (MMDC): adopting the unified database access interface is that the UDBI assembly is that the mainstream data bank interface is unified abstract package: when system carries out SQL for the first time, UDBI meeting buffer memory SQL text and version number thereof, revise if desired, increase SQL, by the SQL version of PATCH new mechanism correspondence; SQL carries out the data set that returns and accepts data by data center's cells D ATASET assembly; By one or more DataSet (data capsule) with preserve the MMDC that the structural information MMDC head of dataset forms; The pattern of DATASET has in the individual process LOCAL pattern of using: or the SHARE pattern of sharing between can process: according to container data address hash value, B* tree and B+ tree index: set up index and delete index:
bool?dropIndex(const?char?*name)；
DATA CENTER backup, recover:
Because MMDC HEAD preserved the data dictionary of data center, and this dictionary information shares, and can serializing, just can carry out backup and recovery by this information; Concrete grammar is that UNIX sends message by the SH script down, and MMDC is by carrying out Dump (), and Recover () carries out, and after the program accident withdraws from, recovers to carry out when start next time.
DataSet can set up in the following way: the order of the store data by field of definition, deleted data, insertion data is set up; When inserting data at every turn, capacity according to present DATASET judges whether to surpass the internal memory that has distributed, if surpass, the continuous internal memory of just reallocating, the size of piece is specified (being defaulted as 4k) when making up DATASET, according to data volume what rationally the size of assignment of allocation piece can improve Memory Allocation efficient.
According to container data address hash value, B* tree and B+ tree index: set up index and deletion index:
DATA CENTER backup, recover: concrete grammar is that UNIX sends message by the SH script down, and MMDC is by carrying out Dump (), and Recover () carries out, and after the program accident withdraws from, recovers to carry out when start next time.
UDBI FRAMEWORK primary clustering:
UDBI: unified database access interface; GenericDAO: database is carried out object;
DataSet: data center unit (MMDC minimum data unit);
SQL Config: carry out SQL xml file configuration; SQL xml:SQL configuration file.
Wherein adopting the UDBI assembly is that the mainstream data bank interface is unified abstract package.When system carried out SQL for the first time, UDBI meeting buffer memory SQL text and version number thereof revised if desired, increase SQL, can be by the SQL version of PATCH new mechanism correspondence, to reach the purpose of heat deployment.SQL carries out the data set that returns and accepts data by the DATASET assembly.
MMDC forms (preserve the structural information of dataset, can be understood as system data dictionary) by one or more DataSet (data capsule) and MMDC head; The pattern of DATASET has the LOCAL pattern: can only use in the individual process, can not share.SHARE pattern: can share between process.Primary clustering is as follows:
Dataset: data capsule; DATA: abstract data (do not deposit real data, only preserve data address);
MemoyObj: internal storage structure information; Allocator: memory allocator;
Row: the record in the data capsule (do not deposit real data, only preserve data address);
Field: container field (being similar to the COLUMN in the database table);
Index: index class masterplate (can set up index to the data container) by it;
Compared with prior art, the present invention has continued the C/C++ language in the advantage of carrying out on the MMDC component efficiency, has improved original drawback.
Description of drawings
Fig. 1 UDBI FRAMEWORK software architecture diagram
Fig. 2 UDBI class-hierarchy structure gragh
Fig. 3 MMDC class-hierarchy structure gragh
Embodiment
Being mainly used in carrier class backstage charging account big data quantity handles.
DataSet can set up in the following way:
DataSet?ds；
ds.addfield(Field(“f1”，FTSTRING，10))；
ds.addfield(Field(“f2”，FTINT))；
ds.addfield(Field(“f3”，FTDOUBLE))；
ds.addfield(Field(“f4”，FTSTRING，20))；
DataSet::iterator it=ds.pushBack (); // store data:
(*it)[0]＝“abc”；
(*it)[1]＝12；
(*it)[2]＝12.56；
(*it)[3]＝“hello”；
Ds.erase (); // deleted data:
Ds.insert () // insertion data:
When inserting data at every turn, can judge whether to surpass the internal memory that has distributed according to the capacity of present DATASET, if surpass, the continuous internal memory of just reallocating, the size of piece can be specified (being defaulted as 4k) when making up DATASET, according to data volume what rationally the size of assignment of allocation piece can improve Memory Allocation efficient.
DATASET sets up index:
ds.buildIndex(“indexname”，”fi?ledName”，SORT_TYPE?type)；
Ds.buildIndex (" indexnamel ", 2, " fieldl ", field2 "); // build composite index
A DATASET can set up a plurality of index, uses when execution is searched.
bool?locate(pair<IndexIterator，IndexIterator>&prLoc，size_tindexNo，...)；
bool?locate(pair<IndexIterator，IndexIterator>&prLoc，si?ze_tindexNo，const?Row&row)；
The deletion index:
bool?dropIndex(const?char*name)；
void?dropAllIndex()；
The index principle: mainly be according to container data address hash value, B* tree and B+ tree index.
Method with the abstract UDBI of C/C++:
The unified main method of access interface AbstractConnection assembly has:
Virtual void connect (const string﹠amp; ConnStr); // be connected to database
Virtual void reconnect (); // reconnect to database
Virtual void di sconnect (); // turn-off data storehouse connects
Virtual void beginTrans (); // beginning affairs
Virtual void commit (); // submit affairs
Virtual void rollback (); // rollback affairs
AbstractStatement assembly main method has:
AbstractStatement(AbstractConnection*conn)；
Void setSQL (const string﹠amp; SqlText); // be provided with and carry out SQL
Void prepareSQL (const string﹠amp; SqlRef); // prepare sql to carry out
Int fetchResultSet (DataSet﹠amp; Ds, int getRows=-1); // obtain set of data structures
Realize above definition uniform database access interface for different database products.
Ratio ORACLE database, behind the realization interface, OracleConnection OracleStatement
Using method:
AbstractConnection*conn＝new?OracleConnection()；
conn->connect(“user/pwd@sid”)；
AbstractStatement*statement＝new?OracleStatment(conn)；
statement->setSQL(“SELECT*FROM?TAB1”)；
DataSet?ds；
Statement-〉fetchResultSet (ds); // obtain data
delete?statement；
statement＝NULL；
conn->disconnect()；
Solve the concrete grammar that unified result returns problem with C/C++:
Carry out after the SQL, database return results collection is mapped to the data type of DataSet according to the data type of field corresponding in the database, is example with ORACLE:
1) DataSet ds; // definition result set makes up the result set field
column_desc*desc＝this->describe_select(desc_len)；
for(int?i＝0；i<desc_len；++i){
if(desc[i].dbtype＝2)//number{
if(desc[i].scale>0){
ds.addField(Field(name.c_str()，FTDOUBLE，8))；}
else{
if(desc[i].prec>9){
ds.addField(Field(name.c_str()，FTSTRING，desc[i].prec))；}
else
ds.addField(Field(name.c_str()，FTINT，4))；}}
else?if(desc[i].dbtype＝1)//varchar2{
ds.addField(Field(name.c_str()，FTSTRING，desc[i].dbsize))；}
else?if(desc[i].dbtype＝12)//date{
ds.addField(Field(name.c_str()，FTSTRING，23))；}
else?if(desc[i].dbtype＝96)//char{
ds.addField(Field(name.c_str()，FTSTRING，desc[i].dbsize))；}
Else....
2) obtain data to DataSet (ds)
while(！this->eof()){
DataSet::iterator?it＝ds.pushBack()；
for(size_t?i＝0；i<ds.fieldCount()；++i){
(*this)>>v；
if((*it)[i].getDataType()＝＝FTSTRING){
(*it)[i]＝v；}
else?if((*it)[i].getDataType()＝＝FTINT){
(*it)[i]＝(int)atol(v.c_str())；}
Else?if((*it)[i].getDataType()＝＝FTDOUBLE){
(*it)[i]＝atof(v.c_str())；}
Else….}}
Realize the concrete grammar that heat is disposed with C/C++:
Each SQL has corresponding version number, generally is version number with the timestamp.
Sql.mmdd.xxxx for example
The SQL text title of Sql--execution
Mmdd.xxxx--SQL version number (mmdd days, xxxx millisecond)
There is one to upgrade tabulation in addition, writes down the version number of present each SQL in service of system, obtain SQL execution version according to upgrading tabulation when program is carried out.
DATA CENTER backup and restoration methods: mainly utilize the mode of IPC under the UNIX (Inter-Process Communication) technology in conjunction with written document.Process is carried out the backup of dump () method or the parameter timed backup is set after receiving specific message, after operation exception appears in DataCenter, from up-to-date backup file, recover (, just not carrying out recovery during startup) when start next time if configuration parameter is manual recovery.
In embodiment, the course of work of this assembly is described in conjunction with example: just taking the background process of writing off is example, and key step has:
1) take the family basic document from database read, the parameter of writing off is waited until DATA CENTER (real data is saved among the DataSet); Referring to unifying method and the hot method of disposing of SQL that the result returns.
2) from file, read the subscriber arrearage bill to DATA CENTER (real data is saved among the DataSet)
DataCenter?dc；dc.open()；
DataSet*ds＝new?DataSet()；
Ds->addfield(Field(“f1”，FTSTRING，10))；
Ds->addfield(Field(“f2”，FTINT))；
Ds->addfield(Field(“f3”，FTDOUBLE))；
Ds->addfield(Field(“f4”，FTSTRING，20))；
ds->loadFromFile(“../data/f001.dat”)；
Set up access index
ds->buildIndex(“indexname”，”filedName”，SORT_TYPE?type)；
dc.commit(“dsname”，ds)；
3) each user of circulating gets the arrearage bill, calls the algorithm of writing off, the data among the final updating DataSet.
DataSet&ds＝dc.getDataSet(“name”)；
pair<IndexIterator，IndexIterator>par；
Bool locate (par, 0, " f1 ", " value "); // according to search index
Calc (); // call professional algorithm
// new data more
for(DataSet::IndexIteratoriit＝par.first；iit！＝par.second；++iit){ (*iit)[0]＝“abc”；
(*iit)[1]＝1234；
(*iit)[2]＝12.34。
Claims (3)
1. the data processing method of a main memory data center: form by general Sybase access interface platform UDBIFRAMEWORK and the MMDC of main memory data center; Database by UDBI FRAMEWORK unified database access interface UDBI and MMDC data interaction, the MMDC of main memory data center assembly is realized by the C/C++ language; It is characterized in that adopting the unified database access interface is that the UDBI assembly is that the mainstream data bank interface is unified abstract package: when system carries out SQL for the first time, and UDBI meeting buffer memory SQL text and version number thereof; Revise if desired, increase SQL, by the SQL version of PATCH new mechanism correspondence; SQL carries out the data set that returns and accepts data by data center's cells D ATASET assembly;
By one or more data capsule DataSet with preserve the MMDC that the structural information MMDC head of dataset forms; The pattern of DATASET has uses the LOCAL pattern in the individual process: or the SHARE pattern of sharing between process:
DataSet sets up in the following way: the order of the store data by field of definition, deleted data, insertion data is set up; When inserting data at every turn, capacity according to present DATASET judges whether to surpass the internal memory that has distributed, if surpass, the continuous internal memory of just reallocating, the size of piece is specified (being defaulted as 4k) when making up DATASET, according to data volume what rationally the size of assignment of allocation piece can improve Memory Allocation efficient;
According to data capsule data address hash value, B* tree and B+ tree index: set up index and deletion index:
bool?buildIndex(“indexname”，”filedName”，SORT_TYPE?type)；
bool?dropIndex(const?char*name)；
DATA CENTER backup, recovery: UNIX send message by the SH script down, and MMDC carries out by carrying out Dump (), Recover (), and after the program accident withdrawed from, recovery was carried out when start next time.
2. the data processing method of main memory according to claim 1 data center is characterized in that DataSet can set up in the following way:
DataSet?ds：
ds.addfield(Field(“f1”，FTSTRING，10))；
ds.addfield(Field(“f2”，FTINT))；
ds.addfield(Field(“f3”，FTDOUBLE))；
ds.addfield(Field(“f4”，FTSTRING，20))；
DataSet::iterator it=ds.pushBack (); // store data:
Ds.erase (); // deleted data:
Ds.insert () // insertion data:
Each when inserting data, judge whether to surpass the internal memory that has distributed according to the capacity of present DATASET, if surpass, the continuous internal memory of just reallocating, the size of piece is specified when making up DATASET, is defaulted as 4k.
3. the data processing method of main memory according to claim 1 data center is characterized in that DATASET sets up index according to container data address hash value, and B* tree and B+ set index and set up:
ds.buildIndex(“indexname”，”filedName”，SORT_TYPE?type)；
Ds.buildIndex (" indexnamel ", 2, " field1 ", field2 "): // composite index built.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB2007101309065A CN100520781C (en) | 2007-08-23 | 2007-08-23 | Main memory data centre data processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB2007101309065A CN100520781C (en) | 2007-08-23 | 2007-08-23 | Main memory data centre data processing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101136029A true CN101136029A (en) | 2008-03-05 |
| CN100520781C CN100520781C (en) | 2009-07-29 |
Family
ID=39160128
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNB2007101309065A Expired - Fee Related CN100520781C (en) | 2007-08-23 | 2007-08-23 | Main memory data centre data processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN100520781C (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101567006B (en) * | 2009-05-25 | 2012-07-04 | 中兴通讯股份有限公司 | Database system and distributed SQL statement execution plan reuse method |
| CN102693170A (en) * | 2011-03-21 | 2012-09-26 | 北京千橡网景科技发展有限公司 | Method and device for memory management |
| CN103513956A (en) * | 2012-06-26 | 2014-01-15 | 阿里巴巴集团控股有限公司 | Data processing method and device of processor |
| CN112069091A (en) * | 2020-08-17 | 2020-12-11 | 北京科技大学 | Access optimization method and device applied to molecular dynamics simulation software |
-
2007
- 2007-08-23 CN CNB2007101309065A patent/CN100520781C/en not_active Expired - Fee Related
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101567006B (en) * | 2009-05-25 | 2012-07-04 | 中兴通讯股份有限公司 | Database system and distributed SQL statement execution plan reuse method |
| CN102693170A (en) * | 2011-03-21 | 2012-09-26 | 北京千橡网景科技发展有限公司 | Method and device for memory management |
| CN103513956A (en) * | 2012-06-26 | 2014-01-15 | 阿里巴巴集团控股有限公司 | Data processing method and device of processor |
| CN103513956B (en) * | 2012-06-26 | 2016-04-06 | 阿里巴巴集团控股有限公司 | A kind of method of processor processes data and device |
| CN112069091A (en) * | 2020-08-17 | 2020-12-11 | 北京科技大学 | Access optimization method and device applied to molecular dynamics simulation software |
| CN112069091B (en) * | 2020-08-17 | 2023-09-01 | 北京科技大学 | Memory access optimization method and device applied to molecular dynamics simulation software |
Also Published As
| Publication number | Publication date |
|---|---|
| CN100520781C (en) | 2009-07-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112534396B (en) | Diary watch in database system | |
| EP4155966A1 (en) | Blockchain data indexing method, and blockchain data storage method | |
| US5519855A (en) | Summary catalogs | |
| CN101046821B (en) | Generic database manipulator | |
| US9672017B2 (en) | Object storage and synchronization hooks for occasionally-connected devices | |
| CN100437569C (en) | Method and apparatus for data processing | |
| US9122738B2 (en) | Selection of rows and values from indexes with updates | |
| US7526513B2 (en) | Multi-level replication and conflict resolution | |
| US7010617B2 (en) | Cluster configuration repository | |
| AU764720B2 (en) | Method and system for fast memory-resident processing of transaction data | |
| US6463439B1 (en) | System for accessing database tables mapped into memory for high performance data retrieval | |
| KR100556594B1 (en) | A method relating to databases | |
| CN102012933B (en) | Distributed file system and method for storing data and providing services by utilizing same | |
| CN102638584B (en) | Data distributing and caching method and data distributing and caching system | |
| CN101799826B (en) | Networking data sharing system and method based on virtual view | |
| CN106599043A (en) | Middleware used for multilevel database and multilevel database system | |
| US20090083341A1 (en) | Ensuring that the archival data deleted in relational source table is already stored in relational target table | |
| CN101080715B (en) | System and method for managing binary large objects | |
| US20120150797A1 (en) | Method and system for safely transporting legacy data to an object semantic form data grid | |
| CN104516967A (en) | Electric power system mass data management system and use method thereof | |
| JP2007526564A (en) | Time-addressed database management system | |
| JPWO2007083371A1 (en) | Data integration apparatus, data integration method, and computer-readable recording medium recording data integration program | |
| JP2005527912A (en) | High-performance change capture for data warehousing | |
| CN100520781C (en) | Main memory data centre data processing method | |
| EP2479675A1 (en) | Database management device using key-value store with attributes, and key-value-store structure caching-device therefor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C56 | Change in the name or address of the patentee |
Owner name: NANJING LIANCHUANG TECHNOLOGY (GROUP) STOCK CO., L Free format text: FORMER NAME: NANJING LIANCHUANG TECHNOLOGY STOCK CO., LTD. |
|
| CP01 | Change in the name or title of a patent holder |
Address after: 210013, No. 12, Huai gate, 16 floor, Nanjing, Jiangsu Patentee after: LINKAGE TECHNOLOGY GROUP Co.,Ltd. Address before: 210013, No. 12, Huai gate, 16 floor, Nanjing, Jiangsu Patentee before: LINKAGE TECHNOLOGY GROUP CO.,LTD. |
|
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20090729 Termination date: 20210823 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |