A kind of distributed NewSQL Database Systems and Query semi-structured for data method
Technical field
The present invention relates to big data technical field, more particularly to a kind of distributed NewSQL Database Systems and semi-structured
Data query method.
Background technology
Hbase units are one of foremost distributed NoSQL databases in Hadoop ecosystems at present.Hbase is mono-
First primary clustering includes HMaster and HRegionsever, provides the user the data model of form types, is drawn by major key scope
It is divided into multiple region, HMaster is responsible for and distributed region, and HRegionserver is responsible for the read-write of region data.
The data of existing Hbase units storage do not have point of data type, are byte arrays, therefore such as this to storage JSON
Semi-structured data can there are problems that in query aspects.JSON formatted datas are stored in Hbase units, then conventional meeting
Whole JSON objects are stored as character string.Following defect be present in which:
Want when filter record, it is necessary to which all records are all read out and then filtered in client, in number
According to measure it is larger in the case of the performance can not be received.
Will more new record when, it is necessary to record is read out be updated again for specific field after re-write
Hbase units are covered.
The content of the invention
The purpose of the embodiment of the present invention is to provide a kind of distributed NewSQL Database Systems and data query method, can be real
The data query of existing JSON forms, effect and performance are bad when solving the problems, such as to handle semi-structured data.
To achieve the above object, the embodiments of the invention provide a kind of distributed NewSQL databases, including:
Control unit, in a manner of database interface accessing user ask, and by the user request be sent to meter
Draw unit;It is additionally operable to Query Result returning to user;Wherein, user's request includes the inquiry for the JSON data that needs are inquired about
Condition, the Query Result are the JSON data that are obtained according to the querying condition;
Planning unit, for parsing user's request, executive plan corresponding to compiling and customization;
Execution unit, for according to executive plan, starting collaboration processing module and obtaining with being looked into described in user request
The corresponding index data of inquiry condition;And tables of data is inquired about according to the index data of acquisition, it is corresponding described so as to obtain
Query Result;And the Query Result is returned to described control unit;
Hbase units, for storing the tables of data and concordance list, wherein, the bottom increase JSON types of Hbase units
Data, the JSON data are stored entirely in bottom HFile;
The Hbase units also include the collaboration processing module, and the collaboration processing module is used for according to the inquiry
Condition query concordance list, obtain the corresponding index data;Wherein, stored in the concordance list by the JSON data
The index data for the inverted index form that the type nested as one is generated.
Compared with prior art, a kind of distributed NewSQL Database Systems disclosed by the invention, it is single by controlling first
Member accessing user in a manner of database interface is asked, and user's request is sent into planning unit;Then planning unit is passed through
Parse user's request, executive plan corresponding to compiling and generation;Then, association is started according to executive plan by execution unit
Index number corresponding with the querying condition of user request in the concordance list of Hbase units is obtained with processing module
According to;And the tables of data of Hbase units is inquired about according to the index data of acquisition, so as to obtain the corresponding Query Result,
Obtain JSON data;And the Query Result is returned to described control unit, finally by the skill of control unit return user
Art scheme, the data query of JSON forms can be realized, effect and performance are bad when solving the problems, such as to handle semi-structured data.
Further, the distributed NewSQL Database Systems also include:Distributed transaction management device, for when described
When being related to distributed transaction in executive plan, coordinate the multi-party completion distributed transaction management in the executive plan.
Further, the Hbase units also include Hbase unit api interfaces, and the execution unit is used for according to acquisition
The index data tables of data is inquired about by the Hbase units api interface, so as to the Query Result corresponding to obtaining.
Further, the database interface is JDBC or ODBC.
The embodiment of the invention also discloses a kind of Query semi-structured for data method, based on described in the embodiments of the present invention
Distributed NewSQL Database Systems, including:
Control unit, accessing user is asked in a manner of database interface, and user request is sent into plan
Unit;Wherein, user's request includes the querying condition for the JSON data that needs are inquired about, and the Query Result is according to
The JSON data that querying condition is obtained;
The user is parsed by planning unit to ask, executive plan corresponding to compiling and customization;
By execution unit according to executive plan, start querying condition inquiry described in the collaboration processing module of Hbase units
Concordance list, obtain the corresponding index data;Wherein, stored in the concordance list by the JSON data as one
The index data for the inverted index form that nested type is generated;The concordance list is stored in the Hbase units;
Tables of data is inquired about according to the index data of acquisition by the execution unit, looked into so as to described corresponding to obtaining
Ask result;And Query Result is returned to described control unit;Wherein, the tables of data is stored in the Hbase units;
The bottom increase JSON categorical datas of Hbase units, the JSON data are stored entirely in bottom HFile;
The Query Result is returned to by user by described control unit.
Compared with prior art, a kind of Query semi-structured for data method disclosed by the invention, passes through control unit first
Accessing user is asked in a manner of database interface, and user's request is sent into planning unit;Then planning unit solution is passed through
Analyse user's request, executive plan corresponding to compiling and generation;Then, collaboration is started according to executive plan by execution unit
The index data corresponding with the querying condition of user request in the concordance list of processing module acquisition Hbase units;
And the tables of data of Hbase units is inquired about according to the index data of acquisition, so as to obtain the corresponding Query Result, that is, obtain
Obtain JSON data;And the Query Result is returned to described control unit, finally by the technical side of control unit return user
Case, the data query of JSON forms can be realized, effect and performance are bad when solving the problems, such as to handle semi-structured data.
Further, by distributed transaction management device when being related to distributed transaction in the executive plan, institute is coordinated
State the multi-party completion distributed transaction management in executive plan.
Further, the Hbase unit APs I of the Hbase units is passed through during the execution unit inquiry tables of data
Interface polls tables of data, so as to obtain corresponding Query Result;
Further, the database interface is JDBC or ODBC.
Brief description of the drawings
Fig. 1 is a kind of structural representation for distributed NewSQL databases that the embodiment of the present invention 1 provides;
Fig. 2 is a kind of schematic flow sheet for Query semi-structured for data method that the embodiment of the present invention 2 provides;
Fig. 3 is to generate to perform meter in a kind of step S2 for Query semi-structured for data method that the embodiment of the present invention 2 provides
The schematic flow sheet drawn.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation describes, it is clear that described embodiment is only part of the embodiment of the present invention, rather than whole embodiments.It is based on
Embodiment in the present invention, those of ordinary skill in the art are obtained every other under the premise of creative work is not made
Embodiment, belong to the scope of protection of the invention.
Referring to Fig. 1, Fig. 1 is a kind of structural representation for distributed NewSQL Database Systems that the embodiment of the present invention 1 provides
Figure;The present embodiment 1 includes following structures:
Control unit 1, in a manner of database interface accessing user ask, and by the user request be sent to meter
Draw unit 2;It is additionally operable to Query Result returning to user;Wherein, user's request includes looking into for the JSON data that needs are inquired about
Inquiry condition, the Query Result are the JSON data that are obtained according to the querying condition;
Planning unit 2, for parsing user's request, executive plan corresponding to compiling and customization;
Execution unit 3, for obtaining described with user request according to executive plan, startup collaboration processing module 41
The corresponding index data of querying condition;And tables of data is inquired about according to the index data of acquisition, so as to obtain corresponding institute
State Query Result;And the Query Result is returned to described control unit 1;
Hbase units 4, for storing the tables of data and concordance list, wherein, the bottom increase JSON classes of Hbase units 4
Type data, the JSON data are stored entirely in bottom HFile;
The Hbase units 4 also include the collaboration processing module 41, and the collaboration processing module 41 is used for according to
Querying condition search index table, obtain the corresponding index data;Wherein, stored in the concordance list by the JSON
The index data for the inverted index form that the data type nested as one is generated.
The present embodiment has increased JSON categorical datas newly in the bottom of Hbase units 4, and in bottom HFile, JSON data are whole
Body is stored, and JSON index column is also served as into a nested type when building secondary index and is indexed, therefore energy
The arbitrary fields inquiry for JSON is supported, index is created and revises.
Further, the distributed NewSQL Database Systems also include:Distributed transaction management device, for when execution
When being related to affairs in the works, coordinate the multi-party completion distributed transaction management in executive plan.Distributed transaction management device utilizes
Java issued transactions API (JTA) realizes distributing real time system and transaction management;Wherein, JTA, i.e. Java Transaction
API, JTA allow application program perform distributing real time system --- on two or more network computer resources access and
Update the data.
Specifically, after user's request of the planning unit 2 for receiving control unit 1, parsing user's request, and pass through height
Fast SQL engines compile SQL, then regenerate executive plan.In addition, execution unit 2 returns to after being additionally operable to executive plan generation
Control unit 1.And control unit 1 is additionally operable to judge whether needs according to the content of executive plan after executive plan is received
The intervention of distributed transaction management device, if it is desired, then start distributed transaction management device.
Further, the Hbase units 4 also include Hbase unit api interfaces, and the execution unit 3 is used for basis and obtained
The index data taken inquires about tables of data by the Hbase units api interface, so as to obtain the corresponding inquiry knot
Fruit.
Further, the database interface is JDBC or ODBC.
Further, control unit 1 is also connected with a monitor, for being responsible for metadata management and for monitoring bottom
Hbase Region load, avoids specific region load too high, and using cooperateing with processing module 41 to redistribute
Region。
In addition, control unit 1 is additionally operable to coordinate data communication, the management overall flow between multiple roles.
Wherein, planning unit 2 is used for the process for generating executive plan, specifically includes:
Judge to whether there is the prestore SQL statement corresponding with SQL statement in common buffer pool, if so, then output and SQL
Executive plan corresponding to sentence, if it is not, then
Syntax check is carried out to SQL statement, if syntax error returns to error message to user, otherwise,
Semantic test is carried out to SQL statement, if semantic error returns to error message to user, otherwise,
View and expression formula conversion, conversion results corresponding to acquisition are carried out to SQL statement;
Optimizer, optimizer selection result corresponding to acquisition are selected according to transformation result;
According to data connection approach and the order of connection corresponding to the selection of optimizer selection result;
According to connected mode and the path of order of connection selection search;
Executive plan is generated according to searching route, and exports executive plan.
When it is implemented, control unit 1, accessing user asks in a manner of database interface first, and please by user
Ask and be sent to planning unit 2;Then parse user by planning unit 2 to ask, executive plan corresponding to compiling and generation;Connect
, judged whether according to the content of executive plan to need the intervention of distributed transaction management device by control unit 1, if needed
Will, then start distributed transaction management device, the multi-party completion coordinated by distributed transaction management device in executive plan is distributed
Transaction management;Then, the rope that collaboration processing module 41 obtains Hbase units 4 is started according to executive plan by execution unit 3
Draw index data corresponding with the querying condition of user request in table;And according to the index number of acquisition it is investigated that
The tables of data of Hbase units 4 is ask, so as to obtain the corresponding Query Result, that is, obtains JSON data;And return to the inquiry
As a result to described control unit 1, user is returned finally by control unit 1.
The distributed NewSQL Database Systems of the present embodiment can realize the data query of JSON forms, solve processing half hitch
The problem of structure data age fruit and bad performance.
Referring to Fig. 2, Fig. 2 is a kind of schematic flow sheet for Query semi-structured for data method that the embodiment of the present invention 2 provides;
The distributed NewSQL Database Systems provided based on the embodiment of the present invention 1, the present embodiment 2 are comprised the steps:
S1, control unit 1, accessing user is asked in a manner of database interface, and user request is sent to
Planning unit 2;Wherein, user's request includes the querying condition for the JSON data that needs are inquired about, and the Query Result is root
The JSON data obtained according to the querying condition;
S2, the user is parsed by planning unit 2 asked, executive plan corresponding to compiling and customization;
S3, by execution unit 3 according to executive plan, inquire about bar described in the collaboration processing module 41 that starts Hbase units 4
Part search index table, obtain the corresponding index data;Wherein, stored in the concordance list and made by the JSON data
The index data of the inverted index form generated by a nested type;The concordance list is stored in the Hbase units 4
In;
S4, by the execution unit 3 according to the index data of acquisition inquire about tables of data, so as to obtain corresponding institute
State Query Result;And Query Result is returned to described control unit 1;Wherein, the tables of data is stored in the Hbase units
4;The bottom increase JSON categorical datas of Hbase units 4, the JSON data are stored entirely in bottom HFile;
S5, by described control unit 1 by the Query Result return user.
The present embodiment has increased JSON categorical datas newly in the bottom of Hbase units 4, and in bottom HFile, JSON data are whole
Body is stored, and JSON index column is also served as into a nested type when building secondary index and is indexed, therefore energy
The arbitrary fields inquiry for JSON is supported, index is created and revises.
Further, the present embodiment step S2 is completed after generating executive plan, in addition to executive plan is returned into control
Unit 1, by control unit 1 after executive plan is received, it is additionally operable to judge whether to need to be distributed according to the content of executive plan
The intervention of formula task manager, if it is desired, then start distributed transaction management device, specifically, passing through distributed transaction management
Device coordinates the multi-party completion distributed transaction management in executive plan when being related to affairs in executive plan;If it is not required, then
Directly perform step S3.
Further, the Hbase units of the Hbase units 4 are passed through when the execution unit 3 inquires about the tables of data
4API interface polls tables of data, so as to obtain corresponding Query Result;
Further, the database interface is JDBC or ODBC.
Wherein, referring to Fig. 3, Fig. 3 is to be used for the schematic flow sheet that generates executive plan by planning unit 2 in step S2,
Specifically include:
S201, judge to whether there is the prestore SQL statement corresponding with SQL statement in common buffer pool, if so, then exporting
Executive plan corresponding with SQL statement, if it is not, then
S202, syntax check is carried out to SQL statement, if syntax error returns to error message to user, otherwise,
S203, semantic test is carried out to SQL statement, if semantic error returns to error message to user, otherwise,
S204, view and expression formula conversion, conversion results corresponding to acquisition are carried out to SQL statement;
S205, according to transformation result select optimizer, optimizer selection result corresponding to acquisition;
S206, data connection approach and the order of connection according to corresponding to the selection of optimizer selection result;
S207, the path for selecting to search for according to connected mode and the order of connection;
S208, executive plan generated according to searching route, and export executive plan.
When it is implemented, control unit 1, accessing user asks in a manner of database interface first, and please by user
Ask and be sent to planning unit 2;Then parse user by planning unit 2 to ask, executive plan corresponding to compiling and generation;Connect
, judged whether according to the content of executive plan to need the intervention of distributed transaction management device by control unit 1, if needed
Will, then start distributed transaction management device, the multi-party completion coordinated by distributed transaction management device in executive plan is distributed
Transaction management;Then, the rope that collaboration processing module 41 obtains Hbase units 4 is started according to executive plan by execution unit 3
Draw index data corresponding with the querying condition of user request in table;And according to the index number of acquisition it is investigated that
The tables of data of Hbase units 4 is ask, so as to obtain the corresponding Query Result, that is, obtains JSON data;And return to the inquiry
As a result to described control unit 1, user is returned finally by control unit 1.
The distributed NewSQL Database Systems of the present embodiment can realize the data query of JSON forms, solve processing half hitch
The problem of structure data age fruit and bad performance.
Described above is the preferred embodiment of the present invention, it is noted that for those skilled in the art
For, under the premise without departing from the principles of the invention, some improvements and modifications can also be made, these improvements and modifications are also considered as
Protection scope of the present invention.