Summary of the invention
For addressing the above problem, a kind of text searching method that uses bitmap index that technical scheme of the present invention provides is applicable to the XML data base management system (DBMS), comprises the steps:
Analyzing XML file obtains all elements node of said XML document;
The all elements node of said XML document is stored in the node table;
Take out all text child nodes of said node element, and all text child nodes are split into several words, constitute a set;
Stop words is carried out in said set filtered, obtain effective set of letters;
Make up the bitmap index table;
Utilize said bitmap index table to carry out full-text search.
Alternatively, said structure bitmap index table is specially: make up the bitmap of each word in said effective set of letters according to the mapping relations between the data line of each word in said effective set of letters and said node table, form said bitmap index table.
Alternatively, utilize following formula to make up the bitmap of each word in said effective set of letters according to the mapping relations between the data line of each word in said effective set of letters and said node table:
block=i/M;
offset=i%M;
Wherein, TID is the address of the data line in the node table, expression become (block, offset), M is a storable number of data lines purpose maximal value in the data page, i representes the position in the bitmap of each word.
Alternatively; Further comprise; Be stored in the process in the node table at said all elements node, also further comprise, be each data line index building key assignments of said bitmap index table said XML document; Be specially: said bitmap index table has a call back function, and said call back function is responsible for each data line index building key assignments of said bitmap index table;
Said call back function is:
IndexKeyRange?IndexBuilder(HeapTuple?htup,Relation?heap,Relation?index);
Wherein, said htup is the data line of said node table, and heap and index are the handle objects of said node table and said bitmap index table, and rreturn value is the object of IndexKeyRange type.
Alternatively, further comprise: the bitmap to each word in effective set of letters of said structure forms is carried out the RLE compression.
Alternatively, the described method of utilizing said bitmap index to carry out full-text search is specially:
When xquery fulltext inquires about, wherein querying condition character string is split into plurality of words;
Use the inactive vocabulary of acquiescence to filter above-mentioned plurality of words, obtain effective looking up words set;
The word that uses the bitmap index table to inquire about successively in effective set of letters obtains several bitmaps;
Described some bitmaps are carried out the target logic computing that meets querying condition obtain final bitmap, also promptly finally met the capable TID set of node data of full-text search condition.
Alternatively, said analyzing XML file is specially with all elements node that obtains said XML document: the XML document resolver of said XML document being sent into the SAX pattern is resolved, to obtain all elements node of said XML document.
The present invention also provides a kind of full-text search device that uses bitmap index, comprising:
The document resolution unit is used for analyzing XML file, obtains all elements node of said XML document;
The node storage unit is used for all elements node of said XML document is stored to a node table;
The text node split cells, all text child nodes that are used to take out said node element, and said text child node split into several words, constitute a set;
Filter element is used for that stop words is carried out in said set and filters, and obtains effective set of letters;
The concordance list construction unit is used to make up the bitmap index table;
Retrieval unit is used to utilize said bitmap index table to carry out full-text search.
Alternatively; Described concordance list construction unit is specially: make up the bitmap of each word in said effective set of letters according to the mapping relations between the data line of each word in said effective set of letters and said node table, form said bitmap index table.
Alternatively, the following formula of described index construct unit by using makes up the bitmap of each word in said effective set of letters according to the mapping relations between the data line of each word in said effective set of letters and said node table:
block=i/M;
offset=i%M;
Wherein, TID is the address of the data line in the node table, expression become (block, offset), M is a storable number of data lines purpose maximal value in the data page, i representes the position in the bitmap of each word.
Alternatively, said bitmap index table has a call back function, and said call back function is responsible for each data line index building key assignments of said bitmap index table;
Said call back function is:
IndexKeyRange?IndexBuilder(HeapTuple?htup,Relation?heap,Relation?index);
Wherein, said htup is the data line of said node table, and heap and index are the handle objects of said node table and said bitmap index table, and rreturn value is the object of IndexKeyRange type.
Alternatively, further comprise a compression unit, described compression unit is used for the bitmap of each word formation of effective set of letters of said structure is carried out the RLE compression.
Alternatively, described retrieval unit further comprises:
The inquiry string split cells is used for when xquery fulltext inquires about, and wherein querying condition character string is split into plurality of words;
The stop words filter element is used to use the inactive vocabulary of acquiescence to filter above-mentioned plurality of words, obtains effective looking up words set;
The bitmap query unit, the word that is used for using the bitmap index table to inquire about effective set of letters successively obtains several bitmaps;
The target bitmap acquiring unit is used for that described some bitmaps are carried out the target logic computing that meets querying condition and obtains final bitmap, is also promptly finally met the capable TID set of node data of full-text search condition.
Alternatively, described document resolution unit is used for said XML document is sent into the XML document resolver of SAX pattern resolves, to obtain all elements node of said XML document.
Compared with prior art, technique scheme has following advantage:
Technical scheme of the present invention makes the XML Data access module in the existing XML data base management system (DBMS) can carry out the existence test of word apace; Because the target of full-text search is exactly to find the node data of some word that contains user's appointment capable; So the test of this existence is most crucial and one of most important function in the full-text search; Compare with traditional B+ tree index, the text searching method of use bitmap index of the present invention can be realized full-text index more efficiently, simultaneously; Owing to used bitmap index, also significantly improved the performance of full-text search to a great extent.
Embodiment
For make above-mentioned purpose of the present invention, feature and advantage can be more obviously understandable, does detailed explanation below in conjunction with the accompanying drawing specific embodiments of the invention.Set forth detail in the following description so that make much of the present invention.But the present invention can be different from alternate manner described here and implements with multiple, and those skilled in the art can do similar popularization under the situation of intension of the present invention.Therefore the present invention does not receive the restriction of following disclosed embodiment.
The entity that those skilled in the art will appreciate that storing X ML document in the XML data base management system (DBMS) is called container, the data content of any a plurality of XML documents of storage in container; And this container is supported by several tables of data, stores the data and the structural information of the various aspects of these XML documents respectively, comprises node data; Relationships between nodes; Path data, various index, various statistic etc.The unit of data table stores is a data line, contains several data lines in the tables of data, and can find specific data line fast through index.The position of each row in the tables of data uses a TID to identify, and a TID identifies a data line uniquely and contains the address of this data line in tables of data, and we can find data line with TID like this.
Bitmap is exactly a bit sequence in fact, and these modes with integer are organized, and is placed on usually such as one 128 bitmaps in the array of 4 32 integers compositions, perhaps is placed in the array of 2 64 integers compositions.Each of bitmap is 0 or is 1 can represent the true and false of a fact.Can contain any a plurality of bitmap in the bitmap index.This shows that bitmap index can be expressed the existence fact of arbitrary number well.
For solving the problems of the prior art, inventor of the present invention has proposed to use the text searching method of bitmap index through research, applicable to the XML data base management system (DBMS).Consult Fig. 1, Fig. 1 is the process flow diagram of text searching method of the use bitmap index of embodiment of the present invention.The text searching method of the use bitmap index of embodiment of the present invention comprises:
At first, get into step S1: analyzing XML file, and obtain all elements node of said XML document;
Wherein, Preferably; Analyzing XML file in the embodiment of the present invention; The all elements node that obtains said XML document is specially: XML document is sent in the XML document resolver of SAX pattern, the XML document resolver can let the query engine in the data base management system (DBMS) obtain all elements node data of XML document through the form of event notice.
Get into step S2: all elements node of said XML document is stored in the node table;
Wherein, Mechanism in execution of this step and the XML data base management system (DBMS) of the prior art in like manner; The node element of query engine in the XML data base management system (DBMS) under text node resolved the SAX event notice that can obtain the XML resolver finish time; The method of query engine treatment S AX incident is to call the data line storage means of storage engines, said node element data storage in node table, thereby accomplish the execution of this step.
In addition; Of the present invention one preferred embodiment in, be stored in the process in the node table at said all elements node said XML document, also further comprise; Data line is being stored in the process of said node table; Be each data line index building key assignments of said bitmap index table, be specially: said bitmap index table has a call back function, and said call back function is responsible for each data line index building key assignments of said bitmap index table;
Said call back function is:
IndexKeyRange?IndexBuilder(HeapTuple?htup,Relation?heap,Relation?index);
Wherein, Said htup is the data line of said node table; Heap and index are the handle objects of said node table and said bitmap index table, and rreturn value is the object of IndexKeyRange type, and this object contains start offset and the length of index key in the htup data line.We just can take out the index key value part among the htup as the index key assignments to have obtained IndexKeyRange.Like this; A data line is being stored in the process of said node table; Through calling the IndexBuilder function of all bitmap index tables successively, be followed successively by each bitmap index table index building key assignments and index key assignments (TID) as capable storing in the bitmap index table of index data.
Get into step S3: take out all text child nodes of said node element and all text child nodes are split into several words, constitute a set;
Get into step S4: stop words is carried out in said set filtered, obtain effective set of letters;
Wherein, in last step, contain the stop words that when inquiry is perhaps retrieved, does not have meaning in the word of the text child node that splits out; Therefore, in order to improve effectiveness of retrieval, need to get into this step; Stop words is filtered, thereby obtain effective set of letters.
Get into step S5: make up the bitmap index table;
Wherein, preferably, in embodiments of the present invention, make up the bitmap of each word in said effective set of letters, form a bitmap index table according to the mapping relations between the data line of each word in said effective set of letters and said node table;
Wherein, utilize following formula to make up the bitmap of each word in said effective set of letters according to the mapping relations between the data line of each word in said effective set of letters and said node table:
block=i/M;
offset=i%M;
Wherein, TID is the address of the data line in the node table, expression become (block, offset), M is a storable number of data lines purpose maximal value in the data page, i representes the position in the bitmap of each word.
As far as each word in the efficient set, because these words are present in the corresponding data line of node table, therefore, the i position of each word in the efficient set can be set to 1 (i=TID.block*M+TID.offset).Through calculating above-mentioned two variate-value block and offset, just accomplished the target that i position in the bitmap of each word is mapped to a data line of a node table.
It is pointed out that when a node data content is updated, for performance considers not remove those original capable bitmap indexs of this node data that points to.Like this, when we obtain a node, also need further to do participle therein, and search target word.But owing to avoided searching target word in a large amount of node in the node table of a container, it is capable to obtain all node datas that contain target word with lower cost, thereby when having quickened the efficient of full-text search, has reduced running cost.
In addition; Since in the bitmap index table, all corresponding bitmap of each word of all documents of document container storage, and perhaps 0 node data is capable corresponding to one for each of the bitmap of each word; Wherein, 1 the expression this word be present in node data capable in; 0 this word of expression is not present in this data line.Therefore, may occur a large amount of continuous 0 and 1 in the bitmap that adopts above-mentioned mode to store.In order further to improve the performance of full-text search, carry out RLE (Run Length Encoding) compression for the bitmap of each word, like this, the bitmap that is stored in the bitmap index table is exactly through the bitmap after the RLE compression.
Get into step S6: utilize said bitmap index table to carry out full-text search.
Referring to Fig. 2, Fig. 2 shows the particular flow sheet that utilizes the bitmap index table to carry out full-text search.The idiographic flow that utilizes the bitmap index table to carry out full-text search is specially:
When xquery fulltext inquires about, wherein querying condition character string is split into plurality of words;
Use the inactive vocabulary of acquiescence to filter above-mentioned plurality of words, obtain effective looking up words set;
The word that uses the bitmap index table to inquire about successively in effective set of letters obtains several bitmaps;
Described some bitmaps are carried out the target logic computing that meets querying condition obtain final bitmap, also promptly finally met the capable TID set of node data of full-text search condition.
When utilizing above-mentioned formation bitmap index table to carry out xquery fulltext inquiry, the querying condition character string is done participle, obtain each word in the querying condition, then, after the inactive vocabulary filtration of using acquiescence, obtain effective looking up words set.Use the word in effective query set the bitmap index table to inquire about successively and obtain working hard a bitmap.According to the difference of querying condition, effectively the logical relation between the word in the looking up words set that is to say the operation relation between the bitmap of these words.These bitmap actuating logic computings are obtained final bitmap, just finally meet the capable TID set of node data of full-text search condition.Because; The position of each row in the tables of data uses a TID to identify; A TID identifies a data line uniquely and contains the address of this data line in tables of data, therefore can find corresponding data line with TID, thereby accomplishes the target of full-text search.
For example, effective looking up words set={ " English ", " teacher " }, " English " wherein and " teacher " and the logical relation between two words be with, that is to say that inquiry contains the XML node element of " English " and " teacher " two words simultaneously.So query engine just obtains from the bitmap index table " English " and " teacher " bitmap of two words; Then " English " and " teacher " bitmap of two words does AND-operation; Obtaining final bitmap, is that 1 the corresponding data line in position is exactly to contain simultaneously in this bitmap " English " and " teacher " node data of two words is capable.
Consult Fig. 3, Fig. 3 shows the full-text search device of the use bitmap index of embodiment of the present invention, comprising:
Document resolution unit 110 is used for analyzing XML file, obtains all elements node of said XML document; Wherein, described document resolution unit 110 is used for said XML document is sent into the XML document resolver of SAX pattern resolves, to obtain all elements node of said XML document.
Node storage unit 120 is used for all elements node of said XML document is stored to a node table;
Text node split cells 130, all text child nodes that are used to take out said node element, and said text child node split into several words, constitute a set;
Filter element 140 is used for that stop words is carried out in said set and filters, and obtains effective set of letters;
Concordance list construction unit 150 is used to make up the bitmap index table; Described concordance list construction unit tool 150 bodies are to utilize following formula to make up the bitmap of each word in said effective set of letters according to the mapping relations between the data line of each word in said effective set of letters and said node table:
block=i/M;
offset=i%M;
Wherein, TID is the address of the data line in the node table, expression become (block, offset), M is a storable number of data lines purpose maximal value in the data page, i representes the position in the bitmap of each word.
Said bitmap index table wherein has a call back function, and said call back function is responsible for each data line index building key assignments of said bitmap index table;
Said call back function is:
IndexKeyRange?IndexBuilder(HeapTuple?htup,Relation?heap,Relation?index);
Wherein, said htup is the data line of said node table, and heap and index are the handle objects of said node table and said bitmap index table, and rreturn value is the object of IndexKeyRange type.
In addition; Since in the bitmap index table, all corresponding bitmap of each word of all documents of document container storage, and perhaps 0 node data is capable corresponding to one for each of the bitmap of each word; Wherein, 1 the expression this word be present in node data capable in; 0 this word of expression is not present in this data line.Therefore, may occur a large amount of continuous 0 and 1 in the bitmap that adopts above-mentioned mode to store.In order further to improve the performance of full-text search; Index construct unit 150 also further comprises a compression unit; Be used for the bitmap of each word is carried out RLE (Run Length Encoding) compression, like this, the bitmap that is stored in the bitmap index table is exactly through the bitmap after the RLE compression.
Retrieval unit 160 is used to utilize said bitmap index table to carry out full-text search.Wherein, described retrieval unit further comprises:
Consult Fig. 4, the retrieval unit 160 that Fig. 4 shows among the present invention further comprises:
Inquiry string split cells 160a is used for when xquery fulltext inquires about, and wherein querying condition character string is split into plurality of words;
Stop words filter element 160b is used to use the inactive vocabulary of acquiescence to filter above-mentioned plurality of words, obtains effective looking up words set;
Bitmap query unit 160c, the word that is used for using the bitmap index table to inquire about effective set of letters successively obtains several bitmaps;
Target bitmap acquiring unit 160d is used for that described some bitmaps are carried out the target logic computing that meets querying condition and obtains final bitmap, is also promptly finally met the capable TID set of node data of full-text search condition.
In sum, technical scheme of the present invention has following advantage:
Technical scheme of the present invention makes the XML Data access module in the existing XML data base management system (DBMS) can carry out the existence test of word apace; Because the target of full-text search is exactly to find the node data of some word that contains user's appointment capable; So the test of this existence is most crucial and one of most important function in the full-text search; Compare with traditional B+ tree index, the text searching method of use bitmap index of the present invention can be realized full-text index more efficiently, simultaneously; Owing to used bitmap index, also significantly improved the performance of full-text search to a great extent.
Should be understood that described method and system can be with various forms of hardware, software, firmware, dedicated processor or their combination realization here.Especially, a part at least of the present invention comprises that the application program of programmed instruction preferably realizes.These programmed instruction positively are included in one or more program storage device and (are included but not limited to hard disk; Magnetic floppy disc, RAM, ROM; CD; ROM etc.) lining, and can be by any equipment or machine that comprises appropriate configuration, for example a kind of universal digital computer with processor, internal memory and input/output interface is carried out.It should also be understood that because the building block of some systems of describing in the accompanying drawing and treatment step are preferably realized with software so the connection between the system module (the perhaps logic flow of method step) maybe be different, this depends on programming mode of the present invention.According to guidance given here, those of ordinary skill in the related art can design these and similar embodiment of the present invention.
More than disclose many aspects of the present invention and embodiment, it will be understood by those skilled in the art that others of the present invention and embodiment.Disclosed many aspects and embodiment just are used to illustrate among the present invention, are not to be to qualification of the present invention, and real protection domain of the present invention and spirit should be as the criterion with claims.