Summary of the invention
Fundamental purpose of the present invention is the extracting method and the device that provide a kind of Webpage information, extracts the low problem of accuracy to solve info web in prior art.
To achieve these goals, according to an aspect of the embodiment of the present invention, a kind of extracting method of Webpage information is provided.
Extracting method according to Webpage information of the present invention comprises: the HTML (Hypertext Markup Language) HTML code obtaining multiple Webpage to be extracted; According to described HTML code, cluster is carried out to multiple described Webpage to be extracted, obtain multiple belonging kinds; Extract the object block element in each described belonging kinds, wherein, described object block element is the block element that the described Webpage to be extracted of difference in same described belonging kinds has; Extract the text in described object block element, obtain the text collection of described object block element; Calculate the desired value of described text collection, wherein, described desired value is for representing the difference degree of the text in described text collection; And extract described desired value and be greater than text in the described text collection of the first predetermined threshold value, obtain described Webpage information.
Further, the desired value calculating described text collection comprises: the occurrence number recording each not identical text in described text collection; According to the occurrence number of text not identical described in each, determine total occurrence number of full text in described text collection; According to occurrence number and described total occurrence number of text not identical described in each, calculate each described in the not identical frequency of occurrences of text in described text collection; And according to the described frequency of occurrences of text in described text collection not identical described in each, determine the desired value of described text collection.
Further, according to the described frequency of occurrences of text in described text collection not identical described in each, determine that the desired value of described text collection comprises: according to formula
calculate the desired value of described text collection, wherein, E
setfor the desired value of described text collection, m is the number of text not identical described in comprising in described text collection, p (text
i) be the frequency of occurrences of text in described text collection not identical described in each.
Further, be greater than the text in the described text collection of the first predetermined threshold value in the described desired value of extraction, after obtaining described Webpage information, described extracting method also comprises: the category attribute recording described text.
Further, determine the belonging kinds of the first Webpage to be extracted and second page to be extracted in the following manner, wherein, described first Webpage to be extracted and described second page to be extracted are any two Webpages to be extracted in multiple described page to be extracted: the HTML code according to described first Webpage to be extracted sets up the first tree structure, and set up the second tree structure according to the HTML code of described second Webpage to be extracted; Extract the block element comprising preset attribute in described first tree structure, obtain first piece of element, and extract the block element comprising preset attribute in described second tree structure, obtain second piece of element; According to described first piece of element and described second piece of element, calculate the similarity mean value of described first Webpage to be extracted and described second Webpage to be extracted; The size of more described similarity mean value and the second predetermined threshold value; And when comparing described similarity mean value and being greater than described second predetermined threshold value, determine that described first Webpage to be extracted and described second page to be extracted are same home classification, or when comparing described similarity mean value and being less than or equal to described second predetermined threshold value, determine that described first Webpage to be extracted and described second page to be extracted are respectively different belonging kinds.
To achieve these goals, according to the another aspect of the embodiment of the present invention, provide a kind of extraction element of Webpage information.
Extraction element according to Webpage information of the present invention comprises: acquiring unit, for obtaining the HTML (Hypertext Markup Language) HTML code of multiple Webpage to be extracted; Cluster cell, for carrying out cluster according to described HTML code to multiple described Webpage to be extracted, obtains multiple belonging kinds; First extraction unit, for extracting the object block element in each described belonging kinds, wherein, described object block element is the block element that the described Webpage to be extracted of difference in same described belonging kinds has; Second extraction unit, for extracting the text in described object block element, obtains the text collection of described object block element; First computing unit, for calculating the desired value of described text collection, wherein, described desired value is for representing the difference degree of the text in described text collection; And the 3rd extraction unit, being greater than text in the described text collection of the first predetermined threshold value for extracting described desired value, obtaining described Webpage information.
Further, described first computing unit comprises: logging modle, for recording the occurrence number of each not identical text in described text collection; First determination module, for the occurrence number according to text not identical described in each, determines total occurrence number of full text in described text collection; Computing module, for according to the occurrence number of text not identical described in each and described total occurrence number, calculate each described in the not identical frequency of occurrences of text in described text collection; And second determination module, for according to the described frequency of occurrences of text in described text collection not identical described in each, determine the desired value of described text collection.
Further, described second determination module comprises: calculating sub module, for according to formula
calculate the desired value of described text collection, wherein, E
setfor the desired value of described text collection, m is the number of text not identical described in comprising in described text collection, p (text
i) be the frequency of occurrences of text in described text collection not identical described in each.
Further, described extraction element also comprises: record cell, for being greater than the text in the described text collection of the first predetermined threshold value in the described desired value of extraction, after obtaining described Webpage information, records the category attribute of described text.
Further, described extraction element also comprises: set up unit, for setting up the first tree structure according to the HTML code of the first Webpage to be extracted, and set up the second tree structure according to the HTML code of the second Webpage to be extracted, wherein, described first Webpage to be extracted and described second page to be extracted are any two Webpages to be extracted in multiple described page to be extracted: the 4th extraction unit, for extracting the block element comprising preset attribute in described first tree structure, obtain first piece of element, and extract the block element comprising preset attribute in described second tree structure, obtain second piece of element, second computing unit, for according to described first piece of element and described second piece of element, calculates the similarity mean value of described first Webpage to be extracted and described second Webpage to be extracted, comparing unit, for the size of more described similarity mean value and the second predetermined threshold value, and processing unit, for when comparing described similarity mean value and being greater than described second predetermined threshold value, determine that described first Webpage to be extracted and described second page to be extracted are same home classification, or when comparing described similarity mean value and being less than or equal to described second predetermined threshold value, determine that described first Webpage to be extracted and described second page to be extracted are respectively different belonging kinds.
According to inventive embodiments, adopt the HTML code obtaining multiple Webpage to be extracted; According to described HTML code, cluster is carried out to multiple described Webpage to be extracted, obtain multiple belonging kinds; Extract the object block element in each described belonging kinds, wherein, described object block element is the block element that the described Webpage to be extracted of difference in same described belonging kinds has; Extract the content of text in described object block element, obtain the text collection of described object block element; Calculate the desired value of described text collection, wherein, described desired value is for representing the difference degree of the text in described text collection; And extract described desired value and be greater than text in the described text collection of the first predetermined threshold value, obtain described Webpage information.By obtaining the HTML code of multiple Webpage to be extracted, the division to multiple Webpage belonging kinds to be extracted can be realized, and then the block element jointly comprised in the difference Webpage to be extracted obtaining under same belonging kinds, the extraction to same block element Chinese version content can be realized, then can according to the comparative result of the difference degree of the content of text got and predetermined threshold value, determine whether text content is the information that the needs in Webpage to be extracted extract, solve info web in prior art and extract the low problem of accuracy, and then reach the effect improving info web extraction accuracy.
Embodiment
The present invention program is understood better in order to make those skilled in the art person, below in conjunction with the accompanying drawing in the embodiment of the present invention, technical scheme in the embodiment of the present invention is clearly and completely described, obviously, described embodiment is only the embodiment of a part of the present invention, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, should belong to the scope of protection of the invention.
It should be noted that, term " first ", " second " etc. in instructions of the present invention and claims and above-mentioned accompanying drawing are for distinguishing similar object, and need not be used for describing specific order or precedence.Should be appreciated that the data used like this can be exchanged in the appropriate case, so as embodiments of the invention described herein can with except here diagram or describe those except order implement.In addition, term " comprises " and " having " and their any distortion, intention is to cover not exclusive comprising, such as, contain those steps or unit that the process of series of steps or unit, method, system, product or equipment is not necessarily limited to clearly list, but can comprise clearly do not list or for intrinsic other step of these processes, method, product or equipment or unit.
Embodiment 1
According to the embodiment of the present invention, provide a kind of embodiment of the method that may be used for implementing the application's device embodiment, it should be noted that, can perform in the computer system of such as one group of computer executable instructions in the step shown in the process flow diagram of accompanying drawing, and, although show logical order in flow charts, in some cases, can be different from the step shown or described by order execution herein.
According to the embodiment of the present invention, provide a kind of extracting method of Webpage information.Fig. 1 is the process flow diagram of the extracting method of Webpage information according to the embodiment of the present invention, and as shown in Figure 1, the method comprises following step S102 to step S112:
S102: the HTML (Hypertext Markup Language) HTML code obtaining multiple Webpage to be extracted.Particularly, the HTML code of multiple Webpage to be extracted can be obtained simultaneously, also can the HTML code of each Webpage to be extracted of acquisition one by one successively.
S104: cluster is carried out to multiple Webpage to be extracted according to HTML code, obtain multiple belonging kinds, namely, according to the HTML code of the Webpage each to be extracted got, multiple Webpage to be extracted is classified, Webpage to be extracted similar in multiple Webpage to be extracted is classified as a classification.It should be noted that, a Webpage to be extracted can only have a belonging kinds.
S106: extract the object block element in each belonging kinds, wherein, object block element is the block element that the difference Webpage to be extracted in same belonging kinds has.Particularly, object block element can be one, also can be multiple.In embodiments of the present invention, the concrete quantity of object block element is that the quantity of the block element had according to the page to be extracted different in same belonging kinds is determined.Total block element refers to the bookmark name of this block element in same belonging kinds in the different page to be extracted, the block element that attribute is all identical, and attribute is herein class attribute or id attribute.Such as: Webpage 1, Webpage 2 and Webpage 3 belong to belonging kinds A, the block unit jointly comprised in each Webpage in Webpage 1, Webpage 2 and Webpage 3 have 3, be div [class=" menu "], div [id=" title "] and p [class=" content "] respectively, the object block element so in belonging kinds A is then 3.
S108: extract the text in object block element, obtain the text collection of object block element.Particularly, comprise multiple text in same object block element, the set of multiple text is the text collection of this object block element.If object block element is multiple, so extract the text in each object block element, obtain the text collection of each object block element.Continue to adopt above-mentioned illustrating, for object block element div [id=" title "], the text collection obtained is { " title 1 ", " title 2 ", " title 3 " }.
S110: the desired value calculating text collection, wherein, desired value is for representing the difference degree of the text in text collection, namely, calculate object block element Chinese version difference degree, difference degree is larger, illustrates that the content difference in the text in this object block element is larger.
S112: extract desired value and be greater than text in the text collection of the first predetermined threshold value, obtain Webpage information, namely only has desired value to be greater than text in the text collection of the first preset value, is only the information needing to extract in Webpage to be extracted.Particularly, the first preset value can be arranged according to demand.
In embodiments of the present invention, by obtaining the HTML code of multiple Webpage to be extracted, the division to multiple Webpage belonging kinds to be extracted can be realized, and then the block element jointly comprised in the difference Webpage to be extracted obtaining under same belonging kinds, the extraction to same block element Chinese version content can be realized, then can according to the comparative result of the difference degree of the content of text got and predetermined threshold value, determine whether text content is the information that the needs in Webpage to be extracted extract, solve info web in prior art and extract the low problem of accuracy, and then reach the effect improving info web extraction accuracy.
It should be noted that, if the quantity of object block element is multiple, need the desired value of the text collection calculating each object block element respectively, and each desired value calculated compared with the first predetermined threshold value respectively, text desired value be greater than in the text collection of the first predetermined threshold value extracts.
Particularly, the desired value of text collection can be calculated to step 1-4 by step 1-1, and step 1-1 is specific as follows to step 1-4:
Step 1-1: the occurrence number of each not identical text in recording text set.Because text collection comprises multiple text, so multiple text may exist the identical text of content, in embodiments of the present invention, the occurrence number of text in text set that content is not identical each other is only added up.
Step 1-2: according to the occurrence number of each not identical text, determines total occurrence number of full text in text collection, and particularly, in text set, total occurrence number of full text equals the occurrence number sum of all not identical texts.
Step 1-3: according to occurrence number and total occurrence number of each not identical text, calculates the frequency of occurrences of each not identical text in text collection.Such as, a text A different from other texts in text set is had in text set, the occurrence number of text A in text set is 3 times, in text set, total occurrence number of full text is 30 times, so for text A, the frequency of occurrences in above-mentioned text collection is 1/10.
Step 1-4: according to the frequency of occurrences of each not identical text in text collection, determine the desired value of text collection.
If object block element is multiple, the desired value of the text collection of so each object block element can be calculated to step 1-4 by repeated execution of steps 1-1.
Particularly, in embodiments of the present invention, according to the frequency of occurrences of each not identical text in text collection, determine that the desired value of text collection comprises: according to formula
calculate the desired value of text collection, wherein, E
setfor the desired value of text collection, m is the number comprising not identical text in text collection, p (text
i) be the frequency of occurrences of each not identical text in text collection.In embodiments of the present invention, text collection E is calculated
setthe middle logarithm by the frequency of occurrences of text not identical with this for the frequency of occurrences of each not identical text is multiplied, and by all result summations obtained, then getting negative, is exactly the desired value of text set.
Preferably, be greater than the text in the text collection of the first predetermined threshold value in extraction desired value, after obtaining Webpage information, the extracting method of the Webpage information that the embodiment of the present invention provides also comprises the category attribute of recording text.Particularly, category attribute can be title, content etc.Namely, the content of text that record extracts is title or content etc. to the embodiment of the present invention.
In embodiments of the present invention, the category attribute of text extracted by record, convenient follow-up carry out large data analysis time, user can filter out required information fast, reaches the effect improving user satisfaction.Such as, user wants to screen in the info web extracted, and content is the information of title, and so user only need select category attribute to be title, can filter out the info web meeting it and require fast.
The embodiment of the present invention additionally provides a kind of concrete mode determining the belonging kinds of the page to be extracted, for any two Webpages to be extracted that the first Webpage to be extracted and second page to be extracted are in multiple page to be extracted, clearly determine the mode of the first Webpage to be extracted and the second page belonging kinds to be extracted, particularly, the belonging kinds of the first Webpage to be extracted and second page to be extracted can be determined by step 2-1 to step 2-5:
Step 2-1: the HTML code according to the first Webpage to be extracted sets up the first tree structure, and set up the second tree structure according to the HTML code of the second Webpage to be extracted.
Step 2-2: extract the block element comprising preset attribute in the first tree structure, obtains first piece of element, and extracts the block element comprising preset attribute in the second tree structure, obtains second piece of element.Particularly, preset attribute is class attribute or id attribute, this step namely only extracts the block element comprising class attribute or id attribute in the HTML code of first page to be extracted, and only extracts the block element comprising class attribute or id attribute in the HTML code of second page to be extracted.
Step 2-3: according to first piece of element and second piece of element, calculates the similarity mean value of the first Webpage to be extracted and the second Webpage to be extracted.In embodiments of the present invention, the similarity mean value of the first Webpage to be extracted and the second Webpage to be extracted can be calculated according to formula V=1/2 (S1+S2), wherein, V is similarity mean value, S1 is the first similarity of the first Webpage to be extracted and the second Webpage to be extracted, and S2 is the second similarity of the first Webpage to be extracted and the second Webpage to be extracted.Particularly, can according to formula
calculate the first similarity S1, wherein, Kp is the block element that the first Webpage to be extracted is identical with in the second Webpage to be extracted, and p gets 1 to m successively, and m is the number of same block element, V
1Kpfor the frequency of occurrence of same block element Kp in the first Webpage to be extracted, K
0kbe first piece of element in the first Webpage to be extracted, N1 is the number of first piece of element in the first Webpage to be extracted,
be first piece of element K
0kfrequency of occurrence in the first Webpage to be extracted; According to formula
calculate the second similarity S2, wherein, V
2Kpfor the frequency of occurrence of same block element Kp in the second Webpage to be extracted, K
1kbe second piece of element in the second Webpage to be extracted, N2 is the number of second piece of element in the second Webpage to be extracted,
be second piece of element K
1kfrequency of occurrence in the second Webpage to be extracted.
Step 2-4: the size comparing similarity mean value and the second predetermined threshold value.Particularly, the second predetermined threshold value also can be arranged according to demand.
Step 2-5: when comparing similarity mean value and being greater than the second predetermined threshold value, determine that the first Webpage to be extracted and second page to be extracted are same home classification, or when comparing similarity mean value and being less than or equal to the second predetermined threshold value, determine that the first Webpage to be extracted and second page to be extracted are respectively different belonging kinds, this step namely, when judging that similarity mean value is greater than the second predetermined threshold value, the first Webpage to be extracted and second page to be extracted belong to same belonging kinds; When judging that similarity mean value is less than or equal to the second predetermined threshold value, the first Webpage to be extracted and second page to be extracted belong to different belonging kinds respectively.
In embodiments of the present invention, can by any two Webpages in multiple page to be extracted respectively as the first Webpage to be extracted and the second Webpage to be extracted, and repeated execution of steps 2-1 is to step 2-5, until determine the belonging kinds of each page to be extracted.It should be noted that, if Webpage A and Webpage B belongs to same belonging kinds, Webpage A and Webpage D also belongs to same belonging kinds, and so Webpage A, Webpage B and Webpage D all belong to same belonging kinds.After two or more Webpage to be extracted belongs to same belonging kinds, the Webpage to be extracted determining belonging kinds is needed for other, as long as a Webpage to be extracted in this Webpage to be extracted and above-mentioned belonging kinds is calculated similarity mean value, and the similarity mean value obtained and the second predetermined threshold value are compared, can determine whether this Webpage to be extracted belongs to above-mentioned belonging kinds.
It should be noted that, for aforesaid each embodiment of the method, in order to simple description, therefore it is all expressed as a series of combination of actions, but those skilled in the art should know, the present invention is not by the restriction of described sequence of movement, because according to the present invention, some step can adopt other orders or carry out simultaneously.Secondly, those skilled in the art also should know, the embodiment described in instructions all belongs to preferred embodiment, and involved action and module might not be that the present invention is necessary.
Through the above description of the embodiments, those skilled in the art can be well understood to the mode that can add required general hardware platform by software according to the method for above-described embodiment and realize, hardware can certainly be passed through, but in a lot of situation, the former is better embodiment.Based on such understanding, technical scheme of the present invention can embody with the form of software product the part that prior art contributes in essence in other words, this computer software product is stored in a storage medium (as ROM/RAM, magnetic disc, CD), comprising some instructions in order to make a station terminal equipment (can be mobile phone, computing machine, server, or the network equipment etc.) perform method described in each embodiment of the present invention.
Embodiment 2
According to the embodiment of the present invention, additionally provide a kind of extraction element of Webpage information of the extracting method for implementing above-mentioned Webpage information, this extraction element is mainly used in the extracting method that execution embodiment of the present invention foregoing provides, and does concrete introduction below to the extraction element of the Webpage information that the embodiment of the present invention provides:
Fig. 2 is the schematic diagram of the extraction element of Webpage information according to the embodiment of the present invention, as shown in Figure 2, this device mainly comprises acquiring unit 10, cluster cell 20, first extraction unit 30, second extraction unit 40, first computing unit 50 and the 3rd extraction unit 60, wherein:
Acquiring unit 10 is for obtaining the HTML (Hypertext Markup Language) HTML code of multiple Webpage to be extracted.Particularly, the HTML code of multiple Webpage to be extracted can be obtained simultaneously, also can the HTML code of each Webpage to be extracted of acquisition one by one successively.
Cluster cell 20 is for carrying out cluster according to HTML code to multiple Webpage to be extracted, obtain multiple belonging kinds, namely, according to the HTML code of the Webpage each to be extracted got, multiple Webpage to be extracted is classified, Webpage to be extracted similar in multiple Webpage to be extracted is classified as a classification.It should be noted that, a Webpage to be extracted can only have a belonging kinds.
First extraction unit 30 is for extracting the object block element in each belonging kinds, and wherein, object block element is the block element that the difference Webpage to be extracted in same belonging kinds has.Particularly, object block element can be one, also can be multiple.In embodiments of the present invention, the concrete quantity of object block element is that the quantity of the block element had according to the page to be extracted different in same belonging kinds is determined.Total block element refers to the bookmark name of this block element in same belonging kinds in the different page to be extracted, the block element that attribute is all identical, and attribute is herein class attribute or id attribute.Such as: Webpage 1, Webpage 2 and Webpage 3 belong to belonging kinds A, the block unit jointly comprised in each Webpage in Webpage 1, Webpage 2 and Webpage 3 have 3, be div [class=" menu "], div [id=" title "] and p [class=" content "] respectively, the object block element so in belonging kinds A is then 3.
Second extraction unit 40, for extracting the text in object block element, obtains the text collection of object block element.Particularly, comprise multiple text in same object block element, the set of multiple text is the text collection of this object block element.If object block element is multiple, so extract the text in each object block element, obtain the text collection of each object block element.Continue to adopt above-mentioned illustrating, for object block element div [id=" title "], the text collection obtained is { " title 1 ", " title 2 ", " title 3 " }.
First computing unit 50 is for calculating the desired value of text collection, and wherein, desired value is for representing the difference degree of the text in text collection, namely, calculate object block element Chinese version difference degree, difference degree is larger, illustrates that the content difference in the text in this object block element is larger.
3rd extraction unit 60 is greater than the text in the text collection of the first predetermined threshold value for extracting desired value, obtain Webpage information, namely only have desired value to be greater than text in the text collection of the first preset value, be only the information needing to extract in Webpage to be extracted.Particularly, the first preset value can be arranged according to demand.
In embodiments of the present invention, by obtaining the HTML code of multiple Webpage to be extracted, the division to multiple Webpage belonging kinds to be extracted can be realized, and then the block element jointly comprised in the difference Webpage to be extracted obtaining under same belonging kinds, the extraction to same block element Chinese version content can be realized, then can according to the comparative result of the difference degree of the content of text got and predetermined threshold value, determine whether text content is the information that the needs in Webpage to be extracted extract, solve info web in prior art and extract the low problem of accuracy, and then reach the effect improving info web extraction accuracy.
It should be noted that, if the quantity of object block element is multiple, need the desired value of the text collection calculating each object block element respectively, and each desired value calculated compared with the first predetermined threshold value respectively, text desired value be greater than in the text collection of the first predetermined threshold value extracts.
Particularly, the first computing unit 50 comprises logging modle, the first determination module, computing module and the second determination module, wherein:
Logging modle is used for the occurrence number of each not identical text in recording text set.Because text collection comprises multiple text, so multiple text may exist the identical text of content, in embodiments of the present invention, the occurrence number of text in text set that content is not identical each other is only added up.
First determination module is used for the occurrence number according to each not identical text, and determine total occurrence number of full text in text collection, particularly, in text set, total occurrence number of full text equals the occurrence number sum of all not identical texts.
Computing module is used for, according to the occurrence number of each not identical text and total occurrence number, calculating the frequency of occurrences of each not identical text in text collection.Such as, a text A different from other texts in text set is had in text set, the occurrence number of text A in text set is 3 times, in text set, total occurrence number of full text is 30 times, so for text A, the frequency of occurrences in above-mentioned text collection is 1/10.
Second determination module is used for according to the frequency of occurrences of each not identical text in text collection, determines the desired value of text collection.
If object block element is multiple, the desired value of the text collection of so each object block element can calculate by repeating to call logging modle, the first determination module, computing module and the second determination module.
Particularly, the second determination module comprises calculating sub module, and calculating sub module is used for according to formula
calculate the desired value of text collection, wherein, E
setfor the desired value of text collection, m is the number comprising not identical text in text collection, p (text
i) be the frequency of occurrences of each not identical text in text collection.In embodiments of the present invention, text collection E is calculated
setthe middle logarithm by the frequency of occurrences of text not identical with this for the frequency of occurrences of each not identical text is multiplied, and by all result summations obtained, then getting negative, is exactly the desired value of text set.
Preferably, the extraction element of the Webpage information that the embodiment of the present invention provides also comprises record cell, record cell is used for the text be greater than in extraction desired value in the text collection of the first predetermined threshold value, after obtaining Webpage information, and the category attribute of recording text.Particularly, category attribute can be title, content etc.Namely, the content of text that record extracts is title or content etc. to the embodiment of the present invention.
In embodiments of the present invention, the category attribute of text extracted by record, convenient follow-up carry out large data analysis time, user can filter out required information fast, reaches the effect improving user satisfaction.Such as, user wants to screen in the info web extracted, and content is the information of title, and so user only need select category attribute to be title, can filter out the info web meeting it and require fast.
Preferably, the embodiment of the present invention additionally provides a kind of concrete mode determining the belonging kinds of the page to be extracted, can be performed by set up unit, the 4th extraction unit, the second computing unit, comparing unit and the processing unit included by the extraction element of Webpage information, wherein:
Set up unit for setting up the first tree structure according to the HTML code of the first Webpage to be extracted, and set up the second tree structure according to the HTML code of the second Webpage to be extracted, wherein, the first Webpage to be extracted and second page to be extracted are any two Webpages to be extracted in multiple page to be extracted.
4th extraction unit, for extracting the block element comprising preset attribute in the first tree structure, obtains first piece of element, and extracts the block element comprising preset attribute in the second tree structure, obtains second piece of element.Particularly, preset attribute is class attribute or id attribute, this unit namely only extracts the block element comprising class attribute or id attribute in the HTML code of first page to be extracted, and only extracts the block element comprising class attribute or id attribute in the HTML code of second page to be extracted.
Second computing unit is used for according to first piece of element and second piece of element, calculates the similarity mean value of the first Webpage to be extracted and the second Webpage to be extracted.In embodiments of the present invention, the similarity mean value of the first Webpage to be extracted and the second Webpage to be extracted can be calculated according to formula V=1/2 (S1+S2), wherein, V is similarity mean value, S1 is the first similarity of the first Webpage to be extracted and the second Webpage to be extracted, and S2 is the second similarity of the first Webpage to be extracted and the second Webpage to be extracted.Particularly, can according to formula
calculate the first similarity S1, wherein, Kp is the block element that the first Webpage to be extracted is identical with in the second Webpage to be extracted, and p gets 1 to m successively, and m is the number of same block element, V
1Kpfor the frequency of occurrence of same block element Kp in the first Webpage to be extracted, K
0kbe first piece of element in the first Webpage to be extracted, N1 is the number of first piece of element in the first Webpage to be extracted,
be first piece of element K
0kfrequency of occurrence in the first Webpage to be extracted; According to formula
calculate the second similarity S2, wherein, V
2Kpfor the frequency of occurrence of same block element Kp in the second Webpage to be extracted, K
1kbe second piece of element in the second Webpage to be extracted, N2 is the number of second piece of element in the second Webpage to be extracted,
be second piece of element K
1kfrequency of occurrence in the second Webpage to be extracted.
Comparing unit is for comparing the size of similarity mean value and the second predetermined threshold value.Particularly, the second predetermined threshold value also can be arranged according to demand.
Processing unit is used for when comparing similarity mean value and being greater than the second predetermined threshold value, determine that the first Webpage to be extracted and second page to be extracted are same home classification, or when comparing similarity mean value and being less than or equal to the second predetermined threshold value, determine that the first Webpage to be extracted and second page to be extracted are respectively different belonging kinds, this unit namely, when judging that similarity mean value is greater than the second predetermined threshold value, the first Webpage to be extracted and second page to be extracted belong to same belonging kinds; When judging that similarity mean value is less than or equal to the second predetermined threshold value, the first Webpage to be extracted and second page to be extracted belong to different belonging kinds respectively.
In embodiments of the present invention, can by any two Webpages in multiple page to be extracted respectively as the first Webpage to be extracted and the second Webpage to be extracted, and repeat call establishment unit, the 4th extraction unit, the second computing unit, comparing unit and processing unit, until determine the belonging kinds of each page to be extracted.It should be noted that, if Webpage A and Webpage B belongs to same belonging kinds, Webpage A and Webpage D also belongs to same belonging kinds, and so Webpage A, Webpage B and Webpage D all belong to same belonging kinds.After two or more Webpage to be extracted belongs to same belonging kinds, the Webpage to be extracted determining belonging kinds is needed for other, as long as a Webpage to be extracted in this Webpage to be extracted and above-mentioned belonging kinds is calculated similarity mean value, and the similarity mean value obtained and the second predetermined threshold value are compared, can determine whether this Webpage to be extracted belongs to above-mentioned belonging kinds.
As can be seen from the above description, the invention solves info web in prior art and extract the low problem of accuracy, reach and improve the effect that info web extracts accuracy.
The invention described above embodiment sequence number, just to describing, does not represent the quality of embodiment.
In the above embodiment of the present invention, the description of each embodiment is all emphasized particularly on different fields, in certain embodiment, there is no the part described in detail, can see the associated description of other embodiments.
In several embodiments that the application provides, should be understood that, disclosed client, the mode by other realizes.Wherein, device embodiment described above is only schematic, the such as division of described unit, be only a kind of logic function to divide, actual can have other dividing mode when realizing, such as multiple unit or assembly can in conjunction with or another system can be integrated into, or some features can be ignored, or do not perform.Another point, shown or discussed coupling each other or direct-coupling or communication connection can be by some interfaces, and the indirect coupling of unit or module or communication connection can be electrical or other form.
The described unit illustrated as separating component or can may not be and physically separates, and the parts as unit display can be or may not be physical location, namely can be positioned at a place, or also can be distributed in multiple network element.Some or all of unit wherein can be selected according to the actual needs to realize the object of the present embodiment scheme.
In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, also can be that the independent physics of unit exists, also can two or more unit in a unit integrated.Above-mentioned integrated unit both can adopt the form of hardware to realize, and the form of SFU software functional unit also can be adopted to realize.
If described integrated unit using the form of SFU software functional unit realize and as independently production marketing or use time, can be stored in a computer read/write memory medium.Based on such understanding, the part that technical scheme of the present invention contributes to prior art in essence in other words or all or part of of this technical scheme can embody with the form of software product, this computer software product is stored in a storage medium, comprises all or part of step of some instructions in order to make a computer equipment (can be personal computer, server or the network equipment etc.) perform method described in each embodiment of the present invention.And aforesaid storage medium comprises: USB flash disk, ROM (read-only memory) (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), portable hard drive, magnetic disc or CD etc. various can be program code stored medium.
The above is only the preferred embodiment of the present invention; it should be pointed out that for those skilled in the art, under the premise without departing from the principles of the invention; can also make some improvements and modifications, these improvements and modifications also should be considered as protection scope of the present invention.