CN105302823A - Overlapped community parallel discovery method and system - Google Patents

Abstract

The present invention discloses an overlapped community parallel fast discovery method and system. Firstly, a node distribution network diagram is read, a community set is established, and the node distribution network diagram is associated with the community set; then first n nodes with the highest node degree are respectively used as center nodes of n initial communities, and the n initial communities are stored in the community set; the n initial communities are simultaneously subjected to the following procedures of: using directly adjacent nodes of each center node as candidate member nodes of the corresponding community, sequentially judging whether the candidate member nodes belong to the community, incorporating the nodes belonging to the community into the community, using directly adjacent nodes of the nodes belonging to the community as candidate member nodes of the community and repeating the step; and removing the nodes which do not belong to the community from a candidate member set; for nodes which are not incorporated to any community, the above steps are repeatedly executed until all the nodes in the node distribution network diagram belong to one community; and finally, community incorporation is performed by a community overlapping degree and the effect of two communities on mutual community gathering degrees.

Description

Overlapping community walks abreast the method and system found

Technical field

The present invention relates to complex network field, in particular to one, based on MapReduce, (Map, maps; Reduce, abbreviation; MapReduce is the software architecture that Google proposes, the concurrent operation for large-scale dataset) overlapping community to walk abreast the method and system found fast.

Background technology

Virtual community refers to the social clustering phenomenon based on internet, it constitutes the personal relationship network of certain scale, has been polymerized the user of same interest in virtual network, enable them to mutually propose, express, exchange the viewpoint of oneself, the user of different community also may be mutual.Community can provide timely for user, reliable, valuable information, is still conducive to businessman simultaneously and finds client exactly.But community is hidden in numerous numerous and diverse relation and is connected behind, it is not dominant existence, this just needs researchist use correlation technique and technology the community discovery hidden and excavate out, and the community relations of discovery then can be utilized to provide personalized application and service for all individualities.

In recent years along with high speed development and the popularization and application of network technology, community discovery has developed into a study hotspot problem interdisciplinary, and the researchist of multiple subjects such as sociology, pedagogy, psychology expands series of studies to it from different perspectives.From more broadly angle, most of system of real world can be described by network, and network is the relation of set by node and these nodes, and the namely limit set of connected node formed.A large amount of real data all shows that complex network is all isomery usually, and that is it is made up of various dissimilar node, associates more between the node of wherein same type, and belong to connect between dissimilar node relatively less.

For large complicated network, the design concept of conventional community discovery method and mode of operation cannot complete fast in real time.

Summary of the invention

The invention provides a kind of overlapping community based on MapReduce to walk abreast discover method and system, in order to the community in the large complicated network of quick-searching.

For achieving the above object, the invention provides a kind of overlapping community and to walk abreast rapid discovery method, comprising the following steps:

S1: read Node distribution network chart from document data set; Establishment community gathers; Described Node distribution network chart is gathered with described community and is associated;

S2: the number setting initial community is n; Calculate the number of degrees of each node in described Node distribution network chart, and obtain the Centroid of the highest node of front n the node number of degrees respectively as n initial community; Now the concentration class of each initial community is 0; N initial community is stored in the set of described community; Wherein n be more than or equal to 1 natural number;

S3: following program is performed simultaneously to each of the initial community of said n: the candidate member's node direct neighbor node of described Centroid being elected as this community, and the candidate member's set described candidate member's node being added this community; Judge whether the described candidate member's node in described candidate member set belongs to this community, the node belonging to this community is incorporated to this community, the direct neighbor node of the node belonging to this community is elected as to candidate member's node of this community, repeat this step successively; Shift out described candidate member set to the node not belonging to this community not deal with; This community data is gathered stored in described community;

S4: judge whether have node not to be incorporated to any community in described Node distribution network chart, if so, then to node repeated execution of steps S2, S3, S4 of not being incorporated to any community, until all nodes in described Node distribution network chart all belong to a community;

S5: its community's degree of overlapping is calculated between two to all communities in the set of described community, if the described community degree of overlapping of Liang Ge community is greater than setting threshold value, the Liang Ge community then community's degree of overlapping described in this being greater than setting threshold value merges into a community, and upgrades the corresponding data of described community set;

S6: to two communities with common node any in the set of described community, calculate the new communities' concentration class being merged into new communities, by this new communities' concentration class, front these two the described community concentration class with the community of common node contrast with merging respectively, if these two have the described community concentration class of the community of common node before this new communities' concentration class is greater than merging respectively, the community then these two being had common node merges into new communities, this new communities' concentration class is community's concentration class of the new communities after these two communities with common node merge, and upgrade the corresponding data of described community set.

Wherein, in step S3, realize in multi-threaded parallel mode each program performed of the initial community of said n simultaneously, the method whether node judging in described candidate member set belongs to this community is: the node contribution degree calculating each described candidate member's node; The node being 1 by described node contribution degree is incorporated to this community, and calculates the initial community concentration class of this community; For the node that described node contribution degree is not 1, sorted on earth by height according to node contribution degree described in it, and from the node that wherein said node contribution degree is the highest, suppose that the node in described candidate member set belongs to this community and calculates the first intermediary communities concentration class of this community successively, to suppose not by the node in described candidate member set and the concentration class of this community fashionable is initial aggregation degree, if this first intermediary communities concentration class of this community is greater than this initial community concentration class, then respective nodes is incorporated to this community; If this first intermediary communities concentration class is less than this initial community concentration class, then respective nodes and node below thereof are not incorporated to this community, be judged as that this community discovery is complete when not having described first intermediary communities concentration class to be greater than this initial community's concentration class, this initial community concentration class is community's concentration class of this community.

Wherein, in step S3, realize under MapReduce framework each program performed of the initial community of said n simultaneously, the method whether node judging in described candidate member set belongs to this community is:

Suppose that all nodes in described candidate member set belong to this community respectively, second intermediary communities concentration class of this community when the node simultaneously calculated respectively in described candidate member set belongs to this community, and all nodes in described candidate member's set are sorted from big to small according to described second intermediary communities concentration class, to suppose not by the node in described candidate member set and the concentration class of this community fashionable is initial aggregation degree, compare with this initial community concentration class by the described second intermediary communities concentration class of each node successively, if the described second intermediary communities concentration class of node is greater than this initial community concentration class, then corresponding node is incorporated to this community, if this second intermediary communities concentration class is less than this initial community concentration class, then respective nodes machine node below is not incorporated to this community, be judged as that this community discovery is complete when not having described first intermediary communities concentration class to be greater than this initial community's concentration class, this initial community concentration class is community's concentration class of this community.

Wherein, the computing method of described node contribution degree are:

Suppose node v _idescribed node contribution degree be c, community n _jinterior node number is N, node V _ibelong to community n _j, node V _iwith community n _jthe limit number that other nodes inner connect is L _in, node V _iwith community n _jthe limit number that exterior node connects is L _out, then node V _idescribed node contribution degree c by following formulae discovery:

$c=\frac{L_{\mathrm{in}}}{L_{\mathrm{in}}+L_{\mathrm{out}}}$

Wherein, i is the natural number from 1 to N-1, and j is the natural number from 1 to n.

Wherein, the computing method of described community concentration class are:

Suppose community n _idescribed community concentration class be M, community n _jinterior node number is N, node V _ibelong to community n _j, and node V _ito community center node V ₀distance be D (V _i, V ₀), node V _iwith community n _jthe limit number that other nodes inner connect is L _in, node V _iwith community n _jthe limit number that exterior node connects is L _out, then community n _jdescribed community concentration class be M by following formulae discovery:

$M=\frac{1}{\underset{i=1}{\overset{N-1}{\mathrm{\Σ}}}D(V_i,V_0)}\×\frac{L_{\mathrm{in}}}{L_{\mathrm{in}}+L_{\mathrm{out}}}$

Wherein, i is the natural number from 1 to N-1, and j is the natural number from 1 to n;

The computing method of described initial community concentration class, described intermediary communities concentration class and described new communities concentration class are all identical with the computing method of described community concentration class.

Wherein, the computing method of the described community degree of overlapping of Liang Ge community are:

Suppose that Liang Ge community is A and B, the node set of community A is designated as C _a, the node set of community B is designated as C _b, the described community degree of overlapping of community A and community B is designated as O _aB, then the described community degree of overlapping O of community A and community B _aBby following formulae discovery:

$O_{\mathrm{AB}}=\frac{C_A\∩C_B}{C_A\∪C_B}$

Wherein, community A and community B is two different communities.

In addition, the present invention also provides a kind of overlapping community to walk abreast Fast Discovery System, comprising:

Community discovery preparation module, for: from document data set, read Node distribution network chart; Establishment community gathers; Described Node distribution network chart is gathered with described community and is associated;

Initial community determination module, for: the number setting initial community is n; Calculate the number of degrees of each node in described Node distribution network chart, and obtain the Centroid of the highest node of front n the node number of degrees respectively as n initial community; Now the concentration class of each initial community is 0; N initial community is stored in the set of described community; Wherein n be more than or equal to 1 natural number;

Community is parallel finds module, for: to each of the initial community of said n, the direct neighbor node (namely having the node that limit is connected) of described Centroid is elected as candidate member's node of this community, and described candidate member's node is added candidate member's set of this community; Calculate the node contribution degree of each described candidate member's node; The node being 1 by described node contribution degree is incorporated to this community, and calculates the initial community concentration class of this community; For the node that described node contribution degree is not 1, sorted on earth by height according to node contribution degree described in it, and from the node that wherein said node contribution degree is the highest, suppose that it belongs to this community and calculates the intermediary communities concentration class of this community successively, if this intermediary communities concentration class is greater than this initial community concentration class, then respective nodes is incorporated to this community, and repeats this step; If this intermediary communities concentration class is less than this initial community concentration class, then respective nodes is not incorporated to this community, and is judged as that this community discovery is complete, and this initial community concentration class is community's concentration class of this community; This community data is gathered stored in described community;

Judge control module, for: judge whether have node not to be incorporated to any community in described Node distribution network chart, if, then call described initial community determination module and described community parallel discovery module, community discovery is re-started to the node not being incorporated to any community, until all nodes in described Node distribution network chart all belong to a community;

First community merges module, for: its community's degree of overlapping is calculated between two to all communities in the set of described community, if the described community degree of overlapping of Liang Ge community is greater than setting threshold value, the Liang Ge community then community's degree of overlapping described in this being greater than setting threshold value merges into a community, and upgrades the corresponding data of described community set;

Second community merges module, for: to two communities with common node any in the set of described community, calculate the new communities' concentration class being merged into new communities, by this new communities' concentration class, front these two the described community concentration class with the community of common node contrast with merging respectively, if these two have the described community concentration class of the community of common node before this new communities' concentration class is greater than merging respectively, the community then these two being had common node merges into new communities, this new communities' concentration class is community's concentration class of the new communities after these two communities with common node merge, and upgrade the corresponding data of described community set.

Wherein, the computing method of described node contribution degree are:

Suppose node v _idescribed node contribution degree be c, community n _jinterior node number is N, node V _ibelong to community n _j, node V _iwith community n _jthe limit number that other nodes inner connect is L _in, node V _iwith community n _jthe limit number that exterior node connects is L _out, then node V _idescribed node contribution degree c by following formulae discovery:

$c=\frac{L_{\mathrm{in}}}{L_{\mathrm{in}}+L_{\mathrm{out}}}$

Wherein, i is the natural number from 1 to N-1, and j is the natural number from 1 to n.

Wherein, the computing method of described community concentration class are:

Suppose community n _idescribed community concentration class be M, community n _jinterior node number is N, node V _ibelong to community n _j, and node V _ito community center node V ₀distance be D (V _i, V ₀), node V _iwith community n _jthe limit number that other nodes inner connect is L _in, node V _iwith community n _jthe limit number that exterior node connects is L _out, then community n _jdescribed community concentration class be M by following formulae discovery:

$M=\frac{1}{\underset{i=1}{\overset{N-1}{\mathrm{\Σ}}}D(V_i,V_0)}\×\frac{L_{\mathrm{in}}}{L_{\mathrm{in}}+L_{\mathrm{out}}}$

Wherein, i is the natural number from 1 to N-1, and j is the natural number from 1 to n;

The computing method of described initial community concentration class, described intermediary communities concentration class and described new communities concentration class are all identical with the computing method of described community concentration class.

Wherein, the computing method of the described community degree of overlapping of Liang Ge community are:

Suppose that Liang Ge community is A and B, the node set of community A is designated as C _a, the node set of community B is designated as C _b, the described community degree of overlapping of community A and community B is designated as O _aB, then the described community degree of overlapping O of community A and community B _aBby following formulae discovery:

$O_{\mathrm{AB}}=\frac{C_A\∩C_B}{C_A\∪C_B}$

Wherein, community A and community B is two different communities.

Compared with prior art, beneficial effect of the present invention is embodied in:

The present invention proposes a kind of overlapping community based on localized clusters degree to walk abreast discover method, multi-threading can be used to realize, also, under can be embodied in MapReduce parallel computation frame, while algorithm can being walked abreast based on the strategy of localized clusters degree, multiple community is found; In strategy of the present invention, community can juxtaposition, crossover node is had between namely different communities, data between different community's detections can not produce association and impact, so the community to different Centroid that can walk abreast based on the parallel communities detecting module of MapReduce framework carries out iteration detection, until Data Convergence and terminating; The parallelization strategies that the present invention proposes under MapReduce framework, parallelization strategies is reasonable in design, effective, the magnanimity computing power of computer node can be made full use of, this parallel model of the Web vector graphic of small data quantity carries out community's detection, and along with the increase of computer node, the communication cost between computer node also increases, model Advantages found is not obvious, but when processing the data of the large order of magnitude, the paralleling tactic Advantages found of the method is especially remarkable, and efficiency of algorithm significantly improves.

Accompanying drawing explanation

In order to be illustrated more clearly in the technical scheme of the embodiment of the present invention, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.

Fig. 1 is that the overlapping community of one embodiment of the invention walks abreast the program flow diagram of rapid discovery method;

Fig. 2 is that the overlapping community of one embodiment of the invention walks abreast the flow chart of steps of rapid discovery method.

Embodiment

Below in conjunction with the accompanying drawing in the embodiment of the present invention, be clearly and completely described the technical scheme in the embodiment of the present invention, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not paying the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.

Shown in Fig. 1, Fig. 2, overlapping community provided by the invention walks abreast rapid discovery method.

Here is first embodiment of the present invention, comprises the following steps:

S1: read Node distribution network chart mgraph from document data set; Create community set Cset and be initialized as empty set; Node distribution network chart mgraph and community are gathered Cset be associated;

S2: the number setting initial community is n; Calculate the number of degrees of each node in described Node distribution network chart, and obtain the Centroid of node respectively as n initial community of front n the node number of degrees the highest (topn); Now each initial community only has a Centroid (hubnode), and community concentration class M is 0; N initial community is stored in community's set Cset;

S3: following program is performed simultaneously (as shown in Figure 1 to each of the initial community of said n, initial community 1 carries out to initial community n is parallel): the candidate member's node direct neighbor node of described Centroid hubnode (namely by node that limit is connected with Centroid) being elected as this community, and the candidate member described candidate member's node being added this community gathers; Calculate the node contribution degree of each described candidate member's node; The node being 1 by described node contribution degree is incorporated to this community, and calculates the initial community concentration class of this community; For the node that described node contribution degree is not 1, sorted on earth by height according to node contribution degree described in it, and from the node that wherein said node contribution degree is the highest, suppose that it belongs to this community and calculates the intermediary communities concentration class of this community successively, if this intermediary communities concentration class is greater than this initial community concentration class, then respective nodes is incorporated to this community, and repeats this step; If this intermediary communities concentration class is less than this initial community concentration class, then respective nodes is not incorporated to this community, and is judged as that this community discovery is complete, and this initial community concentration class is community's concentration class of this community; This community data is gathered Cset stored in community;

S4: judge whether have node not to be incorporated to any community in described Node distribution network chart, if so, then to node repeated execution of steps S2, S3, S4 of not being incorporated to any community, until all nodes in described Node distribution network chart all belong to a community;

S5: its community's degree of overlapping is calculated between two to all communities in the set of described community, if the described community degree of overlapping of Liang Ge community is greater than setting threshold value, the Liang Ge community then community's degree of overlapping described in this being greater than setting threshold value merges into a community, and more new communities gather the corresponding data of Cset;

S6: to two communities with common node any in the set of described community, calculate the new communities' concentration class being merged into new communities, by this new communities' concentration class, front these two the described community concentration class with the community of common node contrast with merging respectively, if these two have the described community concentration class of the community of common node before this new communities' concentration class is greater than merging respectively, the community then these two being had common node merges into new communities, this new communities' concentration class is community's concentration class of the new communities after these two communities with common node merge, and upgrade the corresponding data of described community set.

Here is second embodiment of the present invention, comprises the following steps:

S1: read Node distribution network chart mgraph from document data set; Create community set Cset and be initialized as empty set; Node distribution network chart mgraph and community are gathered Cset be associated;

S2: the number setting initial community is n; Calculate the number of degrees of each node in described Node distribution network chart, and obtain the Centroid of node respectively as n initial community of front n the node number of degrees the highest (topn); Now each initial community only has a Centroid (hubnode), and community concentration class M is 0; N initial community is stored in community's set Cset;

S3: following program is performed simultaneously (as shown in Figure 1 to each of the initial community of said n, initial community 1 carries out to initial community n is parallel): the candidate member's node direct neighbor node of described Centroid hubnode (namely by node that limit is connected with Centroid) being elected as this community, and the candidate member described candidate member's node being added this community gathers; The task iteration be made up of two MapReduce is run, judge whether the described candidate member's node in described candidate member set belongs to this community, the node belonging to this community is incorporated to this community, the direct neighbor node of the node belonging to this community is elected as to candidate member's node of this community, repeat this step; Described candidate member set is shifted out to the node not belonging to this community and does not do other process; This community data is gathered stored in described community; Be specially:

Set up global variable, comprise the distance sum etc. of the inner existing inner fillet number in community and all nodes in community, the number of the existing node in community, total number of degrees of the existing node in community, the existing node in community and Centroid, put into a HDFS (HadoopDistributedFileSystem, distributed file system) in file, all computer nodes can obtain their value;

Mapper1 reads the node data in Node distribution network chart mgraph, the form of this node data is [<Node, HubNode>, <FlagDistanceM{adjNodes}>], wherein HubNode is the Centroid of certain initial community, and Node is present node, two tuple <Node, HubNode> is key (index), for searching node; <FlagDistanceM{adjNodes}> is the text that nodal information generates is value (content of storage), Flag is node label position, value has four kinds: represent when being 0 that node is untreated, represent when being 1 node be certain community both candidate nodes, represent when being 2 node be newly add certain community node, represent that node processing is complete when being 3, Distance is the distance between present node and HubNode, and M is community's concentration class present node being added this community behind the community of HubNode place.After initial community determines, only have the flag initial value of each Centroid HubNode to be 2, the flag initial value of other nodes is 0; When a node is added into both candidate nodes set, its flag value is assigned 1; When a node is determined to belong to certain community, its flag value is assigned 2; After all adjacent nodes of this node are added to both candidate nodes collection, its flag value is assigned 3.Mapper1 reads in the data block of node, if the Flag value of node equals 2, represent that this node is the node being newly incorporated to community, community's both candidate nodes set is added with it by the node that limit is connected all, and export corresponding <k ', the value of v ' > (k is key, and v is value, and this form represents new data block); If the node Flag value of the data block of reading in is not equal to 2, then the direct former state of former data block exports;

The task of Reducer1 is that the data that Mapper1 generates are arranged, carry out because Mapper1 walks abreast according to n initial community, crossover node (namely simultaneously processed in multiple community node) may be had, therefore Mapper1 export data in a node may have many data, their information is carried out merging gather, the data layout that last Reducer1 exports remains [<Node, HubNode>, <FlagDistanceM{adjNodes}>], but a node only has data,

Mapper2 reads in the node data block [<Node that Reducer1 exports, HubNode>, <FlagDistanceM{adjNodes}>], if the value of the Flag of node is 1, then this node is a member that community's both candidate nodes is concentrated, so suppose that this node is incorporated to the community of node centered by node HubNode, calculate now the value of this community's accumulations degree assignment to the M of this node, i.e. the intermediary communities concentration class of community; After Mapper2 upgrades M value, this node is exported again, and output format is [<HubNode, Flag>, <NodeDistanceM{adjNodes}>].Two tuple <HubNode, Flag> export Key value, and the key value of all both candidate nodes of same like this community is all identical, and they can be output to same reduce node;

Reducer2 reads in all both candidate nodes of certain community, compare their M value, and by M value order from big to small, all both candidate nodes are sorted, from the node that M value is maximum, by node M value compared with the initial community concentration class of this community, if node M value is greater than the initial community concentration class of this community, then the Flag of this node be set to 2 (being namely included into this community) and continue more next, if node M value is less than the initial community concentration class of this community, then below do not need compare, this node and later node do not belong to this community, this community discovery is complete, upgrade global variable Aggregate, wherein global variable Aggregate represents whether the discovery procedure of a community completes, before MapReduce iteration is run each time, the value of write global variable Aggregate (set) is 0, when MapReduce performs, if there is node to be incorporated to any one community, the value of then putting Aggregate is 1, at the end of one takes turns Mapper1-Reducer1-Mapper2-Reducer2 task, detect the value of Aggregate, if Aggregate still equals 0, representing all just all has been found that complete in the community of parallel detecting, if the value of Aggregate is 1, then continue to perform new one and take turns Mapper1-Reducer1-Mapper2-Reducer2 task until algorithm convergence (namely Aggregate remains 0).

S4: judge whether have node not to be incorporated to any community in described Node distribution network chart, if so, then to node repeated execution of steps S2, S3, S4 of not being incorporated to any community, until all nodes in described Node distribution network chart all belong to a community;

S5: its community's degree of overlapping is calculated between two to all communities in the set of described community, if the described community degree of overlapping of Liang Ge community is greater than setting threshold value, the Liang Ge community then community's degree of overlapping described in this being greater than setting threshold value merges into a community, and more new communities gather the corresponding data of Cset;

Community's degree of overlapping detects and merges execution map and the reduce task of module iteration, principal function is set as first property the target community that current consideration merges and writes global data, all map nodes read in community's record respectively, target community's information is read from global data, calculate community's degree of overlapping of target community and this community, if community's degree of overlapping is greater than threshold value (in one embodiment of the invention, this threshold value is set as 0.7), then this community OverlapFlag (overlapping mark) be set to 1, otherwise OverlapFlag is set to 0; Map is used as the community ID of target community as key value, the information (comprising overlapping mark) of this community and target community is used as value value, exports <key, value > data pair.Reduce checks the <key that map exports, the data of list<value>> (information list that all communities are relevant to target community), whether check in list<value> has OverlapFlag to equal the record of 1, Article 1 OverlapFlag equal 1 community and target community merge, generate new community and export the corresponding <key of new communities, list<value>> data, the new communities generated are upgraded global data as target community simultaneously, the MergeFlag (merging mark) putting global data is again masked as 1.The meaning merging mark MergeFlag is that set to 0 by MergeFlag before each iteration task, when there being community to merge, MergeFlag is set to 1.If record OverlapFlag values all in list<value> all equals 0, then represent that community's degree of overlapping of target community and all communities is all less than and set threshold value 0.7, it does not need to merge with any community, and so the value of MergeFlag is 0.OverlapFlag value is equaled to community's record of 1, adjustment formatted output <key ', value ' > data pair.The task of Map judges whether community's degree of overlapping of target community and all communities exceedes threshold value, and the task of Reduce is in the community exceeding threshold value, get one and target community's merging, if do not exceed the community of threshold value, gives and merge mark MergeFlag assignment.Map-Reduce runs once, principal function determining program end condition (global data community process of aggregation complete and current MergeFlag be masked as 0), if condition meets, EOP (end of program) now obtains all community datas after merging; If condition does not meet, continue iteration and perform new one and take turns map-reduce task, degree of overlapping inspection is carried out to new target community.

S6: to two communities with common node any in the set of described community, calculate the new communities' concentration class being merged into new communities, by this new communities' concentration class, front these two the described community concentration class with the community of common node contrast with merging respectively, if these two have the described community concentration class of the community of common node before this new communities' concentration class is greater than merging respectively, the community then these two being had common node merges into new communities, this new communities' concentration class is community's concentration class of the new communities after these two communities with common node merge, and upgrade the corresponding data of described community set.

In one embodiment of the invention, the computing method of described node contribution degree are:

Suppose node v _idescribed node contribution degree be c, community n _jinterior node number is N, node V _ibelong to community n _j, node V _iwith community n _jthe limit number that other nodes inner connect is L _in, node V _iwith community n _jthe limit number that exterior node connects is L _out, then node V _idescribed node contribution degree c by following formulae discovery:

$c=\frac{L_{\mathrm{in}}}{L_{\mathrm{in}}+L_{\mathrm{out}}}$

Wherein, i is the natural number from 1 to N-1, and j is the natural number from 1 to n.

In one embodiment of the invention, the computing method of described community concentration class are:

Suppose community n _idescribed community concentration class be M, community n _jinterior node number is N, node V _ibelong to community n _j, and node V _ito community center node V ₀distance be D (V _i, V ₀), node V _iwith community n _jthe limit number that other nodes inner connect is L _in, node V _iwith community n _jthe limit number that exterior node connects is L _out, then community n _jdescribed community concentration class be M by following formulae discovery:

$M=\frac{1}{\underset{i=1}{\overset{N-1}{\mathrm{\&Sigma;}}}D(V_i,V_0)}\&times;\frac{L_{\mathrm{in}}}{L_{\mathrm{in}}+L_{\mathrm{out}}}$

Wherein, i is the natural number from 1 to N-1, and j is the natural number from 1 to n;

The computing method of described initial community concentration class, described intermediary communities concentration class and described new communities concentration class are all identical with the computing method of described community concentration class.

In one embodiment of the invention, the computing method of the described community degree of overlapping of Liang Ge community are:

Suppose that Liang Ge community is A and B, the node set of community A is designated as C _a, the node set of community B is designated as C _b, the described community degree of overlapping of community A and community B is designated as O _aB, then the described community degree of overlapping O of community A and community B _aBby following formulae discovery:

$O_{\mathrm{AB}}=\frac{C_A\&cap;C_B}{C_A\&cup;C_B}$

Wherein, community A and community B is two different communities.

In addition, the present invention also provides a kind of overlapping community to walk abreast Fast Discovery System, comprising:

Community discovery preparation module, for: from document data set, read Node distribution network chart; Establishment community gathers; Described Node distribution network chart is gathered with described community and is associated;

Initial community determination module, for: the number setting initial community is n; Calculate the number of degrees of each node in described Node distribution network chart, and obtain the Centroid of the highest node of front n the node number of degrees respectively as n initial community; Now the concentration class of each initial community is 0; N initial community is stored in the set of described community;

Community is parallel finds module, for: to each of the initial community of said n, the direct neighbor node (namely having the node that limit is connected) of described Centroid is elected as candidate member's node of this community, and described candidate member's node is added candidate member's set of this community; Calculate the node contribution degree of each described candidate member's node; The node being 1 by described node contribution degree is incorporated to this community, and calculates the initial community concentration class of this community; For the node that described node contribution degree is not 1, sorted on earth by height according to node contribution degree described in it, and from the node that wherein said node contribution degree is the highest, suppose that it belongs to this community and calculates the intermediary communities concentration class of this community successively, if this intermediary communities concentration class is greater than this initial community concentration class, then respective nodes is incorporated to this community, and repeats this step; If this intermediary communities concentration class is less than this initial community concentration class, then respective nodes is not incorporated to this community, and is judged as that this community discovery is complete, and this initial community concentration class is community's concentration class of this community; This community data is gathered stored in described community;

Judge control module, for: judge whether have node not to be incorporated to any community in described Node distribution network chart, if, then initial community's determination module described in electrophoresis and described community parallel discovery module, community discovery is re-started to the node not being incorporated to any community, until all nodes in described Node distribution network chart all belong to a community;

First community merges module, for: its community's degree of overlapping is calculated between two to all communities in the set of described community, if the described community degree of overlapping of Liang Ge community is greater than setting threshold value, the Liang Ge community then community's degree of overlapping described in this being greater than setting threshold value merges into a community, and upgrades the corresponding data of described community set;

Second community merges module, for: to two communities with common node any in the set of described community, calculate the new communities' concentration class being merged into new communities, by this new communities' concentration class, front these two the described community concentration class with the community of common node contrast with merging respectively, if these two have the described community concentration class of the community of common node before this new communities' concentration class is greater than merging respectively, the community then these two being had common node merges into new communities, this new communities' concentration class is community's concentration class of the new communities after these two communities with common node merge, and upgrade the corresponding data of described community set.

In one embodiment of the invention, the computing method of described node contribution degree are:

Suppose node v _idescribed node contribution degree be c, community n _jinterior node number is N, node V _ibelong to community n _j, node V _iwith community n _jthe limit number that other nodes inner connect is L _in, node V _iwith community n _jthe limit number that exterior node connects is L _out, then node V _idescribed node contribution degree c by following formulae discovery:

$c=\frac{L_{\mathrm{in}}}{L_{\mathrm{in}}+L_{\mathrm{out}}}$

Wherein, i is the natural number from 1 to N-1, and j is the natural number from 1 to n.

In one embodiment of the invention, the computing method of described community concentration class are:

Suppose community n _idescribed community concentration class be M, community n _jinterior node number is N, node V _ibelong to community n _j, and node V _ito community center node V ₀distance be D (V _i, V ₀), node V _iwith community n _jthe limit number that other nodes inner connect is L _in, node V _iwith community n _jthe limit number that exterior node connects is L _out, then community n _jdescribed community concentration class be M by following formulae discovery:

$M=\frac{1}{\underset{i=1}{\overset{N-1}{\mathrm{\&Sigma;}}}D(V_i,V_0)}\&times;\frac{L_{\mathrm{in}}}{L_{\mathrm{in}}+L_{\mathrm{out}}}$

Wherein, i is the natural number from 1 to N-1, and j is the natural number from 1 to n;

The computing method of described initial community concentration class, described intermediary communities concentration class and described new communities concentration class are all identical with the computing method of described community concentration class.

In one embodiment of the invention, the computing method of the described community degree of overlapping of Liang Ge community are:

Suppose that Liang Ge community is A and B, the node set of community A is designated as C _a, the node set of community B is designated as C _b, the described community degree of overlapping of community A and community B is designated as O _aB, then the described community degree of overlapping O of community A and community B _aBby following formulae discovery:

$O_{\mathrm{AB}}=\frac{C_A\&cap;C_B}{C_A\&cup;C_B}$

Wherein, community A and community B is two different communities.

One of ordinary skill in the art will appreciate that: accompanying drawing is the schematic diagram of an embodiment, the module in accompanying drawing or flow process might not be that enforcement the present invention is necessary.

One of ordinary skill in the art will appreciate that: the module in the device in embodiment can describe according to embodiment and be distributed in the device of embodiment, also can carry out respective change and be arranged in the one or more devices being different from the present embodiment.The module of above-described embodiment can merge into a module, also can split into multiple submodule further.

Last it is noted that above embodiment is only in order to illustrate technical scheme of the present invention, be not intended to limit; Although with reference to previous embodiment to invention has been detailed description, those of ordinary skill in the art is to be understood that: it still can be modified to the technical scheme described in previous embodiment, or carries out equivalent replacement to wherein portion of techniques feature; And these amendments or replacement, do not make the essence of appropriate technical solution depart from the spirit and scope of embodiment of the present invention technical scheme.

Claims (10)

1. overlapping community walks abreast a rapid discovery method, it is characterized in that, comprises the following steps:

S1: read Node distribution network chart from document data set; Establishment community gathers; Described Node distribution network chart is gathered with described community and is associated;

S2: the number setting initial community is n; Calculate the number of degrees of each node in described Node distribution network chart, and obtain the Centroid of the highest node of front n the node number of degrees respectively as n initial community; Now the concentration class of each initial community is 0; N initial community is stored in the set of described community;

S3: following program is performed simultaneously to each of the initial community of said n: the candidate member's node direct neighbor node of described Centroid being elected as this community, and the candidate member's set described candidate member's node being added this community; Judge whether the described candidate member's node in described candidate member set belongs to this community, the node belonging to this community is incorporated to this community, the direct neighbor node of the node belonging to this community is elected as to candidate member's node of this community, repeat this step successively; Described candidate member set is shifted out to the node not belonging to this community; This community data is gathered stored in described community;

S4: judge whether have node not to be incorporated to any community in described Node distribution network chart, if so, then to node repeated execution of steps S2, S3, S4 of not being incorporated to any community, until all nodes in described Node distribution network chart all belong to a community;

S5: its community's degree of overlapping is calculated between two to all communities in the set of described community, if the described community degree of overlapping of Liang Ge community is greater than setting threshold value, the Liang Ge community then community's degree of overlapping described in this being greater than setting threshold value merges into a community, and upgrades the corresponding data of described community set;

S6: to two communities with common node any in the set of described community, calculate the new communities' concentration class being merged into new communities, by this new communities' concentration class, front these two the described community concentration class with the community of common node contrast with merging respectively, if these two have the described community concentration class of the community of common node before this new communities' concentration class is greater than merging respectively, the community then these two being had common node merges into new communities, this new communities' concentration class is community's concentration class of the new communities after these two communities with common node merge, and upgrade the corresponding data of described community set.

2. overlapping community according to claim 1 walks abreast rapid discovery method, it is characterized in that, in step S3, realize in multi-threaded parallel mode each program performed of the initial community of said n simultaneously, the method whether node judging in described candidate member set belongs to this community is: the node contribution degree calculating each described candidate member's node; The node being 1 by described node contribution degree is incorporated to this community, and calculates the initial community concentration class of this community; For the node that described node contribution degree is not 1, sorted on earth by height according to node contribution degree described in it, and from the node that wherein said node contribution degree is the highest, suppose that the node in described candidate member set belongs to this community and calculates the first intermediary communities concentration class of this community successively, to suppose not by the node in described candidate member set and the concentration class of this community fashionable is initial aggregation degree, if this first intermediary communities concentration class of this community is greater than this initial community concentration class, then respective nodes is incorporated to this community; If this first intermediary communities concentration class is less than this initial community concentration class, then respective nodes and node below thereof are not incorporated to this community, be judged as that this community discovery is complete when not having described first intermediary communities concentration class to be greater than this initial community's concentration class, this initial community concentration class is community's concentration class of this community.

3. overlapping community according to claim 1 walks abreast rapid discovery method, it is characterized in that, in step S3, realize under MapReduce framework each program performed of the initial community of said n simultaneously, the method whether node judging in described candidate member set belongs to this community is:

Suppose that all nodes in described candidate member set belong to this community respectively, second intermediary communities concentration class of this community when the node simultaneously calculated respectively in described candidate member set belongs to this community, and all nodes in described candidate member's set are sorted from big to small according to described second intermediary communities concentration class, to suppose not by the node in described candidate member set and the concentration class of this community fashionable is initial aggregation degree, compare with this initial community concentration class by the described second intermediary communities concentration class of each node successively, if the described second intermediary communities concentration class of node is greater than this initial community concentration class, then corresponding node is incorporated to this community, if this second intermediary communities concentration class is less than this initial community concentration class, then respective nodes machine node below is not incorporated to this community, be judged as that this community discovery is complete when not having described first intermediary communities concentration class to be greater than this initial community's concentration class, this initial community concentration class is community's concentration class of this community.

4. overlapping community according to claim 1 walks abreast rapid discovery method, and it is characterized in that, the computing method of described node contribution degree are:

Suppose node v _idescribed node contribution degree be c, community n _jinterior node number is N, node V _ibelong to community n _j, node V _iwith community n _jthe limit number that other nodes inner connect is L _in, node V _iwith community n _jthe limit number that exterior node connects is L _out, then node V _idescribed node contribution degree c by following formulae discovery:

$c=\frac{L_{\mathrm{in}}}{L_{\mathrm{in}}+L_{\mathrm{out}}}$

Wherein, i is the natural number from 1 to N-1, and j is the natural number from 1 to n.

5. overlapping community according to claim 1 walks abreast rapid discovery method, and it is characterized in that, the computing method of described community concentration class are:

Suppose community n _idescribed community concentration class be M, community n _jinterior node number is N, node V _ibelong to community n _j, and node V _ito community center node V ₀distance be D (V _i, V ₀), node V _iwith community n _jthe limit number that other nodes inner connect is L _in, node V _iwith community n _jthe limit number that exterior node connects is L _out, then community n _jdescribed community concentration class be M by following formulae discovery:

$M=\frac{1}{\underset{i=1}{\overset{N-1}{\mathrm{\&Sigma;}}}D(V_i,V_0)}\&times;\frac{L_{\mathrm{in}}}{L_{\mathrm{in}}+L_{\mathrm{out}}}$

Wherein, i is the natural number from 1 to N-1, and j is the natural number from 1 to n;

The computing method of described initial community concentration class, described intermediary communities concentration class and described new communities concentration class are all identical with the computing method of described community concentration class.

6. overlapping community according to claim 1 walks abreast rapid discovery method, and it is characterized in that, the computing method of the described community degree of overlapping of Liang Ge community are:

Suppose that Liang Ge community is A and B, the node set of community A is designated as C _a, the node set of community B is designated as C _b, the described community degree of overlapping of community A and community B is designated as O _aB, then the described community degree of overlapping O of community A and community B _aBby following formulae discovery:

$O_{\mathrm{AB}}=\frac{C_A\&cap;C_B}{C_A\&cup;C_B}$

Wherein, community A and community B is two different communities.

7. overlapping community walks abreast a Fast Discovery System, it is characterized in that, comprising:

Community discovery preparation module, for: from document data set, read Node distribution network chart; Establishment community gathers; Described Node distribution network chart is gathered with described community and is associated;

Initial community determination module, for: the number setting initial community is n; Calculate the number of degrees of each node in described Node distribution network chart, and obtain the Centroid of the highest node of front n the node number of degrees respectively as n initial community; Now the concentration class of each initial community is 0; N initial community is stored in the set of described community;

Community is parallel finds module, for: to each of a said n initial community, the direct neighbor node of described Centroid is elected as candidate member's node of this community, and described candidate member's node is added candidate member's set of this community; Calculate the node contribution degree of each described candidate member's node; The node being 1 by described node contribution degree is incorporated to this community, and calculates the initial community concentration class of this community; For the node that described node contribution degree is not 1, sorted on earth by height according to node contribution degree described in it, and from the node that wherein said node contribution degree is the highest, suppose that it belongs to this community and calculates the intermediary communities concentration class of this community successively, if this intermediary communities concentration class is greater than this initial community concentration class, then respective nodes is incorporated to this community, and repeats this step; If this intermediary communities concentration class is less than this initial community concentration class, then respective nodes is not incorporated to this community, and is judged as that this community discovery is complete, and this initial community concentration class is community's concentration class of this community; This community data is gathered stored in described community;

Judge control module, for: judge whether have node not to be incorporated to any community in described Node distribution network chart, if, then call described initial community determination module and described community parallel discovery module, community discovery is re-started to the node not being incorporated to any community, until all nodes in described Node distribution network chart all belong to a community;

First community merges module, for: its community's degree of overlapping is calculated between two to all communities in the set of described community, if the described community degree of overlapping of Liang Ge community is greater than setting threshold value, the Liang Ge community then community's degree of overlapping described in this being greater than setting threshold value merges into a community, and upgrades the corresponding data of described community set;

Second community merges module, for: to two communities with common node any in the set of described community, calculate the new communities' concentration class being merged into new communities, by this new communities' concentration class, front these two the described community concentration class with the community of common node contrast with merging respectively, if these two have the described community concentration class of the community of common node before this new communities' concentration class is greater than merging respectively, the community then these two being had common node merges into new communities, this new communities' concentration class is community's concentration class of the new communities after these two communities with common node merge, and upgrade the corresponding data of described community set.

8. overlapping community according to claim 7 walks abreast Fast Discovery System, and it is characterized in that, the computing method of described node contribution degree are:

Suppose node v _idescribed node contribution degree be c, community n _jinterior node number is N, node V _ibelong to community n _j, node V _iwith community n _jthe limit number that other nodes inner connect is L _in, node V _iwith community n _jthe limit number that exterior node connects is L _out, then node V _idescribed node contribution degree c by following formulae discovery:

$c=\frac{L_{\mathrm{in}}}{L_{\mathrm{in}}+L_{\mathrm{out}}}$

Wherein, i is the natural number from 1 to N-1, and j is the natural number from 1 to n.

9. overlapping community according to claim 7 walks abreast Fast Discovery System, and it is characterized in that, the computing method of described community concentration class are:

Suppose community n _idescribed community concentration class be M, community n _jinterior node number is N, node V _ibelong to community n _j, and node V _ito community center node V ₀distance be D (V _i, V ₀), node V _iwith community n _jthe limit number that other nodes inner connect is L _in, node V _iwith community n _jthe limit number that exterior node connects is L _out, then community n _jdescribed community concentration class be M by following formulae discovery:

$M=\frac{1}{\underset{i=1}{\overset{N-1}{\mathrm{\&Sigma;}}}D(V_i,V_0)}\&times;\frac{L_{\mathrm{in}}}{L_{\mathrm{in}}+L_{\mathrm{out}}}$

Wherein, i is the natural number from 1 to N-1, and j is the natural number from 1 to n;

The computing method of described initial community concentration class, described intermediary communities concentration class and described new communities concentration class are all identical with the computing method of described community concentration class.

10. overlapping community according to claim 7 walks abreast Fast Discovery System, and it is characterized in that, the computing method of the described community degree of overlapping of Liang Ge community are:

Suppose that Liang Ge community is A and B, the node set of community A is designated as C _a, the node set of community B is designated as C _b, the described community degree of overlapping of community A and community B is designated as O _aB, then the described community degree of overlapping O of community A and community B _aBby following formulae discovery:

$O_{\mathrm{AB}}=\frac{C_A\&cap;C_B}{C_A\&cup;C_B}$

Wherein, community A and community B is two different communities.