CN106452899B - A kind of distributed data digging system and method - Google Patents
A kind of distributed data digging system and method Download PDFInfo
- Publication number
- CN106452899B CN106452899B CN201610957904.2A CN201610957904A CN106452899B CN 106452899 B CN106452899 B CN 106452899B CN 201610957904 A CN201610957904 A CN 201610957904A CN 106452899 B CN106452899 B CN 106452899B
- Authority
- CN
- China
- Prior art keywords
- server
- control server
- task
- information
- calculation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
- H04L41/0663—Performing the actions predefined by failover planning, e.g. switching to standby network elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Hardware Redundancy (AREA)
Abstract
The present invention provides a kind of distributed data digging system and methods, are related to data mining technology field.First by the first control server as working node in system；And monitored in real time by the second control server, when confirming that the first control server breaks down, working node is sent to front server and substitutes request, completes the replacement of working node.To restore the normal work of entire distributed data digging system after completing working node switching.The problem of task schedule system that the present invention can will lead to entire hadoop frame to avoid the current JobTracker server as main controlled node if breaking down paralyses, is unable to complete task schedule and processing.
Description
Technical field
The present invention relates to data mining technology field more particularly to a kind of distributed data digging system and methods.
Background technique
In recent years, with big data, cloud computing, mobile Internet etc. for the generation information technology of representative rise in the whole world
" third time IT revolution " has been started in range.With the arrival of big data era, data analysis capabilities how are further promoted, it is deep
Enter the business value of mining data, change that is close and driving products innovation, operation flow and management system is realized really " with number
It is said that words ", become a new issue.The essence of data mining be excavated from mass data it is implicit, have to decision it is potential
Relationship, mode and the trend of value.Due to being related to the calculating of mass data, have higher requirements to the frame of the system of realization.
The big data digging system of mainstream is using distributed mining structure, i.e., by a big data analysis task at present
It is decomposed, then by multiple servers parallel computation, finally subitem result is summarized.Such as based on hadoop frame
The distributed system basic framework to be represented is widely used.In hadoop frame, JobTracker server undertakes total tune
The big data analysis operation that the key player of degree, i.e. reception user terminal submit, realizes task according to map/reduce algorithm
It decomposes, then task is distributed on idle calculation server according to the busy degree of calculation server (TaskTracker).
JobTracker server monitors the executive condition of task on calculation server (TaskTracker) simultaneously, if task execution has
It is abnormal, then task is redistributed.This frame haves the defects that apparent, an i.e. risk for Single Point of Faliure.As master
The task schedule system that the JobTracker server of control node will lead to entire hadoop frame if breaking down is fallen into
Paralysis.It is embodied in the submission that user terminal is unable to complete task due to the main controlled node failure as service entrance；Operation
The implementation procedure of server (TaskTracker) cannot monitor and cause execution failed tasks that cannot redistribute；Simultaneously
Calculation server (TaskTracker) makes resource be in idle state because losing task allocator.
Summary of the invention
The embodiment of the present invention provides a kind of distributed data digging system and method, current as main controlled node to solve
JobTracker server will lead to the task schedule system of entire hadoop frame if breaking down and paralyse, nothing
Method completes the problem of task schedule and processing.
In order to achieve the above objectives, the present invention adopts the following technical scheme:
A kind of distributed data digging system, comprising: user terminal group, front server group, the first control server,
Two control servers and calculation server group；The user terminal group includes multiple user terminals；The front server group
Including multiple front servers；The calculation server group includes multiple calculation servers；The user terminal group and it is described before
Set the communication connection of server group；The front server group respectively with first control server, the second control server and
The calculation server group communication connection；
The user terminal, for sending data mining task request to the front server；
The front server is dug for parsing the domain-name information of the data mining task request according to the data
The domain-name information for digging task requests requests the data mining task to be submitted to the first control server as working node；
First control server, for requesting corresponding data mining task to be divided the data mining task
Solution, forms multiple data mining subtasks；The multiple data mining subtask is sent to the front server；
The front server, be also used to for being assigned at multiple calculation servers the multiple data mining subtask into
Row processing, and the task feedback information of calculation server is received, and the task feedback information is sent to first control
Server；
First control server is also used to send the second control service for the task feedback information real-time synchronization
At device；
Second control server is confirming first control for monitoring first control server in real time
When server fail, Xiang Suoshu front server sends working node and substitutes request；
The front server is also used to be substituted according to the working node and request, the second control clothes described in more new record
The network address of business device, so that second control server is as working node；
Second control server is also used to collect to each calculation server broadcasting tasks information and request；
The calculation server is also used to when listening to the broadcasting tasks information collection request, to the preposition clothes
Business device feeds back task situation information；
The front server is also used to the task situation information being sent to second control server；
Second control server, is also used to carry out the task situation information and the task feedback information pair
Than, determine the different information of the task situation information and the task feedback information, and according to pre-set processing strategie,
The different information is handled.
Further, the front server is also used to obtain the fortune of other front servers in front server group
Row status information；When the operating status of other front servers is malfunction, receives and connect with other front servers
The connection request of the user terminal connect, and establish communication connection.
Further, the front server is also used to record the network of the first control server as working node
The network address of address or the second control server as working node.
In addition, the front server, specifically for receiving the heartbeat message of calculation server；The calculation server
Heartbeat message includes the task feedback information of calculation server processing data mining subtask and the cpu resource letter of calculation server
Breath；The heartbeat message of the calculation server is sent to first control server.
In addition, first control server, specifically for multiple data mining subtasks are sent to it is described preposition
When server, data synchronization information is sent to the second control server；The data synchronization information includes data mining subtask
The corresponding calculation server of mission number and each data mining subtask IP address；
After receiving the heartbeat message of calculation server, is sent by the heartbeat message real-time synchronization of calculation server
At two control servers.
In addition, second control server, is specifically used for taking with prefixed time interval timing to first control
Business device sends heartbeat request；If not receiving the first control after continuous n times send heartbeat request to first control server
The heartbeat response message of control server, it is determined that first control server breaks down, and Xiang Suoshu front server is sent
Working node substitutes request；Wherein n is pre-set frequency threshold value.
In addition, second control server, is specifically used for:
According to the task situation information and the task feedback information, two parts of task list lists are generated；The task
Inventory list includes the IP address of calculation server and the cpu resource information of calculation server；
According to two parts of task list lists, different information is determined；
If the different information is that the first control server has been distributed to after calculation server due to the first control server
Failure is not synchronized to the task of the second control server, according to the number of the second control server of the task situation information update
According to synchronizing information；
If the different information is that the first control server has distributed to calculation server, and calculation server handles task
After failure, due to the first control server failure, it is not synchronized to the task of the second control server, from the task situation information
Middle acquisition mission failure information, and the corresponding data mining subtask of the mission failure information is redistributed；
If the different information is the still unappropriated data mining subtask of the first control server, will still unappropriated number
It is distributed to according to excavation subtask by front server and is handled at calculation server.
In addition, first control server, specifically for the cpu resource information according to calculation server, by a data
It excavates subtask and distributes to the maximum calculation server of cpu resource in each calculation server.
A kind of distributed data digging method, applied to above-mentioned distributed data digging system, which includes: user
Set of terminal, front server group, the first control server, the second control server and calculation server group；The user is whole
End group includes multiple user terminals；The front server group includes multiple front servers；The calculation server group includes
Multiple calculation servers；The user terminal group and the front server group communicate to connect；The front server group difference
It is communicated to connect with first control server, the second control server and the calculation server group；
The described method includes:
User terminal sends data mining task request to the front server；
Front server parses the domain-name information of the data mining task request, is requested according to the data mining task
Domain-name information data mining task request is submitted to the first control server as working node；
The data mining task is requested corresponding data mining task to decompose by the first control server, is formed more
A data mining subtask, and the multiple data mining subtask is sent to the front server；
The multiple data mining subtask is assigned at multiple calculation servers and handles by the front server,
And the task feedback information of calculation server is received, and the task feedback information is sent to first control server；
First control server sends the task feedback information real-time synchronization at the second control server；
Second control server monitors first control server in real time, is confirming first control server
When breaking down, Xiang Suoshu front server sends working node and substitutes request；
The front server substitutes according to the working node requests, the net of the second control server described in more new record
Network address, so that second control server is as working node；
Second control server is collected to each calculation server broadcasting tasks information and is requested；
The calculation server is when listening to the broadcasting tasks information collection request, Xiang Suoshu front server feedback
Task situation information；
The task situation information is sent to second control server by the front server；
Second control server compares the task situation information and the task feedback information, determines institute
The different information of task situation information and the task feedback information is stated, and according to pre-set processing strategie, to the difference
Different information is handled.
In addition, the distributed data digging method, further includes:
The front server obtains the running state information of other front servers in front server group；Described
When the operating status of other front servers is malfunction, the connection for the user terminal connecting with other front servers is received
Request, and establish communication connection.
In addition, the distributed data digging method, further includes:
The network address of first control server of the front server record as working node is saved as work
The network address of second control server of point.
Specifically, the front server by the multiple data mining subtask be assigned at multiple calculation servers into
Row processing, and the task feedback information of calculation server is received, and the task feedback information is sent to first control
Server, comprising:
The front server receives the heartbeat message of calculation server；The heartbeat message of the calculation server includes fortune
Calculate the task feedback information of server process data mining subtask and the cpu resource information of calculation server；
The heartbeat message of the calculation server is sent to first control server.
Furthermore the distributed data digging method, further includes:
First control server is when being sent to the front server for multiple data mining subtasks, to second
Control server sends data synchronization information；The data synchronization information include data mining subtask mission number and each number
According to the IP address for excavating the corresponding calculation server in subtask；
First control server sends the task feedback information real-time synchronization at the second control server, packet
It includes:
First control server believes the heartbeat of calculation server after receiving the heartbeat message of calculation server
Breath real-time synchronization is sent at the second control server.
Specifically, second control server monitors first control server in real time, first control is being confirmed
When control server breaks down, Xiang Suoshu front server sends working node and substitutes request, comprising:
Second control server sends heartbeat to first control server with prefixed time interval timing and asks
It asks；
If not receiving the first control server after continuous n times send heartbeat request to first control server
Heartbeat response message, it is determined that first control server breaks down, Xiang Suoshu front server send working node
Substitute request；Wherein n is pre-set frequency threshold value.
Specifically, second control server carries out pair the task situation information and the task feedback information
Than, determine the different information of the task situation information and the task feedback information, and according to pre-set processing strategie,
The different information is handled, comprising:
Second control server generates two parts of tasks according to the task situation information and the task feedback information
Inventory list；The task list list includes the IP address of calculation server and the cpu resource information of calculation server；
Second control server determines different information according to two parts of task list lists；
If the different information is that the first control server has been distributed to after calculation server due to the first control server
Failure is not synchronized to the task of the second control server, according to the number of the second control server of the task situation information update
According to synchronizing information；
If the different information is that the first control server has distributed to calculation server, and calculation server handles task
After failure, due to the first control server failure, it is not synchronized to the task of the second control server, from the task situation information
Middle acquisition mission failure information, and the corresponding data mining subtask of the mission failure information is redistributed；
If the different information is the still unappropriated data mining subtask of the first control server, will still unappropriated number
It is distributed to according to excavation subtask by front server and is handled at calculation server.
Further, the distributed data digging method, further includes:
First control server distributes a data mining subtask according to the cpu resource information of calculation server
To the maximum calculation server of cpu resource in each calculation server.
A kind of distributed data digging system and method provided in an embodiment of the present invention, in the first control as working node
When control server breaks down, the second control server, which can substitute first control server, becomes new working node,
To restore the normal work of entire distributed data digging system after completing working node switching.Certainly, when the second control
When server fail, the first control server can also substitute second control server, and two control servers can
To realize the hot standby pattern of principal and subordinate, avoiding the current JobTracker server as main controlled node will if breaking down
The problem of causing the task schedule system of entire hadoop frame to paralyse, being unable to complete task schedule and processing.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention without any creative labor, may be used also for those of ordinary skill in the art
To obtain other drawings based on these drawings.
Fig. 1 is a kind of structural schematic diagram one of distributed data digging system provided in an embodiment of the present invention；
Fig. 2 is a kind of flow chart one of distributed data digging method provided in an embodiment of the present invention；
Fig. 3 is a kind of flowchart 2 of distributed data digging method provided in an embodiment of the present invention.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall within the protection scope of the present invention.
As shown in Figure 1, the embodiment of the present invention provides a kind of distributed data digging system 10, comprising: user terminal group 11,
Front server group 12, the first control server 13, the second control server 14 and calculation server group 15；The user is whole
End group 11 includes multiple user terminals 111；The front server group 12 includes multiple front servers 121；The operation clothes
Device group 15 of being engaged in includes multiple calculation servers 151；The user terminal group 11 is communicated to connect with the front server group 12；Institute
State front server group 12 respectively with first control server 13, the second control server 14 and the calculation server group
15 communication connections.
Wherein, the user terminal 111, for sending data mining task request to the front server 121.
Herein, which can run the procedure script etc. of hadoop client or data mining task.
The front server 121, for parsing the domain-name information of the data mining task request, according to the data
Data mining task request is submitted to and services as the first control of working node by the domain-name information of mining task request
Device 13.
First control server 13, for requesting corresponding data mining task to carry out the data mining task
It decomposes, forms multiple data mining subtasks.The multiple data mining subtask is sent to the front server 121.
The front server 121 is also used to the multiple data mining subtask being assigned to multiple calculation servers
It is handled at 151, and receives the task feedback information of calculation server 151, and the task feedback information is sent to institute
State the first control server 13.
First control server 13 is also used to send the task feedback information real-time synchronization to the second control clothes
It is engaged at device 14.
Second control server 14, for monitoring first control server 13 in real time, in confirmation described first
When control server 13 breaks down, Xiang Suoshu front server 121 sends working node and substitutes request.
The front server 121 is also used to be substituted according to the working node and request, the second control described in more new record
The network address of server 14, so that second control server 14 is used as working node.
Second control server 14 is also used to collect to each 151 broadcasting tasks information of calculation server and request.
The calculation server 151 is also used to when listening to the broadcasting tasks information and collecting request, to described preposition
Server 121 feeds back task situation information.
The front server 121 is also used to the task situation information being sent to second control server 14.
Second control server 14, is also used to carry out the task situation information and the task feedback information pair
Than, determine the different information of the task situation information and the task feedback information, and according to pre-set processing strategie,
The different information is handled.
Further, the front server 121 is also used to obtain other preposition services in 121 groups of front server
The running state information of device 121.Other front servers 121 operating status be malfunction when, receive and other
The connection request for the user terminal that front server 121 connects, and establish communication connection.
Further, the front server 121 is also used to record the first control server 13 as working node
The network address of network address or the second control server 14 as working node.
In addition, the front server 121, specifically for receiving the heartbeat message of calculation server 151.The operation clothes
The heartbeat message of business device 151 includes the task feedback information and computational service that calculation server 151 handles data mining subtask
The cpu resource information of device 151.The heartbeat message of the calculation server 151 is sent to first control server 13.
In addition, first control server 13, specifically for multiple data mining subtasks are sent to it is described before
When setting server 121, data synchronization information is sent to the second control server 14.The data synchronization information includes data mining
The IP address of the corresponding calculation server 151 of the mission number of subtask and each data mining subtask.
After the heartbeat message for receiving calculation server 151, the heartbeat message real-time synchronization of calculation server 151 is sent out
It is sent at the second control server 14.
In addition, second control server 14, is specifically used for controlling with prefixed time interval timing to described first
Server 13 sends heartbeat request.If not received after continuous n times send heartbeat request to first control server 13
The heartbeat response message of first control server 13, it is determined that first control server 13 breaks down, to described preposition
Server 121 sends working node and substitutes request.Wherein n is pre-set frequency threshold value.
In addition, second control server 14, is specifically used for:
According to the task situation information and the task feedback information, two parts of task list lists are generated.The task
Inventory list includes the IP address of calculation server 151 and the cpu resource information of calculation server 151.
According to two parts of task list lists, different information is determined.
If the different information is that the first control server 13 has been distributed to after calculation server 151 due to the first control clothes
Business 13 failure of device, is not synchronized to the task of the second control server 14, is controlled and is taken according to the task situation information update second
The data synchronization information of business device 14.
If the different information is that the first control server 13 has distributed to calculation server 151, and calculation server 151
After handling mission failure, due to 13 failure of the first control server, it is not synchronized to the task of the second control server 14, from described
Mission failure information is obtained in task situation information, and the corresponding data mining subtask of the mission failure information is divided again
Match.
If the different information is the still unappropriated data mining subtask of the first control server 13, will be still unappropriated
It is distributed to by front server 121 and is handled at calculation server 151 in data mining subtask.
In addition, first control server 13, specifically for the cpu resource information according to calculation server 151, by one
Distribute to the maximum calculation server 151 of cpu resource in each calculation server 151 in data mining subtask.
A kind of distributed data digging system provided in an embodiment of the present invention, in the first control service as working node
When device breaks down, the second control server can substitute first control server as new working node, thus
After completing working node switching, restore the normal work of entire distributed data digging system.Certainly, when the second control server
When breaking down, the first control server can also substitute second control server, and two control servers may be implemented
The hot standby pattern of principal and subordinate, avoid the current JobTracker server as main controlled node will lead to if failure it is whole
The problem of task schedule system of a hadoop frame paralyses, and is unable to complete task schedule and processing.
As shown in Fig. 2, the embodiment of the present invention provides a kind of distributed data digging method, it is applied to above-mentioned shown in FIG. 1
Distributed data digging system, which comprises
Step 201, user terminal send data mining task request to the front server.
Step 202, front server parse the domain-name information of the data mining task request, according to the data mining
Data mining task request is submitted to the first control server as working node by the domain-name information of task requests.
The data mining task is requested corresponding data mining task to be divided by step 203, the first control server
Solution, forms multiple data mining subtasks, and the multiple data mining subtask is sent to the front server.
The multiple data mining subtask is assigned at multiple calculation servers by step 204, the front server
It is handled, and receives the task feedback information of calculation server, and the task feedback information is sent to first control
Control server.
The task feedback information real-time synchronization is sent the second control clothes by step 205, first control server
It is engaged at device.
Step 206, second control server monitor first control server in real time, are confirming first control
When control server breaks down, Xiang Suoshu front server sends working node and substitutes request.
Step 207, the front server substitute according to the working node requests, the second control clothes described in more new record
The network address of business device, so that second control server is as working node.
Step 208, second control server are collected to each calculation server broadcasting tasks information and are requested.
Step 209, the calculation server are when listening to the broadcasting tasks information collection request, to the preposition clothes
Business device feeds back task situation information.
The task situation information is sent to second control server by step 210, the front server.
Step 211, second control server carry out pair the task situation information and the task feedback information
Than, determine the different information of the task situation information and the task feedback information, and according to pre-set processing strategie,
The different information is handled.
A kind of distributed data digging method provided in an embodiment of the present invention, in the first control service as working node
When device breaks down, the second control server can substitute first control server as new working node, thus
After completing working node switching, restore the normal work of entire distributed data digging system.Certainly, when the second control server
When breaking down, the first control server can also substitute second control server, and two control servers may be implemented
The hot standby pattern of principal and subordinate, avoid the current JobTracker server as main controlled node will lead to if failure it is whole
The problem of task schedule system of a hadoop frame paralyses, and is unable to complete task schedule and processing.
In order to make those skilled in the art be better understood by the present invention, a more detailed embodiment is set forth below,
As shown in figure 3, the embodiment of the present invention provides a kind of distributed data digging method, comprising:
Step 301, user terminal and a front server, which are established, to be communicated to connect.
Step 302, each front server obtain the running state information of other front servers in front server group；
When the operating status of other front servers is malfunction, the user terminal connecting with other front servers is received
Connection request, and establish communication connection.
It herein, is the load balancing for realizing front server group, the present invention is using ZooKeeper as Distributed Application journey
Sequence coordination service, Zookeeper are usually made of multiple nodes (being herein each front server), and each node leads between each other
Cross the running state information that heartbeat obtains others, and in the memory of each node in store full dose data, node single-point therefore
Barrier will not influence the service ability of entire cluster.In the present invention, a section of each front server as Zookeeper
Point, after client is connected on some front server, client will remain this connection, and by this connection come
It sends request, obtain event notice, and send heartbeat, if connected break down, client can be connected to automatically
In addition on available node, the service cluster of a High Availabitity is finally formed, realizes the association between multiple distributed variable-frequencypumps
Biconditional operation.
Step 303, user terminal send data mining task request to the front server.
Step 304, front server parse the domain-name information of the data mining task request, according to the data mining
Data mining task request is submitted to the first control server as working node by the domain-name information of task requests.
Wherein, the front server record has the network address of the first control server as working node.At this point,
Second control server is as standby node.
The data mining task is requested corresponding data mining task to be divided by step 305, the first control server
Solution, forms multiple data mining subtasks, and the multiple data mining subtask is sent to the front server.
Step 306, the first control server when multiple data mining subtasks are sent to the front server, to
Second control server sends data synchronization information；The data synchronization information include data mining subtask mission number and
The IP address of the corresponding calculation server in each data mining subtask.
The multiple data mining subtask is assigned at multiple calculation servers by step 307, the front server
It is handled.
Step 308, front server receive the heartbeat message of calculation server, and the heartbeat of the calculation server is believed
Breath is sent to first control server.
Wherein, the heartbeat message of the calculation server includes that the task of calculation server processing data mining subtask is anti-
The cpu resource information of feedforward information and calculation server.
Step 309, the first control server can continue according to the cpu resource information of calculation server by a data mining
Distribute to the maximum calculation server of cpu resource in each calculation server in subtask.
Step 310, the first control server are after receiving the heartbeat message of calculation server, by the heart of calculation server
Hop-information real-time synchronization is sent at the second control server.
Step 311, the second control server send the heart to first control server with prefixed time interval timing
Request is jumped, if not receiving the first control server after continuous n times send heartbeat request to first control server
Heartbeat response message, it is determined that first control server breaks down, and Xiang Suoshu front server sends working node more
For request.
Wherein n is pre-set frequency threshold value.
Step 312, the front server substitute according to the working node requests, the second control clothes described in more new record
The network address of business device, so that second control server is as working node.
At this point, the front server needs to record the network address of the second control server as working node.
Step 313, second control server are collected to each calculation server broadcasting tasks information and are requested.
Step 314, the calculation server are when listening to the broadcasting tasks information collection request, to the preposition clothes
Business device feeds back task situation information.
The task situation information is sent to second control server by step 315, the front server.
Step 316, second control server are generated according to the task situation information and the task feedback information
Two parts of task list lists.
The task list list includes the IP address of calculation server and the cpu resource information of calculation server.
Step 317, second control server determine different information according to two parts of task list lists.
After step 317, step 318,319 or step 320 can be executed.
If step 318, the different information are that the first control server has been distributed to after calculation server due to the first control
Control server failure is not synchronized to the task of the second control server, is controlled and is taken according to the task situation information update second
The data synchronization information of business device.
If step 319, the different information are that the first control server has distributed to calculation server, and calculation server
After handling mission failure, due to the first control server failure, it is not synchronized to the task of the second control server, from the task
Mission failure information is obtained in situation information, and the corresponding data mining subtask of the mission failure information is redistributed.
If step 320, the different information are the still unappropriated data mining subtask of the first control server, will not yet
The data mining subtask of distribution is distributed to by front server to be handled at calculation server.
After the completion of step 318, step 319 or step 320, the second control server has been disposed as working node
Finish, working node is used as by the second control server, the continuation whole flow process of return step 304, second control server at this time
The course of work is identical as the first control server, it is subsequent monitoring the second control server work by the first control server Lai
It carries out, two such control server cycle alternation constitutes the hot standby pattern of principal and subordinate.It is worth noting that the first control clothes
Business device and " first ", " second " in the second control server are used for the purpose of two control servers of difference, and two control
The structure and function of server is essentially identical, and when one of them is as working node, another is as standby node.
A kind of distributed data digging method provided in an embodiment of the present invention, in the first control service as working node
When device breaks down, the second control server can substitute first control server as new working node, thus
After completing working node switching, restore the normal work of entire distributed data digging system.Certainly, when the second control server
When breaking down, the first control server can also substitute second control server, and two control servers may be implemented
The hot standby pattern of principal and subordinate, avoid the current JobTracker server as main controlled node will lead to if failure it is whole
The problem of task schedule system of a hadoop frame paralyses, and is unable to complete task schedule and processing.
It should be understood by those skilled in the art that, the embodiment of the present invention can provide as method, system or computer program
Product.Therefore, complete hardware embodiment, complete software embodiment or reality combining software and hardware aspects can be used in the present invention
Apply the form of example.Moreover, it wherein includes the computer of computer usable program code that the present invention, which can be used in one or more,
The computer program implemented in usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) produces
The form of product.
The present invention be referring to according to the method for the embodiment of the present invention, the process of equipment (system) and computer program product
Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions
The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer programs
Instruct the processor of general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices to produce
A raw machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for real
The device for the function of being specified in present one or more flows of the flowchart and/or one or more blocks of the block diagram.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates,
Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or
The function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting
Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or
The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one
The step of function of being specified in a box or multiple boxes.
Specific embodiment is applied in the present invention, and principle and implementation of the present invention are described, above embodiments
Explanation be merely used to help understand method and its core concept of the invention；At the same time, for those skilled in the art,
According to the thought of the present invention, there will be changes in the specific implementation manner and application range, in conclusion in this specification
Appearance should not be construed as limiting the invention.
Claims (14)
1. a kind of distributed data digging system characterized by comprising user terminal group, front server group, the first control
Server, the second control server and calculation server group；The user terminal group includes multiple user terminals；It is described preposition
Server group includes multiple front servers；The calculation server group includes multiple calculation servers；The user terminal group
It is communicated to connect with the front server group；The front server group is controlled with first control server, second respectively
Server and calculation server group communication connection；
The user terminal, for sending data mining task request to the front server；
The front server is appointed for parsing the domain-name information of the data mining task request according to the data mining
Data mining task request is submitted to the first control server as working node by the domain-name information of business request；
First control server, for requesting corresponding data mining task to decompose the data mining task,
Form multiple data mining subtasks；The multiple data mining subtask is sent to the front server；
The front server is also used to for being assigned at multiple calculation servers the multiple data mining subtask
Reason, and the task feedback information of calculation server is received, and the task feedback information is sent to first control and is serviced
Device；
First control server is also used to send the second control server for the task feedback information real-time synchronization
Place；
Second control server is confirming the first control service for monitoring first control server in real time
When device breaks down, Xiang Suoshu front server sends working node and substitutes request；
The front server is also used to be substituted according to the working node and request, the second control server described in more new record
Network address so that second control server is as working node；
Second control server is also used to collect to each calculation server broadcasting tasks information and request；
The calculation server is also used to when listening to the broadcasting tasks information collection request, Xiang Suoshu front server
Feedback task situation information；
The front server is also used to the task situation information being sent to second control server；
Second control server is also used to compare the task situation information and the task feedback information, really
The different information of fixed the task situation information and the task feedback information, and according to pre-set processing strategie, to institute
Different information is stated to be handled；
The front server is also used to obtain the running state information of other front servers in front server group；?
When the operating status of other front servers is malfunction, the user terminal connecting with other front servers is received
Connection request, and establish communication connection.
2. distributed data digging system according to claim 1, which is characterized in that the front server is also used to
Record the network address of the first control server as working node or the net of the second control server as working node
Network address.
3. distributed data digging system according to claim 2, which is characterized in that the front server, it is specific to use
In the heartbeat message for receiving calculation server；The heartbeat message of the calculation server includes calculation server processing data mining
The task feedback information of subtask and the cpu resource information of calculation server；The heartbeat message of the calculation server is sent
To first control server.
4. distributed data digging system according to claim 3, which is characterized in that first control server, tool
Body is used for when multiple data mining subtasks are sent to the front server, and it is same to send data to the second control server
Walk information；The data synchronization information includes the corresponding fortune of mission number and each data mining subtask of data mining subtask
Calculate the IP address of server；
After receiving the heartbeat message of calculation server, the second control is sent by the heartbeat message real-time synchronization of calculation server
At control server.
5. distributed data digging system according to claim 4, which is characterized in that second control server, tool
Body is used to send heartbeat request to first control server with prefixed time interval timing；If continuous n times are to described the
After one control server sends heartbeat request, the heartbeat response message of the first control server is not received, it is determined that described
First control server breaks down, and Xiang Suoshu front server sends working node and substitutes request；Wherein n is pre-set
Frequency threshold value.
6. distributed data digging system according to claim 5, which is characterized in that second control server, tool
Body is used for:
According to the task situation information and the task feedback information, two parts of task list lists are generated；The task list
List includes the IP address of calculation server and the cpu resource information of calculation server；
According to two parts of task list lists, different information is determined；
If the different information is that the first control server has been distributed to after calculation server due to the first control server failure,
It is not synchronized to the task of the second control server, it is synchronous according to the data of the second control server of the task situation information update
Information；
If the different information is that the first control server has distributed to calculation server, and calculation server handles mission failure
Afterwards, due to the first control server failure, it is not synchronized to the task of the second control server, is obtained from the task situation information
Mission failure information is taken, and the corresponding data mining subtask of the mission failure information is redistributed；
If the different information is the still unappropriated data mining subtask of the first control server, still unappropriated data are dug
Pick subtask is distributed to by front server to be handled at calculation server.
7. distributed data digging system according to claim 6, which is characterized in that first control server, tool
Body is used for the cpu resource information according to calculation server, and CPU in each calculation server is distributed in a data mining subtask and is provided
The maximum calculation server in source.
8. a kind of distributed data digging method, which is characterized in that be applied to the described in any item distributions of claim 1 to 7
Data digging system, the system include: user terminal group, front server group, the first control server, the second control server
And calculation server group；The user terminal group includes multiple user terminals；The front server group includes multiple preposition
Server；The calculation server group includes multiple calculation servers；The user terminal group and the front server group are logical
Letter connection；The front server group respectively with first control server, the second control server and the computational service
The communication connection of device group；
The described method includes:
User terminal sends data mining task request to the front server；
Front server parses the domain-name information of the data mining task request, the domain requested according to the data mining task
Data mining task request is submitted to the first control server as working node by name information；
The data mining task is requested corresponding data mining task to decompose by the first control server, forms multiple numbers
According to excavation subtask, and the multiple data mining subtask is sent to the front server；
The multiple data mining subtask is assigned at multiple calculation servers and handles by the front server, and connects
The task feedback information of calculation server is received, and the task feedback information is sent to first control server；
First control server sends the task feedback information real-time synchronization at the second control server；
Second control server monitors first control server in real time, is confirming the first control server generation
When failure, Xiang Suoshu front server sends working node and substitutes request；
The front server substitutes according to the working node requests, the network of the second control server described in more new record
Location, so that second control server is as working node；
Second control server is collected to each calculation server broadcasting tasks information and is requested；
For the calculation server when listening to the broadcasting tasks information collection request, Xiang Suoshu front server feeds back task
Situation information；
The task situation information is sent to second control server by the front server；
Second control server compares the task situation information and the task feedback information, determines described appoint
The different information for the situation information and the task feedback information of being engaged in, and according to pre-set processing strategie, the difference is believed
Breath is handled；
The front server obtains the running state information of other front servers in front server group；It is described other
When the operating status of front server is malfunction, the connection for receiving the user terminal connecting with other front servers is asked
It asks, and establishes communication connection.
9. distributed data digging method according to claim 8, which is characterized in that further include:
The front server records the network address of the first control server as working node or as working node
The network address of second control server.
10. distributed data digging method according to claim 9, which is characterized in that the front server will be described
Multiple data mining subtasks are assigned at multiple calculation servers and are handled, and receive the task feedback letter of calculation server
Breath, and the task feedback information is sent to first control server, comprising:
The front server receives the heartbeat message of calculation server；The heartbeat message of the calculation server includes operation clothes
The task feedback information of business device processing data mining subtask and the cpu resource information of calculation server；
The heartbeat message of the calculation server is sent to first control server.
11. distributed data digging method according to claim 10, which is characterized in that further include:
First control server is when being sent to the front server for multiple data mining subtasks, to the second control
Server sends data synchronization information；The data synchronization information includes that the mission number of data mining subtask and each data are dug
Dig the IP address of the corresponding calculation server in subtask；
First control server sends the task feedback information real-time synchronization at the second control server, comprising:
First control server is real by the heartbeat message of calculation server after receiving the heartbeat message of calculation server
When be simultaneously sent at the second control server.
12. distributed data digging method according to claim 11, which is characterized in that second control server is real
When monitor first control server, when confirming that first control server breaks down, Xiang Suoshu front server
It sends working node and substitutes request, comprising:
Second control server sends heartbeat request to first control server with prefixed time interval timing；
If not receiving the heart of the first control server after continuous n times send heartbeat request to first control server
Jump response message, it is determined that first control server breaks down, and Xiang Suoshu front server sends working node and substitutes
Request；Wherein n is pre-set frequency threshold value.
13. distributed data digging method according to claim 12, which is characterized in that second control server pair
The task situation information and the task feedback information compare, and determine the task situation information and task feedback
The different information of information, and according to pre-set processing strategie, the different information is handled, comprising:
Second control server generates two parts of task lists according to the task situation information and the task feedback information
List；The task list list includes the IP address of calculation server and the cpu resource information of calculation server；
Second control server determines different information according to two parts of task list lists；
If the different information is that the first control server has been distributed to after calculation server due to the first control server failure,
It is not synchronized to the task of the second control server, it is synchronous according to the data of the second control server of the task situation information update
Information；
If the different information is that the first control server has distributed to calculation server, and calculation server handles mission failure
Afterwards, due to the first control server failure, it is not synchronized to the task of the second control server, is obtained from the task situation information
Mission failure information is taken, and the corresponding data mining subtask of the mission failure information is redistributed；
If the different information is the still unappropriated data mining subtask of the first control server, still unappropriated data are dug
Pick subtask is distributed to by front server to be handled at calculation server.
14. distributed data digging method according to claim 13, which is characterized in that further include:
First control server distributes to a data mining subtask respectively according to the cpu resource information of calculation server
The maximum calculation server of cpu resource in calculation server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610957904.2A CN106452899B (en) | 2016-10-27 | 2016-10-27 | A kind of distributed data digging system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610957904.2A CN106452899B (en) | 2016-10-27 | 2016-10-27 | A kind of distributed data digging system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106452899A CN106452899A (en) | 2017-02-22 |
| CN106452899B true CN106452899B (en) | 2019-09-06 |
Family
ID=58179730
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610957904.2A Active CN106452899B (en) | 2016-10-27 | 2016-10-27 | A kind of distributed data digging system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106452899B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107025288A (en) * | 2017-04-14 | 2017-08-08 | 四川九鼎瑞信软件开发有限公司 | Distributed data digging method and system |
| CN109033196A (en) * | 2018-06-28 | 2018-12-18 | 北京奇虎科技有限公司 | A kind of distributed data scheduling system and method |
| CN110298549B (en) * | 2019-05-30 | 2022-05-06 | 广东省机场管理集团有限公司工程建设指挥部 | Project task processing method, device, equipment and medium for airport construction engineering |
| CN110503431A (en) * | 2019-08-07 | 2019-11-26 | 广州子牛区块链技术有限公司 | A kind of personal node type dividend cell system based on block chain technology |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102567396A (en) * | 2010-12-30 | 2012-07-11 | ***通信集团公司 | Method, system and device for data mining on basis of cloud computing |
| CN102841822A (en) * | 2011-06-23 | 2012-12-26 | 腾讯科技（深圳）有限公司 | Method and system for performing crash protection on jobTracker hosts |
| CN103220183A (en) * | 2013-05-02 | 2013-07-24 | 杭州电子科技大学 | Implement method of Hadoop high-availability system based on double-main-engine warm backup |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8799006B2 (en) * | 2004-12-30 | 2014-08-05 | Cerner Innovation, Inc. | System and methods for distributed analysis of patient records |
| US9122535B2 (en) * | 2011-11-22 | 2015-09-01 | Netapp, Inc. | Optimizing distributed data analytics for shared storage |
-
2016
- 2016-10-27 CN CN201610957904.2A patent/CN106452899B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102567396A (en) * | 2010-12-30 | 2012-07-11 | ***通信集团公司 | Method, system and device for data mining on basis of cloud computing |
| CN102841822A (en) * | 2011-06-23 | 2012-12-26 | 腾讯科技（深圳）有限公司 | Method and system for performing crash protection on jobTracker hosts |
| CN103220183A (en) * | 2013-05-02 | 2013-07-24 | 杭州电子科技大学 | Implement method of Hadoop high-availability system based on double-main-engine warm backup |
Non-Patent Citations (1)
| Title |
|---|
| 《MapReduce故障容错研究与作业调度器优化》;刘明刚;《中国优秀硕士学位论文全文数据库 信息科技辑》;20131215;摘要，第2-3章 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106452899A (en) | 2017-02-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106452899B (en) | A kind of distributed data digging system and method | |
| Taherizadeh et al. | Key influencing factors of the Kubernetes auto-scaler for computing-intensive microservice-native cloud-based applications | |
| CN106663030B (en) | Scalable failover communication in distributed clusters | |
| CN107959705B (en) | Distribution method of streaming computing task and control server | |
| CN110209492B (en) | Data processing method and device | |
| CN104461752B (en) | A kind of multimedia distributed task processing method of two-stage failure tolerant | |
| Meng et al. | State monitoring in cloud datacenters | |
| CN103383689A (en) | Service process fault detection method, device and service node | |
| EP3958508A1 (en) | Data processing method and device, storage medium and electronic device | |
| CN110855737B (en) | Consistency level controllable self-adaptive data synchronization method and system | |
| CN109343939A (en) | A kind of distributed type assemblies and parallel computation method for scheduling task | |
| WO2018131556A1 (en) | Resource setting control device, resource setting control system, resource setting control method, and computer-readable recording medium | |
| Hao et al. | {EdgeCons}: Achieving Efficient Consensus in Edge Computing Networks | |
| CN112492022A (en) | Cluster, method, system and storage medium for improving database availability | |
| Guerraoui et al. | Can 100 machines agree? | |
| CN103108000A (en) | Task synchronization method and system and host node and work nodes in system | |
| Ali et al. | Probabilistic normed load monitoring in large scale distributed systems using mobile agents | |
| CN116346834A (en) | Session synchronization method, device, computing equipment and computer storage medium | |
| Stanik et al. | Failover pattern with a self-healing mechanism for high availability cloud solutions | |
| CN110795225B (en) | Method and system for executing scheduling task | |
| CN114760304A (en) | Computing power information processing method and system and computing power gateway | |
| CN107040388A (en) | Charge system and method | |
| CN109388502A (en) | A kind of service identification distribution method and device | |
| CN110647440A (en) | Big data task processing method and system based on state machine | |
| JP2008182411A (en) | Information distribution program, information distribution device, and information distribution method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |