CN106385355B - The method, apparatus and system of operation are submitted to hadoop cluster - Google Patents
The method, apparatus and system of operation are submitted to hadoop cluster Download PDFInfo
- Publication number
- CN106385355B CN106385355B CN201610806358.2A CN201610806358A CN106385355B CN 106385355 B CN106385355 B CN 106385355B CN 201610806358 A CN201610806358 A CN 201610806358A CN 106385355 B CN106385355 B CN 106385355B
- Authority
- CN
- China
- Prior art keywords
- equipment
- address
- vpn
- subnet
- ide
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
- H04L12/46—Interconnection of networks
- H04L12/4641—Virtual LANs, VLANs, e.g. virtual private networks [VPN]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Small-Scale Networks (AREA)
Abstract
The invention discloses the method, apparatus and system that operation is submitted to hadoop cluster, the described method includes: when IDE equipment needs to submit operation to the hadoop cluster in different network environments, by cooperating with the management equipment in hadoop cluster, the network environment with hadoop cluster is got through, and the operation of generation is submitted to the working equipment for running the operation in hadoop cluster.Using scheme of the present invention, the success rate etc. of operation submission can be improved.
Description
[technical field]
The present invention relates to network technologies, in particular to the method, apparatus and system of operation are submitted to hadoop cluster.
[background technique]
Hadoop is a distributed system infrastructure developed by apache foundation, and user can not know about
In the case where distributed bottom level details, distributed program is developed, the power of cluster is made full use of to carry out high speed computing and storage.
Currently, debugging hadoop is an extremely complex process, need by writing, compiling, be packaged etc. to submit work
Industry runs operation, checks the series of steps such as job run result.
Wherein, the mode of operation is submitted can include: pass through Integrated Development Environment (IDE, Integrated Devolopment
Environment) tool submits the mode of operation, and, in such a way that plug-in unit submits operation etc..
May include following steps 1 in such a way that IDE tool submits operation by taking Eclipse as an example)~4).
1) hadoop cluster is built in home environment, and starts another resource coordination person (YARN, Yet
Another Resource Negotiator), distributed file system (HDFS, Hadoop Distributed File
The related services process such as System).
2) on Eclipse create a Java engineering, and import hadoop application programming interface (API,
Application Programming Interface), i.e. Jar file under hadoop installation file lib catalogue.
3) operation procedure is write, a WordCount operation is such as write, the number that word occurs in statistics file, and will
Its compiling is packaged into Jar file, is placed under Java engineering.
4) a Main method is write, a Job object is constructed, sets the parameters such as HDFS, JobTracker, Jar file,
Then this Main method is run, Fig. 1 is the schematic diagram of existing Main method.
Similar in such a way that IDE tool submits operation by way of plug-in unit submits operation, difference, which is only that, to be needed
One hadoop plug-in unit to be installed in Eclipse, is packaged so as to automatically be compiled operation by the hadoop plug-in unit
At Jar file, and then it is committed to hadoop cluster, without being packaged manually.
But above two mode can have certain problems in practical applications, such as: IDE tool and hadoop collection
Group necessarily is in the same network environment, such as when hadoop cluster is in virtual private cloud (VPC, Virtual Private
When Cloud) in subnet, then operation can not be submitted by above two mode, to reduce the success rate of operation submission.
[summary of the invention]
The present invention provides the method, apparatus and system that operation is submitted to hadoop cluster, can be improved operation submission
Success rate.
Specific technical solution is as follows:
A kind of management equipment in hadoop cluster, comprising: first processing module；
The first processing module, for needing when Integrated Development Environment IDE equipment in different network environments
When hadoop cluster submits operation, by cooperating with the IDE equipment, the IDE equipment and the hadoop collection are got through
Network environment locating for group, and the operation that the IDE equipment is submitted is transmitted in the hadoop cluster and runs the operation
Working equipment.
A kind of Integrated Development Environment IDE equipment, comprising: IDE module and Second processing module；
The Second processing module, for submitting operation to the hadoop cluster in different network environments when needs
When, by cooperating with the management equipment in the hadoop cluster, get through the IDE equipment and the hadoop cluster
Network environment；
The IDE module, for the operation of generation to be submitted to the work for running the operation in the hadoop cluster
Equipment.
A kind of system from operation to hadoop cluster that submitting, comprising:
Management equipment as described above, and, IDE equipment as described above.
A method of operation is submitted to hadoop cluster, comprising:
When Integrated Development Environment IDE equipment needs to submit operation to the hadoop cluster in different network environments,
Management equipment in the hadoop cluster gets through the IDE equipment and described by cooperating with the IDE equipment
Network environment locating for hadoop cluster, and the operation that the IDE equipment is submitted is transmitted in the hadoop cluster and is run
The working equipment of the operation.
A method of operation is submitted to hadoop cluster, comprising:
When IDE equipment needs to submit operation to the hadoop cluster in the different network environments, by with it is described
Management equipment in hadoop cluster cooperates, and gets through the network environment with the hadoop cluster, and by the operation of generation
Submit to the working equipment that the operation is run in the hadoop cluster.
It can be seen that based on above-mentioned introduction using scheme of the present invention, when IDE equipment and hadoop cluster are in difference
Network environment when, the network environment that both can be got through, and based on the network environment after getting through, by IDE tool come to
Hadoop cluster submits operation, to overcome in the prior art when IDE equipment and hadoop cluster are in different network rings
The problem of operation can not be submitted when border, and then improve the success rate of operation submission.
[Detailed description of the invention]
Fig. 1 is the schematic diagram of existing Main method.
Fig. 2 is the composed structure schematic diagram of the management equipment embodiment in hadoop cluster of the present invention.
Fig. 3 is the network distribution schematic diagram of IDE equipment of the present invention and hadoop cluster.
Fig. 4 is the composed structure schematic diagram of IDE apparatus embodiments of the present invention.
Fig. 5 is the schematic diagram of Main method of the present invention.
Fig. 6 is the schematic diagram of the operation result of WordCount operation of the present invention
[specific embodiment]
In the prior art, operation can not be submitted to the hadoop cluster in VPC subnet by IDE tool, mainly
Because they are in different network environments, therefore, if it is desired to by IDE tool come to the hadoop in the VPC subnet
Cluster submits operation, then the network environment for both needing to get through, and allows equipment i.e. IDE equipment and hadoop cluster where IDE tool
In same local area network.
In order to be clearer and more clear technical solution of the present invention, hereinafter, referring to the drawings and the embodiments, to institute of the present invention
The scheme of stating is described in further detail.
Embodiment one
Fig. 2 is the composed structure schematic diagram of management equipment (master) embodiment in hadoop cluster of the present invention,
As shown in Figure 2, comprising: first processing module 21.
First processing module 21, for needing to submit to the hadoop cluster in different network environments when IDE equipment
When operation, by cooperating with IDE equipment, network environment locating for IDE equipment and hadoop cluster is got through, and IDE is set
The standby operation submitted is transmitted to the working equipment (slave) that operation is run in hadoop cluster.
As previously mentioned, hadoop cluster can be in VPC subnet, the network environment after getting through can be Virtual Private Network
(VPN, Virtual Private Network) subnet, i.e., using VPN technologies construct a VPN subnet, allow IDE equipment and
Hadoop cluster is in the same subnet.
It certainly, is above the technical solution for example, being not intended to restrict the invention, such as when hadoop cluster is in
When in other subnets, scheme of the present invention is equally applicable, and constructed subnet is also not necessarily limited to VPN subnet, can be according to reality
Depending on border needs.
When hadoop cluster is in VPC subnet, constructed subnet is VPN subnet, Fig. 3 is IDE of the present invention
The network distribution schematic diagram of equipment and hadoop cluster, as shown in figure 3, being directed to three networks, respectively public network, VPN
Net and VPC subnet.
Correspondingly, in first processing module 21 can include: VPN service terminal unit 211 and VPC processing unit 212.
VPN service terminal unit 211, for establishing connection by the VPN client unit in public network address and IDE equipment,
A VPN subnet is constructed, and returns to the network segment address and VPN service terminal unit 211 of VPC subnet to VPN client unit
VPN subnet address, to be routed below VPN client building unit: destination address be VPC subnet network segment address message
It is sent to the VPN subnet address of VPN service terminal unit 211.
It is provided with iptables rule on VPN service terminal unit 211, is VPN by institute's source address according to the rule
The network segment address and destination address of net are the message of the VPC subnet address of working equipment, are sent to VPC processing unit 212.
VPC processing unit 212 have forwarding capability (IP forwarding), by it is received from VPN service terminal unit 211,
IDE module in IDE tool is transmitted to the work that operation is run in hadoop cluster by the operation that VPN client unit is submitted
Equipment.
That is, VPN client unit and VPN service terminal unit 211 have a public network address and a VPN respectively
Subnet address, by public network address, the accessible VPN service terminal unit 211 of VPN client unit, to construct a VPN
Subnet.
As VPN subnet and the difference of network segment locating for VPC subnet, directly can not also be accessed in VPC by VPN subnet
Hadoop cluster in net, therefore iptables rule can be set on VPN service terminal unit 211, to allow institute's source address
Network segment address and destination address for VPN subnet are the message of the VPC subnet address of working equipment, are sent to VPC processing unit
212 corresponding VPC subnet address, i.e. the IP address 192.168.0.1 of network interface card nic3 shown in Fig. 3, in this way, being set by IDE
Standby upper nic2 can access the IP address 192.168.0.1 of the nic3 in management equipment with IP address 10.0.0.6 how
Setting iptables is the prior art.
But aforesaid way is the nic3 got through in IDE equipment and management equipment, in order to which IDE equipment can be visited
Ask the working equipment in hadoop cluster, it is also necessary to which the forwarding capability for opening VPC processing unit 212 forwards mesh by nic3
Address be hadoop cluster in working equipment VPC subnet address message.
It needs to explain, why not directly get through public network address and VPC subnet? the reason is that can not know public affairs in advance
What net address is, if all public network address messages to be imported to the nic3 in management equipment, no matter what public affairs is passed through in this way
Net address can access VPC subnet, not can guarantee the safety of VPC subnet.And subnet is constructed by VPN technologies and is able to use
The IP address of fixed network segment, so as to only allow the source address of this network segment to access VPC subnet, and VPC subnet has safety
Verifying function only can be added VPN subnet by the VPN client that certificate verifies, and then ensure that the safety of VPC subnet
Property.
By above-mentioned processing, IDE equipment and hadoop cluster only have been got through at IP layers, and it is general to access hadoop cluster
Domain name is used, therefore needs to build name server in VPC subnet also to carry out domain name mapping.Scheme of the present invention
In bind external member tool can be used, name server is built in management equipment, correspondingly, as shown in Fig. 2, in management equipment also
Can further comprise: domain name service module 22.
Domain name service module 22, for distinguish preservation management equipment and working equipment domain name and corresponding VPC subnet
Location；When the domain name for receiving the management equipment that IDE module is sent by VPN client unit and VPN service terminal unit 211
When, the corresponding VPC subnet address of the domain name of management equipment is returned into IDE module；Pass through VPN client when receiving IDE module
When the domain name of the working equipment for the operation operation that end unit and VPN service terminal unit 211 are sent, the work for running operation is set
The corresponding VPC subnet address of standby domain name returns to IDE module.
VPN service terminal unit 211 can be further used for, and be by network segment address and destination address that source address is VPN subnet
The message of the VPC subnet address of management equipment, is sent to VPC processing unit 212；The VPC subnet address of management equipment is equal to VPC
The corresponding VPC subnet address of processing unit 212.
VPC processing unit 212 can be further used for, and be serviced when receiving IDE module by VPN client unit and VPN
When the parameter information for the operation that end unit 211 is sent, by analyzing the parameter information of operation, operation operation is determined
Working equipment, and the domain name for running the working equipment of operation is returned into IDE module.
Embodiment two
Fig. 4 is the composed structure schematic diagram of IDE apparatus embodiments of the present invention, as shown in figure 4, including IDE module 41
With Second processing module 42.
Second processing module 42, for when needing to submit operation to the hadoop cluster in the different network environments,
By cooperating with the management equipment in hadoop cluster, the network environment of IDE equipment and hadoop cluster is got through；
IDE module 41, for the operation of generation to be submitted to the working equipment for running operation in hadoop cluster.
Hadoop cluster can be in VPC subnet, and the network environment after getting through can be VPN subnet, correspondingly, second processing
In module 42 can include: VPN client unit 421.
VPN client unit 421 connects for being established by the VPN service terminal unit 211 in public network address and management equipment
It connects, constructs a VPN subnet, and the network segment address of the VPC subnet returned according to VPN service terminal unit 211 and VPN are serviced
The following routing of VPN subnet address building of end unit 211: destination address is that the message of the network segment address of VPC subnet is sent to VPN
Service the VPN subnet address of end unit 211.
Wherein, it is provided with iptables rule on VPN service terminal unit 211, according to the rule, is by institute's source address
The network segment address and destination address of VPN subnet are the message of the VPC subnet address of working equipment, are sent in management equipment
VPC processing unit 212, VPC processing unit 212 has forwarding capability, by the IDE module 41 received from VPN service terminal unit 211
The operation submitted by VPN client unit 421 is transmitted to the working equipment that operation is run in hadoop cluster.
In addition, IDE module 41 can be further used for,
By VPN client unit 421 and VPN service terminal unit 211, the domain name of management equipment is sent to management equipment
In domain name service module 22, and receive domain name service module 22 return management equipment the corresponding VPC subnet address of domain name；
According to the corresponding VPC subnet address of the domain name of management equipment, the parameter information of operation is sent to VPC processing unit
212, and receive the working equipment of the operation operation returned after VPC processing unit 212 analyzes the parameter information of operation
Domain name；
The domain name for running the working equipment of operation is sent to domain name service module 22, and receives domain name service module 22 and returns
The corresponding VPC subnet address of domain name of the working equipment for the operation operation returned, according to the VPC subnet of the working equipment of operation operation
Address submits a job to the working equipment of operation operation.
Embodiment three
A kind of system for submitting operation to hadoop cluster is disclosed in the present embodiment, comprising: as described in embodiment one
Management equipment and the IDE equipment as described in embodiment two.
Based on the introduction in embodiment one and embodiment two, and referring to fig. 3, the work of system described in the present embodiment
Process may include following steps 1)~4).
1) VPN client unit 421 obtains VPN configuration information, such as may include the public network address of VPN service terminal unit 211
And certification certificate etc., it how to be retrieved as the prior art, and connection is established by public network address and VPN service terminal unit 211.
Later, VPN service terminal unit 211 can push the network segment address of VPC subnet to VPN client unit 421
192.168.0.0/24, the VPN subnet address 10.0.0.1 of VPN service terminal unit 211 and VPC of domain name service module 22
The information such as net address.
VPN client unit 421 can be routed according to the information architecture received one, i.e., destination address is
192.168.0.0/24 message be sent to nic2:10.0.0.1.
2) IDE module 41 obtains hadoop cluster configuration information, and creates a Java engineering, and hadoop cluster is configured
Information imports Java engineering.
It may include the management in the operating parameter and hadoop cluster of hadoop cluster in hadoop cluster configuration information
How equipment and the domain name of working equipment etc. are retrieved as the prior art.
3) IDE module 41 writes operation procedure, WordCount operation as described above, and is compiled and be packaged into Jar
File is placed under Java engineering.
4) IDE module 41 writes a Main method, specify 3) obtained in Jar file, then run this side Main
Method can submit operation to hadoop cluster.
Specifically, the domain name of management equipment can be sent to domain name service module 22 by IDE module 41, inquire domain name service mould
Block 22 is with the corresponding VPC subnet address of the domain name for obtaining management equipment；Later, IDE module 41 can be according to the VPC subnet got
The parameter information of operation is sent to VPC processing unit 212 by address, which content the parameter information of operation specifically includes can root
Depending on actual needs, after VPC processing unit 212 analyzes the parameter information of operation, the work of operation operation is determined
Equipment, and the domain name for running the working equipment of operation is returned into IDE module 41, i.e. notice IDE module 41 needs to mention operation
Which working equipment given；Later, the domain name for running the working equipment of operation can be sent to domain name service module by IDE module 41
22, and the corresponding VPC subnet address of domain name of the working equipment of the operation operation of the return of domain name service module 22 is obtained, and then will
The working equipment of operation operation is submitted in operation.
Wherein, the communication between IDE module 41 and VPC processing unit 212 and domain name service module 22 etc. is required to borrow
Help VPN client unit 421 and VPN service terminal unit 211.
In addition, job run result can be checked in the standard output of IDE module 41 according to the prior art.
Fig. 5 is the schematic diagram of Main method of the present invention, and Fig. 6 is the operation knot of WordCount operation of the present invention
The schematic diagram of fruit.
Example IV
A kind of method for submitting operation to hadoop cluster is disclosed in the present embodiment, comprising: when IDE equipment is needed to place
When hadoop cluster in different network environments submits operation, management equipment in hadoop cluster by with IDE equipment into
Row cooperation, gets through the network environment of IDE equipment and hadoop cluster, and the operation that IDE equipment is submitted is transmitted to hadoop collection
The working equipment of operation is run in group.
When network environment after hadoop cluster is in VPC subnet, gets through is VPN subnet, get through IDE equipment and
The network environment of hadoop cluster, and the work that the operation that IDE equipment is submitted is transmitted to operation operation in hadoop cluster is set
Standby specific implementation can include:
Management equipment establishes connection by public network address and IDE equipment, constructs a VPN subnet, and return to IDE equipment
The network segment address of VPC subnet and the VPN subnet address of management equipment are returned, to route below IDE device build: destination address
The VPN subnet address of management equipment is sent to for the message of the network segment address of VPC subnet；
It is provided with iptables rule in management equipment, is the network segment address of VPN subnet by institute's source address according to rule
And destination address is the message of the VPC subnet address of working equipment, is sent to the VPC subnet address of management equipment, and management is set
Standby open has forwarding capability, can by the VPC subnet address for being sent to management equipment, IDE equipment submit to the work of operation operation
The operation for making equipment is transmitted to the working equipment of operation operation.
Embodiment five
Also disclosed in the present embodiment it is a kind of to hadoop cluster submit operation method, comprising: IDE equipment need to
Hadoop cluster in different network environments submits operation, by cooperating with the management equipment in hadoop cluster,
The network environment with hadoop cluster is got through, and the work that operation operation in hadoop cluster is submitted in the operation of generation is set
It is standby.
When network environment after hadoop cluster is in VPC subnet, gets through is VPN subnet, get through and hadoop collection
The network environment of group, and the specific implementation that the working equipment of operation operation in hadoop cluster is submitted in the operation of generation can be wrapped
It includes:
IDE equipment establishes connection by public network address and management equipment, constructs a VPN subnet, and set according to management
The following routing of VPN subnet address building of the network segment address and management equipment of the standby VPC subnet returned: destination address VPC
The message of the network segment address of subnet is sent to the VPN subnet address of management equipment.
Wherein, it is provided with iptables rule in management equipment, is the net of VPN subnet by institute's source address according to rule
Sector address and destination address are the message of the VPC subnet address of working equipment, are sent to the VPC subnet address of management equipment, and
Management equipment unlatching have forwarding capability, can by the VPC subnet address for being sent to management equipment, IDE equipment submit to operation make
The operation of the working equipment of industry is transmitted to the working equipment of operation operation.
It, can when IDE equipment and hadoop cluster are in different network environments in short, using scheme of the present invention
The network environment of the two is got through, and based on the network environment after getting through, operation is submitted to hadoop cluster by IDE tool,
Asking for operation can not be submitted when IDE equipment and hadoop cluster are in different network environments in the prior art to overcome
Topic, and then improve the success rate of operation submission.
In several embodiments provided by the present invention, it should be understood that disclosed system, apparatus and method, it can be with
It realizes by another way.For example, apparatus embodiments described above are merely indicative, for example, the unit
It divides, only a kind of logical function partition, there may be another division manner in actual implementation.
The unit as illustrated by the separation member may or may not be physically separated, aobvious as unit
The component shown may or may not be physical unit, it can and it is in one place, or may be distributed over multiple
In network unit.It can select some or all of unit therein according to the actual needs to realize the mesh of this embodiment scheme
's.
It, can also be in addition, the functional units in various embodiments of the present invention may be integrated into one processing unit
It is that each unit physically exists alone, can also be integrated in one unit with two or more units.Above-mentioned integrated list
Member both can take the form of hardware realization, can also realize in the form of hardware adds SFU software functional unit.
The above-mentioned integrated unit being realized in the form of SFU software functional unit can store and computer-readable deposit at one
In storage media.Above-mentioned SFU software functional unit is stored in a storage medium, including some instructions are used so that a computer
It is each that equipment (can be personal computer, server or the network equipment etc.) or processor (processor) execute the present invention
The part steps of embodiment the method.And storage medium above-mentioned include: USB flash disk, mobile hard disk, read-only memory (ROM,
Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic or disk etc. it is various
It can store the medium of program code.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all in essence of the invention
Within mind and principle, any modification, equivalent substitution, improvement and etc. done be should be included within the scope of the present invention.
Claims (11)
1. the management equipment in a kind of hadoop cluster characterized by comprising first processing module；
The first processing module, for needing when Integrated Development Environment IDE equipment in different network environments
When hadoop cluster submits operation, by cooperating with the IDE equipment, the IDE equipment and the hadoop collection are got through
Network environment locating for group, and the operation that the IDE equipment is submitted is transmitted in the hadoop cluster and runs the operation
Working equipment；
Wherein, the hadoop cluster is in virtual private cloud VPC subnet；
It is described get through after network environment include: Virtual Private Network VPN subnet.
2. management equipment according to claim 1, which is characterized in that
It include: VPN service terminal unit and VPC processing unit in the first processing module；
The VPN service terminal unit, for establishing connection by the VPN client unit in public network address and the IDE equipment,
A VPN subnet is constructed, and returns to the network segment address and VPN clothes of the VPC subnet to the VPN client unit
The VPN subnet address of business end unit, to route below the VPN client building unit: destination address is the VPC subnet
The message of network segment address be sent to the VPN subnet address of the VPN service terminal unit；
It is provided with iptables rule on the VPN service terminal unit, is the VPN by institute's source address according to the rule
The network segment address and destination address of subnet are the message of the VPC subnet address of the working equipment, and it is single to be sent to the VPC processing
Member；
The VPC processing unit has forwarding capability, will be received from the VPN service terminal unit, described IDE equipment
The operation that IDE module is submitted by the VPN client unit is transmitted to the work that the operation is run in the hadoop cluster
Make equipment.
3. management equipment according to claim 2, which is characterized in that
The management equipment further comprises: domain name service module；
Domain name service module, for save respectively the management equipment and the working equipment domain name and corresponding VPC
Subnet address；When receiving what the IDE module was sent by the VPN client unit and the VPN service terminal unit
When the domain name of the management equipment, the corresponding VPC subnet address of the domain name of the management equipment is returned into the IDE module；
When the operation work for receiving the IDE module and being sent by the VPN client unit and the VPN service terminal unit
When the domain name of the working equipment of industry, the corresponding VPC subnet address of the domain name for running the working equipment of the operation is returned into institute
State IDE module；
The VPN service terminal unit is further used for, and is institute by network segment address and destination address that source address is the VPN subnet
The message for stating the VPC subnet address of management equipment is sent to the VPC processing unit；
The VPC processing unit is further used for, and passes through the VPN client unit and described when receiving the IDE module
When the parameter information for the operation that VPN service terminal unit is sent, by analyzing the parameter information of the operation, really
The working equipment for running the operation is made, and the domain name for running the working equipment of the operation is returned into the IDE module.
4. a kind of Integrated Development Environment IDE equipment characterized by comprising IDE module and Second processing module；
The Second processing module, for leading to when needing to submit operation to the hadoop cluster in different network environments
It crosses and cooperates with the management equipment in the hadoop cluster, get through the network of the IDE equipment and the hadoop cluster
Environment；
The IDE module, for the operation of generation to be submitted to the working equipment for running the operation in the hadoop cluster；
Wherein, the hadoop cluster is in virtual private cloud VPC subnet；
It is described get through after network environment include: Virtual Private Network VPN subnet.
5. IDE equipment according to claim 4, which is characterized in that
It include: VPN client unit in the Second processing module；
The VPN client unit connects for being established by the VPN service terminal unit in public network address and the management equipment
It connects, constructs a VPN subnet, and according to the network segment address of the VPC subnet of VPN service terminal unit return and institute
State the following routing of VPN subnet address building of VPN service terminal unit: destination address is the report of the network segment address of the VPC subnet
Wen Jun is sent to the VPN subnet address of the VPN service terminal unit；
Wherein, it is provided with iptables rule on the VPN service terminal unit, is institute by institute's source address according to the rule
The network segment address of VPN subnet and destination address are stated as the message of the VPC subnet address of the working equipment, is sent to the management
VPC processing unit in equipment, the VPC processing unit have forwarding capability, by it is received from the VPN service terminal unit,
The operation that the IDE module is submitted by the VPN client unit, is transmitted in the hadoop cluster and runs the operation
Working equipment.
6. IDE equipment according to claim 5, which is characterized in that
The IDE module is further used for,
By the VPN client unit and the VPN service terminal unit, the domain name of the management equipment is sent to the pipe
The domain name service module in equipment is managed, and the domain name for receiving the management equipment of domain name service module return is corresponding
VPC subnet address；
According to the corresponding VPC subnet address of the domain name of the management equipment, the parameter information of the operation is sent to the VPC
Processing unit, and receive the operation work returned after the VPC processing unit analyzes the parameter information of the operation
The domain name of the working equipment of industry；
The domain name for running the working equipment of the operation is sent to domain name service module, and receives domain name service mould
The corresponding VPC subnet address of domain name of the working equipment for the operation operation that block returns, according to the work for running the operation
The working equipment for running the operation is submitted in the operation by the VPC subnet address of equipment.
7. a kind of system for submitting operation to hadoop cluster characterized by comprising
Management equipment according to any one of claims 1 to 3, and, the IDE as described in any one of claim 4~6
Equipment.
8. a kind of method for submitting operation to hadoop cluster characterized by comprising
It is described when Integrated Development Environment IDE equipment needs to submit operation to the hadoop cluster in different network environments
Management equipment in hadoop cluster gets through the IDE equipment and the hadoop collection by cooperating with the IDE equipment
Network environment locating for group, and the operation that the IDE equipment is submitted is transmitted in the hadoop cluster and runs the operation
Working equipment；
Wherein, the hadoop cluster is in virtual private cloud VPC subnet；
It is described get through after network environment include: Virtual Private Network VPN subnet.
9. according to the method described in claim 8, it is characterized in that,
The network environment for getting through the IDE equipment and the hadoop cluster, and the operation that the IDE equipment is submitted turns
It issues and runs the working equipment of the operation in the hadoop cluster and include:
The management equipment establishes connection by public network address and the IDE equipment, constructs a VPN subnet, and to described
IDE equipment returns to the network segment address of the VPC subnet and the VPN subnet address of the management equipment, so as to the IDE equipment
The following routing of building: destination address is that the message of the network segment address of the VPC subnet is sent to the VPN subnet of the management equipment
Address；
It is provided with iptables rule in the management equipment, is the VPN subnet by institute's source address according to the rule
Network segment address and destination address are that the message of the VPC subnet address of the working equipment is sent to VPC of the management equipment
Net address, and management equipment unlatching has forwarding capability, can will be sent to the VPC subnet address of the management equipment, institute
It states IDE equipment and submits to and run the operation of the working equipment of the operation and be transmitted to the working equipment for running the operation.
10. a kind of method for submitting operation to hadoop cluster characterized by comprising
When IDE equipment needs to submit operation to the hadoop cluster in the different network environments, by with the hadoop
Management equipment in cluster cooperates, and gets through the network environment with the hadoop cluster, and the operation of generation is submitted to
The working equipment of the operation is run in the hadoop cluster；
Wherein, the hadoop cluster is in virtual private cloud VPC subnet；
It is described get through after network environment include: Virtual Private Network VPN subnet.
11. according to the method described in claim 10, it is characterized in that,
The network environment got through with the hadoop cluster, and the operation of generation is submitted in the hadoop cluster and is transported
The working equipment of the row operation includes:
The IDE equipment establishes connection by public network address and the management equipment, constructs a VPN subnet, and according to institute
The VPN subnet address of the network segment address and the management equipment of stating the VPC subnet of management equipment return constructs following road
By: destination address is that the message of the network segment address of the VPC subnet is sent to the VPN subnet address of the management equipment；
Wherein, it is provided with iptables rule in the management equipment, is the VPN by institute's source address according to the rule
The network segment address and destination address of subnet are that the message of the VPC subnet address of the working equipment is sent to the management equipment
VPC subnet address, and management equipment unlatching has forwarding capability, can will be sent to the VPC subnet address of the management equipment
, the IDE equipment submits to and runs the operation of the working equipment of the operation and be transmitted to the working equipment for running the operation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610806358.2A CN106385355B (en) | 2016-09-06 | 2016-09-06 | The method, apparatus and system of operation are submitted to hadoop cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610806358.2A CN106385355B (en) | 2016-09-06 | 2016-09-06 | The method, apparatus and system of operation are submitted to hadoop cluster |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106385355A CN106385355A (en) | 2017-02-08 |
| CN106385355B true CN106385355B (en) | 2019-07-23 |
Family
ID=57938039
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610806358.2A Active CN106385355B (en) | 2016-09-06 | 2016-09-06 | The method, apparatus and system of operation are submitted to hadoop cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106385355B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102170440A (en) * | 2011-03-24 | 2011-08-31 | 北京大学 | Method suitable for safely migrating data between storage clouds |
| CN103064664A (en) * | 2012-11-28 | 2013-04-24 | 华中科技大学 | Hadoop parameter automatic optimization method and system based on performance pre-evaluation |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9178935B2 (en) * | 2009-03-05 | 2015-11-03 | Paypal, Inc. | Distributed steam processing |
-
2016
- 2016-09-06 CN CN201610806358.2A patent/CN106385355B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102170440A (en) * | 2011-03-24 | 2011-08-31 | 北京大学 | Method suitable for safely migrating data between storage clouds |
| CN103064664A (en) * | 2012-11-28 | 2013-04-24 | 华中科技大学 | Hadoop parameter automatic optimization method and system based on performance pre-evaluation |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106385355A (en) | 2017-02-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112470436B (en) | Systems, methods, and computer-readable media for providing multi-cloud connectivity | |
| US10862852B1 (en) | Resolution of domain name requests in heterogeneous network environments | |
| Medhat et al. | Service function chaining in next generation networks: State of the art and research challenges | |
| CN103403707B (en) | The system and method exchanged for database proxy request | |
| US9794370B2 (en) | Systems and methods for distributed network-aware service placement | |
| CN111917649B (en) | Virtual private cloud communication and configuration method and related device | |
| CN103179192B (en) | The message forwarding method that virtual server moves, system and NAT service equipment | |
| CN103155524B (en) | The system and method for IIP address is shared between the multiple cores in multiple nucleus system | |
| CN102334111B (en) | Providing logical networking functionality for managed computer networks | |
| RU2595540C9 (en) | Chassis controllers for converting universal flows | |
| CN104322010B (en) | System and method for comparing configuration file and generation corrective command | |
| CN104364761B (en) | For the system and method for the converting flow in cluster network | |
| JP2018088293A (en) | Database system providing single tenant environment and a plurality of tenant environments | |
| CN107771320A (en) | For improving security socket layer（SSL）The system and method for communications security | |
| JP2020504405A (en) | Virtual network verification service | |
| CN103119907A (en) | Systems and methods for providing a smart group | |
| CN102460391A (en) | Systems and methods for providing virtual appliance in application delivery fabric | |
| US20180069787A1 (en) | Exposing a subset of hosts on an overlay network to components external to the overlay network without exposing another subset of hosts on the overlay network | |
| CN113301116A (en) | Cross-network communication method, device, system and equipment for microservice application | |
| CN111818081B (en) | Virtual encryption machine management method, device, computer equipment and storage medium | |
| CN113709810A (en) | Method, device and medium for configuring network service quality | |
| Baldin et al. | The future of multi-clouds: A survey of essential architectural elements | |
| CN106385355B (en) | The method, apparatus and system of operation are submitted to hadoop cluster | |
| CN116389599A (en) | Gateway service request processing method and device and cloud native gateway system management method and device | |
| Allahvirdi et al. | Placement of dynamic service function chains in partially VNF-enabled networks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |