CN103218263B - The dynamic defining method of MapReduce parameter and device - Google Patents

Abstract

The invention provides a kind of dynamic defining method and device of MapReduce parameter, wherein, described method comprises: obtain MapReduce job request, and described MapReduce job request comprises the quantity for the treatment of work data collection, default Reduce task and for representing the parameter whether allowing the Regulation mechanism starting Reduce task quantity; If whether allow the parameter of the Regulation mechanism starting Reduce task quantity for allowing, then monitor the execution of Map task for representing; If the quantity of executed Map task meets default first threshold, then determine the quantity of the Reduce task after adjusting; According to the quantity of the Reduce task after adjustment, each unenforced default Reduce task is corresponded to the Reduce task after each adjustment, thus realize the quantity dynamically determining a rational Reduce task in MapReduce operation.

Description

The dynamic defining method of MapReduce parameter and device

Technical field

The present invention relates to distributed computing technology field, particularly relate to a kind of dynamic defining method and device of MapReduce parameter.

Background technology

Mapping abbreviation MapReduce is a kind of distributed computing framework, and it uses for reference the thought of functional expression programming, carries out Distributed Calculation efficiently to large-scale dataset.A computational tasks Job is divided into several and maps Map task and abbreviation Reduce task by MapReduce framework, the data model of the input and output of Map task phase and Reduce task phase is all Key-Value form, and Reduce task phase relies on the input of output as oneself of Map task phase.And the quantity of Map task is determined by the data set inputted, the quantity of Reduce task is specified by user.Because the data set of input is general larger, therefore, data set can be cut into multiple data block chunk, after MapReduce Hand up homework, the quantity of the data block that the scheduler Master of MapReduce framework can comprise according to input data set is determined the quantity of corresponding Map task to make each Map task process data block.

And the data block of each Map task input is converted into Key-Value form, through Map computing, export the intermediate result of Key-Value form, intermediate result can sort by Key, ranking results will be output on the local disk of this Map task run place computing node, MapReduce framework can by Key sequence and the quantity of the Reduce task of specifying by user do to key subregion and to the Value dropped in same subregion and be polymerized.The input data of each Reduce task are parts for the intermediate result that multiple Map task exports, such as, if user specifies n Reduce task, just there is n subregion, the intermediate result belonging to each subregion by Internet Transmission to performing in Reduce task that this subregion calculates, perform the Reduce algorithm that user specifies, last Output rusults.

Due in existing MapReduce framework, the accurate quantity of Reduce task just must be known before Map tasks carrying, thus the quantity of the Reduce task can specified according to user in Map task phase carries out subregion to the intermediate result exported, and the quantity of Reduce task is normally artificially specified by user, therefore, the intermediate result no matter Map task phase exports has how many, is all the operation carrying out Reduce task according to the quantity of set Reduce task.When the intermediate result that Map task phase exports is little, can perform in 1 ~ 2 Reduce task completely, but the quantity of the Reduce task that possible user specifies is much larger than 2, if the quantity operation of the Reduce task of also specifying according to user, the unnecessary wasting of resources can be caused; And when the quantity of the Reduce task that the intermediate result that Map task phase exports is a lot, user specifies is relatively less, if the quantity operation of the Reduce task of specifying according to user, the problem that the execution time is long can be caused.

Summary of the invention

The object of the present invention is to provide a kind of dynamic defining method and device of MapReduce parameter, thus realize the quantity dynamically determining a rational Reduce task in MapReduce operation.

First aspect of the present invention is to provide a kind of dynamic defining method of MapReduce parameter, comprising:

Obtain MapReduce job request, described MapReduce job request comprises the quantity for the treatment of work data collection, default Reduce task and for representing the parameter whether allowing the Regulation mechanism starting Reduce task quantity;

Whether allow the parameter of the Regulation mechanism starting Reduce task quantity for allowing, then monitor the execution of Map task if described for representing;

If the quantity of executed Map task meets default first threshold, the Output rusults of described executed Map task is mapped to the subregion of the quantity of default Reduce task, then determine the quantity of the Reduce task after adjusting;

According to the quantity of the Reduce task after described adjustment, each unenforced default Reduce task is corresponded to the Reduce task after each adjustment, to make the Reduce task after each adjustment of execution.

Another aspect of the present invention is to provide a kind of dynamic determining device of MapReduce parameter, comprising:

Job request acquisition module, for obtaining MapReduce job request, described MapReduce job request comprises the quantity for the treatment of work data collection, default Reduce task and for representing the parameter whether allowing the Regulation mechanism starting Reduce task quantity;

Whether monitoring module, if allow the parameter of the Regulation mechanism starting Reduce task quantity for allowing, then monitor the execution of Map task for described for representing;

Determination module, if meet default first threshold for the quantity of executed Map task, the subregion of the quantity of the Reduce task preset that the Output rusults of described executed Map task is mapped to, then determine the quantity of the Reduce task after adjusting;

Mapping block, for the quantity according to the Reduce task after described adjustment, corresponds to the Reduce task after each adjustment, to make the Reduce task after each adjustment of execution by each unenforced default Reduce task.

The beneficial effect of employing the invention described above technical scheme is: the present embodiment is by monitoring the Master of MapReduce framework, thus can according to the implementation status of Map task, dynamic conditioning is carried out to the quantity of Reduce task in MapReduce job request, thus perform Reduce task according to the quantity of the Reduce task after adjustment, specify the quantity of Reduce task by user's static state in prior art and the wasting of resources that causes or the problem such as the execution time is long to solve.

Accompanying drawing explanation

The process flow diagram of the dynamic defining method of a kind of MapReduce parameter that Fig. 1 provides for the embodiment of the present invention one;

The structural representation of the dynamic determining device of a kind of MapReduce parameter that Fig. 2 provides for the embodiment of the present invention two.

Embodiment

The process flow diagram of the dynamic defining method of a kind of MapReduce parameter that Fig. 1 provides for the embodiment of the present invention one, as shown in Figure 1, described method can comprise the steps:

Step 101, obtains MapReduce job request; Described MapReduce job request comprises the quantity for the treatment of work data collection, default Reduce task and for representing the parameter whether allowing the Regulation mechanism starting Reduce task quantity;

It should be noted that, the executive agent of the present embodiment method can be the dynamic determining device of MapReduce parameter, and the Master of this device to MapReduce framework monitors, thus can be obtained the job request of MapReduce by Master.Wherein, the job request of MapReduce can comprise the quantity for the treatment of work data collection, default Reduce task and for representing the parameter whether allowing the Regulation mechanism starting Reduce task quantity.Due to the accurate quantity of Reduce task just must be known before performing Map task, subregion can be carried out according to the quantity of Reduce task to the intermediate result exported in Map task phase, therefore, the job request of each MapReduce is preset to the quantity of a Reduce task, to perform Map task, in addition, in the present embodiment, the dynamic determining device of MapReduce parameter is according to for representing the adjustment whether allowing the parameter of the Regulation mechanism starting Reduce task quantity to determine whether to start Reduce task quantity, for example, this parameter can be represent the parameter value allowing to start or do not allow to start, as being True, False or default, in the present embodiment, True can represent to treat when work data collection carries out MapReduce operation and allows the dynamic determining device of MapReduce parameter to carry out dynamic conditioning to the quantity of Reduce task, Reduce task can be performed according to the quantity of the Reduce task after adjustment when performing Reduce task, False or default, can represent to treat when work data collection carries out MapReduce operation and not allow the dynamic determining device of MapReduce parameter to carry out dynamic conditioning to the quantity of Reduce task, and can only perform Reduce task according to the quantity of the Reduce task preset when performing Reduce task.

Whether step 102, allow the parameter of the Regulation mechanism starting Reduce task quantity for allowing, then monitor the execution of Map task if described for representing;

In the present embodiment, when for representing that the parameter whether allowing the Regulation mechanism starting Reduce task quantity is permission, then this parameter can trigger the adjustment of the dynamic determining device startup Reduce task quantity of MapReduce parameter, therefore, the dynamic determining device of MapReduce parameter continues to monitor the Master of MapReduce framework, and according to treating that the quantity of Map task determined by work data collection, or obtained the quantity of Map task in this MapReduce operation by Master, simultaneously according to the execution of job request monitoring Map task.

Step 103, if the quantity of executed Map task meets default first threshold, then determines the quantity of the Reduce task after adjusting;

In the present embodiment, in order to reasonably adjust the quantity of Reduce task, therefore, can after Map tasks carrying a period of time, the data volume of the intermediate result exported in conjunction with the quantity of executed Map task and executed Map task and determine the input data total amount of Reduce task in this MapReduce operation, specifically, because the data volume of each Map task input is identical, therefore, the data volume of its intermediate result exported is also substantially identical, again due to output that the input of Reduce task is exactly Map task, therefore, in the data volume of the intermediate result that can export according to the quantity of executed Map task and executed Map task and this MapReduce operation Map task quantity and determine the input data total amount of Reduce task, thus adjust according to the quantity of data total amount to Reduce task of Reduce task.In the present embodiment, due to start when performing Map task be according to preset Reduce task quantity to export intermediate result carry out subregion, therefore, in the subregion of the quantity of the Reduce task preset that the intermediate result that executed Map task exports is mapped to.In the present embodiment, started the adjustment opportunity of Reduce task quantity as the dynamic determining device triggering MapReduce parameter by the first threshold that arranges, specifically, when the quantity of executed Map task meets default first threshold, the dynamic determining device then triggering MapReduce parameter carries out dynamic conditioning according to executed Map task to the quantity of Reduce task, thus determine the quantity of the Reduce task after adjusting, under normal circumstances, the quantity of the Reduce task after adjustment is less than the quantity of default Reduce task.

Step 104, according to the quantity of the Reduce task after described adjustment, corresponds to the Reduce task after each adjustment, to make the Reduce task after each adjustment of execution by each unenforced default Reduce task.

In the present embodiment, due to start when performing Map task be according to preset Reduce task quantity to export intermediate result carry out subregion, i.e. each default Reduce task corresponding subregion respectively, therefore, after the quantity of Reduce task is adjusted, then by the Reduce task after the partition map in the quantity of default Reduce task corresponding to unenforced Reduce task to adjustment, namely a Reduce task after adjustment can subregion corresponding to corresponding one or more default Reduce task, thus make Master can scheduled for executing adjustment after Reduce task.

The present embodiment is by monitoring the Master of MapReduce framework, thus can according to the implementation status of Map task, dynamic conditioning is carried out to the quantity of Reduce task in MapReduce job request, thus perform Reduce task according to the quantity of the Reduce task after adjustment, specify the quantity of Reduce task by user's static state in prior art and the wasting of resources that causes or the problem such as the execution time is long to solve.

Concrete, the first threshold preset described in above-described embodiment can be the amount threshold of default Map task, such as, can be 30 Map tasks or 100 Map task dispatchings, choosing of concrete numerical value can be arranged according to actual job situation, and the present embodiment does not limit this.Therefore, when the quantity of executed Map task meets the amount threshold of default Map task, then the dynamic determining device triggering MapReduce parameter starts the adjustment of Reduce task quantity, thus determines the quantity of the Reduce task after adjusting.

Preferably, the first threshold preset described in above-described embodiment can be default ratio, and such as, can be 1/5 or 1/3 etc., choosing of concrete numerical value can be arranged according to actual job situation, and the present embodiment does not limit this.Therefore, when ratio between the quantity and the total quantity of Map task of executed Map task meets default ratio, the dynamic determining device then triggering MapReduce parameter starts the adjustment of Reduce task quantity, thus determine the quantity of the Reduce task after adjusting, in the present embodiment, the total quantity of Map task according to treating that work data collection is determined, also can be obtained by Master.

Further, on the basis of above-mentioned any embodiment, the Second Threshold preset for forward scheduling Reduce tasks carrying can also be comprised in MapReduce job request, wherein, Second Threshold also can be amount threshold or the ratio of default Map task, namely, when the quantity of executed Map task meets Second Threshold, Master can be triggered and start to perform default Reduce task.

In the present embodiment, when Second Threshold is less than first threshold, namely when the quantity of executed Map task meets default Second Threshold but does not meet the first threshold preset, then the dynamic determining device of MapReduce parameter can also monitor the execution of default Reduce task, wherein, each default Reduce task distinguishes a subregion in the subregion of the quantity of the corresponding Reduce task preset; And after the quantity of executed Map task meets default first threshold, then the dynamic determining device of MapReduce parameter can also indicate and stop unenforced default Reduce task.And determine the data volume of unenforced Reduce task according to the quantity of executed Map task with the data total amount of Reduce task in this MapReduce operation, thus the quantity of corresponding unenforced Reduce task is determined according to the data volume of unenforced Reduce task, concrete, when performing unenforced Map task, the intermediate result exported is mapped in the subregion of the quantity of the Reduce task redefined, i.e. the Reduce tasks carrying of unenforced default Reduce task by redefining.

For a MapReduce operation, in two stages of Map and Reduce, quantity due to Map task is fixing, therefore, the cost of Map task is also relatively-stationary, and by MapReduce performance model, calculate respectively operation when the quantity of same MapReduce operation corresponding different Reduce task T.T. cost and the execution time time find, the T.T. cost of operation can increase along with the increase of the quantity of Reduce task usually, simultaneously, the quantity of increase Reduce task can improve the degree of parallelism between the computing power of cluster and task, shorten the execution time of operation, vice versa.Therefore, in a specific implementation of the present invention, by MapReduce performance model, calculate the cost of Reduce task, and between time cost and execution time, find equilibrium point to adjust the quantity of Reduce task.

For example, in MapReduce performance model, the time cost of Reduce task comprises TR1_init, TR2_read, TR3_net, TR4_merge, TR5_serial, TR6_io, TR7_parse, TR8_Reducer, TR9_net and TR10_write, wherein, the time cost of system when TR1_init represents initialization Reduce task, namely initialization task, open time of task, closedown task, loading procedure etc., usual TR1_init=RedSysCost+RedInit; TR2_read represents the IO cost of read data when starting to perform Reduce task, usual TR2_read=ReduceInput/seqRead; TR3_net transmits the network cost of data when representing and perform Reduce task, usually cPU cost needed for TR4_merge sorts when representing and perform Reduce task, usual TR4_merge=SortCEF*RIRNumber*logNumberofMap; TR5_serial represents the time cost performing Reduce task time series, usual TR5_serial=(se1*RIRNumber+se2*ReduceInput); The disk I/O cost that TR6_io relates to when representing and perform Reduce task, usually $\mathrm{TR}6\_\mathrm{io}=\mathrm{ReduceInput}*(\frac{1}{\mathrm{seqRead}}+\frac{1}{\mathrm{seqWrite}});$ TR7_parse represents time cost needed for resolution data when performing Reduce task, usual TR7_parse=pa1*RIRNumber+pa2*ReduceInput; TR8_Reducer represent perform Reduce task time function calculate needed for time cost, usual TR8_Reducer=ReduceInput*ComplexOfReduce*CEF; TR9_net transmits the network cost of data after representing execution Reduce task, usually $\mathrm{TR}9\_\mathrm{net}=\mathrm{HDFSReplica}*\frac{\mathrm{ReduceOutput}}{\mathrm{BandWidth}};$ TR10_write represents the IO cost performed to write after Reduce task, usually $\mathrm{TR}10\_\mathrm{write}=$ $\mathrm{HDFSReplica}*\frac{\mathrm{ReduceOutput}}{\mathrm{SeqWrite}}.$

Therefore, the time cost TR of each Reduce task equals above-mentioned every sum, that is: $\mathrm{TR}=\underset{k=1}{\overset{10}{\mathrm{\Σ}}}T{R}_k;$

Suppose there be n Reduce task in a MapReduce operation, therefore, in a MapReduce operation, the T.T. cost TRS of all Reduce tasks is:

TRS=n*TR。

By above-mentioned various substitution pricing formula TR, then the time cost TR performing a Reduce task equals: $\mathrm{TR}=\mathrm{RedSysCost}+\mathrm{RedInit}+\mathrm{ReduceInput}/\mathrm{seqRead}+\frac{\mathrm{ReduceInput}}{\mathrm{BandWidth}}$

$\begin{array}{c}+\mathrm{SortCEF}*\mathrm{RIRNunber}*\log \mathrm{NunberofMap}\\ +(\mathrm{sel}*\mathrm{RIRNumber}+\mathrm{se}2*\mathrm{ReduceInput})\\ +\mathrm{ReduceInput}*(\frac{1}{\mathrm{seqRead}}+\frac{1}{\mathrm{seqWrite}})\\ +\mathrm{pa}1*\mathrm{RIRNumber}+\mathrm{pa}2*\mathrm{ReduceInput}\\ +\mathrm{ReduceInput}*\mathrm{ComplexOfReduce}*\mathrm{CEF}\\ +\mathrm{HDFSReplica}*\frac{\mathrm{ReduceOutput}}{\mathrm{BandWidth}}+\mathrm{HDFSReplica}*\frac{\mathrm{ReduceOutput}}{\mathrm{SeqWrite}}\end{array};$

Wherein, initialization when RedSysCost represents initialization Reduce task, open, close the time cost of task, be traditionally arranged to be 2000 milliseconds (ms); RedInit loads additional programs or the time needed for data when representing initialization Reduce task; ReduceInput represents the data volume that execution Reduce task inputs; SeqRead represents in Preset Time the data volume reading disk; Bandwidth represents network transfer speeds in Preset Time; SortCEF represents sequence coefficient; The quantity of the data that RIRNumber inputs when representing and perform Reduce task; NumberOfMap represents the quantity of Map task; SeqWrite represents in Preset Time the data volume writing disk; The complexity that ComplexOfReduce calculates when representing and perform Reduce task; HDFSReplica represents the quantity of copy; The data volume that ReduceOutput exports after representing execution Reduce task; Se1 and se2 is two parameters in serializing stage, in Mapredcue performance model, and linear with data volume of the time cost required for serializing, se1 and se2 is exactly two parameters of this linear relationship, and saying for a cluster, is constant, therefore, constant can be thought; Pa1 and pa2 is two constant parameter of phase sorting; CEF is the time required for the computing of clustered machine cleanup standard, is also constant.Except ReduceInput, ReduceOutput, outside the Pass the data volume of Map task and Reduce task has in RIRNumber, NumberOfMap and MapReduce operation, other parameter is the Given information of systemic presupposition in above-mentioned each parameter.

Further, if make n be the quantity of Reduce task in MapReduce operation, then ReduceInput=Input/n, wherein, Input represents total input data volume of all Reduce tasks in MapReduce operation, if Y is the data transformations rate of Reduce, i.e. Reduce output data quantity and the ratio inputting data volume, Y=ReduceOutput/ReduceInput, then ReduceOutput=Y*ReduceInput=Y*Input/n, then

RIRNumber=ReduceInput/RIRLength=Input/n/RIRLength; Wherein, RIRlength represents the average length of input data.

Substitute into above-mentioned formula and by after abbreviation, obtain:

$\begin{array}{c}\mathrm{TR}=\mathrm{RedSysCost}+\mathrm{RedInit}\\ +[2/\mathrm{SeqRead}+(1+\mathrm{HDFSReplica}*Y)*(\frac{1}{\mathrm{SeqWrite}}+\frac{1}{\mathrm{BandWidth}})\\ +(\mathrm{se}1+\mathrm{pa}1+\mathrm{SortCEF}*\log \mathrm{NumberofMap})/\mathrm{RIRLength}\\ +(\mathrm{se}2+\mathrm{pa}2+\mathrm{ComplexOfReduce}*\mathrm{CEF})]*\mathrm{Input}/n\end{array};$

If make a=RedSysCost+RedInit,

$\begin{array}{c}b=2/\mathrm{SeqRed}+(1+\mathrm{HDFSReplica}*Y)*(\frac{1}{\mathrm{SeqWrite}}+\frac{1}{\mathrm{BandWidth}})\\ +(\mathrm{se}1+\mathrm{pa}1+\mathrm{SortCEF}*\log \mathrm{NumberofMap})/\mathrm{RIRLength}\\ +(\mathrm{se}2+\mathrm{pa}2+\mathrm{ComplexOfReduce}*\mathrm{CEF})\end{array};$

For given MapReduce operation, the value of a and b can be tried to achieve by above-mentioned parameter, is equivalent to constant, therefore performs the time cost TR=a+b*Input/n of a Reduce task; Correspondingly, TRS=n*TR=a*n+b*Input.

For the formation of the T.T. cost TRS of all Reduce tasks in a time cost TR and MapReduce operation of a Reduce task, the first order derivative sum of TR and TRS can be selected to be that the point of 0 is as equilibrium point, thus the quantity of adjustment Reduce task, that is: $\frac{\∂\left(\mathrm{TR}\right)}{\∂\left(n\right)}+\frac{\∂\left(\mathrm{TRS}\right)}{\∂\left(n\right)}=0;$ Substitute into equation: $n=\sqrt{b*\mathrm{Input}/a}=\sqrt{\frac{b}{a}}*\sqrt{\mathrm{Input}}.$

Thus the quantity of Reduce task can be determined according to above formula, further, in order to avoid occurring that the data volume of each Reduce process is more thus the situation that the execution time that is single Reduce task is longer, the maximum process data volume that can limit single Reduce task is further U, therefore, the all data of Reduce task process using Input/U quantity are needed for each MapReduce operation is minimum, the value of U is specifically as follows: U=ReduceJVM*ShuffleBufferPercent*ShuffleMergePercent, therefore:

$n=\{\begin{array}{c}\sqrt{b/a}*\sqrt{\mathrm{Input}},\mathrm{ifinput}<U^2*b/a\\ \frac{\mathrm{Input}}{U},\mathrm{else}\end{array}.$

The present embodiment is by monitoring the Master of MapReduce framework, thus can according to the implementation status of Map task, determine the total amount of data of Reduce task in MapReduce operation, thus consider the time cost that performs Reduce task according to the total amount of data of Reduce task and the quantity of Reduce task is adjusted, to determine the quantity of a rational Reduce task, and perform Reduce task according to the quantity of the Reduce task after adjustment, the quantity of Reduce task is specified by user's static state in prior art and the wasting of resources that causes or the problem such as the execution time is long to solve, make the time cost performing Reduce task be tending towards minimum simultaneously.

One of ordinary skill in the art will appreciate that: all or part of step realizing above-mentioned each embodiment of the method can have been come by the hardware that programmed instruction is relevant.Aforesaid program can be stored in a computer read/write memory medium.This program, when performing, performs the step comprising above-mentioned each embodiment of the method; And aforesaid storage medium comprises: ROM, RAM, magnetic disc or CD etc. various can be program code stored medium.

The structural representation of the dynamic determining device of a kind of MapReduce parameter that Fig. 2 provides for the embodiment of the present invention two, as shown in Figure 2, the dynamic determining device of the MapReduce parameter of the present embodiment can comprise:

Job request acquisition module 201, for obtaining MapReduce job request, described MapReduce job request comprises the quantity for the treatment of work data collection, default Reduce task and for representing the parameter whether allowing the Regulation mechanism starting Reduce task quantity;

Whether monitoring module 202, if for allowing the parameter of the Regulation mechanism starting Reduce task quantity for allowing, then monitor the execution of Map task for representing;

Determination module 203, if meet default first threshold for the quantity of executed Map task, the subregion of the quantity of the Reduce task preset that the Output rusults of described executed Map task is mapped to, then determine the quantity of the Reduce task after adjusting;

Mapping block 204, for the quantity according to the Reduce task after adjustment, corresponds to the Reduce task after each adjustment, to make the Reduce task after each adjustment of execution by each unenforced default Reduce task.

The dynamic determining device of the MapReduce parameter of the present embodiment may be used for the technical scheme performing embodiment of the method shown in Fig. 1, and it realizes principle and technique effect is similar, repeats no more herein.

Further, determination module specifically may be used for: according to the data total amount of unenforced Reduce task, determines the quantity of the Reduce task after adjusting.

Preferably, determination module specifically may be used for:

According to following formula, determine the quantity of the Reduce task after adjusting:

$\frac{\&PartialD;\left(\mathrm{TR}\right)}{\&PartialD;\left(n\right)}+\frac{\&PartialD;\left(\mathrm{TRS}\right)}{\&PartialD;\left(n\right)}=0;$

Wherein, n is the quantity of the Reduce task after adjustment, TR is the time cost of a Reduce task after performing adjustment, TRS is the T.T. cost of all Reduce tasks after performing adjustment, and TR depends on the quantity of the data total amount of unenforced Reduce task and the Reduce task after adjusting.

Concrete, the MapReduce job request in above-described embodiment can also comprise the Second Threshold preset for forward scheduling Reduce tasks carrying, if Second Threshold is less than first threshold, then monitoring module specifically may be used for:

If the quantity of executed Map task meets described Second Threshold and do not meet the first threshold preset, then the execution of Reduce task is preset in monitoring, a subregion in the subregion of the quantity of the corresponding described default Reduce task of each default Reduce task difference;

And after the quantity of executed Map task meets default first threshold, instruction stops performing unenforced default Reduce task.

Concrete, in above-mentioned any embodiment, the first threshold preset can be the amount threshold of Map task preset or default ratio, then determination module specifically for:

If the quantity of executed Map task meets the amount threshold of default Map task, then start the Regulation mechanism of Reduce task quantity, determine the quantity of the Reduce task after adjusting; Or,

If the ratio between the quantity of executed Map task and the total quantity of Map task meets default ratio, then start the Regulation mechanism of Reduce task quantity, determine the quantity of Reduce task after adjusting, wherein, the total quantity of Map task according to described in treat that work data collection is determined.

Last it is noted that above each embodiment is only in order to illustrate technical scheme of the present invention, be not intended to limit; Although with reference to foregoing embodiments to invention has been detailed description, those of ordinary skill in the art is to be understood that: it still can be modified to the technical scheme described in foregoing embodiments, or carries out equivalent replacement to wherein some or all of technical characteristic; And these amendments or replacement, do not make the essence of appropriate technical solution depart from the scope of various embodiments of the present invention technical scheme.

Claims (6)

1. a dynamic defining method for MapReduce parameter, is characterized in that, comprising:

Obtain MapReduce job request, described MapReduce job request comprises the quantity for the treatment of work data collection, default Reduce task and for representing the parameter whether allowing the Regulation mechanism starting Reduce task quantity;

Whether allow the parameter of the Regulation mechanism starting Reduce task quantity for allowing, then monitor the execution of Map task if described for representing;

If the quantity of executed Map task meets default first threshold, the Output rusults of described executed Map task is mapped to the subregion of the quantity of default Reduce task, then determine the quantity of the Reduce task after adjusting;

According to the quantity of the Reduce task after described adjustment, each unenforced default Reduce task is corresponded to the Reduce task after each adjustment, to make the Reduce task after each adjustment of execution;

The described quantity determining the Reduce task after adjusting, specifically comprises:

According to the data total amount of unenforced Reduce task, determine the quantity of the Reduce task after adjusting;

The described data total amount according to unenforced Reduce task, determine the quantity of the Reduce task after adjusting, specifically comprise:

According to following formula, determine the quantity of the Reduce task after adjusting:

$\frac{\&part;\left(TR\right)}{\&part;\left(n\right)}+\frac{\&part;\left(TRS\right)}{\&part;\left(n\right)}=0;$

Wherein, n is the quantity of the Reduce task after adjustment, TR is the time cost of a Reduce task after performing adjustment, TRS is the T.T. cost of all Reduce tasks after performing adjustment, and TR depends on the quantity of the data total amount of unenforced Reduce task and the Reduce task after adjusting.

2. method according to claim 1, it is characterized in that, described MapReduce job request also comprises the Second Threshold preset for forward scheduling Reduce tasks carrying, if described Second Threshold is less than described first threshold, if before then the quantity of described executed Map task meets default first threshold, also comprise:

If the quantity of described executed Map task meets described Second Threshold and do not meet the first threshold preset, then the execution of Reduce task is preset in monitoring, a subregion in the subregion of the quantity of the corresponding described default Reduce task of each default Reduce task difference;

If the quantity of described executed Map task also comprises after meeting default first threshold:

Instruction stops performing unenforced default Reduce task.

3. method according to claim 1 and 2, it is characterized in that, described default first threshold is the amount threshold of default Map task or default ratio, if the quantity of described executed Map task meets default first threshold, then determine the quantity of the Reduce task after adjusting, be specially:

If the quantity of described executed Map task meets the amount threshold of described default Map task, then start the Regulation mechanism of Reduce task quantity, determine the quantity of the Reduce task after adjusting; Or,

If the ratio between the quantity of described executed Map task and the total quantity of Map task meets default ratio, then start the Regulation mechanism of Reduce task quantity, determine the quantity of Reduce task after adjusting, the total quantity of described Map task according to described in treat that work data collection is determined.

4. a dynamic determining device for MapReduce parameter, is characterized in that, comprising:

Job request acquisition module, for obtaining MapReduce job request, described MapReduce job request comprises the quantity for the treatment of work data collection, default Reduce task and for representing the parameter whether allowing the Regulation mechanism starting Reduce task quantity;

Whether monitoring module, if allow the parameter of the Regulation mechanism starting Reduce task quantity for allowing, then monitor the execution of Map task for described for representing;

Determination module, if meet default first threshold for the quantity of executed Map task, the subregion of the quantity of the Reduce task preset that the Output rusults of described executed Map task is mapped to, then determine the quantity of the Reduce task after adjusting;

Mapping block, for the quantity according to the Reduce task after described adjustment, corresponds to the Reduce task after each adjustment, to make the Reduce task after each adjustment of execution by each unenforced default Reduce task;

Described determination module specifically for:

According to the data total amount of unenforced Reduce task, according to following formula, determine the quantity of the Reduce task after adjusting:

$\frac{\&part;\left(TR\right)}{\&part;\left(n\right)}+\frac{\&part;\left(TRS\right)}{\&part;\left(n\right)}=0;$

Wherein, n is the quantity of the Reduce task after adjustment, TR is the time cost of a Reduce task after performing adjustment, TRS is the T.T. cost of all Reduce tasks after performing adjustment, and TR depends on the quantity of the data total amount of unenforced Reduce task and the Reduce task after adjusting.

5. device according to claim 4, it is characterized in that, described MapReduce job request also comprise for forward scheduling Reduce tasks carrying preset Second Threshold, if described Second Threshold is less than described first threshold, then described monitoring module specifically for:

If the quantity of described executed Map task meets described Second Threshold and do not meet the first threshold preset, then the execution of Reduce task is preset in monitoring, a subregion in the subregion of the quantity of the corresponding described default Reduce task of each default Reduce task difference;

And after the quantity of described executed Map task meets default first threshold, instruction stops performing unenforced default Reduce task.

6. the device according to claim 4 or 5, is characterized in that, described default first threshold is the amount threshold of default Map task or default ratio, described determination module specifically for:

If the quantity of described executed Map task meets the amount threshold of described default Map task, then start the Regulation mechanism of Reduce task quantity, determine the quantity of the Reduce task after adjusting; Or,

If the ratio between the quantity of described executed Map task and the total quantity of Map task meets default ratio, then start the Regulation mechanism of Reduce task quantity, determine the quantity of Reduce task after adjusting, the total quantity of described Map task according to described in treat that work data collection is determined.