CN103218263A - Dynamic determining method and device for MapReduce parameter - Google Patents

Abstract

The invention provides a dynamic determining method and device for MapReduce parameters. The method comprises the steps of acquiring a MapReduce operation request, wherein the MapReduce operation request comprises a dataset to be operated, preset Reduce task quantity and parameters which are used for expressing whether an adjusting mechanism for the Reduce task quantity is allowed to be started or not; if the parameters which are used for expressing whether the adjusting mechanism for the Reduce task quantity is allowed to be started or not are allowable, monitoring the execution of Map tasks; if the quantity of the executed Map tasks satisfies a preset first threshold value, determining the adjusted Reduce task quantity; and according to the adjusted Reduce task quantity, enabling unexecuted preset Reduce tasks to correspond to the adjusted Reduce tasks, so as to realize the goal of dynamically determining a reasonable Reduce task quantity during MapReduce operation.

Description

The dynamically definite method and the device of MapReduce parameter

Technical field

The present invention relates to the distributed computing technology field, relate in particular to a kind of dynamically definite method and device of MapReduce parameter.

Background technology

Mapping abbreviation MapReduce is a kind of distributed computing framework, and it uses for reference the thought of functional expression programming, and large-scale dataset is carried out Distributed Calculation efficiently.The MapReduce framework is divided into several mapping Map task and abbreviation Reduce tasks with a computational tasks Job, the data model of the input and output of Map task phase and Reduce task phase all is the Key-Value form, and the Reduce task phase relies on the input of the output of Map task phase as oneself.And the quantity of Map task determines that by the data set of importing the quantity of Reduce task is specified by the user.Because the data set of input is generally bigger, therefore, data set can be cut into a plurality of data block chunk, after the MapReduce operation is submitted to, the quantity of the data block that the scheduler Master of MapReduce framework can comprise according to input data set is determined the quantity of corresponding M ap task, makes each Map task handle a data block.

And the data block of each Map task input is converted into the Key-Value form, through the Map computing, the intermediate result of output Key-Value form, intermediate result can sort by Key, ranking results will be output on the local disk of this Map task run place computing node, and the MapReduce framework can be done polymerization to the key subregion and to the Value that drops in the same subregion by Key ordering and by the quantity of the Reduce task of user's appointment.The input data of each Reduce task are the parts of the intermediate result of a plurality of Map task outputs, for example, if the user has specified n Reduce task, just there be n subregion, the intermediate result that belongs to each subregion by Network Transmission to carrying out in the Reduce task that this subregion calculates, carry out the Reduce algorithm of user's appointment, export the result at last.

Because in existing MapReduce framework, before carrying out, the Map task just must know the accurate quantity of Reduce task, thereby can carry out subregion to the intermediate result of output according to the quantity of the Reduce task of user's appointment in the Map task phase, and the quantity of Reduce task is normally artificially specified by the user, therefore, how much intermediate result no matter the Map task phase is exported has, and all is to carry out the operation of Reduce task according to the quantity of set Reduce task.When the intermediate result of Map task phase output seldom the time, can in 1～2 Reduce task, carry out fully, but the quantity of the Reduce task of possible user's appointment is much larger than 2, if also according to the quantity operation of the Reduce task of user's appointment then can cause the unnecessary wasting of resources; And the quantity of the Reduce task of, user appointment a lot of when the intermediate result of Map task phase output relatively more after a little while, if according to the quantity operation of the Reduce task of user's appointment then can cause long problem of execution time.

Summary of the invention

The object of the present invention is to provide a kind of dynamically definite method and device of MapReduce parameter, thereby be implemented in the quantity of dynamically determining a rational Reduce task in the MapReduce operation.

First aspect of the present invention provides a kind of method of dynamically determining of MapReduce parameter, comprising:

Obtain the MapReduce job request, the parameter that described MapReduce job request comprises the quantity for the treatment of work data collection, default Reduce task and is used to represent whether allow start the adjustment mechanism of Reduce task quantity;

If the parameter of the described adjustment mechanism that is used to represent whether to allow to start Reduce task quantity is then monitored the Map task executions for allowing;

If the quantity of executed Map task satisfies presetting first threshold, the output result of described executed Map task is mapped to the subregion of the quantity of default Reduce task, then determines the quantity of adjusted Reduce task;

According to the quantity of described adjusted Reduce task, each unenforced default Reduce task is corresponded to each adjusted Reduce task, so that carry out each adjusted Reduce task.

Another aspect of the present invention provides a kind of dynamically definite device of MapReduce parameter, comprising:

The job request acquisition module, be used to obtain the MapReduce job request, the parameter that described MapReduce job request comprises the quantity for the treatment of work data collection, default Reduce task and is used to represent whether allow start the adjustment mechanism of Reduce task quantity;

Monitoring module is used for if the parameter of the described adjustment mechanism that is used to represent whether to allow to start Reduce task quantity for allowing, is then monitored the Map task executions;

Determination module is used for if the quantity of executed Map task satisfies presetting first threshold, the subregion of the quantity of the default Reduce task that the output result of described executed Map task is mapped to, the then quantity of definite adjusted Reduce task;

Mapping block is used for the quantity according to described adjusted Reduce task, and each unenforced default Reduce task is corresponded to each adjusted Reduce task, so that carry out each adjusted Reduce task.

Adopt the beneficial effect of the invention described above technical scheme to be: present embodiment is monitored by the Master to the MapReduce framework, thereby can be according to Map task executions situation, quantity to Reduce task in the MapReduce job request is dynamically adjusted, thereby carry out the Reduce task according to the quantity of adjusted Reduce task, to solve problems such as the wasting of resources that caused by the quantity of the static Reduce of appointment of user task in the prior art or execution time be long.

Description of drawings

The process flow diagram of the method for dynamically determining of a kind of MapReduce parameter that Fig. 1 provides for the embodiment of the invention one;

The structural representation of dynamically determining device of a kind of MapReduce parameter that Fig. 2 provides for the embodiment of the invention two.

Embodiment

The process flow diagram of the method for dynamically determining of a kind of MapReduce parameter that Fig. 1 provides for the embodiment of the invention one, as shown in Figure 1, described method can comprise the steps:

Step 101 is obtained the MapReduce job request; The parameter that described MapReduce job request comprises the quantity for the treatment of work data collection, default Reduce task and is used to represent whether allow start the adjustment mechanism of Reduce task quantity;

Need to prove that the executive agent of present embodiment method can be dynamically definite device of MapReduce parameter, this device is monitored the Master of MapReduce framework, thereby can obtain the job request of MapReduce by Master.Wherein, the job request of the MapReduce parameter that can comprise the quantity for the treatment of work data collection, default Reduce task and be used to represent whether allow start the adjustment mechanism of Reduce task quantity.Owing to before carrying out the Map task, just must know the accurate quantity of Reduce task, so that can carry out subregion to the intermediate result of output according to the quantity of Reduce task in the Map task phase, therefore, the job request of each MapReduce is all preset the quantity of a Reduce task, so that carry out the Map task; In addition, the device of dynamically determining of MapReduce parameter determines whether to start the adjustment of Reduce task quantity in the present embodiment according to the parameter that is used to represent whether to allow to start the adjustment mechanism of Reduce task quantity, for instance, this parameter can be the parameter value that expression allows startup or do not allow to start, as being True, False or default, in the present embodiment, True can represent to allow when treating the work data collection carries out the MapReduce operation device of dynamically determining of MapReduce parameter that the quantity of Reduce task is dynamically adjusted, so that can carry out the Reduce task according to the quantity of adjusted Reduce task when carrying out the Reduce task; False or defaultly then can represent not allow when treating the work data collection carries out the MapReduce operation device of dynamically determining of MapReduce parameter that the quantity of Reduce task is dynamically adjusted, and when carrying out the Reduce task, can only carry out the Reduce task according to the quantity of default Reduce task.

Step 102 is if the parameter of the described adjustment mechanism that is used to represent whether to allow to start Reduce task quantity is then monitored the Map task executions for allowing;

In the present embodiment, when the parameter of the adjustment mechanism that is used to represent whether to allow to start Reduce task quantity when allowing, then this parameter can trigger the adjustment of dynamically determining device startup Reduce task quantity of MapReduce parameter, therefore, dynamically definite device of MapReduce parameter continues the Master of MapReduce framework is monitored, and according to treating that the work data collection determines the quantity of Map task, or, monitor the Map task executions according to job request simultaneously by the quantity that Master obtains Map task in this MapReduce operation.

Step 103 if the quantity of executed Map task satisfies presetting first threshold, is then determined the quantity of adjusted Reduce task;

In the present embodiment, in order can reasonably to adjust to the quantity of Reduce task, therefore, can be after the Map task be carried out a period of time, determine the input data total amount of Reduce task in this MapReduce operation in conjunction with the data volume of the intermediate result of the quantity of executed Map task and the output of executed Map task, specifically, because the data volume of each Map task input is identical, therefore, the data volume of the intermediate result of its output is also basic identical, again because the input of Reduce task is exactly the output of Map task, therefore, can determine the input data total amount of Reduce task according to the quantity of Map task in the data volume of the intermediate result of the quantity of executed Map task and the output of executed Map task and this MapReduce operation, thereby the quantity of Reduce task be adjusted according to the data total amount of Reduce task.In the present embodiment, owing to be the intermediate result of output to be carried out subregion when beginning to carry out the Map task according to the quantity of default Reduce task, therefore, in the subregion of the quantity of the default Reduce task that is mapped to of the intermediate result of executed Map task output.In the present embodiment, start the adjustment opportunity of Reduce task quantity by the first threshold that is provided with as the device of dynamically determining that triggers the MapReduce parameter, specifically, when the quantity of executed Map task satisfies presetting first threshold, then triggering the device of dynamically determining of MapReduce parameter dynamically adjusts the quantity of Reduce task according to executed Map task, thereby determine the quantity of adjusted Reduce task, generally, the quantity of adjusted Reduce task is less than the quantity of default Reduce task.

Step 104 according to the quantity of described adjusted Reduce task, corresponds to each adjusted Reduce task with each unenforced default Reduce task, so that carry out each adjusted Reduce task.

In the present embodiment, owing to be the intermediate result of output to be carried out subregion when beginning to carry out the Map task according to the quantity of default Reduce task, it is the respectively corresponding subregion of each default Reduce task, therefore, after the quantity of Reduce task is adjusted, then the pairing partition map of unenforced Reduce task in the quantity of default Reduce task is arrived adjusted Reduce task, be the subregion that an adjusted Reduce task can corresponding one or more default Reduce task correspondences, thereby make that Master can the adjusted Reduce task of scheduled for executing.

Present embodiment is monitored by the Master to the MapReduce framework, thereby can be according to Map task executions situation, quantity to Reduce task in the MapReduce job request is dynamically adjusted, thereby carry out the Reduce task according to the quantity of adjusted Reduce task, to solve problems such as the wasting of resources that caused by the quantity of the static Reduce of appointment of user task in the prior art or execution time be long.

Concrete, presetting first threshold described in the foregoing description can be the amount threshold of the Map task preset, for example, can be 30 Map tasks or 100 Map task dispatchings, concrete choosing of numerical value can be provided with according to the actual job situation, and present embodiment does not limit this.Therefore, when the quantity of executed Map task satisfied the amount threshold of default Map task, the device of dynamically determining that then triggers the MapReduce parameter started the adjustment of Reduce task quantity, thereby determines the quantity of adjusted Reduce task.

Preferably, the presetting first threshold described in the foregoing description can be the ratio of presetting, and for example, can be 1/5 or 1/3 etc., and concrete choosing of numerical value can be provided with according to the actual job situation, and present embodiment does not limit this.Therefore, when the ratio between the total quantity of the quantity of executed Map task and Map task satisfies default ratio, the device of dynamically determining that then triggers the MapReduce parameter starts the adjustment of Reduce task quantity, thereby determine the quantity of adjusted Reduce task, in the present embodiment, the total quantity of Map task can be definite according to treating the work data collection, also can obtain by Master.

Further, on the basis of above-mentioned arbitrary embodiment, can also comprise in the MapReduce job request and be used for the second default threshold value that forward scheduling Reduce task is carried out, wherein, second threshold value also can be the amount threshold or the ratio of the Map task preset, promptly when the quantity of executed Map task satisfies second threshold value, can trigger Master and begin to carry out default Reduce task.

In the present embodiment, when second threshold value during less than first threshold, be that the quantity of executed Map task satisfies the second default threshold value but when not satisfying presetting first threshold, then the device of dynamically determining of MapReduce parameter can also be monitored default Reduce task executions, wherein, a subregion in the subregion of the quantity of the corresponding Reduce task of presetting of each default Reduce task difference; And after the quantity of executed Map task satisfied presetting first threshold, then dynamically definite device of MapReduce parameter can also be indicated and be stopped unenforced default Reduce task.And determine the data volume of unenforced Reduce task according to the data total amount of Reduce task in the quantity of executed Map task and this MapReduce operation, thereby determine the quantity of corresponding unenforced Reduce task according to the data volume of unenforced Reduce task, concrete, when carrying out unenforced Map task, the intermediate result of its output is mapped in the subregion of quantity of the Reduce task that redefines, promptly unenforced default Reduce task is carried out by the Reduce task that redefines.

For a MapReduce operation, at Map and Reduce in two stages, because the quantity of Map task is fixed, therefore, the cost of Map task also is relatively-stationary, and by the MapReduce performance model, calculate the T.T. cost and finding during the execution time of operation under the situation of quantity of the corresponding different Reduce tasks of same MapReduce operation respectively, the common meeting of T.T. cost of operation increases along with the increase of the quantity of Reduce task, simultaneously, the quantity that increases the Reduce task can improve the computing power of cluster and the degree of parallelism between the task, shortens the execution time of operation; Vice versa.Therefore, in a specific implementation of the present invention,, calculate the cost of Reduce task, and seek equilibrium point so that the quantity of Reduce task is adjusted at time cost and between the execution time by the MapReduce performance model.

For instance, in the MapReduce performance model, the time cost of Reduce task comprises TR1_init, TR2_read, TR3_net, TR4_merge, TR5_serial, TR6_io, TR7_parse, TR8_Reducer, TR9_net and TR10_write, wherein, the time cost of system when TR1_init represents initialization Reduce task, be initialization task, open task, close task, time of loading procedure etc., TR1_init=RedSysCost+RedInit usually; The IO cost of read data when TR2_read represents to begin to carry out the Reduce task, TR2_read=ReduceInput/seqRead usually; The network cost of transmission data when TR3_net represents to carry out the Reduce task, usually The TR4_merge required CPU cost that sorts when representing to carry out the Reduce task, TR4_merge=SortCEF*RIRNumber*logNumberofMap usually; TR5_serial represents to carry out the time cost of Reduce task time seriesization, usually TR5_serial=(se1*RIRNumber+se2*ReduceInput); The disk I cost that TR6_io relates to when representing to carry out the Reduce task, usually $\mathrm{TR}6\_\mathrm{io}=\mathrm{ReduceInput}*(\frac{1}{\mathrm{seqRead}}+\frac{1}{\mathrm{seqWrite}});$ The required time cost of resolution data when TR7_parse represents to carry out the Reduce task, TR7_parse=pa1*RIRNumber+pa2*ReduceInput usually; The required time cost of function calculation when TR8_Reducer represents to carry out the Reduce task, TR8_Reducer=ReduceInput*ComplexOfReduce*CEF usually; TR9_net represents to carry out the network cost of transmission data after the Reduce task, usually $\mathrm{TR}9\_\mathrm{net}=\mathrm{HDFSReplica}*\frac{\mathrm{ReduceOutput}}{\mathrm{BandWidth}};$ TR10_write represents to carry out after the Reduce task to the IO of disk write data cost, usually $\mathrm{TR}10\_\mathrm{write}=$ $\mathrm{HDFSReplica}*\frac{\mathrm{ReduceOutput}}{\mathrm{SeqWrite}}.$

Therefore, the time cost TR of each Reduce task equals above-mentioned every sum, that is: $\mathrm{TR}=\underset{k=1}{\overset{10}{\mathrm{\Σ}}}T{R}_k;$

Supposing has n Reduce task in the MapReduce operation, therefore, the T.T. cost TRS of all Reduce tasks is in MapReduce operation:

TRS=n*TR。

With above-mentioned various substitution pricing formula TR, the time cost TR that then carries out a Reduce task equals: $\mathrm{TR}=\mathrm{RedSysCost}+\mathrm{RedInit}+\mathrm{ReduceInput}/\mathrm{seqRead}+\frac{\mathrm{ReduceInput}}{\mathrm{BandWidth}}$

$\begin{array}{c}+\mathrm{SortCEF}*\mathrm{RIRNunber}*\log \mathrm{NunberofMap}\\ +(\mathrm{sel}*\mathrm{RIRNumber}+\mathrm{se}2*\mathrm{ReduceInput})\\ +\mathrm{ReduceInput}*(\frac{1}{\mathrm{seqRead}}+\frac{1}{\mathrm{seqWrite}})\\ +\mathrm{pa}1*\mathrm{RIRNumber}+\mathrm{pa}2*\mathrm{ReduceInput}\\ +\mathrm{ReduceInput}*\mathrm{ComplexOfReduce}*\mathrm{CEF}\\ +\mathrm{HDFSReplica}*\frac{\mathrm{ReduceOutput}}{\mathrm{BandWidth}}+\mathrm{HDFSReplica}*\frac{\mathrm{ReduceOutput}}{\mathrm{SeqWrite}}\end{array};$

Wherein, initialization when RedSysCost represents initialization Reduce task, open, close the time cost of task, be traditionally arranged to be 2000 milliseconds (ms); When representing initialization Reduce task, RedInit loads additional programs or required time of data; ReduceInput represents to carry out the data volume that a Reduce task is imported; SeqRead represents to read in the Preset Time data volume of disk; Bandwidth represents network transfer speeds in the Preset Time; SortCEF represents the coefficient that sorts; The quantity of the data that RIRNumber imports when representing to carry out the Reduce task; NumberOfMap represents the quantity of Map task; SeqWrite represents to write in the Preset Time data volume of disk; The complexity that ComplexOfReduce calculates when representing to carry out the Reduce task; HDFSReplica represents the quantity of copy; ReduceOutput represents to carry out the data volume of being exported after the Reduce task; Se1 and se2 are two parameters in serializing stage, and in the Mapredcue performance model, the needed time cost of serializing is linear with data volume, se1 and se2 are exactly two parameters of this linear relationship, say for a cluster, and be constant, therefore, can think constant; Pa1 and pa2 are two constant parameters of phase sorting; CEF is that clustered machine is handled the needed time of standard operation, also is constant.Remove in above-mentioned each parameter outside the Pass in ReduceInput, ReduceOutput, RIRNumber, NumberOfMap and the MapReduce operation there being the data volume of Map task and Reduce task, other parameter is the Given information of systemic presupposition.

Further, n is the quantity of Reduce task in the MapReduce operation if make, ReduceInput=Input/n then, wherein, Input represents total input data volume of all Reduce tasks in the MapReduce operation, and establishing Y is the data conversion rate of Reduce, be Reduce output data quantity and the ratio of importing data volume, Y=ReduceOutput/ReduceInput, ReduceOutput=Y*ReduceInput=Y*Input/n then, then

RIRNumber=ReduceInput/RIRLength=Input/n/RIRLength; Wherein, RIRlength represents to import the average length of data.

The above-mentioned formula of substitution also by behind the abbreviation, obtains:

$\begin{array}{c}\mathrm{TR}=\mathrm{RedSysCost}+\mathrm{RedInit}\\ +[2/\mathrm{SeqRead}+(1+\mathrm{HDFSReplica}*Y)*(\frac{1}{\mathrm{SeqWrite}}+\frac{1}{\mathrm{BandWidth}})\\ +(\mathrm{se}1+\mathrm{pa}1+\mathrm{SortCEF}*\log \mathrm{NumberofMap})/\mathrm{RIRLength}\\ +(\mathrm{se}2+\mathrm{pa}2+\mathrm{ComplexOfReduce}*\mathrm{CEF})]*\mathrm{Input}/n\end{array};$

If make a=RedSysCost+RedInit,

$\begin{array}{c}b=2/\mathrm{SeqRed}+(1+\mathrm{HDFSReplica}*Y)*(\frac{1}{\mathrm{SeqWrite}}+\frac{1}{\mathrm{BandWidth}})\\ +(\mathrm{se}1+\mathrm{pa}1+\mathrm{SortCEF}*\log \mathrm{NumberofMap})/\mathrm{RIRLength}\\ +(\mathrm{se}2+\mathrm{pa}2+\mathrm{ComplexOfReduce}*\mathrm{CEF})\end{array};$

For given MapReduce operation, the value of a and b can be tried to achieve by above-mentioned parameter, is equivalent to constant, therefore carries out the time cost TR=a+b*Input/n of a Reduce task; Correspondingly, TRS=n*TR=a*n+b*Input.

Formation at the T.T. cost TRS of all Reduce tasks in the time cost TR of a Reduce task and the MapReduce operation, the first order derivative sum that can select TR and TRS is that 0 point is as equilibrium point, thereby adjust the quantity of Reduce task, that is: $\frac{\∂\left(\mathrm{TR}\right)}{\∂\left(n\right)}+\frac{\∂\left(\mathrm{TRS}\right)}{\∂\left(n\right)}=0;$ The substitution equation: $n=\sqrt{b*\mathrm{Input}/a}=\sqrt{\frac{b}{a}}*\sqrt{\mathrm{Input}}.$

Thereby can determine the quantity of Reduce task according to following formula, further, for fear of thereby the long situation of more single Reduce task executions time of data volume that each Reduce handles occurring, the maximum processing data volume that can further limit single Reduce task is U, therefore, use the Reduce task of Input/U quantity at the minimum needs of each MapReduce operation and handle all data, the value of U is specifically as follows: U=ReduceJVM*ShuffleBufferPercent*ShuffleMergePercent, therefore:

$n=\{\begin{array}{c}\sqrt{b/a}*\sqrt{\mathrm{Input}},\mathrm{ifinput}<U^2*b/a\\ \frac{\mathrm{Input}}{U},\mathrm{else}\end{array}.$

Present embodiment is monitored by the Master to the MapReduce framework, thereby can be according to Map task executions situation, determine the total amount of data of Reduce task in the MapReduce operation, thereby according to the total amount of data of Reduce task and take all factors into consideration the time cost of carrying out the Reduce task and the quantity of Reduce task is adjusted, to determine the quantity of a rational Reduce task, and carry out the Reduce task according to the quantity of adjusted Reduce task, to solve problems such as the wasting of resources that caused by the quantity of the static Reduce of appointment of user task in the prior art or execution time be long, make the time cost of carrying out the Reduce task be tending towards minimum simultaneously.

One of ordinary skill in the art will appreciate that: all or part of step that realizes above-mentioned each method embodiment can be finished by the relevant hardware of programmed instruction.Aforesaid program can be stored in the computer read/write memory medium.This program is carried out the step that comprises above-mentioned each method embodiment when carrying out; And aforesaid storage medium comprises: various media that can be program code stored such as ROM, RAM, magnetic disc or CD.

The structural representation of dynamically determining device of a kind of MapReduce parameter that Fig. 2 provides for the embodiment of the invention two, as shown in Figure 2, the MapReduce parameter of present embodiment determine that dynamically device can comprise:

Job request acquisition module 201, be used to obtain the MapReduce job request, the parameter that described MapReduce job request comprises the quantity for the treatment of work data collection, default Reduce task and is used to represent whether allow start the adjustment mechanism of Reduce task quantity;

Monitoring module 202 is used for then monitoring the Map task executions if being used to represent whether allow start the parameter of the adjustment mechanism of Reduce task quantity is to allow;

Determination module 203 is used for if the quantity of executed Map task satisfies presetting first threshold, the subregion of the quantity of the default Reduce task that the output result of described executed Map task is mapped to, the then quantity of definite adjusted Reduce task;

Mapping block 204 is used for the quantity according to adjusted Reduce task, and each unenforced default Reduce task is corresponded to each adjusted Reduce task, so that carry out each adjusted Reduce task.

The device of dynamically determining of the MapReduce parameter of present embodiment can be used to carry out the technical scheme of method embodiment shown in Figure 1, and its realization principle and technique effect are similar, repeat no more herein.

Further, determination module specifically can be used for: according to the data total amount of unenforced Reduce task, determine the quantity of adjusted Reduce task.

Preferably, determination module specifically can be used for:

According to following formula, determine the quantity of adjusted Reduce task:

$\frac{\&PartialD;\left(\mathrm{TR}\right)}{\&PartialD;\left(n\right)}+\frac{\&PartialD;\left(\mathrm{TRS}\right)}{\&PartialD;\left(n\right)}=0;$

Wherein, n is the quantity of adjusted Reduce task, TR is for carrying out the time cost of an adjusted Reduce task, TRS is for carrying out the T.T. cost of adjusted all Reduce tasks, and TR depends on the data total amount of unenforced Reduce task and the quantity of adjusted Reduce task.

Concrete, the MapReduce job request in the foregoing description can also comprise the second default threshold value that is used for the execution of forward scheduling Reduce task, less than first threshold, then monitoring module specifically can be used for as if second threshold value:

If the quantity of executed Map task satisfies described second threshold value and does not satisfy presetting first threshold, the default Reduce task executions of monitoring then, a subregion in the subregion of the quantity of the corresponding respectively described default Reduce task of each default Reduce task;

And after the quantity of executed Map task satisfied presetting first threshold, indication stopped to carry out unenforced default Reduce task.

Concrete, in above-mentioned arbitrary embodiment, presetting first threshold can be the amount threshold of default Map task or default ratio, then determination module specifically is used for:

If the quantity of executed Map task satisfies the amount threshold of default Map task, then start the adjustment mechanism of Reduce task quantity, determine the quantity of adjusted Reduce task; Or,

If the ratio between the total quantity of the quantity of executed Map task and Map task satisfies default ratio, then start the adjustment mechanism of Reduce task quantity, determine the quantity of adjusted Reduce task, wherein, the total quantity of Map task is determined according to the described work data collection for the treatment of.

It should be noted that at last: above each embodiment is not intended to limit only in order to technical scheme of the present invention to be described; Although the present invention is had been described in detail with reference to aforementioned each embodiment, those of ordinary skill in the art is to be understood that: it still can be made amendment to the technical scheme that aforementioned each embodiment put down in writing, and perhaps some or all of technical characterictic wherein is equal to replacement; And these modifications or replacement do not make the essence of appropriate technical solution break away from the scope of various embodiments of the present invention technical scheme.

Claims (10)

1. the method for dynamically determining of a MapReduce parameter is characterized in that, comprising:

Obtain the MapReduce job request, the parameter that described MapReduce job request comprises the quantity for the treatment of work data collection, default Reduce task and is used to represent whether allow start the adjustment mechanism of Reduce task quantity;

If the parameter of the described adjustment mechanism that is used to represent whether to allow to start Reduce task quantity is then monitored the Map task executions for allowing;

If the quantity of executed Map task satisfies presetting first threshold, the output result of described executed Map task is mapped to the subregion of the quantity of default Reduce task, then determines the quantity of adjusted Reduce task;

According to the quantity of described adjusted Reduce task, each unenforced default Reduce task is corresponded to each adjusted Reduce task, so that carry out each adjusted Reduce task.

2. method according to claim 1 is characterized in that, the described quantity of determining adjusted Reduce task specifically comprises:

According to the data total amount of unenforced Reduce task, determine the quantity of adjusted Reduce task.

3. method according to claim 2 is characterized in that, described data total amount according to unenforced Reduce task is determined specifically to comprise the quantity of adjusted Reduce task:

According to following formula, determine the quantity of adjusted Reduce task:

$\frac{\&PartialD;\left(\mathrm{TR}\right)}{\&PartialD;\left(n\right)}+\frac{\&PartialD;\left(\mathrm{TRS}\right)}{\&PartialD;\left(n\right)}=0;$

Wherein, n is the quantity of adjusted Reduce task, TR is for carrying out the time cost of an adjusted Reduce task, TRS is for carrying out the T.T. cost of adjusted all Reduce tasks, and TR depends on the data total amount of unenforced Reduce task and the quantity of adjusted Reduce task.

4. method according to claim 1, it is characterized in that, described MapReduce job request also comprises the second default threshold value that is used for the execution of forward scheduling Reduce task, if described second threshold value is less than described first threshold, then described if the quantity of executed Map task satisfies before the presetting first threshold, also comprise:

If the quantity of described executed Map task satisfies described second threshold value and does not satisfy presetting first threshold, the default Reduce task executions of monitoring then, a subregion in the subregion of the quantity of the corresponding respectively described default Reduce task of each default Reduce task;

Described if the quantity of executed Map task satisfies after the presetting first threshold, also comprise:

Indication stops to carry out unenforced default Reduce task.

5. according to each described method in the claim 1～4, it is characterized in that, described presetting first threshold is the amount threshold of default Map task or default ratio, it is described if the quantity of described executed Map task satisfies presetting first threshold, then determine the quantity of adjusted Reduce task, be specially:

If the quantity of described executed Map task satisfies the amount threshold of described default Map task, then start the adjustment mechanism of Reduce task quantity, determine the quantity of adjusted Reduce task; Perhaps,

If the ratio between the total quantity of the quantity of described executed Map task and Map task satisfies default ratio, then start the adjustment mechanism of Reduce task quantity, determine the quantity of adjusted Reduce task, the total quantity of described Map task is determined according to the described work data collection for the treatment of.

6. dynamically definite device of a MapReduce parameter is characterized in that, comprising:

The job request acquisition module, be used to obtain the MapReduce job request, the parameter that described MapReduce job request comprises the quantity for the treatment of work data collection, default Reduce task and is used to represent whether allow start the adjustment mechanism of Reduce task quantity;

Monitoring module is used for if the parameter of the described adjustment mechanism that is used to represent whether to allow to start Reduce task quantity for allowing, is then monitored the Map task executions;

Determination module is used for if the quantity of executed Map task satisfies presetting first threshold, the subregion of the quantity of the default Reduce task that the output result of described executed Map task is mapped to, the then quantity of definite adjusted Reduce task;

Mapping block is used for the quantity according to described adjusted Reduce task, and each unenforced default Reduce task is corresponded to each adjusted Reduce task, so that carry out each adjusted Reduce task.

7. device according to claim 6 is characterized in that, described determination module specifically is used for:

According to the data total amount of unenforced Reduce task, determine the quantity of adjusted Reduce task.

8. device according to claim 7 is characterized in that, described determination module specifically is used for:

According to following formula, determine the quantity of adjusted Reduce task:

$\frac{\&PartialD;\left(\mathrm{TR}\right)}{\&PartialD;\left(n\right)}+\frac{\&PartialD;\left(\mathrm{TRS}\right)}{\&PartialD;\left(n\right)}=0;$

Wherein, n is the quantity of adjusted Reduce task, TR is for carrying out the time cost of an adjusted Reduce task, TRS is for carrying out the T.T. cost of adjusted all Reduce tasks, and TR depends on the data total amount of unenforced Reduce task and the quantity of adjusted Reduce task.

9. device according to claim 6, it is characterized in that, described MapReduce job request also comprises the second default threshold value that is used for the execution of forward scheduling Reduce task, and less than described first threshold, then described monitoring module specifically is used for as if described second threshold value:

If the quantity of described executed Map task satisfies described second threshold value and does not satisfy presetting first threshold, the default Reduce task executions of monitoring then, a subregion in the subregion of the quantity of the corresponding respectively described default Reduce task of each default Reduce task;

And after the quantity of described executed Map task satisfied presetting first threshold, indication stopped to carry out unenforced default Reduce task.

10. according to each described device of claim 6～9, it is characterized in that described presetting first threshold is the amount threshold of default Map task or default ratio, described determination module specifically is used for:

If the quantity of described executed Map task satisfies the amount threshold of described default Map task, then start the adjustment mechanism of Reduce task quantity, determine the quantity of adjusted Reduce task; Or,

If the ratio between the total quantity of the quantity of described executed Map task and Map task satisfies default ratio, then start the adjustment mechanism of Reduce task quantity, determine the quantity of adjusted Reduce task, the total quantity of described Map task is determined according to the described work data collection for the treatment of.

Images