CN109189572A - A kind of resource predictor method and system, electronic equipment and storage medium - Google Patents
A kind of resource predictor method and system, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN109189572A CN109189572A CN201810868916.7A CN201810868916A CN109189572A CN 109189572 A CN109189572 A CN 109189572A CN 201810868916 A CN201810868916 A CN 201810868916A CN 109189572 A CN109189572 A CN 109189572A
- Authority
- CN
- China
- Prior art keywords
- physics
- execution
- cost
- executes
- logic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 62
- 230000015654 memory Effects 0.000 claims description 24
- 238000011156 evaluation Methods 0.000 claims description 23
- 238000005192 partition Methods 0.000 claims description 12
- 238000013507 mapping Methods 0.000 claims description 6
- 238000004891 communication Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 description 13
- 230000006870 function Effects 0.000 description 6
- 238000005457 optimization Methods 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 241000404172 Minois dryas Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000008358 core component Substances 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3442—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for planning or managing the needed capacity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/501—Performance criteria
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5017—Task decomposition
Abstract
The present embodiments relate to big data technical field, a kind of resource predictor method and system, electronic equipment and storage medium are disclosed.The resource predictor method includes: that whole physics execution figures is generated according to the logic execution figure got；Wherein, the logic executes the physics for scheming to correspond at least one and executes figure；The cost value of each physics execution figure is determined according to cost estimation model；The cost value of each physics execution figure is matched with the performance strategy got, determines optimal physics execution figure；Wherein, the optimal physics executes the value at cost minimum of figure.Resource predictor method in the present invention obtains the index request of user so that avoiding obtaining data volume using the method for " estimating " during cost estimation in calculating process, it is ensured that the physics executive plan being calculated meets user's requirement.
Description
Technical field
The present embodiments relate to big data technical field, in particular to a kind of resource predictor method and system, electronics are set
Standby and storage medium.
Background technique
DAG is the abbreviation of directed acyclic graph (Directed Acyclic Graph).In big data processing, DAG is calculated
Calculating task is often referred to as become into several subtasks in internal breakup, according to the logical relation between several subtasks
Or sequential build is at DAG (directed acyclic graph) structure.
DAG is a kind of very common structure in distributed computing, has application in each subdivision field, such as: Dryad
(Microsoft's concurrent software platform), Flumejava (concurrency programming frame) and Tez etc., DAG is generally divided into logic execution figure and physics
Execute figure.Logic executes figure, is that calculating task is resolved into the DAG structure formed by several subtasks by certain means,
Its core is the convenience of expression, and application developer is mainly facilitated quickly to describe or construct application.Physics execution figure is that DAG is held
Row engine layers, main purpose are that the DAG calculating task for expressing on upper layer in a special manner passes through conversion and mapping, are deployed to
Run in the physical machine cluster of lower layer, this layer be DAG calculate core component, the scheduling of calculating task, bottom hardware it is fault-tolerant,
The transmitting of data and management information, the management of whole system require to be completed by this layer with running well etc..Physics executes figure
It is finally distributed on physical cluster to execute.
During logic execution figure is converted to physics execution figure, need according to the characteristics of data to the distribution plan of data
It is slightly distinguished with executive mode, this process is known as " physics executive plan optimization ".Such as under normal circumstances, data are carried out
Data should be avoided to transmit between different calculate nodes as far as possible when operation, reduced IO (Input/Output, input/output)
Delay.At this time often transmitted between the Partition (subregion) by data between the same calculate node.But when number
According to amount very little, and when can not be evenly distributed in each Partition, data above distribution policy be not then it is efficient,
And data should be rebalanced on each Partition, other processing are then carried out again, it in this way can be adequately sharp
Parallel processing is carried out with each Partition.It can be seen that attribute of optimization and data that physics executes etc. one from above-mentioned example
A little external conditions are related, are not unalterable, and the optimization is most important for the entire distributed performance executed.
Current mainstream big data system, which is all based on cost, to be estimated (Cost Estimate) model as physics and executes meter
Draw the foundation of optimization.It encapsulates the factor of some cost estimations while providing some calculation methods for cost objective
(add, subtract, multiplication and division) and identification and verification to these factor unknown-values.The factor of cost estimation is generally divided by they
Two major classes: quantifiable cost estimation factor: referring to can be with calculated cost by one quantifiable measurement index of tracking
Estimation factor (such as byte number of network or I/O)；Didactic cost estimation factor: refer to those can not quantitatively calculate at
This estimation factor, therefore some qualitative empirical values can only be provided.Simultaneously be included into cost estimation factor it is as follows: network at
This；Magnetic disc i/o cost；Central processing unit (CPU, Central Processing Unit) cost；Heuristic network cost；It opens
Hairdo disk cost；Heuristic CPU cost.
At least there are the following problems in the prior art for inventor's discovery: although incorporating network I/O, magnetic in cost estimation
Disk IO, CPU etc. index, but these indexs are all to be measured by " input data amount ", and there is a problem of inaccuracy,
Trace it to its cause is that these indexs have been obtained by the way of " estimating " before DAG execution.Such as in cluster a small number of nodes net
Network bandwidth is there are when bottleneck, and the network I/O of these nodes certainly exists performance issue, and the bottleneck of this network is difficult by " defeated
Entering data volume " this index estimates.In addition, the index of its concern of different users may be different for same service logic,
If some users wish " low latency ", and other user may want to " height is handled up ", in this case, pre- by " data volume "
Estimate that the physics executive plan got is less likely to be and fully meets user's requirement.
Summary of the invention
Embodiment of the present invention be designed to provide a kind of resource predictor method and system, electronic equipment and storage are situated between
Matter obtains user's so that avoiding obtaining data volume using the method for " estimating " during cost estimation in calculating process
Index request, it is ensured that the physics executive plan being calculated meets user's requirement.
In order to solve the above technical problems, embodiments of the present invention provide a kind of resource predictor method, including following step
It is rapid:
Whole physics execution figures is generated according to the logic execution figure got；Wherein, logic executes figure corresponding at least one
A physics executes figure；
The cost value of each physics execution figure is determined according to cost estimation model；
The cost value of each physics execution figure is matched with the performance strategy got, determines that optimal physics is held
Row figure；Wherein, optimal physics executes the value at cost minimum of figure.
Embodiments of the present invention additionally provide a kind of resource Prediction System, comprising: generation module, the first determining module and
Second determining module；
Generation module, for generating whole physics execution figures according to the logic execution figure got；Wherein, logic executes
The physics that figure corresponds at least one executes figure；
First determining module, for determining the cost value of each physics execution figure according to cost estimation model；
Second determining module is determined most for matching the cost value of each physics execution figure with performance strategy
Excellent physics executes figure；Wherein, optimal physics executes the value at cost minimum of figure.
Embodiments of the present invention additionally provide a kind of electronic equipment, comprising: at least one processor；And at least
The memory of one processor communication connection；Wherein, memory is stored with the instruction that can be executed by least one processor, instruction
It is executed by least one processor, so that at least one processor is able to carry out above-mentioned resource predictor method.
Embodiments of the present invention additionally provide a kind of computer readable storage medium, are stored with computer program, wherein
Above-mentioned resource predictor method is realized when computer program is executed by processor.
Embodiment of the present invention in terms of existing technologies, after generating whole physics execution figures, is estimated according to cost
The cost value that model determines each physics execution figure is calculated, avoids directly determining physics executory cost using preset data volume, lead
Cause achievement data inaccuracy problem, also, by the cost value of each physics execution figure with get performance strategy progress
Match, wherein performance strategy can timely requirement of the feedback user to the resource Prediction System so that the optimal object determined
Reason execution figure can satisfy user's requirement, promote user experience.
In addition, cost estimation model includes the evaluation index of at least one；
The cost value of each physics execution figure is determined according to cost estimation model, comprising: transport in preset time span
The each physics of row executes figure；Acquire each evaluation index value in each physics execution figure operational process；Referred to according to each assessment
Scale value determines the cost value of physics execution figure.
It include the evaluation index of at least one in the embodiment, in cost estimation model, and using long in preset time
The method of operation in degree determines evaluation index value, improves the accuracy that resource is estimated, and can better meet the demand of user.
In addition, performance strategy includes the mapping relations between worth of data and performance requirement；
The cost value of each physics execution figure is matched with the performance strategy got, determines that optimal physics is held
Row figure, comprising: corresponding worth of data is determined according to the performance requirement got；Worth of data and each physics are executed into figure
Cost value matched；Determine the smallest physics execution figure of the cost value for meeting performance strategy.
In the embodiment, the corresponding specific worth of data of the performance requirement of user can want abstract performance
It asks and is converted into quantitative requirement, the user experience is improved, also further improves the accuracy that resource is estimated.
In addition, the cost value of each physics execution figure is matched with performance strategy, determine that optimal physics executes
Before figure, resource predictor method further include: obtain the performance strategy of user's input.
In the embodiment, the performance strategy of user's input is obtained during resource is estimated, so that finally determine
Physics execution figure more meets user demand, meets the requirement of user, the standard that further promotion user experience and resource are estimated
True property.
In addition, evaluation index include: network inputs/output cost, disk input/output cost, central processor CPU at
Sheet, heuristic network inputs/output cost, heuristic disk cost, heuristic CPU cost.
In addition, resource predictor method also wraps before generating whole physics execution figures according to the logic execution figure got
It includes: according to the Program Generating operator for having write completion；Figure is executed according to the logic that operator generates directed acyclic graph DAG；Wherein, it patrols
Collecting and executing figure includes operator.
In addition, generating whole physics execution figures according to the logic execution figure got, comprising: executed in figure according to logic
Operator determine whole partition data distribution policies；Wherein, at least two subregions in different partition data distribution policies
Data distribution is different；It determines the corresponding physics execution figure of each partition data distribution policy, obtains whole physics and execute
Figure.
Detailed description of the invention
One or more embodiments are illustrated by the picture in corresponding attached drawing, these exemplary theorys
The bright restriction not constituted to embodiment.
Fig. 1 is the flow chart of resource predictor method in first embodiment of the invention；
Fig. 2 is the flow chart of resource predictor method in second embodiment of the invention；
Fig. 3 is the structure chart of resource Prediction System in third embodiment of the invention；
Fig. 4 is the structure chart of resource Prediction System in four embodiment of the invention；
Fig. 5 is the structure chart of electronic equipment in fifth embodiment of the invention.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with attached drawing to the present invention
Each embodiment be explained in detail.However, it will be understood by those skilled in the art that in each embodiment party of the present invention
In formula, in order to make the reader understand this application better, many technical details are proposed.But even if without these technical details
And various changes and modifications based on the following respective embodiments, the application technical solution claimed also may be implemented.
The first embodiment of the present invention is related to a kind of resource predictor methods.Detailed process is as shown in Figure 1.Including walking as follows
It is rapid:
Step 101: whole physics execution figures is generated according to the logic execution figure got.
Wherein, logic executes the physics for scheming to correspond at least one and executes figure.Specifically, in the program for getting user and writing
Later, the Program Generating logic execution figure write according to user, then determine whole physics execution figures.
Specifically, step 101 specifically includes: executing the operator in figure according to logic and determine that whole partition datas is distributed
Strategy；Wherein, the data distribution of at least two subregions is different in different partition data distribution policies；Determine each partition data
The corresponding physics of distribution policy executes figure, obtains whole physics execution figures.
One in the specific implementation, the program that user writes is also referred to as the service logic that user writes, user under normal circumstances
It is API (Application Programming Interface, the application programming provided by writing the system of program
Interface) write program, wherein it include preset defined function in API, so that user carries out writing for service logic, wherein pre-
If defined function in include operator, operator is the mapping relations in function, therefore, user write finishing service logic it
Afterwards, the logic execution figure based on operator can be generated, wherein the logic execution figure, which is laid particular emphasis on, to be described user by operator and write
Service logic.
It is noted that user is handled data when writing service logic, using API, specific processing mode
Including but not limited to: Map (data conversion treatment): handling individual element, and output is still individual element；FlatMap
(data conversion treatment): handling individual element, when output multiple elements；Filter (filtration treatment): data are carried out
Filtering；Count (statistical disposition): statistical counting is carried out to data.Specifically, after user edits finishing service logic,
It is that logic execution figure is generated according to the logical order of the API called during user written program.
Specifically, possible subregion distribution policy is determined according to the operator in logic execution figure, according to data distribution strategy
Determine that whole physics executive plans, the determination of subregion distribution policy include but is not limited to following several as far as possible: same
Data are transmitted between the subregion of the identical number of two operators in a calculate node, due to not being related to network transmission, are transmitted
Efficiency is very high；Purpose subregion is carried out according to the cryptographic Hash (Hash) of the attribute (Key) in data between all calculate nodes
Transmitting guarantees that the data mutual transmission of identical Key is defeated on the same subregion, because the data being related between each node pass
It is defeated, so the output to network is related；Data are distributed at random between the subregion in all calculate nodes.Wherein, it is generating entirely
The physics execution figure in portion is all possibility of the meeting according to data distribution in each subregion, generates different physics execution figures, in turn
Determine whole physics execution figures.
Step 102: the cost value of each physics execution figure is determined according to cost estimation model.
Specifically, cost estimation model includes the evaluation index of at least one；Wherein, evaluation index include: network inputs/
Export cost, disk input/output cost, central processing unit (Central Processing Unit, referred to as " CPU ") cost,
Heuristic network inputs/output cost, heuristic disk cost, heuristic CPU cost.
Wherein, it can determine that the value at cost of physics execution figure according to the data in evaluation index, specifically, evaluation index
In further include, physics execute figure implementation procedure in CPU utilization rate, network inputs/output (Input/Output, I/O), disk
I/O network delay etc., specifically, the utilization rate of CPU indicates that physics executes the average CPU's of the calculate node in figure implementation procedure
Utilization rate, network I/O indicate that physics executes the average value of network I/O in figure implementation procedure, and magnetic disc i/o indicates that physics executes figure and exists
The average value of magnetic disc i/o in implementation procedure, network delay indicate that physics executes the end-to-end of figure network channel in the process of implementation
Time delay.
It should be noted that the data for obtaining evaluation index in the implementation procedure of physics execution figure are capable of determining that the object
Reason executes the value at cost of figure.The the data of the evaluation index of acquisition the more more can embody cost in the physics execution figure
Value, evaluation index mentioned above are merely illustrative, are not particularly limited herein for evaluation index.
Specifically, step 102 specifically includes: running each physics execution figure in preset time span；Acquisition is every
A physics executes each evaluation index value in figure operational process；The cost of physics execution figure is determined according to each evaluation index value
Value.
Wherein, in preset time span according to physics execute figure execution after, such as obtain CPU utilization rate,
The assessed value of the evaluation indexes such as network I/O, magnetic disc i/o, network delay is obtained according to the assessed value of each evaluation index got
The cost value total to one such as referred to as Cn, wherein n indicates n-th of physics executive plan figure.
It should be noted that determining cost value according to the physics execution figure executed in preset time, can guarantee providing
Current system is obtained when the process that source is estimated to the value at cost of physics execution figure, which can be interpreted as to dynamic acquisition user
Current system runs the cost value that physics executes figure.
Step 103: the cost value of each physics execution figure being matched with the performance strategy got, is determined optimal
Physics execute figure.
Wherein, optimal physics executes the value at cost minimum of figure.
Specifically, performance strategy includes the mapping relations between worth of data and performance requirement.Then carrying out step 103
It also needs to be implemented before: obtaining the performance strategy of user's input.For example, the performance strategy of user can be with are as follows: height is handled up
Or low delay.Wherein, low delay (LOW_LATENCY) indicates to guarantee to arrive in network channel middle-end as far as possible under the performance strategy
The delay at end is minimum；Height is handled up (HIGH_THROGHOUTPUT), indicates the number handled in guarantor unit's time under the performance strategy
Maximum according to amount, then in corresponding performance indicator, the utilization rate of utilization rate, magnetic disc i/o, network I/O of CPU etc. are all maximized.On
The performance strategy stated is merely illustrative.
Specifically, system needs to convert numerical value for the performance requirement of user since user is after setting performance strategy
On restriction, it is also necessary to convert performance indicator to the requirement of specific evaluation index, e.g., under the strategy of LOW_LATENCY,
The utilization rate of CPU is not more than 10%, and the average reading of magnetic disc i/o is not higher than 10000 bytes-per-seconds, the delay average out to of network I/O
0.001ms (every 64 byte of message) etc..In addition, can also add evaluation index again according to demand when user sets performance strategy
Requirement, be not specifically limited herein.
Specifically, step 103 specifically includes: determining corresponding worth of data according to the performance requirement got；It will
Worth of data is matched with the cost value of each physics execution figure；Determine the smallest physics of the cost value for meeting performance strategy
Execute figure.
Wherein, after getting the performance requirement of user, according to the mapping relations between performance requirement and worth of data,
Cost Value Data is matched with the cost value of each physics execution figure and is exactly screened by the value for determining corresponding cost value
The physics execution figure for meeting the performance requirement of user out, will meet user requirement physics execution figure cost value according to from it is small to
Big sequence is arranged, and selects the smallest physics execution figure of cost value as final physics execution figure.
It should be noted that during determining optimal physics execution figure, the performance requirement of user is obtained, and
It scans for, within a preset time respectively once gets the operation of each physics execution figure each in whole physics execution figures
Physics executes the cost value of figure, avoids falling into local optimum, and compare the data processing of general open source, more can satisfy user
Requirement, and execution efficiency is higher.
In terms of existing technologies, it after generating whole physics execution figures, is determined according to cost estimation model each
Physics executes the cost value of figure, avoids directly determining physics executory cost using preset data volume, causes achievement data inaccurate
True problem, also, the cost value of each physics execution figure is matched with the performance strategy got, wherein performance plan
Slightly can timely requirement of the feedback user to the resource Prediction System so that the optimal physics execution figure determined be satisfaction
User requires, and promotes user experience.
Second embodiment of the present invention is related to a kind of resource predictor method.Second embodiment is big with first embodiment
It causes identical, is in place of the main distinction: in second embodiment of the invention, holding before generating whole physics execution figures
Row step.Detailed process is as shown in Figure 2.
Specifically, the resource predictor method includes the following steps:
It should be noted that step 203 is identical to step 103 as the step 101 in first embodiment to step 205,
Details are not described herein again.
Step 201: according to the Program Generating operator for having write completion.
Step 202: figure is executed according to the logic that operator generates directed acyclic graph DAG；Wherein, it includes calculating that logic, which executes figure,
Son.
Specifically, the corresponding DAG of Program Generating first write according to user is needed to patrol before determining physics execution figure
It collects and executes figure, the logic execution figure of DAG structure, which refers to, also is understood as the program that user writes for calculating task, in internal breakup
For several subtasks, by between these subtasks logical relation or sequential build at the logic of DAG structure execute figure, it is practical
On, when the logic for constructing DAG executes figure, the operator in program for needing to be write according to user is generated, and DAG logic can be claimed to execute
Figure is that the logic generated based on operator executes figure.
It should be noted that step 101 of the user written program with the relationship of corresponding operator in the first embodiment
In have been described, details are not described herein again.
The step of various methods divide above, be intended merely to describe it is clear, when realization can be merged into a step or
Certain steps are split, multiple steps are decomposed into, as long as including identical logical relation, all in the protection scope of this patent
It is interior；To adding inessential modification in algorithm or in process or introducing inessential design, but its algorithm is not changed
Core design with process is all in the protection scope of the patent.
Third embodiment of the invention is related to a kind of resource Prediction System, as shown in Figure 3, comprising: generation module 301,
One determining module 302 and the second determining module 303；
Generation module 301, for generating whole physics execution figures according to the logic execution figure got；Wherein, logic
It executes the physics for scheming to correspond at least one and executes figure；
First determining module 302, for determining the cost value of each physics execution figure according to cost estimation model；
Second determining module 303 is determined for matching the cost value of each physics execution figure with performance strategy
Optimal physics executes figure；Wherein, optimal physics executes the value at cost minimum of figure.
It is not difficult to find that present embodiment is system embodiment corresponding with first embodiment, present embodiment can
It works in coordination implementation with first embodiment.The relevant technical details mentioned in first embodiment are in the present embodiment still
Effectively, in order to reduce repetition, which is not described herein again.
It is noted that each module involved in present embodiment is logic module, and in practical applications, one
A logic unit can be a physical unit, be also possible to a part of a physical unit, can also be with multiple physics lists
The combination of member is realized.In addition, in order to protrude innovative part of the invention, it will not be with solution institute of the present invention in present embodiment
The technical issues of proposition, the less close unit of relationship introduced, but this does not indicate that there is no other single in present embodiment
Member.
Four embodiment of the invention is related to a kind of resource Prediction System.4th embodiment and third embodiment are substantially
Identical, be in place of the main distinction: in four embodiment of the invention, specifically illustrating in resource Prediction System further includes calculating
Sub- generation module and DAG generation module, specific structure are as shown in Figure 4.
It should be noted that only illustrate increased module in present embodiment, in third embodiment it is stated that module
It repeats no more.
It include operator generation module 401 and DAG generation module 402 in the resource Prediction System.
Operator generation module 401, for according to the Program Generating operator for having write completion.
DAG generation module 402, the logic for generating directed acyclic graph DAG according to operator execute figure；Wherein, logic is held
Row figure includes operator.
Since second embodiment is corresponded to each other with present embodiment, present embodiment can be mutual with second embodiment
Match implementation.The relevant technical details mentioned in second embodiment are still effective in the present embodiment, implement second
The attainable technical effect of institute similarly may be implemented in the present embodiment in mode, no longer superfluous here in order to reduce repetition
It states.
Fifth embodiment of the invention is related to a kind of electronic equipment, and specific structure is as shown in Figure 5.Including at least one processing
Device 501；And the memory 502 with the communication connection of at least one processor 501.Wherein, be stored with can be by extremely for memory 502
The instruction that a few processor 501 executes, instruction is executed by least one processor 501, so that at least one 501 energy of processor
Enough execute resource predictor method.
In present embodiment, for processor 501 is with central processing unit (Central Processing Unit, CPU),
For memory 502 is with readable and writable memory (Random Access Memory, RAM).Processor 501, memory 502 can be with
It is connected by bus or other modes, in Fig. 5 for being connected by bus.Memory 502 is used as a kind of non-volatile meter
Calculation machine readable storage medium storing program for executing can be used for storing non-volatile software program, non-volatile computer executable program and module,
As realized in the application embodiment, the program of resource predictor method is stored in memory 502.Processor 501 passes through operation
Non-volatile software program, instruction and the module being stored in memory 502, thereby executing equipment various function application with
And data processing, that is, realize above-mentioned resource predictor method.
Memory 502 may include storing program area and storage data area, wherein storing program area can store operation system
Application program required for system, at least one function；It storage data area can the Save option list etc..In addition, memory can wrap
High-speed random access memory is included, can also include nonvolatile memory, for example, at least disk memory, a flash memories
Part or other non-volatile solid state memory parts.In some embodiments, it includes relative to processor that memory 502 is optional
501 remotely located memories, these remote memories can pass through network connection to external equipment.The example packet of above-mentioned network
Include but be not limited to internet, intranet, local area network, mobile radio communication and combinations thereof.
One or more program module is stored in memory 502, is executed when by one or more processor 501
When, execute the resource predictor method in above-mentioned first or second method implementation.
Resource predictor method provided by the application embodiment can be performed in the said goods, has the corresponding function of execution method
Can module and beneficial effect, the not technical detail of detailed description in the present embodiment, reference can be made to the application embodiment is mentioned
The resource predictor method of confession.
Sixth embodiment of the invention is related to a kind of computer readable storage medium, which is computer
Readable storage medium storing program for executing is stored with computer instruction in the computer readable storage medium, which enables a computer to
Execute resource predictor method involved in the application first or second method implementation.
It will be appreciated by those skilled in the art that realizing that all or part of the steps in above embodiment method is can to lead to
Program is crossed to instruct relevant hardware and complete, which is stored in a storage medium, including some instructions use so that
One equipment (can be single-chip microcontroller, chip etc.) or processor (processor) execute side described in each embodiment of the application
The all or part of the steps of method.And storage medium above-mentioned includes: USB flash disk, mobile hard disk, read-only memory (ROM, Read-Only
Memory), the various media that can store program code such as random access memory, magnetic or disk.
It will be understood by those skilled in the art that the respective embodiments described above are to realize specific embodiment party of the invention
Formula, and in practical applications, can to it, various changes can be made in the form and details, without departing from spirit and model of the invention
It encloses.
Claims (10)
1. a kind of resource predictor method characterized by comprising
Whole physics execution figures is generated according to the logic execution figure got；Wherein, the logic executes figure corresponding at least one
A physics executes figure；
The cost value of each physics execution figure is determined according to cost estimation model；
The cost value of each physics execution figure is matched with the performance strategy got, determines that optimal physics is held
Row figure；Wherein, the optimal physics executes the value at cost minimum of figure.
2. resource predictor method according to claim 1, which is characterized in that the cost estimation model includes at least one
Evaluation index；
The cost value of each physics execution figure is determined according to cost estimation model, comprising:
Each physics execution figure is run in preset time span；
Acquire each of each physics execution figure operational process evaluation index value；
The cost value of the physics execution figure is determined according to each evaluation index value.
3. resource predictor method according to any one of claim 1 to 2, which is characterized in that the performance strategy includes
Mapping relations between worth of data and performance requirement；
The cost value of each physics execution figure is matched with the performance strategy got, determines that optimal physics is held
Row figure, comprising:
Corresponding worth of data is determined according to the performance requirement got；
The worth of data is matched with the cost value of each physics execution figure；
Determine the smallest physics execution figure of the cost value for meeting the performance strategy.
4. resource predictor method according to claim 3, which is characterized in that each physics is executed to the cost value of figure
It is matched with performance strategy, before determining optimal physics execution figure, the resource predictor method further include:
Obtain the performance strategy of user's input.
5. resource predictor method according to claim 2, which is characterized in that the evaluation index includes: network inputs/defeated
Cost, disk input/output cost, central processor CPU cost, heuristic network inputs/output cost, heuristic disk out
Cost, heuristic CPU cost.
6. according to claim 1 to 2 described in any item resource predictor methods, which is characterized in that held according to the logic got
Before row figure generates whole physics execution figures, the resource predictor method further include:
According to the Program Generating operator for having write completion；
Figure is executed according to the logic that the operator generates directed acyclic graph DAG；Wherein, it includes the calculation that the logic, which executes figure,
Son.
7. resource predictor method according to claim 6, which is characterized in that generated according to the logic execution figure got
Whole physics execute figure, comprising:
The operator in figure, which is executed, according to the logic determines whole partition data distribution policies；Wherein, the different subregion
The data distribution of at least two subregions is different in data distribution strategy；
It determines each corresponding physics execution figure of the partition data distribution policy, obtains whole physics execution figures.
8. a kind of resource Prediction System, which is characterized in that including generation module, the first determining module and the second determining module；
The generation module, for generating whole physics execution figures according to the logic execution figure got；Wherein, the logic
It executes the physics for scheming to correspond at least one and executes figure；
First determining module, for determining the cost value of each physics execution figure according to cost estimation model；
Second determining module is determined for matching the cost value of each physics execution figure with performance strategy
Optimal physics executes figure out；Wherein, the optimal physics executes the value at cost minimum of figure.
9. a kind of electronic equipment characterized by comprising
At least one processor；And the memory being connect at least one described processor communication；Wherein, the memory
It is stored with the instruction that can be executed by least one described processor, described instruction is executed by least one described processor, so that
At least one described processor is able to carry out resource predictor method as claimed in claim 1.
10. a kind of computer readable storage medium, is stored with computer program, wherein the computer program is held by processor
Claim 1-7 described in any item resource predictor methods are realized when row.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810868916.7A CN109189572B (en) | 2018-08-02 | 2018-08-02 | Resource estimation method and system, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810868916.7A CN109189572B (en) | 2018-08-02 | 2018-08-02 | Resource estimation method and system, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109189572A true CN109189572A (en) | 2019-01-11 |
| CN109189572B CN109189572B (en) | 2021-06-04 |
Family
ID=64920460
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810868916.7A Active CN109189572B (en) | 2018-08-02 | 2018-08-02 | Resource estimation method and system, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109189572B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110532291A (en) * | 2019-07-25 | 2019-12-03 | 中国科学院计算技术研究所 | Model conversion method and system between deep learning frame based on minimum Executing Cost |
| WO2020207393A1 (en) * | 2019-04-09 | 2020-10-15 | 华为技术有限公司 | Operator operation scheduling method and apparatus |
| CN111967902A (en) * | 2020-08-04 | 2020-11-20 | 甘棠软件***（上海）有限公司 | Method and apparatus for estimating cost |
| WO2021114757A1 (en) * | 2019-12-09 | 2021-06-17 | 北京迈格威科技有限公司 | Optimization method and apparatus for computation graph, computer device, and storage medium |
| CN116545770A (en) * | 2023-07-03 | 2023-08-04 | 上海观安信息技术股份有限公司 | Scene detection method, device, medium and equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130061203A1 (en) * | 2011-09-06 | 2013-03-07 | International Business Machines Corporation | Modeling Task-Site Allocation Networks |
| CN105117286A (en) * | 2015-09-22 | 2015-12-02 | 北京大学 | Task scheduling and pipelining executing method in MapReduce |
| CN105868019A (en) * | 2016-02-01 | 2016-08-17 | 中国科学院大学 | Automatic optimization method for performance of Spark platform |
| US20170185457A1 (en) * | 2015-03-27 | 2017-06-29 | Intel Corporation | Technologies for offloading and on-loading data for processor/coprocessor arrangements |
| CN107038070A (en) * | 2017-04-10 | 2017-08-11 | 郑州轻工业学院 | The Parallel Task Scheduling method that reliability is perceived is performed under a kind of cloud environment |
-
2018
- 2018-08-02 CN CN201810868916.7A patent/CN109189572B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130061203A1 (en) * | 2011-09-06 | 2013-03-07 | International Business Machines Corporation | Modeling Task-Site Allocation Networks |
| US20170185457A1 (en) * | 2015-03-27 | 2017-06-29 | Intel Corporation | Technologies for offloading and on-loading data for processor/coprocessor arrangements |
| CN105117286A (en) * | 2015-09-22 | 2015-12-02 | 北京大学 | Task scheduling and pipelining executing method in MapReduce |
| CN105868019A (en) * | 2016-02-01 | 2016-08-17 | 中国科学院大学 | Automatic optimization method for performance of Spark platform |
| CN107038070A (en) * | 2017-04-10 | 2017-08-11 | 郑州轻工业学院 | The Parallel Task Scheduling method that reliability is perceived is performed under a kind of cloud environment |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020207393A1 (en) * | 2019-04-09 | 2020-10-15 | 华为技术有限公司 | Operator operation scheduling method and apparatus |
| CN111796917A (en) * | 2019-04-09 | 2020-10-20 | 华为技术有限公司 | Operator operation scheduling method and device |
| US11934866B2 (en) | 2019-04-09 | 2024-03-19 | Huawei Technologies Co., Ltd. | Operator operation scheduling method and apparatus to determine an optimal scheduling policy for an operator operation |
| CN110532291A (en) * | 2019-07-25 | 2019-12-03 | 中国科学院计算技术研究所 | Model conversion method and system between deep learning frame based on minimum Executing Cost |
| CN110532291B (en) * | 2019-07-25 | 2022-07-12 | 中国科学院计算技术研究所 | Method and system for converting deep learning frame model based on minimum execution cost |
| WO2021114757A1 (en) * | 2019-12-09 | 2021-06-17 | 北京迈格威科技有限公司 | Optimization method and apparatus for computation graph, computer device, and storage medium |
| CN111967902A (en) * | 2020-08-04 | 2020-11-20 | 甘棠软件***（上海）有限公司 | Method and apparatus for estimating cost |
| CN116545770A (en) * | 2023-07-03 | 2023-08-04 | 上海观安信息技术股份有限公司 | Scene detection method, device, medium and equipment |
| CN116545770B (en) * | 2023-07-03 | 2023-09-01 | 上海观安信息技术股份有限公司 | Scene detection method, device, medium and equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109189572B (en) | 2021-06-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109189572A (en) | A kind of resource predictor method and system, electronic equipment and storage medium | |
| US9727383B2 (en) | Predicting datacenter performance to improve provisioning | |
| US7752239B2 (en) | Risk-modulated proactive data migration for maximizing utility in storage systems | |
| US7890315B2 (en) | Performance engineering and the application life cycle | |
| US9031826B2 (en) | Method and apparatus for simulating operation in a data processing system | |
| US10120724B2 (en) | Optimized resource metering in a multi tenanted distributed file system | |
| Gibilisco et al. | Stage aware performance modeling of dag based in memory analytic platforms | |
| US20150095089A1 (en) | Workload management for license cost optimization | |
| CN104035747B (en) | Method and apparatus for parallel computation | |
| Li et al. | Amoeba: Qos-awareness and reduced resource usage of microservices with serverless computing | |
| CN105431815A (en) | Input-output prioritization for database workload | |
| Lattuada et al. | Optimal resource allocation of cloud-based spark applications | |
| Feljan et al. | Task allocation optimization for multicore embedded systems | |
| CN116501503B (en) | Architecture mapping method and device for load task, computer equipment and medium | |
| WO2021114757A1 (en) | Optimization method and apparatus for computation graph, computer device, and storage medium | |
| Kiran et al. | Execution time prediction of imperative paradigm tasks for grid scheduling optimization | |
| Wang et al. | Test case prioritization for service-oriented workflow applications: A perspective of modification impact analysis | |
| CN107562520A (en) | The method and apparatus of the internal memory of dilatation virtual machine | |
| CN105871618A (en) | Multi-resource scheduling method for coping with uncertain requirement in cloud scheduler | |
| CN109324886A (en) | cluster resource scheduling method and device | |
| WO2013070226A1 (en) | Computer system performance management with control variables, performance metrics and/or desirability functions | |
| Abdel-Basset et al. | Some appraisal criteria for multi-mode scheduling problem | |
| Na et al. | An adaptive replanning mechanism for dependable service-based systems | |
| Pucher et al. | EXFed: Efficient Cross-Federation with Availability SLAs on Preemptible IaaS Instances | |
| CN110209645A (en) | Task processing method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A resource estimation method and its system, electronic devices, and storage media Granted publication date: 20210604 Pledgee: Bank of China Limited by Share Ltd. Nanjing Jiangning branch Pledgor: YI TAI FEI LIU INFORMATION TECHNOLOGY LLC Registration number: Y2024980008211 |