WO2021088207A1 - 云计算集群混部作业调度方法、装置、服务器及存储装置 - Google Patents
云计算集群混部作业调度方法、装置、服务器及存储装置 Download PDFInfo
- Publication number
- WO2021088207A1 WO2021088207A1 PCT/CN2019/125432 CN2019125432W WO2021088207A1 WO 2021088207 A1 WO2021088207 A1 WO 2021088207A1 CN 2019125432 W CN2019125432 W CN 2019125432W WO 2021088207 A1 WO2021088207 A1 WO 2021088207A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- job scheduling
- server
- scheduling request
- hybrid
- cloud computing
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Definitions
- This application relates to the field of cloud computing technology, and in particular to a cloud computing cluster hybrid job scheduling method, device, server, and storage device.
- Cloud computing services are widely used in data center service platforms due to their high scalability, on-demand services, and extremely cheap features. More and more companies and individuals choose to use cloud computing platforms to run applications. As the types of services in cloud clusters become more and more diversified, the requirements for job scheduling are getting higher and higher. In order to improve the overall resource utilization of the cloud computing cluster, the relevant personnel of the cloud data center began to gradually pay attention to the mixed use of physical hosts.
- the first is load characteristic analysis and resource prediction. This method analyzes the historical operation logs of the load, and uses server learning models to predict the next moment of resource consumption, so as to carry out reasonable resource allocation. This method shortens the gap between resource requests and actual resource consumption to a certain extent, and reduces the oversold situation of server resources.
- the second is the deployment of the hybrid strategy in the cluster, which is mainly to mix real-time services and computing services in the same physical machine.
- most of the current job scheduling schemes consider a single level, that is, the resource requirements of the resource layer or the scheduling priority of the job itself.
- the methods used include classical heuristic algorithms such as neural networks and ant colony algorithm, queuing theory, etc., or It is an improvement strategy based on the classic algorithm model, which ignores the problem of server cluster performance degradation caused by interference between different types of loads during hybrid deployment.
- the present application provides a cloud computing cluster hybrid job scheduling method, device, server, and storage device to solve the problem of server cluster performance degradation caused by interference between loads in the existing job scheduling scheme.
- a technical solution adopted in this application is to provide a cloud computing cluster hybrid job scheduling method, which includes:
- System scheduling of job scheduling requests is performed according to the target hybrid deployment mode.
- the calculation formula for the recommendation score of each hybrid deployment mode is:
- the method before the step of simulating the mixed deployment of the multiple load types to the server based on the maximum resource value, the method further includes:
- the load type closest to the new load type is confirmed through similarity analysis to confirm the maximum resource value required by the new load type.
- the step of screening out a list of servers that meet the required resources includes:
- the step of confirming the target hybrid deployment mode with the highest recommended score from the server list includes:
- the hybrid deployment model with the highest recommended score among the hybrid deployment models that meet the job scheduling request and multiple load types is used as the target hybrid deployment model;
- the present invention also provides a cloud computing cluster hybrid deployment job scheduling device, which includes:
- the analysis module is used to analyze the historical operation log data of multiple load types obtained in advance, and confirm the maximum resource value required for each load type during operation;
- the simulation module is used to simulate the mixed deployment of multiple load types to the server based on the maximum resource value, and obtain the frequency of hardware events in each mixed deployment mode;
- the calculation module is used to calculate the recommended score for each hybrid deployment mode based on the frequency of hardware events
- the confirmation module is used to confirm that the required resources of the job scheduling request are met when the job scheduling request is received;
- the screening module is used to screen out the server list that meets the required resources, and confirm the target hybrid deployment mode with the highest recommended score from the server list;
- the scheduling module is used to systematically schedule the job scheduling request according to the target hybrid deployment mode.
- the present invention also provides a server.
- the server includes a processor and a memory coupled with the processor, wherein:
- the memory stores program instructions for implementing any one of the foregoing cloud computing cluster hybrid job scheduling methods
- the processor is used to execute program instructions stored in the memory to schedule job scheduling requests.
- the present invention also provides a storage device that stores program files that can implement any one of the above-mentioned cloud computing cluster hybrid job scheduling methods.
- the beneficial effect of the present application is that the present invention obtains the recommended score of the hybrid deployment mode formed by the mixed deployment of different load types by analyzing the characteristics of each hardware event when different load types are mixed deployment, and confirms when the job scheduling request is received.
- the resources required for job scheduling requests are selected from the servers that meet the required resources, and the hybrid deployment mode with the highest score is recommended when the job scheduling request is mixed with other loads, and the job scheduling request is systematically scheduled according to this hybrid deployment mode.
- FIG. 1 is a schematic flowchart of a cloud computing cluster hybrid job scheduling method according to the first embodiment of the present invention
- FIG. 2 is a schematic flowchart of a cloud computing cluster hybrid job scheduling method according to a second embodiment of the present invention
- FIG. 3 is a schematic flowchart of a cloud computing cluster hybrid job scheduling method according to a third embodiment of the present invention.
- FIG. 4 is a schematic flowchart of a cloud computing cluster hybrid job scheduling method according to a fourth embodiment of the present invention.
- FIG. 5 is a schematic structural diagram of a cloud computing cluster hybrid job scheduling device according to an embodiment of the present invention.
- FIG. 6 is a schematic structural diagram of a server according to an embodiment of the present invention.
- FIG. 7 is a schematic structural diagram of a storage device according to an embodiment of the present invention.
- FIG. 1 is a schematic flowchart of a cloud computing cluster hybrid job scheduling method according to the first embodiment of the present invention. It should be noted that, if there is substantially the same result, the method of the present invention is not limited to the sequence of the process shown in FIG. 1. As shown in Figure 1, the method includes steps:
- Step S1 Analyze the pre-obtained historical operation log data of multiple load types, and confirm the maximum resource value required for each load type during operation.
- historical running log data of multiple load types needs to be collected in advance, and historical running log data includes attribute values such as CPU utilization, memory utilization, disk usage, network I/O, etc.
- the sampling frequency can be set For sampling every 60 seconds.
- the maximum resource value required during the operation of each load type can be obtained, and the resource value includes the CPU, memory and other resources required during the load operation.
- Step S2 Simulate mixed deployment of multiple load types to the server based on the maximum resource value, and obtain the frequency of occurrence of hardware events in each mixed deployment mode.
- hardware events include Instructions per Cycle, Branch prediction misses, Context switches, Cache misses, L1 data cache load misses, Last level cache misses, dTLB load misses, iTLB load misses, etc.
- the resource situation of the server is also obtained, and the mixed deployment of each load type to the server is simulated according to the maximum resource value required for operation of each load type. Then, by analyzing the characteristics of hardware events in the hybrid deployment mode, we can learn the frequency of hardware events.
- Step S3 Calculate the recommendation score of each hybrid deployment mode based on the frequency of hardware events.
- the frequency of hardware events when w1 is deployed alone can be obtained in advance.
- ⁇ i is a preset hardware event. the weight of.
- Step S4 When the job scheduling request is received, it is confirmed that the required resources of the job scheduling request are met.
- step S5 the server list that meets the required resources is filtered out, and the target hybrid deployment mode with the highest recommended score is confirmed from the server list.
- Step S6 system scheduling the job scheduling request according to the target hybrid deployment mode.
- the recommended score for the mixed deployment mode formed by the mixed deployment of different load types is obtained.
- the resources required for the job scheduling request are confirmed , And then select the highest-scoring hybrid deployment mode recommended when the job scheduling request is mixed with other loads from the servers that meet the required resources, and perform system scheduling on the job scheduling request based on this hybrid deployment mode, which comprehensively considers resources and differences
- There are two aspects of interference between loads which avoids the problem of server performance degradation caused by mutual interference between loads, and improves the overall resource utilization of cloud servers.
- step S2 it also includes:
- step S10 when a new load type appears, the load type closest to the new load type is confirmed through similarity analysis to confirm the maximum resource value required by the new load type.
- the similarity analysis between the new load type and the existing load type is performed, so as to confirm the load type closest to the new load type, and then compare the new load type.
- the required maximum resource value of the load type is predicted, and then the new load type and the existing load type are simulated mixed deployment scenarios according to the predicted maximum resource value, thereby improving the cloud computing cluster hybrid provided by this embodiment of the present invention.
- step S4 the method further includes:
- step S20 it is judged whether there is a server that satisfies the required resources of the job scheduling request. If yes, go to step S5 to step S6; if no, go to step 21.
- Step S21 Keep the job scheduling request and continue to wait until there is a server that meets the required resources.
- step S4 it further includes:
- Step S30 Initialize the priority of the job scheduling request.
- step S31 it is judged whether the priority is zero. If the priority is zero, execute step S32; if the priority is not zero, execute step S5 to step S6.
- Step S32 directly execute the system scheduling operation.
- step S5 includes:
- Step S33 It is judged whether there is a server that meets the required resources. If it exists, execute step S34 to step S35; if it does not exist, reduce the priority by one, and execute step S31 and subsequent steps again.
- step S34 the servers that meet the required resources are filtered out, and a server list is obtained.
- Step S35 Determine whether there is a mixed deployment mode that meets the job scheduling request and multiple load types in the server list. If it exists, execute step S36; if it does not exist, reduce the priority by one, and execute step S31 and subsequent steps again.
- step S36 the mixed deployment mode with the highest recommended score among the mixed deployment modes that meet the job scheduling request and multiple load types is used as the target mixed deployment mode.
- the job scheduling request in order to avoid too long waiting time for the job scheduling request, each time it is determined that there is no server that meets the required resources or the server list does not exist in a mixed deployment mode that meets the job scheduling request and multiple load types, the job The priority of the scheduling request is reduced by one until the priority drops to zero, and the job scheduling request is scheduled for the system.
- FIG. 5 shows a schematic structural diagram of a cloud computing cluster hybrid deployment job scheduling device of the present invention.
- the cloud computing cluster hybrid deployment job scheduling device 1 includes an analysis module 10, a simulation module 11, a calculation module 12, a confirmation module 13, a screening module 14 and a scheduling module 15.
- the analysis module 10 is used to analyze the pre-obtained historical operation log data of multiple load types to confirm the maximum resource value required for each load type to run; the simulation module 11 is used to simulate multiple Load types are deployed to the server in a mixed manner, and the frequency of hardware events in each mixed deployment mode is obtained; the calculation module 12 is used to calculate the recommended score of each mixed deployment mode based on the frequency of hardware events; the confirmation module 13 is used to receive When it comes to the job scheduling request, confirm the required resources that meet the job scheduling request; the screening module 14 is used to filter the list of servers that meet the required resources, and confirm the target hybrid deployment mode with the highest recommended score from the server list; scheduling module 15 , Used for system scheduling of job scheduling requests according to the target hybrid deployment mode.
- the calculation formula for the recommendation score of each hybrid deployment mode is:
- the simulation module 11 simulates the mixed deployment of multiple load types to the server based on the maximum resource value, it also includes: when a new load type appears, confirming the load that is closest to the new load type through similarity analysis Type to confirm the maximum resource value required by the new load type.
- the confirmation module 13 confirms the operation of satisfying the required resources of the job scheduling request, it further includes determining whether there is a server that meets the required resources of the job scheduling request; if so, the screening module 14 and the scheduling module 15 perform subsequent operations; If not, keep the job scheduling request and continue to wait until there is a server that meets the required resources.
- the confirmation module 13 confirms the operation of satisfying the required resources of the job scheduling request, it further includes initializing the priority of the job scheduling request; judging whether the priority is zero; if the priority is zero, the system scheduling operation is directly executed; If the priority is not zero, the screening module 14 and the scheduling module 15 perform subsequent operations; the screening module 14 screens out a list of servers that meet the required resources, and confirms from the server list that the target hybrid deployment mode with the highest score is recommended.
- the operation can be : Determine whether there is a server that meets the required resources; if it exists, filter out the server that meets the required resources and get the server list; if it does not exist, reduce the priority by one, and execute again to determine whether the priority is zero and subsequent Operation; after obtaining the server list, it also includes: determining whether there is a mixed deployment mode that meets the job scheduling request and multiple load types in the server list; if it exists, it will be in the hybrid deployment mode that meets the job scheduling request and multiple load types The hybrid deployment mode with the highest score is recommended as the target hybrid deployment mode; if it does not exist, the priority is reduced by one, and the judgment whether the priority is zero and the subsequent operations are executed again.
- FIG. 6 is a schematic structural diagram of a server according to an embodiment of the present invention.
- the server 60 includes a processor 61 and a memory 62 coupled to the processor 61.
- the memory 62 stores program instructions for implementing the cloud computing cluster hybrid job scheduling method described in any of the above embodiments.
- the processor 61 is configured to execute program instructions stored in the memory 62 to schedule job scheduling requests.
- the processor 61 may also be referred to as a CPU (Central Processing Unit, central processing unit).
- the processor 61 may be an integrated circuit chip with signal processing capability.
- the processor 61 may also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component .
- the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
- FIG. 7 is a schematic structural diagram of a storage device according to an embodiment of the present invention.
- the storage device in the embodiment of the present invention stores a program file 71 that can implement all the above methods.
- the program file 71 can be stored in the above storage device in the form of a software product, and includes several instructions to enable a computer device (which can It is a personal computer, a server, or a network device, etc.) or a processor (processor) that executes all or part of the steps of the methods described in the various embodiments of the present application.
- the aforementioned storage devices include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disks or optical disks and other media that can store program codes. , Or computer, server, mobile phone, tablet and other server equipment.
- the disclosed system, device, and method may be implemented in other ways.
- the device embodiments described above are merely illustrative, for example, the division of units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or integrated. To another system, or some features can be ignored, or not implemented.
- the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
- the functional units in the various embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
- the above-mentioned integrated unit can be implemented in the form of hardware or software functional unit. The above are only implementations of this application, and do not limit the scope of this application. Any equivalent structure or equivalent process transformation made using the content of the description and drawings of this application, or directly or indirectly applied to other related technical fields, The same reasoning is included in the scope of patent protection of this application.
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
- Multi Processors (AREA)
Abstract
一种云计算集群混部作业调度方法、装置、服务器及存储装置,其中方法包括:分析预先获取的多种负载类型的历史运行日志数据,确认每种负载类型运行时所需的最大资源值(S1);基于最大资源值模拟将多种负载类型混合部署至服务器,并获取每种混合部署模式下硬件事件的发生频率(S2);基于硬件事件发生频率计算每种混合部署模式的推荐评分(S3);当接收到作业调度请求时,确认满足作业调度请求的所需资源(S4);筛选出符合所需资源的服务器列表,并从服务器列表中确认推荐评分最高的目标混合部署模式(S5);根据目标混合部署模式对作业调度请求进行***调度(S6)。该方案通过在资源层面和硬件事件层面对作业调度请求进行双层混合部署,以提升云服务器整体资源利用率。
Description
本申请涉及云计算技术领域,特别是涉及一种云计算集群混部作业调度方法、装置、服务器及存储装置。
云计算服务因其高扩展性、按需服务以及极其廉价的特性而被广泛应用于数据中心服务平台,越来越多的企业和个人选择利用云计算平台运行应用程序。随之云集群中的服务类型越来越多样化,对作业调度的要求也越来越高。为了提升云计算集群整体的资源利用率,云数据中心相关人员开始逐步重视对物理主机的混合利用。
目前,提升云计算集群资源利用率主要从两个方面进行。一是负载特征分析与资源预测,该方法通过对负载的历史运行日志进行分析,采用服务器学习模型等进行预测下一时刻的资源消耗情况,从而进行合理的资源分配。该方式在一定程度上缩短了资源请求与实际资源消耗的差距,减少服务器资源超售的情况。二是集群中的混部策略的部署,主要是将实时型业务和计算型业务混部在同一台物理机中。但是,目前大部分的作业调度方案考虑的是单个层面,即资源层的资源需求或者作业本身的调度优先级,采用的方法有神经网络、蚁群算法等经典启发式算法,排队论等,或者是基于经典算法模型的改进策略,其忽略了混合部署时,不同类型的负载之间互相干扰而导致服务器集群性能下降的问题。
发明内容
本申请提供一种云计算集群混部作业调度方法、装置、服务器及存储装置,以解决现有的作业调度方案因负载之间互相干扰而导致服务器集群性能下降的问题。
为解决上述技术问题,本申请采用的一个技术方案是:提供一种云计算集群混部作业调度方法,其包括:
分析预先获取的多种负载类型的历史运行日志数据,确认每种负载类型运 行时所需的最大资源值;
基于最大资源值模拟将多种负载类型混合部署至服务器,并获取每种混合部署模式下硬件事件的发生频率;
基于硬件事件发生频率计算每种混合部署模式的推荐评分;
当接收到作业调度请求时,确认满足作业调度请求的所需资源;
筛选出符合所需资源的服务器列表,并从服务器列表中确认推荐评分最高的目标混合部署模式;
根据目标混合部署模式对作业调度请求进行***调度。
作为本发明的进一步改进,所述每种混合部署模式的推荐评分的计算公式为:
其中,T
w1w2为w1、w2两种负载类型混合部署时的推荐评分,S
HEM={IPC,BPM,LLCM……},w1、w2分别代表不同类型的负载,rate
i表示w1、w2混合部署运行时w1的硬件事件发生频率与w1单独部署运行时的硬件事件发生频率的比值,α
i为预先设定的硬件事件的权重。
作为本发明的进一步改进,基于所述最大资源值模拟将所述多种负载类型混合部署至服务器的步骤之前,还包括:
当出现新的负载类型时,通过相似性分析确认与新的负载类型最接近的负载类型,以确认新的负载类型所需的最大资源值。
作为本发明的进一步改进,确认满足作业调度请求的所需资源的步骤之后,还包括:
判断是否有服务器满足作业调度请求的所需资源;
若有,则执行筛选出符合所需资源的服务器列表以及后续步骤。
若无,则保持作业调度请求,并继续等待,直至有满足所需资源的服务器。
作为本发明的进一步改进,确认满足作业调度请求的所需资源的步骤之后,还包括:
初始化作业调度请求的优先级;
判断优先级是否为零;
若优先级为零,则直接执行***调度操作;
若优先级不为零,则执行筛选出符合所需资源的服务器列表以及后续步骤。
作为本发明的进一步改进,筛选出符合所需资源的服务器列表的步骤,包括:
判断是否存在符合所需资源的服务器;
若存在,则筛选出符合所需资源的服务器,得到服务器列表;
若不存在,则将优先级减一,并再次执行判断优先级是否为零以及后续步骤。
作为本发明的进一步改进,从服务器列表中确认推荐评分最高的目标混合部署模式的步骤,包括:
判断服务器列表中是否存在符合作业调度请求与多种负载类型的混合部署模式;
若存在,则将符合作业调度请求与多种负载类型的混合部署模式中推荐评分最高的混合部署模式作为目标混合部署模式;
若不存在,则将优先级减一,并再次执行判断优先级是否为零以及后续步骤。
为了解决上述问题,本发明还提供了一种云计算集群混合部署作业调度装置,其包括:
分析模块,用于分析预先获取的多种负载类型的历史运行日志数据,确认每种负载类型运行时所需的最大资源值;
模拟模块,用于基于最大资源值模拟将多种负载类型混合部署至服务器,并获取每种混合部署模式下硬件事件的发生频率;
计算模块,用于基于硬件事件发生频率计算每种混合部署模式的推荐评分;
确认模块,用于当接收到作业调度请求时,确认满足作业调度请求的所需 资源;
筛选模块,用于筛选出符合所需资源的服务器列表,并从服务器列表中确认推荐评分最高的目标混合部署模式;
调度模块,用于根据目标混合部署模式对作业调度请求进行***调度。
为了解决上述问题,本发明还提供了一种服务器,服务器包括处理器、与处理器耦接的存储器,其中,
存储器存储有用于实现上述中任一项的云计算集群混部作业调度方法的程序指令;
处理器用于执行存储器存储的程序指令以对作业调度请求进行调度。
为了解决上述问题,本发明还提供了一种存储装置,存储有能够实现上述中任一项的云计算集群混部作业调度方法的程序文件。
本申请的有益效果是:本发明通过分析不同负载类型混合部署时,各硬件事件发生特征,从而得到不同负载类型混合部署形成的混合部署模式的推荐评分,在接接收到作业调度请求时,确认作业调度请求所需资源,再从满足所需资源的服务器中筛选出作业调度请求与其他负载混合部署时推荐评分最高的混合部署模式,并根据此混合部署模式对作业调度请求进行***调度,其综合考虑了资源和不同负载之间的干扰情况两个方面,避免了因负载之间互相干扰而导致服务器性能下降的问题,提升云服务器整体资源利用率。
图1是本发明第一实施例的云计算集群混部作业调度方法的流程示意图;
图2是本发明第二实施例的云计算集群混部作业调度方法的流程示意图;
图3是本发明第三实施例的云计算集群混部作业调度方法的流程示意图;
图4是本发明第四实施例的云计算集群混部作业调度方法的流程示意图;
图5是本发明实施例的云计算集群混部作业调度装置的结构示意图;
图6是本发明实施例的服务器的结构示意图;
图7是本发明实施例的存储装置的结构示意图。
为了使本申请的目的、技术方案及优点更加清楚明白,以下结合附图及实施例,对本申请进行进一步详细说明。应当理解,此处所描述的具体实施例仅用以解释本申请,并不用于限定本申请。
图1是本发明第一实施例的云计算集群混部作业调度方法的流程示意图。需注意的是,若有实质上相同的结果,本发明的方法并不以图1所示的流程顺序为限。如图1所示,该方法包括步骤:
步骤S1,分析预先获取的多种负载类型的历史运行日志数据,确认每种负载类型运行时所需的最大资源值。
需要说明的是,多种负载类型的历史运行日志数据需要预先进行采集,并且,历史运行日志数据包括CPU利用率、内存利用率、磁盘占用大小、网络I/O等属性值,采样频率可以设置为每隔60秒采样一次。
具体地,通过分析每种负载类型的历史运行日志数据,即可得到每种负载类型运行时所需的最大资源值,该资源值包括负载运行时所需的CPU、内存等资源。
步骤S2,基于最大资源值模拟将多种负载类型混合部署至服务器,并获取每种混合部署模式下硬件事件的发生频率。
需要说明的是,硬件事件包括Instructions per Cycle、Branch prediction misses、Context switches、Cache misses、L1 data cache load misses、Last level cache misses、dTLB load misses、iTLB load misses等。
具体地,在获取每种负载类型运行时所需的最大资源值之后,同样获取服务器的资源情况,根据每种负载类型运行时所需的最大资源值模拟将每种负载类型混合部署至服务器,再通过分析混合部署模式下硬件事件的特征,从而获知硬件事件的发生频率。
步骤S3,基于硬件事件发生频率计算每种混合部署模式的推荐评分。
需要说明的是,每种混合部署模式的推荐评分的计算公式为:
其中,T
w1w2为w1、w2两种负载类型混合部署时的推荐评分,S
HEM={IPC,BPM,LLCM……}(即硬件事件),w1、w2分别代表不同类型的负载,rate
i表示w1、w2混合部署运行时w1的硬件事件发生频率与w1单独部署运行时的硬件事件发生频率的比值,w1单独部署运行时的硬件事件发生频率可提前获取,α
i为预先设定的硬件事件的权重。
步骤S4,当接收到作业调度请求时,确认满足作业调度请求的所需资源。
步骤S5,筛选出符合所需资源的服务器列表,并从服务器列表中确认推荐评分最高的目标混合部署模式。
具体地,获取各个服务器的当前剩余的资源,再根据所需资源筛选出符合要求的服务器,组合成一个服务器列表,再根据作业调度请求和已经运行的负载确认服务器列表中推荐评分最高的目标混合部署模式。
步骤S6,根据目标混合部署模式对作业调度请求进行***调度。
本实施例通过分析不同负载类型混合部署时,各硬件事件发生特征,从而得到不同负载类型混合部署形成的混合部署模式的推荐评分,在接接收到作业调度请求时,确认作业调度请求所需资源,再从满足所需资源的服务器中筛选出作业调度请求与其他负载混合部署时推荐评分最高的混合部署模式,并根据此混合部署模式对作业调度请求进行***调度,其综合考虑了资源和不同负载之间的干扰情况两个方面,避免了因负载之间互相干扰而导致服务器性能下降的问题,提升云服务器整体资源利用率。
将本发明的云计算集群混部作业调度方法应用于云计算集群混部作业调度装置的过程中,还存在新的类型的负载进入的情况,因此,上述实施例的基础上,其他实施例中,如图2所示,步骤S2之前,还包括:
步骤S10,当出现新的负载类型时,通过相似性分析确认与新的负载类型最接近的负载类型,以确认新的负载类型所需的最大资源值。
在本实施例中,当出现新的负载类型时,对该新出现的负载类型与已有的 负载类型进行相似性分析,从而确认与该新的负载类型最接近的负载类型,进而对新的负载类型的所需的最大资源值作出预测,再根据预测的最大资源值将新的负载类型和已有的负载类型模拟混合部署的场景,从而提高该本发明实施例提供的云计算集群混部作业调度方法的泛用性。
将本发明的云计算集群混部作业调度方法应用于云计算集群混部作业调度装置的过程中,还需要判断是否有服务器满足作业调度请求所需资源,因此,上述实施例的基础上,其他实施例中,如图3所示,步骤S4之后,还包括:
步骤S20,判断是否有服务器满足作业调度请求的所需资源。若有,则执行步骤S5~步骤S6;若无,则执行步骤21。
步骤S21,保持作业调度请求,并继续等待,直至有满足所需资源的服务器。
本实施例中,在确认作业调度请求的所需资源之后,获取所有的服务器的剩余资源,并判断其中是否有满足作业调度请求的所需资源,若无,则保持该作业调度请求,并继续等待,并循环判断是否有满足作业调度请求所需资源的服务器,直至有满足所需资源的服务器为止。
将本发明的云计算集群混部作业调度方法应用于云计算集群混部作业调度装置的过程中,还需要对作业调度请求设置优先级,因此,上述实施例的基础上,其他实施例中,如图4所示,步骤S4之后,还包括:
步骤S30,初始化作业调度请求的优先级。
具体地,初始化作业调度请求的优先级P=M,其中,M为预先设定的优先级阈值,即允许的最大重复调度次数。
步骤S31,判断优先级是否为零。若优先级为零,执行步骤S32;若优先级不为零,则执行步骤S5~步骤S6。
步骤S32,直接执行***调度操作。
本实施例中,根据作业调度请求的优先级确认是否需要优先对作业调度请求进行处理。
进一步的,在上述实施例的基础上,步骤S5包括:
步骤S33,判断是否存在符合所需资源的服务器。若存在,则执行步骤S34~步骤S35;若不存在,则将优先级减一,并再次执行步骤S31及后续步骤。
步骤S34,筛选出符合所需资源的服务器,得到服务器列表。
步骤S35,判断服务器列表中是否存在符合作业调度请求与多种负载类型的混合部署模式。若存在,则执行步骤S36;若不存在,则将优先级减一,并再次执行步骤S31及后续步骤。
步骤S36,将符合作业调度请求与多种负载类型的混合部署模式中推荐评分最高的混合部署模式作为目标混合部署模式。
本实施例中,为了避免作业调度请求等待的时间过长,每判定一次不存在符合所需资源的服务器或服务器列表中不存在符合作业调度请求与多种负载类型的混合部署模式时,将作业调度请求的优先级减一,直至优先级降为零时,对该作业调度请求进行***调度。
图5展示了本发明云计算集群混合部署作业调度装置的结构示意图。如图5所示,该云计算集群混合部署作业调度装置1包括分析模块10、模拟模块11、计算模块12、确认模块13、筛选模块14和调度模块15。
其中,分析模块10,用于分析预先获取的多种负载类型的历史运行日志数据,确认每种负载类型运行时所需的最大资源值;模拟模块11,用于基于最大资源值模拟将多种负载类型混合部署至服务器,并获取每种混合部署模式下硬件事件的发生频率;计算模块12,用于基于硬件事件发生频率计算每种混合部署模式的推荐评分;确认模块13,用于当接收到作业调度请求时,确认满足作业调度请求的所需资源;筛选模块14,用于筛选出符合所需资源的服务器列表,并从服务器列表中确认推荐评分最高的目标混合部署模式;调度模块15,用于根据目标混合部署模式对作业调度请求进行***调度。
可选地,每种混合部署模式的推荐评分的计算公式为:
其中,T
w1w2为w1、w2两种负载类型混合部署时的推荐评分,S
HEM={IPC,BPM,LLCM……},w1、w2分别代表不同类型的负载,rate
i表示w1、w2混合部署运行时w1的硬件事件发生频率与w1单独部署运行时的硬件事件发生频率的比值,α
i为预先设定的硬件事件的权重。
可选地,模拟模块11基于最大资源值模拟将多种负载类型混合部署至服务器的操作之前,还包括:当出现新的负载类型时,通过相似性分析确认与新的负载类型最接近的负载类型,以确认新的负载类型所需的最大资源值。
可选地,确认模块13确认满足作业调度请求的所需资源的操作之后,还包括判断是否有服务器满足作业调度请求的所需资源;若有,则筛选模块14和调度模块15执行后续操作;若无,则保持作业调度请求,并继续等待,直至有满足所需资源的服务器。
可选地,确认模块13确认满足作业调度请求的所需资源的操作之后,还包括初始化作业调度请求的优先级;判断优先级是否为零;若优先级为零,则直接执行***调度操作;若优先级不为零,则筛选模块14和调度模块15执行后续操作;筛选模块14筛选出符合所需资源的服务器列表,并从服务器列表中确认推荐评分最高的目标混合部署模式的操作可以为:判断是否存在符合所需资源的服务器;若存在,则筛选出符合所需资源的服务器,得到服务器列表;若不存在,则将优先级减一,并再次执行判断优先级是否为零以及后续操作;在得到服务器列表之后,还包括:判断服务器列表中是否存在符合作业调度请求与多种负载类型的混合部署模式;若存在,则将符合作业调度请求与多种负载类型的混合部署模式中推荐评分最高的混合部署模式作为目标混合部署模式;若不存在,则将优先级减一,并再次执行判断优先级是否为零以及后续操作。
请参阅图6,图6为本发明实施例的服务器的结构示意图。如图6所示,该服务器60包括处理器61及和处理器61耦接的存储器62。
存储器62存储有用于实现上述任一实施例所述的云计算集群混部作业调 度方法的程序指令。
处理器61用于执行存储器62存储的程序指令以对作业调度请求进行调度。
其中,处理器61还可以称为CPU(Central Processing Unit,中央处理单元)。处理器61可能是一种集成电路芯片,具有信号的处理能力。处理器61还可以是通用处理器、数字信号处理器(DSP)、专用集成电路(ASIC)、现成可编程门阵列(FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
参阅图7,图7为本发明实施例的存储装置的结构示意图。本发明实施例的存储装置存储有能够实现上述所有方法的程序文件71,其中,该程序文件71可以以软件产品的形式存储在上述存储装置中,包括若干指令用以使得一台计算机设备(可以是个人计算机,服务器,或者网络设备等)或处理器(processor)执行本申请各个实施方式所述方法的全部或部分步骤。而前述的存储装置包括:U盘、移动硬盘、只读存储器(ROM,Read-Only Memory)、随机存取存储器(RAM,Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质,或者是计算机、服务器、手机、平板等服务器设备。
在本申请所提供的几个实施例中,应该理解到,所揭露的***,装置和方法,可以通过其它的方式实现。例如,以上所描述的装置实施例仅仅是示意性的,例如,单元的划分,仅仅为一种逻辑功能划分,实际实现时可以有另外的划分方式,例如多个单元或组件可以结合或者可以集成到另一个***,或一些特征可以忽略,或不执行。另一点,所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口,装置或单元的间接耦合或通信连接,可以是电性,机械或其它的形式。
另外,在本发明各个实施例中的各功能单元可以集成在一个处理单元中,也可以是各个单元单独物理存在,也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现,也可以采用软件功能单元的 形式实现。以上仅为本申请的实施方式,并非因此限制本申请的专利范围,凡是利用本申请说明书及附图内容所作的等效结构或等效流程变换,或直接或间接运用在其他相关的技术领域,均同理包括在本申请的专利保护范围内。
以上所述仅是本发明的优选实施方式,应当指出,对于本技术领域的普通技术人员来说,在不脱离本发明原理的前提下,还可以做出若干改进和润饰,这些改进和润饰也应视为本发明的保护范围。
Claims (10)
- 一种云计算集群混部作业调度方法,其特征在于,其包括:分析预先获取的多种负载类型的历史运行日志数据,确认每种负载类型运行时所需的最大资源值;基于所述最大资源值模拟将所述多种负载类型混合部署至服务器,并获取每种混合部署模式下硬件事件的发生频率;基于所述硬件事件发生频率计算每种混合部署模式的推荐评分;当接收到作业调度请求时,确认满足所述作业调度请求的所需资源;筛选出符合所述所需资源的服务器列表,并从所述服务器列表中确认推荐评分最高的目标混合部署模式;根据所述目标混合部署模式对所述作业调度请求进行***调度。
- 根据权利要求1所述的云计算集群混部作业调度方法,其特征在于,所述基于所述最大资源值模拟将所述多种负载类型混合部署至服务器的步骤之前,还包括:当出现新的负载类型时,通过相似性分析确认与所述新的负载类型最接近的负载类型,以确认所述新的负载类型所需的最大资源值。
- 根据权利要求1所述的云计算集群混部作业调度方法,其特征在于,所述确认满足所述作业调度请求的所需资源的步骤之后,还包括:判断是否有服务器满足所述作业调度请求的所需资源;若有,则执行筛选出符合所述所需资源的服务器列表以及后续步骤;若无,则保持所述作业调度请求,并继续等待,直至有满足所需资源的服务器。
- 根据权利要求1所述的云计算集群混部作业调度方法,其特征在于,所述确认满足所述作业调度请求的所需资源的步骤之后,还包括:初始化所述作业调度请求的优先级;判断所述优先级是否为零;若所述优先级为零,则直接执行***调度操作;若所述优先级不为零,则执行筛选出符合所述所需资源的服务器列表以及后续步骤。
- 根据权利要求5所述的云计算集群混部作业调度方法,其特征在于,所述筛选出符合所述所需资源的服务器列表的步骤,包括:判断是否存在符合所述所需资源的服务器;若存在,则筛选出符合所述所需资源的服务器,得到服务器列表;若不存在,则将所述优先级减一,并再次执行判断所述优先级是否为零以及后续步骤。
- 根据权利要求6所述的云计算集群混部作业调度方法,其特征在于,所述从所述服务器列表中确认推荐评分最高的目标混合部署模式的步骤,包括:判断所述服务器列表中是否存在符合所述作业调度请求与所述多种负载类型的混合部署模式;若存在,则将符合所述作业调度请求与所述多种负载类型的混合部署模式中推荐评分最高的混合部署模式作为目标混合部署模式;若不存在,则将所述优先级减一,并再次执行判断所述优先级是否为零以及后续步骤。
- 一种云计算集群混合部署作业调度装置,其特征在于,其包括:分析模块,用于分析预先获取的多种负载类型的历史运行日志数据,确认每种负载类型运行时所需的最大资源值;模拟模块,用于基于所述最大资源值模拟将所述多种负载类型混合部署至服务器,并获取每种混合部署模式下硬件事件的发生频率;计算模块,用于基于所述硬件事件发生频率计算每种混合部署模式的推荐评分;确认模块,用于当接收到作业调度请求时,确认满足所述作业调度请求的所需资源;筛选模块,用于筛选出符合所述所需资源的服务器列表,并从所述服务器列表中确认推荐评分最高的目标混合部署模式;调度模块,用于根据所述目标混合部署模式对所述作业调度请求进行***调度。
- 一种服务器,其特征在于,所述服务器包括处理器、与所述处理器连接的存储器,其中,所述存储器存储有用于实现如权利要求1-7中任一项所述的云计算集群混部作业调度方法的程序指令;所述处理器用于执行所述存储器存储的所述程序指令以对作业调度请求进行调度。
- 一种存储装置,其特征在于,存储有能够实现如权利要求1-7中任一项所述的云计算集群混部作业调度方法的程序文件。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911065641.4 | 2019-11-04 | ||
CN201911065641.4A CN110908795B (zh) | 2019-11-04 | 2019-11-04 | 云计算集群混部作业调度方法、装置、服务器及存储装置 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021088207A1 true WO2021088207A1 (zh) | 2021-05-14 |
Family
ID=69814818
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2019/125432 WO2021088207A1 (zh) | 2019-11-04 | 2019-12-14 | 云计算集群混部作业调度方法、装置、服务器及存储装置 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN110908795B (zh) |
WO (1) | WO2021088207A1 (zh) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113434258A (zh) * | 2021-07-07 | 2021-09-24 | 京东科技控股股份有限公司 | 模型部署方法、装置、设备和计算机存储介质 |
CN114338683A (zh) * | 2021-12-27 | 2022-04-12 | 中国电信股份有限公司 | 调度请求处理方法、装置、存储介质及电子设备 |
CN116932228A (zh) * | 2023-09-14 | 2023-10-24 | 湖南希赛网络科技有限公司 | 一种基于志愿者计算的边缘ai任务调度和资源管理*** |
CN117893334A (zh) * | 2024-03-15 | 2024-04-16 | 国任财产保险股份有限公司 | 一种基于大数据的保险任务分配方法及*** |
Families Citing this family (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111858070B (zh) * | 2020-08-05 | 2023-12-01 | 中国工商银行股份有限公司 | 计算资源配置方法、装置、设备以及存储介质 |
CN112162791B (zh) * | 2020-09-25 | 2022-12-30 | 天津大学 | 一种分布式深度学习任务混部后的性能预测方法 |
CN112559282A (zh) * | 2020-12-08 | 2021-03-26 | 海光信息技术股份有限公司 | 功耗监测***、相关方法、装置、处理器及介质 |
CN112506619B (zh) * | 2020-12-18 | 2023-08-04 | 北京百度网讯科技有限公司 | 作业处理方法、装置、电子设备和存储介质 |
CN112668011B (zh) * | 2020-12-29 | 2022-03-01 | 广东电网有限责任公司电力调度控制中心 | 一种云容器集群的部署方法及*** |
CN113075994B (zh) * | 2021-04-26 | 2023-05-30 | 华南理工大学 | 面向x86和arm混合云计算的节能调度***、方法和存储介质 |
CN113407192B (zh) * | 2021-06-30 | 2023-07-04 | 竹间智能科技(上海)有限公司 | 模型部署方法和设备 |
CN113672382B (zh) * | 2021-07-20 | 2024-03-26 | 北京达佳互联信息技术有限公司 | 一种业务资源分配方法、装置、电子设备和存储介质 |
CN114143322B (zh) * | 2021-11-25 | 2024-02-27 | ***数智科技有限公司 | 一种混合云场景中计算资源的配置行动评估方法 |
CN114020584B (zh) * | 2022-01-05 | 2022-05-03 | 北京并行科技股份有限公司 | 一种作业分流方法、装置及计算设备 |
CN116382892B (zh) * | 2023-02-08 | 2023-10-27 | 深圳市融聚汇信息科技有限公司 | 一种基于多云融合以及云服务的负载均衡方法及装置 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106095550A (zh) * | 2016-06-07 | 2016-11-09 | 百度在线网络技术(北京)有限公司 | 一种计算任务调度方法和装置 |
US20170083381A1 (en) * | 2015-09-21 | 2017-03-23 | Alibaba Group Holding Limited | System and method for processing task resources |
CN106878389A (zh) * | 2017-01-04 | 2017-06-20 | 北京百度网讯科技有限公司 | 用于在云***中进行资源调度的方法和装置 |
CN108023958A (zh) * | 2017-12-08 | 2018-05-11 | 中国电子科技集团公司第二十八研究所 | 一种基于云平台资源监视的资源调度*** |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107656806A (zh) * | 2016-07-25 | 2018-02-02 | 华为技术有限公司 | 一种资源分配方法和资源分配装置 |
CN106407137A (zh) * | 2016-09-18 | 2017-02-15 | 中国科学技术大学苏州研究院 | 基于邻域模型的协同过滤推荐算法的硬件加速器和方法 |
CN107404523A (zh) * | 2017-07-21 | 2017-11-28 | 中国石油大学(华东) | 云平台自适应资源调度***和方法 |
US10346279B2 (en) * | 2017-07-25 | 2019-07-09 | Citrix Systems, Inc. | Method for optimized canary deployments for improved customer experience |
CN108234612B (zh) * | 2017-12-23 | 2021-07-06 | 北京卫星信息工程研究所 | 多终端模式的高分资源动态分配与按需服务发布*** |
CN109308222B (zh) * | 2018-09-12 | 2021-09-03 | 上海丕休智能科技有限公司 | 基于统一建模环境的去中心化软硬件资源云平台搭建方法 |
-
2019
- 2019-11-04 CN CN201911065641.4A patent/CN110908795B/zh active Active
- 2019-12-14 WO PCT/CN2019/125432 patent/WO2021088207A1/zh active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170083381A1 (en) * | 2015-09-21 | 2017-03-23 | Alibaba Group Holding Limited | System and method for processing task resources |
CN106095550A (zh) * | 2016-06-07 | 2016-11-09 | 百度在线网络技术(北京)有限公司 | 一种计算任务调度方法和装置 |
CN106878389A (zh) * | 2017-01-04 | 2017-06-20 | 北京百度网讯科技有限公司 | 用于在云***中进行资源调度的方法和装置 |
CN108023958A (zh) * | 2017-12-08 | 2018-05-11 | 中国电子科技集团公司第二十八研究所 | 一种基于云平台资源监视的资源调度*** |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113434258A (zh) * | 2021-07-07 | 2021-09-24 | 京东科技控股股份有限公司 | 模型部署方法、装置、设备和计算机存储介质 |
CN113434258B (zh) * | 2021-07-07 | 2024-04-12 | 京东科技控股股份有限公司 | 模型部署方法、装置、设备和计算机存储介质 |
CN114338683A (zh) * | 2021-12-27 | 2022-04-12 | 中国电信股份有限公司 | 调度请求处理方法、装置、存储介质及电子设备 |
CN114338683B (zh) * | 2021-12-27 | 2024-04-30 | 中国电信股份有限公司 | 调度请求处理方法、装置、存储介质及电子设备 |
CN116932228A (zh) * | 2023-09-14 | 2023-10-24 | 湖南希赛网络科技有限公司 | 一种基于志愿者计算的边缘ai任务调度和资源管理*** |
CN116932228B (zh) * | 2023-09-14 | 2023-12-12 | 湖南希赛网络科技有限公司 | 一种基于志愿者计算的边缘ai任务调度和资源管理*** |
CN117893334A (zh) * | 2024-03-15 | 2024-04-16 | 国任财产保险股份有限公司 | 一种基于大数据的保险任务分配方法及*** |
Also Published As
Publication number | Publication date |
---|---|
CN110908795A (zh) | 2020-03-24 |
CN110908795B (zh) | 2022-08-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2021088207A1 (zh) | 云计算集群混部作业调度方法、装置、服务器及存储装置 | |
CN108829494B (zh) | 基于负载预测的容器云平台智能资源优化方法 | |
US10565083B2 (en) | Simulating hosted application performance | |
WO2018103595A1 (zh) | 一种授权策略推荐方法及装置、服务器、存储介质 | |
Liu et al. | Resource preprocessing and optimal task scheduling in cloud computing environments | |
CN1610311A (zh) | 通过把推理用于it***进行自动建模的方法和设备 | |
CN111444019B (zh) | 云端协同的深度学习模型分布式训练方法及*** | |
CN116614385A (zh) | 基于数字孪生的业务调度路径规划方法、装置和设备 | |
CN111860853A (zh) | 在线预测***、设备、方法及电子设备 | |
Li et al. | An intelligent collaborative inference approach of service partitioning and task offloading for deep learning based service in mobile edge computing networks | |
CN105760227A (zh) | 云环境下资源调度方法及*** | |
CN108037998A (zh) | 一种面向Spark Streaming平台的数据接收通道动态分配方法 | |
CN112633823A (zh) | 投产进度实时跟踪方法、存储介质及计算机设备 | |
CN114785693B (zh) | 基于分层强化学习的虚拟网络功能迁移方法及装置 | |
Cao et al. | Online cost-rejection rate scheduling for resource requests in hybrid clouds | |
CN108052378B (zh) | 一种面向复杂仿真工作流应用的利润敏感的任务调度方法 | |
CN112866358B (zh) | 一种物联网服务重调度的方法、***及装置 | |
CN115713216A (zh) | 一种机器人调度方法及相关设备 | |
CN114298813A (zh) | 一种基于资源消耗的企业用户管理方法及*** | |
CN114596054A (zh) | 针对数字化办公的业务信息管理方法及*** | |
Ruby et al. | RenderSelect: a cloud broker framework for cloud renderfarm services | |
Abbes et al. | Toward a framework for improving the execution of the big data applications | |
CN112073239B (zh) | 一种云计算环境分布式应用性能预测方法 | |
CN116155835B (zh) | 一种基于排队论的云资源服务质量评估方法及*** | |
Rong et al. | A large-scale device collaboration resource selection method with multi-QoS constraint supported |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19951898 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19951898 Country of ref document: EP Kind code of ref document: A1 |