CN116755902A

CN116755902A - Data communication method and device, processing system, electronic equipment and storage medium

Info

Publication number: CN116755902A
Application number: CN202310685760.XA
Authority: CN
Inventors: 钟渊; 李静
Original assignee: Haiguang Information Technology Co Ltd
Current assignee: Haiguang Information Technology Co Ltd
Priority date: 2023-06-09
Filing date: 2023-06-09
Publication date: 2023-09-15

Abstract

A data communication method and device, a processing system, an electronic device and a storage medium. The data communication method comprises the following steps: determining a storage address of target data for data communication between a first process and a second process, wherein the first process and the second process belong to the same computing node, the target data is stored in a first physical resource, and a mapping relation is established between the first physical resource and a first virtual address space in the first process and between the first physical resource and a second virtual address space in the second process; and sending the storage address belonging to the first virtual address space to the second process so that the second process acquires the target data based on the storage address. For example, the data communication method can make data copies in a user mode without switching to a kernel mode; and the intra-node communication can be realized only by one copy, the intra-node communication time is shortened, the intra-node communication bandwidth is increased, the system operation speed and operation efficiency are improved, and the system performance is optimized.

Description

Data communication method and device, processing system, electronic equipment and storage medium

Technical Field

Embodiments of the present disclosure relate to a data communication method, a data communication apparatus, a processing system, an electronic device, a non-transitory computer readable storage medium.

Background

High performance computing (highperformance computing) refers to computing systems and environments that typically use many processors (as part of a single machine) or several computers organized in a cluster (operating as a single computing resource). Application programs running on a high-performance cluster generally use parallel algorithms to divide a large common problem into a plurality of small sub-problems according to a certain rule, and calculate the sub-problems on different nodes in the cluster, and the processing results of the small problems can be combined into the final result of the original problem after processing. Since the computation of these small problems can generally be done in parallel, the processing time of the problem can be shortened.

Disclosure of Invention

At least one embodiment of the present disclosure provides a data communication method for a processing system including a plurality of computing nodes, the data communication method including: determining a storage address of target data for data communication between a first process and a second process, wherein the first process and the second process belong to the same computing node, the target data is stored in a first physical resource, and a mapping relation is established between the first physical resource and a first virtual address space in the first process and between the first physical resource and a second virtual address space in the second process; and sending the storage address belonging to the first virtual address space to the second process so that the second process acquires the target data based on the storage address.

For example, in a data communication method provided in at least one embodiment of the present disclosure, before determining a storage address of target data for data communication between a first process and a second process, the data communication method further includes: applying for the first physical resource to be used for data communication between the first process and the second process, and establishing a mapping relationship between the first physical resource and the first virtual address space, wherein the first physical resource is at least part of physical resources in physical resources provided by the same computing node; and sending the related information of the first physical resource to the second process so that the second process establishes a mapping relation between the first physical resource and the second virtual address space.

For example, in a data communication method provided in at least one embodiment of the present disclosure, applying the first physical resource for data communication between the first process and the second process, and establishing a mapping relationship between the first physical resource and the first virtual address space includes: setting the first process to enter a kernel mode, and applying for the first physical resource; applying for an identification number corresponding to data communication between the first process and the second process, wherein the identification number is used for indicating the first physical resource; establishing a mapping relation between the first physical resource and a first virtual address space in the first process; and setting the first process to exit the kernel mode.

For example, in the data communication method provided in at least one embodiment of the present disclosure, the information about the first physical resource includes the identification number.

For example, the data communication method provided in at least one embodiment of the present disclosure further includes: applying a second physical resource for transmission of the target data according to the length of the target data in response to the length of the target data being greater than the number of resources of the first physical resource, wherein the number of resources of the second physical resource is determined by a difference between the length of the target data and the number of resources of the first physical resource; establishing a mapping relation between the second physical resource and a third virtual address space in the first process; and sending the related information of the second physical resource to the second process so that the second process establishes a mapping relation between the second physical resource and a fourth virtual address space in the second process.

For example, in a data communication method provided in at least one embodiment of the present disclosure, the resource amount of the first physical resource is determined by a single maximum data traffic of data communication between the first process and the second process.

For example, in a data communication method provided in at least one embodiment of the present disclosure, the storage address includes a first virtual first address corresponding to the target data in the first virtual address space and a length of the target data, and sending the storage address belonging to the first virtual address space to the second process, so that the second process obtains the target data based on the storage address includes: determining an offset of the first virtual head address relative to a starting address of the first virtual address space; and sending the offset and the length of the target data to the second process, so that the second process acquires the target data based on the offset and the length of the target data.

For example, in a data communication method provided by at least one embodiment of the present disclosure, the processing system is a high performance computing system.

For example, in the data communication method provided in at least one embodiment of the present disclosure, a start address of the first virtual address space and a start address of the second virtual address space are different, and lengths of the first virtual address space and the second virtual address space are the same.

For example, in the data communication method provided in at least one embodiment of the present disclosure, in a process that the first process sends the storage address to the second process, the first process is in a user state.

At least one embodiment of the present disclosure provides a data communication method for a processing system including a plurality of computing nodes, the data communication method including: and responding to the received storage address from the first process, and acquiring target data sent to the second process by the first process based on the storage address, wherein the first process and the second process belong to the same computing node, the target data is stored in a first physical resource, and the first physical resource and a first virtual address space in the first process and a second virtual address space in the second process are respectively established with a mapping relation.

For example, in a data communication method provided in at least one embodiment of the present disclosure, the storage address includes an offset of a first virtual first address corresponding to the target data in the first virtual address space relative to a start address of the first virtual address space, and a length of the target data, and based on the storage address, acquiring the target data sent by the first process to the second process includes: determining a source address based on the offset and a start address of the second virtual address space; and copying the space with the length of the target data to a target address by taking the source address as a starting position so as to copy the target data to the target address.

For example, in the data communication method provided in at least one embodiment of the present disclosure, before acquiring the target data sent by the first process to the second process based on the storage address, the data communication method further includes: and receiving related information of the first physical resource from the first process, and establishing a mapping relation between the first physical resource and a second virtual address space in the second process.

For example, in a data communication method provided in at least one embodiment of the present disclosure, the related information of the first physical resource includes an identification number corresponding to communication between the first process and the second process, where the identification number is used to indicate the first physical resource, receive the related information of the first physical resource from the first process, and establish a mapping relationship between the first physical resource and a second virtual address space in the second process, and includes: receiving the identification number and setting the second process to enter a kernel mode; determining the first physical resource by the identification number; establishing a mapping relation between the first physical resource and a second virtual address space in the second process; and setting the second process to exit the kernel mode.

For example, in the data communication method provided in at least one embodiment of the present disclosure, in a process that the second process obtains the target data based on the storage address, the second process is in a user state.

At least one embodiment of the present disclosure provides a data communication method for a processing system including a plurality of computing nodes, the data communication method including: providing a first physical resource for data communication between a first process and a second process, wherein the first process and the second process belong to the same computing node, the first physical resource and a first virtual address space in the first process and a second virtual address space in the second process are respectively established with a mapping relation, and the first physical resource is used for caching data related to the operation of the first process and the second process; in response to a data communication between the first process and the second process, dividing a memory space required for the data communication from the first physical resource for the data communication.

For example, in a data communication method provided in at least one embodiment of the present disclosure, providing a first physical resource for data communication between a first process and a second process includes: the first process applies for the first physical resource to be used for data communication between the first process and the second process, and establishes a mapping relation between the first physical resource and a first virtual address space in the first process; the first process sends the related information of the first physical resource to the second process; and the second process establishes a mapping relation between the first physical resource and a second virtual address space in the second process according to the related information of the first physical resource.

For example, in a data communication method provided in at least one embodiment of the present disclosure, in response to data communication between the first process and the second process, dividing a memory space required for the data communication from the first physical resource for the data communication, includes: in response to a data communication between the first process and the second process, the first process determining a storage address for target data for the data communication between the first process and the second process; the first process sends the storage address belonging to the first virtual address space to the second process; the second process obtains the target data based on the memory address.

For example, in a data communication method provided in at least one embodiment of the present disclosure, during a data communication process between the first process and the second process, the first process and the second process are in a user state.

At least one embodiment of the present disclosure provides a data communication apparatus for a processing system including a plurality of computing nodes, the data communication apparatus comprising: a storage address determining unit configured to determine a storage address of target data for data communication between a first process and a second process, wherein the first process and the second process belong to the same computing node, the target data is stored in a first physical resource, and a mapping relationship is established between the first physical resource and a first virtual address space in the first process and a second virtual address space in the second process; and a transmitting unit configured to transmit the storage address belonging to the first virtual address space to the second process so that the second process acquires the target data based on the storage address.

At least one embodiment of the present disclosure provides a data communication apparatus for a processing system including a plurality of computing nodes, the data communication apparatus comprising: the target data acquisition unit is configured to respond to receiving a storage address from a first process, acquire target data sent to a second process by the first process based on the storage address, wherein the first process and the second process belong to the same computing node, the target data is stored in a first physical resource, and the first physical resource and a first virtual address space in the first process and a second virtual address space in the second process are respectively established with a mapping relation.

At least one embodiment of the present disclosure provides a processing system including a first computing node including a first process and a second process, the processing system including: the first physical resource is used for data communication between the first process and the second process, wherein a mapping relation is established between the first physical resource and a first virtual address space in the first process and a second virtual address space in the second process, and the first physical resource is used for caching data related to the operation of the first process and the second process; and a communication unit configured to divide a storage space required for the data communication from the first physical resource for the data communication in response to the data communication between the first process and the second process.

For example, in a processing system provided by at least one embodiment of the present disclosure, the first physical resource is at least a portion of the physical resources provided by the first computing node.

For example, in a processing system provided by at least one embodiment of the present disclosure, the plurality of computing nodes are configured to cooperatively perform parallel computing tasks.

At least one embodiment of the present disclosure provides an electronic device, including: a memory non-transitory storing computer-executable instructions; a processor configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement a data communication method in accordance with at least one embodiment of the present disclosure.

At least one embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement a data communication method according to at least one embodiment of the present disclosure.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly described below, and it is apparent that the drawings in the following description relate only to some embodiments of the present disclosure, not to limit the present disclosure.

FIG. 1 is a schematic diagram of a computing node;

FIG. 2 is a schematic diagram of intra-node communication;

FIG. 3 is a schematic flow chart diagram of a data communication method provided by at least one embodiment of the present disclosure;

FIG. 4 is a schematic flow chart diagram of another data communication method provided by at least one embodiment of the present disclosure;

FIG. 5 is a schematic flow chart diagram of another data communication method provided by at least one embodiment of the present disclosure;

FIG. 6 is a schematic process diagram of a data communication method provided by an embodiment of the present disclosure;

FIG. 7 is a timing diagram of a process according to an embodiment of the present disclosure;

FIG. 8 is a schematic block diagram of a data communication apparatus provided in accordance with at least one embodiment of the present disclosure;

FIG. 9 is a schematic block diagram of another data communication apparatus provided in accordance with at least one embodiment of the present disclosure;

FIG. 10 is a schematic block diagram of a processing system provided in accordance with at least one embodiment of the present disclosure;

FIG. 11 is a schematic block diagram of an electronic device provided in accordance with at least one embodiment of the present disclosure;

fig. 12 is a schematic diagram of a non-transitory computer readable storage medium according to at least one embodiment of the present disclosure.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present disclosure. It will be apparent that the described embodiments are some, but not all, of the embodiments of the present disclosure. All other embodiments, which can be made by one of ordinary skill in the art without the need for inventive faculty, are within the scope of the present disclosure, based on the described embodiments of the present disclosure.

Unless defined otherwise, technical or scientific terms used in this disclosure should be given the ordinary meaning as understood by one of ordinary skill in the art to which this disclosure belongs. The terms "first," "second," and the like, as used in this disclosure, do not denote any order, quantity, or importance, but rather are used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that elements or items preceding the word are included in the element or item listed after the word and equivalents thereof, but does not exclude other elements or items. The terms "connected" or "connected," and the like, are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", etc. are used merely to indicate relative positional relationships, which may also be changed when the absolute position of the object to be described is changed.

In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits a detailed description of some known functions and known components.

The high-performance cluster is characterized in that all nodes are cooperated in the calculation process, process part of big problems respectively, exchange data according to the need in the process, and the processing result of each node is part of the final result. Such clusters mainly solve the computation of large-scale scientific problems and the processing of mass data, such as scientific research, weather forecast, computation simulation, military research, biopharmaceutical, genetic sequencing, image processing and the like.

The main purpose of constructing a high-performance computing system is to increase the computing speed, and the requirements on the aspects of a processor, a memory bandwidth, a computing mode, a system I/O, storage and the like of the system are very high in order to achieve the computing speed of trillion secondary parts per second, and each link directly influences the computing speed of the system.

The high-performance computing system comprises a plurality of computing nodes, integrates the computing capabilities of the computing nodes, and effectively completes the data-intensive computing task. For example, multiple computing nodes may cooperatively execute parallel computing tasks, e.g., a parallel computing task may be multiple sub-tasks that may be executed in parallel in a large computing task, multiple sub-tasks may not be associated with each other (e.g., high-throughput computing), or multiple sub-tasks may be closely related to each other and require a large amount of data exchange (e.g., distributed computing), etc.

FIG. 1 is a schematic diagram of a computing node. FIG. 1 illustrates a processing system, such as an HPC cluster, made up of a plurality of compute nodes (compute node 0, compute nodes 1, …, compute node N-1, and compute node N).

For example, each computing node may include at least one processor Core (Core), each having independent local memory, input-output slots, etc., with the computing nodes being connected and information interacted with each other through an interconnection switch (cross bar switch) or an interconnection network.

For example, in a hardware implementation, the computing node may be a processor, or a portion of a processor core in a processor (such as a virtual machine or an operating system), or the computing node may be a shared memory processing unit including a plurality of processor cores as shown in fig. 1, where the implementation of the computing node is related to the architecture of the processing system.

For example, the computing nodes of the main stream include both homogeneous nodes and heterogeneous nodes. The isomorphic computing nodes refer to that each computing node in the cluster is completely composed of CPU (central processing unit) computing resources, and currently, one isomorphic computing node can support CPU computing nodes such as one-way, two-way, four-way, eight-way and the like; heterogeneous compute nodes refer to each compute node in a cluster being composed of a variety of computing resources, such as a cpu+gpu (graphics processing unit), a cpu+fpga (field programmable gate array), and the like.

In the HPC application scenario, the communication problem is a core problem of overall system performance, and is also a major bottleneck. At present, an interface provided by an MPI (message passing interface) protocol is mainly adopted in HPC to realize communication between computing tasks, and the MPI protocol is a set of standards for multi-node data communication. There are many current frame implementations following the MPI protocol, such as OpenMPI, intelMPI, MVAPICH (wireless bandwidth abbreviation on VAPI layer, an implementation of the MPI standard), UCX (Unified CommunicationX, unified abstract communication interface), etc.

With the computing nodes as communication dividing units, MPI communication can be divided into Inter-node communication (Intra-node) and Intra-node communication (Intra-node), wherein the Inter-node communication refers to the distribution of computing tasks of the Inter-node communication on different computing nodes, and the Intra-node communication refers to the distribution of computing tasks of the Inter-node communication on the same computing node.

At present, communication in a node has communication modes such as SYSV (system v, an inter-process communication mode in a linux system)/POSIX (portable operating system interfaces for computing systems)/CMA (continuous memory allocation), and the like, and these modes are all to copy data to be transmitted from a transmitting process to a shared memory, and then from the shared memory to a receiving process. The shared memory directly modifies the kernel area in the process to realize the communication interaction of information between two different processes without affinity, and the kernel creates the one-to-one mapping relation between the shared memory and the information of the process, so that the interaction of communication content information can be realized like a mirror.

Fig. 2 is a schematic diagram of intra-node communication.

As shown in fig. 2, when a sending process needs to send data to a receiving process, a computing node including the sending process and the receiving process enters a kernel mode from a user mode, the sending process copies data cached at a sending end to a shared memory, then the receiving process copies data from the shared memory to a receiving end cache, and the computing node exits the kernel mode and switches to the user mode.

Because the use of shared memory requires modification of the kernel area, this communication mode requires entry from the user mode into the kernel mode when a copy of the data occurs. In the operating system, the user mode has no direct control authority to the hardware and can not directly access the memory of the address, the program accesses the hardware and the memory by calling a system interface (System call APIs), and the kernel mode executes the code of the kernel space and has all operation authorities to the hardware. The kernel mode has higher operation authority, and the switching between the user mode and the kernel mode can involve interrupt processing, saving and recovering field information, and a series of security checks are performed, so that the time is consumed.

In addition, this way of communication between nodes requires two copies, which are also time consuming. And because the shared memory of the kernel area is used, the length of the shared memory is usually not too large, and the single data volume of the inter-process interaction of the high-performance computing system may be large, for example, the size of the shared memory is 2MB or 4KB, and the data volume of the single inter-process interaction may reach tens or hundreds of MB, which may need more memory copies to be realized, thereby further increasing the delay of data communication. For a computing system such as a high-performance computing system, which needs to complete mass computing every second, the delay of data interaction has a great influence on the computing efficiency, and multiple data copies and switching between a kernel mode and a user mode can reduce the computing performance of the system and reduce the computing efficiency of the system.

At least one embodiment of the present disclosure provides a data communication method and apparatus, a processing system, an electronic device, and a non-transitory computer readable storage medium. The data communication method comprises the following steps: determining a storage address of target data for data communication between a first process and a second process, wherein the first process and the second process belong to the same computing node, the target data is stored in a first physical resource, and a mapping relation is established between the first physical resource and a first virtual address space in the first process and between the first physical resource and a second virtual address space in the second process; and sending the storage address belonging to the first virtual address space to the second process so that the second process acquires the target data based on the storage address.

For example, a data communication method provided in at least one embodiment of the present disclosure includes: and responding to the received storage address from the first process, and acquiring target data sent to the second process by the first process based on the storage address, wherein the first process and the second process belong to the same computing node, the target data is stored in a first physical resource, and the first physical resource and a first virtual address space in the first process and a second virtual address space in the second process are respectively established with a mapping relation.

For example, the data communication method provides the first physical resource for communication between the first process and the second process, and establishes a mapping relation between the first virtual address space in the first process and the second virtual address space in the second process and the first physical resource, so that data can be directly copied through the virtual address in the process during copying, and the data can be copied in a user mode without switching to a kernel mode; and the first process sends the storage address to the second process during copying, the first process does not need to copy the data to the shared memory, intra-node communication can be realized only by one copy, intra-node communication time is shortened, intra-node communication bandwidth is increased, system operation speed and operation efficiency are improved, and system performance is optimized.

Embodiments of the present disclosure will be described in detail below with reference to the attached drawings, but the present disclosure is not limited to these specific embodiments.

Fig. 3 is a schematic flow chart of a data communication method according to at least one embodiment of the present disclosure.

For example, the data communication method is used in a processing system including a plurality of computing nodes, for example, the plurality of computing nodes are used to cooperatively execute parallel computing tasks, each computing node may be a processor, or a part of processor cores (such as a virtual machine or an operating system) in a processor, or the computing nodes may be a shared memory processing unit including a plurality of processor cores, which is not limited in the form of the computing node in this disclosure. For more description of computing nodes, reference may be made to the foregoing description of FIG. 1 and the like.

For example, the processing system may be a high performance computing system, or the processing system may be another processing system including a plurality of computing nodes, such as a NUMA (non-uniform memory access) system, etc., and the architecture of the processing system is not particularly limited by the present disclosure.

The data communication method provided by at least one embodiment of the present disclosure is applicable to any scenario of intra-node communication.

For any one computing node, such as a first computing node, the first computing node includes a first process and a second process. In this disclosure, a process is a running application (App). One program may contain multiple processes, but one process cannot belong to multiple programs at the same time. The processes have independent running environments (memory, registers, CPU execution time and the like) which are independent executable basic units in an operating system, and the resources occupied by each process are independent and are not shared with other processes.

For example, the data communication method is used for a first process, for example, a first process is a transmitting process that transmits data, and a second process is a receiving process that receives data.

For example, as shown in fig. 3, the data communication method provided in the embodiment of the present disclosure includes at least steps S10 to S20.

In step S10, a storage address of target data for data communication between the first process and the second process is determined.

In step S20, a storage address belonging to the first virtual address space is transmitted to the second process, so that the second process acquires the target data based on the storage address.

For example, the target data is stored in a first physical resource, and the first physical resource and a first virtual address space in a first process and a second virtual address space in a second process are both established with a mapping relationship.

For example, the starting address of the first virtual address space and the starting address of the second virtual address space are different, and the lengths of the first virtual address space and the second virtual address space are the same. For example, the first virtual address space is a virtual address space managed by the first process, and the second virtual address space is a virtual address space managed by the second process, which are independent of each other, so that the start address of the first virtual address space and the start address of the second virtual address space are different. However, since the first virtual address space and the second virtual address space both establish a complete mapping relationship with the first physical resource, the lengths of the first virtual address space and the second virtual address space are the same.

For example, the first physical resource is at least a portion of the physical resources of the first computing node, the first physical resource being used for data communication between the first process and the second process. For example, the physical resource of the first computing node may be physical memory in the processor, or may be shared memory shared by multiple processor cores within one computing node as shown in fig. 1, or the like.

For example, the first physical resource is a shared resource pool allocated by the processing system for data communication between the first process and the second process, both exchange target data to be communicated stored in the first physical resource in a contracted manner, and resources required for each data communication or data exchange are divided from the first physical resource.

Generally, memory resources between processes are not shared, and a process cannot access other process memory spaces, and other processes cannot access the process memory spaces. However, the mapping relation is established between the first physical resource and the first virtual address space in the first process and between the first physical resource and the second virtual address space in the second process, so that the first process and the second process can access the first physical resource through the respective virtual addresses to exchange data, and the first process and the second process can access in a user mode without entering the kernel mode because the operation on the kernel area is not required directly or the physical address of the memory is not required, and the delay caused by switching between the user mode and the kernel mode is reduced.

Before normal data communication is performed, for example, when a program is initialized, or before a communication connection is established between a first process and a second process, a first physical resource needs to be applied first and mapping between the first physical resource and a first virtual address space and a second virtual address space is completed.

For example, before step S10, the data communication method provided in at least one embodiment of the present disclosure further includes: applying for a first physical resource for data communication between a first process and a second process, and establishing a mapping relation between the first physical resource and a first virtual address space, wherein the first physical resource is at least part of physical resources in physical resources provided by the same computing node; and sending the related information of the first physical resource to the second process so that the second process establishes a mapping relation between the first physical resource and the second virtual address space.

For example, in some embodiments, applying for the first physical resource for data communication between the first process and the second process, and establishing a mapping relationship between the first physical resource and the first virtual address space may include: setting a first process to enter a kernel mode, and applying for a first physical resource; applying for an identification number corresponding to data communication between the first process and the second process, wherein the identification number is used for indicating the first physical resource; establishing a mapping relation between a first physical resource and a first virtual address space in a first process; and setting the first process to exit from the kernel mode.

For example, in one example, a first process applies for a first physical resource of 100M, and a first computing node allocates a contiguous memory space of 100M from a physical memory as the first physical resource to the first process for data exchange between the first process and a second process. When the first process is allocated, the first process enters a kernel state firstly because of using a memory physical address, and obtains an identification number for indicating data communication of the first process and the second process, wherein the identification number uniquely indicates that both communication parties are the first process and the second process, and the identification number is associated with information of the first physical resource in the kernel, for example, the identification number can be used as an index, and the information of the first physical resource, such as a starting address, a length and the like of the first physical resource in the memory, can be found through the identification number.

Then, a mapping relationship is established between the first physical resource and a first virtual address space in the first process, for example, virtual addresses A1-a1000 are selected from the virtual address space managed by the first process to be used as the first virtual address space, and a mapping relationship between the virtual addresses A1-a1000 and the first physical resource is established between the virtual addresses and the physical addresses, wherein the specific mapping process is the same as the conventional address mapping process, and details are not repeated here.

And then, the computing node exits the kernel mode to enter the user mode, and does not need to enter the kernel mode again in the data exchange or data communication process of the subsequent first process and the second process. That is, in the data communication method provided in the present disclosure, except for the process of applying for physical resources and establishing a mapping relationship with the virtual address space, the process is generally in a user state, except for other special needs.

The first process also needs to send the related information of the first physical resource to the second process, for example, directly send the start address, the length, etc. of the first physical resource to the second process. For example, the first process may send the identifier of the application to the second process, for example, the second process may find information of the first physical resource of the application, such as a start address, a length, etc. of the first physical resource of the application, so that a mapping relationship may be established between the first physical resource and a second virtual address space in the second process, which is described in related parts hereinafter, and details are not repeated herein.

For example, the first physical resource may store data related to the operation of the first process and the second process, e.g., the first process may store data to be transmitted directly in the first physical resource by accessing the virtual address, in such a way that the data access may be accomplished without directly accessing the kernel area and without high permissions, thereby avoiding copying the data from the transmitting process to the first copy of the shared memory and switching from the user state to the kernel state.

For example, in some examples, the number of resources of the first physical resource is determined by a single maximum data traffic of data communications between the first process and the second process. For example, the single maximum data traffic between the first process and the second process may be predicted to be 100M in advance, so that the first physical resource of 100M is applied, and the physical resource does not need to be applied again in the whole communication process, so that the number of times of entering the kernel mode due to the application of the physical resource is reduced to the greatest extent. And the physical resource for data communication between the first process and the second process is larger than that of the traditional shared memory, so that the copying times can be further reduced, and the communication delay in the node is reduced.

For example, in other examples, if the length of the target data to be sent is less than or equal to the number of the first physical resources already applied, the subsequent data communication is directly performed; and if the length of the target data to be transmitted is greater than the number of the first physical resources already applied, applying the physical resources again for transmitting the target data.

For example, the determination of applying for the physical resource may be performed before communication, if the length of the target data to be sent is determined to be less than or equal to the number of the first physical resources already applied, the operation of applying for the physical resource is skipped to directly perform data communication, and if the length of the target data to be sent is determined to be greater than the number of the first physical resources already applied, the application of the physical resource is performed.

For example, in some embodiments, the data communication method provided in at least one embodiment of the present disclosure further includes: applying for a second physical resource for transmission of the target data according to the length of the target data in response to the length of the target data being greater than the number of resources of the first physical resource, wherein the number of resources of the second physical resource is determined by a difference between the length of the target data and the number of resources of the first physical resource; establishing a mapping relation between the second physical resource and a third virtual address space in the first process; and sending the related information of the second physical resource to the second process so that the second process establishes a mapping relation between the second physical resource and a fourth virtual resource space in the second process.

For example, in one example, the number of resources of the first physical resource is 100M, and the length of the target data that needs to be sent this time is 200M, then the application for the second physical resource with the number of resources being 100M is continued, where the second physical resource is used together with the first physical resource for communication between the first process and the second process.

For example, the first process enters a kernel state, selects a1001-a2000 from a virtual address space of the first process as a third virtual address space, establishes a mapping relationship between the third virtual address space and a second physical resource, updates information in the kernel, such as a start address, a length, etc., of the second physical resource, which indicates a physical resource corresponding to an identification number communicated between the first process and the second process, and sends the identification number to the second process, so that the second process searches for information of the second physical resource according to the identification number, establishes a mapping relationship between the second physical resource and a fourth virtual address space in the second process, and then the first process exits the kernel state.

At this time, for the first process and the second process, the number of physical resources used for performing data communication is 200M, in the subsequent communication process, if the length of the target data is less than 200M, the data communication is performed directly, and if the length of the target data is greater than 200M, the above process is re-performed, and more physical resources are applied for transmission of the target data.

In this embodiment, the physical resources for the data communication of the first process and the second process can be dynamically adjusted according to the current traffic of the data communication, without occupying a large amount of physical resources at a time, and the number of times of entering the kernel mode due to the application of the physical resources is reduced as much as possible. And the physical resource for data communication between the first process and the second process is larger than that of the traditional shared memory, so that the copying times can be further reduced, and the communication delay in the node is reduced.

In the above embodiment, because the mapping relationship between the first virtual address space and the second virtual address space and the first physical resource is established before the formal communication, the page fault abnormality caused by that the logical address is not allocated to the physical address or the physical address does not exist can be avoided, the communication speed is improved, and the calculation efficiency and the system performance are improved.

For example, since the first process is a sending process, the first process applies for the first physical resource, and of course, the second process applies for the first physical resource, which is not described in detail.

For example, after the first physical resource is applied and the mapping relationship is established, step S10 and step S20 may be performed in the user state.

For example, the first process may directly store the calculated target data into the first physical resource, determine a storage address of the target data in the first physical resource, where the storage address belongs to the first virtual address space, and the storage address may include a first virtual first address corresponding to the target data in the first virtual address space and a length of the target data.

For example, step S20 may include: determining an offset of the first virtual head address relative to a starting address of the first virtual address space; and sending the offset and the length of the target data to a second process, so that the second process acquires the target data based on the offset and the length of the target data.

Because the first virtual address space and the second virtual address space are independent from each other, and the first virtual address space and the second virtual address space both establish a mapping relationship with the first physical resource, when the storage address is sent, relevant information of the storage address may be actually sent, for example, the relevant information of the storage address may include an offset of the first virtual first address corresponding to the target data relative to a start address of the first virtual address space and a length of the target data, for example, an offset of the first virtual first address corresponding to the target data relative to the start address of the first virtual address space and the length of the target data may be sent to the second process, and the second process may determine the actual start address of the target data through the offset. For example, the second process adds the offset and the start address of the second virtual address space to obtain the source address, and copies the space of the length of the target data to the target address with the source address as the start position to copy the target data to the target address.

For example, the storage address includes a first virtual first address a120 corresponding to the target data in the first virtual address space and a length 20 of the target data, and the offset 120 and the length 20 of the target data are sent to the second process on the assumption that the offset of the first virtual first address a120 with respect to the start address A1 of the first virtual address space is determined to be 120. The second process adds the received offset 120 to the start address B1 of the second virtual address space to obtain a source address, e.g., the source address is B120, and copies the space of the length (20) of the target data to the target address with the source address as a start position to copy the target data to the target address.

For example, in the process that the first process sends the storage address to the second process, the first process is in a user state. In fact, as described above, in the data communication method provided in the present disclosure, the first process is usually in a user state except for the process of applying for physical resources and establishing a mapping relationship with the virtual address space, except for other special needs.

In the data communication method provided by at least one embodiment of the present disclosure, interaction with the kernel is greatly reduced, switching times from the user state to the kernel state are reduced, and in a formal data communication or interaction process, operations can be completed directly in the user state, and the data communication needs only one copy, so that communication time in the node is shortened, and system performance and operation efficiency are improved.

At least one embodiment of the present disclosure further provides a data communication method, and fig. 4 is a schematic flowchart of another data communication method provided by at least one embodiment of the present disclosure.

For example, the data communication method is used in a processing system including a plurality of computing nodes, and the description of the computing nodes and the processing system may refer to the foregoing, which is not repeated herein.

For example, the data communication method is used for the second process, and the relation and meaning of the first process and the second process can be referred to the related description, which is not repeated here.

For example, as shown in fig. 4, the data communication method provided in the embodiment of the present disclosure at least includes step S30.

In step S30, in response to receiving the memory address from the first process, target data transmitted by the first process to the second process is acquired based on the memory address.

For example, the first process and the second process belong to the same computing node, e.g., a first computing node. The target data is stored in a first physical resource, and the first physical resource and a first virtual address space in a first process and a second virtual address space in a second process are respectively established with a mapping relation.

For example, the first physical resource is at least a portion of the physical resources of the first computing node, the first physical resource being used for data communication between the first process and the second process. The related description of the first physical resource may refer to the foregoing, and will not be repeated here.

For example, the storage address includes an offset of a corresponding first virtual head address of the target data in the first virtual address space relative to a start address of the first virtual address space, and a length of the target data. For example, as described above, since the first virtual address space and the second virtual address space are independent from each other, and the first virtual address space and the second virtual address space each establish a mapping relationship with the first physical resource, when the storage address is sent, the offset of the first virtual head address corresponding to the target data with respect to the start address of the first virtual address space and the length of the target data can be sent to the second process.

For example, based on the memory address, obtaining the target data sent by the first process to the second process may include: determining a source address based on the offset and a start address of the second virtual address space; and copying the space with the length of the target data to the target address by taking the source address as a starting position so as to copy the target data to the target address.

For example, the source address may be the sum of the offset and the starting address of the second virtual address space.

For example, in copying the target data, the memcpy function may be used to copy the target data from the first physical resource to a target address in the second process, and store the target data in a memory space of the second process.

Similarly, before normal data communication is performed, for example, when the program is initialized, or when the second process receives the related information of the first physical resource sent by the first process, the second process needs to complete mapping between the first physical resource and the second virtual address space.

For example, the related information of the first physical resource includes an identification number corresponding to communication between the first process and the second process, the identification number being used to indicate the first physical resource.

For example, as described above, when the first process applies for the first physical resource, the identification number is obtained, where the identification number uniquely indicates that both the communication parties are the first process and the second process, and the identification number is associated with the information of the first physical resource in the kernel, for example, the identification number may be used as an index, and the information of the first physical resource, such as the start address, the length, etc. of the first physical resource in the memory, may be found by the identification number.

For example, receiving information about a first physical resource from a first process, establishing a mapping relationship between the first physical resource and a second virtual address space in a second process, including: receiving the identification number and setting the second process to enter a kernel mode; determining a first physical resource by the identification number; establishing a mapping relation between the first physical resource and a second virtual address space in a second process; and setting the second process to exit from the kernel mode.

For example, in the process of applying for the first physical resource and establishing the mapping relationship between the first physical resource and the first virtual address space and between the first virtual address space and the second virtual address space, the first process and the second process are in a kernel mode, and after the process is ended, the first process and the second process exit the kernel mode.

For example, after receiving the identification number, the second process uses the identification number as an index to find information of the first physical resource from the kernel, for example, a starting address, a length, and the like of the first physical resource in the memory. Then, a mapping relationship is established between the first physical resource and a second virtual address space in the second process, for example, virtual addresses B1-B1000 are selected from the virtual address space managed by the second process to be used as the second virtual address space, and a mapping relationship between the virtual addresses B1-B1000 and the first physical resource is established, wherein the specific mapping process is the same as the conventional address mapping process, and details are not repeated here.

And then, the second process exits the kernel mode to enter the user mode, and the second process does not need to enter the kernel mode again in the data exchange or data communication process of the subsequent first process and the second process. For example, in the process of the second process acquiring the target data based on the storage address, the second process is in the user state.

In the data communication method provided by at least one embodiment of the present disclosure, interaction with a kernel is greatly reduced, switching times from a user state to a kernel state are reduced, and in a formal data communication or interaction process, an operation can be completed directly in the user state, and the data communication only needs one copy, so that target data can be directly copied to a target address in a second process based on a storage address, thereby shortening intra-node communication time, and improving system performance and operation efficiency.

For example, the first computing node may further include a third process, and for the data communication process between the first process and the third process or the second process and the third process, the data communication process between the first process and the second process is similar, for example, for the data communication between the first process and the third process, a third physical resource may be applied for data communication between two processes, and for the data communication between the second process and the third process, a fourth physical resource may be applied for data communication between two processes, which is not described herein in detail. For example, in the present disclosure, a first physical resource may be dedicated to data communication between a first process and a second process, the data communication of the first process with other processes occurring with other physical resources.

At least one embodiment of the present disclosure also provides a data communication method. Fig. 5 is a schematic flow chart diagram of another data communication method provided in at least one embodiment of the present disclosure.

For example, as shown in fig. 5, the data communication method provided in the embodiment of the present disclosure at least includes steps S40 to S50.

In step S40, a first physical resource is provided for data communication between a first process and a second process.

For example, a mapping relationship is established between a first physical resource and a first virtual address space in a first process and a second virtual address space in a second process, and the first physical resource is used for caching data related to the operation of the first process and the second process.

For example, reference may be made to the foregoing related description for the relationship and meaning of the first process and the second process, which are not repeated here.

For example, reference may be made to the foregoing for a description of the first physical resource, which is not repeated here.

For example, step S40 may include: the first process applies for a first physical resource for data communication between the first process and the second process, and establishes a mapping relation between the first physical resource and a first virtual address space in the first process; the first process sends the related information of the first physical resource to the second process; and the second process establishes a mapping relation between the first physical resource and a second virtual address space in the second process according to the related information of the first physical resource.

In step S50, in response to the data communication between the first process and the second process, a memory space required for the data communication is divided from the first physical resource for the data communication.

For example, the first physical resource may store data related to the operation of the first process and the second process, for example, the first process may store target data to be transmitted in the first physical resource directly by accessing the virtual address in a user state, and during data communication, a storage space of the target data is divided from the first physical resource for data communication.

For example, step S50 may include: responsive to data communication between the first process and the second process, the first process determining a storage address for target data for the data communication between the first process and the second process; the first process sends a storage address belonging to a first virtual address space to the second process; the second process obtains the target data based on the memory address.

Fig. 6 is a schematic process diagram of a data communication method provided in an embodiment of the present disclosure.

The overall flow of the data communication method provided in at least one embodiment of the present disclosure is fully described below with reference to fig. 6.

For example, fig. 6 shows a data communication process between a first process and a second process in a first computing node, a physical address space is a physical resource in the computing node, a virtual address space in a first process block is a virtual address space allocated by the first computing node for the first process, and a virtual address space in a second process block is a virtual address space allocated by the first computing node for the second process.

As shown in fig. 6, the overall process includes four steps.

First, as shown in fig. 6, in step (1), resource registration is completed. For example, the resource registration includes, when the program is initialized, or before the first process and the second process establish a communication connection, applying for the first physical resource and completing mapping between the first physical resource and the first virtual address space.

Specifically, first, a first process enters a kernel mode, the first process applies for a first physical resource, a computing node allocates a corresponding memory space (the first physical resource in fig. 6) from a physical memory (the physical address space in fig. 6) to the first process, and meanwhile, the first process obtains an identification number indicating data communication of the first process and a second process, wherein the identification number is associated with information of the first physical resource, so that the second process can obtain the information of the first physical resource according to the identification number.

Then, the first physical resource is mapped to a first virtual address space in the first process, which will not be described in detail herein.

Thereafter, as shown in fig. 6, in step (2), the resource binding is completed. The resource binding includes that the first process sends related information of the first physical resource, such as an identification number, to the second process, the second process enters a kernel mode after receiving the identification number, and the information of the first physical resource applied by the first process is searched from the kernel area through the identification number.

Then, as shown in fig. 6, in step (3), resource mapping is completed, where the resource mapping includes that the second process establishes a mapping relationship between the first physical resource and a second virtual address space in the virtual address space of the second process based on the obtained information of the first physical resource, thereby completing application and mapping of the first physical resource.

And then, the first process and the second process exit from the kernel state and enter a user state. In the subsequent data exchange or data communication process of the first process and the second process, the first process and the second process are in a user state except for other special requirements and the process of applying for physical resources and establishing a mapping relation with a virtual address space.

When the first process and the second process need to perform data communication, first, the first process determines a storage address of target data to be sent, where the storage address includes, for example, a first virtual first address corresponding to the target data in a first virtual address space and a length of the target data. Then, an offset of the first virtual head address relative to a start address of the first virtual address space is determined, and the offset and a length of the target data are sent to the second process.

As shown in fig. 6, in step (4), for example, after receiving the offset and the length of the target data, the second process adds the offset to the start address of the second virtual address space to obtain a source address, takes the source address as a start position, copies the length of the target data to the target address, copies the target data to the target address, and caches the target data in the storage space of the second process.

In the embodiment, the method enters the kernel mode only in the process of applying for physical resources and establishing a mapping relation with the virtual address space, and the kernel mode does not need to be entered every time data communication, so that interaction with the kernel is greatly reduced, switching times from a user mode to the kernel mode are reduced, in the formal data communication or interaction process, the operation can be completed directly in the user mode, the data communication only needs to be copied once, the communication time in the node is shortened, and the system performance and the operation efficiency are improved. In actual test, compared with the traditional communication mode, the performance is obviously improved, and especially when the target data (message size) is within the range of 256 KB-8 MB, the improvement amplitude can reach 30% -100%.

Fig. 7 is a program timing diagram according to an embodiment of the disclosure.

As shown in fig. 7, the left side of the dotted line is the relevant interface of the first process, and the right side of the dotted line is the relevant interface of the second process.

MPI represents a protocol specification, and MPI provides only one tag that indicates which object to issue.

The communication interface is based on MPI protocol specification, and is hardware realization of MPI protocol. For example, the communication interface may be a UCX (unified abstract communication interface) module, which is capable of adapting to any communication device and supporting the requirements of various applications, and is a high-performance network communication library. A hardware implementation of a specific intra-node communication or inter-node communication is encapsulated in a communication interface, for example, a related implementation of a data communication method provided in at least one embodiment of the present disclosure may be encapsulated in a communication interface, for example, a UCX module.

As shown in fig. 7, when the first process performs intra-node data communication with the second process, the first process side cyclically transmits information such as a memory address of transmission target data. Specifically, the first process sends data to the MPI, the MPI sends a flag indicating the receiver to the communication module, the communication module determines, according to the flag, a receiving object, for example, the receiving object is a second process belonging to the same computing node, and the communication module encapsulates a message according to the intra-node communication mode and sends the message to the communication module of the second process.

The communication module of the second process side circularly receives the information, for example, copies the data after receiving the storage address, and the specific process is as shown before, and stores the copied target data into the storage space of the second process. In addition, after receiving the storage address, the communication module of the second process also sends a response message to the communication module of the first process, which indicates that the message is received, and the first process confirms that the data communication is correctly ended after receiving the response.

Corresponding to the above-mentioned data communication method, at least one embodiment of the present disclosure further provides a data communication device, and fig. 8 is a schematic block diagram of a data communication device provided by at least one embodiment of the present disclosure.

For example, as shown in fig. 8, the data communication apparatus 100 includes at least: a storage address determining unit 101, a transmitting unit 102.

For example, the data communication apparatus is used in a processing system including a plurality of computing nodes, and the concept of the computing node may refer to the related description of the foregoing data communication method, which is not repeated herein.

For example, the data communication means may be implemented as a communication interface of the first process side, such as a UCX module of the first process side. For example, here the first process may be a sending process.

The memory address determination unit 101 is configured to determine a memory address of target data for data communication between the first process and the second process.

For example, the first process and the second process belong to the same computing node, the target data is stored in a first physical resource, and a mapping relationship is established between the first physical resource and a first virtual address space in the first process and between the first physical resource and a second virtual address space in the second process;

and a transmitting unit 102 configured to transmit a storage address belonging to the first virtual address space to the second process so that the second process acquires the target data based on the storage address.

For example, as shown in fig. 8, the data communication apparatus 100 further includes a resource registration unit 103.

The resource registration unit 103 is configured to apply for a first physical resource for data communication between the first process and the second process, and establish a mapping relationship between the first physical resource and the first virtual address space, where the first physical resource is at least part of physical resources in physical resources provided by the same computing node; and sending the related information of the first physical resource to the second process so that the second process establishes a mapping relation between the first physical resource and the second virtual address space.

For example, when the resource registration unit 103 performs application of the first physical resource for data communication between the first process and the second process, and establishes a mapping relationship between the first physical resource and the first virtual address space, the following operations are performed: setting a first process to enter a kernel mode, and applying for a first physical resource; applying for an identification number corresponding to data communication between the first process and the second process, wherein the identification number is used for indicating the first physical resource; establishing a mapping relation between a first physical resource and a first virtual address space in a first process; and setting the first process to exit from the kernel mode.

For example, the related information of the first physical resource includes an identification number.

For example, the resource registration unit 103 is further configured to: applying for a second physical resource for transmission of the target data according to the length of the target data in response to the length of the target data being greater than the number of resources of the first physical resource, wherein the number of resources of the second physical resource is determined by a difference between the length of the target data and the number of resources of the first physical resource; establishing a mapping relation between the second physical resource and a third virtual address space in the first process; and sending the related information of the second physical resource to the second process so that the second process establishes a mapping relation between the second physical resource and a fourth virtual address space in the second process.

For example, the number of resources of the first physical resource is determined by a single maximum data traffic of data communication between the first process and the second process.

For example, the storage address includes a first virtual head address corresponding to the target data in the first virtual address space and a length of the target data. The transmitting unit performs transmission of the memory address belonging to the first virtual address space to the second process so that the second process acquires the target data based on the memory address, including performing the following operations: determining an offset of the first virtual head address relative to a starting address of the first virtual address space; and sending the offset and the length of the target data to the second process, so that the second process acquires the target data based on the offset and the length of the target data.

For example, the processing system is a high performance computing system.

For example, the starting address of the first virtual address space and the starting address of the second virtual address space are different, and the lengths of the first virtual address space and the second virtual address space are the same.

For example, in the process of the first process sending a storage address to the second process, the computing node is in a user state.

For example, the memory address determination unit 101, the transmission unit 102 include codes and programs stored in a memory; the processor may execute the code and program to implement some or all of the functions of the memory address determination unit 101, the transmission unit 102 as described above. For example, the memory address determining unit 101, the transmitting unit 102 may be dedicated hardware devices for implementing some or all of the functions of the memory address determining unit 101, the transmitting unit 102 as described above. For example, the memory address determination unit 101, the transmission unit 102 may be one circuit board or a combination of a plurality of circuit boards for realizing the functions as described above. In an embodiment of the present application, the circuit board or the combination of the circuit boards may include: (1) one or more processors; (2) One or more non-transitory memories coupled to the processor; and (3) firmware stored in the memory that is executable by the processor.

The storage address determining unit 101 is configured to implement step S10 shown in fig. 3, and the transmitting unit 102 is configured to implement step S20 shown in fig. 3. Thus, the specific description about the memory address determining unit 101 may refer to the related description of step S10 shown in fig. 3 in the above-described embodiment of the data communication method, and the specific description about the transmitting unit 102 may refer to the related description of step S20 shown in fig. 3 in the above-described embodiment of the data communication method. In addition, the data communication apparatus may achieve similar technical effects to those of the foregoing data communication method, and will not be described herein.

At least one embodiment of the present disclosure also provides another data communication apparatus. Fig. 9 is a schematic block diagram of another data communication apparatus provided in accordance with at least one embodiment of the present disclosure.

For example, as shown in fig. 9, the data communication apparatus 200 includes at least: a target data acquisition unit 201.

For example, the data communication means may be implemented as a communication interface of the second process side, such as a UCX module of the second process side. The second process may be, for example, a receiving process.

For example, the target data acquisition unit 201 is configured to acquire target data transmitted by the first process to the second process based on a storage address in response to receiving the storage address from the first process.

The first process and the second process belong to the same computing node, target data are stored in a first physical resource, and a mapping relation is established between the first physical resource and a first virtual address space in the first process and a second virtual address space in the second process.

For example, the storage address includes an offset of a corresponding first virtual head address of the target data in the first virtual address space relative to a start address of the first virtual address space, and a length of the target data. For example, when the target data acquisition unit 201 performs acquisition of target data transmitted from the first process to the second process based on the storage address, the following operations are performed: determining a source address based on the offset and a start address of the second virtual address space; and copying the space with the length of the target data to the target address by taking the source address as a starting position so as to copy the target data to the target address.

For example, as shown in fig. 9, the data communication apparatus 200 further includes a resource mapping unit 202.

The resource mapping unit 202 is configured to receive information about a first physical resource from a first process, and establish a mapping relationship between the first physical resource and a second virtual address space in a second process.

For example, the related information of the first physical resource includes an identification number corresponding to communication between the first process and the second process, the identification number is used to indicate the first physical resource, and the resource mapping unit 202 performs the following operations when receiving the related information of the first physical resource from the first process and establishing a mapping relationship between the first physical resource and the second virtual address space in the second process: receiving the identification number and setting the second process to enter a kernel mode; determining a first physical resource by the identification number; establishing a mapping relation between the first physical resource and a second virtual address space in a second process; and setting the second process to exit from the kernel mode.

For example, in the process of the second process acquiring the target data based on the storage address, the second process is in the user state.

For example, the target data acquisition unit 201, the resource mapping unit 202 include codes and programs stored in a memory; the processor may execute the code and program to implement some or all of the functions of the target data acquisition unit 201, the resource mapping unit 202 as described above. For example, the target data acquisition unit 201, the resource mapping unit 202 may be dedicated hardware devices for implementing some or all of the functions of the target data acquisition unit 201, the resource mapping unit 202 as described above. For example, the target data acquisition unit 201, the resource mapping unit 202 may be one circuit board or a combination of a plurality of circuit boards for realizing the functions as described above. In an embodiment of the present application, the circuit board or the combination of the circuit boards may include: (1) one or more processors; (2) One or more non-transitory memories coupled to the processor; and (3) firmware stored in the memory that is executable by the processor.

The target data obtaining unit 201 is configured to implement step S30 shown in fig. 4. So that a detailed description of the target data acquisition unit 201 can be referred to the related description of step S30 shown in fig. 4 in the embodiment of the data communication method described above. In addition, the data communication apparatus may achieve similar technical effects to those of the foregoing data communication method, and will not be described herein.

At least one embodiment of the present disclosure also provides a processing system. Fig. 10 is a schematic block diagram of a processing system provided in accordance with at least one embodiment of the present disclosure.

For example, as shown in FIG. 10, processing system 300 includes a first computing node, although processing system 300 may include other computing nodes not shown in FIG. 10.

For example, as shown in fig. 10, the first computing node includes a first process and a second process, and the meaning of the first process and the second process may refer to the related description of the foregoing data communication method, which is not repeated herein.

For example, the first computing node further includes a first physical resource 302, the first physical resource 302 being used for data communication between the first process and the second process.

For example, a mapping relationship is established between a first physical resource and a first virtual address space in a first process and a second virtual address space in a second process, and the first physical resource is used for caching data related to the operation of the first process and the second process. The related description of the first physical resource may refer to the content of the foregoing data communication method, and will not be repeated here.

In addition, regarding the related content of applying for the first physical resource and establishing the mapping relationship between the first physical resource and the first virtual address space and the second virtual address space, reference may be made to the related description of the foregoing data communication method, or the related description of the foregoing resource registration unit 103 and the resource mapping unit 202, which are not repeated herein.

For example, the first computing node further comprises a communication unit 301 configured to divide a memory space required for data communication from the first physical resource for data communication in response to data communication between the first process and the second process.

For example, when the communication unit 301 performs dividing a memory space required for data communication from the first physical resource for data communication in response to data communication between the first process and the second process, the following operations are performed: responsive to data communication between the first process and the second process, the first process determining a storage address for target data for the data communication between the first process and the second process; the first process sends a storage address belonging to a first virtual address space to the second process; the second process obtains the target data based on the memory address.

For example, the first physical resource is at least a portion of the physical resources provided by the first computing node.

For example, multiple compute nodes are used to cooperatively perform parallel computing tasks. For example, processing system 300 may be a HPC cluster, or other system in which intra-node communication exists, and the specific architecture of processing system 300 is not limited by this disclosure.

The communication unit 301 is configured to implement step S50 shown in fig. 5. So that a detailed description of the communication unit 301 can be referred to with respect to step S50 shown in fig. 5 in the above-described embodiment of the data communication method.

The processing system provided by at least one embodiment of the present disclosure greatly reduces interaction with the kernel, reduces switching times from a user mode to a kernel mode, and can complete operation directly in the user mode in a formal data communication or interaction process, and the data communication only needs one copy, so that communication time in a node is shortened, and system performance and operation efficiency are improved.

At least one embodiment of the present disclosure further provides an electronic device, and fig. 11 is a schematic block diagram of an electronic device provided by at least one embodiment of the present disclosure.

For example, as shown in fig. 11, the electronic device includes a processor 1001, a communication interface 1002, a memory 1003, and a communication bus 1004. The processor 1001, the communication interface 1002, and the memory 1003 communicate with each other via the communication bus 1004, and the components of the processor 1001, the communication interface 1002, and the memory 1003 may communicate with each other via a network connection. The present disclosure is not limited herein with respect to the type and functionality of the network.

For example, the memory 1003 is used to store computer-executable instructions non-transitory. The processor 1001 is configured to execute computer-executable instructions that, when executed by the processor 1001, implement a data communication method according to any of the embodiments described above. For specific implementation of each step of the data communication method and related explanation, reference may be made to the above embodiment of the data communication method, which is not described herein.

For example, the implementation manner of the data communication method by the processor 1001 executing the program stored in the memory 1003 is the same as the implementation manner mentioned in the foregoing embodiment portion of the data communication method, and will not be described herein again.

For example, communication bus 1004 may be a peripheral component interconnect standard (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.

For example, the communication interface 1002 is used to enable communication between an electronic device and other devices.

For example, the processor 1001 may control other components in the electronic device to perform desired functions. The processor 1001 may be a Central Processing Unit (CPU), a Network Processor (NP), etc., and may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The Central Processing Unit (CPU) can be an X86 or ARM architecture, etc.

For example, memory 1003 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. Volatile memory can include, for example, random Access Memory (RAM) and/or cache memory (cache) and the like. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, and the like. One or more computer-executable instructions may be stored on the computer-readable storage medium that may be executed by the processor 1001 to implement various functions of an electronic device. Various applications and various data, etc. may also be stored in the storage medium.

For example, a detailed description of a process of the electronic device performing the design of the digital circuit may refer to a related description in the embodiment of the data communication method, and the repetition is not repeated.

Fig. 12 is a schematic diagram of a non-transitory computer readable storage medium according to at least one embodiment of the present disclosure. For example, as shown in fig. 12, one or more computer-executable instructions 1101 may be stored non-transitory on the storage medium 1100. For example, the computer-executable instructions 1101, when executed by a processor, may perform one or more steps in accordance with the data communication methods described above.

For example, the storage medium 1100 may be applied to the electronic device and/or the data communication apparatus described above. For example, storage medium 1100 may include memory 1003 in an electronic device.

For example, the description of the storage medium 1100 may refer to the description of the memory in the embodiment of the electronic device, and the repetition is omitted.

Those skilled in the art will appreciate that various modifications and improvements can be made to the disclosure. For example, the various devices or components described above may be implemented in hardware, or may be implemented in software, firmware, or a combination of some or all of the three.

Further, while the present disclosure makes various references to certain elements in a system according to embodiments of the present disclosure, any number of different elements may be used and run on a client and/or server. The units are merely illustrative and different aspects of the systems and methods may use different units.

A flowchart is used in this disclosure to describe the steps of a method according to an embodiment of the present disclosure. It should be understood that the steps that follow or before do not have to be performed in exact order. Rather, the various steps may be processed in reverse order or simultaneously. Also, other operations may be added to these processes.

Those of ordinary skill in the art will appreciate that all or a portion of the steps of the methods described above may be implemented by a computer program to instruct related hardware, and the program may be stored in a computer readable storage medium, such as a read only memory, a magnetic disk, or an optical disk. Alternatively, all or part of the steps of the above embodiments may be implemented using one or more integrated circuits. Accordingly, each module/unit in the above embodiment may be implemented in the form of hardware, or may be implemented in the form of a software functional module. The present disclosure is not limited to any specific form of combination of hardware and software.

Unless defined otherwise, all terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

The foregoing is illustrative of the present disclosure and is not to be construed as limiting thereof. Although a few exemplary embodiments of this disclosure have been described, those skilled in the art will readily appreciate that many modifications are possible in the exemplary embodiments without materially departing from the novel teachings and advantages of this disclosure. Accordingly, all such modifications are intended to be included within the scope of this disclosure as defined in the claims. It is to be understood that the foregoing is illustrative of the present disclosure and is not to be construed as limited to the specific embodiments disclosed, and that modifications to the disclosed embodiments, as well as other embodiments, are intended to be included within the scope of the appended claims. The disclosure is defined by the claims and their equivalents.

Claims

1. A data communication method for a processing system including a plurality of computing nodes, the data communication method comprising:

determining a storage address of target data for data communication between a first process and a second process, wherein the first process and the second process belong to the same computing node, the target data is stored in a first physical resource, and a mapping relation is established between the first physical resource and a first virtual address space in the first process and between the first physical resource and a second virtual address space in the second process;

and sending the storage address belonging to the first virtual address space to the second process so that the second process acquires the target data based on the storage address.

2. The data communication method according to claim 1, wherein before determining a storage address of target data for data communication between the first process and the second process, the data communication method further comprises:

applying for the first physical resource to be used for data communication between the first process and the second process, and establishing a mapping relationship between the first physical resource and the first virtual address space, wherein the first physical resource is at least part of physical resources in physical resources provided by the same computing node;

And sending the related information of the first physical resource to the second process so that the second process establishes a mapping relation between the first physical resource and the second virtual address space.

3. The data communication method according to claim 2, wherein applying the first physical resource for data communication between the first process and the second process, and establishing a mapping relationship between the first physical resource and the first virtual address space, includes:

setting the first process to enter a kernel mode, and applying for the first physical resource;

applying for an identification number corresponding to data communication between the first process and the second process, wherein the identification number is used for indicating the first physical resource;

establishing a mapping relation between the first physical resource and a first virtual address space in the first process;

and setting the first process to exit the kernel mode.

4. A data communication method according to claim 3, wherein the information about the first physical resource comprises the identification number.

5. The data communication method of claim 2, further comprising:

in response to the length of the target data being greater than the number of resources of the first physical resource,

Applying for a second physical resource for transmission of the target data according to the length of the target data, wherein the resource number of the second physical resource is determined by a difference value between the length of the target data and the resource number of the first physical resource;

establishing a mapping relation between the second physical resource and a third virtual address space in the first process;

and sending the related information of the second physical resource to the second process so that the second process establishes a mapping relation between the second physical resource and a fourth virtual address space in the second process.

6. The data communication method of claim 1, wherein the number of resources of the first physical resource is determined by a single maximum data traffic of data communication between the first process and the second process.

7. The data communication method of claim 1, wherein the storage address includes a first virtual head address corresponding to the target data in the first virtual address space and a length of the target data,

sending the storage address belonging to the first virtual address space to the second process so that the second process obtains the target data based on the storage address, including:

Determining an offset of the first virtual head address relative to a starting address of the first virtual address space;

and sending the offset and the length of the target data to the second process, so that the second process acquires the target data based on the offset and the length of the target data.

8. The data communication method of any of claims 1-7, wherein the processing system is a high performance computing system.

9. The data communication method according to any one of claims 1 to 7, wherein a start address of the first virtual address space and a start address of the second virtual address space are different, and a length of the first virtual address space and a length of the second virtual address space are the same.

10. The data communication method according to any one of claims 1 to 7, wherein the first process is in a user state in a process in which the first process transmits the storage address to the second process.

11. A data communication method for a processing system including a plurality of computing nodes, the data communication method comprising:

and responding to the received storage address from the first process, and acquiring target data sent to the second process by the first process based on the storage address, wherein the first process and the second process belong to the same computing node, the target data is stored in a first physical resource, and the first physical resource and a first virtual address space in the first process and a second virtual address space in the second process are respectively established with a mapping relation.

12. The data communication method of claim 11, wherein the memory address includes an offset of a corresponding first virtual head address of the target data in the first virtual address space relative to a start address of the first virtual address space, and a length of the target data,

based on the storage address, obtaining target data sent by the first process to the second process, including:

determining a source address based on the offset and a start address of the second virtual address space;

and copying the space with the length of the target data to a target address by taking the source address as a starting position so as to copy the target data to the target address.

13. The data communication method according to claim 11, wherein before acquiring the target data sent by the first process to the second process based on the storage address, the data communication method further comprises:

and receiving related information of the first physical resource from the first process, and establishing a mapping relation between the first physical resource and a second virtual address space in the second process.

14. The data communication method of claim 13, wherein the related information of the first physical resource includes an identification number corresponding to communication between the first process and the second process, the identification number indicating the first physical resource,

Receiving information about the first physical resource from the first process, and establishing a mapping relationship between the first physical resource and a second virtual address space in the second process, including:

receiving the identification number and setting the second process to enter a kernel mode;

determining the first physical resource by the identification number;

establishing a mapping relation between the first physical resource and a second virtual address space in the second process;

and setting the second process to exit the kernel mode.

15. The data communication method according to any one of claims 11 to 14, wherein the second process is in a user state in a process of acquiring the target data based on the storage address.

16. A data communication method for a processing system including a plurality of computing nodes, the data communication method comprising:

providing a first physical resource for data communication between a first process and a second process, wherein the first process and the second process belong to the same computing node, the first physical resource and a first virtual address space in the first process and a second virtual address space in the second process are respectively established with a mapping relation, and the first physical resource is used for caching data related to the operation of the first process and the second process;

In response to a data communication between the first process and the second process, dividing a memory space required for the data communication from the first physical resource for the data communication.

17. The data communication method of claim 16, wherein providing the first physical resource for data communication between the first process and the second process comprises:

the first process applies for the first physical resource to be used for data communication between the first process and the second process, and establishes a mapping relation between the first physical resource and a first virtual address space in the first process;

the first process sends the related information of the first physical resource to the second process;

and the second process establishes a mapping relation between the first physical resource and a second virtual address space in the second process according to the related information of the first physical resource.

18. The data communication method of claim 16, wherein responsive to the data communication between the first process and the second process, dividing the memory space required for the data communication from the first physical resource for the data communication comprises:

In response to a data communication between the first process and the second process, the first process determining a storage address for target data for the data communication between the first process and the second process;

the first process sends the storage address belonging to the first virtual address space to the second process;

the second process obtains the target data based on the memory address.

19. The data communication method according to any one of claims 16 to 18, wherein the first process and the second process are in a user state during data communication between the first process and the second process.

20. A data communication apparatus for a processing system comprising a plurality of computing nodes, the data communication apparatus comprising:

a storage address determining unit configured to determine a storage address of target data for data communication between a first process and a second process, wherein the first process and the second process belong to the same computing node, the target data is stored in a first physical resource, and a mapping relationship is established between the first physical resource and a first virtual address space in the first process and a second virtual address space in the second process;

And a transmitting unit configured to transmit the storage address belonging to the first virtual address space to the second process so that the second process acquires the target data based on the storage address.

21. A data communication apparatus for a processing system comprising a plurality of computing nodes, the data communication apparatus comprising:

the target data acquisition unit is configured to respond to receiving a storage address from a first process, acquire target data sent to a second process by the first process based on the storage address, wherein the first process and the second process belong to the same computing node, the target data is stored in a first physical resource, and the first physical resource and a first virtual address space in the first process and a second virtual address space in the second process are respectively established with a mapping relation.

22. A processing system includes a first computing node including a first process and a second process,

the processing system includes:

the first physical resource is used for data communication between the first process and the second process, wherein a mapping relation is established between the first physical resource and a first virtual address space in the first process and a second virtual address space in the second process, and the first physical resource is used for caching data related to the operation of the first process and the second process;

And a communication unit configured to divide a storage space required for the data communication from the first physical resource for the data communication in response to the data communication between the first process and the second process.

23. The processing system of claim 22, wherein the first physical resource is at least a portion of a physical resource provided by the first computing node.

24. The processing system of claim 22, wherein the plurality of compute nodes are to cooperatively perform parallel computing tasks.

25. An electronic device, comprising:

a memory non-transitory storing computer-executable instructions;

a processor configured to execute the computer-executable instructions,

wherein the computer executable instructions when executed by the processor implement a data communication method according to any of claims 1-19.

26. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions,

the computer executable instructions, when executed by a processor, implement the data communication method according to any one of claims 1-19.