CN114253883B

CN114253883B - Method and system for accessing endpoint equipment and endpoint equipment

Info

Publication number: CN114253883B
Application number: CN202111475014.5A
Authority: CN
Inventors: 尹淇; 毛磊冰
Original assignee: Fiberhome Telecommunication Technologies Co Ltd
Current assignee: Wuhan Changjiang Computing Technology Co ltd; Fiberhome Telecommunication Technologies Co Ltd
Priority date: 2021-12-03
Filing date: 2021-12-03
Publication date: 2024-05-10
Anticipated expiration: 2041-12-03
Also published as: CN114253883A

Abstract

The application relates to an access method and system for an endpoint device and the endpoint device, wherein a DMA controller is preset in the endpoint device and is used for writing data into a host CPU according to control information stored in a preset DMA controller register; the method comprises the following steps: the host CPU writes control information of data to be read into a DMA controller register arranged in the endpoint device; and the DMA controller writes the data to be read of the endpoint device into the host CPU according to the control information. The method can solve the problem that the CPU of the host is suspended when the PCIe equipment is pulled out in the related technology.

Description

Method and system for accessing endpoint equipment and endpoint equipment

Technical Field

The present invention relates to the field of data communications technologies, and in particular, to an endpoint device access method and system, and an endpoint device.

Background

With the gradual evolution of PCIe standards, PCIe protocols have become the dominant physical layer communication method for peripheral devices and CPU high-speed data exchange. The device has the capability of hot plug (hotplug) and is incorporated in the evolution process of PCIe standard, a series of requirements are set for hardware design of PCIe peripheral equipment, for example, under the condition that external hardware intervention is needed, hardware interrupt is given to ensure that software is terminated to access PCIe peripheral equipment resources in advance, and then hardware unplugging actions are completed. However, this approach is still not secure enough, and in extreme cases may cause a CPU instruction level to hang, corresponding to the Linux system, appearing as a system hang, corresponding to Windows, and we may observe a blue screen. This is determined by PCIe protocol implementation mechanisms and CPU implementation mechanisms, and standard PCIe hot plug cannot completely avoid this phenomenon.

Standard PCIe device communications have two behaviors, post and non-post, respectively. The post is one-way interaction, for example, the host sends data to the slave, and the slave sends data to the host, which are all considered as the post; the non-post indicates that the interaction between the host and the slave is bidirectional, the corresponding operation must be completed, in general, the read operation of the CPU on the PCIe peripheral is the non-post, the instruction corresponding to the CPU is an atomic instruction, the non-post is not detachable, and not all the CPUs design a timeout exception response mechanism for the atomic operation. If the non-post operation is executed when the peripheral is pulled out, the CPU instruction does not continue to execute, so that the system is suspended; even if the CPU designs a timeout response mechanism, there may be a hang up caused by the inability to obtain valid data.

Disclosure of Invention

The embodiment of the invention provides PCIe equipment, a PCIe equipment access method and a PCIe equipment access system, which are used for solving the problem that a host CPU is suspended when the PCIe equipment is pulled out in the related technology.

On one hand, the embodiment of the invention provides an access method of an endpoint device, which is characterized in that a DMA controller is preset in the endpoint device, and the DMA controller is used for writing data into a CPU of a host according to control information stored in a preset DMA controller register; the method comprises the following steps:

The host CPU writes control information of data to be read into a DMA controller register arranged in the endpoint device;

and the DMA controller writes the data to be read of the endpoint device into the host CPU according to the control information.

In some embodiments, the writing control information of data to be read into a DMA controller register set in an endpoint device by a host cpu includes the steps of:

Applying for a memory segment matched with a corresponding DMA controller register on a root complex side, wherein the memory segment is used for storing the written control information of a host CPU in the corresponding DMA controller register;

writing the control information into a DMA controller register corresponding to the memory segment on the root complex side;

the control information comprises a data source address to be read, a data destination address to be read, a data length to be read and transmission control information, wherein the transmission control information comprises transmission enabling and transmission closing which are used for controlling the DMA controller to start or stop writing data into the CPU.

In some embodiments, when there is a DMA transfer requirement for multiple segments of data, a DMA controller in an endpoint device initializes multiple segments of DMA controller registers;

The method for applying for the memory segment matched with the corresponding DMA controller register on the root complex side comprises the following steps:

applying for a plurality of data spaces corresponding to the multiple DMA controller registers in the endpoint device on a root complex side, wherein each data space is used for storing a data segment corresponding to DMA transmission;

And on the root complex side, the value of the last member in each data space is led to the address corresponding to the DMA transmission control register of the next section.

In some embodiments, the control information includes transmission status information including transmission not initiated and transmission completed;

The DMA controller writes the data to be read of the endpoint device into the host CPU according to the control information, and the method comprises the following steps:

the DMA controller writes the data to be read into a destination address corresponding to the root complex side according to the control information;

And when the transmission is completed, the DMA controller modifies the transmission state information in the DMA controller register in the endpoint device to be completed and reports the completion to the host CPU.

In some embodiments, the method comprises:

the host CPU judges whether the to-be-accessed endpoint equipment works abnormally according to the transmission state information acquired by the root complex side from the DMA controller of the to-be-accessed endpoint equipment, and sends out a prompt of the abnormal working of the endpoint equipment when judging the abnormal working.

In a second aspect, an embodiment of the present invention provides an endpoint device, where a DMA controller is preset in the endpoint device, where the DMA controller is configured to write data to a host CPU according to control information stored in a preset DMA controller register, where the control information is written by a host CPU to the DMA controller register according to a need of reading the data.

In a third aspect, an embodiment of the present invention further provides an endpoint device access system, which is characterized in that the system includes:

A host cpu, configured to write control information of data to be read into a DMA controller register set in the endpoint device when accessing the endpoint device;

And the endpoint equipment is internally preset with a DMA controller, and the DMA controller is used for writing data to be read into the host CPU according to control information stored in a preset DMA controller register.

In some embodiments, the host cpu is configured to:

In some embodiments, the endpoint device is further to:

Initializing a multi-segment DMA controller register and storing control information required by transmission of corresponding data segments in each segment DMA controller register when DMA transmission requirements of the multi-segment data exist;

the host cpu is further configured to:

the DMA controller is configured to:

writing the data to be read into a destination address corresponding to the root complex side according to the control information;

After the transmission is completed, the DMA controller modifies the transmission state information in a DMA controller register in the endpoint device to be completed and reports the completion to the host CPU;

the host cpu is further configured to:

Judging whether the to-be-accessed endpoint equipment works abnormally according to the transmission state information acquired by the root complex side from the DMA controller of the to-be-accessed endpoint equipment, and sending out a prompt of the endpoint equipment working abnormality when judging that the working abnormality.

The beneficial effects of the embodiment of the invention include:

Because of the preset DMA controller in the endpoint device, the host CPU does not access the slave computer in a non-post bidirectional interaction mode, but writes the control information of the data to be read into the DMA controller in a post unidirectional interaction mode, so as to realize the access to the endpoint device. Hot plug of the endpoint device can be realized without intervention of external hardware (such as plug board pieces). The CPU cannot be suspended due to the fact that non-post mode interaction is adopted when the external PCIe equipment is pulled up, and the external endpoint equipment is in a problem in extreme cases in time, so that the CPU cannot be suspended, the problems of dead halt, blue screen and the like are avoided, hardware design is simplified, and meanwhile, system reliability is improved. In addition, under the big data throughput scene, the host cpu finishes the read operation through the DMA controller of the endpoint device, and because the PCIe protocol design is friendly to big data space equipment, the actual data acquisition efficiency is higher than that of direct reading according to 32-bit data for many times, and the efficiency is improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic flow chart of an endpoint device access method provided by an embodiment of the present invention;

FIG. 2 is a schematic diagram of an endpoint device access system according to an embodiment of the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

As shown in FIG. 1, the embodiment of the invention provides an access method of an endpoint device, wherein a DMA controller is preset in the endpoint device, and the DMA controller is used for writing data into a host CPU according to control information stored in a preset DMA controller register; the method comprises the following steps:

S100, a host CPU writes control information of data to be read into a DMA controller register arranged in an endpoint device;

and S200, the controller writes the data to be read of the endpoint equipment into the host CPU according to the control information.

It should be noted that, the DMA controller is a DMA (Direct Memory Access ) controller; the Endpoint device includes a DMA ctrl (DMA controller), a DMA mem (DMA controller register for holding configuration DATA of the DMA controller), and a DATA mem (EP subsystem memory unit for storing DATA of all EP sides).

It will be appreciated that prior to step S100, when the host cpu discovers an endpoint device, specific information (including data length and address, etc.) of the endpoint configuration space and data space is obtained using the standard endpoint device enumeration method in the kernel.

According to the embodiment of the invention, because of the DMA controller preset in the endpoint device, the host CPU does not access the slave computer in a non-post type bidirectional interaction mode, and writes the control information of the data to be read into the DMA controller in a post type unidirectional interaction mode, so that the access to the endpoint device is realized. The hot plug of the endpoint device can be realized without the intervention of external hardware (such as a plug board piece). The CPU can not be suspended due to the fact that non-post mode interaction is adopted when the external endpoint equipment is pulled up, problems occur to the external PCIe equipment under extreme conditions in time, the CPU can not be suspended, the problems of dead halt, blue screen and the like are avoided, hardware design is simplified, and meanwhile reliability of the system is improved. In addition, under the big data throughput scene, the host cpu finishes the read operation through the DMA controller of the endpoint device, and because the PCIe protocol design is friendly to big data space equipment, the actual data acquisition efficiency is higher than that of direct reading according to 32-bit data for many times, and the efficiency is improved.

In some embodiments, step S100 includes:

S110, applying for a memory segment matched with a corresponding DMA controller register on a root complex side, wherein the memory segment is used for storing the written control information of a host CPU in the corresponding DMA controller register;

s120, writing the control information into a DMA controller register corresponding to a memory segment on the root complex side;

The control information of the data to be read comprises a source address of the data to be read, a destination address of the data to be read, the length of the data to be read and transmission control information; the transfer control information includes transfer enable and transfer disable, which are used to control the DMA controller to start or stop transferring data.

It should be noted that, the root complex (abbreviated as RC or RC) is a hub of the host cpu and the slave device, and may specifically select a function of supporting different inter-domain message routing, where the host cpu further includes a kernel, a memory unit mem, and so on.

It can be understood that, in the content stored in the DMA controller register, the source address of the data to be read refers to the source address of the data to be read on the endpoint device side, the destination address of the data to be read refers to the destination address where the data to be read is finally stored on the root complex side, and the length of the data to be read refers to the length of the data to be read by a DMA read operation.

When the RC side needs to read the data of the endpoint side, the source address, the destination address and the data length of the data to be read can be written into the DMA controller register, and when transmission enabling (which can be an enabling bit of transmission control information) is written, the DMA controller starts data transmission to the host side correspondingly.

It can be appreciated that in step S110, the application on the root complex side for the memory segment space matching with the DMA controller register in the slave endpoint device is convenient for buffering and managing data. The buffer setting of the data depends on the actual data space size of the EP and the performance requirements and is not strictly limited. The management space that works with the DMA controller may contain the following pseudo code:

struct{

ARCH_POINTER*host_cfg_addr；

ARCH_POINTER*srcaddr；

ARCH_POINTER*distaddr；

int datalen；

int transfer_status；

int transfer_ctrl；

void*next；

}EP_DMACTL；

The ARCH_ POINTERd is the instruction bit width change of the CPU, if the CPU is 64 bits, the address is 64 bits, and if the CPU is 32 bits, the address is 32 bits; since PCIe protocols are currently accessed with 32-bit data bit width, datalen uses the int type; transfer_status represents the transfer status bit, filled in by the DMA controller at the EP side after the DMA operation is completed.

In some embodiments, when there is a DMA transfer of multi-segment data, the DMA controller in the endpoint initializes the multi-segment DMA controller registers;

Step S110 includes the steps of:

S111, applying a plurality of data spaces corresponding to the multiple DMA controller registers in the endpoint device on a root complex side, wherein each data space is used for storing a data segment corresponding to DMA transmission;

s112, on the root complex side, the value of the last member in each data space is led to the address corresponding to the DMA transmission control register of the next segment;

In this embodiment, considering the case of multi-segment DMA transfer, the host cpu applies for a plurality of data spaces on the root complex side for accommodating data required for the multi-segment DMA transfer. For the case of multi-segment DMA transmission, when control information is written from RC side to endpoint, the writing can be simplified to 1 time, and the data reading efficiency is improved.

Specifically, the expression allows initializing multiple DMA control segments according to the void x and next used in the pseudo code segment structure, thus having the capability of multi-segment DMA operation.

In some embodiments, step S200 includes the steps of:

S210, the DMA controller writes the data to be read into a destination address corresponding to the root complex side according to the control information;

S220, after the transmission is completed, the DMA controller modifies the transmission state information in the DMA controller register in the endpoint device to be the transmission completion and reports the transmission state information to the host CPU.

It should be noted that the control information includes transmission status information, where the transmission status information is used to reflect a status of data transmitted by the DMA controller, and includes that transmission is not started and transmission is completed. In actual operation, the status bits of the transmission status information may be respectively represented by 0 and 1 (0 indicates that the transmission is not started, and 1 indicates that the transmission is completed), and the DMA controller may modify the transmission status according to the actual transmission situation.

Meanwhile, it can be understood that the control information further includes a host configuration address, which represents an address on the root complex side for storing data in a corresponding DMA controller register; in step S220, the DMA controller first modifies the transmission status information in the DMA controller register of the endpoint, calculates the transmission status bit offset of the root complex side according to the DMA controller register structure and the host configuration address, executes an independent DMA transmission operation, and writes the transmission status information indicating the completion of the transmission into the corresponding address of the root complex side.

In a specific implementation, specific instructions (pseudo code) include:

Write(*ep_dmactl_src_addr,ep_data_addr,addrlen)；

Write(*ep_dmactl_dist_addr,cpu_data_addr,addrlen)；

Write(*ep_dmactl_rdlen_addr,datalen,addrlen)；

Write(*ep_dmactl_transfer,START,sizeof(START))；

in actual operation, the structure configuration data is written once using memset as:

struct EP_DMA_CTL*ep_dma_ctl；

ep_dma_ctl->host_cfg_addr＝host_cfg_addr；

ep_dma_ctl->srcaddr＝ep_data_addr；

ep_dma_ctl->distaddr＝cpu_data_addr；

ep_dma_ctl->datalen＝datalen

ep_dma_ctl->transfer_ctrl＝START；

ep_dma_ctl->transfer_status＝TRANSFERING；

memcpy(ep_dma_ctl,ep_dmactl_addr,(sizeof(struct EP_DMA_CTL)))；

The host_cfg_addr refers to the management space (the memory segment address matched with the corresponding DMA controller register and applied on the root complex side) applied in the cpu subsystem memory and used for cooperating with the DMA controller;

After the DMA controller in the Endpoint device completes operation, the transfer_status offset is superimposed with the host_cfg_addr as the base address according to the configuration data in ep_ dmactl, and the transfer state is set as DONE.

The atomic operation of read is changed to multiple instruction operations, and is all write operation, with no read operation.

Based on data processing efficiency requirements, the DMA data may be allowed to be divided into multiple slices and transferred multiple times.

In some embodiments, the method further comprises:

S300, the host CPU judges whether the to-be-accessed endpoint equipment works abnormally according to the transmission state information acquired from the DMA controller of the to-be-accessed endpoint equipment at the root complex side, and sends out a prompt of the abnormal work of the endpoint equipment when judging the abnormal work.

As can be appreciated, an endpoint device operational exception includes multiple failures, hot-pull actions, or failures to initiate a DMA operation. When the endpoint has the abnormal working conditions, the host cpu can identify and judge the transmission state and the time of the normal data to be transmitted by comparing. After the upper software obtains the end point equipment work abnormality prompt sent by the host cpu, the end point equipment work abnormality condition can be further diagnosed or unloaded according to the requirement.

As shown in FIG. 2, an embodiment of the present invention provides an endpoint device (ep subsystem) in which a DMA controller (DMA ctrl) is preset, the DMA controller is used for writing data to a host CPU according to control information stored in a preset DMA controller register, and the control information is written to the DMA controller register by a host CPU according to the need of reading the data.

Wherein RC (RC) refers to root complex (hub of host cpu and slave device, which can support different inter-domain message routing according to specific selection), and DMA controller is DMA (Direct Memory Access ) controller; the host cpu further comprises a kernel and a memory unit mem; the endpoint (ep subsystem) of the PCIe device also includes a DMA ctrl (DMA controller), a DMA mem (DMA controller register for holding configuration DATA of the DMA controller), a DATA mem (ep subsystem memory unit for storing DATA of all ep subsystem sides).

It should be noted that there are various ways to preset DMA controllers for an endpoint device, including a way to implement it through logic devices or configure an embedded processor as an endpoint, which can implement the ep subsystem in this embodiment. The construction of an endpoint device with a DMA controller using Programmable logic devices can be implemented based on FPGA (Field-Programmable gate array) of manufacturer Altera and Xilinx. Instead, using an embedded processor that can flexibly configure PCIe controllers (either as root complexes or as endpoint), it may be set as an endpoint device instead of a root complex in this embodiment, and using the DMA controller of the CPU, a dedicated DMA control driver may be developed according to the embodiment.

It will be appreciated that an endpoint device with a pre-configured DMA controller, constructed in either of the above-described ways, has features including: a certain address space (CFG MEM as shown in fig. 1 for holding the configuration data of the DMA controller) and a completely exclusive DMA controller (for performing DMA read actions based on the data written in CFG MEM).

By adopting the endpoint device of the embodiment, the host cpu accesses the slave computer not in a non-post bidirectional interaction mode but writes the control information of the data to be read into the DMA controller in a post unidirectional interaction mode through the DMA controller preset in the endpoint device, so as to realize the access to the endpoint device. Hot plug of the endpoint device can be realized without intervention of external hardware (such as plug board pieces). The CPU can not be suspended due to the non-post mode interaction when the external endpoint equipment is pulled up, so that the problem of the external endpoint equipment in extreme cases is solved, the problems of dead halt, blue screen and the like are avoided, the hardware design is simplified, and the reliability of the system is improved. In addition, under the big data throughput scene, the host cpu finishes the read operation through the DMA controller of the endpoint device, and because the PCIe protocol design is friendly to big data space equipment, the actual data acquisition efficiency is higher than that of direct reading according to 32-bit data for many times, and the efficiency is improved.

As shown in fig. 2, an embodiment of the present invention provides a PCIe device access system, including:

A host cpu for writing control information of data to be read into a DMA controller register provided in an endpoint device when accessing the endpoint device;

Wherein RC (RC) refers to root complex (hub of host cpu and slave device, which can support different inter-domain message routing according to specific selection), and DMA controller is DMA (Direct Memory Access ) controller; the host cpu further comprises a kernel and a memory unit mem; the endpoint (ep subsystem) of the PCIe device also includes a DMA ctrl (DMA controller), a DMA mem (DMA controller register for holding configuration DATA of the DMA controller), a DATA mem (ep subsystem memory unit for storing DATA of all ep subsystems).

In some embodiments, the host cpu is configured to:

Writing control information into a DMA controller register corresponding to a memory segment on the root complex side;

In some embodiments, the endpoint device is also used to:

when there is a DMA transfer demand for the pieces of data, initializing the pieces of DMA controller registers and storing control information required for transfer of the corresponding pieces of data in each piece of DMA controller registers.

The host cpu is also configured to:

The DMA controller is configured to:

The host cpu is also configured to:

judging whether the to-be-accessed endpoint device is abnormal in operation or not according to the transmission state information acquired by the root complex side from the DMA controller of the to-be-accessed endpoint device, and in some embodiments of prompting that the endpoint device is abnormal in operation when judging that the operation is abnormal, the host CPU is used for:

judging whether the PCIe device endpoint to be accessed is abnormal in work or not according to the transmission state information acquired by the root complex side from the DMA controller of the PCIe device endpoint to be accessed, and sending out a prompt of the PCIe device endpoint work abnormality when judging the work abnormality.

Those of ordinary skill in the art will appreciate that all or some of the steps of the methods, systems, functional modules/units in the devices disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between the functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed cooperatively by several physical components. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer-readable storage media, which may include computer-readable storage media (or non-transitory media) and communication media (or transitory media).

The foregoing is only a specific embodiment of the invention to enable those skilled in the art to understand or practice the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An access method of an endpoint device is characterized in that a DMA controller is preset in the endpoint device, and the DMA controller is used for writing data into a host CPU according to control information stored in a preset DMA controller register; the method comprises the following steps:

the DMA controller writes the data to be read of the endpoint device into the host CPU according to the control information;

The host cpu writes control information of data to be read into a DMA controller register built in an endpoint device, including the steps of:

2. The method of accessing an endpoint device of claim 1,

When DMA transmission requirements of multiple segments of data exist, a DMA controller in the endpoint device initializes multiple segments of DMA controller registers;

3. The method of accessing an endpoint device of claim 1,

The control information comprises transmission state information, wherein the transmission state information comprises transmission non-starting and transmission completion;

4. The endpoint device access method of claim 3, wherein the method comprises:

5. An endpoint device is characterized in that a DMA controller is preset in the endpoint device, the DMA controller is used for writing data into a host CPU according to control information stored in a preset DMA controller register, and the control information is written into the DMA controller register corresponding to a memory segment on the root complex side by a host CPU according to a memory segment which is applied on the root complex side and is matched with the corresponding DMA controller register.

6. An endpoint device access system, comprising:

The host CPU is used for applying for a memory segment matched with a corresponding DMA controller register on the root complex side, and the memory segment is used for storing the control information written by the host CPU in the corresponding DMA controller register; writing the control information into a DMA controller register corresponding to the memory segment on the root complex side; the control information comprises a data source address to be read, a data destination address to be read, a data length to be read and transmission control information, wherein the transmission control information comprises transmission enabling and transmission closing which are used for controlling a DMA controller to start or stop writing data into a CPU;

7. The endpoint device access system of claim 6,

The endpoint device is also to:

the host cpu is further configured to:

8. The endpoint device access system of claim 7,

the DMA controller is configured to:

the host cpu is further configured to: