CN116225315A - Broadband data high-speed recording system, storage architecture and method based on PCI-E optical fiber card - Google Patents

Abstract

The invention discloses a broadband data high-speed recording system, a storage architecture and a method based on a PCI-E optical fiber card, wherein the system comprises a plurality of distributed storage servers and an Infiniband switch; the Infiniband network switch is respectively connected with the distributed storage servers and used for data transmission; the distributed storage server comprises an optical fiber card and a RAID card and is used for providing distributed storage service; the optical fiber card comprises an FPGA module and is used for receiving high-speed data streams output by the high-speed AD chip. The invention adopts the optical fiber card based on PCIE3.0 bus, solves the problems of receiving and buffering broadband high-speed AD data, and can reach the transmission rate of more than 4 GB/s. A plurality of universal servers are adopted to construct distributed storage, so that the problem of broadband high-speed AD data storage is solved, the 10GB/s write disk speed is provided, and broadband data high-speed recording is realized.

Description

Broadband data high-speed recording system, storage architecture and method based on PCI-E optical fiber card

Technical Field

The invention relates to a high-speed recording method. And more particularly, to a PCI-E fiber optic card based broadband data high-speed recording system, storage architecture and method.

Background

The current requirements of complex battlefield environments and electronic countermeasure require that the radar not only can measure distance, speed and angle information of a target, but also can image and identify the target, and has anti-interference capability. The radar system can be assisted to optimize an imaging algorithm, extract target feature information to form a target feature library and study an anti-interference algorithm by carrying out post-fine analysis after broadband data recording. In response to the above requirements, the bandwidth of the radar signal is continuously improved, and the maximum bandwidth at present exceeds 5GHz. Thus, high-speed recording of broadband data is highly required.

The main technical difficulties to be solved in the high-speed recording of broadband data are as follows: high-speed AD data receiving and caching, and high-speed data storage. The current high-speed AD data receiving and caching is mainly realized by using a high-performance optical fiber data acquisition card, the highest continuous transmission rate is difficult to exceed 2GB/s based on a PCIE2.0 bus, and the transmission rate requirement of more than 4GB/s at present cannot be met. At present, the high-speed data storage is mainly realized by using a universal server and a RAID disk array, and the writing speed which can be supported by a single RAID cannot meet the speed requirement of more than 4GB/s at present.

Accordingly, there is a need to provide a PCI-E fiber optic card based broadband data high-speed recording system, storage architecture and method.

Disclosure of Invention

The invention aims to provide a broadband data high-speed recording system, a storage architecture and a method based on a PCI-E optical fiber card, which solve at least one of the problems.

In order to achieve the above purpose, the invention adopts the following technical scheme:

a broadband data high-speed recording system based on PCI-E optical fiber card comprises a plurality of distributed storage servers and an Infiniband network switch;

the Infiniband network switch is respectively connected with the distributed storage servers and used for data transmission;

the distributed storage server comprises an optical fiber card and a RAID card and is used for providing distributed storage service;

the optical fiber card comprises an FPGA module and is used for receiving high-speed data streams output by the high-speed AD chip.

Preferably, the storage space of the optical fiber card comprises a DDR4 space hung outside the FPGA and a BRAM space inside the FPGA;

the BRAM space includes REG group space and RAM space.

Preferably, the DDR4 space is used for storing broadband data received by an optical fiber;

the REG group space is used for storing control register values and parameter configuration register values acquired by each fiber channel;

the RAM space is used to store broadband data buffer descriptors.

Preferably, the optical fiber card further comprises:

the AXIS interface is used for outputting the broadband acquisition data to the S2MM module;

the S2MM module is used for realizing the mapping from AXI flow to memory and writing the acquired broadband AD data into a preset DDR4 buffer area;

the SGDMA module is used for moving the data of the DDR4 buffer area to the acquisition buffer area of the user;

the Register Files Register module is used for configuring and controlling the S2MM module and returning the state of the S2MM module;

the inter Interrupt module is used for forwarding the Interrupt request of the S2MM and sending MSI Interrupt to the CPU;

and the PCI-E module is used for realizing PCIe protocol specification and transmitting the broadband data to the CPU memory.

Preferably, the FPGA further comprises an XDMA module and a user logic register, where the user logic register is used to implement control and parameter configuration of the distributed storage server on the optical fiber acquisition card.

Preferably, the XDMA module comprises an AXI4 bus, and is used for controlling reading and writing of DDR4 outside the FPGA; and the PCIE AXI4-Lite bus is used for accessing the user logic register or the BRAM module equipment.

Preferably, a plurality of RAID cards are installed in each storage server, and an N+M distributed RAID mechanism is adopted.

The distributed storage architecture of the PCI-E optical fiber card-based broadband data high-speed recording system comprises a physical resource layer, a persistence layer, an access protocol layer, an application layer and a platform management layer, wherein the physical resource layer comprises a plurality of distributed storage servers and is used for providing distributed storage services;

the persistence layer is used for supporting a multi-copy and erasure code data redundancy mechanism;

the access protocol layer is used for providing a block storage protocol accessible to the iSER client;

the application layer is used for providing broadband data high-speed write disc and NFS shared network disc service;

the platform management layer is used for carrying out storage management.

A high-speed broadband data recording method based on PCI-E optical fiber card according to the system comprises the following steps:

installing an optical fiber card and a RAID card in a general server to form a distributed storage server;

receiving a high-speed data stream output by a high-speed AD chip through an FPGA module in the optical fiber card, carrying out high-speed data caching through a plug-in high-capacity DDR4 chip, and transmitting the cached data to a CPU memory of a server through PCIE;

and constructing a distributed storage space by using a plurality of universal servers to realize high-speed recording of broadband data.

Preferably, the method further comprises:

opening PCI-E optical fiber card character equipment;

allocating space in the memory of the storage server to store broadband data;

creating and opening a broadband data recording file;

configuring the size of a data packet received each time, and enabling data interruption;

starting broadband data receiving, reading a broadband data packet from DDR4 to a memory space of a storage server after receiving a data interrupt, updating a DDR4 receiving data address pointer to write into a pre-created recording file, and repeating the step to wait for receiving the next data;

stopping broadband data recording after the data receiving is finished, and prohibiting interruption and stopping memory data transmission;

closing the broadband data recording file;

closing the character device, closing the thread and releasing the mapping memory space.

The beneficial effects of the invention are as follows:

the optical fiber data acquisition card based on PCIE3.0 bus is adopted, the problems of broadband high-speed AD data receiving and buffering are solved, and the transmission rate can reach more than 4 GB/s. A plurality of universal servers are adopted to construct distributed storage, so that the problem of broadband high-speed AD data storage is solved, the 10GB/s write disk speed is provided, and broadband data high-speed recording is realized.

Drawings

The following describes the embodiments of the present invention in further detail with reference to the drawings.

FIG. 1 shows a schematic diagram of a broadband data high-speed recording system based on a PCI-E optical fiber card.

FIG. 2 illustrates a schematic diagram of a PCI-E fiber optic card framework.

FIG. 3 shows a schematic diagram of a PCI-E fiber optic card FPGA design.

Fig. 4 shows a BRAM space division schematic inside an FPGA.

Fig. 5 shows a schematic diagram of a distributed storage architecture.

Detailed Description

In order to more clearly illustrate the present invention, the present invention will be further described with reference to preferred embodiments and the accompanying drawings. Like parts in the drawings are denoted by the same reference numerals. It is to be understood by persons skilled in the art that the following detailed description is illustrative and not restrictive, and that this invention is not limited to the details given herein.

As shown in fig. 1-5, the invention provides a broadband data high-speed recording system, a storage architecture and a method based on a PCI-E optical fiber card, which adopt an optical fiber data acquisition card based on a PCIE3.0 bus, receive a high-speed data stream output by a high-speed AD chip through a high-performance FPGA, cache the high-speed data through an externally-hung high-capacity DDR4 chip, and transmit the cached data to a CPU memory of a server through PCIE, so that the transmission rate can reach more than 4 GB/s. The optical fiber data acquisition card is arranged in a universal server, 3 RAID cards are configured in the server, 3 universal servers are utilized to construct a distributed storage space, the 10GB/s write disk speed is provided, and the high-speed recording of broadband data is realized.

1-5, taking three distributed storage service areas and each distributed storage service area including three RAID cards as an example, one embodiment of the present invention provides a broadband data high-speed recording system based on PCI-E optical fiber cards, which includes three distributed storage servers and an Infiniband network switch;

the Infiniband network switch is respectively connected with the three distributed storage servers and is used for data transmission;

the distributed storage server comprises an optical fiber data acquisition card and a RAID card and is used for providing distributed storage service;

the optical fiber data acquisition card comprises an FPGA module and is used for receiving high-speed data streams output by the high-speed AD chip.

Specifically, in the system, the PCI-E optical fiber card is arranged in a general server, a 12-way optical fiber interface is adopted to realize the receiving of high-speed AD data, and the high-speed AD data is transmitted to the CPU memory of the general server through the PCI-E interface.

And 3 distributed storage servers are constructed by adopting a general server, a high-speed distributed unified storage resource pool is constructed by deploying distributed storage software, multiple paths of optical fiber data access are processed in parallel, N/M (erasure code) data redundancy protection among servers is supported, and the requirement of high-bandwidth and high-concurrency file recording is met.

The high-speed Infiniband network switch is used for connecting 3 distributed storage servers and providing high-bandwidth and low-delay data transmission service.

In an alternative embodiment, the storage space of the optical fiber card comprises a DDR4 space hung on the FPGA and a BRAM space inside the FPGA;

the BRAM space includes REG group space and RAM space.

In an alternative embodiment, the DDR4 space is used to store broadband data received by the optical fiber;

the REG group space is used for storing control register values and parameter configuration register values acquired by each fiber channel;

the RAM space is used to store broadband data buffer descriptors.

Specifically, the design of the PCI-E optical fiber card storage space mainly comprises a large-capacity DDR4 space hung on the FPGA and a BRAM space inside the FPGA, wherein the DDR4 space is used for storing broadband data received by an optical fiber.

The BRAM space is a memory mapping space in the FPGA and is divided into a REG group space and a RAM space, the REG group stores a control register value and a parameter configuration register value acquired by each fiber channel, and the RAM space stores DDR4 cache addresses and sizes of each fiber channel data packet, namely a broadband data buffer descriptor. The BRAM space division inside the FPGA is shown in fig. 3.

In an alternative embodiment, the fiber card further comprises:

the AXIS interface is used for outputting the broadband acquisition data to the S2MM module;

the S2MM module is used for realizing the mapping from AXI flow to memory and writing the acquired broadband AD data into a preset DDR4 buffer area;

the SGDMA module is used for moving the data of the DDR4 buffer area to the acquisition buffer area of the user;

the Register Files Register module is used for configuring and controlling the S2MM module and returning the state of the S2MM module;

the inter Interrupt module is used for forwarding the Interrupt request of the S2MM and sending MSI Interrupt to the CPU;

and the PCI-E module is used for realizing PCIe protocol specification and transmitting the broadband data to the CPU memory.

Specifically, the FPGA system architecture of the PCI-E optical fiber card in the method is shown in FIG. 2. And realizing a PCIe bus control interface by using the XDMA IPCORE design of the FPGA of the Xilinx company. The broadband data source outputs broadband acquisition data to the S2MM module with an AXIS interface.

The S2MM module realizes AXI flow to memory mapping and is responsible for writing acquired broadband AD data into a preset DDR4 buffer area.

SGDMA is responsible for moving DDR4 buffer data to the user's acquisition buffer. The Register Files Register module is used to configure and control the S2MM modules and return the state of these modules.

The Interrupt module forwards the Interrupt request of the S2MM and sends MSI Interrupt to the CPU.

PCI-E module realizes PCIe protocol specification, and transmits broadband data to CPU memory. The PCIe fiber card in the server is driven, WDF framework development is adopted in Windows system development, and character equipment is adopted in Linux system development. Finally, the user application program accesses the acquisition buffer area to obtain broadband acquisition data.

In an alternative embodiment, the FPGA further includes an XDMA module and a user logic register for implementing control and parameter configuration of the fiber acquisition card by the distributed storage server.

In an alternative embodiment, the XDMA module includes an AXI4 bus for read-write control of DDR4 external to the FPGA; and

and the PCIE AXI4-Lite bus is used for accessing the user logic register or the BRAM module equipment.

Specifically, as shown in fig. 3, the XDMA module is generated by a DMA Subsystem for PCIExpress IP core. And the broadband data is received by connecting the PCIE AXI4 bus with an external optical fiber module. The AXI4 bus is connected to the MIG to realize read-write control of DDR4 outside the FPGA and is used for high-speed and large-capacity caching of optical fiber data.

User logic register or Block RAM Block memory module equipment access is carried out through PCIE AXI4-Lite bus, wherein the user logic register is used for realizing control and parameter configuration of a storage server to an optical fiber acquisition card, and the Block RAM is used for realizing descriptors (storing address and length information of a cache data packet in DDR 4)

In an alternative embodiment, a 3-block RAID card is installed within each storage server and an N+M distributed RAID mechanism is employed.

Specifically, the 3 RAID cards are installed in each storage server to improve the write disk bandwidth of a single storage server, support copy and erasure code redundancy protection, and an N+M distributed RAID mechanism is used for improving the effective storage utilization rate, the scale can be transversely expanded, and all-solid-state SSD resources of a plurality of storage servers form a high-performance, high-availability and easily-expanded block storage resource pool through a high-speed Infiniband network

The invention provides a distributed storage architecture of the PCI-E optical fiber card-based broadband data high-speed recording system, which comprises a physical resource layer, a persistence layer, an access protocol layer, an application layer and a platform management layer, wherein the physical resource layer comprises three distributed storage servers and is used for providing distributed storage services;

the persistence layer is used for supporting a multi-copy and erasure code data redundancy mechanism;

the access protocol layer is used for providing a block storage protocol accessible to the iSER client;

the application layer is used for providing broadband data high-speed write disc and NFS shared network disc service;

the platform management layer is used for carrying out storage management.

Specifically, as shown in fig. 4, the distributed storage architecture in the method mainly includes a physical resource layer, a persistence layer, an access protocol layer, an application layer and a platform management layer.

The physical resource layer provides 3 storage servers. The persistence layer supports data redundancy mechanisms such as multiple copies, erasure codes and the like. An access protocol layer provides a block storage protocol accessible to iSER clients. The application layer provides broadband data high-speed write disk and NFS shared network disk services. The platform management layer performs storage management based on the CLI command line and supports log management, performance monitoring, fault monitoring and disk management.

The distributed storage system is mainly used for providing distributed storage service by 3 storage servers, adopts an all-solid-state SSD storage disk hardware platform, improves the disk writing bandwidth of a single storage server by installing 3 RAID cards in each storage server, supports copy and erasure code redundancy protection, improves the effective storage utilization rate by an N+M distributed RAID mechanism, can transversely expand the scale, forms a high-performance, high-availability and easily-expanded block storage resource pool by all-solid-state SSD resources of a plurality of storage servers through a high-speed Infiniband network, externally provides ultra-high-performance storage service through an iSER storage interface protocol, and supports development and operation of high-speed recording application software.

Another embodiment of the present invention provides a method for high-speed recording broadband data based on a PCI-E optical fiber card according to the system, comprising the steps of:

installing an optical fiber card and a RAID card in a general server to form a distributed storage server;

receiving a high-speed data stream output by a high-speed AD chip through an FPGA module in the optical fiber card, carrying out high-speed data caching through a plug-in high-capacity DDR4 chip, and transmitting the cached data to a CPU memory of a server through PCIE;

and constructing a distributed storage space by using three universal servers, so as to realize high-speed recording of broadband data.

In an alternative embodiment, the method further comprises:

opening PCI-E optical fiber card character equipment;

allocating space in the memory of the storage server to store broadband data;

creating and opening a broadband data recording file;

configuring the size of a data packet received each time, and enabling data interruption;

starting broadband data receiving, reading a broadband data packet from DDR4 to a memory space of a storage server after receiving a data interrupt, updating a DDR4 receiving data address pointer to write into a pre-created recording file, and repeating the step to wait for receiving the next data;

stopping broadband data recording after the data receiving is finished, and prohibiting interruption and stopping memory data transmission;

closing the broadband data recording file;

closing the character device, closing the thread and releasing the mapping memory space.

In a specific embodiment, the implementation of high-speed broadband data recording based on PCI-E fiber optic cards comprises the following specific steps:

and step 1, designing and realizing the FPGA in the PCI-E optical fiber card. An address space allocation table of the internal register of the FPGA is shown in table 1; the address space allocation table of the internal RAM of the FPGA is shown in table 2.

As shown in FIG. 3, the XDMA modules are generated by DMA Subsystem for PCIExpress IP cores. And the broadband data is received by connecting the PCIE AXI4 bus with an external optical fiber module. The AXI4 bus is connected to the MIG to realize read-write control of DDR4 outside the FPGA and is used for high-speed and large-capacity caching of optical fiber data.

And accessing a user logic register or a Block RAM Block memory module device through the PCIE AXI4-Lite bus, wherein the user logic register is used for realizing control and parameter configuration of a storage server to the optical fiber acquisition card, and the Block RAM is used for realizing descriptors (storing address and length information of a cache data packet in DDR 4).

And 2, designing and realizing a PCI-E optical fiber card driving API function.

The XDMA kernel driver packages PCIE DMA into a plurality of character devices, such as xdma_user, xdma_c2h. The application layer can access the axilite interface of the XDMA through the xdma_user device file, thereby operating the user logic register. The application layer can copy the broadband data of DDR4 cached outside the FPGA to the memory space of the storage server through the xdma_c2h. The application layer may be used to read user interrupts through xdma_events and obtain broadband data from DDR4 when a data interrupt occurs.

Step 3, designing and implementing a distributed storage system with reference to fig. 5. And designing according to 4 layers of a persistence layer, an access protocol layer, an application layer and a platform management layer, and testing the read-write performance by using a fio performance testing tool.

And 4, designing broadband data recording application software.

Application API functions such as XPcie_Open, XPcie_write_reg, XPcie_read_reg, XPcie_Init_C2h, XPcie_UnInit_C2h and XPcie_Dma_C2h are designed and developed.

XPcie_Open is used for opening character devices such as xmda_user, xdma_c2h, xdma_events and the like. XPcie_Write_Reg is used to Write user logical registers or RAM space. XPcie_Read_Reg is used to Read user logical register space or RAM space. XPcie_Init_C2h is used for initial parameter configuration: stopping DMA, configuring the size of the data packet, interruption, threshold, closing interruption and soft reset. XPcie_UnInit_C2h is used to release C2h, prohibit interrupt, stop DMA. XPcie_Dma_C2h is used to run 1 S2MM_DMA operation, read broadband data packets from DDR4 to storage server memory space and write into disk array. XPcie_Close is used to shut down character devices, shut down threads, stop interrupts, and free up mapped memory space.

The high-speed broadband data recording working steps based on the PCI-E optical fiber card are as follows:

and step 1, calling an XPcie_Open function, and opening the PCI-E optical fiber card character equipment.

And step 2, allocating space in the memory of the storage server for storing the broadband data.

And step 3, creating and opening the broadband data recording file.

And 4, calling an XPcie_Init_C2h function, soft resetting the PCI-E optical fiber card, configuring the size of a data packet received each time, and enabling data interruption.

And 5, calling an XPcie_Dma_C2h function, starting broadband data receiving, reading a broadband data packet from the DDR4 to a memory space of a storage server after receiving a data interrupt, calling the XPcie_write_Reg to update a DDR received data address pointer, and then writing the data address pointer into a previously created admission file.

Step 6, jumping to step 5, and waiting for receiving the next data.

Step 7, calling XPcie_UnInit_C2h to stop broadband data recording, and prohibiting interruption and stopping DMA

And 8, closing the broadband data recording file.

And 9, calling an XPcie_close function, closing character equipment, closing threads and releasing mapping memory space.

Table 1 address space allocation table for FPGA internal registers

Table 2 address space allocation table for internal RAM of FPGA

It should be understood that the foregoing examples of the present invention are provided merely for clearly illustrating the present invention and are not intended to limit the embodiments of the present invention, and that various other changes and modifications may be made therein by one skilled in the art without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims (10)

1. The broadband data high-speed recording system based on the PCI-E optical fiber card is characterized by comprising a plurality of distributed storage servers and an Infiniband network switch;

the Infiniband switch is respectively connected with the distributed storage servers and is used for data transmission;

the distributed storage server comprises an optical fiber card and a RAID card and is used for providing distributed storage service;

the optical fiber card comprises an FPGA module and is used for receiving high-speed data streams output by the high-speed AD chip.

2. The broadband data high-speed recording system according to claim 1, wherein the storage space of the optical fiber card comprises a DDR4 space externally hung in the FPGA and a BRAM space inside the FPGA;

the BRAM space includes REG group space and RAM space.

3. The broadband data high-speed recording system according to claim 2, wherein the DDR4 space is used for storing broadband data received by an optical fiber;

the REG group space is used for storing control register values and parameter configuration register values acquired by each fiber channel;

the RAM space is used to store broadband data buffer descriptors.

4. The broadband data high-speed recording system of claim 2, wherein the fiber optic card further comprises:

the AXIS interface is used for outputting the broadband acquisition data to the S2MM module;

the S2MM module is used for realizing the mapping from AXI flow to memory and writing the acquired broadband AD data into a preset DDR4 buffer area;

the SGDMA module is used for moving the data of the DDR4 buffer area to the acquisition buffer area of the user;

the Register Files Register module is used for configuring and controlling the S2MM module and returning the state of the S2MM module;

the inter Interrupt module is used for forwarding the Interrupt request of the S2MM and sending MSI Interrupt to the CPU;

and the PCI-E module is used for realizing PCIe protocol specification and transmitting the broadband data to the CPU memory.

5. The broadband data high-speed recording system according to claim 2, wherein the FPGA further comprises an XDMA module and a user logic register, the user logic register being configured to implement control and parameter configuration of the optical fiber acquisition card by the distributed storage server.

6. The broadband data high-speed recording system according to claim 5, wherein the XDMA module comprises an AXI4 bus for controlling read-write of DDR4 outside the FPGA; and the PCIE AXI4-Lite bus is used for accessing the user logic register or the BRAM module equipment.

7. The high-speed broadband data recording system according to claim 1, wherein a plurality of RAID cards are installed in each storage server, and an N+M distributed RAID mechanism is adopted.

8. A distributed storage architecture of a PCI-E optical fiber card based broadband data high-speed recording system according to any one of claims 1-7, characterized by comprising a physical resource layer, a persistence layer, an access protocol layer, an application layer and a platform management layer, wherein the physical resource layer comprises a plurality of distributed storage servers for providing distributed storage services;

the persistence layer is used for supporting a multi-copy and erasure code data redundancy mechanism;

the access protocol layer is used for providing a block storage protocol accessible to the iSER client;

the application layer is used for providing broadband data high-speed write disc and NFS shared network disc service;

the platform management layer is used for carrying out storage management.

9. A method for high-speed recording of broadband data based on a PCI-E optical fiber card according to any one of claims 1 to 7, characterized in that it comprises the steps of:

installing an optical fiber card and a RAID card in a general server to form a distributed storage server;

receiving a high-speed data stream output by a high-speed AD chip through an FPGA module in the optical fiber card, carrying out high-speed data caching through a plug-in high-capacity DDR4 chip, and transmitting the cached data to a CPU memory of a server through PCIE;

and constructing a distributed storage space by using a plurality of universal servers to realize high-speed recording of broadband data.

10. The method for high-speed recording of broadband data according to claim 9, wherein said method further comprises:

opening PCI-E optical fiber card character equipment;

allocating space in the memory of the storage server to store broadband data;

creating and opening a broadband data recording file;

configuring the size of a data packet received each time, and enabling data interruption;

starting broadband data receiving, reading a broadband data packet from DDR4 to a memory space of a storage server after receiving a data interrupt, updating a DDR4 receiving data address pointer to write into a pre-created recording file, and repeating the step to wait for receiving the next data;

stopping broadband data recording after the data receiving is finished, and prohibiting interruption and stopping memory data transmission;

closing the broadband data recording file;

closing the character device, closing the thread and releasing the mapping memory space.

Images