Disclosure of Invention
In view of this, an object of the embodiments of the present invention is to provide a method, an apparatus, a computer device and a computer readable medium for transmitting network data, which improve receiving efficiency by directly using a data frame to transmit data; through the coroutine scheduling structure, the locking overhead among threads is avoided; and the data transmission not based on flow control expands the application range.
In view of the foregoing, an aspect of the embodiments of the present invention provides a network data transmission method, which may include the following steps: acquiring user data and converting the user data into a data frame; adding the data frame into a blank ring linked list frame in a ring linked list, and calculating metadata information; in response to receiving a sending command, acquiring the corresponding data frame based on metadata information in the sending command; and sending the data frame to a receiving end.
In some embodiments, the methods of the present invention may further comprise: and in response to receiving the acknowledgement message of the receiving end, releasing the corresponding data frame in the ring linked list based on the metadata information in the acknowledgement message.
In some embodiments, the methods of the present invention may further comprise: and setting the read-write operation and the circulation thread of the application layer as coroutine scheduling.
In some embodiments, converting the user data into a data frame may include: and mapping the address pointer of the kernel segment of the operating system to the memory address in the application layer through mmap.
In some embodiments, releasing the corresponding data frame in the ring linked list based on the metadata information in the acknowledgement message may include: and releasing the corresponding data frame in the link list through a release interface of the lower-layer equipment.
In another aspect of the embodiments of the present invention, the present invention further provides a network data transmission apparatus, where the apparatus may include: a first module configurable to obtain user data and convert the user data into a data frame; a second module, which may be configured to add the data frame to a blank ring linked list frame in a ring linked list and calculate metadata information; a third module, which may be configured to, in response to receiving a send command, obtain the corresponding data frame based on metadata information in the send command; and a fourth module, which may be configured to send the data frame to a receiving end.
In some embodiments, the apparatus of the present invention may further include a fifth module, which may be configured to: and in response to receiving the acknowledgement message of the receiving end, releasing the corresponding data frame in the ring linked list based on the metadata information in the acknowledgement message.
In some embodiments, the apparatus of the present invention further comprises a sixth module, which may be configured to: and setting the read-write operation and the circulation thread of the application layer as coroutine scheduling.
In some embodiments, the first module may be further configured to: and mapping the address pointer of the kernel segment of the operating system to the memory address in the application layer through mmap.
In some embodiments, the fifth module may be further configured to: and releasing the corresponding data frame in the link list through a release interface of the lower-layer equipment.
In another aspect of the embodiments of the present invention, the present invention further provides a computer device, where the computer device may include: at least one processor; and a memory, the memory may store computer instructions executable on the processor, the computer instructions when executed by the processor may implement steps of a method comprising: acquiring user data and converting the user data into a data frame; adding the data frame into a blank ring linked list frame in a ring linked list, and calculating metadata information; in response to receiving a sending command, acquiring the corresponding data frame based on metadata information in the sending command; and sending the data frame to a receiving end.
In some embodiments, the computer instructions, when executed by the processor, may implement the method further comprising: and in response to receiving the acknowledgement message of the receiving end, releasing the corresponding data frame in the ring linked list based on the metadata information in the acknowledgement message.
In some embodiments, the computer instructions, when executed by the processor, may implement the method further comprising: and setting the read-write operation and the circulation thread of the application layer as coroutine scheduling.
In some embodiments, converting the user data into a data frame may include: and mapping the address pointer of the kernel segment of the operating system to the memory address in the application layer through mmap.
In some embodiments, releasing the corresponding data frame in the ring linked list based on the metadata information in the acknowledgement message may include: and releasing the corresponding data frame in the link list through a release interface of the lower-layer equipment.
In yet another aspect of the embodiments of the present invention, the present invention further provides a computer readable storage medium, which may store a computer program for implementing the above method steps when executed by a processor.
The invention can have at least the following beneficial technical effects: the running mode of directly using frame data is utilized, the limitation that the original kcp protocol only supports a tcp/ip protocol stack is changed, and meanwhile, due to the fact that data are directly operated by using a bottom frame, multiple memory copying and checking processes can be omitted, and sending/receiving efficiency is optimized; meanwhile, by borrowing the coroutine structure, the lock expense among threads is avoided in the using process, the development burden in development is also reduced, and a user can develop high-efficiency and lock-free io transceiving request logic only by calling simple logics of creating the coroutine, operating, yield and completing the flow; finally, because the tcp/ip protocol stack is stripped to form the kcp protocol, and the operation flow of applying for a frame/releasing the frame in the kcp protocol is saved, the modified logic can operate in an embedded system with a tight resource or a large program with a strict requirement on resource management, and the application range is expanded.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it is understood that "first" and "second" are only used for convenience of expression and should not be construed as limitations to the embodiments of the present invention, and the descriptions thereof in the following embodiments are omitted.
In view of the above object, a first aspect of the embodiments of the present invention provides an embodiment of a network data transmission method. Fig. 1 is a schematic diagram illustrating an embodiment of a network data transmission method provided by the present invention. As shown in fig. 1, a network data transmission method according to an embodiment of the present invention may include the following steps:
s01, acquiring user data and converting the user data into a data frame;
s02, adding the data frame into a blank ring linked list frame in a ring linked list, and calculating metadata information;
s03, responding to the received sending command, acquiring the corresponding data frame based on the metadata information in the sending command; and
and S04, sending the data frame to a receiving end.
In this embodiment, the logic for applying for frame memory and managing in the original kcp protocol is modified to the operation logic for directly scheduling the device frame, and the operation logic for transmitting/receiving the request is scheduled by using the protocol. The kcp protocol is an ARQ (Automatic Repeat-reQuest) protocol with reliability, and the ARQ protocol is one of error correction protocols of a data link layer in an OSI Model (Open System Interconnection Reference Model). It includes stop-and-wait ARQ protocol and continuous ARQ protocol, Error Detection (Error Detection), Positive acknowledgement (Positive acknowledgement), time-out Retransmission (Retransmission after Timeout), and Negative acknowledgement followed by Retransmission (Negative acknowledgement and Retransmission).
In this embodiment, for the characteristic that the kcp protocol is based on the flow control, the technical solution of the embodiment of the present invention can modify the read and write data interfaces of the kcp protocol, modify the original write and read functions based on the flow control into interfaces based on data frames, and directly write the bottom layer data frames into the device transmission queue, bypassing the restriction of the tcp/ip protocol.
In this embodiment, for stop-and-wait protocol caching, the native kcp protocol uses an array queue to store the queue to be retransmitted and then copies it into the output stream, which causes an extra memory copy consumption. Aiming at the problem, the retransmission queue is replaced by a ring link table structure provided by bottom layer frame data, when sending data is written, a blank ring link table frame is directly written and meta (metadata) information is calculated, when receiving confirmation, a frame memory is directly released, and memory copy and application layer/kernel layer scheduling overhead are avoided.
In this embodiment, for a network transmission request in a general situation, an application layer design may need a seletc/iocp/epoll/libev mechanism to implement asynchronous data scheduling, and an internal implementation of the mechanism is to construct a thread pool and trigger an asynchronous callback function. But in cases where performance optimization requirements are high (e.g., tcp/ip stack), either thread overhead alone or memory allocation functionality alone may not be acceptable. In view of the above situation, the technical solution of the embodiment of the present invention may modify a kcp protocol data transmission cycle, and schedule the transmission cycle into a coroutine process, thereby avoiding system switching overhead and an additional data lock function.
In some embodiments of the present invention, the methods provided herein may further comprise: and in response to receiving the acknowledgement message of the receiving end, releasing the corresponding data frame in the ring linked list based on the metadata information in the acknowledgement message.
In this embodiment, after the successful transmission of the acknowledgement frame, the conventional free function is not used, and the release interface of the lower layer device itself is called. Therefore, the problem of memory fragments generated by one-time memory application/release can be avoided, and the one-time memory copying process from the user mode to the kernel mode is saved; in practical use, mechanisms like device DMA memory copy intel avx512 instruction set can be used to speed up the most time-consuming memory copy process. And in the process, the transmission mechanism of the jumbo frame (>4k byte) is supported naturally.
In some embodiments of the present invention, the methods provided herein may further comprise: and setting the read-write operation and the circulation thread of the application layer as coroutine scheduling.
In this embodiment, the common usage manner in the original protocol is to use one thread to circularly wait, process the read/write message, and execute the read/write operation and the corresponding subsequent logic by other multiple threads. This has the advantage of ease of use and understanding, but inter-thread data synchronization has always been a performance-costly operation. We now set the application layer read-write operations and the loop thread as coroutine. Coroutines are also called micro threads, fiber threads, and the switching between coroutines occurs in a user mode. And no mechanism such as clock interrupt, system call and the like exists in the user mode. The greatest advantage is the extremely high execution efficiency of the coroutines, and the second is the lock mechanism without multiple threads.
By means of a coroutine mechanism, any coroutine task actually runs in the current thread when sending data, and no thread conflict exists with other task scheduling. And after the scheduling is finished, performing yield operation and switching to any other yield operation to be sent or kcp core cycle. In the core cycle, the kcp protocol executes the transceiving and management of data, and executes resume after the current task is completed, and then the thread context is switched to the coroutine logic which executes yield operation before, and other logic behaviors of successful sending/receiving are continued. By the method, the locking expense among threads is avoided, the development burden during development is also reduced, and a user can develop efficient and lock-free io transceiving request logic only by calling simple logics of creating coroutines, operating, yield and completing the flow.
In some embodiments of the invention, converting the user data into data frames comprises: and mapping the address pointer of the kernel segment of the operating system to the memory address in the application layer through mmap.
In this embodiment, in the conventional protocol, the original data incoming from the upper layer needs to allocate frame memory-set frame header-call tcp/udp protocol for transmission, which means two memory copy processes. After improvement, the processes of allocating the application frame memory and scheduling frame transmission/reception do not use the malloc (dynamic memory allocation) dynamic allocation frame memory in the prior art, but use mmap (a memory mapping file method) to directly map the kernel segment address pointer of the operating system to the memory address in the application layer.
The dynamic memory allocation is used for applying for a continuous memory block region with a specified size to return an allocated memory region address in a void type, when the specific position of the memory cannot be known, if a real memory space is bound, the dynamic allocated memory is needed, and the allocated size is the size required by a program.
And mmap maps a file or other object into memory. The file is mapped to multiple pages and if the size of the file is not the sum of the sizes of all pages, the unused space of the last page is cleared. mmap plays a significant role in the user-space mapping invocation system. mmap operations provide a mechanism for user programs to directly access device memory, which is significantly more efficient than copying data between user space and kernel space. Are commonly used in applications requiring high performance. The mmap mapping memory must be integer multiple of the page size, the stream-oriented device cannot perform mmap, and the mmap implementation is related to hardware.
In this embodiment, the limitation that the original kcp protocol only supports the tcp/ip protocol stack is changed by using the running mode of directly using the frame data, and meanwhile, as the data is directly operated by using the bottom frame, the memory copy and check processes can be omitted for many times, and the sending/receiving efficiency is optimized.
In some embodiments of the present invention, releasing the corresponding data frame in the ring linked list based on the metadata information in the acknowledgement message may include: and releasing the corresponding data frame in the link list through a release interface of the lower-layer equipment.
In this embodiment, the writing of data in the protocol in the prior art is performed by calling a socket interface, and is modified into a data interaction interface that directly sends frame data with a lower application, so that a strong correlation relationship of a tcp/ip protocol layer on data requirement sequentiality is avoided.
In some embodiments of the present invention, because the tcp/ip protocol stack is stripped to the kcp protocol, and the operation flow of applying for a frame/releasing a frame in the kcp protocol is omitted, the modified logic can be operated in an embedded system with a tight resource or a large program with a strict requirement on resource management, thereby expanding the application range.
It should be particularly noted that, the steps in the embodiments of the network data transmission method described above may be mutually intersected, replaced, added, or deleted, and therefore, the network data transmission method based on these reasonable permutation and combination transformations shall also belong to the scope of the present invention, and shall not limit the scope of the present invention to the embodiments.
In view of the above object, a second aspect of the embodiments of the present invention provides a network data transmission device. Fig. 2 is a schematic diagram of an embodiment of a network data transmission device provided in the present invention. As shown in fig. 2, the network data transmission apparatus according to the embodiment of the present invention may include the following modules: a first module S11, the first module S11 may be configured to obtain user data and convert the user data into data frames; a second module S12, the second module S12 may be configured to add the data frame to a blank ring list frame in a ring list and calculate metadata information; a third module S13, the third module S13 may be configured to, in response to receiving a send command, obtain the corresponding data frame based on metadata information in the send command; and a fourth module S14, the fourth module S14 may be configured to send the data frame to a receiving end.
In some embodiments of the present invention, the apparatus provided by the present invention may further include a fifth module (not shown in the figures), and the fifth module may be configured to: and in response to receiving the acknowledgement message of the receiving end, releasing the corresponding data frame in the ring linked list based on the metadata information in the acknowledgement message.
In some embodiments of the invention, the apparatus further comprises a sixth module, which may be configured to: and setting the read-write operation and the circulation thread of the application layer as coroutine scheduling.
In some embodiments of the invention, the first module S11 may be further configured to: and mapping the address pointer of the kernel segment of the operating system to the memory address in the application layer through mmap.
In some embodiments of the invention, the fifth module may be further configured to: and releasing the corresponding data frame in the link list through a release interface of the lower-layer equipment.
In view of the above object, a third aspect of the embodiments of the present invention provides a computer device. Fig. 3 is a schematic diagram of an embodiment of a computer device provided by the present invention. As shown in fig. 3, a computer apparatus of an embodiment of the present invention may include the following means: at least one processor S21; and a memory S22, the memory S22 may store computer instructions S23 executable on the processor, the computer instructions S23 when executed by the processor may implement steps of a method comprising: acquiring user data and converting the user data into a data frame; adding the data frame into a blank ring linked list frame in a ring linked list, and calculating metadata information; in response to receiving a sending command, acquiring the corresponding data frame based on metadata information in the sending command; and sending the data frame to a receiving end.
In some embodiments of the invention, the computer instructions S23 when executed by the processor may implement the method further comprising: and in response to receiving the acknowledgement message of the receiving end, releasing the corresponding data frame in the ring linked list based on the metadata information in the acknowledgement message.
In some embodiments of the invention, the computer instructions S23 when executed by the processor may implement the method further comprising: and setting the read-write operation and the circulation thread of the application layer as coroutine scheduling.
In some embodiments of the invention, converting the user data into a data frame may comprise: and mapping the address pointer of the kernel segment of the operating system to the memory address in the application layer through mmap.
In some embodiments of the present invention, releasing the corresponding data frame in the ring linked list based on the metadata information in the acknowledgement message may include: and releasing the corresponding data frame in the link list through a release interface of the lower-layer equipment.
In the embodiment shown in FIG. 3, the processor S21 and the memory S22 may be connected by a bus or other means, such as being connected by a bus in FIG. 3.
The memory S22 is a non-volatile computer-readable storage medium and can be used for storing non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions/modules corresponding to the method for predicting the lifetime of a solid state disk in the embodiments of the present application. The processor S21 executes various functional applications of the server and data processing, i.e., implements a network data transmission method, by executing nonvolatile software programs, instructions, and modules stored in the memory S22.
The invention also provides a computer readable storage medium. FIG. 4 is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by the present invention. As shown in fig. 4, the computer readable storage medium S31 stores a computer program S32 which can be executed by a processor to perform the above method.
Finally, it should be noted that, as one of ordinary skill in the art can appreciate, all or part of the processes in the methods of the above embodiments may be implemented by a computer program to instruct related hardware, and the program of the network data transmission method may be stored in a computer readable storage medium, and when executed, may include the processes of the embodiments of the methods as described above. The storage medium of the program may be a magnetic disk, an optical disk, a Read Only Memory (ROM), a Random Access Memory (RAM), or the like. The embodiments of the computer program may achieve the same or similar effects as any of the above-described method embodiments.
Furthermore, the methods disclosed according to embodiments of the present invention may also be implemented as a computer program executed by a processor, which may be stored in a computer-readable storage medium. Which when executed by a processor performs the above-described functions defined in the methods disclosed in embodiments of the invention.
Further, the above method steps and system elements may also be implemented using a controller and a computer readable storage medium for storing a computer program for causing the controller to implement the functions of the above steps or elements.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments of the present invention.
In one or more exemplary designs, the functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes Compact Disc (CD), laser disc, optical disc, Digital Versatile Disc (DVD), floppy disk, blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.