CN113852666A

CN113852666A - Method for acquiring HTTP (hyper text transport protocol) resources in real time through FTP (file transfer protocol)

Info

Publication number: CN113852666A
Application number: CN202110980462.4A
Authority: CN
Inventors: 陆涛; 孙为国; 许文华
Original assignee: Tianyi Digital Life Technology Co Ltd
Current assignee: Tianyi Digital Life Technology Co Ltd
Priority date: 2021-08-25
Filing date: 2021-08-25
Publication date: 2021-12-28
Anticipated expiration: 2041-08-25
Also published as: CN113852666B

Abstract

The invention provides a method for acquiring HTTP resources in real time through an FTP protocol. The invention provides an FTP proxy module which converts FTP requests into HTTP requests in real time and converts HTTP responses into FTP responses in real time. In the invention, a client initiates an FTP request to an FTP proxy module firstly, completes authentication login after TCP three-way handshake connection, then initiates the FTP request to the FTP proxy module to request HTTP resources, the FTP proxy module splices URL used for initiating the HTTP request to an HTTP source station based on the FTP request, and initiates the request to the HTTP source station by the spliced URL. The HTTP source station sends HTTP response to the FTP proxy module, the FTP proxy module receives and analyzes the corresponding TCP message, the FTP proxy module applies for a memory, stores temporary TCP message data, packages the TCP message in an FTP format and sends the TCP message to the client, and therefore the purpose of real-time HTTP source return is achieved.

Description

Method for acquiring HTTP (hyper text transport protocol) resources in real time through FTP (file transfer protocol)

Technical Field

The invention relates to the field of file transmission, in particular to the field of transmission of internet files (such as pictures, videos and the like).

Background

FTP was proposed in 1971 and is the most long-standing network tool on the Internet; HTTP was proposed in 1991 and is the most popular application protocol on the internet. With the continuous development of communication technology, FTP and HTTP, as file transfer protocols, are always very important services for the internet, and are widely applied to the fields of DVB, IPTV, OTT, and the like.

The DVB and IPTV technologies have been developed for over a decade, the scale of users has broken through 7 hundred million users, the video inventory has exceeded ten million hours, and both work orders and content media are transmitted by the FTP protocol as default according to the requirements of the related technical specifications of video injection.

Compared with DVB and IPTV, audio-visual entertainment files on the Internet and OTT are transmitted based on HTTP protocol, and on the basis of HTTP, the transmission efficiency and effect are improved through CDN technology, so that audio-visual entertainment services such as popular films, short videos, self-media and the like are more enjoyable.

At present, a few service platforms upgrade a content injection system to be compatible with HTTP back sources, and accordingly, the content is introduced into the audio-visual entertainment content of the Internet in an HTTP back source mode, however, the existing platforms are greatly improved, and a plurality of platforms do not have the conditions for improving and upgrading. Therefore, how to obtain HTTP resources in real time through the FTP protocol becomes important without changing the existing system.

Disclosure of Invention

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

According to an embodiment of the present invention, there is provided a method for acquiring an HTTP resource in real time through an FTP protocol, including: receiving an FTP request in a URL form; splicing into a URL for initiating an HTTP request to an HTTP source station based on the FTP requests; initiating an HTTP request to the HTTP source station by using the spliced URL; receiving an HTTP response from the HTTP source station; analyzing a corresponding TCP message in the HTTP response; packaging the TCP message by using an FTP format; wherein the method is implemented by the FTP proxy module.

According to another embodiment of the present invention, there is provided a system for acquiring an HTTP resource in real time by using an FTP protocol, including: an FTP proxy module configured to: receiving an FTP request in a URL form; splicing into a URL for initiating an HTTP request to an HTTP source station based on the FTP requests; initiating an HTTP request to the HTTP source station by using the spliced URL; receiving an HTTP response from the HTTP source station; analyzing a corresponding TCP message in the HTTP response; and packaging the TCP message by using an FTP format.

According to another embodiment of the present invention, there is provided a computing device for acquiring HTTP resources in real time through an FTP protocol, including: a processor; a memory storing instructions that, when executed by the processor, are capable of performing the method as described above.

These and other features and advantages will become apparent upon reading the following detailed description and upon reference to the accompanying drawings. It is to be understood that both the foregoing general description and the following detailed description are explanatory only and are not restrictive of aspects as claimed.

Drawings

So that the manner in which the above recited features of the present invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only some typical aspects of this invention and are therefore not to be considered limiting of its scope, for the description may admit to other equally effective aspects.

FIG. 1 shows a schematic diagram 100 of a prior art system for downloading and transferring files using FTP technology;

FIG. 2 shows a schematic diagram 200 of a system for real-time acquisition of HTTP resources via the FTP protocol, in accordance with one embodiment of the present invention;

FIG. 3 illustrates a hierarchy of a network model in the prior art;

FIG. 4 illustrates a prior art TCP message graph;

FIG. 5 shows a flow diagram of a method 500 for obtaining HTTP resources in real-time via FTP protocol according to one embodiment of the present invention;

FIG. 6 shows schematic diagrams 600-1 through 600-13 illustrating the steps of FIG. 5, according to one embodiment of the invention; and

FIG. 7 illustrates a block diagram of a computing device 700 that may be used in hardware devices in accordance with aspects of the invention.

Detailed Description

The present invention will be described in detail below with reference to the attached drawings, and the features of the present invention will be further apparent from the following detailed description.

Interpretation of industry terms:

FTP (File Transfer Protocol) is a Protocol for two computers to Transfer files on a network, operates on TCP, is a way to Transfer files from one computer to another computer over the Internet, and is widely used for trusted File Transfer in various industries by means of the authentication mechanism that is native to the Protocol.

HTTP (HyperText Transfer Protocol) transfers data (HTML files, picture files, query results, etc.) based on TCP/IP communication Protocol, which is the most widely used network Transfer Protocol on the internet, and all WWW files must comply with this standard.

An Internet Protocol Television (IPTV) is a technology that integrates a plurality of technologies such as Internet, multimedia, and communication by using a broadband cable Television network, and provides a plurality of interactive services including digital Television to home users.

A CDN (Content Delivery Network) is an intelligent virtual Network constructed on the basis of an existing Network, and by means of edge servers deployed in various places, a user can obtain required Content nearby through functional modules of load balancing, Content Delivery, scheduling, and the like of a central platform, so that Network congestion is reduced, and the access response speed of the user is increased.

Generally speaking, the existing FTP file downloading needs to be stored in advance and then forwarded, and the invention provides real-time analysis and encapsulation of the FTP and HTTP protocol messages transmitted on the basis of TCP, so as to realize real-time conversion.

Fig. 1 shows a schematic diagram 100 of a prior art system for downloading and transferring files using FTP technology. Although HTTP has been widely used in various industries, there still exist some services that require downloading and transferring files via FTP, because FTP was proposed 20 years earlier than HTTP. When a file is to be acquired from the internet, the file needs to be downloaded and stored in advance, and then the FTP download service is distributed and provided, so that the method not only occupies a larger storage space, but also has long operation time consumption and untimely file update.

For example, referring to FIG. 1, there is shown an HTTP source station 101, a file management system 102, and a business system 103. Generally, the business system 103 is used to provide various services such as video services, file services, and the like to clients (such as set-top boxes, smart homes, and the like). The business system 103 receives requests from clients and initiates requests for videos, files, pictures, etc. to the file management system 102. The file management system 102 downloads content (e.g., video, pictures, files, etc.) from the HTTP source station 101 by HTTP and stores and manages the content, and the service system 103 acquires the content from the file management system 102 by FTP transmission.

Specifically, in the related art related to the IPTV service, the file management system 102 provides a download service module 106, an FTP service module 107, and a distributed storage module 108 in addition to the work order management module 104 that manages the work orders and the file review module 105 that reviews files. The work order management module 104 is configured to manage a work order, which generally refers to a list of information for each time a download request is initiated, such as an ID for initiating the request, a time, a file name for applying for downloading, and the like. The file audit module 105 is configured to confirm whether the requested file has a problem, such as illegal content. The file management system 102 downloads internet contents (e.g., videos, pictures, files, etc.) from the HTTP source station 101 via HTTP technology through the download service module 106 in advance on the premise that the URL of the internet contents to be downloaded is known, and stores the downloaded contents in the distributed storage module 108. The file management system 102 then publishes the content available for download in the form of an FTP service. When the service system 103 initiates an FTP request to transmit contents by means of the URL to which the FTP is spliced, the file management system 102 transmits the stored contents to the service system 103 by means of the FTP.

However, as business progresses, the amount of video content stocked at the file management system 102 increases, causing the following problems: 1. the demand of FTP storage space is increasing; 2. downloading and storing content in advance reduces the speed of content introduction, resulting in low business operation efficiency.

Fig. 2 shows a schematic diagram 200 of a system for acquiring HTTP resources in real time via FTP protocol according to an embodiment of the present invention.

Compared with the prior art of fig. 1, in the file management system 102, an FTP proxy module 201 is used in fig. 2 instead of the download service module 106, the FTP service module 107 and the distributed storage module 107 in fig. 1. Specifically, on the premise that the URL of the internet content to be downloaded is known, when the service system 103 splices the URL of the FTP and requests transmission, the FTP proxy module 201 converts the FTP request into an HTTP request in real time, and converts an HTTP response into an FTP response in real time, thereby providing FTP transmission.

In the present invention, the FTP proxy module 201 is configured to provide real-time conversion of HTTP and FTP, i.e., real-time conversion of FTP requests to HTTP requests, and real-time conversion of HTTP responses to FTP responses. The implementation logic of the FTP proxy module 201 is transparent to the service system 103, provides FTP download service for the service system 103, and can be used for downloading files such as videos, pictures, installation packages and the like on the internet. Therefore, the invention replaces the modes of downloading files in advance, storing files and releasing files through FTP in the application scene in the prior art with the FTP proxy module which can convert HTTP and FTP in real time, thereby saving the storage space and improving the downloading speed.

Those skilled in the art will appreciate that any module may communicate with any other module at the file management system 102, but not all connections are shown for ease of illustration. Also, it is fully understood by those skilled in the art that the various modules described above are illustrated herein for illustrative purposes only, and that the functionality of one or more of the modules described above may be combined into a single module or split into multiple modules. Also, one or more of the above modules may be implemented in software, hardware, or a combination thereof. Furthermore, the communication mode between the HTTP source station 101, the file management system 102 and the service system 103 is performed in a manner known in the art, and is not within the protection scope of the present invention.

It is practically feasible to implement real-time conversion of FTP and HTTP. Fig. 3 shows a hierarchy of a network model in the prior art. Fig. 4 shows a TCP message diagram in the prior art. As can be seen from fig. 3, the FTP protocol and the HTTP protocol both belong to the application layer of the network model, and are transmitted based on the TCP protocol, and the transmitted messages both conform to the TCP format. The invention is realized based on TCP Socket programming, and the protocol of the upper layer of the TCP message needs to be analyzed and processed.

Fig. 5 shows a flow diagram of a method 500 for obtaining HTTP resources in real time via FTP protocol, in accordance with one embodiment of the present invention. Fig. 6 shows schematic diagrams 600-1 through 600-13 illustrating the steps of fig. 5, according to an embodiment of the invention. Wherein the FTP proxy module may be implemented in the file management system 102. The scenario of FIG. 5 is conducted in the case where the business system knows the URL of the HTTP resource to be acquired at the file management system (e.g., directory location and name at the file management system).

Generally speaking, a service system firstly initiates an FTP connection request to an FTP proxy module, authentication login is completed after TCP three-way handshake connection, the service system then initiates an FTP request to the FTP proxy module to request a file, the FTP proxy module converts and splices request URLs, and the spliced URLs are used for initiating a request to an HTTP source station. The HTTP source station sends HTTP response to the FTP proxy module, the FTP proxy module receives and analyzes the corresponding TCP message, the FTP proxy module applies for a memory, stores temporary TCP message data, packages the TCP message in an FTP format and sends the TCP message to a service system, and therefore the purpose of real-time HTTP source return is achieved.

In step 501, the service system initiates an FTP connection establishment request to the FTP proxy module, and establishes a TCP connection. According to one embodiment of the invention, referring to fig. 600-1, the TCP connection is established after a TCP three-way handshake.

In step 502, the FTP proxy module authenticates the log-in of the business system. According to one embodiment of the invention, the authentication comprises: (1) and the FTP agent module sends a message to the service system. For example, referring to fig. 600-2, in the message, the code field 220 indicates that a new user service is ready, the arg field is a server name and version number, and ftpProxy 1.0 is a service name and version number of the FTP proxy module; (2) and the service system sends a request message to the FTP proxy module. For example, see fig. 600-3, where the command field indicates that the message includes a user name, and the arg field indicates the content of the user name; (3) and the FTP agent module replies a response message. For example, referring to fig. 600-4, where the code value is 331, indicating that a username exists, a password needs to be entered for authentication; (4) and the service system sends a request message to the FTP proxy module. For example, see fig. 600-5, where the command field indicates that the message contains a password, and the arg field indicates the password content; (5) and the FTP agent module replies a response message. For example, referring to FIGS. 600-6, where the code value is 230, indicating that the user is logged in, operation may continue.

In step 503, the service system establishes an FTP passive connection with the FTP proxy module. As shown in fig. 4, the present invention supports both active and passive modes of FTP, and the passive mode is taken as an example in the present invention. According to one embodiment of the invention, the service system sends a request to the FTP proxy module to establish a passive connection. The FTP proxy module responds, agrees to establish the passive connection and sends the IP and the monitoring port of the FTP proxy module to the service system. In accordance with one embodiment of the present invention, see, e.g., FIGS. 600-7, where the IP addresses of the passive connections are: 221.228.56.5, port 10028.

In step 504, the service system sends an FTP request to the FTP proxy module for HTThe TP resource makes a request. According to one embodiment of the invention, the FTP request has the format of an FTP URL. According to one embodiment of the invention, the FTP request comprises at least the following: IP address and port of the FTP proxy module (generally, port of the FTP proxy module is 21); directory location and name of the requested HTTP resource at the file management system. For example, the FTP request may beftp:// 221.228.56.5:21/config.txt. Fig. 600-8 illustrate TCP messages for the FTP request, according to one embodiment of the present invention.

In step 505, the FTP proxy module splices into URLs for initiating HTTP requests to HTTP source stations based on the received FTP requests. According to one embodiment of the invention, for example, referring to FIGS. 600-8, the splice URL includes: A) the destination IP and destination port of the TCP packet of the FTP request in step 504 (i.e., IP address and port of FTP proxy module) are parsed, e.g., 221.228.56.5: 21; B) analyzing a Request/Rest keyword in the TCP message, and taking a directory after intercepting a "/" symbol through a regular expression as a root directory, for example, the file name is config.txt; C) querying the configuration file to obtain the HTTP source address corresponding to the destination IP, for example, 221.228.39.45: 9080; D) the obtained address of the HTTP source station is spliced with the URL indicating the position and the name of the file directory in the FTP request to obtain the URL of the HTTP source station corresponding to the HTTP resource, for example, the URL ishttp://221.228.39.45:9080/config.txt. According to one embodiment of the invention, the configuration file describes the correspondence between the IP of the FTP proxy module and the HTTP source station address.

At step 506, the FTP proxy module initiates an HTTP request to the HTTP origin station with the spliced URL. According to one embodiment of the invention, for example, referring to fig. 600-9, the FTP proxy module establishes a TCP connection with the HTTP source station through a three-way handshake. According to another embodiment of the invention, the FTP agent module carries out configuration according to the address of a common source station in the configuration file, and selects whether to use the long connection pool, thereby reducing the time consumption of connection and disconnection.

The HTTP source station sends an HTTP response to the FTP proxy module in step 507. According to one embodiment of the invention, for example, referring to FIGS. 600-10, the HTTP response includes file data.

In step 508, the FTP proxy module receives and parses the corresponding TCP message in the HTTP response and applies for memory to store temporary data. According to one embodiment of the invention, the FTP proxy module applies for memory to store temporary TCP message data. Specifically, the FTP proxy module parses the HTTP message obtained in step 507, removes the fields of the HTTP message, and temporarily stores the HTTP message. A memory pool of 256 size is created for each HTTP connection and a memory pool of 4096 size is created for each request. The memory pool is managed through a linked list and a pointer, and the management comprises application, occupation, statistics, release and the like. For example, the diagrams 600-11 illustrate a memory pool model.

In step 509, the FTP proxy module encapsulates the TCP packet and sends the encapsulated TCP packet to the client. According to an embodiment of the present invention, for example, referring to fig. 600-12, the FTP proxy module encapsulates the response data temporarily stored in step 508 into TCP packets in FTP format, and sends the encapsulated TCP packets to the client.

At step 510, after the data transmission is completed, the TCP connection between the service system and the FTP proxy is disconnected. According to one embodiment of the invention, for example, referring to FIGS. 600-13, a TCP connection is broken by a TCP four-wave.

Meanwhile, the invention also optimizes the FTP agent module in the following aspects:

1. breakpoint resuming: the FTP protocol realizes the breakpoint continuous transmission instruction through the REST instruction, and the HTTP protocol realizes the RANGE request through the RANGE instruction, so that the protocol conversion of the breakpoint continuous transmission function is realized on the basis of realizing the whole file transmission.

2. Hot spot caching: considering that a service system may request the same file for multiple times in a short time, the response speed of the file can be accelerated by temporarily caching the hot file, and meanwhile, the function can be controlled to be switched on and off by referring to a no-cache instruction of HTTP.

3. Connection multiplexing: as the agent, the TCP connection is required to be continuously established with the source station, and the source station is configured, so that the number of the source stations is not too large, a long link pool with the specified source station is introduced, the time of three-way handshake and four-way waving is saved, and the condition that a large number of TIMEWAIT states occur in a large number of concurrent ways to cause channel overload can be prevented.

Compared with the prior art, the invention at least has the following advantages:

1) in the technical scheme of the invention, the file management system does not need to provide FTP storage service for downloading the content media, and only needs to deploy the proxy service of the FTP, thereby reducing the storage space demand;

2) in the technical scheme of the invention, the file management system does not need to download the file in advance, thereby saving time, improving efficiency and improving the overall transmission speed of the content medium between the network and the platform.

With reference to FIG. 7, a computing device 700, which is one example of a hardware device that may be employed in connection with aspects of the present invention, will now be described. For example, the client, proxy module, and/or HTTP origin station in fig. 5 may be implemented as computing device 700. Computing device 700 may be any machine that may be configured to implement processing and/or computing, and may be, but is not limited to, a workstation, a server, a desktop computer, a laptop computer, a tablet computer, personal digital processing, a smart phone, an in-vehicle computer, or any combination thereof. The various methods/apparatus/servers/client devices described above may be implemented in whole or at least in part by a computing device 700 or similar device or system.

Computing device 700 may include components that may be connected or communicate via one or more interfaces and a bus 702. For example, computing device 700 may include a bus 702, one or more processors 704, one or more input devices 706, and one or more output devices 708. The one or more processors 704 may be any type of processor and may include, but are not limited to, one or more general purpose processors and/or one or more special purpose processors (e.g., dedicated processing chips). Input device 706 may be any type of device capable of inputting information to a computing device and may include, but is not limited to, a mouse, a keyboard, a touch screen, a microphone, and/or a remote controller. Output device 708 may be any type of device capable of presenting information and may include, but is not limited to, a display, speakers, a video/audio output terminal, a vibrator, and/or a printer. Computing device 700 may also include or be connected with non-transitory storage device 710, which may be any storage device that is non-transitory and that enables data storage, and which may include, but is not limited to, a disk drive, an optical storage device, a solid-state memory, a floppy disk, a flexible disk, a hard disk, a tape, or any other magnetic medium, an optical disk or any other optical medium, a ROM (read only memory), a RAM (random access memory), a cache memory, and/or any memory chip or cartridge, and/or any other medium from which a computer can read data, instructions, and/or code. Non-transitory storage device 610 may be detached from the interface. The non-transitory storage device 710 may have data/instructions/code for implementing the above-described methods and steps. Computing device 700 may also include a communication device 712. The communication device 712 may be any type of device or system capable of communicating with internal apparatus and/or with a network and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication device, and/or a chipset, such as a bluetooth device, an IEEE 1302.11 device, a WiFi device, a WiMax device, a cellular communication device, and/or the like.

The bus 702 may include, but is not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an enhanced ISA (eisa) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

Computing device 700 may also include a working memory 714, which working memory 714 may be any type of working memory capable of storing instructions and/or data that facilitate the operation of processor 704 and may include, but is not limited to, random access memory and/or read only memory devices.

Software components may be located in the working memory 714, including, but not limited to, an operating system 716, one or more application programs 718, drivers, and/or other data and code. Instructions for implementing the above-described methods and steps of the invention may be contained in the one or more application programs 718, and the above-described method 500 of the invention may be implemented by the processor 704 reading and executing the instructions of the one or more application programs 718.

It should also be appreciated that variations may be made according to particular needs. For example, customized hardware might also be used, and/or particular components might be implemented in hardware, software, firmware, middleware, microcode, hardware description speech, or any combination thereof. In addition, connections to other computing devices, such as network input/output devices and the like, may be employed. For example, some or all of the disclosed methods and apparatus can be implemented with logic and algorithms in accordance with the present invention through programming hardware (e.g., programmable logic circuitry including Field Programmable Gate Arrays (FPGAs) and/or Programmable Logic Arrays (PLAs)) having assembly language or hardware programming languages (e.g., VERILOG, VHDL, C + +).

Although the various aspects of the present invention have been described thus far with reference to the accompanying drawings, the above-described methods, systems, and apparatuses are merely examples, and the scope of the present invention is not limited to these aspects but only by the appended claims and equivalents thereof. Various components may be omitted or may be replaced with equivalent components. In addition, the steps may also be performed in a different order than described in the present invention. Further, the various components may be combined in various ways. It is also important that as technology develops that many of the described components can be replaced by equivalent components appearing later.

Claims

1. A method for acquiring HTTP resources in real time through FTP protocol includes:

receiving an FTP request in a URL form;

splicing into a URL for initiating an HTTP request to an HTTP source station based on the FTP requests;

initiating an HTTP request to the HTTP source station by using the spliced URL;

receiving an HTTP response from the HTTP source station;

analyzing a corresponding TCP message in the HTTP response;

packaging the TCP message by using an FTP format;

wherein the method is implemented by an FTP proxy module.

2. The method of claim 1, wherein the FTP request is received from a business system.

3. The method of claim 2, further comprising:

before receiving the FTP request, the service system initiates an FTP connection establishment request to the FTP agent module, and completes authentication login after TCP three-way handshake connection establishment.

4. The method of claim 1, further comprising: and applying for a memory to store temporary TCP message data.

5. The method of claim 2, further comprising: and sending the encapsulated TCP message to the client.

6. The method of claim 1, wherein the FTP proxy module is included in a file management system, wherein stitching based on the FTP request to a URL for initiating an HTTP request to an HTTP source station further comprises:

analyzing the FTP request to obtain an HTTP source station address corresponding to the IP of the FTP proxy module indicated in the FTP request, and obtaining a directory position and a name of the HTTP resource indicated in the FTP request in the file management system;

and splicing the HTTP source station address with the directory position and the name of the HTTP resource in the file management system to obtain the URL of the HTTP source station corresponding to the HTTP resource.

7. A system for acquiring HTTP resources in real time through FTP protocol comprises:

an FTP proxy module configured to:

receiving an FTP request in a URL form;

initiating an HTTP request to the HTTP source station by using the spliced URL;

receiving an HTTP response from the HTTP source station;

analyzing a corresponding TCP message in the HTTP response;

and packaging the TCP message by using an FTP format.

8. The system of claim 7, further comprising: a business system configured to:

initiating an FTP connection establishing request to the FTP agent module and establishing a TCP connection;

sending the FTP request to the FTP agent module;

and receiving the encapsulated TCP message sent by the FTP proxy module.

9. The system of claim 7, wherein the FTP proxy module is included in a file management system, wherein stitching to URLs for initiating HTTP requests to HTTP source stations based on the FTP requests further comprises:

10. A computing device for real-time acquisition of HTTP resources via FTP protocol, comprising:

a processor;

a memory storing instructions that, when executed by the processor, are capable of performing the method of claims 1-6.