CN113207009B - Method, device and system for testing audio-video interaction system with out-of-step sound and picture - Google Patents

Method, device and system for testing audio-video interaction system with out-of-step sound and picture Download PDF

Info

Publication number
CN113207009B
CN113207009B CN202110509258.4A CN202110509258A CN113207009B CN 113207009 B CN113207009 B CN 113207009B CN 202110509258 A CN202110509258 A CN 202110509258A CN 113207009 B CN113207009 B CN 113207009B
Authority
CN
China
Prior art keywords
audio
video
terminal
test
tested
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110509258.4A
Other languages
Chinese (zh)
Other versions
CN113207009A (en
Inventor
金潇泽
高伟
罗亚明
陈堃
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Industrial and Commercial Bank of China Ltd ICBC
Original Assignee
Industrial and Commercial Bank of China Ltd ICBC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Industrial and Commercial Bank of China Ltd ICBC filed Critical Industrial and Commercial Bank of China Ltd ICBC
Priority to CN202110509258.4A priority Critical patent/CN113207009B/en
Publication of CN113207009A publication Critical patent/CN113207009A/en
Application granted granted Critical
Publication of CN113207009B publication Critical patent/CN113207009B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/2187—Live feed
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/80—Responding to QoS
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N17/00—Diagnosis, testing or measuring for television systems or their details
    • H04N17/004—Diagnosis, testing or measuring for television systems or their details for digital television systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/637—Control signals issued by the client directed to the server or network components
    • H04N21/6373—Control signals issued by the client directed to the server or network components for rate control, e.g. request to the server to modify its transmission rate
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/845—Structuring of content, e.g. decomposing content into time segments
    • H04N21/8456—Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/85—Assembly of content; Generation of multimedia applications
    • H04N21/854—Content authoring
    • H04N21/8547—Content authoring involving timestamps for synchronizing content

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • General Health & Medical Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Telephonic Communication Services (AREA)

Abstract

The invention provides a method, a device and a system for testing the audio-video interaction system with the loss of synchronism of sound and pictures, and relates to the field of real-time audio-video communication technology and finance. The method comprises the following steps: configuring the network states of a sending terminal and a receiving terminal of the audio and video interactive system to be tested according to preset network configuration parameters; sending a test starting instruction to an audio and video playing terminal arranged at the sending terminal so that the audio and video playing terminal synthesizes and plays a test audio and video segment; receiving the out-of-step time length of a test audio and video clip transmitted by the audio and video interactive system to be tested and sent by an audio and video acquisition terminal arranged at the receiving terminal; and generating a sound-picture out-of-step report according to the out-of-step duration and the current network state. The invention synthesizes the special test audio and video segments by synchronizing the network state of the conference terminal, realizes the accurate measurement of the audio and video interaction system sound and picture desynchronization, and is beneficial to the development personnel to find the defects of the conference system.

Description

Method, device and system for testing audio-video interaction system with out-of-step sound and picture
Technical Field
The invention relates to the field of real-time audio and video communication technology and finance, in particular to a method, a device and a system for testing the audio and video desynchronization of an audio and video interaction system.
Background
With the migration of bank business handling to online, the real-time audio and video communication technology is more and more used in business handling in the fields of bank credit and wind control. Research shows that in the real-time audio and video communication process, when the audio and the video are not synchronous and the out-of-step time length exceeds 80 milliseconds, communication participants can feel out-of-step. When the out-of-sync period exceeds 120 ms, the communication participants feel obvious discomfort. Therefore, the test of the out-of-step sound picture in the real-time audio-video interactive system is necessary.
Meanwhile, the communication effect of the real-time audio and video communication system is easily influenced by factors such as network bandwidth, network packet loss rate, video resolution, video code rate and the like, so that the influence of different network environments on the step loss of sound pictures is also an important index for evaluating the quality of the audio and video interaction system.
However, in the prior art, a system and a method for testing the audio-visual desynchronization of the network environment are not considered.
Disclosure of Invention
In order to solve the problems in the prior art, the application provides a method for testing the audio-video interaction system with the out-of-step sound and picture, and relates to the field of real-time audio-video communication technology and finance. The method for testing the audio-video interaction system with the out-of-step sound and picture comprises the following steps:
configuring the network states of a sending terminal and a receiving terminal of the audio and video interactive system to be tested according to preset network configuration parameters;
sending a test starting instruction to an audio and video playing terminal arranged at the sending terminal so that the audio and video playing terminal synthesizes and plays a test audio and video segment;
receiving the out-of-step time length of a test audio and video clip transmitted by the audio and video interactive system to be tested and sent by an audio and video acquisition terminal arranged at the receiving terminal; the step-out duration is obtained by matching the video frame and the audio of the audio and video acquisition terminal based on the test audio and video clip transmitted by the audio and video interaction system to be tested;
and generating a sound-picture out-of-step report according to the out-of-step duration and the current network state.
In an embodiment, the method for testing the audio-visual interaction system out of sync further includes:
receiving a test stopping instruction sent by the audio and video acquisition terminal;
and sending the test stopping instruction to the audio and video playing terminal.
The application also provides another method for testing the sound and picture desynchronization of the audio and video interactive system, which comprises the following steps:
synthesizing a test audio-video clip according to a test starting instruction of the master control server;
playing the test audio and video clip at a sending terminal of the audio and video interactive system to be tested, so that the test audio and video clip is transmitted through the sending terminal, the conference server and a receiving terminal of the audio and video interactive system to be tested, and an audio and video acquisition terminal arranged at the receiving terminal analyzes the test audio and video clip to obtain the out-of-step time length of the test audio and video clip transmitted by the audio and video interactive system to be tested;
and the network states of the sending terminal and the receiving terminal are the same and are kept unchanged.
In one embodiment, the synthesizing according to the test starting instruction of the grandmaster server to obtain the test audio-video clip comprises:
respectively generating unique corresponding video frame numbers for a plurality of audio frequencies with different pre-stored frequency values;
respectively coding the frequency value of each audio and the corresponding video frame number to obtain a plurality of video frames;
and respectively coding each audio and the corresponding video frame to obtain a plurality of test audio and video segments.
The application also provides another method for testing the audio-video interaction system with the loss of synchronism of the sound and the picture, which comprises the following steps:
collecting video frames and audios of a plurality of test audio and video clips received by a receiving terminal of an audio and video interactive system to be tested; the test audio/video clip is generated by an audio/video playing terminal and is transmitted to the receiving terminal through a sending terminal and a conference server of the audio/video interactive system to be tested;
matching the video frame corresponding to the audio according to the frequency value of the collected audio;
and obtaining the out-of-step duration of the test audio and video clip transmitted by the audio and video interactive system to be tested according to the frame number corresponding to the video frame matched with the audio and the frame number of the video frame acquired simultaneously with the audio.
In an embodiment, the method for testing the out-of-step of the sound picture of the audio/video interactive system further includes:
analyzing each video frame to obtain a frame number, a frequency value and an acquisition timestamp corresponding to each video frame;
and analyzing each audio to obtain the frequency value and the acquisition time stamp of each audio.
In an embodiment, the obtaining the out-of-step duration of the test audio/video segment transmitted by the audio/video interactive system to be tested according to the frame number corresponding to the video frame matched with the audio and the frame number of the video frame simultaneously acquired with the audio includes:
matching video frames with the same acquisition time stamp according to the acquisition time stamp of the audio;
acquiring a frame number of a video frame with a collection timestamp matched with the audio;
acquiring a frame number of a video frame with a frequency value matched with the audio;
and calculating to obtain the step-out duration according to the frame number of the video frame with the frequency value matched with the audio, the frame number of the video frame with the acquisition timestamp matched with the audio, and the video frame rate of a test audio-video clip transmitted by the audio-video interaction system to be tested.
The invention also provides a device for testing the audio-video interaction system with the loss of synchronism of sound and picture, which comprises:
the network configuration module is used for configuring the network states of a sending terminal and a receiving terminal of the audio and video interactive system to be tested according to preset network configuration parameters;
the start-stop control module is used for sending a test start instruction to an audio and video playing terminal arranged at the sending terminal so as to enable the audio and video playing terminal to synthesize and play a test audio and video clip;
the data receiving module is used for receiving the out-of-step time length of a test audio and video clip transmitted by the audio and video interactive system to be tested, which is sent by an audio and video acquisition terminal arranged at the receiving terminal;
the step-out duration is obtained by matching the video frame and the audio of a test audio and video clip transmitted by the audio and video interactive system to be tested by the audio and video acquisition terminal;
and the report generation module is used for generating a sound picture desynchronizing report according to the desynchronizing time and the current network state.
In an embodiment, the start-stop control module is further configured to:
receiving a test stopping instruction sent by the audio and video acquisition terminal; and
and sending the test stopping instruction to the audio and video playing terminal.
The invention also provides another testing device for the audio-video interaction system with the loss of synchronism of the sound picture, which comprises:
the audio and video synthesis module is used for synthesizing a test audio and video segment according to the test starting instruction of the master control server;
the audio and video playing module is used for playing the test audio and video clip at the sending terminal of the audio and video interactive system to be tested, so that the test audio and video clip is transmitted through the sending terminal, the conference server and the receiving terminal of the audio and video interactive system to be tested, and an audio and video acquisition terminal arranged at the receiving terminal analyzes the test audio and video clip to obtain the step-out time length of the test audio and video clip transmitted through the audio and video interactive system to be tested;
and the network states of the sending terminal and the receiving terminal are the same and are kept unchanged.
In one embodiment, the audio/video synthesis module includes:
the video frame number generating unit is used for respectively generating unique corresponding video frame numbers for a plurality of audios with different pre-stored frequency values;
the video frame synthesis unit is used for coding the frequency value of each audio and the corresponding video frame number to obtain a plurality of video frames;
and the audio and video synthesis unit is used for respectively coding each audio and the corresponding video frame to obtain a plurality of test audio and video segments.
The invention also provides another testing device for the audio-video interaction system with the loss of synchronism of the sound picture, which comprises:
the audio and video acquisition module is used for acquiring video frames and audio of a plurality of test audio and video clips received by a receiving terminal of the audio and video interactive system to be tested; the test audio/video clip is generated by an audio/video playing terminal and is transmitted to the receiving terminal through a sending terminal and a conference server of the audio/video interactive system to be tested;
the audio and video matching module is used for matching the video frame corresponding to the audio according to the frequency value of the collected audio;
and the step-out duration calculation module is used for obtaining the step-out duration of the test audio and video clip transmitted by the audio and video interactive system to be tested according to the frame number corresponding to the video frame matched with the audio and the frame number of the video frame simultaneously acquired with the audio.
In an embodiment, the apparatus for testing the loss of synchronism of the sound picture of the audio/video interactive system further includes:
the video analysis module is used for analyzing each video frame to obtain a frame number, a frequency value and an acquisition timestamp corresponding to each video frame;
and the audio analysis module is used for analyzing each audio to obtain the frequency value and the acquisition timestamp of each audio.
In one embodiment, the out-of-step duration calculation module includes:
the video frame determining unit is used for matching video frames with the same acquisition time stamps according to the acquisition time stamps of the audios;
the video frame number determining unit is used for acquiring the frame number of the video frame of which the acquisition timestamp is matched with the audio; acquiring a frame number of a video frame with a frequency value matched with the audio;
and the step-out duration calculation unit is used for calculating the step-out duration according to the frequency value, the frame number of the video frame matched with the audio, the collection timestamp, the frame number of the video frame matched with the audio and the video frame rate of the test audio-video clip transmitted by the audio-video interactive system to be tested.
The invention also provides a system for testing the audio-video interaction system with the audio-video step-out, which comprises a system for testing the audio-video step-out and an audio-video interaction system to be tested;
the audio and video interaction system to be tested comprises: the system comprises a sending terminal, a receiving terminal and a conference server which is respectively connected with the sending terminal and the receiving terminal;
the sound picture step-out test system comprises three sound picture step-out test devices of the audio and video interaction system.
According to the method, the device and the system for testing the audio-video interaction system audio-video step-out, the audio-video interaction system audio-video step-out is accurately measured by synthesizing the special test audio-video segment; through the network state of the terminal synchronously participating in the conference, a result report of the network state and the loss of synchronism of the sound picture is generated, and therefore developers can find the defects of the conference system and conduct automatic regression testing. The device and the system for testing the audio-video interaction system with the loss of synchronism of the sound and the picture are not limited by physical distance, and can respectively simulate the real feeling of participants in a conference before the terminals participating in the conference.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic view of an application scenario of a method for testing audio-video out-of-step in an audio-video interactive system.
Fig. 2 is a schematic diagram of a first method for testing audio-video interaction system audio-video step loss.
Fig. 3 is another schematic diagram of a first method for testing audio-visual interaction system out-of-sync.
Fig. 4 is a schematic diagram of a second method for testing audio-visual interaction system out-of-sync.
FIG. 5 is a diagram illustrating a method for synthesizing a test audio/video segment.
Fig. 6 is a schematic diagram of a third method for testing the loss of synchronism of the sound and the picture of the audio-video interactive system.
Fig. 7 is a schematic diagram of a method for analyzing the step-out duration.
Fig. 8 is a schematic diagram of a device for testing the loss of synchronism of the sound picture of the audio-video interactive system.
Fig. 9 is a schematic diagram of another device for testing audio-visual interaction system out-of-sync.
Fig. 10 is a schematic diagram of another testing device for audio-video interaction system with out-of-step sound painting.
Fig. 11 is a schematic diagram of another testing device for audio-video interaction system with out-of-step sound painting.
Fig. 12 is a schematic diagram of another testing device for audio-video interaction system with out-of-step sound picture.
Fig. 13 is a schematic diagram of another testing device for audio-video interaction system with out-of-step sound painting.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
The method for testing the audio-video interaction system with the out-of-step sound and picture is suitable for systems for audio-video interaction, such as conference systems and the like. Fig. 1 is a schematic view of an application scenario of the method for testing audio-video interaction system audio-video step loss provided by the present application. As shown in fig. 1, the application scenario includes an audio/video interaction system to be tested and a sound and picture step-out test system.
The audio/video interaction system to be tested is exemplified by a conference system, specifically, the audio/video interaction system to be tested includes at least two conference terminals used by participants and a conference server respectively connected to each conference terminal, and fig. 1 is exemplified by including two conference terminals. When the conference is in progress, any terminal can be used as a sending terminal and a receiving terminal. When the audio and video out-of-step of the audio and video interaction system is measured, the audio and video out-of-step state of audio and video information received by a receiving terminal is mainly measured, so that in the application scene, it can be assumed that one conference terminal is used as a sending terminal 11, the other conference terminal is used as a receiving terminal 12, and the sending terminal 11 and the receiving terminal 12 perform data interaction through a conference server 13, so as to explain the audio and video out-of-step testing method of the audio and video interaction system.
As shown in fig. 1, the audio-video step-out testing system includes an audio-video playing terminal 21 disposed at the transmitting terminal 11, an audio-video collecting terminal 22 disposed at the receiving terminal 12, and a master control server 23 connected to the audio-video transmitting terminal 21 and the audio-video collecting terminal 22, where the master control server 23 is further connected to the transmitting terminal 11 and the receiving terminal 12 of the audio-video interactive system to be tested, respectively.
The audio/video playing terminal 21 is configured to play an audio/video clip in front of the sending terminal 11, simulate the voice and the picture of a speaker during a conference, and the sending terminal 11 collects the audio/video clip played by the audio/video playing terminal 21 and simulates the voice and the picture of the speaker. For example, the sending terminal 11 may capture a video picture of an audio/video clip played by the audio/video playing terminal through a camera, and collect an audio of the audio/video clip played by the audio/video playing terminal through a microphone. The transmitting terminal 11 transmits the captured video picture and audio to the receiving terminal 12 through the corresponding transmission channel via the conference server 13. In practical application, the audio/video playing terminal can be a mobile terminal or a PC terminal or other devices or equipment.
The receiving terminal 12 may play the received audio through a speaker, play the received video through a display screen, and simulate to show the sound and the picture of the current speaker to the local conference participants; the audio/video capture terminal 22 captures the video pictures and audio played by the receiving terminal 12, and simulates local participants to watch the video pictures and listen to the audio. The video picture and audio collected by the audio/video collecting terminal 22 are audio/video clips transmitted through the audio/video interactive system to be tested, and the audio/video clips may have a step-out problem. In practical application, the audio and video acquisition terminal can be a mobile terminal or a device or equipment such as a PC terminal.
The master control server 23 is used for performing data interaction with the audio/video playing terminal and the audio/video collecting terminal, for example, issuing a control instruction. In practical application, the master control server can be remotely accessed through another terminal, test data of audio and video interaction system audio and video desynchronization are set, test results of audio and video interaction system audio and video desynchronization are checked, and the like. The general control server can also be replaced by other terminals, devices or electronic equipment which can realize the whole functions of the general control server. The general control server is only an example given for convenience of describing the technical scheme of the application, and is not used for limiting the application.
Based on the application scenario of fig. 1, the application provides a method for testing audio-video step-out of an audio-video interactive system, which will be described below from the perspective of a master control server, an audio-video playing terminal and an audio-video acquisition terminal.
Fig. 2 is a schematic diagram of a first method for testing audio-video interaction system step-out in sound and picture provided by the present application, and is described from the perspective of a master control server. As shown in fig. 2, a first method for testing audio-visual interaction system out-of-sync includes the following steps:
step S201, configuring the network state of a sending terminal and a receiving terminal of the audio/video interactive system to be tested according to preset network configuration parameters.
Specifically, referring to fig. 1, the general control server 23 is connected to the sending terminal 11 and the receiving terminal 12, and the sending terminal 11 and the receiving terminal 12 are respectively installed with network control software. The general control server 23 sends a group of network configuration parameters to the sending terminal 11 and the receiving terminal 12 at the same time, so that the network control software on the sending terminal 11 and the receiving terminal 12 configures the network state according to the received network configuration parameters. The step ensures that the network states of the conference terminals of the audio and video interactive system to be tested are the same, so that the condition that the sound picture of the audio and video clip transmitted by the audio and video interactive system to be tested is out of step in the network state can be tested.
The network configuration parameters include, but are not limited to, network bandwidth, network packet loss rate, network delay, and other parameters.
Because the network states of all the conference terminals can be configured, the master control server can measure the sound picture desynchronizing condition of the audio and video clips transmitted by the audio and video interactive system to be tested under different network states by issuing different network configuration parameters, thereby facilitating conference system developers to find the defects of the conference system and carry out automatic regression test, and improving the conference quality.
Step S202, a test start instruction is sent to an audio and video playing terminal arranged at the sending terminal, so that the audio and video playing terminal synthesizes and plays the test audio and video segments.
Specifically, referring to fig. 1, the master control server 23 sends a test start instruction to the audio/video playing terminal 21 and the audio/video collecting terminal 22 at the same time, so that the audio/video playing terminal 21 starts to synthesize and play a test audio/video clip, and the audio/video collecting terminal 22 starts to collect the test audio/video clip transmitted by the audio/video interactive system to be tested.
The audio and video testing clip is synthesized by the audio and video playing terminal 21 according to a preset synthesis rule. And when playing, the audio and video playing terminal synchronously plays the video frame of the tested audio and video clip and the audio with the corresponding frequency. For example, suppose there are a plurality of monophonic audios S1, S2, \8230ofdifferent frequencies, SN, each audio corresponding to a respective video frame P1, P2, \8230, PN, and each video frame is not repeated. Combining the audio and the video to form a test audio-video segment, wherein the first frame of video frame of the test audio-video segment is P1, the corresponding audio is S1, the second frame of video frame of the test audio-video segment is P2, the corresponding audio is S2, and so on, the test audio-video segment comprises N video frames and N sections of audio. The specific test audio-video clip synthesis process is detailed in the following examples.
It is understood that the audio/video playing terminal 21 may play the test audio/video segment at a certain frame rate, for example, 30 frames per second. The frame rate can be preset through the master control server.
Step S203, receiving the out-of-step time length in the current network state sent by the audio and video acquisition terminal arranged at the receiving terminal, wherein the out-of-step time length refers to the out-of-step time length of the test audio and video clip transmitted by the audio and video interactive system to be tested; the step-out duration is obtained by matching the video frame and the audio of the audio and video acquisition terminal based on the test audio and video clip transmitted by the audio and video interaction system to be tested.
Specifically, after receiving the test start instruction of the master control server 23, the audio/video acquisition terminal 22 starts to acquire the audio/video segment to be tested received by the receiving terminal 12. Although the audio/video playing terminal 21 plays the video frame and the corresponding audio synchronously, and the sending terminal also receives the video frame and the audio synchronously, because the video frame and the audio of the test audio/video segment are transmitted through different signal channels in the audio/video interactive system to be tested, the video frame and the audio received through different signal channels are received by the receiving terminal 22, for example, the video frame P1 is received by the receiving terminal 22, the audio frame is not received, and the audio S1 is received while the video frame P2 is received, at this time, the video frame and the audio of the audio/video segment to be tested transmitted through the audio/video interactive system to be tested are out of step.
In order to accurately simulate the real experience of the user, the audio/video capture terminal 22 captures the picture of the display screen of the receiving terminal 12 in real time through a camera, and captures the audio played by the speaker of the receiving terminal 12 in real time through a microphone. The audio and video acquisition terminal 22 analyzes the acquired audio and video frames respectively to obtain the out-of-step time of the audio and video clip to be detected transmitted through the audio and video interactive system to be detected, and then sends the out-of-step time to the master control server 23 for the next operation. The specific method for calculating the out-of-step duration is detailed in the following embodiments, and the method is related to the synthesis process of the test audio/video clip.
And step S204, generating a sound picture desynchronizing report according to the desynchronizing time and the current network state.
Specifically, the master control server 23 records the step-out time and the corresponding network configuration parameters after receiving the step-out time data, and generates a sound-picture step-out report. The staff can access the master control server 23 through another terminal to check the audio-video out-of-step report.
After step S204, the master control server 23 may also configure the network states of the sending terminal 11 and the receiving terminal 12 according to the next set of network configuration parameters, and repeatedly execute the above steps, so as to obtain the out-of-step duration of the audio/video interactive system to be tested in different network states.
In an embodiment, as shown in fig. 3, the method for testing the out-of-sync of the sound picture of the audio-video interactive system further includes:
and step S205, receiving a test stop instruction sent by the audio and video acquisition terminal.
When the audio/video acquisition terminal 22 receives the step-out time length, it indicates that the test process is finished, so that after the operation of the audio/video acquisition terminal 22 is finished, a test stop instruction can be sent to the master control server to notify the master control server of finishing the test. The test stop instruction may be information that the audio/video acquisition terminal 22 sends the test completion to the master control server 23 alone, or may be a test stop instruction that the out-of-step time sent to the master control server 23 is used as well, and the master control server 23 can obtain the test completion information after receiving the out-of-step time data.
And step S206, sending the test stop instruction to the audio and video playing terminal.
Here, the master control server 23 sends a test stop instruction to the audio/video playing terminal 21, so that the audio/video playing terminal 21 stops playing the test audio/video clips.
According to the description of the method for testing the audio-video interaction system audio-video desynchronization from the perspective of the master control server, the influence of the network state on the data transmission performance of the audio-video interaction system to be tested is considered when the desynchronization time of the audio-video clip to be tested transmitted by the audio-video interaction system to be tested is tested. The method has the advantages that the out-of-step duration of audio and video clips transmitted in different network states is obtained by setting and changing the network states of all conference terminals in the audio and video interaction system to be tested, and a sound and picture out-of-step report is generated, so that developers can find the defects of the conference system and perform automatic regression testing, and further the influence of the network states on the data transmission performance is solved to a certain extent.
Fig. 4 is a schematic diagram of a second method for testing audio-visual desynchronization of an audio-video interactive system, which is provided by the present application and is described from the perspective of an audio-video playing terminal. As shown in fig. 4, the second method for testing the audio-visual interaction system out of sync includes the following steps:
and S401, synthesizing the testing audio and video clips according to the testing start instruction of the master control server.
Specifically, referring to fig. 1, after receiving a test start instruction from the master control server 23, the audio/video playing terminal 21 starts to synthesize a test audio/video segment, and plays the synthesized test audio/video segment at the sending terminal 11.
Specifically, the test audio/video clips used in the present application are synthesized by the audio/video playing terminal 21 according to a specific audio/video synthesis method, and the selection of synthesizing the test audio/video clips according to the specific method is also convenient for the audio/video acquisition terminal 22 to analyze the time length of step loss of the audio/video clips to be detected transmitted through the audio/video interactive system to be detected. The method for synthesizing the test audio/video clip is shown in fig. 5, and step S401 specifically includes the following steps:
step S4011, unique corresponding video frame numbers are generated for a plurality of audio frequencies with different frequency values stored in advance.
The audio frequencies are all single-tone audio frequencies, and the frequencies of the single-tone audio frequencies are different and have larger difference, so that the audio frequencies are convenient to distinguish. For example, the frequency of audio S1 is 100Hz, the frequency of audio S2 is 200Hz, and the frequency of audio SN is (N-1) 100Hz. In practice, a single tone audio with other frequency values may be selected, as long as each audio is easy to distinguish, and the application is not limited thereto.
And generating a unique corresponding video frame number for each single-tone audio, wherein for example, the video frame number corresponding to the audio S1 is 1, the video frame number corresponding to the audio S2 is 2, and the video frame number corresponding to the audio SN is N.
Step S4012, respectively encoding the frequency value of each audio and the corresponding video frame number to obtain a plurality of video frames.
Specifically, the frequency value F of the monaural audio is combined with the corresponding video frame number and encoded to obtain a plurality of video frames. For example, the frequency value F1 of the audio S1 is combined with the corresponding video frame number 1 and encoded to obtain a two-dimensional code image, the image is used as a first video frame V1, the frequency value F2 of the audio S2 is combined with the corresponding video frame number 2 and encoded to obtain a two-dimensional code image, the two-dimensional code image is used as a second video frame V2, and the frequency value FN of the audio SN is combined with the corresponding video frame number N and encoded to obtain a two-dimensional code image, the two-dimensional code image is used as an nth video frame VN. The subsequent audio/video capture terminal 22 may obtain the frame number of the video frame and the frequency value of the corresponding audio by recognizing the two-dimensional code image of the video frame.
And step S4013, respectively coding each audio and the corresponding video frame to obtain a plurality of test audio and video segments.
Specifically, the audio S1 and the video frame V1 are coded, the audio S2 and the video frame V2, \ 8230, and the audio SN and the video frame VN are coded and sequenced in sequence, so that the test audio and video segment can be obtained.
Step S402, playing the test audio and video clip at the sending terminal of the audio and video interactive system to be tested, so that the test audio and video clip is transmitted through the sending terminal, the conference server and the receiving terminal of the audio and video interactive system to be tested, and an audio and video acquisition terminal arranged at the receiving terminal analyzes the test audio and video clip to obtain the step-out time length of the test audio and video clip transmitted through the audio and video interactive system to be tested. And the network states of the sending terminal and the receiving terminal are the same and are kept unchanged.
Specifically, after the audio/video playing terminal 21 synthesizes and plays the test audio/video segment according to the method shown in fig. 5, the receiving terminal 11 collects the video frame of the test audio/video segment through the camera, and collects the audio of the test audio/video segment through the microphone. It can be understood that the receiving terminal 11 is configured to simultaneously capture the video frame and the audio of the test audio-video segment, and no out-of-synchronization problem occurs.
Subsequently, the receiving terminal 11 transmits the test audio and video clip to the receiving terminal through the conference server 13. In the audio and video interactive system, audio and video frames are transmitted through different signal channels, so that complete synchronization of the audio and the corresponding video frames cannot be ensured, and the problem of step-out of a test audio and video clip transmitted through the audio and video interactive system to be tested can occur. The audio/video capture terminal 22 captures a picture of the display screen of the receiving terminal 12 in real time through the camera, and captures an audio played by the speaker of the receiving terminal 12 in real time through the microphone. The audio/video acquisition terminal 22 analyzes the frequency of the acquired audio and the frame number and frequency value corresponding to the video frame, respectively, and the step-out duration of the audio/video segment to be detected transmitted through the audio/video interactive system to be detected can be obtained.
According to the description of the audio and video interaction system audio and video desynchronization test method from the perspective of the audio and video playing terminal, the audio and video capture terminal obtains the corresponding relation of each audio and video frame based on the principle of the audio and video synthesis method by using the test audio and video fragments synthesized by the specific audio and video synthesis method as the data for testing the audio and video desynchronization, and determines the desynchronization duration of the audio and video fragments to be tested transmitted by the audio and video interaction system to be tested. The method and the device reduce the analysis difficulty and interference of the voice picture step-out and improve the accuracy of the voice picture step-out test result.
Fig. 6 is a schematic diagram of a third method for testing audio-visual desynchronization of an audio-video interactive system provided by the present application, and is described from the perspective of an audio-video acquisition terminal. As shown in fig. 6, the third method for testing the loss of synchronism of the sound and the picture of the audio-video interactive system comprises the following steps:
step S601, collecting video frames and audios of a plurality of test audio and video clips received by a receiving terminal of an audio and video interactive system to be tested; the test audio and video clips are generated by an audio and video playing terminal and transmitted to the receiving terminal through the sending terminal of the audio and video interactive system to be tested and the conference server.
Specifically, referring to fig. 1, after receiving a test start instruction of the master control server 23, the audio/video acquisition terminal 22 starts to acquire an audio/video clip to be tested, which is transmitted through the audio/video interaction system to be tested, for example, the audio/video acquisition terminal 22 shoots a picture of a display screen of the receiving terminal 12 through a camera, and thus, each video frame of the test audio/video clip can be acquired; audio/video capture terminal 22 captures audio signals played by the speaker of receiving terminal 12 via a microphone. For example, the audio/video capture terminal 22 captures video frames P1, P2, \ 8230;, PN by a camera, and audio S1, S2, \ 8230;, SN by a microphone.
In order to facilitate the proceeding of the subsequent steps, after step S601 is finished, the audio/video capture terminal 22 may also pre-process the captured video frames and audio. Specifically, for the video frames, the audio/video capture terminal 22 parses each captured video frame to obtain a frame number and a frequency value corresponding to each video frame and a capture timestamp, where the two-dimensional code image of each video frame is mainly identified, and since the two-dimensional code image of each video frame is encoded by the frame number of the video frame and the frequency value of the corresponding audio, the frame number and the frequency value corresponding to the video frame can be obtained by identifying the two-dimensional code image. For example, the frame number 1 and the frequency value F1 can be obtained by recognizing the two-dimensional code image of the video frame P1. Meanwhile, the audio/video acquisition terminal 22 analyzes each acquired audio to obtain a frequency value and an acquisition time stamp of each audio. For example, the frequency value F1 can be obtained by analyzing the audio S1. At this time, the parsing result and the capture timestamp of each video frame and audio may be stored.
It can be understood that the collecting time stamps of the video frames and the audio are recorded at the same time, and the purpose is to reserve the collecting time of the video frames and the audio so as to subsequently judge the corresponding relationship between the video frames and the audio of the audio-video clip to be detected transmitted through the audio-video interactive system to be detected, where the judgment basis of the corresponding relationship is the same time stamp, and the basis of determining the corresponding relationship between the video frames and the audio when the audio-video clip is synthesized in the audio-video playing terminal 21 is the same frequency value of the audio as the frequency value contained in the two-dimensional code image of the video frames.
Step S602, matching the video frame corresponding to the audio according to the frequency value of the collected audio.
In step S602, the audio and video frames are matched according to the frequency value, and thus the matching result in step S602 is the video frame corresponding to each audio when the step is not out of step. Specifically, for each audio, a video frame containing the same frequency value as the audio is matched.
The step is actually to restore the corresponding relation between the video frame and the audio of the test audio and video clip synthesized by the audio and video playing terminal before transmission, namely the corresponding relation during synchronization. For example, according to the frequency value F1 of the audio S1, the video frame P1 containing the frequency value F1 can be matched, and in the correspondence relationship during synchronization, the audio S1 corresponds to the video frame P1.
And step S603, obtaining the out-of-step duration of the test audio and video clip transmitted by the audio and video interactive system to be tested according to the frame number corresponding to the video frame matched with the audio and the frame number of the video frame acquired simultaneously with the audio.
Specifically, as shown in fig. 7, the specific step of obtaining the out-of-step duration in step S603 includes:
and step S6031, matching video frames with the same acquisition time stamps according to the acquisition time stamps of the audio.
The step is to match the video frames corresponding to the audios according to the corresponding relation between the video frames and the audios of the test audio-video clips transmitted by the audio-video interactive system to be tested. For example, assuming that at time t, the audio/video capture terminal 22 captures a video frame P2 and an audio S1, and at time v (v > t), the audio/video capture terminal 22 captures a video frame P6 and an audio S5, the video frame P6 captured at the same time v as the audio S5 is a video frame with the same capture timestamp as the audio S5, and the frame number of the video frame P6 is 6.
Step S6032, acquiring the frame number of the video frame of which the acquisition timestamp is matched with the audio.
As stated in step S6031, the frame number of the video frame P6 whose capture timestamp matches the audio is 6.
Step S6033, acquiring a frame number of a video frame of which the frequency value is matched with the audio;
the step is to match the video frames corresponding to the audios according to the corresponding relationship between the video frames and the audios of the test audio-video clips synthesized by the audio-video playing terminal 21. For example, to match the video frame corresponding to the audio S5 with the frequency value F5, the video frame P5 with the frequency value F5 included in the two-dimensional code image may be found, and the frame number of the video frame P5 is 5.
And step S6034, calculating to obtain the step-out duration according to the frequency value, the frame number of the video frame matched with the audio, the collection timestamp, the frame number of the video frame matched with the audio and the video frame rate of a test audio-video clip transmitted by the audio-video interaction system to be tested.
Specifically, the out-of-step duration = | the frame number-frequency value of the video frame of which the collection timestamp matches the audio and the frame number | ÷ frame rate of the video frame of which the audio matches.
For example, when the video frame rate is 30, the out-of-step duration = |6-5| ÷ 30 ≈ 33ms of the test audio-video segment transmitted through the audiovisual interactive system to be tested in the above example.
In practical applications, the audio and video acquisition terminal may calculate the out-of-step time lengths for a plurality of different audio frequencies, and only when a difference between two adjacent out-of-step time lengths is within a preset error threshold range or a variance of the out-of-step time lengths satisfies a preset condition, the audio and video acquisition terminal may feed back the out-of-step time length to the master control server. The error threshold value can be preset through the master control server.
According to the description of the audio and video interaction system audio and video desynchronization test method from the perspective of the audio and video playing terminal, the audio and video capture terminal obtains the corresponding relation of each audio and video frame based on the principle of the audio and video synthesis method by using the test audio and video fragments synthesized by the specific audio and video synthesis method as the data for testing the audio and video desynchronization, and determines the desynchronization duration of the audio and video fragments to be tested transmitted by the audio and video interaction system to be tested. The method and the device reduce the analysis difficulty and interference of the voice picture step-out and improve the accuracy of the voice picture step-out test result.
Finally, a complete example is taken to explain the method for testing the audio-video interaction system with the loss of synchronism of the sound painting:
1) The master control server 23 sends the preset network configuration parameters to the sending terminal 11 and the receiving terminal 12, so as to configure the network states of the sending terminal 11 and the receiving terminal 12; for example, the first set of network configuration parameters is:
the network bandwidth is 500Kbps, the packet loss rate of the conference terminal is 20%, and the network delay is 500ms; the second set of network configuration parameters is: the network bandwidth is 5MBps, the packet loss rate of the conference terminal is 0%, and the network delay is 10ms.
2) The master control server 23 sends a test start instruction to the audio/video playing terminal 21 and the audio/video acquisition terminal 22;
3) The audio/video playing terminal 21 initializes the frequency value of the single-tone audio to 100Hz according to the test start instruction
(in this example, a monophonic audio frequency with a frequency difference of 100Hz, such as 100Hz, 200Hz, and 300Hz, is taken as an example), the frame number is initialized to 1; coding 100Hz and 1 to obtain a two-dimensional code picture, namely a video frame P1, and coding the video frame P1 and a single-tone audio S1 with the frequency value of 100 Hz; then adding 1 to the frequency value and 1 to the frame number, repeating the above steps and sequencing in sequence, and synthesizing a section of test audio and video clip. A maximum value can be set for the frame number, when the frame number reaches the maximum value (such as 1000), the step 3 is not repeated any more, and the synthesized test audio/video clip is played circularly.
4) The audio/video playing terminal 21 plays the test audio/video clip through the display screen and the speaker.
5) The audio/video capture terminal 22 captures the video frame and the audio of the test audio/video clip received by the sending terminal 12 through a camera and a microphone respectively.
6) The audio and video acquisition terminal 22 decodes the two-dimensional code picture of the video frame to obtain a frequency value and a frame number corresponding to the video frame; the audio is decoded to obtain a frequency value. And recording the decoded data and the acquisition time stamps of the video frames and the audio.
7) For one audio, matching video frames with the same acquisition time stamp according to the acquisition time stamp of the audio, and acquiring the frame number of the video frame, wherein N is assumed;
8) For the same audio in the step 7), matching the video frame containing the same frequency value according to the frequency value of the audio, and acquiring the frame number of the video frame, wherein the assumption is N0;
9) Calculating out-of-step time length T = | N-N0| ÷ frame rate according to a preset out-of-step time length calculation formula
10 Step 7), step 8) and step 9) are repeated by the audio and video acquisition terminal 22 for another audio, when the difference value of the out-of-step time lengths obtained at two adjacent times is within a preset error threshold range (such as 10 ms) or the variance of a plurality of out-of-step time lengths meets a preset condition, the audio and video acquisition terminal feeds back the out-of-step time lengths to the master control server;
11 The master control server 23 generates a sound picture out-of-step report according to the received out-of-step duration and the corresponding network configuration parameters, and stores the sound picture out-of-step report into the storage space of the master control server.
Based on the same inventive concept, the embodiment of the application also provides a device for testing the audio-video interaction system audio-video desynchronization and a system for testing the audio-video interaction system audio-video desynchronization, may be used to implement the methods described in the above embodiments, as described in the following embodiments. Device for testing audio-video interaction system audio-video desynchronizing the principle of the test system for solving the problems is similar to the test method for the audio-video interaction system for the step-out of the sound picture, therefore, the implementation of the device for testing the audio-video interaction system audio-video desynchronization and the system for testing the audio-video interaction system audio-video desynchronization can refer to the implementation of the distributed short sequence number generation method, and repeated parts are not repeated. As used hereinafter, the term "unit" or "module" may be a combination of software and/or hardware that implements a predetermined function. While the system described in the embodiments below is preferably implemented in software, implementations in hardware, or a combination of software and hardware are also possible and contemplated.
The invention provides a device for testing audio and video interaction system audio and video desynchronization, wherein the functions realized by each functional module of the device correspond to the steps of the first method for testing audio and video interaction system audio and video desynchronization, as shown in figure 8, the device comprises:
the network configuration module 801 is used for configuring the network states of a sending terminal and a receiving terminal of the audio/video interactive system to be tested according to preset network configuration parameters;
the start-stop control module 802 is configured to send a test start instruction to an audio/video playing terminal disposed at the sending terminal, so that the audio/video playing terminal synthesizes and plays a test audio/video segment;
the data receiving module 803 is configured to receive the step-out duration of a test audio/video clip transmitted through the audio/video interactive system to be tested, where the test audio/video clip is sent by an audio/video acquisition terminal disposed at the receiving terminal;
the step-out duration is obtained by matching the video frame and the audio of a test audio and video clip transmitted by the audio and video interactive system to be tested by the audio and video acquisition terminal;
and the report generating module 804 is configured to generate a sound-picture out-of-step report according to the out-of-step duration and the current network state.
In an embodiment, the start-stop control module is further configured to:
receiving a test stopping instruction sent by the audio and video acquisition terminal; and
and sending the test stopping instruction to the audio and video playing terminal.
In another testing apparatus for audio-video interaction system out-of-step in sound and picture provided by the present invention, the functions implemented by each functional module of the apparatus correspond to the steps of the second testing method for audio-video interaction system out-of-step in sound and picture provided by the present application, as shown in fig. 9, the apparatus includes:
the audio and video synthesis module 901 is used for synthesizing a test audio and video segment according to a test starting instruction of the master control server;
the audio and video playing module 902 is configured to play the test audio and video clip at a sending terminal of the audio and video interactive system to be tested, so that the test audio and video clip is transmitted through the sending terminal, the conference server and a receiving terminal of the audio and video interactive system to be tested, and an audio and video acquisition terminal arranged at the receiving terminal analyzes the test audio and video clip to obtain the out-of-step time length of the test audio and video clip transmitted through the audio and video interactive system to be tested;
and the network states of the sending terminal and the receiving terminal are the same and are kept unchanged.
In an embodiment, as shown in fig. 10, the audio/video synthesis module 902 includes:
a video frame number generating unit 9021, configured to generate unique corresponding video frame numbers for multiple audio frequencies with different pre-stored frequency values respectively;
a video frame synthesizing unit 9022, configured to encode the frequency value of each audio and the corresponding video frame number respectively to obtain multiple video frames;
and the audio/video synthesis unit 9023 is configured to encode each audio and the corresponding video frame to obtain multiple test audio/video segments.
In another testing apparatus for audio-visual desynchronization of an audio-visual interactive system, which is provided by the present invention, functions realized by functional modules of the apparatus correspond to steps of a testing method for audio-visual desynchronization of a third audio-visual interactive system, which is provided by the present application, and as shown in fig. 11, the apparatus includes:
the audio/video acquisition module 111 is used for acquiring video frames and audio of a plurality of test audio/video clips received by a receiving terminal of the audio/video interactive system to be tested; the test audio/video clip is generated by an audio/video playing terminal and is transmitted to the receiving terminal through a sending terminal and a conference server of the audio/video interactive system to be tested;
the audio and video matching module 112 is configured to match a video frame corresponding to the audio according to the frequency value of the acquired audio;
and the step-out duration calculation module 113 is configured to obtain the step-out duration of the test audio and video segment transmitted by the audio and video interactive system to be tested according to the frame number corresponding to the video frame matched with the audio and the frame number of the video frame simultaneously acquired with the audio.
In an embodiment, as shown in fig. 12, the apparatus for testing the out-of-sync of the sound picture of the audio-video interactive system further includes:
the video analyzing module 114 is configured to analyze each video frame to obtain a frame number, a frequency value, and an acquisition timestamp corresponding to each video frame;
and the audio analyzing module 115 is configured to analyze each audio to obtain a frequency value and a collection timestamp of each audio.
In one embodiment, as shown in fig. 13, the step-out duration calculation module 113 includes:
a video frame determining unit 1131, configured to match video frames with the same acquisition timestamp according to the acquisition timestamp of the audio;
a video frame number determining unit 1132, configured to acquire a frame number of a video frame whose acquisition timestamp matches the audio; acquiring a frame number of a video frame with a frequency value matched with the audio;
and the step-out duration calculation unit 1133 is configured to calculate the step-out duration according to the frame number of the video frame matched with the audio by the frequency value, the frame number of the video frame matched with the audio by the acquisition timestamp, and the video frame rate of the test audio/video segment transmitted by the audio/video interaction system to be tested.
The device for testing the audio-video interaction system audio-video desynchronization realizes the accurate measurement of the audio-video interaction system audio-video desynchronization by synthesizing the special testing audio-video segment; through the network state of the terminal synchronously participating in the conference, a result report of the network state and the loss of synchronism of the sound picture is generated, and therefore developers can find the defects of the conference system and conduct automatic regression testing. The device and the system for testing the audio-video interaction system with the loss of synchronism of the sound and the picture are not limited by physical distance, and can respectively simulate the real feeling of participants in a conference before the terminals participating in the conference.
The present application further provides a system for testing audio-video interaction system audio-video loss of synchronism, which includes a device for testing audio-video interaction system audio-video loss of synchronism, as shown in fig. 1, the system includes:
the system comprises an audio and video playing terminal 21 arranged at the transmitting terminal 11, an audio and video acquisition terminal 22 arranged at the receiving terminal 12, and a master control server 23 respectively connected with the audio and video playing terminal 21, the audio and video acquisition terminal 22, the transmitting terminal 11 and the receiving terminal 12;
the total control server 23 includes:
the network configuration module is used for configuring the network states of a sending terminal and a receiving terminal of the audio and video interactive system to be tested according to preset network configuration parameters;
the start-stop control module is used for sending a test starting instruction to the audio and video playing terminal;
the data receiving module is used for receiving the out-of-step time length of the test audio and video clip transmitted by the audio and video interactive system to be tested and sent by the audio and video acquisition terminal; and
the report generation module is used for generating a sound picture desynchronizing report according to the desynchronizing time and the current network state;
the audio/video playing terminal 21 includes:
the audio and video synthesis module is used for synthesizing a test audio and video segment according to the test starting instruction of the master control server;
the audio and video playing module is used for playing the test audio and video clip at the sending terminal of the audio and video interactive system to be tested so that the test audio and video clip is transmitted through the sending terminal, the conference server and the receiving terminal of the audio and video interactive system to be tested;
the audio and video acquisition terminal 22 includes:
the audio and video acquisition module is used for acquiring the video frame and the audio of the test audio and video clip transmitted by the audio and video interactive system to be tested;
the audio and video matching module is used for matching the video frame corresponding to the audio according to the frequency value of the collected audio;
and the out-of-step duration calculation module is used for obtaining the out-of-step duration of the test audio and video clip transmitted by the audio and video interactive system to be tested according to the frame number corresponding to the video frame matched with the audio and the frame number of the video frame acquired simultaneously with the audio.
The system for testing the audio-video interaction system audio-video desynchronization provided by the invention realizes the accurate measurement of the audio-video interaction system audio-video desynchronization by synthesizing the special test audio-video clip; through the network state of the terminal synchronously participating in the conference, a result report of the network state and the loss of synchronism of the sound picture is generated, and therefore developers can find the defects of the conference system and conduct automatic regression testing. The device and the system for testing the audio-video interaction system with the loss of synchronism of the sound and the picture are not limited by physical distance, and can respectively simulate the real feeling of participants in a conference before the terminals participating in the conference.
As will be appreciated by one skilled in the art, embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the embodiments described herein may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein. The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the system embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and reference may be made to the partial description of the method embodiment for relevant points. In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of an embodiment of the specification.
In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction. The above description is only an example of the embodiments of the present disclosure, and is not intended to limit the embodiments of the present disclosure. Various modifications and alterations to the embodiments described herein will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement or the like made within the spirit and principle of the embodiments of the present invention should be included in the scope of the claims of the embodiments of the present invention.

Claims (15)

1. A method for testing the sound and picture step loss of an audio and video interactive system is characterized by comprising the following steps:
configuring the network states of a sending terminal and a receiving terminal of the audio and video interactive system to be tested according to preset network configuration parameters so as to ensure that the network states of the sending terminal and the receiving terminal of the audio and video interactive conference system to be tested are the same and are kept unchanged;
sending a test starting instruction to an audio and video playing terminal arranged at the sending terminal so that the audio and video playing terminal synthesizes and plays a test audio and video clip; the audio and video playing terminal plays the test audio and video clips for simulating the participants of the sending terminal to show the sound and the picture of the current speaker;
receiving the out-of-step time length of a test audio and video clip transmitted by the audio and video interactive system to be tested and sent by an audio and video acquisition terminal arranged at the receiving terminal; the step-out duration is obtained by matching the video frame and the audio of the test audio-video clip transmitted by the audio-video acquisition terminal based on the audio-video interactive system to be tested;
and generating a sound-picture out-of-step report according to the out-of-step duration and the current network state.
2. The method for testing the audio-visual interaction system voice-visual desynchronization according to claim 1, further comprising:
receiving a test stopping instruction sent by the audio and video acquisition terminal;
and sending the test stopping instruction to the audio and video playing terminal.
3. A method for testing the sound and picture step loss of an audio and video interactive system is characterized by comprising the following steps:
synthesizing a test audio-video clip according to a test starting instruction of the master control server;
playing the test audio and video clip at a sending terminal of the audio and video interactive system to be tested, so that the test audio and video clip is transmitted through the sending terminal, the conference server and a receiving terminal of the audio and video interactive system to be tested, and an audio and video acquisition terminal arranged at the receiving terminal analyzes the test audio and video clip to obtain the step-out time length of the test audio and video clip transmitted by the audio and video interactive system to be tested; the test audio and video clips are played at a sending terminal of the audio and video interaction system to be tested and used for simulating the participants of the sending terminal to display the sound and the picture of the current speaker;
the network states of the sending terminal and the receiving terminal are configured according to preset network configuration parameters, so that the network states of the sending terminal and the receiving terminal of the audio and video interactive conference system to be tested are the same and are kept unchanged.
4. The method for testing the audio-visual interaction system with the audio-visual desynchronizing function according to claim 3, wherein the step of synthesizing the test audio-visual segments according to the test starting instruction of the master control server comprises the following steps:
respectively generating unique corresponding video frame numbers for a plurality of audio frequencies with different pre-stored frequency values;
respectively coding the frequency value of each audio and the corresponding video frame number to obtain a plurality of video frames;
and respectively coding each audio and the corresponding video frame to obtain a plurality of test audio and video segments.
5. A method for testing the out-of-step of sound and picture of an audio and video interactive system is characterized by comprising the following steps:
collecting video frames and audios of a plurality of test audio and video clips received by a receiving terminal of an audio and video interactive system to be tested; the audio and video playing terminal plays the test audio and video clip at the sending terminal of the audio and video interactive system to be tested, and the audio and video playing terminal is used for simulating the participants of the sending terminal to show the sound and the picture of the current speaker; the test audio and video clips are transmitted to the receiving terminal through the sending terminal of the audio and video interaction system to be tested and the conference server; the network states of a sending terminal and a receiving terminal of the audio and video interactive system to be tested are configured according to preset network configuration parameters so as to ensure that the network states of the sending terminal and the receiving terminal of the audio and video interactive conference system to be tested are the same and keep unchanged;
matching the video frame corresponding to the audio according to the frequency value of the collected audio;
and obtaining the out-of-step duration of the test audio and video clip transmitted by the audio and video interactive system to be tested according to the frame number corresponding to the video frame matched with the audio and the frame number of the video frame acquired simultaneously with the audio.
6. The method for testing the audio-visual interaction system voice-visual desynchronization according to claim 5, further comprising:
analyzing each video frame to obtain a frame number, a frequency value and an acquisition timestamp corresponding to each video frame;
and analyzing each audio frequency to obtain the frequency value and the acquisition time stamp of each audio frequency.
7. The method for testing the audio-visual desynchronization of the audio-visual interactive system according to claim 6, wherein the step-out duration of the test audio-visual segment transmitted by the audio-visual interactive system to be tested is obtained according to the frame number corresponding to the video frame matched with the audio and the frame number of the video frame simultaneously acquired with the audio, and comprises the following steps:
matching video frames with the same acquisition time stamp according to the acquisition time stamp of the audio;
acquiring a frame number of a video frame of which the acquisition timestamp is matched with the audio;
acquiring a frame number of a video frame of which the frequency value is matched with the audio;
and calculating to obtain the out-of-step duration according to the frame number of the video frame with the frequency value matched with the audio, the frame number of the video frame with the acquisition timestamp matched with the audio, and the video frame rate of a test audio-video fragment transmitted by the audio-video interactive system to be tested.
8. A testing device for the audio-video interaction system with the loss of synchronism of sound and pictures is characterized by comprising:
the network configuration module is used for configuring the network states of the sending terminal and the receiving terminal of the audio-video interactive system to be tested according to preset network configuration parameters so as to ensure that the network states of the sending terminal and the receiving terminal of the audio-video interactive conference system to be tested are the same and are kept unchanged;
the start-stop control module is used for sending a test start instruction to an audio and video playing terminal arranged at the sending terminal so as to enable the audio and video playing terminal to synthesize and play a test audio and video segment; the audio and video playing terminal plays the test audio and video clips for simulating the participants of the sending terminal to display the sound and the picture of the current speaker;
the data receiving module is used for receiving the out-of-step time length of a test audio and video clip transmitted by the audio and video interactive system to be tested, which is sent by an audio and video acquisition terminal arranged at the receiving terminal;
the step-out duration is obtained by matching the video frame and the audio of a test audio and video clip transmitted by the audio and video interactive system to be tested by the audio and video acquisition terminal;
and the report generation module is used for generating a sound picture out-of-step report according to the out-of-step time length and the current network state.
9. The device for testing the audio-visual interaction system with the loss of synchronism of sound painting according to claim 8, wherein the start-stop control module is further configured to:
receiving a test stopping instruction sent by the audio and video acquisition terminal; and
and sending the test stopping instruction to the audio and video playing terminal.
10. A testing device for the audio-video interaction system with the loss of synchronism of sound and pictures is characterized by comprising:
the audio and video synthesis module is used for synthesizing a test audio and video segment according to the test starting instruction of the master control server;
the audio and video playing module is used for playing the test audio and video clip at the sending terminal of the audio and video interactive system to be tested, so that the test audio and video clip is transmitted through the sending terminal, the conference server and the receiving terminal of the audio and video interactive system to be tested, and an audio and video acquisition terminal arranged at the receiving terminal analyzes the test audio and video clip to obtain the step-out time length of the test audio and video clip transmitted through the audio and video interactive system to be tested; the test audio and video clips are played at a sending terminal of the audio and video interaction system to be tested and used for simulating the participants of the sending terminal to display the sound and the picture of the current speaker;
the network states of the sending terminal and the receiving terminal are configured according to preset network configuration parameters, so that the network states of the sending terminal and the receiving terminal of the audio and video interactive conference system to be tested are the same and are kept unchanged.
11. The apparatus for testing the audio-visual interaction system out-of-sync according to claim 10, wherein the audio-visual synthesis module comprises:
the video frame number generating unit is used for respectively generating unique corresponding video frame numbers for a plurality of audios with different pre-stored frequency values;
the video frame synthesis unit is used for respectively coding the frequency value of each audio and the corresponding video frame number to obtain a plurality of video frames;
and the audio and video synthesis unit is used for respectively coding each audio and the corresponding video frame to obtain a plurality of test audio and video segments.
12. A testing device for audio-video interaction system with out-of-step sound and picture is characterized by comprising:
the audio and video acquisition module is used for acquiring video frames and audio of a plurality of test audio and video clips received by a receiving terminal of the audio and video interactive system to be tested; the audio and video playing terminal plays the test audio and video clip at the sending terminal of the audio and video interactive system to be tested, and the audio and video playing terminal is used for simulating the participants of the sending terminal to show the sound and the picture of the current speaker; the test audio and video clips are transmitted to the receiving terminal through the sending terminal of the audio and video interaction system to be tested and the conference server; the network states of a sending terminal and a receiving terminal of the audio-video interactive system to be tested are configured according to preset network configuration parameters so as to ensure that the network states of the sending terminal and the receiving terminal of the audio-video interactive conference system to be tested are the same and are kept unchanged;
the audio and video matching module is used for matching the video frame corresponding to the audio according to the frequency value of the collected audio;
and the step-out duration calculation module is used for obtaining the step-out duration of the test audio and video clip transmitted by the audio and video interactive system to be tested according to the frame number corresponding to the video frame matched with the audio and the frame number of the video frame simultaneously acquired with the audio.
13. The apparatus for testing the audio-visual interaction system with the loss of synchronism of the sound painting according to claim 12, further comprising:
the video analysis module is used for analyzing each video frame to obtain a frame number, a frequency value and an acquisition timestamp corresponding to each video frame;
and the audio analysis module is used for analyzing each audio to obtain the frequency value and the acquisition timestamp of each audio.
14. The device for testing the audio-visual interaction system with the loss of synchronism of the sound frame according to claim 13, wherein the loss of synchronism duration calculation module includes:
the video frame determining unit is used for matching video frames with the same acquisition time stamps according to the acquisition time stamps of the audios;
the video frame number determining unit is used for acquiring the frame number of the video frame of which the acquisition timestamp is matched with the audio; acquiring a frame number of a video frame with a frequency value matched with the audio;
and the step-out duration calculation unit is used for calculating the step-out duration according to the frequency value, the frame number of the video frame matched with the audio, the collection timestamp, the frame number of the video frame matched with the audio and the video frame rate of the test audio-video clip transmitted by the audio-video interactive system to be tested.
15. A test system for audio-video interaction system with out-of-step sound and picture is characterized by comprising:
the system comprises an audio and video playing terminal arranged at a sending terminal of an audio and video interactive system to be tested, an audio and video acquisition terminal arranged at a receiving terminal of the audio and video interactive system to be tested, and a master control server respectively connected with the audio and video playing terminal, the audio and video acquisition terminal, the sending terminal and the receiving terminal;
the total control server comprises:
the network configuration module is used for configuring the network states of the sending terminal and the receiving terminal of the audio and video interactive system to be tested according to preset network configuration parameters so as to ensure that the network states of the sending terminal and the receiving terminal of the audio and video interactive conference system to be tested are the same and are kept unchanged;
the start-stop control module is used for sending a test starting instruction to the audio and video playing terminal;
the data receiving module is used for receiving the out-of-step time length of the test audio and video clip transmitted by the audio and video interactive system to be tested and sent by the audio and video acquisition terminal; and
the report generation module is used for generating a sound picture desynchronizing report according to the desynchronizing time and the current network state;
the audio and video playing terminal comprises:
the audio and video synthesis module is used for synthesizing a test audio and video segment according to the test starting instruction of the master control server;
the audio and video playing module is used for playing the test audio and video clip at the sending terminal of the audio and video interactive system to be tested so that the test audio and video clip is transmitted through the sending terminal, the conference server and the receiving terminal of the audio and video interactive system to be tested; the audio and video playing terminal plays the test audio and video clips for simulating the participants of the sending terminal to display the sound and the picture of the current speaker;
the audio and video acquisition terminal comprises:
the audio and video acquisition module is used for acquiring the video frame and the audio of the test audio and video clip transmitted by the audio and video interactive system to be tested;
the audio and video matching module is used for matching the video frame corresponding to the audio according to the frequency value of the collected audio;
and the out-of-step duration calculation module is used for obtaining the out-of-step duration of the test audio and video clip transmitted by the audio and video interactive system to be tested according to the frame number corresponding to the video frame matched with the audio and the frame number of the video frame acquired simultaneously with the audio.
CN202110509258.4A 2021-05-11 2021-05-11 Method, device and system for testing audio-video interaction system with out-of-step sound and picture Active CN113207009B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110509258.4A CN113207009B (en) 2021-05-11 2021-05-11 Method, device and system for testing audio-video interaction system with out-of-step sound and picture

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110509258.4A CN113207009B (en) 2021-05-11 2021-05-11 Method, device and system for testing audio-video interaction system with out-of-step sound and picture

Publications (2)

Publication Number Publication Date
CN113207009A CN113207009A (en) 2021-08-03
CN113207009B true CN113207009B (en) 2022-11-29

Family

ID=77030750

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110509258.4A Active CN113207009B (en) 2021-05-11 2021-05-11 Method, device and system for testing audio-video interaction system with out-of-step sound and picture

Country Status (1)

Country Link
CN (1) CN113207009B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113766215B (en) * 2021-09-07 2024-06-25 中电科航空电子有限公司 Method and system for synchronously testing broadcast of passengers in airborne passenger cabin
CN113495846A (en) * 2021-09-07 2021-10-12 浙江华创视讯科技有限公司 Terminal test system and test device based on cluster
CN114071134A (en) * 2022-01-13 2022-02-18 荣耀终端有限公司 Sound-picture synchronization detection method, sound-picture synchronization generation method, electronic equipment and storage medium
CN117061730A (en) * 2023-10-11 2023-11-14 天津华来科技股份有限公司 Method for testing sound and picture synchronization performance of intelligent camera

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1645944A (en) * 2004-12-13 2005-07-27 武汉大学 Method for testing audio and video frequency step falling out of audio and video frequency coding-decoding system
CN106358039A (en) * 2016-09-07 2017-01-25 深圳Tcl数字技术有限公司 Sound and image synchronization test method and device
CN108270622A (en) * 2018-01-22 2018-07-10 杭州当虹科技有限公司 A kind of method and system that concurrent emulation testing is carried out to video communication service system
CN109963184A (en) * 2017-12-14 2019-07-02 阿里巴巴集团控股有限公司 A kind of method, apparatus and electronic equipment of audio-video network broadcasting

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8379735B2 (en) * 2006-05-15 2013-02-19 Microsoft Corporation Automatic video glitch detection and audio-video synchronization assessment

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1645944A (en) * 2004-12-13 2005-07-27 武汉大学 Method for testing audio and video frequency step falling out of audio and video frequency coding-decoding system
CN106358039A (en) * 2016-09-07 2017-01-25 深圳Tcl数字技术有限公司 Sound and image synchronization test method and device
CN109963184A (en) * 2017-12-14 2019-07-02 阿里巴巴集团控股有限公司 A kind of method, apparatus and electronic equipment of audio-video network broadcasting
CN108270622A (en) * 2018-01-22 2018-07-10 杭州当虹科技有限公司 A kind of method and system that concurrent emulation testing is carried out to video communication service system

Also Published As

Publication number Publication date
CN113207009A (en) 2021-08-03

Similar Documents

Publication Publication Date Title
CN113207009B (en) Method, device and system for testing audio-video interaction system with out-of-step sound and picture
EP1639816B1 (en) Method and system for testing lip-synchronization of a digital television receiver
CN108600815B (en) Method and system for on-line real-time chorus
CN102577360B (en) Synchronized playback of media players
CN103077734A (en) Time alignment of recorded audio signals
JP2022550372A (en) Method and system for creating binaural immersive audio for audiovisual content
CN113766215B (en) Method and system for synchronously testing broadcast of passengers in airborne passenger cabin
CN109168059A (en) A kind of labial synchronization method playing audio & video respectively on different devices
CN110430104A (en) Test method, device, storage medium and the electronic equipment of audio transmission delay
CN114845004B (en) Audio and video synchronization implementation method and acoustic imaging method
CN111383656A (en) Voiceprint live broadcast method, voiceprint live broadcast device, server, client equipment and storage medium
CN110428798B (en) Method for synchronizing voice and accompaniment, Bluetooth device, terminal and storage medium
CN111726686B (en) Virtual karaoke system and method based on television
CN107968942B (en) Method and system for measuring audio and video time difference of live broadcast platform
Comanducci et al. Investigating networked music performances in pedagogical scenarios for the intermusic project
CN110446103B (en) Audio and video testing method and device and storage medium
CN109246664A (en) A kind of tone testing method and device
CN107135407B (en) Synchronous method and system in a kind of piano video teaching
CN116233411A (en) Method, device, equipment and computer storage medium for audio and video synchronous test
CN116962746A (en) Online chorus method and device based on continuous wheat live broadcast and online chorus system
CN110602521B (en) Method, system, computer readable medium and device for measuring mixed drawing time delay
WO2000001165A1 (en) Measurement of performance of communications systems
Celebi et al. Is virtual reality streaming ready for remote medical education? measuring latency of stereoscopic VR for telementoring
CN110753261B (en) Audio and video time delay testing method and device, computer equipment and storage medium
CN112769645A (en) Terminal testing method and device, terminal equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant