WO2023051245A1 - 视频处理方法、装置、设备及存储介质 - Google Patents
视频处理方法、装置、设备及存储介质 Download PDFInfo
- Publication number
- WO2023051245A1 WO2023051245A1 PCT/CN2022/118679 CN2022118679W WO2023051245A1 WO 2023051245 A1 WO2023051245 A1 WO 2023051245A1 CN 2022118679 W CN2022118679 W CN 2022118679W WO 2023051245 A1 WO2023051245 A1 WO 2023051245A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- video
- target
- audio
- target object
- original
- Prior art date
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 21
- 238000000034 method Methods 0.000 claims abstract description 66
- 230000008859 change Effects 0.000 claims description 44
- 239000013598 vector Substances 0.000 claims description 44
- 238000012545 processing Methods 0.000 claims description 40
- 238000003709 image segmentation Methods 0.000 claims description 28
- 230000011218 segmentation Effects 0.000 claims description 24
- 230000004913 activation Effects 0.000 claims description 22
- 230000009466 transformation Effects 0.000 claims description 22
- 238000004422 calculation algorithm Methods 0.000 claims description 14
- 238000001514 detection method Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 10
- 238000011282 treatment Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 description 17
- 238000010586 diagram Methods 0.000 description 15
- 230000000694 effects Effects 0.000 description 13
- 230000006870 function Effects 0.000 description 11
- 238000004891 communication Methods 0.000 description 7
- 241001465754 Metazoa Species 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 6
- 230000007704 transition Effects 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 230000007423 decrease Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 3
- 230000033764 rhythmic process Effects 0.000 description 3
- 235000013399 edible fruits Nutrition 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
Definitions
- Embodiments of the present disclosure relate to the technical field of image processing, for example, to a video processing method, device, device, and storage medium.
- Embodiments of the present disclosure provide a video processing method, device, device, and storage medium, which can not only improve the efficiency of video processing, but also improve the playback effect of the video, and enrich the effect of the processed video presentation.
- an embodiment of the present disclosure provides a video processing method, including:
- an embodiment of the present disclosure further provides a video processing device, including:
- the original audio acquisition module is configured to acquire original images and original audio
- An image segmentation module configured to segment the target object on the original image to obtain a target object image and a background image
- An accent recognition module configured to perform accent recognition on the original audio to obtain accent audio
- the target object image size adjustment module is configured to adjust the size of the target object image according to different adjustment ratios to obtain multiple adjusted target object images
- a target image acquisition module configured to fuse the multiple adjusted target object images with the background image respectively to obtain multiple target images
- the target video acquisition module is configured to perform audio and video encoding on the plurality of target images and the stress audio to obtain the target video.
- an embodiment of the present disclosure further provides an electronic device, and the electronic device includes:
- a storage device configured to store one or more programs
- the one or more processing devices When the one or more programs are executed by the one or more processing devices, the one or more processing devices implement the video processing method according to the embodiments of the present disclosure.
- the embodiments of the present disclosure further provide a computer-readable medium on which a computer program is stored, and when the program is executed by a processing device, the video processing method as described in the embodiments of the present disclosure is implemented.
- FIG. 1 is a flowchart of a video processing method in an embodiment of the present disclosure
- FIG. 2 is an example diagram of target object segmentation for an original image or video frame in an embodiment of the present disclosure
- Fig. 3 is a schematic diagram of an image segmentation model in an embodiment of the present disclosure
- FIG. 4 is a schematic structural diagram of a video processing device in an embodiment of the present disclosure.
- FIG. 5 is a schematic structural diagram of an electronic device in an embodiment of the present disclosure.
- FIG. 6 is a flowchart of a video processing method in another embodiment of the present disclosure.
- Fig. 7 is a schematic structural diagram of a video processing device in another embodiment of the present disclosure.
- the term “comprise” and its variations are open-ended, ie “including but not limited to”.
- the term “based on” is “based at least in part on”.
- the term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one further embodiment”; the term “some embodiments” means “at least some embodiments.” Relevant definitions of other terms will be given in the description below.
- “ghost animal” generally has the following characteristics: the same segment is played repeatedly, segment playback needs to be combined with accents, mirror flip and zoom in/out special effects, etc. will be performed. In order to achieve the above effect, it is necessary to perform the processing of the technical solution disclosed in this implementation on the picture.
- Fig. 1 is a flow chart of a video processing method provided by Embodiment 1 of the present disclosure. This embodiment is applicable to the situation of generating video based on pictures.
- the method can be executed by a video processing device, which can be implemented by hardware and/or software. Composition, and can generally be integrated in equipment with video processing functions, such equipment can be electronic equipment such as servers, mobile terminals, or server clusters. As shown in Figure 1, the method includes the following steps:
- Step 110 acquiring the original image and the original audio matched with the original image.
- the original image may be taken by the user through the camera of the smart terminal, stored locally, downloaded from a picture library in the network, or sent by other users.
- the source of the original image is not limited here.
- the original audio may be audio with a strong sense of rhythm.
- the way to obtain the original audio that matches the original image may be: obtain the original audio that matches the original image according to the user's selection operation; or identify the type information of the original image; to match the original audio.
- the method selected by the user may be audio specified by the user, which is selected by the user after obtaining the audio template provided by the APP.
- the manner of identifying the type information of the original image may be: input the original image into the type recognition model, and obtain the type to which the original image belongs.
- the type recognition model can be obtained by training a neural network. For example, after the type information of the original image is determined, a piece of audio is randomly selected from the audio library corresponding to the type information as the original audio.
- the types may include: types of natural scenery, types of people, types of buildings, and the like.
- Step 120 segment the target object on the original image to obtain a target object image and a background image.
- the target object may be a human body or a main object contained in the original image.
- the target object in the original image needs to be recognized first, and then the recognized target object and the background are segmented to obtain the target object image and the background image.
- FIG. 2 is a group of example diagrams of target object segmentation on an original image in this embodiment.
- the target object can be fruit, animal, human body or vehicle, etc.
- the process of segmenting the target object on the original image and obtaining the target object image and the background image may be: performing portrait recognition on the original image; if a portrait is recognized, the recognized portrait is determined as the target object; For the portrait, the main object is recognized in the original image, and the recognized main object is determined as the target object; the target object and the background are segmented to obtain the target object image and the background image.
- a human body is firstly used as a target object, and when there is no human figure in the original image, a saliency segmentation algorithm may be used to identify the main object in the original image. For example, first perform portrait recognition on the original image, and if a portrait is recognized, the portrait and background are segmented to obtain a human body image and a background image; Identify and segment the main object and the background to obtain the main object image and the background image.
- the portrait with the largest size ratio of the original image may be used as the target object.
- the method of segmenting the target object on the original image to obtain the target object image and the background image may also be: input the original image into the image segmentation model to obtain the target object image and the background image.
- the convolutional network is a depthwise separable convolutional network.
- Fig. 3 is a schematic diagram of an image segmentation model in this embodiment.
- the image segmentation model includes: channel switching network, channel segmentation network and depth separable convolutional network.
- the depthwise separable convolutional network includes a first-channel convolutional subnetwork, a deep convolutional subnetwork, a second-channel convolutional subnetwork, and a channel-merging layer.
- the channel switching network, the channel segmentation network, the first channel convolutional subnetwork, the deep convolutional subnetwork, the second channel convolutional subnetwork and the channel merging layer are sequentially connected; and the output of the channel slicing network and the input of the channel merging layer are skipped connect.
- the first channel convolution sub-network includes the first channel convolution layer, nonlinear activation layer and linear transformation layer;
- the depth convolution sub-network includes depthwise convolution layer (Depthwise Convolution), nonlinear activation layer and linear transformation layer;
- the second The channel convolution subnetwork includes the second channel convolution layer (Pointwise Convolution), a nonlinear activation layer and a linear transformation layer;
- the depth convolution layer consists of multiple parallel convolution kernels.
- both the first channel convolution layer and the second channel convolution layer may be composed of 1 ⁇ 1 convolution kernels.
- the depth convolution layer can be composed of a 3 ⁇ 3 convolution kernel, and the 3 ⁇ 3 convolution kernel is composed of three parallel convolution kernels, and the sizes of the three parallel convolution kernels are divided into 3 ⁇ 3, 3 ⁇ 1 and 1 ⁇ 3.
- the channel switching network can be implemented by channel shuffle, the nonlinear activation layer can be implemented by a linear rectification function (Rectified Linear Unit, ReLU), and the linear transformation layer can be implemented by a batch normalization (Batch Normalization, BN) algorithm.
- the vector field prediction model provided by this embodiment has low time-consuming work and can be applied to mobile terminals with high time-consuming requirements.
- Step 130 performing accent recognition on the original audio to obtain accent audio.
- stress can be understood as a note with a strong sense of rhythm.
- the accent recognition is performed on the original audio, and the way to obtain the accent audio can be: denoise the original audio; detect the note onset on the denoised original audio to obtain the note onset; use a peak detection algorithm Detect the peak of the original audio after denoising, and obtain the peak point that meets the set conditions; determine the accented audio according to the peak point and the start point of the note.
- the onset function can be used to detect the starting point of the note on the audio.
- the principle of peak-picking algorithm can be: obtain the waveform corresponding to the accent audio, calculate the first-order difference value of each point of the waveform, if a point satisfies: the difference value before the point is greater than 0, after the point The difference value of is less than 0, then this point can be considered as the peak point.
- the peak point it is also necessary to judge whether its amplitude is greater than the set threshold, if the amplitude of the peak point is greater than the set threshold, then the peak point is a peak point that satisfies the set condition, if the peak value The amplitude of the point is less than or equal to the set threshold, and the peak point does not meet the set condition.
- the process of determining the accent audio frequency according to the peak point and the note onset can be: obtain two note onsets adjacent to the peak point before and after, the audio frequency between the front adjacent note onset and the rear adjacent note onset is Accented audio.
- Step 140 adjusting the size of the target object image according to different adjustment ratios to obtain multiple adjusted target object images.
- the adjustment ratio may be any value greater than 1. Since the adjustment ratio is greater than 1, the adjusted target object image is larger than the original target object image.
- the adjustment ratio can first increase and then decrease according to a certain step size, so that the effect in the video is that the target object first gradually increases and then gradually decreases to the original picture. Exemplarily, assuming that there are 20 frames of images in total, the adjustment ratio of the first 15 images is changed from 1 to 2 according to the first change step, and the adjustment ratio of the last 5 images is changed from 2 to 1 according to the second change step.
- the size of the target object image is adjusted according to different adjustment ratios
- the process of obtaining multiple adjusted target object images may be: determine the required number of images according to the duration of the accent audio; determine the change of the adjustment ratio according to the number of images
- the method obtains a plurality of different adjustment ratios; respectively adjusts the size of the target object image according to the plurality of different adjustment ratios, and obtains the adjusted target object image of the number of images.
- the change method includes a change trend and a change step.
- the change trend may first increase and then decrease, and the change step size is determined by the number of images and the maximum adjustment ratio.
- the amount of rescaling is the same as the number of images.
- the duration of the accent audio can be multiplied by the frame rate of the video to obtain the required number of images. Exemplarily, assuming that the duration of the accent audio is 2s and the frame rate of the video is 15, the number of required images is 30.
- the method of changing the adjustment ratio is determined according to the number of images, and the process of obtaining multiple different adjustment ratios can be: assuming that the maximum adjustment ratio is M and the number of images is N, set the adjustment ratio of the number of images in the top a% from small to large Change, that is, change from 1 to M, then the first change step is (M-1)/(a%*N-1); after setting, the adjustment ratio of the number of images of 1-a% changes from large to small, That is, when changing from M to 1, the second change step size is (M-1)/((1-a%)*N-1).
- the target object image is sequentially adjusted according to the different adjustment ratios, thereby obtaining a plurality of adjusted target object images.
- step 150 a plurality of adjusted target object images are respectively fused with a background image to obtain a plurality of target images.
- the process of fusing multiple adjusted target object images with the background image may be: first determine the position information of the target object image in the original image, and then directly paste the target object image back into the original image according to the position, so that Get the target image.
- Step 160 perform audio-video coding on multiple target images and accented audio to obtain target video.
- the accent audio includes an accent start point and an accent end point
- multiple target images are encoded with the accent audio
- the process of obtaining the target video may be: aligning the first frame in the multiple target images with the accent start point, combining multiple The end frame in the target image is aligned with the end point of the accent; audio and video encoding is performed based on the aligned target image and the accent audio to obtain the target video.
- the following steps are also included: extracting target areas from multiple target images; performing at least one of the following processes on the target areas: randomly enlarging the target area, randomly reducing the target area Or mirror rotate the target area.
- the target area includes some or all pixels of the target object, and the center point of the target area is the pixel point of the target object.
- Randomly zooming in on the target area can be understood as being able to zoom in on any direction of the target area instead of proportionally zooming in.
- randomly shrinking the target area can be understood as being able to zoom in along any direction of the target area instead of scaling down proportionally.
- the processes performed by multiple target areas may be the same or different. For example: the target area in the first frame performs random zoom-in and mirror rotation processing, and the second frame performs random zoom-out processing, etc.
- the original image and the original audio matching the original image are acquired; the original image is segmented into the target object to obtain the target object image and the background image; the accent recognition is performed on the original audio to obtain the accent audio;
- the size of the target object image is adjusted according to different adjustment ratios to obtain multiple adjusted target object images; the multiple adjusted target object images are respectively fused with the background image to obtain multiple target images; the multiple target image Perform audio and video encoding with the accent audio to obtain the target video.
- the video processing method provided by the embodiments of the present disclosure performs audio and video encoding on the resized target object image and accented audio to obtain the target video, so that the target video has the effect of "ghost animal", which can not only improve the efficiency of video generation, but also can Enrich the rendering effect of the generated video.
- Fig. 4 is a schematic structural diagram of a video processing device provided by an embodiment of the present disclosure. As shown in Figure 4, the device includes:
- the original audio acquisition module 210 is configured to acquire the original image and the original audio matched with the original image
- the image segmentation module 220 is configured to segment the target object on the original image to obtain the target object image and the background image;
- the stress recognition module 230 is configured to carry out stress recognition to the original audio to obtain the stress audio;
- the target object image size adjustment module 240 is configured to adjust the size of the target object image according to different adjustment ratios to obtain multiple adjusted target object images;
- the target image acquisition module 250 is configured to fuse the multiple adjusted target object images with the background image respectively to obtain multiple target images
- the target video acquisition module 260 is configured to perform audio and video encoding on multiple target images and accent audio to obtain the target video.
- the original audio acquisition module 210 is also set to:
- the type information of the original image is identified; and the original audio matching the original image is acquired based on the type information.
- the image segmentation module 220 is also set to:
- the main object is identified on the original image, and the identified main object is determined as the target object;
- the stress recognition module 230 is also set to:
- the target object image size adjustment module 240 is also set to:
- the change method includes a change trend and a change step
- the size of the target object image is adjusted respectively according to a plurality of different adjustment ratios to obtain the adjusted target object image of the number of images.
- the target video acquisition module 260 is also set to:
- target area processing module set to:
- Extracting a target area from multiple target images wherein, the target area includes some or all pixels of the target object, and the center point of the target area is the pixel point of the target object;
- the image segmentation module 220 is also set to:
- the image segmentation model includes: channel switching network, channel segmentation network and depth separable convolutional network;
- the depth separable convolutional network includes a first channel convolutional subnetwork, a deep convolutional subnetwork, a second channel convolutional subnetwork and a channel merging layer;
- the channel switching network, the channel segmentation network, the first channel convolutional subnetwork, the deep convolutional subnetwork, the second channel convolutional subnetwork and the channel merging layer are sequentially connected; and the output of the channel slicing network and the input of the channel merging layer are skipped connect;
- the first channel convolution sub-network includes the first channel convolution layer, nonlinear activation layer and linear transformation layer;
- the depth convolution sub-network includes depth convolution layer, nonlinear activation layer and linear transformation layer;
- the second channel convolution sub-network The network includes a second channel convolution layer, a nonlinear activation layer, and a linear transformation layer; the depth convolution layer consists of multiple parallel convolution kernels.
- the above-mentioned device can execute the methods provided by all the foregoing embodiments of the present disclosure, and has corresponding functional modules and advantageous effects for executing the above-mentioned methods.
- the above-mentioned device can execute the methods provided by all the foregoing embodiments of the present disclosure, and has corresponding functional modules and advantageous effects for executing the above-mentioned methods.
- FIG. 5 it shows a schematic structural diagram of an electronic device 300 suitable for implementing the embodiments of the present disclosure.
- the electronic equipment in the embodiment of the present disclosure may include but not limited to such as mobile phone, notebook computer, digital broadcast receiver, PDA (personal digital assistant), PAD (tablet computer), PMP (portable multimedia player), vehicle terminal (such as Mobile terminals such as car navigation terminals) and fixed terminals such as digital TVs, desktop computers, etc., or various forms of servers, such as independent servers or server clusters.
- the electronic device shown in FIG. 5 is only an example, and should not limit the functions and scope of use of the embodiments of the present disclosure.
- an electronic device 300 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 301, which may be stored in a read-only storage device (ROM) 302 or loaded into a random access device from a storage device 305.
- ROM read-only storage device
- RAM random access device
- various appropriate actions and processes are executed by accessing programs in the storage device (RAM) 303 .
- RAM random access device
- various programs and data necessary for the operation of the electronic device 300 are also stored.
- the processing device 301, ROM 302, and RAM 303 are connected to each other through a bus 304.
- An input/output (I/O) interface 305 is also connected to the bus 304 .
- the following devices can be connected to the I/O interface 305: input devices 306 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speaker, vibrating an output device 307 such as a computer; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309.
- the communication means 309 may allow the electronic device 300 to perform wireless or wired communication with other devices to exchange data. While FIG. 5 shows electronic device 300 having various means, it should be understood that implementing or having all of the means shown is not a requirement. More or fewer means may alternatively be implemented or provided.
- embodiments of the present disclosure include a computer program product including a computer program carried on a computer-readable medium, the computer program including program code for executing a video processing method.
- the computer program may be downloaded and installed from the network via the communication means 309, or from the storage means 305, or from the ROM 302.
- the processing device 301 When the computer program is executed by the processing device 301, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.
- the above-mentioned computer-readable medium in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two.
- a computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
- a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
- a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
- the computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, and the computer-readable signal medium may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device .
- Program code embodied on a computer readable medium may be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
- the computer readable storage medium may be a non-transitory computer readable storage medium.
- the client and the server can communicate using any currently known or future network protocols such as HTTP (HyperText Transfer Protocol, Hypertext Transfer Protocol), and can communicate with digital data in any form or medium Communications (eg, communication networks) are interconnected.
- Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network of.
- the above-mentioned computer-readable medium may be included in the above-mentioned electronic device, or may exist independently without being incorporated into the electronic device.
- the above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device: acquires the original image and the original audio matched with the original image; Segmenting the target object on the original image to obtain the target object image and the background image; performing accent recognition on the original audio to obtain the accent audio; adjusting the size of the target object image according to different adjustment ratios to obtain multiple adjusted The target object image; the plurality of adjusted target object images are respectively fused with the background image to obtain a plurality of target images; the plurality of target images and the accent audio are audio-video encoded to obtain the target video.
- Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, or combinations thereof, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and Included are conventional procedural programming languages - such as the "C" language or similar programming languages.
- the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
- the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as through an Internet service provider). Internet connection).
- LAN local area network
- WAN wide area network
- Internet service provider such as AT&T, MCI, Sprint, EarthLink, MSN, GTE, etc.
- each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more logical functions for implementing specified executable instructions.
- the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved.
- each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.
- the units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. Wherein, the name of a unit does not constitute a limitation of the unit itself under certain circumstances.
- FPGAs Field Programmable Gate Arrays
- ASICs Application Specific Integrated Circuits
- ASSPs Application Specific Standard Products
- SOCs System on Chips
- CPLD Complex Programmable Logical device
- a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device.
- a machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
- a machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing.
- machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.
- RAM random access memory
- ROM read only memory
- EPROM or flash memory erasable programmable read only memory
- CD-ROM compact disk read only memory
- magnetic storage or any suitable combination of the foregoing.
- the embodiments of the present disclosure disclose a video processing method, including:
- obtaining the original audio matching the original image includes:
- the original audio matching the original image is obtained based on the type information.
- the target object is segmented on the original image to obtain the target object image and background image, including:
- the main object is identified on the original image, and the identified main object is determined as the target object;
- the target object and the background are segmented to obtain the target object image and the background image.
- performing accent recognition on the original audio to obtain accent audio includes:
- Accent audio is determined according to the peak point and the note onset point.
- the size of the target object image is adjusted according to different adjustment ratios to obtain multiple adjusted target object images, including:
- the change mode includes a change trend and a change step
- the sizes of the target object images are respectively adjusted according to the plurality of different adjustment ratios to obtain the number of adjusted target object images.
- the accent audio includes an accent start point and an accent end point
- encoding the plurality of target images and the accent audio to obtain the target video includes:
- the accent audio before performing audio and video encoding on the plurality of images and the accent audio, it also includes:
- the target object is segmented on the original image to obtain the target object image and background image, including:
- the original image is input into an image segmentation model to obtain a target object image and a background image;
- the image segmentation model includes: a channel switching network, a channel segmentation network, and a depthwise separable convolutional network;
- the depth separable convolutional network includes a first channel convolutional subnetwork, a deep convolutional subnetwork, a second channel convolutional subnetwork and a channel merging layer;
- the channel switching network, the channel splitting network, the first channel convolutional subnetwork, the deep convolutional subnetwork, the second channel convolutional subnetwork and the channel merging layer are sequentially connected; and The output of the channel segmentation network is skip-connected to the input of the channel merging layer;
- the first channel convolution sub-network includes a first channel convolution layer, a nonlinear activation layer and a linear transformation layer;
- the depth convolution sub-network includes a depth convolution layer, a nonlinear activation layer and a linear transformation layer;
- the second channel convolution sub-network includes a second channel convolution layer, a nonlinear activation layer and a linear transformation layer;
- the depth convolution layer is composed of multiple parallel convolution kernels.
- Fig. 6 is a flowchart of a video processing method provided by another embodiment of the present disclosure. This embodiment is applicable to the situation of generating a target video based on original video processing, and the method can be executed by a video processing device, which can be composed of hardware and/or software, and can generally be integrated in a device with a video processing function.
- the device may be an electronic device such as a server, a mobile terminal, or a server cluster.
- the method includes the following steps:
- Step 610 acquire the original video and the original audio matched with the original video.
- the original video may be taken by the user through the camera of the smart terminal, stored locally, downloaded from a video library in the network, or sent by other users.
- the source of the original video is not limited here.
- the original audio may be audio with a strong sense of rhythm.
- the method of obtaining the original audio matching the original video may be: obtaining the original audio matching the original video according to the user's selection operation; or identifying the type information of the original video; Match the original audio.
- the method selected by the user may be audio specified by the user, or selected by the user after the APP provides an audio template.
- the manner of identifying the type information of the original video may be: input the original video into the type recognition model, and obtain the type to which the original video belongs.
- the type recognition model can be obtained by training a neural network. For example, after determining the type information of the original video, a piece of audio is randomly selected from the audio library corresponding to the type information as the original audio.
- the types may include: types of natural scenery, types of people, types of buildings, and the like.
- Step 620 extracting video segments satisfying the set conditions from the original video to obtain target video segments.
- the target video segment may be understood as a video segment including a transition video frame, or a video frequency segment in which a gap between video frames is smaller than a certain value.
- the transition video frame can be understood as the gap between the video frame and the previous frame is greater than a certain value, for example: a video frame with other objects entering the screen; the gap between video frames is smaller than a certain value, which means that the same object has been photographed for a long time.
- the video clips that meet the set conditions are extracted from the original video
- the method of obtaining the target video clip can be: obtaining the feature vector of each video frame in the original video; clustering the feature vectors, and obtaining the clustered a plurality of initial video segments; extract video segments satisfying the set conditions from the plurality of initial video segments based on feature vectors, and obtain target video segments.
- the feature vector may be feature information that characterizes image elements and attributes included in each video frame in the original video, and may be quantified in the form of an array, for example.
- Image elements may include foreground images, background images, etc.
- attribute information may refer to at least one of information such as the structure of the image, the color, size, position, shape, and style of the image element. For example, the image of the image element in the picture layer position, the color of the image, the contrast of the image, the brightness of the image, etc.
- the method for obtaining the feature vector may include but not limited to at least one of the following: neural network method, scale-invariant feature transform (Scale-invariant feature transform, SIFT) method, accelerated robust feature (Speeded Up Robust Features, SURF) method, etc.
- neural network method Scale-invariant feature transform, SIFT
- accelerated robust feature Speeded Up Robust Features, SURF
- Clustering feature vectors where multiple video frames in each cluster are related to each other, e.g. similarity exceeds a set threshold.
- the clustering analysis method may be a k-means algorithm (k-means), a spectral clustering algorithm, and the like.
- the clustering is performed according to the image elements shown in the video frames.
- the image elements include human bodies or main objects.
- video clips that meet the set conditions are respectively extracted from a plurality of initial video clips, and the way to obtain the target video clips can be: calculate the distance between the feature vectors of adjacent video frames; When it is greater than the first threshold, then the video segment of the set duration that includes adjacent video frames is determined as the target video segment; when the video segment in the first duration satisfies the following conditions, the video segment of the first duration is Determined as the target video segment: the distances between the feature vectors of adjacent video frames are all less than the second threshold, and the distance between the feature vectors of the Nth frame and the weighted and summed feature vectors of the previous N-1 frames is less than the third threshold.
- 1 ⁇ N the number of frames included in the video segment of the first duration.
- Calculating the distance between feature vectors of adjacent video frames may be understood as: calculating the distance between feature vectors of two adjacent video frames in a video segment. The distance between the feature vectors of adjacent video frames can be calculated using the Euclidean distance formula or the Mahalanobis distance formula. If the obtained distance is greater than the first threshold, it indicates that a large change has taken place in the adjacent video frames, and the change value has exceeded the set value.
- the first threshold that is, it can be considered that a transition occurs in a video frame, and a video segment of a set duration including adjacent video frames of the transition is determined as a target video segment.
- the first duration, the first threshold, the second threshold, and the third threshold can be set according to requirements. It can be clearly seen that both the second threshold and the third threshold are smaller than the first threshold, and the second threshold and the third threshold may be the same or different.
- the feature vectors of adjacent video frames are expressed as: x1, x2, x3, ..., xn, wherein n represents the number of video frames, if the distance between the feature vector xn and x(n-1) is greater than the first A threshold value, then will comprise the video segment of the setting duration of the corresponding video frame of feature vector xn and x(n-1) and determine as target video segment, for example: select respectively x(n-1) before 2 seconds and xn rear 2 seconds second video segment, and xn and x(n-1) form the target video segment.
- the feature vectors of adjacent video frames are expressed as: x1, x2, x3, ...., xn in turn, and the corresponding weights are: p1, p2, p3, ....pn, where 1 ⁇ n ⁇ the first duration
- the number of frames contained in the video clip, if within the first duration, the feature vectors x1, x2, x3, ..., xn of adjacent video frames are all less than the second threshold, and the weighted sum of the previous n-1 frames
- step 630 the target object is segmented for each video frame of the target video segment, and target object images and background images respectively corresponding to multiple video frames are obtained.
- the target object may be a human body or a main object contained in the original video.
- the target object in the original video needs to be recognized first, and then the recognized target object and the background are segmented to obtain the target object image and the background image.
- FIG. 2 is a set of example diagrams for segmenting a video frame into a target object in this embodiment.
- the target object may be fruit, animal, human body, or vehicle.
- each video frame of the target video segment is respectively segmented into the target object
- the process of obtaining target object images and background images respectively corresponding to a plurality of video frames may be: performing portrait recognition on each video frame of the target video segment; If a portrait is identified, then the identified portrait is determined as the target object; if the portrait is not recognized, the main object is identified for each video frame of the target video clip, and the identified main object is determined as the target object;
- the target object and the background are segmented to obtain target object images and background images respectively corresponding to multiple video frames.
- the human body is firstly used as the target object.
- the saliency segmentation algorithm may be used to identify the main object in the video frame of the target video segment. For example, first perform portrait recognition on each video frame of the target video clip, if a portrait is recognized, then segment the portrait from the background to obtain a human body image and a background image; if no portrait is recognized, use the saliency segmentation algorithm
- the main object is identified in the video frame of the video clip, and the main object and the background are segmented to obtain the main object image and the background image.
- the segmentation of the target object is carried out for each video frame of the target video clip
- the mode of obtaining the target object image and the background image can also be: each video frame of the target video clip is input into the image segmentation model, and the target object image and the background image are obtained. background image.
- the convolutional network is a depthwise separable convolutional network.
- Step 640 perform accent recognition on the original audio to obtain accent audio.
- Step 650 sequentially adjust the size of the target object images in multiple video frames according to different adjustment ratios, and fuse the adjusted target object images with corresponding background images to obtain multiple target frames.
- the adjustment ratio when adjusting the size of the target object image in multiple video frames, the adjustment ratio can first increase and then decrease according to a certain step size, so that the effect in the video is that the target object first gradually increases Then gradually reduce to the original image.
- the process of sequentially adjusting the size of the target object image in multiple video frames according to different adjustment ratios may be: obtaining the number of video frames contained in the target video segment; determining the change mode of the adjustment ratio according to the number of video frames, and obtaining the video The adjustment ratio of the number of frames; according to the adjustment ratio of the number of video frames, the size of the target object image in the multiple video frames is adjusted in sequence.
- the amount of rescaling is the same as the number of video frames.
- the change mode of the adjustment ratio is determined according to the number of video frames, and the process of obtaining the adjustment ratio of the number of video frames can be as follows: Assuming that the maximum adjustment ratio is M, and the number of video frames is N, the adjustment ratio of the number of video frames of the previous a% is set according to Change from small to large, that is, from 1 to M, then the first change step is (M-1)/(a%*N-1); after setting, the adjustment ratio of 1-a% video frame number is in accordance with the large To a small change, that is, from M to 1, the second change step size is (M-1)/((1-a%)*N-1).
- the target object image is sequentially adjusted according to the different adjustment ratios, thereby obtaining a plurality of adjusted target object images.
- the position information of the target object image in the original video frame is determined, and then the target object image is directly pasted back into the original video frame according to the position, so as to obtain the target frame.
- Step 660 perform audio and video coding on multiple target frames and accent audio to obtain target video.
- the accent audio includes the accent starting point and the accent ending point
- multiple target frames are encoded with the accent audio
- the process of obtaining the target video may be: aligning the first frame of the multiple target frames with the accent starting point, and aligning multiple target frames with the accent starting point.
- the end frame in the target frame is aligned with the accent termination point; audio and video encoding is performed based on the aligned video frame and the accent audio to obtain the target video.
- multiple target frames and the accent audio are subjected to audio-video encoding
- the process of obtaining the target video may be: for each accent audio, randomly select a target video segment from one or more target video segments, Perform audio and video encoding on multiple target frames and accent audio corresponding to the selected target video segment to obtain multiple target videos; splicing multiple target videos to obtain the spliced target video.
- the following steps are also included: extracting the target area from multiple target frames; performing at least one of the following processes on the target area: randomly enlarging the target area, randomly reducing the target area Or mirror rotate the target area.
- the embodiment of the present disclosure discloses a video processing method, device, equipment and storage medium.
- the image is fused with the corresponding background image to obtain multiple target frames; audio and video encoding is performed on the multiple target frames and accent audio to obtain the target video.
- the video processing method provided by the embodiments of the present disclosure performs audio and video encoding on the resized target object image and accented audio to obtain the target video, so that the target video has the effect of "ghost animal", which can not only improve the efficiency of video processing, but also can Enrich the presentation effect of the processed video.
- Fig. 7 is a schematic structural diagram of a video processing device provided by an embodiment of the present disclosure. As shown in Figure 7, the device includes:
- the original audio acquisition module 710 is configured to obtain the original video and the original audio matched with the original video;
- the target video segment acquisition module 720 is configured to extract a video segment satisfying the set condition from the original video, and obtain the target video segment;
- the image segmentation module 730 is configured to segment the target object respectively for each video frame of the target video segment, and obtain target object images and background images respectively corresponding to a plurality of video frames;
- Accent recognition module 740 is configured to carry out accent recognition to original audio frequency, obtains accent audio frequency
- the target frame acquisition module 750 is configured to sequentially adjust the size of the target object images in multiple video frames according to different adjustment ratios, and fuse the adjusted target object images with the corresponding background images to obtain multiple target frames ;
- the target video acquisition module 760 is configured to perform audio and video encoding on multiple target frames and accent audio to obtain the target video.
- the original audio acquisition module 710 is also set to:
- Identify the type information of the original video obtain the original audio matching the original video based on the type information.
- the target video clip acquisition module 720 includes:
- a feature vector obtaining unit is configured to obtain the feature vector of each video frame in the original video
- the initial video segment acquisition unit is configured to cluster the feature vectors to obtain a plurality of initial video segments after clustering
- the target video clip acquisition unit is configured to extract video clips satisfying the set conditions from a plurality of initial video clips based on the feature vector to obtain the target video clip.
- the target video clip acquisition unit is set to:
- the video segment containing the set duration of the adjacent video frame is determined as the target video segment
- the video clips within the first duration satisfy the following conditions, the video clips of the first duration are determined as the target video clips:
- the distances between the feature vectors of adjacent video frames are all less than the second threshold, and the distance between the feature vectors of the Nth frame and the weighted and summed feature vectors of the previous N-1 frames is less than the third threshold; wherein, 1 ⁇ N ⁇ The number of frames contained in the video segment of the first duration.
- the image segmentation module 730 is also set to:
- the main object is identified for each video frame of the target video clip, and the identified main object is determined as the target object;
- the target object and the background are segmented to obtain target object images and background images respectively corresponding to multiple video frames.
- the stress recognition module 740 is also set to:
- the target frame acquisition module 750 is also set to:
- the change method includes a change trend and a change step
- the size of the target object image in the multiple video frames is adjusted sequentially according to the adjustment ratio of the number of video frames.
- the target video acquisition module 760 is also set to:
- the target video acquisition module 760 is also set to:
- the accent audio includes multiple, for each accent audio, randomly select a target video segment from one or more target video segments, and perform audio and video encoding on multiple target frames corresponding to the selected target video segment and the accent audio, Obtain multiple target videos;
- Multiple target videos are spliced to obtain a spliced target video.
- target area processing module set to:
- Extracting a target area from multiple target frames wherein, the target area includes some or all pixels of the target object, and the center point of the target area is the pixel point of the target object;
- the image segmentation module 730 is also set to:
- the image segmentation model includes: channel switching network, channel segmentation network and depth separable convolutional network;
- the depth separable convolutional network includes a first channel convolutional subnetwork, a deep convolutional subnetwork, a second channel convolutional subnetwork and a channel merging layer;
- the channel switching network, the channel segmentation network, the first channel convolutional subnetwork, the deep convolutional subnetwork, the second channel convolutional subnetwork and the channel merging layer are sequentially connected; and the output of the channel slicing network and the input of the channel merging layer are skipped connect;
- the first channel convolution sub-network includes the first channel convolution layer, nonlinear activation layer and linear transformation layer;
- the depth convolution sub-network includes depth convolution layer, nonlinear activation layer and linear transformation layer;
- the second channel convolution sub-network The network includes a second channel convolution layer, a nonlinear activation layer, and a linear transformation layer; the depth convolution layer consists of multiple parallel convolution kernels.
- the above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains the original video and the original audio matching the original video; Extract the video segment that satisfies the setting condition in the video, obtain the target video segment; Carry out the segmentation of the target object respectively to each video frame of the target video segment, obtain target object images and background images corresponding to a plurality of video frames respectively; Perform accent recognition on the original audio to obtain accent audio; adjust the size of the target object images in the plurality of video frames according to different adjustment ratios, and fuse the adjusted target object images with the corresponding background images , obtaining a plurality of target frames; performing audio and video encoding on the plurality of target frames and the accent audio to obtain a target video.
- the embodiments of the present disclosure disclose a video processing method, including:
- the original audio matching the original video including:
- extracting video segments that meet the set conditions from the original video to obtain target video segments includes:
- video clips satisfying the set conditions are respectively extracted from the plurality of initial video clips to obtain target video clips.
- video segments that meet the set conditions are respectively extracted from the plurality of initial video segments to obtain a target video segment, including:
- the video segment containing the set duration of the adjacent video frame is determined as the target video segment
- the video clips within the first duration meet the following conditions, the video clips of the first duration are determined as target video clips:
- the distances between the feature vectors of adjacent video frames are all less than the second threshold, and the distance between the feature vectors of the Nth frame and the weighted and summed feature vectors of the previous N-1 frames is less than the third threshold; wherein, 1 ⁇ N ⁇ The number of frames contained in the video segment of the first duration.
- the segmentation of the target object is performed on each video frame of the target video segment, and the corresponding target object images and background images of multiple video frames are obtained, including:
- the target object and the background are segmented to obtain target object images and background images respectively corresponding to a plurality of video frames.
- performing accent recognition on the original audio to obtain accent audio includes:
- Accent audio is determined according to the peak point and the note onset point.
- the size of the target object image in the plurality of video frames is sequentially adjusted according to different adjustment ratios, including:
- the change mode includes a change trend and a change step
- the sizes of the target object images in the plurality of video frames are sequentially adjusted according to the adjustment ratio of the number of video frames.
- the accent audio includes an accent start point and an accent end point, and performing audio-video encoding on the multiple target frames and the accent audio to obtain the target video, including:
- the multiple target frames and the accent audio are audio-video encoded to obtain the target video, including:
- a target video segment is randomly selected from one or more target video segments, and a plurality of target frames corresponding to the selected target video segment are audio-video encoded with the accent audio to obtain multiple target videos;
- the plurality of target videos are spliced to obtain a spliced target video.
- the accent audio before performing audio and video encoding on the plurality of target frames and the accent audio, it also includes:
- the segmentation of the target object is performed on each video frame of the target video segment, and the corresponding target object images and background images of multiple video frames are obtained, including:
- each video frame of the target video segment into the image segmentation model respectively, and obtain target object images and background images respectively corresponding to a plurality of video frames;
- the image segmentation model includes: channel switching network, channel segmentation network And depth separable convolutional network;
- the depth separable convolutional network includes a first channel convolutional subnetwork, a deep convolutional subnetwork, a second channel convolutional subnetwork and a channel merging layer;
- the channel switching network, the channel splitting network, the first channel convolutional subnetwork, the deep convolutional subnetwork, the second channel convolutional subnetwork and the channel merging layer are sequentially connected; and The output of the channel segmentation network is skip-connected to the input of the channel merging layer;
- the first channel convolution sub-network includes a first channel convolution layer, a nonlinear activation layer and a linear transformation layer;
- the depth convolution sub-network includes a depth convolution layer, a nonlinear activation layer and a linear transformation layer;
- the second channel convolution sub-network includes a second channel convolution layer, a nonlinear activation layer and a linear transformation layer;
- the depth convolution layer is composed of multiple parallel convolution kernels.
- the embodiment of the present disclosure discloses a video processing method, including:
- An embodiment of the present disclosure discloses a video processing device, including:
- the original audio acquisition module is configured to acquire original images and original audio
- An image segmentation module configured to segment the target object on the original image to obtain a target object image and a background image
- An accent recognition module configured to perform accent recognition on the original audio to obtain accent audio
- the target object image size adjustment module is configured to adjust the size of the target object image according to different adjustment ratios to obtain multiple adjusted target object images
- a target image acquisition module configured to fuse the multiple adjusted target object images with the background image respectively to obtain multiple target images
- the target video acquisition module is configured to perform audio and video encoding on the plurality of target images and the stress audio to obtain the target video.
- the original image is the video frame corresponding to the target video segment extracted from the original video, and the original audio matches the original video;
- the original audio acquisition module is also configured to acquire the original video
- the video processing device also includes a target video segment acquisition module, which is configured to extract video segments satisfying the set conditions from the original video to obtain the target video segment;
- the image segmentation module is also configured to segment the target object respectively for each video frame of the target video segment, and obtain target object images and background images corresponding to a plurality of video frames respectively;
- the target object image size adjustment module is also configured to sequentially adjust the size of the target object images in multiple video frames according to different adjustment ratios
- the video processing device also includes a target frame acquisition module, which is configured to fuse the adjusted target object image in multiple video frames with the corresponding background image to obtain multiple target frames;
- the target video acquisition module is also configured to perform audio and video encoding on multiple target frames and accent audio to obtain the target video.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
本公开实施例公开了一种视频处理方法、装置、设备及存储介质。获取原始图像及原始音频;对原始图像进行目标对象的分割,获得目标对象图像和背景图像;对原始音频进行重音识别,获得重音音频;对目标对象图像的尺寸按照不同的调整比例进行调整,获得多个调整后的目标对象图像;将多个调整后的目标对象图像分别与背景图像进行融合,获得多个目标图像;将多个目标图像与重音音频进行音视频编码,获得目标视频。
Description
本申请要求在2021年9月29日提交中国专利局、申请号为202111154474.8的中国专利申请,以及在2021年9月29日提交中国专利局、申请号为202111154001.8的中国专利申请的优先权,该申请的全部内容通过引用结合在本申请中。
本公开实施例涉及图像处理技术领域,例如涉及一种视频处理方法、装置、设备及存储介质。
随着智能终端中拍照技术的不断成熟,用户越来越喜欢利用智能终端进行拍照和录制视频以记录生活,这样就获得的大量的照片和视频,并将拍摄的视频发布于网络进行分享。在实际场景中,对于终端存储的照片和视频,用户喜欢进行二次加工再进行分享,例如:对照片进行精修,或者将照片制作成视频,以增加趣味性。相关技术中,通常需要用户手动对图片进行处理以生成视频,或者对视频进行编辑,不仅效率低下,且编辑后的图片和视频达不到想要的效果。
发明内容
本公开实施例提供一种视频处理方法、装置、设备及存储介质,不仅可以提高视频处理的效率,且可以提高视频的播放效果,可以丰富处理后视频呈现的效果。
第一方面,本公开实施例提供了一种视频处理方法,包括:
获取原始图像及原始音频;
对所述原始图像进行目标对象的分割,获得目标对象图像和背景图像;
对所述原始音频进行重音识别,获得重音音频;
对所述目标对象图像的尺寸按照不同的调整比例进行调整,获得多个调整后的目标对象图像;
将所述多个调整后的目标对象图像分别与所述背景图像进行融合,获得多个目标图像;
将所述多个目标图像与所述重音音频进行音视频编码,获得目标视频。
第二方面,本公开实施例还提供了一种视频处理装置,包括:
原始音频获取模块,设置为获取原始图像及原始音频;
图像分割模块,设置为对所述原始图像进行目标对象的分割,获得目标对象图像和背景图像;
重音识别模块,设置为对所述原始音频进行重音识别,获得重音音频;
目标对象图像尺寸调整模块,设置为对所述目标对象图像的尺寸按照不同的调整比例进行调整,获得多个调整后的目标对象图像;
目标图像获取模块,设置为将所述多个调整后的目标对象图像分别与所述背景图像进行融合,获得多个目标图像;
目标视频获取模块,设置为将所述多个目标图像与所述重音音频进行音视频编码,获得目标视频。
第三方面,本公开实施例还提供了一种电子设备,所述电子设备包括:
一个或多个处理装置;
存储装置,设置为存储一个或多个程序;
当所述一个或多个程序被所述一个或多个处理装置执行,使得所述一个或多个处理装置实现如本公开实施例所述的视频处理方法。
第四方面,本公开实施例还提供了一种计算机可读介质,其上存储有计算机程序,该程序被处理装置执行时实现如本公开实施例所述的视频处理方法。
图1是本公开实施例中的一种视频处理方法的流程图;
图2是本公开实施例中的对原始图像或视频帧进行目标对象分割的示例图;
图3是本公开实施例中的一种图像分割模型的示意图;
图4是本公开实施例中的一种视频处理装置的结构示意图;
图5是本公开实施例中的一种电子设备的结构示意图;
图6是本公开另一实施例中的一种视频处理方法的流程图;
图7是本公开另一实施例中的一种视频处理装置的结构示意图。
应当理解,本公开的方法实施方式中记载的多个步骤可以按照不同的顺序执行,和/或并行执行。此外,方法实施方式可以包括附加的步骤和/或省略执行示出的步骤。本公开的范围在此方面不受限制。
本文使用的术语“包括”及其变形是开放性包括,即“包括但不限于”。术语“基于”是“至少部分地基于”。术语“一个实施例”表示“至少一个实施例”;术语“另一实施例”表示“至少一个另外的实施例”;术语“一些实施例”表示“至少一些实施例”。其他术语的相关定义将在下文描述中给出。
需要注意,本公开中提及的“第一”、“第二”等概念仅用于对不同的装置、模块或单元进行区分,并非用于限定这些装置、模块或单元所执行的功能的顺序或者相互依存关系。
需要注意,本公开中提及的“一个”、“多个”的修饰是示意性而非限制性的,本领域技术人员应当理解,除非在上下文另有明确指出,否则应该理解为“一个或多个”。
本公开实施方式中的多个装置之间所交互的消息或者信息的名称仅用于说明性的目的,而并不是用于对这些消息或信息的范围进行限制。
本实施例中,要使得生成的视频具有“鬼畜”效果,“鬼畜”一般具有如下特点:相同片段重复播放、片段播放需要结合重音、会进行镜像翻转以及放大/缩小特效等。为了实现上述效果,需要对图片执行本实施公开的技术方案的处理。
图1为本公开实施例一提供的一种视频处理方法的流程图,本实施例可适用于基于图片生成视频的情况,该方法可以由视频处理装置来执行,该装置可由硬件和/或软件组成,并一般可集成在具有视频处理功能的设备中,该设备可以是服务器、移动终端或服务器集群等电子设备。如图1所示,该方法包括如下步骤:
步骤110,获取原始图像及与原始图像相匹配的原始音频。
其中,原始图像可以是用户通过智能终端的摄像头拍摄的、本地存储的、从网络中图片库中下载获得的或者是其他用户发送过来的。此处对原始图像的来源不做限定。原始音频可以是节奏感较强的音频。
本实施例中,获取与原始图像相匹配的原始音频的方式可以是:根据用户的选择操作获取与原始图像相匹配的原始音频;或者,识别原始图像的类型信息;基于类型信息获取与原始图像相匹配的原始音频。
其中,用户选择的方式可以是用户指定的音频,获得APP提供音频模板后用户选择的。
其中,识别原始图像的类型信息的方式可以是:将原始图像输入至类型识别模型中,获得原始图像所属的类型。类型识别模型可以是由设定神经网络训练获得的。例如,在确定了原始图像的类型信息后,从该类型信息对应的音频库中随机选择一段音频作为原始音频。类型可以包括:自然风景类型、人物类型、建筑物类型等。
步骤120,对原始图像进行目标对象的分割,获得目标对象图像和背景图像。
其中,目标对象可以是原始图像中包含的人体或者主体物体。本实施例中,需要首先对原始图像中的目标对象进行识别,然后将识别到的目标对象与背景进行分割,获得目标对象图像和背景图像。示例性的,图2是本实施例中一组对原始图像进行目标对象分割的示例图。如图2所示,目标对象可以是水果、动物、人体或者车辆等。
例如,对原始图像进行目标对象的分割,获得目标对象图像和背景图像的过程可以是:对原始图像进行人像识别;若识别到人像,则将识别到的人像确定为目标对象;若未识别到人像,则对原始图像进行主体物体的识别,将识别到的主体物体确定为目标对象;将目标对象与背景进行分割,获得目标对象图像和背景图像。
本实施例中,首先将人体作为目标对象,当原始图像中不存在人像时,可以采用显著性分割算法识别原始图像中的主体物体。例如,首先对原始图像中进行人像识别,若识别到人像,则将人像与背景进行分割,获得人体图像和背景图像;若未识别到人像,则采用显著性分割算法对原始图像进行主体物体的识别,并将主体物体和背景进行分割,获得主体物体图像和背景图像。
例如,若在原始图像中识别到多个人像,则可以将占原始图像的尺寸比例最大的人像作为目标对象。
例如,对原始图像进行目标对象的分割,获得目标对象图像和背景图像的方式还可以是:将原始图像输入图像分割模型中,获得目标对象图像和背景图像。
本实例中,为了模型能够部署于移动终端上,需要模型计算量小、计算高效且简单,本公开实施例中,卷积网络为深度可分卷积网络。
图3是本实施例中一种图像分割模型的示意图。如图3所示,图像分割模型包括:通道交换网络、通道切分网络及深度可分卷积网络。深度可分卷积网络包括第一通道卷积子网络、深度卷积子网络、第二通道卷积子网络和通道合并层。通道交换网络、通道切分网络、第一通道卷积子网络、深度卷积子网络、第 二通道卷积子网络和通道合并层依次连接;且通道切分网络输出与通道合并层的输入跳跃连接。第一通道卷积子网络包括第一通道卷积层、非线性激活层和线性变换层;深度卷积子网络包括深度卷积层(Depthwise Convolution)、非线性激活层和线性变换层;第二通道卷积子网络包括第二通道卷积层(Pointwise Convolution)、非线性激活层和线性变换层;深度卷积层有多个并行的卷积核组成。
其中,第一通道卷积层和第二通道卷积层均可以由1×1的卷积核构成。深度卷积层可以由3×3的卷积核构成,且3×3的卷积核是由三个并行的卷积核构成,三个并行的卷积核的尺寸分为3×3、3×1和1×3。通道交换网络可以通过channel shuffle方式实现,非线性激活层可以由线性整流函数(Rectified Linear Unit,ReLU)实现,线性变换层可以由批量标准化(Batch Normalization,BN)算法实现。本实施例提供的向量场预测模型工作耗时低,可以应用于对耗时要求高的移动终端上。
步骤130,对原始音频进行重音识别,获得重音音频。
其中,重音可以理解为节奏感较强的音符。
本实施例中,对原始音频进行重音识别,获得重音音频的方式可以是:对原始音频进行去噪处理;对去噪后的原始音频进行音符起始点检测,获得音符起始点;采用峰值检测算法对去噪后的原始音频的峰值进行检测,获得满足设定条件的峰值点;根据峰值点和音符起始点确定重音音频。
其中,可以采用onset函数对音频进行音符起始点的检测。峰值检测算法(peak-picking)的原理可以是:获取重音音频对应的波形,计算该波形每个点的一阶差分值,若某个点满足:该点之前的差分值大于0,该点之后的差分值小于0,则该点可以认为是峰值点。本实施例中,对于提取出的峰值点,还需要判断其幅值是否大于设定阈值,若峰值点的幅值大于设定阈值,则该峰值点为满足设定条件的峰值点,若峰值点的幅值小于或等于设定阈值,该峰值点不满足设定条件。
其中,根据峰值点和音符起始点确定重音音频的过程可以是:获取与该峰值点前后相邻的两个音符起始点,前相邻音符起始点和后相邻音符起始点之间的音频为重音音频。
步骤140,对目标对象图像的尺寸按照不同的调整比例进行调整,获得多个调整后的目标对象图像。
其中,调整比例可以是大于1的任意值。由于调整比例大于1,则调整后的目标对象图像大于原目标对象图像。本实施例中,在对目标对象图像的尺寸进行调整时,调整比例可以按照一定的步长先增大后减小,这样在视频中的效果为目标对象先逐渐增大再逐渐减小到原图。示例性的,假设共有20帧图像,则设置前15张图像按照第一变化步长使得调整比例从1变到2,后5张图像按照第二变化步长使得调整比例从2变到1。
例如,对目标对象图像的尺寸按照不同的调整比例进行调整,获得多个调整后的目标对象图像的过程可以是:根据重音音频的时长确定所需的图像数量;根据图像数量确定调整比例的变化方式,获得多个不同的调整比例;根据多个不同的调整比例分别对目标对象图像的尺寸进行调整,获得图像数量的调整后的目标对象图像。
其中,变化方式包括变化趋势以及变化步长。变化趋势可以是先增大后减小,变化步长由图像数量及最大调整比例来确定。调整比例的数量与图像的数量相同。本实施例中,可以将重音音频的时长与视频的帧率相乘,以获得所需的图像数量。示例性的,假设重音音频的时长为2s,视频的帧率为15,则所需的图像数量为30。
例如,根据图像数量确定调整比例的变化方式,获得多个不同的调整比例的过程可以是:假设最大调整比例为M,图像数量为N,设置前a%的图像数量的调整比例按照从小到大变化,即从1变化到M,则第一变化步长为(M-1)/(a%*N-1);设置后1-a%的图像数量的调整比例按照从大到小变化,即从M变化到1,则第二变化步长为(M-1)/((1-a%)*N-1)。在获得多个不同的调整比例后,对目标对象图像依次按照不同的调整比例进行调整,从而获得多个调整后的目标对象图像。
步骤150,将多个调整后的目标对象图像分别与背景图像进行融合,获得多个目标图像。
例如,将多个调整后的目标对象图像分别与背景图像进行融合的过程可以是:首先确定目标对象图像在原始图像的位置信息,然后按照该位置将目标对象图像直接贴回原始图像中,从而获得目标图像。
步骤160,将多个目标图像与重音音频进行音视频编码,获得目标视频。
本实施例中,需要将多个目标图像与重音音频对齐后再进行音视频编码。
其中,重音音频包括重音起始点和重音终止点,将多个目标图像与重音音频进行编码,获得目标视频的过程可以是:将多个目标图像中的首帧与重音起始点对齐,将多个目标图像中的尾帧与重音终止点对齐;基于对齐后的目标图像和重音音频进行音视频编码,获得目标视频。
其中,音视频编码的方式可以采用相关技术中任意的方式实现,此处不做限定。
例如,在将多个图像与重音音频进行音视频编码之前,还包括如下步骤:从多个目标图像中提取目标区域;对目标区域执行如下至少一项处理:随机放大目标区域、随机缩小目标区域或者对目标区域进行镜 像旋转。
其中,目标区域包含目标对象的部分或者全部像素点,且目标区域的中心点为目标对象的像素点。随机放大目标区域可以理解为可以沿目标区域的任意一方向放大,而不是等比例放大,类似的,随机缩小目标区域可以理解为可以沿目标区域的任意一方向放大,而不是等比例缩小。本实施例中,多个目标区域执行的处理可以相同也可以不同。例如:第一帧中目标区域执行随机放大和镜像旋转处理,第2帧执行随机缩小的处理等。
本公开实施例的技术方案,获取原始图像及与原始图像相匹配的原始音频;对原始图像进行目标对象的分割,获得目标对象图像和背景图像;对原始音频进行重音识别,获得重音音频;对目标对象图像的尺寸按照不同的调整比例进行调整,获得多个调整后的目标对象图像;将多个调整后的目标对象图像分别与背景图像进行融合,获得多个目标图像;将多个目标图像与重音音频进行音视频编码,获得目标视频。本公开实施例提供的视频处理方法,将调整尺寸后的目标对象图像与重音音频进行音视频编码,获得目标视频,使得目标视频具有“鬼畜”的效果,不仅可以提高视频生成的效率,且可以丰富生成视频的呈现效果。
图4是本公开实施例提供的一种视频处理装置的结构示意图。如图4所示,该装置包括:
原始音频获取模块210,设置为获取原始图像及与原始图像相匹配的原始音频;
图像分割模块220,设置为对原始图像进行目标对象的分割,获得目标对象图像和背景图像;
重音识别模块230,设置为对原始音频进行重音识别,获得重音音频;
目标对象图像尺寸调整模块240,设置为对目标对象图像的尺寸按照不同的调整比例进行调整,获得多个调整后的目标对象图像;
目标图像获取模块250,设置为将多个调整后的目标对象图像分别与背景图像进行融合,获得多个目标图像;
目标视频获取模块260,设置为将多个目标图像与重音音频进行音视频编码,获得目标视频。
例如,原始音频获取模块210,还设置为:
根据用户的选择操作获取与原始图像相匹配的原始音频;或者,
识别原始图像的类型信息;基于类型信息获取与原始图像相匹配的原始音频。
例如,图像分割模块220,还设置为:
对原始图像进行人像识别;
若识别到人像,则将识别到的人像确定为目标对象;
若未识别到人像,则对原始图像进行主体物体的识别,将识别到的主体物体确定为目标对象;
将目标对象与背景进行分割,获得目标对象图像和背景图像。
例如,重音识别模块230,还设置为:
对原始音频进行去噪处理;
对去噪后的原始音频进行音符起始点检测,获得音符起始点;
采用峰值检测算法对去噪后的原始音频的峰值进行检测,获得满足设定条件的峰值点;
根据峰值点和音符起始点确定重音音频。
例如,目标对象图像尺寸调整模块240,还设置为:
根据重音音频的时长确定所需的图像数量;
根据图像数量确定调整比例的变化方式,获得多个不同的调整比例;其中,变化方式包括变化趋势以及变化步长;
根据多个不同的调整比例分别对目标对象图像的尺寸进行调整,获得图像数量的调整后的目标对象图像。
例如,目标视频获取模块260,还设置为:
将多个目标图像中的首帧与重音起始点对齐,将多个目标图像中的尾帧与重音终止点对齐;
基于对齐后的目标图像和重音音频进行音视频编码,获得目标视频。
例如,还包括:目标区域处理模块,设置为:
从多个目标图像中提取目标区域;其中,目标区域包含目标对象的部分或者全部像素点,且目标区域的中心点为目标对象的像素点;
对目标区域执行如下至少一项处理:
随机放大目标区域、随机缩小目标区域或者对目标区域进行镜像旋转。
例如,图像分割模块220,还设置为:
将原始图像输入图像分割模型中,获得目标对象图像和背景图像;其中,图像分割模型包括:通道交换网络、通道切分网络及深度可分卷积网络;
其中,深度可分卷积网络包括第一通道卷积子网络、深度卷积子网络、第二通道卷积子网络和通道合并层;
通道交换网络、通道切分网络、第一通道卷积子网络、深度卷积子网络、第二通道卷积子网络和通道合并层依次连接;且通道切分网络输出与通道合并层的输入跳跃连接;
第一通道卷积子网络包括第一通道卷积层、非线性激活层和线性变换层;深度卷积子网络包括深度卷积层、非线性激活层和线性变换层;第二通道卷积子网络包括第二通道卷积层、非线性激活层和线性变换层;深度卷积层有多个并行的卷积核组成。
上述装置可执行本公开前述所有实施例所提供的方法,具备执行上述方法相应的功能模块和有益效果。未在本实施例中详尽描述的技术细节,可参见本公开前述所有实施例所提供的方法。
下面参考图5,其示出了适于用来实现本公开实施例的电子设备300的结构示意图。本公开实施例中的电子设备可以包括但不限于诸如移动电话、笔记本电脑、数字广播接收器、PDA(个人数字助理)、PAD(平板电脑)、PMP(便携式多媒体播放器)、车载终端(例如车载导航终端)等等的移动终端以及诸如数字TV、台式计算机等等的固定终端,或者多种形式的服务器,如独立服务器或者服务器集群。图5示出的电子设备仅仅是一个示例,不应对本公开实施例的功能和使用范围带来任何限制。
如图5所示,电子设备300可以包括处理装置(例如中央处理器、图形处理器等)301,其可以根据存储在只读存储装置(ROM)302中的程序或者从存储装置305加载到随机访问存储装置(RAM)303中的程序而执行多种适当的动作和处理。在RAM 303中,还存储有电子设备300操作所需的多种程序和数据。处理装置301、ROM 302以及RAM 303通过总线304彼此相连。输入/输出(I/O)接口305也连接至总线304。
通常,以下装置可以连接至I/O接口305:包括例如触摸屏、触摸板、键盘、鼠标、摄像头、麦克风、加速度计、陀螺仪等的输入装置306;包括例如液晶显示器(LCD)、扬声器、振动器等的输出装置307;包括例如磁带、硬盘等的存储装置308;以及通信装置309。通信装置309可以允许电子设备300与其他设备进行无线或有线通信以交换数据。虽然图5示出了具有多种装置的电子设备300,但是应理解的是,并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。
根据本公开的实施例,上文参考流程图描述的过程可以被实现为计算机软件程序。例如,本公开的实施例包括一种计算机程序产品,其包括承载在计算机可读介质上的计算机程序,该计算机程序包含用于执行视频处理方法的程序代码。在这样的实施例中,该计算机程序可以通过通信装置309从网络上被下载和安装,或者从存储装置305被安装,或者从ROM 302被安装。在该计算机程序被处理装置301执行时,执行本公开实施例的方法中限定的上述功能。
需要说明的是,本公开上述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的***、装置或器件,或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于:具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中,计算机可读存储介质可以是任何包含或存储程序的有形介质,该程序可以被指令执行***、装置或者器件使用或者与其结合使用。而在本公开中,计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号,其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式,包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质,该计算机可读信号介质可以发送、传播或者传输用于由指令执行***、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输,包括但不限于:电线、光缆、RF(射频)等等,或者上述的任意合适的组合。计算机可读存储介质可以为非暂态计算机可读存储介质。
在一些实施方式中,客户端、服务器可以利用诸如HTTP(HyperText Transfer Protocol,超文本传输协议)之类的任何当前已知或未来研发的网络协议进行通信,并且可以与任意形式或介质的数字数据通信(例如,通信网络)互连。通信网络的示例包括局域网(“LAN”),广域网(“WAN”),网际网(例如,互联网)以及端对端网络(例如,ad hoc端对端网络),以及任何当前已知或未来研发的网络。
上述计算机可读介质可以是上述电子设备中所包含的;也可以是单独存在,而未装配入该电子设备中。
上述计算机可读介质承载有一个或者多个程序,当上述一个或者多个程序被该电子设备执行时,使得该电子设备:获取原始图像及与所述原始图像相匹配的原始音频;对所述原始图像进行目标对象的分割,获得目标对象图像和背景图像;对所述原始音频进行重音识别,获得重音音频;对所述目标对象图像的尺寸按照不同的调整比例进行调整,获得多个调整后的目标对象图像;将所述多个调整后的目标对象图像分 别与所述背景图像进行融合,获得多个目标图像;将所述多个目标图像与所述重音音频进行音视频编码,获得目标视频。
可以以一种或多种程序设计语言或其组合来编写用于执行本公开的操作的计算机程序代码,上述程序设计语言包括但不限于面向对象的程序设计语言-诸如Java、Smalltalk、C++,还包括常规的过程式程序设计语言-诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中,远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)-连接到用户计算机,或者,可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图,图示了按照本公开多种实施例的***、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上,流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分,该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意,在有些作为替换的实现中,方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如,两个接连地表示的方框实际上可以基本并行地执行,它们有时也可以按相反的顺序执行,这依所涉及的功能而定。也要注意的是,框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合,可以用执行规定的功能或操作的专用的基于硬件的***来实现,或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的单元可以通过软件的方式实现,也可以通过硬件的方式来实现。其中,单元的名称在某种情况下并不构成对该单元本身的限定。
本文中以上描述的功能可以至少部分地由一个或多个硬件逻辑部件来执行。例如,非限制性地,可以使用的示范类型的硬件逻辑部件包括:现场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准产品(ASSP)、片上***(SOC)、复杂可编程逻辑设备(CPLD)等等。
在本公开的上下文中,机器可读介质可以是有形的介质,其可以包含或存储以供指令执行***、装置或设备使用或与指令执行***、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体***、装置或设备,或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
根据本公开实施例的一个或多个实施例,本公开实施例公开了一种视频处理方法,包括:
获取原始图像及与所述原始图像相匹配的原始音频;
对所述原始图像进行目标对象的分割,获得目标对象图像和背景图像;
对所述原始音频进行重音识别,获得重音音频;
对所述目标对象图像的尺寸按照不同的调整比例进行调整,获得多个调整后的目标对象图像;
将所述多个调整后的目标对象图像分别与所述背景图像进行融合,获得多个目标图像;
将所述多个目标图像与所述重音音频进行音视频编码,获得目标视频。
例如,获取与所述原始图像相匹配的原始音频,包括:
根据用户的选择操作获取与所述原始图像相匹配的原始音频;或者,
识别所述原始图像的类型信息;
基于所述类型信息获取与所述原始图像相匹配的原始音频。
例如,对所述原始图像进行目标对象的分割,获得目标对象图像和背景图像,包括:
对所述原始图像进行人像识别;
若识别到人像,则将识别到的人像确定为目标对象;
若未识别到人像,则对所述原始图像进行主体物体的识别,将识别到的主体物体确定为目标对象;
将所述目标对象与背景进行分割,获得目标对象图像和背景图像。
例如,对所述原始音频进行重音识别,获得重音音频,包括:
对所述原始音频进行去噪处理;
对去噪后的原始音频进行音符起始点检测,获得音符起始点;
采用峰值检测算法对去噪后的原始音频的峰值进行检测,获得满足设定条件的峰值点;
根据所述峰值点和所述音符起始点确定重音音频。
例如,对所述目标对象图像的尺寸按照不同的调整比例进行调整,获得多个调整后的目标对象图像,包括:
根据所述重音音频的时长确定所需的图像数量;
根据所述图像数量确定调整比例的变化方式,获得多个不同的调整比例;其中,变化方式包括变化趋势以及变化步长;
根据所述多个不同的调整比例分别对所述目标对象图像的尺寸进行调整,获得所述图像数量的调整后的目标对象图像。
例如,所述重音音频包括重音起始点和重音终止点,将所述多个目标图像与所述重音音频进行编码,获得目标视频,包括:
将所述多个目标图像中的首帧与所述重音起始点对齐,将所述多个目标图像中的尾帧与所述重音终止点对齐;
基于对齐后的目标图像和重音音频进行音视频编码,获得目标视频。
例如,在将所述多个图像与所述重音音频进行音视频编码之前,还包括:
从所述多个目标图像中提取目标区域;其中,所述目标区域包含所述目标对象的部分或者全部像素点,且所述目标区域的中心点为所述目标对象的像素点;
对所述目标区域执行如下至少一项处理:
随机放大所述目标区域、随机缩小所述目标区域或者对所述目标区域进行镜像旋转。
例如,对所述原始图像进行目标对象的分割,获得目标对象图像和背景图像,包括:
将所述原始图像输入图像分割模型中,获得目标对象图像和背景图像;其中,所述图像分割模型包括:通道交换网络、通道切分网络及深度可分卷积网络;
其中,所述深度可分卷积网络包括第一通道卷积子网络、深度卷积子网络、第二通道卷积子网络和通道合并层;
所述通道交换网络、所述通道切分网络、所述第一通道卷积子网络、所述深度卷积子网络、所述第二通道卷积子网络和所述通道合并层依次连接;且所述通道切分网络输出与所述通道合并层的输入跳跃连接;
所述第一通道卷积子网络包括第一通道卷积层、非线性激活层和线性变换层;所述深度卷积子网络包括深度卷积层、非线性激活层和线性变换层;所述第二通道卷积子网络包括第二通道卷积层、非线性激活层和线性变换层;所述深度卷积层有多个并行的卷积核组成。
图6为本公开另一实施例提供的一种视频处理方法的流程图。本实施例可适用于基于原始视频处理生成目标视频的情况,该方法可以由视频处理装置来执行,该装置可由硬件和/或软件组成,并一般可集成在具有视频处理功能的设备中,该设备可以是服务器、移动终端或服务器集群等电子设备。如图6所示,该方法包括如下步骤:
步骤610,获取原始视频及与原始视频匹配的原始音频。
其中,原始视频可以是用户通过智能终端的摄像头拍摄的、本地存储的、从网络中视频库中下载获得的或者是其他用户发送过来的。此处对原始视频的来源不做限定。原始音频可以是节奏感较强的音频。
本实施例中,获取与原始视频匹配的原始音频的方式可以是:根据用户的选择操作获取与原始视频相匹配的原始音频;或者,识别原始视频的类型信息;基于类型信息获取与原始视频相匹配的原始音频。
其中,用户选择的方式可以是用户指定的音频,或者APP提供音频模板后用户选择的。
其中,识别原始视频的类型信息的方式可以是:将原始视频输入至类型识别模型中,获得原始视频所属的类型。类型识别模型可以是由设定神经网络训练获得的。例如,在确定了原始视频的类型信息后,从该类型信息对应的音频库中随机选择一段音频作为原始音频。类型可以包括:自然风景类型、人物类型、建筑物类型等。
步骤620,从原始视频中提取满足设定条件的视频片段,获得目标视频片段。
其中,目标视频片段可以理解为包含转场视频帧的视频片段、或者视频帧间的差距小于一定值的视频频段。其中,转场视频帧可以理解为该视频帧与前一帧的差距大于一定值,例如:有其他物体进入画面的视频帧;视频帧间的差距小于一定值可以是长时间拍摄同一个物体。
本实施例中,从原始视频中提取满足设定条件的视频片段,获得目标视频片段的方式可以是:获取原始视频中每个视频帧的特征向量;对特征向量进行聚类,获得聚类后的多个初始视频片段;基于特征向量从多个初始视频片段中分别提取满足设定条件的视频片段,获得目标视频片段。
其中,特征向量可以是表征原始视频中每个视频帧包括的图像元素及属性等特征信息,例如可以以数组的形式进行量化表示。图像元素可以是包括前景图像、背景图像等,属性信息可以是指图像的结构、图像元素的颜色、尺寸、位置、形状和样式等信息中的至少一种,例如,图像元素在图片中的图层位置,该图像的颜色、图像的对比度和图像的亮度等。获取特征向量的方法可以包括但不限于以下至少一种:神经网络方法、尺度不变特征变换(Scale-invariant feature transform,SIFT)方法、加速稳健特征(Speeded Up Robust Features,SURF)方法等。
对特征向量进行聚类,其中,每个类集合中的多个视频帧彼此相关,例如相似度超过设定阈值。其中,聚类分析方法可以是k均值算法(k-means)、谱聚类算法等。例如是根据视频帧中展示的图像元素进行聚类,示例性的,图像元素包括人体或者主体物体。将每个类集合中的特征向量进行聚类,生成多个初始视频片段。
例如,本实施例中,基于特征向量从多个初始视频片段中分别提取满足设定条件的视频片段,获得目标视频片段的方式可以是:计算相邻视频帧的特征向量间的距离;在距离大于第一阈值的情况下,则将包含相邻视频帧的设定时长的视频片段确定为目标视频片段;在第一时长内的视频片段满足如下条件的情况下,将第一时长的视频片段确定为目标视频片段:相邻视频帧的特征向量间的距离均小于第二阈值,且第N帧的特征向量与前N-1帧加权求和后的特征向量间的距离小于第三阈值。
其中,1≤N≤第一时长的视频片段包含的帧数量。计算相邻视频帧的特征向量间的距离可以理解为:计算视频片段中的两两相邻视频帧的特征向量间的距离。计算相邻视频帧的特征向量间的距离可以采用欧式距离公式或者马氏距离公式计算,若得到的距离大于第一阈值,表明相邻视频帧发生了较大变化,变化值已超过设定的第一阈值,即可认为视频帧发生转场,则将包含该转场相邻视频帧的设定时长的视频片段确定为目标视频片段。若在第一时长内相邻视频帧的特征向量间的距离小于设定阈值且第N帧的特征向量与前N-1帧加权求和后的特征向量间的距离小于第三阈值,即可认为在设定时长内视频片段中的视频帧未发生转场,则将第一时长的视频片段确定为目标视频片段确定为目标视频片段。本实施例中,第一时长、第一阈值、第二阈值和第三阈值可以根据需求进行设定。可以清楚的是,第二阈值和第三阈值均小于第一阈值,第二阈值和第三阈值可以相同或者不同。
示例性的,相邻视频帧的特征向量依次表示为:x1,x2,x3,....,xn,其中n代表视频帧数量,若特征向量xn与x(n-1)的距离大于第一阈值,则将包含特征向量xn和x(n-1)对应的视频帧的设定时长的视频片段确定为目标视频片段,例如:分别选择x(n-1)前2秒以及xn后2秒的视频片段,与xn和x(n-1)组成目标视频片段。
相邻视频帧的特征向量依次表示为:x1,x2,x3,....,xn,对应权重分别为:p1,p2,p3,....pn,其中,1≤n≤第一时长的视频片段包含的帧数量,若在第一时长内,相邻视频帧的特征向量x1,x2,x3,....,xn均小于第二阈值,且前n-1帧加权求和后的特征向量可以表示为S=p1*x1+p2*x2+p3*x3+...+p(n-1)*x(n-1),特征向量xn与S的距离小于第三阈值,则将第一时长的视频片段确定为目标视频片段。其中,对于距离第n帧越近的视频帧,权重分配越大。
步骤630,对目标视频片段的每个视频帧分别进行目标对象的分割,获得多个视频帧分别对应的目标对象图像和背景图像。
其中,目标对象可以是原始视频中包含的人体或者主体物体。本实施例中,需要首先对原始视频中的目标对象进行识别,然后将识别到的目标对象与背景进行分割,获得目标对象图像和背景图像。示例性的,图2是本实施例中一组对视频帧进行目标对象分割的示例图,如图2所示,目标对象可以是水果、动物、人体或者车辆等。
例如,对目标视频片段的每个视频帧分别进行目标对象的分割,获得多个视频帧分别对应的目标对象图像和背景图像的过程可以是:对目标视频片段的每个视频帧进行人像识别;若识别到人像,则将识别到的人像确定为目标对象;若未识别到人像,则对目标视频片段的每个视频帧进行主体物体的识别,将识别到的主体物体确定为目标对象;将目标对象与背景进行分割,获得多个视频帧分别对应的目标对象图像和背景图像。
本实施例中,首先将人体作为目标对象,当目标视频片段的视频帧中不存在人像时,可以采用显著性分割算法识别目标视频片段的视频帧中的主体物体。例如,首先对目标视频片段的每个视频帧进行人像识别,若识别到人像,则将人像与背景进行分割,获得人体图像和背景图像;若未识别到人像,则采用显著性分割算法对目标视频片段的视频帧进行主体物体的识别,并将主体物体和背景进行分割,获得主体物体图像和背景图像。
例如,对目标视频片段的每个视频帧进行目标对象的分割,获得目标对象图像和背景图像的方式还可以是:将目标视频片段的每个视频帧输入图像分割模型中,获得目标对象图像和背景图像。
本实例中,为了模型能够部署于移动终端上,需要模型计算量小、计算高效且简单,本公开实施例中,卷积网络为深度可分卷积网络。
步骤640,对原始音频进行重音识别,获得重音音频。
步骤650,对多个视频帧中的目标对象图像的尺寸按照不同的调整比例依次进行调整,并将调整后的目标对象图像与对应的背景图像进行融合,获得多个目标帧。
本实施例中,在对多个视频帧中的目标对象图像的尺寸进行调整时,调整比例可以按照一定的步长先增大后减小,这样在视频中的效果为目标对象先逐渐增大再逐渐减小到原图。例如,对多个视频帧中的目标对象图像的尺寸按照不同的调整比例依次进行调整的过程可以是:获取目标视频片段包含的视频帧数量;根据视频帧数量确定调整比例的变化方式,获得视频帧数量的调整比例;根据视频帧数量的调整比例依次对多个视频帧中的目标对象图像的尺寸进行调整。
调整比例的数量与视频帧的数量相同。
例如,根据视频帧数量确定调整比例的变化方式,获得视频帧数量的调整比例的过程可以是:假设最大调整比例为M,视频帧数量为N,设置前a%的视频帧数量的调整比例按照从小到大变化,即从1变化到M,则第一变化步长为(M-1)/(a%*N-1);设置后1-a%的视频帧数量的调整比例按照从大到小变化,即从M变化到1,则第二变化步长为(M-1)/((1-a%)*N-1)。在获得多个不同的调整比例后,对目标对象图像依次按照不同的调整比例进行调整,从而获得多个调整后的目标对象图像。确定目标对象图像在原始视频帧的位置信息,然后按照该位置将目标对象图像直接贴回原始视频帧中,从而获得目标帧。
步骤660,将多个目标帧与重音音频进行音视频编码,获得目标视频。
本实施例中,需要将多个目标帧与重音音频对齐后再进行音视频编码。
其中,重音音频包括重音起始点和重音终止点,将多个目标帧与重音音频进行编码,获得目标视频的过程可以是:将多个目标帧中的首帧与重音起始点对齐,将多个目标帧中的尾帧与重音终止点对齐;基于对齐后的视频帧和重音音频进行音视频编码,获得目标视频。
例如,若重音音频均包括多个,则将多个目标帧与所述重音音频进行音视频编码,获得目标视频。
本实施例中,将多个目标帧与所述重音音频进行音视频编码,获得目标视频的过程可以是:对于每个重音音频,从一个或多个目标视频片段中随机选择一个目标视频片段,将选择的目标视频片段对应的多个目标帧与重音音频进行音视频编码,获得多个目标视频;将多个目标视频进行拼接,获得拼接后的目标视频。
例如,在将多个图像与重音音频进行音视频编码之前,还包括如下步骤:从多个目标帧中提取目标区域;对目标区域执行如下至少一项处理:随机放大目标区域、随机缩小目标区域或者对目标区域进行镜像旋转。
本公开实施例公开了一种视频处理方法、装置、设备及存储介质。获取原始视频及与原始视频匹配的原始音频;从原始视频中提取满足设定条件的视频片段,获得目标视频片段;对目标视频片段的每个视频帧分别进行目标对象的分割,获得多个视频帧分别对应的目标对象图像和背景图像;对原始音频进行重音识别,获得重音音频;对多个视频帧中的目标对象图像的尺寸按照不同的调整比例依次进行调整,并将调整后的目标对象图像与对应的背景图像进行融合,获得多个目标帧;将多个目标帧与重音音频进行音视频编码,获得目标视频。本公开实施例提供的视频处理方法,将调整尺寸后的目标对象图像与重音音频进行音视频编码,获得目标视频,使得目标视频具有“鬼畜”的效果,不仅可以提高视频处理的效率,且可以丰富处理后视频的呈现效果。
图7是本公开实施例提供的一种视频处理装置的结构示意图。如图7所示,该装置包括:
原始音频获取模块710,设置为获取原始视频及与原始视频匹配的原始音频;
目标视频片段获取模块720,设置为从原始视频中提取满足设定条件的视频片段,获得目标视频片段;
图像分割模块730,设置为对目标视频片段的每个视频帧分别进行目标对象的分割,获得多个视频帧分别对应的目标对象图像和背景图像;
重音识别模块740,设置为对原始音频进行重音识别,获得重音音频;
目标帧获取模块750,设置为对多个视频帧中的目标对象图像的尺寸按照不同的调整比例依次进行调整,并将调整后的目标对象图像与对应的背景图像进行融合,获得多个目标帧;
目标视频获取模块760,设置为将多个目标帧与重音音频进行音视频编码,获得目标视频。
例如,原始音频获取模块710,还设置为:
根据用户的选择操作获取与原始视频相匹配的原始音频;或者,
识别原始视频的类型信息;基于类型信息获取与原始视频相匹配的原始音频。
例如,目标视频片段获取模块720,包括:
特征向量获取单元,设置为获取原始视频中每个视频帧的特征向量;
初始视频片段获取单元,设置为对特征向量进行聚类,获得聚类后的多个初始视频片段;
目标视频片段获取单元,设置为基于特征向量从多个初始视频片段中分别提取满足设定条件的视频片段,获得目标视频片段。
例如,目标视频片段获取单元,设置为:
计算相邻视频帧的特征向量间的距离;
在距离大于第一阈值的情况下,则将包含相邻视频帧的设定时长的视频片段确定为目标视频片段;
在第一时长内的视频片段满足如下条件的情况下,则将第一时长的视频片段确定为目标视频片段:
相邻视频帧的特征向量间的距离均小于第二阈值,且第N帧的特征向量与前N-1帧加权求和后的特征向量间的距离小于第三阈值;其中,1≤N≤第一时长的视频片段包含的帧数量。
例如,图像分割模块730,还设置为:
对目标视频片段的每个视频帧进行人像识别;
若识别到人像,则将识别到的人像确定为目标对象;
若未识别到人像,则对目标视频片段的每个视频帧进行主体物体的识别,并将识别到的主体物体确定为目标对象;
将目标对象与背景进行分割,获得多个视频帧分别对应的目标对象图像和背景图像。
例如,重音识别模块740,还设置为:
对原始音频进行去噪处理;
对去噪后的原始音频进行音符起始点检测,获得音符起始点;
采用峰值检测算法对去噪后的原始音频的峰值进行检测,获得满足设定条件的峰值点;
根据峰值点和音符起始点确定重音音频。
例如,目标帧获取模块750,还设置为:
获取目标视频片段包含的视频帧数量;
根据视频帧数量确定调整比例的变化方式,获得视频帧数量的调整比例;其中,变化方式包括变化趋势以及变化步长;
根据视频帧数量的调整比例依次对多个视频帧中的目标对象图像的尺寸进行调整。
例如,目标视频获取模块760,还设置为:
将多个目标帧中的首帧与重音起始点对齐,将多个目标帧像中的尾帧与重音终止点对齐;
基于对齐后的视频帧和重音音频进行音视频编码,获得目标视频。
例如,目标视频获取模块760,还设置为:
若重音音频均包括多个,对于每个重音音频,从一个或多个目标视频片段中随机选择一个目标视频片段,将选择的目标视频片段对应的多个目标帧与重音音频进行音视频编码,获得多个目标视频;
将多个目标视频进行拼接,获得拼接后的目标视频。
例如,还包括:目标区域处理模块,设置为:
从多个目标帧中提取目标区域;其中,目标区域包含目标对象的部分或者全部像素点,且目标区域的中心点为目标对象的像素点;
对目标区域执行如下至少一项处理:
随机放大目标区域、随机缩小目标区域或者对目标区域进行镜像旋转。
例如,图像分割模块730,还设置为:
将目标视频片段的每个视频帧分别输入图像分割模型中,获得多个视频帧分别对应的目标对象图像和背景图像;其中,图像分割模型包括:通道交换网络、通道切分网络及深度可分卷积网络;
其中,深度可分卷积网络包括第一通道卷积子网络、深度卷积子网络、第二通道卷积子网络和通道合并层;
通道交换网络、通道切分网络、第一通道卷积子网络、深度卷积子网络、第二通道卷积子网络和通道合并层依次连接;且通道切分网络输出与通道合并层的输入跳跃连接;
第一通道卷积子网络包括第一通道卷积层、非线性激活层和线性变换层;深度卷积子网络包括深度卷积层、非线性激活层和线性变换层;第二通道卷积子网络包括第二通道卷积层、非线性激活层和线性变换层;深度卷积层有多个并行的卷积核组成。
上述计算机可读介质承载有一个或者多个程序,当上述一个或者多个程序被该电子设备执行时,使得该电子设备:获取原始视频及与所述原始视频匹配的原始音频;从所述原始视频中提取满足设定条件的视频片段,获得目标视频片段;对所述目标视频片段的每个视频帧分别进行目标对象的分割,获得多个视频帧分别对应的目标对象图像和背景图像;对所述原始音频进行重音识别,获得重音音频;对所述多个视频帧中的目标对象图像的尺寸按照不同的调整比例依次进行调整,并将调整后的目标对象图像与对应的背景图像进行融合,获得多个目标帧;将所述多个目标帧与所述重音音频进行音视频编码,获得目标视频。
根据本公开实施例的一个或多个实施例,本公开实施例公开了一种视频处理方法,包括:
获取原始视频及与所述原始视频匹配的原始音频;
从所述原始视频中提取满足设定条件的视频片段,获得目标视频片段;
对所述目标视频片段的每个视频帧分别进行目标对象的分割,获得多个视频帧分别对应的目标对象图像和背景图像;
对所述原始音频进行重音识别,获得重音音频;
对所述多个视频帧中的目标对象图像的尺寸按照不同的调整比例依次进行调整,并将调整后的目标对象图像与对应的背景图像进行融合,获得多个目标帧;
将所述多个目标帧与所述重音音频进行音视频编码,获得目标视频。
例如,获取与所述原始视频匹配的原始音频,包括:
根据用户的选择操作获取与所述原始视频相匹配的原始音频;或者,
识别所述原始视频的类型信息;
基于所述类型信息获取与所述原始视频相匹配的原始音频。
例如,从所述原始视频中提取满足设定条件的视频片段,获得目标视频片段,包括:
获取所述原始视频中每个视频帧的特征向量;
对所述特征向量进行聚类,获得聚类后的多个初始视频片段;
基于所述特征向量从所述多个初始视频片段中分别提取满足设定条件的视频片段,获得目标视频片段。
例如,基于所述特征向量从所述多个初始视频片段中分别提取满足设定条件的视频片段,获得目标视频片段,包括:
计算相邻视频帧的特征向量间的距离;
在所述距离大于第一阈值的情况下,则将包含所述相邻视频帧的设定时长的视频片段确定为目标视频片段;
在第一时长内的视频片段满足如下条件的情况下,则将所述第一时长的视频片段确定为目标视频片段:
相邻视频帧的特征向量间的距离均小于第二阈值,且第N帧的特征向量与前N-1帧加权求和后的特征向量间的距离小于第三阈值;其中,1≤N≤第一时长的视频片段包含的帧数量。
例如,对所述目标视频片段的每个视频帧分别进行目标对象的分割,获得多个视频帧分别对应的目标对象图像和背景图像,包括:
对所述目标视频片段的每个视频帧进行人像识别;
若识别到人像,则将识别到的人像确定为目标对象;
若未识别到人像,则对所述目标视频片段的每个视频帧进行主体物体的识别,并将识别到的主体物体确定为目标对象;
将所述目标对象与背景进行分割,获得多个视频帧分别对应的目标对象图像和背景图像。
例如,对所述原始音频进行重音识别,获得重音音频,包括:
对所述原始音频进行去噪处理;
对去噪后的原始音频进行音符起始点检测,获得音符起始点;
采用峰值检测算法对去噪后的原始音频的峰值进行检测,获得满足设定条件的峰值点;
根据所述峰值点和所述音符起始点确定重音音频。
例如,对所述多个视频帧中的目标对象图像的尺寸按照不同的调整比例依次进行调整,包括:
获取所述目标视频片段包含的视频帧数量;
根据所述视频帧数量确定调整比例的变化方式,获得视频帧数量的调整比例;其中,变化方式包括变化趋势以及变化步长;
根据视频帧数量的调整比例依次对所述多个视频帧中的目标对象图像的尺寸进行调整。
例如,所述重音音频包括重音起始点和重音终止点,将所述多个目标帧与所述重音音频进行音视频编码,获得目标视频,包括:
将所述多个目标帧中的首帧与所述重音起始点对齐,将所述多个目标帧像中的尾帧与所述重音终止点对齐;
基于对齐后的视频帧和重音音频进行音视频编码,获得目标视频。
例如,若所述重音音频均包括多个,则将所述多个目标帧与所述重音音频进行音视频编码,获得目标视频,包括:
对于每个重音音频,从一个或多个目标视频片段中随机选择一个目标视频片段,将选择的目标视频片段对应的多个目标帧与所述重音音频进行音视频编码,获得多个目标视频;
将所述多个目标视频进行拼接,获得拼接后的目标视频。
例如,在将所述多个目标帧与所述重音音频进行音视频编码之前,还包括:
从所述多个目标帧中提取目标区域;其中,所述目标区域包含所述目标对象的部分或者全部像素点,且所述目标区域的中心点为所述目标对象的像素点;
对所述目标区域执行如下至少一项处理:
随机放大所述目标区域、随机缩小所述目标区域或者对所述目标区域进行镜像旋转。
例如,对所述目标视频片段的每个视频帧分别进行目标对象的分割,获得多个视频帧分别对应的目标对象图像和背景图像,包括:
将所述目标视频片段的每个视频帧分别输入图像分割模型中,获得多个视频帧分别对应的目标对象图像和背景图像;其中,所述图像分割模型包括:通道交换网络、通道切分网络及深度可分卷积网络;
其中,所述深度可分卷积网络包括第一通道卷积子网络、深度卷积子网络、第二通道卷积子网络和通道合并层;
所述通道交换网络、所述通道切分网络、所述第一通道卷积子网络、所述深度卷积子网络、所述第二通道卷积子网络和所述通道合并层依次连接;且所述通道切分网络输出与所述通道合并层的输入跳跃连接;
所述第一通道卷积子网络包括第一通道卷积层、非线性激活层和线性变换层;所述深度卷积子网络包括深度卷积层、非线性激活层和线性变换层;所述第二通道卷积子网络包括第二通道卷积层、非线性激活层和线性变换层;所述深度卷积层有多个并行的卷积核组成。
本公开实施例公开了一种视频处理方法,包括:
获取原始图像及原始音频;
对所述原始图像进行目标对象的分割,获得目标对象图像和背景图像;
对所述原始音频进行重音识别,获得重音音频;
对所述目标对象图像的尺寸按照不同的调整比例进行调整,获得多个调整后的目标对象图像;
将所述多个调整后的目标对象图像分别与所述背景图像进行融合,获得多个目标图像;
将所述多个目标图像与所述重音音频进行音视频编码,获得目标视频。
本公开实施例公开了一种视频处理装置,包括:
原始音频获取模块,设置为获取原始图像及原始音频;
图像分割模块,设置为对所述原始图像进行目标对象的分割,获得目标对象图像和背景图像;
重音识别模块,设置为对所述原始音频进行重音识别,获得重音音频;
目标对象图像尺寸调整模块,设置为对所述目标对象图像的尺寸按照不同的调整比例进行调整,获得多个调整后的目标对象图像;
目标图像获取模块,设置为将所述多个调整后的目标对象图像分别与所述背景图像进行融合,获得多个目标图像;
目标视频获取模块,设置为将所述多个目标图像与所述重音音频进行音视频编码,获得目标视频。
例如,原始图像是从原始视频中提取的目标视频片段对应的视频帧,原始音频与原始视频相匹配;
原始音频获取模块,还设置为获取所述原始视频;
视频处理装置还包括目标视频片段获取模块,设置为从原始视频中提取满足设定条件的视频片段,获得目标视频片段;
图像分割模块,还设置为对目标视频片段的每个视频帧分别进行目标对象的分割,获得多个视频帧分别对应的目标对象图像和背景图像;
目标对象图像尺寸调整模块,还设置为对多个视频帧中的目标对象图像的尺寸按照不同的调整比例依次进行调整;
视频处理装置还包括目标帧获取模块,设置为将多个视频帧中调整后的目标对象图像与对应的背景图像进行融合,获得多个目标帧;
目标视频获取模块,还设置为将多个目标帧与重音音频进行音视频编码,获得目标视频。
Claims (25)
- 一种视频处理方法,包括:获取原始图像及原始音频;对所述原始图像进行目标对象的分割,获得目标对象图像和背景图像;对所述原始音频进行重音识别,获得重音音频;对所述目标对象图像的尺寸按照不同的调整比例进行调整,获得多个调整后的目标对象图像;将所述多个调整后的目标对象图像分别与所述背景图像进行融合,获得多个目标图像;将所述多个目标图像与所述重音音频进行音视频编码,获得目标视频。
- 根据权利要求1所述的方法,其中,所述原始音频与所述原始图像相匹配。
- 根据权利要求2所述的方法,其中,所述获取原始音频,包括:根据用户的选择操作获取所述原始音频;或者,识别所述原始图像的类型信息;基于所述类型信息获取所述原始音频。
- 根据权利要求2所述的方法,其中,所述对所述原始图像进行目标对象的分割,获得目标对象图像和背景图像,包括:对所述原始图像进行人像识别;响应于确定识别到人像,将识别到的人像确定为目标对象;响应于确定未识别到人像,对所述原始图像进行主体物体的识别,将识别到的主体物体确定为目标对象;将所述目标对象与背景进行分割,获得目标对象图像和背景图像。
- 根据权利要求2所述的方法,其中,所述对所述原始音频进行重音识别,获得重音音频,包括:对所述原始音频进行去噪处理;对去噪后的原始音频进行音符起始点检测,获得音符起始点;采用峰值检测算法对去噪后的原始音频的峰值进行检测,获得满足设定条件的峰值点;根据所述峰值点和所述音符起始点确定重音音频。
- 根据权利要求2所述的方法,其中,所述对所述目标对象图像的尺寸按照不同的调整比例进行调整,获得多个调整后的目标对象图像,包括:根据所述重音音频的时长确定所需的图像数量;根据所述图像数量确定调整比例的变化方式,获得多个不同的调整比例;其中,变化方式包括变化趋势以及变化步长;根据所述多个不同的调整比例分别对所述目标对象图像的尺寸进行调整,获得所述图像数量的调整后的目标对象图像。
- 根据权利要求2所述的方法,其中,所述重音音频包括重音起始点和重音终止点,所述将所述多个目标图像与所述重音音频进行编码,获得目标视频,包括:将所述多个目标图像中的首帧与所述重音起始点对齐,将所述多个目标图像中的尾帧与所述重音终止点对齐;基于对齐后的多个目标图像和重音音频进行音视频编码,获得目标视频。
- 根据权利要求2所述的方法,在所述将所述多个目标图像与所述重音音频进行音视频编码之前,还包括:从所述多个目标图像中提取目标区域;其中,所述目标区域包含所述目标对象的部分或者全部像素点,且所述目标区域的中心点为所述目标对象的像素点;对所述目标区域执行如下至少一项处理:随机放大所述目标区域、随机缩小所述目标区域,对所述目标区域进行镜像旋转。
- 根据权利要求2所述的方法,其中,所述对所述原始图像进行目标对象的分割,获得目标对象图像和背景图像,包括:将所述原始图像输入图像分割模型中,获得目标对象图像和背景图像;其中,所述图像分割模型包括:通道交换网络、通道切分网络及深度可分卷积网络;其中,所述深度可分卷积网络包括第一通道卷积子网络、深度卷积子网络、第二通道卷积子网络和通道合并层;所述通道交换网络、所述通道切分网络、所述第一通道卷积子网络、所述深度卷积子网络、所述第二通道卷积子网络和所述通道合并层依次连接;且所述通道切分网络的输出与所述通道合并层的输入跳跃连接;所述第一通道卷积子网络包括第一通道卷积层、非线性激活层和线性变换层;所述深度卷积子网络包括深度卷积层、非线性激活层和线性变换层;所述第二通道卷积子网络包括第二通道卷积层、非线性激活层和线性变换层;所述深度卷积层由多个并行的卷积核组成。
- 根据权利要求1所述的方法,其中,所述原始图像是从原始视频中提取的目标视频片段对应的视频帧,所述原始音频与所述原始视频相匹配,所述方法还包括:获取所述原始视频;从所述原始视频中提取满足设定条件的视频片段,获得所述目标视频片段;所述对所述原始图像进行目标对象的分割,获得目标对象图像和背景图像,包括:对所述目标视频片段的每个视频帧分别进行目标对象的分割,获得多个视频帧分别对应的目标对象图像和背景图像;所述对所述目标对象图像的尺寸按照不同的调整比例进行调整,包括:对所述多个视频帧中的目标对象图像的尺寸按照不同的调整比例依次进行调整;所述将所述多个目标图像与所述重音音频进行音视频编码,包括:将多个目标帧与所述重音音频进行音视频编码,获得目标视频,其中,所述多个目标帧通过将所述多个视频帧中调整后的目标对象图像与对应的背景图像进行融合获得。
- 根据权利要求10所述的方法,其中,所述获取所述原始音频,包括:根据用户的选择操作获取与所述原始视频相匹配的原始音频;或者,识别所述原始视频的类型信息;基于所述类型信息获取与所述原始视频相匹配的原始音频。
- 根据权利要求10所述的方法,其中,所述从所述原始视频中提取满足设定条件的视频片段,获得所述目标视频片段,包括:获取所述原始视频中每个视频帧的特征向量;对所述特征向量进行聚类,获得聚类后的多个初始视频片段;基于所述特征向量从所述多个初始视频片段中分别提取满足设定条件的视频片段,获得所述目标视频片段。
- 根据权利要求12所述的方法,其中,所述基于所述特征向量从所述多个初始视频片段中分别提取满足设定条件的视频片段,获得所述目标视频片段,包括:计算相邻视频帧的特征向量间的距离;响应于确定所述距离大于第一阈值,将包含所述相邻视频帧的设定时长的视频片段确定为目标视频片段;响应于确定第一时长内的视频片段满足如下条件,将所述第一时长的视频片段确定为目标视频片段:相邻视频帧的特征向量间的距离小于第二阈值,且第N帧的特征向量与前N-1帧加权求和后的特征向量间的距离小于第三阈值;其中,1≤N≤第一时长的视频片段包含的帧数量。
- 根据权利要求10所述的方法,其中,所述对所述目标视频片段的每个视频帧分别进行目标对象的分割,获得多个视频帧分别对应的目标对象图像和背景图像,包括:对所述目标视频片段的每个视频帧进行人像识别;响应于确定识别到人像,将识别到的人像确定为目标对象;响应于确定未识别到人像,对所述目标视频片段的每个视频帧进行主体物体的识别,并将识别到的主体物体确定为目标对象;将所述目标对象与背景进行分割,获得多个视频帧分别对应的目标对象图像和背景图像。
- 根据权利要求10所述的方法,其中,所述对所述原始音频进行重音识别,获得重音音频,包括:对所述原始音频进行去噪处理;对去噪后的原始音频进行音符起始点检测,获得音符起始点;采用峰值检测算法对去噪后的原始音频的峰值进行检测,获得满足设定条件的峰值点;根据所述峰值点和所述音符起始点确定重音音频。
- 根据权利要求10所述的方法,其中,所述对所述多个视频帧中的目标对象图像的尺寸按照不同的调整比例依次进行调整,包括:获取所述目标视频片段包含的视频帧数量;根据所述视频帧数量确定调整比例的变化方式,获得视频帧数量的调整比例;其中,变化方式包括变化趋势以及变化步长;根据所述视频帧数量的调整比例依次对所述多个视频帧中的目标对象图像的尺寸进行调整。
- 根据权利要求10所述的方法,其中,所述重音音频包括重音起始点和重音终止点,所述将多个 目标帧与所述重音音频进行音视频编码,获得目标视频,包括:将所述多个目标帧中的首帧与所述重音起始点对齐,将所述多个目标帧中的尾帧与所述重音终止点对齐;基于对齐后的多个视频帧和重音音频进行音视频编码,获得目标视频。
- 根据权利要求17所述的方法,其中,响应于确定所述重音音频包括多个,所述将多个目标帧与所述重音音频进行音视频编码,获得目标视频,包括:对于每个重音音频,从一个或多个目标视频片段中随机选择一个目标视频片段,将选择的目标视频片段对应的多个目标帧与所述重音音频进行音视频编码,获得多个目标视频;将所述多个目标视频进行拼接,获得拼接后的目标视频。
- 根据权利要求10所述的方法,在所述将多个目标帧与所述重音音频进行音视频编码之前,还包括:从所述多个目标帧中提取目标区域;其中,所述目标区域包含所述目标对象的部分或者全部像素点,且所述目标区域的中心点为所述目标对象的像素点;对所述目标区域执行如下至少一项处理:随机放大所述目标区域、随机缩小所述目标区域,对所述目标区域进行镜像旋转。
- 根据权利要求10所述的方法,其中,所述对所述目标视频片段的每个视频帧分别进行目标对象的分割,获得多个视频帧分别对应的目标对象图像和背景图像,包括:将所述目标视频片段的每个视频帧分别输入图像分割模型中,获得多个视频帧分别对应的目标对象图像和背景图像;其中,所述图像分割模型包括:通道交换网络、通道切分网络及深度可分卷积网络;其中,所述深度可分卷积网络包括第一通道卷积子网络、深度卷积子网络、第二通道卷积子网络和通道合并层;所述通道交换网络、所述通道切分网络、所述第一通道卷积子网络、所述深度卷积子网络、所述第二通道卷积子网络和所述通道合并层依次连接;且所述通道切分网络的输出与所述通道合并层的输入跳跃连接;所述第一通道卷积子网络包括第一通道卷积层、非线性激活层和线性变换层;所述深度卷积子网络包括深度卷积层、非线性激活层和线性变换层;所述第二通道卷积子网络包括第二通道卷积层、非线性激活层和线性变换层;所述深度卷积层由多个并行的卷积核组成。
- 一种视频处理装置,包括:原始音频获取模块,设置为获取原始图像及原始音频;图像分割模块,设置为对所述原始图像进行目标对象的分割,获得目标对象图像和背景图像;重音识别模块,设置为对所述原始音频进行重音识别,获得重音音频;目标对象图像尺寸调整模块,设置为对所述目标对象图像的尺寸按照不同的调整比例进行调整,获得多个调整后的目标对象图像;目标图像获取模块,设置为将所述多个调整后的目标对象图像分别与所述背景图像进行融合,获得多个目标图像;目标视频获取模块,设置为将所述多个目标图像与所述重音音频进行音视频编码,获得目标视频。
- 根据权利要求21所述的装置,其中,所述原始音频与所述原始图像相匹配。
- 根据权利要求21所述的装置,其中,所述原始图像是从原始视频中提取的目标视频片段对应的视频帧,所述原始音频与所述原始视频相匹配;所述原始音频获取模块,还设置为获取所述原始视频;所述装置还包括目标视频片段获取模块,设置为从所述原始视频中提取满足设定条件的视频片段,获得目标视频片段;所述图像分割模块,还设置为对所述目标视频片段的每个视频帧分别进行目标对象的分割,获得多个视频帧分别对应的目标对象图像和背景图像;所述目标对象图像尺寸调整模块,还设置为对所述多个视频帧中的目标对象图像的尺寸按照不同的调整比例依次进行调整;所述装置还包括目标帧获取模块,设置为将所述多个视频帧中调整后的目标对象图像与对应的背景图像进行融合,获得多个目标帧;所述目标视频获取模块,还设置为将所述多个目标帧与所述重音音频进行音视频编码,获得目标视频。
- 一种电子设备,包括:一个或多个处理装置;存储装置,设置为存储一个或多个程序;当所述一个或多个程序被所述一个或多个处理装置执行,使得所述一个或多个处理装置实现如权利要求1-20中任一所述的视频处理方法。
- 一种计算机可读介质,其上存储有计算机程序,所述计算机程序被处理装置执行时实现如权利要求1-20中任一所述的视频处理方法。
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202111154001.8A CN113905177B (zh) | 2021-09-29 | 2021-09-29 | 视频生成方法、装置、设备及存储介质 |
CN202111154474.8 | 2021-09-29 | ||
CN202111154474.8A CN113923378B (zh) | 2021-09-29 | 2021-09-29 | 视频处理方法、装置、设备及存储介质 |
CN202111154001.8 | 2021-09-29 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2023051245A1 true WO2023051245A1 (zh) | 2023-04-06 |
Family
ID=85780422
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2022/118679 WO2023051245A1 (zh) | 2021-09-29 | 2022-09-14 | 视频处理方法、装置、设备及存储介质 |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2023051245A1 (zh) |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100110232A1 (en) * | 2008-10-31 | 2010-05-06 | Fortemedia, Inc. | Electronic apparatus and method for receiving sounds with auxiliary information from camera system |
CN110336960A (zh) * | 2019-07-17 | 2019-10-15 | 广州酷狗计算机科技有限公司 | 视频合成的方法、装置、终端及存储介质 |
CN111065001A (zh) * | 2019-12-25 | 2020-04-24 | 广州酷狗计算机科技有限公司 | 视频制作的方法、装置、设备及存储介质 |
CN113055738A (zh) * | 2019-12-26 | 2021-06-29 | 北京字节跳动网络技术有限公司 | 视频特效处理方法及装置 |
CN113905177A (zh) * | 2021-09-29 | 2022-01-07 | 北京字跳网络技术有限公司 | 视频生成方法、装置、设备及存储介质 |
CN113923378A (zh) * | 2021-09-29 | 2022-01-11 | 北京字跳网络技术有限公司 | 视频处理方法、装置、设备及存储介质 |
-
2022
- 2022-09-14 WO PCT/CN2022/118679 patent/WO2023051245A1/zh active Application Filing
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100110232A1 (en) * | 2008-10-31 | 2010-05-06 | Fortemedia, Inc. | Electronic apparatus and method for receiving sounds with auxiliary information from camera system |
CN110336960A (zh) * | 2019-07-17 | 2019-10-15 | 广州酷狗计算机科技有限公司 | 视频合成的方法、装置、终端及存储介质 |
CN111065001A (zh) * | 2019-12-25 | 2020-04-24 | 广州酷狗计算机科技有限公司 | 视频制作的方法、装置、设备及存储介质 |
CN113055738A (zh) * | 2019-12-26 | 2021-06-29 | 北京字节跳动网络技术有限公司 | 视频特效处理方法及装置 |
CN113905177A (zh) * | 2021-09-29 | 2022-01-07 | 北京字跳网络技术有限公司 | 视频生成方法、装置、设备及存储介质 |
CN113923378A (zh) * | 2021-09-29 | 2022-01-11 | 北京字跳网络技术有限公司 | 视频处理方法、装置、设备及存储介质 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP7228682B2 (ja) | 動画解析のためのゲーティングモデル | |
WO2020119350A1 (zh) | 视频分类方法、装置、计算机设备和存储介质 | |
CN106682632B (zh) | 用于处理人脸图像的方法和装置 | |
WO2020082870A1 (zh) | 即时视频显示方法、装置、终端设备及存储介质 | |
JP7222008B2 (ja) | 動画クリップの検索方法及び装置 | |
WO2023125374A1 (zh) | 图像处理方法、装置、电子设备及存储介质 | |
CN110446066B (zh) | 用于生成视频的方法和装置 | |
WO2021254502A1 (zh) | 目标对象显示方法、装置及电子设备 | |
CN111491187B (zh) | 视频的推荐方法、装置、设备及存储介质 | |
JP7209851B2 (ja) | 画像変形の制御方法、装置およびハードウェア装置 | |
WO2020259449A1 (zh) | 一种短视频的生成方法及装置 | |
WO2019227429A1 (zh) | 多媒体内容生成方法、装置和设备/终端/服务器 | |
WO2021057740A1 (zh) | 视频生成方法、装置、电子设备和计算机可读介质 | |
CN113923378B (zh) | 视频处理方法、装置、设备及存储介质 | |
EP4207195A1 (en) | Speech separation method, electronic device, chip and computer-readable storage medium | |
KR102550305B1 (ko) | 머신 러닝 기반의 영상 자동 편집 방법 및 시스템 | |
WO2021190625A1 (zh) | 拍摄方法和设备 | |
WO2023197648A1 (zh) | 截图处理方法及装置、电子设备和计算机可读介质 | |
CN112785669B (zh) | 一种虚拟形象合成方法、装置、设备及存储介质 | |
CN113905177B (zh) | 视频生成方法、装置、设备及存储介质 | |
CN111507142A (zh) | 人脸表情图像处理方法、装置和电子设备 | |
WO2023088029A1 (zh) | 一种封面生成方法、装置、设备及介质 | |
WO2023138441A1 (zh) | 视频生成方法、装置、设备及存储介质 | |
WO2023051245A1 (zh) | 视频处理方法、装置、设备及存储介质 | |
WO2022262473A1 (zh) | 图像处理方法、装置、设备及存储介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22874632 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 18569566 Country of ref document: US |
|
NENP | Non-entry into the national phase |
Ref country code: DE |