WO2019095979A1 - 视频图像处理方法、装置及终端 - Google Patents

视频图像处理方法、装置及终端 Download PDF

Info

Publication number
WO2019095979A1
WO2019095979A1 PCT/CN2018/112304 CN2018112304W WO2019095979A1 WO 2019095979 A1 WO2019095979 A1 WO 2019095979A1 CN 2018112304 W CN2018112304 W CN 2018112304W WO 2019095979 A1 WO2019095979 A1 WO 2019095979A1
Authority
WO
WIPO (PCT)
Prior art keywords
coordinates
canvas
video image
target
screen
Prior art date
Application number
PCT/CN2018/112304
Other languages
English (en)
French (fr)
Inventor
罗爽
Original Assignee
腾讯科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 腾讯科技（深圳）有限公司 filed Critical 腾讯科技（深圳）有限公司
Priority to EP18879128.9A priority Critical patent/EP3713220B1/en
Publication of WO2019095979A1 publication Critical patent/WO2019095979A1/zh
Priority to US16/854,807 priority patent/US11140339B2/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/2621—Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04845—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range for image manipulation, e.g. dragging, rotation, expansion or change of colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/60—Editing figures and text; Combining figures or text
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G—PHYSICS
    • G11—INFORMATION STORAGE
    • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/02—Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
    • G11B27/031—Electronic editing of digitised analogue information signals, e.g. audio or video signals
    • G—PHYSICS
    • G11—INFORMATION STORAGE
    • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • G11B27/34—Indicating arrangements 
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/426—Internal components of the client ; Characteristics thereof
    • H04N21/42653—Internal components of the client ; Characteristics thereof for processing graphics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H04N21/4312—Generation of visual interfaces for content selection or interaction; Content or additional data rendering involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
    • H04N21/4316—Generation of visual interfaces for content selection or interaction; Content or additional data rendering involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
    • H04N21/4335—Housekeeping operations, e.g. prioritizing content for deletion because of storage space restrictions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
    • H04N21/47205—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/8146—Monomedia components thereof involving graphical data, e.g. 3D object, 2D graphics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/61—Control of cameras or camera modules based on recognised objects
    • H04N23/611—Control of cameras or camera modules based on recognised objects where the recognised objects include parts of the human body
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/631—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/631—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
    • H04N23/632—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/265—Mixing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/24—Indexing scheme for image data processing or generation, in general involving graphical user interfaces [GUIs]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation

Definitions

  • the present application relates to the field of image processing technologies, and in particular, to a video image processing method, apparatus, and terminal.
  • the source of the special effect material is mainly that the producer uploads the special effect material and uploads it to the webpage, or the service provider presets the special effect material in the application; when the terminal uses the special effect material, the special effect material needs to be downloaded or downloaded in the field.
  • these special effects materials are often used for still images.
  • the special effects materials that can be used have certain limitations. How to reduce the usage limitations of the special effects materials, and more flexible video images. The addition of special effects has become a problem to be considered by those skilled in the art.
  • the embodiments of the present application provide a video image processing method, apparatus, and terminal, to reduce the use limitation of the special effect material, and improve the flexibility of adding special effects to the video image.
  • a video image processing method includes:
  • a target canvas drawn with a graphic is superimposed on the video image.
  • the embodiment of the present application further provides a video image processing apparatus, including:
  • a video image display module configured to display the captured video image on the screen
  • a coordinate conversion module configured to detect a user at each operation point of the screen, and convert screen coordinates of each operation point into target canvas coordinates of the target canvas;
  • a graphics drawing module configured to draw a graphic on the target canvas according to target canvas coordinates of each operation point
  • An overlay module for superimposing a target canvas drawn with graphics on the video image.
  • the embodiment of the present application further provides a terminal, including: a memory, a processor, and a graphics processor; the memory stores an executable program, where the program can be executed by the processor or the graphics processor, The program is used to:
  • a target canvas drawn with a graphic is superimposed on the video image.
  • the embodiment of the present application further provides a storage medium for storing executable instructions, when the executable instructions are executed on a computer, causing the computer to execute the video image processing method described in the present application.
  • Embodiments of the present application also provide a computer program product comprising instructions that, when run on a computer, cause the computer to perform the video image processing method described herein.
  • the video image processing method provided by the embodiment of the present application includes: after displaying the captured video image on the screen, detecting each operation point involved in the graffiti on the screen by the user, and converting the screen coordinates of each operation point into Target canvas coordinates of the target canvas; further, according to the target canvas coordinates of each operation point, drawing a graphic on the target canvas, and superimposing the target canvas drawn with the graphic on the video image, thereby adding a user to the video image
  • the effects of the effects of the graffiti graphics includes: after displaying the captured video image on the screen, detecting each operation point involved in the graffiti on the screen by the user, and converting the screen coordinates of each operation point into Target canvas coordinates of the target canvas; further, according to the target canvas coordinates of each operation point, drawing a graphic on the target canvas, and superimposing the target canvas drawn with the graphic on the video image, thereby adding a user to the video image
  • the effects of the effects of the graffiti graphics includes: after displaying the captured video image on the screen, detecting each operation point involved in the graffiti on
  • the corresponding graphic is drawn on the target canvas in the memory, and the target canvas drawn with the graphic can be used as the special effect material, so the content of the special effect material can be
  • the user's different graffiti operations on the screen ie, the difference in the operating points involved in the user's graffiti on the screen
  • the content is more varied and varied, reducing the limitations of the use of special effects materials, and improving the flexibility of adding special effects to video images.
  • the video image processing method provided by the embodiment of the present application may be used to implement the addition of the special effect material drawn by the user on the self-timer image, the video call image, and the like; further, the drawn The effect material is added to the video image in the form of a sticker, so that the effect of the face effect in the dynamic video image can be achieved by the corresponding effect material in the dynamic video image collected in the self-timer, video call and the like.
  • FIG. 1 is a flowchart of a video image processing method according to an embodiment of the present application
  • FIG. 2 is a schematic diagram of an interface operation of entering a graffiti mode
  • FIG. 3 is a schematic diagram of an application example of an embodiment of the present application.
  • FIG. 4 is a flow chart of a method for converting screen coordinates of an operating point to target canvas coordinates
  • Figure 5 is a schematic diagram of a normalized virtual canvas
  • FIG. 6 is a flow chart of another method of converting screen coordinates of an operating point to target canvas coordinates
  • FIG. 7 is a flow chart of still another method of converting screen coordinates of an operating point to target canvas coordinates
  • Figure 8 is a schematic view of the virtual canvas after rotation
  • Figure 9 is a schematic diagram of an application framework incorporating a brush filter
  • FIG. 10 is a structural block diagram of a video image processing apparatus according to an embodiment of the present application.
  • 11 is a block diagram showing the hardware structure of the terminal.
  • the embodiment of the present application can support the user to perform a graffiti operation on the video image, and can draw a corresponding graphic on the canvas according to the user's graffiti operation, thereby superimposing the canvas on the video image, thereby achieving the purpose of adding special effects to the video image;
  • the embodiment can draw a canvas of the corresponding graphic based on the user's graffiti operation, and use the canvas as a special effect material, so that the content of the special effect material can be flexibly changed by different graffiti operations of the user, thereby reducing the use limitation of the special effect material and improving the use.
  • Video images add the flexibility of special effects.
  • FIG. 1 is a flowchart of a video image processing method provided by an embodiment of the present application.
  • the method may be applied to a terminal, and the terminal may be a user equipment such as a smart phone, a tablet computer, or a notebook computer used by the user;
  • the embodiment of the present application can implement the video image processing method provided by the embodiment of the present application by loading a program corresponding to the video image processing method in the terminal, and executing the program by the terminal;
  • the program may be executed by a GPU (Graphics Processing Unit) of the terminal.
  • the embodiment of the present application may implement the implementation of the application by an OpenGL (Open Graphics Library) of the GPU.
  • OpenGL Open Graphics Library
  • the video image processing method provided by the embodiment of the present invention is implemented by the method of the OpenGL of the GPU.
  • the video image processing method may include:
  • Step S100 displaying the captured video image on the screen.
  • the user can start an application for supporting video recording loaded in the terminal, and the user can call the image capture device such as the camera of the terminal to collect the video image and collect the video image.
  • the video image can be recalled to the app and displayed on the screen of the terminal.
  • the video image referred to in the embodiment of the present application may be an image during video recording, or may be a preview image before video recording.
  • the preview image refers to an image captured by an image capturing device such as a camera before shooting, and the preview image may change as the camera moves.
  • the user when the user performs a video recording operation by calling the camera of the terminal, the user can move the position of the terminal to move the position of the camera. During the movement, the image of the corresponding position is displayed on the screen of the terminal, and the image is a preview image.
  • the video image processing method in the embodiment of the present application can also be used in other scenarios. For example, before the user takes a photo with the camera, the image captured by the avatar acquisition device such as the camera can also be used as the video image. As the camera moves, the image acquired by the image acquisition device also changes, that is, the video image displayed on the screen changes.
  • the form of the application supporting the video recording may be determined according to the actual situation; for example, in the self-timer scene, it may be a video photography application; in a video call scenario, it may be an instant messaging application; in a live video scene, It can be a live video application.
  • Step S110 Detecting the user at each operation point of the screen, converting the screen coordinates of each operation point to the target canvas coordinates of the target canvas, and obtaining the target canvas coordinates of each operation point.
  • the user can use a finger or a touch pen or the like as a brush to perform graffiti on the screen of the terminal.
  • the embodiment of the present application can detect each operation point of the user on the screen, and record each operation based on the operation sequence of the operation point.
  • the screen coordinates of the points so that the screen coordinates of the respective operation points are respectively converted into the target canvas coordinates on the target canvas according to the operation sequence of the operation points.
  • the terminal may detect the user's operation points on the screen in response to the user-triggered operation of entering the graffiti mode. There are a plurality of operations triggered by the user to enter the graffiti mode. One operation may be clicking an icon corresponding to the graffiti mode in the effect material panel, and the other operation may be a long press operation of the user on the screen of the terminal.
  • the manner of entering the graffiti mode based on the user operation is as shown in FIG. 2, and the special effect material panel can be displayed on the screen based on the user operation, and the special effect material panel can display the special virtual face pendant, the filter and other special effect materials.
  • an icon for triggering into the graffiti mode is added to the special effect material panel, and when the user clicks the icon, the graffiti mode can be triggered; after entering the graffiti mode, the user can perform the graffiti operation on the screen for displaying the video image;
  • the icon that triggers to enter the graffiti mode is not necessarily added to the effect material panel, but can also be displayed separately on the screen;
  • the screen may display prompt information that the user can perform the graffiti operation, for example, the screen may display a prompt message that “can be painted on the screen” to prompt the user to perform the graffiti operation.
  • the manner of entering the graffiti mode by clicking the icon is only optional.
  • the embodiment of the present application can also detect whether the user performs a long press operation on the screen after the video image is displayed on the screen (long Press operation and other setting operations can be set to be the initial operation of the user's graffiti.
  • the setting operation may also be double-clicking on the screen, etc., and the form is not solid. If so, the user is considered to have the intention of graffiti on the screen. Triggers into the graffiti mode.
  • the embodiment of the present application can apply for a memory space for the target canvas in the memory; wherein the role of the target canvas is mainly based on the user's operation on the screen to draw a corresponding graphic on the target canvas, so as to obtain the special effect material.
  • the target canvas referred to here is a canvas that has applied for memory space in memory.
  • the size of the memory space requested for the target canvas can be determined according to the preset target canvas size (the target canvas size here refers to the resolution size of the target canvas), and the memory occupied by each pixel, such as the target canvas size.
  • the resolution size is 512*512 (only an optional example), and the memory space requested for the target canvas can be 512*512* per pixel.
  • the target canvas coordinates refer to the canvas coordinates of the screen coordinates of the operation point mapped on the target canvas to which the memory space is applied; since the embodiment of the present application is the memory space applied for the target canvas, the operation points of each operation point are The screen coordinates are converted to the target canvas coordinates of the target canvas, and it can be considered that the operation points are mapped to the memory space corresponding to the target canvas.
  • Step S120 Drawing a graphic on the target canvas according to target canvas coordinates of each operation point.
  • the embodiment of the present application can determine the mapping points of the operation points on the target canvas according to the target canvas coordinates of the operation points, thereby Algorithms (such as line brush, texture brush, etc.), connect the mapping points of each operation point on the target canvas (such as connecting in the order of operation of each operation point), and draw the corresponding graphics.
  • Algorithms such as line brush, texture brush, etc.
  • the graffiti referred to in the embodiment of the present application can be considered as that the user can perform a drawing operation on the screen by using a finger or a touch pen or the like as a brush.
  • the drawing points on the target canvas can be realized on the target canvas by the mapping points of the operation points involved in the drawing operation on the target canvas.
  • the graffiti mode referred to in the embodiment of the present application is considered to be a mode that allows the user to perform a drawing operation on the screen by a finger or a touch pen or the like.
  • the user can perform at least one operation track by each operation point on the screen by performing operations such as sliding on the screen by a finger or a touch pen or the like, and performing graphic drawing based on the mapping points of the operation points on the target canvas.
  • a graphic resembling the operation track is drawn on the target canvas.
  • Step S130 superimposing a target canvas drawn with a graphic on the video image.
  • the target canvas drawn with the graphic can be used as a special effect material of the sticker type, and the target canvas on which the graphic is drawn can be superimposed on the video image based on the principle that the sticker is superimposed on the video image.
  • the graphic drawn on the target canvas can be superimposed on the video image in real time, so that the user can adjust the graffiti operation based on the graphic of the target canvas superimposed on the video image in real time, and modify and adjust the graphic.
  • the video image processing method provided by the embodiment of the present application supports the user to perform graffiti on the video image. Specifically, the video image captured on the screen is displayed, and the user can trigger to enter the graffiti mode based on the video image. , detecting each operation point of the user performing graffiti on the screen, converting the screen coordinates of each operation point to the target canvas coordinates of the target canvas, and drawing the target canvas coordinates on the target canvas by the target canvas coordinates of each operation point, and then drawing The graphic target canvas is superimposed as a special effect material on the video image to achieve the effect of graffiti on the video image.
  • the video image processing method of the embodiment of the present application makes real-time graffiti based on dynamic video images possible, and there is an essential difference from the conventional method of graffitiing a video image in the form of a still picture as a canvas.
  • the user opens a video photography application of the terminal to perform self-timer, and the front camera of the terminal can collect a video image containing the user's face and display it on the terminal screen; the user clicks on the special effect material.
  • the "graffiti" icon of the panel can be triggered to enter the graffiti mode, the terminal can initialize the target canvas, and apply for memory space for the target canvas; and the user can refer to the user's face position of the video image displayed on the screen, and use the finger to perform graffiti operation on the screen.
  • the graffiti operation is performed by sliding the user's finger on the screen, and the sliding track of the finger on the screen is shaped like “glasses”;
  • the terminal can record the screen coordinates of each operation point of the screen involved in the sliding process, and convert the screen coordinates of each operation point into the target canvas coordinates of the target canvas, and obtain the operation points at Corresponding mapping points of the target canvas; thus, according to the target canvas coordinates of each operation point, each operation point may be connected at a corresponding mapping point of the target canvas, and a graphic resembling "glasses" is drawn on the target canvas;
  • a target canvas that draws a graphic with a shape like "glasses” can be superimposed on the face of the user of the video image in the form of a sticker, and achieves the effect of the "eyeglass"-like graphic on the target canvas moving with the face of the video image (the effect is as follows)
  • a virtual effect sticker such as a virtual face pendant moves with the face of the video image to achieve the purpose of graffiti on the face of the user of the video image;
  • the user's finger actually moves the graffiti on the screen, and the effect of adding the graphic to the video image displayed on the screen is actually based on the moving graffiti of the user's finger on the screen, and the graphic drawing is performed on the target canvas. After that, the effect of the target canvas superimposed on the video image is presented.
  • the manner in which the triggering enters the graffiti mode shown in the corresponding part of FIG. 3 is only optional, and can be considered as an example for easy understanding.
  • the video image processing method provided by the embodiment of the present application includes: after displaying the captured video image on the screen, if the user enters the graffiti mode according to the user operation, the memory space can be requested for the target canvas, and the target canvas is cached in the memory; After the mode, the operation points involved in the graffiti on the screen can be detected, and the screen coordinates of each operation point are converted into the target canvas coordinates of the target canvas, and the target canvas coordinates of each operation point are obtained; The target canvas coordinates of the operation point, the graphic is drawn on the target canvas, and the target canvas on which the graphic is drawn is superimposed on the video image, thereby realizing the effect of adding the special effect of the graphic doodled by the user on the video image.
  • the corresponding graphic is drawn on the target canvas in the memory, and the target canvas drawn with the graphic can be used as the special effect material, so the content of the special effect material can be
  • the user's different graffiti operations on the screen ie, the difference in the operating points involved in the user's graffiti on the screen
  • the content is more varied and varied, reducing the limitations of the use of special effects materials, and improving the flexibility of adding special effects to video images.
  • the video image processing method provided by the embodiment of the present application may be used to implement the addition of the special effect material drawn by the user on the self-timer image, the video call image, and the like; further, the drawn The effect material is added to the video image in the form of a sticker, and the effect of the drawn effect material on the face movement in the dynamic video image can be realized in the dynamic video image correspondingly collected by the scene such as the self-timer and the video call.
  • the embodiment of the present application may first draw a virtual canvas (the virtual canvas is different from the target canvas that has applied the memory space).
  • the size of the virtual canvas is adjustable. Specifically, the size of the virtual canvas is in a preset proportional relationship with the size of the facial feature region in the video image. The preset proportional relationship may be set according to actual needs.
  • the virtual canvas may be centered on the nose bone of the face, and scaled according to the size of the face.
  • the virtual canvas may be assumed to be a square, and the width may be two of the distance between the two eyes of the face. Times.
  • the screen coordinates of the operation point are mapped to the virtual canvas, the virtual canvas coordinates of each operation point are obtained, and the virtual canvas coordinates of each operation point are mapped to the target canvas, and the target canvas coordinates of each operation point on the target canvas are obtained. ;
  • this method can be applied to a scene in which a video effect is added to a face on a video image.
  • the size of the target canvas does not apply to the diversity of the face.
  • Sexuality so the diversity of the face can be applied through the virtual canvas.
  • the virtual canvas can be used as the intermediate state of the coordinate conversion of the screen coordinates of the operation point to the target canvas coordinates, so that the screen coordinates of the operation point can be made to the target canvas coordinates of the target canvas. The conversion is more accurate.
  • FIG. 4 is a flowchart of a method for converting screen coordinates of each operation point into target canvas coordinates according to an embodiment of the present application.
  • the method may be applied to a terminal, such as may be implemented by a GPU of the terminal;
  • the method can include:
  • Step S200 Determine a virtual canvas according to a facial feature in the video image.
  • the virtual canvas is different from the target canvas mentioned above, and the size of the virtual canvas is in a predetermined proportional relationship with the size of the facial feature region in the video image.
  • the size may be represented by a resolution
  • the size of the target canvas that is, the resolution of the target canvas is preset
  • the size of the virtual canvas that is, the resolution of the virtual canvas
  • the resolution of the face in the image is adjusted. That is, for the video images with different resolutions of the face, the resolution of the corresponding virtual canvas may be different.
  • the embodiment of the present application can draw a virtual canvas according to the facial features in the video image.
  • the embodiment of the present application can detect the coordinates of the first feature point of the face in the video image by using a face detection technology (taking the face as an example, the first feature point can be set as a face)
  • the characteristics of the facial features such as the nose of the face, the eyebrows, the bridge of the nose, etc., the form of the first feature point of the face can be set according to the actual situation), and then determined according to the preset proportional relationship and the size of the face region in the video image.
  • the size of the virtual canvas, and then the coordinates of the first feature point of the face are the center point of the virtual canvas, and the virtual canvas is determined according to the size of the virtual canvas.
  • the preset proportional relationship can be set according to actual conditions, and the specific value can be adjusted by itself, so that the virtual canvas can cover most of the screens.
  • Step S210 Mapping screen coordinates of each operation point to the virtual canvas to obtain virtual canvas coordinates of each operation point.
  • the embodiment of the present application may be based on the screen coordinates of the operation point and the screen coordinates of the center point of the virtual canvas (ie, the coordinates of the first feature point of the face in the video image). Converts the screen coordinates of the operating point to the coordinates in the virtual canvas to obtain the virtual canvas coordinates of the operating point.
  • the terminal may calculate a coordinate translation vector according to the screen coordinates of the operation point and the screen coordinates of the center point of the virtual canvas, and then determine the virtual canvas coordinates of the operation point according to the coordinate translation vector.
  • the manner of converting the screen coordinates of the operating point to the coordinates of the virtual canvas is only optional, and the embodiment of the present application can also record the relationship between the coordinate system of the recording screen and the coordinate system of the virtual canvas according to the mapping relationship. Determines the coordinates of the screen coordinates of the operating point converted in the virtual canvas.
  • step S220 the coordinates of the virtual canvas of each operation point are normalized to obtain normalized coordinates of each operation point.
  • the embodiment of the present application can set the size of the normalized virtual canvas, which can be recorded as a specified normalized size for convenience of description. If the virtual canvas is normalized and the range of the x-axis and y-axis directions of the virtual canvas is -1 to 1, the virtual canvas can be considered to have a size of 2*2, as shown in FIG.
  • the specified normalized size can be customized according to the actual situation, except that the virtual canvas that can be set and normalized as shown in FIG. 5 is square, and the width is 2, and the normalized can also be set.
  • the width of the virtual canvas is 4 (such as the range of the virtual canvas's x-axis and y-axis directions - 2 to 2).
  • the embodiment of the present application may determine the first coordinate scaling ratio based on the size of the virtual canvas and the specified normalized size; for example, dividing the width of the normalized virtual canvas by the virtual canvas The width of the drawing, the first coordinate scaling;
  • the initial normalized coordinate of the operating point on the virtual canvas may be determined according to the operating point's coordinate of the virtual canvas and the first coordinate scaling; if the operating point is multiplied by the coordinates of the virtual canvas The first coordinate is scaled to obtain the initial normalized coordinates of the operating point on the virtual canvas.
  • the manner of determining the initial normalized coordinates of the virtual canvas on the virtual canvas is optional.
  • the embodiment of the present application may also set the coordinate system of the virtual canvas and the coordinates of the normalized virtual canvas. According to the mapping relationship, the initial normalized coordinates of each operation point are obtained.
  • the rotation angle of the face in the video image may be divided into 0 and not 0, and subsequent processing of normalizing coordinates of each operation point may be performed;
  • the initial normalized coordinates of the operation point on the virtual canvas can be directly used as the normalized coordinates of each operation point, that is, the operation point is obtained at the initial of the virtual canvas. Normalized coordinates, it is considered that the normalized coordinates of each operating point are obtained;
  • the initial normalized coordinates of the operation points are also reversely rotated by the rotation angle to obtain the operation points. Normalized coordinates.
  • Step S230 Mapping normalized coordinates of each operation point to the target canvas to obtain target canvas coordinates of each operation point.
  • the embodiment of the present application can determine the second coordinate scaling according to the size of the target canvas and the specified normalized size; if the width of the target canvas is divided, normalized. The width of the virtual canvas to get the second coordinate scaling;
  • the target canvas coordinates of the operating point on the target canvas are obtained according to the normalized coordinates of the operating point and the second coordinate scaling; if the set value is added to the normalized x-axis of the operating point After the coordinates, multiply the second coordinate scaling to obtain the x-axis target canvas coordinates of the operating point, subtract the normalized y-axis coordinate of the operating point from the set value, and multiply the second coordinate scaling to obtain the operating point.
  • the y-axis target canvas coordinates, combined with the x-axis target canvas coordinates of the operation point and the y-axis target canvas coordinates, the target canvas coordinates of the operation point on the target canvas are obtained;
  • the set value can be customized according to the actual situation; the specific value can be determined according to the size of the virtual canvas.
  • the virtual canvas size is 2*2
  • the center point is the origin
  • the target canvas is the origin of the target canvas in the upper left corner, with the x-axis positive direction to the right and the y-axis positive direction.
  • the target canvas coordinates of the operating point on the target canvas can be obtained as follows: (1+x' , 1-y') * 512/2.
  • the step S220 to the step S230 are only an alternative manner of converting the coordinates of the operation points on the virtual canvas to the target canvas coordinates of the operation points on the target canvas, and the virtual canvas can also be set in the embodiment of the present application.
  • the coordinate mapping relationship of the target canvas according to the coordinate mapping relationship, the conversion of each operation point from the coordinates of the virtual canvas to the target canvas coordinates on the target canvas is realized.
  • the virtual canvas when the face of the video image is graffiti, can be used as the intermediate state of the coordinate conversion of the screen coordinates of the operation point to the coordinate of the target canvas coordinate, and the face is drawn by the virtual canvas drawn by the face of the video image.
  • the application of the diversity can make the conversion of the screen coordinates of the operating point to the target canvas coordinates of the target canvas more accurate, and improve the accuracy of the subsequent drawing of the target canvas with the graphic on the face of the video image.
  • FIG. 6 is a flowchart of another method for converting the screen coordinates of each operation point into the target canvas coordinates provided by the embodiment of the present application.
  • the method can include:
  • Step S300 Detecting screen coordinates of the first feature point of the face in the video image.
  • Step S310 The screen coordinates of the first feature point of the face are the center point of the virtual canvas, and the size of the virtual canvas is determined according to the preset proportional relationship and the size of the face region in the video image, according to the virtual canvas. The size determines the virtual canvas.
  • step S300 to step S310 can be considered as an optional implementation of step S200 shown in FIG. 4, wherein the rotation angle of the face in the video image is 0.
  • Step S320 For each operation point, calculate a coordinate translation vector according to the screen coordinates of the operation point and the screen coordinates of the center point of the virtual canvas, and then determine the virtual canvas coordinates of the operation point according to the coordinate translation vector.
  • the coordinate translation vector can be obtained by subtracting the screen coordinates of the operating point and the screen coordinates of the center point of the virtual canvas. Since the virtual canvas is obtained by shifting the center point, the component in the x direction of the coordinate translation vector can be regarded as the x-axis coordinate of the virtual canvas, and the component in the y direction can be regarded as the y-axis coordinate of the virtual canvas, and thus, the operation will be performed.
  • the x-axis screen coordinate of the point is subtracted from the x-axis coordinate of the center point of the virtual canvas, and the y-axis coordinate of the y-axis of the operation point is subtracted from the y-axis coordinate of the center point of the virtual canvas to obtain the coordinates of the operation point on the virtual canvas.
  • step S320 can be considered as step S210 shown in FIG. 4 to map screen coordinates of each operation point into the virtual canvas to obtain an implementation of virtual canvas coordinates of each operation point.
  • the coordinates of the center point of the virtual canvas ie, the coordinates of the first feature point of the face in the video image
  • the screen coordinates of an operation point are P0 (120, 660).
  • the center of the virtual canvas can be subtracted from the x-axis screen coordinates of the operation point.
  • the x-axis coordinate of the point is obtained by subtracting the y-axis coordinate of the center point of the virtual canvas from the y-axis screen coordinate of the operating point, and obtaining the coordinates of the virtual canvas after the operation point is converted;
  • the coordinates P1 (P1x, P1y) of the center point of the virtual canvas and the screen coordinates of the operation point are P0 (P0x, P0y)
  • step S320 is based on processing in a two-dimensional coordinate system.
  • the face detection also supports the Euler angle detection (the angle between the rotation angle of the face and the x, y, and z axes in the three-dimensional space) ), the x-axis screen coordinate of the operating point is subtracted from the x-axis coordinate of the center point of the virtual canvas, and the y-axis coordinate of the operating point is subtracted from the y-axis coordinate of the center point of the virtual canvas, and the result is multiplied.
  • Step S330 determining a first coordinate scaling according to the specified normalized size and the size of the virtual canvas.
  • the width of the normalized virtual canvas is divided by the width of the virtual canvas when drawn, the first coordinate scaling is obtained.
  • Step S340 For each operation point, determine the normalized coordinates of each operation point according to the virtual canvas coordinates of the operation point and the first coordinate scaling ratio.
  • the normalized coordinates of the operating point are obtained.
  • the embodiment of the present application may multiply the virtual canvas coordinate P2 of the operating point by the first coordinate scaling scale to obtain the normalized coordinate of the operating point on the virtual canvas; for example, set the operating point to the coordinate P2 of the virtual canvas.
  • the first coordinate scaling scale is 2/680
  • step S330 to step S340 may be considered as an implementation of step S220 of FIG. 4.
  • Step S350 determining a second coordinate scaling according to the size of the target canvas and the specified normalized size.
  • the width of the target canvas may be divided by the width of the normalized virtual canvas to obtain a second coordinate scaling.
  • Step S360 For each operation point, obtain the target canvas coordinates of the operation point according to the normalized coordinates of the operation point and the second coordinate scaling.
  • the set value is added to the normalized x-axis coordinate of the operating point, and then multiplied by the second coordinate scaling to obtain the x-axis target canvas coordinates of the operating point, and the set value is subtracted from the operating point.
  • the y-axis coordinate is multiplied by the second coordinate scaling, the y-axis target canvas coordinates of the operating point are obtained, and the x-axis target canvas coordinates and the y-axis target canvas coordinates of the operating point are combined to obtain the target of the operating point on the target canvas. Canvas coordinates.
  • FIG. 7 is a flowchart of still another method for converting the screen coordinates of each operation point into the target canvas coordinates provided by the embodiment of the present application.
  • the method can include:
  • Step S400 Detecting coordinates of a first feature point of the face in the video image.
  • Step S410 The screen coordinates of the first feature point of the face are the center point of the virtual canvas, and the size of the virtual canvas is determined according to the preset proportional relationship and the size of the face region in the video image, according to the virtual canvas. The size determines the virtual canvas.
  • the virtual canvas may be a square.
  • the coordinates of the face and nose of the video image may be the center point of the virtual canvas, and the virtual canvas may be determined by the width of the virtual canvas twice the width of the virtual canvas;
  • step S400 and step S410 may be considered as an optional implementation of step S200 shown in FIG. 4, wherein the rotation angle of the face in the video image is not zero.
  • step S300 and step S310 shown in FIG. 6 and step S400 and step S410 shown in FIG. 7 is that the rotation angle of the face in the video image of FIG. 6 is 0, and in the video image of FIG. The rotation angle of the face is not zero.
  • Step S420 For each operation point, calculate a coordinate translation vector according to the screen coordinates of the operation point and the screen coordinates of the center point of the virtual canvas, and determine the virtual canvas coordinates of the operation point according to the coordinate translation vector.
  • the x-axis screen coordinate of the operating point is subtracted from the x-axis coordinate of the center point of the virtual canvas, and the y-axis coordinate of the operating point is subtracted from the y-axis coordinate of the center point of the virtual canvas, and the operating point is virtualized.
  • the coordinates of the canvas is subtracted from the x-axis coordinate of the center point of the virtual canvas.
  • step S420 can be considered as an implementation of step S210 shown in FIG. 4, and can be referred to step S320 shown in FIG. 6.
  • the face detection also supports the Euler angle detection (the angle between the face rotation angle and the x, y, and z axes in the three-dimensional space)
  • the x-axis screen coordinates of the operation point are subtracted from the center of the virtual canvas.
  • the x-axis coordinate of the point after subtracting the y-axis coordinate of the center point of the virtual canvas from the y-axis screen coordinate of the operation point, multiply the multi-dimensional rotation matrix to obtain the coordinates of the operation point on the virtual canvas.
  • Step S430 determining a first coordinate scaling according to the specified normalized size and the size of the virtual canvas.
  • the width of the normalized virtual canvas is divided by the width of the virtual canvas when drawn, the first coordinate scaling is obtained.
  • Step S440 For each operation point, determine initial normalized coordinates of each operation point according to the coordinates of the virtual canvas and the first coordinate scaling ratio of the operation point.
  • the initial normalized coordinates of the operating point are obtained.
  • step S450 the initial normalized coordinates of the operation points are respectively reversed by the rotation angle to obtain normalized coordinates of the operation points.
  • the embodiment of the present application may determine a two-dimensional rotation matrix corresponding to the rotation angle, and for each operation point, determine the normalization of each operation point according to the initial normalized coordinates of the operation point and the two-dimensional rotation matrix.
  • the coordinates can be obtained by multiplying the initial normalized coordinates of the operating point by the two-dimensional rotation matrix to obtain the normalized coordinates of the operating point.
  • the normalized coordinates of the operation point can be obtained as follows:
  • the virtual canvas coordinates need to be rotated, which is equivalent to rotating the virtual canvas.
  • the positive and negative directions of the rotation angle may be customized with reference to the center line of the video image, which is not limited in this embodiment of the present application.
  • step S430 to step S450 can be considered as an implementation of step S220 of FIG. 4, and the difference from step S330 to step S340 shown in FIG. 6 is that the method shown in FIG. 7 is based on the angle of rotation of the video image.
  • the result obtained in step S340 of FIG. 6 can be regarded as the initial normalized coordinate of the method shown in FIG. 7.
  • the method shown in FIG. 7 needs to perform initial return of each operation point based on the rotation angle. The reverse rotation of a coordinate.
  • Step S460 determining a second coordinate scaling according to the size of the target canvas and the specified normalized size.
  • the width of the target canvas may be divided by the width of the normalized virtual canvas to obtain a second coordinate scaling.
  • Step S470 For each operation point, obtain a target canvas coordinate of the operation point according to the normalized coordinate of the operation point and the second coordinate scaling.
  • the set value may be added to the normalized x-axis coordinate of the operating point, and then multiplied by the second coordinate scaling to obtain the x-axis target canvas coordinate of the operating point, and the set value is subtracted from the normalized operating point.
  • the y-axis coordinate is multiplied by the second coordinate scaling, the y-axis target canvas coordinates of the operating point are obtained, and the x-axis target canvas coordinates and the y-axis target canvas coordinates of the operating point are combined to obtain the target canvas of the operating point on the target canvas. coordinate.
  • the embodiment of the present application supports the processing in the two-dimensional coordinate system, and also supports the processing in the three-dimensional coordinate system.
  • One of the differences between the two is that the three-dimensional coordinate system is compared with the two-dimensional coordinate system.
  • the three-dimensional coordinate system is compared with the two-dimensional coordinate system.
  • the above shows the application of the virtual canvas as the intermediate state of the coordinate conversion of the screen coordinates of the operation point to the target canvas coordinate in the scene of graffitiing the face of the video image, and the application of the diversity of the face is achieved; It is illustrated that the coordinate conversion of the screen coordinates of the operating point to the target canvas coordinate through the virtual canvas is only an alternative, as an alternative: if the face of the video image is not graffiti, etc.
  • the embodiment of the present application may also display a graffiti area corresponding to the target canvas size on the screen (optionally, further, the position of the graffiti area may be adjusted by the user operation, but the size is unchanged), so that the user is in the graffiti area.
  • Graffiti to achieve the direct conversion of the screen coordinates of the operating point to the target canvas coordinates (here directly relative to the indirect way of using the virtual canvas as an intermediate state); correspondingly, the target canvas drawn with the graphics can be based on the graffiti area The position in the video image is made to superimpose the video image.
  • the target canvas drawn with the graphics may be added to the dynamic video image in the form of a sticker, thereby realizing the drawing.
  • the target canvas of the graphic follows the effect of face movement in the video image;
  • the target canvas drawn with the graphic may be used as a special effect material of the sticker type and superimposed on the video image; for example, using openGL as an example, the present application
  • the embodiment may pass the vertex coordinates and the rotation angle of the target canvas drawn with the graphics (corresponding to the rotation angle of the face in the video image) to the vertex shader, and the vertex shader draws the vertex of the target canvas according to the rotation angle and the graphic.
  • Coordinates the coordinates of the superimposed position of the target canvas on which the graphic is drawn are calculated, so that the target canvas on which the graphic is drawn is superimposed on the video image according to the coordinates of the superimposed position; further, the target canvas to be drawn with the graphic
  • the target canvas drawn with the graphic can follow the face movement in the video image.
  • the superposition principle of the three-dimensional space sticker to the video image may be adopted, for example, a perspective projection technology.
  • the video image processing method provided by the embodiment of the present application may exist in the form of a painter filter, and may be added to a filter chain of an application supporting video recording, and a filter still exists in the filter chain.
  • the squeegee filter, the whitening filter, the face detection, the deformation filter, etc., the video image processing method provided by the embodiment of the present application can be added to the end of the filter chain in the form of a brush filter, and the video is processed in the form of a filter.
  • the graffiti of the image, after the graffiti is superimposed, the video image data of the target canvas on which the graphic is drawn can be output to the screen or written to the local video file for saving.
  • the application framework that adds the brush filter can be seen in Figure 9, which can be referenced.
  • the video image processing method provided by the embodiment of the present application can support the user to perform graffiti on the video image, and the target canvas drawn by the user based on the user graffiti can be used as the special effect material, and the content of the special effect material can be displayed on the screen with the user.
  • the adjustment of the graffiti operation is different, so that the content of the special effect material can be personally adjusted by the user, so that the content of the video special effect added on the video image is richer and more varied, the use limit of the special effect material is reduced, and the video is improved.
  • the video image processing apparatus provided in the embodiment of the present application is described below.
  • the video image processing apparatus described below can be considered as a program module required for the terminal to implement the video image processing method provided by the embodiment of the present application.
  • the content of the video image processing apparatus described below may be referred to in correspondence with the content of the video image processing method described above.
  • FIG. 10 is a structural block diagram of a video image processing apparatus according to an embodiment of the present disclosure.
  • the video image processing apparatus may be applied to a terminal, and may be specifically applied to a GPU of a terminal.
  • the video image processing apparatus may include:
  • the video image display module 100 is configured to display the captured video image on the screen
  • the coordinate conversion module 200 is configured to detect, at each operation point of the screen, convert the screen coordinates of each operation point to the target canvas coordinates of the target canvas;
  • a graphics drawing module 300 configured to draw a graphic on the target canvas according to target canvas coordinates of each operation point
  • the overlay module 400 is configured to superimpose the drawn target canvas on the video image.
  • the coordinate conversion module 200 is configured to convert the screen coordinates of each operation point into the target canvas coordinates of the target canvas, and specifically includes:
  • the operation points are mapped to the target canvas in the virtual canvas coordinates, and the target canvas coordinates of the respective operation points on the target canvas are obtained.
  • the coordinate conversion module 200 is configured to determine the virtual canvas according to the facial features in the video image, and specifically includes:
  • the coordinate conversion module 200 is configured to map the screen coordinates of each operation point to the virtual canvas, and obtain the virtual canvas coordinates of each operation point, which specifically includes:
  • the screen coordinates of the operation point are converted into the coordinates of the virtual canvas, and the virtual canvas coordinates of the operation point are obtained.
  • the coordinate conversion module 200 is configured to convert the screen coordinates of the operation point into the coordinates of the virtual canvas according to the screen coordinates of the operation point and the coordinates of the center point of the virtual canvas, and specifically includes:
  • the coordinate translation vector is calculated according to the screen coordinates of the operation point and the screen coordinates of the center point of the virtual canvas;
  • a virtual canvas coordinate of the operating point is determined based on the coordinate translation vector.
  • the coordinate conversion module 200 is configured to convert the screen coordinates of the operation point into the coordinates of the virtual canvas according to the screen coordinates of the operation point and the coordinates of the center point of the virtual canvas, and specifically includes:
  • the x-axis screen coordinate of the operation point is subtracted from the x-axis coordinate of the center point of the virtual canvas, and the y-axis coordinate of the operation point is subtracted from the y-axis coordinate of the center point of the virtual canvas, and the result is obtained.
  • the coordinate conversion module 200 is configured to map the virtual canvas coordinates of the operation points to the target canvas, and obtain the target canvas coordinates of each operation point, which specifically includes:
  • the normalized coordinates of each operation point are mapped into the target canvas, and the target canvas coordinates of each operation point are obtained.
  • the coordinate conversion module 200 is configured to normalize the virtual canvas coordinates of each operation point to obtain normalized coordinates of each operation point, specifically including :
  • the initial normalized coordinates of the respective operating points are respectively reversed by the rotation angle to obtain normalized coordinates of the respective operating points.
  • the coordinate conversion module 200 may be specifically configured to: divide the width of the normalized virtual canvas by the virtual canvas when drawing The width of the first coordinate is obtained.
  • the coordinate conversion module 200 may be specifically configured to: multiply the virtual canvas coordinates of the operation point by the first A coordinate scaling is obtained to obtain the initial normalized coordinates of the operating point.
  • the coordinate conversion module 200 is configured to: when initializing the normalized coordinates of the operation points, respectively, to rotate the rotation angle to obtain normalized coordinates of each operation point, specifically: initializing the operation points
  • the coordinate is multiplied by the two-dimensional rotation matrix corresponding to the rotation angle to obtain the normalized coordinates of the operation point.
  • the coordinate conversion module 200 is configured to normalize the virtual canvas coordinates of each operation point to obtain normalized coordinates of each operation point, which specifically includes:
  • the normalized coordinates of each operating point are determined according to the virtual canvas coordinates of the operating point and the first coordinate scaling.
  • the coordinate conversion module 200 determines the normalized coordinates of each operation point according to the virtual canvas coordinates and the first coordinate scaling ratio of the operation point, specifically It can be used to: multiply the virtual canvas coordinates of the operating point by the first coordinate scaling to obtain the normalized coordinates of the operating point.
  • the coordinate conversion module 200 is configured to map the normalized coordinates of the operation points to the target canvas, and obtain the target canvas coordinates of each operation point, which specifically includes:
  • the target canvas coordinates of the operating point are obtained according to the normalized coordinates of the operating point and the second coordinate scaling.
  • the coordinate conversion module 200 may be specifically configured to: divide the width of the target canvas by the width of the normalized virtual canvas. , get the second coordinate scaling.
  • the coordinate conversion module 200 when the coordinate conversion module 200 obtains the target canvas coordinates of the operation point according to the normalized coordinates and the second coordinate scaling ratio of the operation point, the coordinate conversion module 200 may be specifically configured to:
  • the coordinate conversion module 200 is configured to detect each operation point of the user on the screen, and specifically includes:
  • the user In response to the user-triggered operation of entering the graffiti mode, the user is detected at various operating points of the screen.
  • the superimposing module 400 is configured to superimpose the drawn target canvas on the video image, and specifically includes:
  • the target canvas on which the graphic is drawn is superimposed on the video image according to the superimposed position coordinates, and the target canvas on which the graphic is drawn follows the face movement in the video image.
  • the coordinate conversion module 200 is configured to detect, at each operation point of the screen, the screen coordinates of each operation point to be converted to the target canvas coordinates of the target canvas, which may specifically include:
  • An operation point of the graffiti area displayed by the user on the screen corresponding to the target canvas size is detected, and the screen coordinates of the operation point of the graffiti area are converted into the target canvas coordinates of the target canvas.
  • the overlay module 400 may superimpose the drawn target canvas on the video image using a perspective projection technique.
  • FIG. 11 shows a hardware structural block diagram of the terminal.
  • the terminal may include: at least one processor 1, at least one communication interface 2, at least one memory 3, at least one communication bus 4 and at least one graphic.
  • the communication interface 2 can be an interface of the communication module, such as an interface of the GSM module;
  • the processor 1 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.
  • CPU central processing unit
  • ASIC Application Specific Integrated Circuit
  • the memory 3 may include a high speed RAM memory and may also include a non-volatile memory such as at least one disk memory.
  • the memory 3 stores an executable program, which can be executed by a processor or a graphics processor (in some special cases, the corresponding program of the video image processing method provided by the embodiment of the present application can also be implemented by the processor), This program is specifically used to:
  • a target canvas drawn with a graphic is superimposed on the video image.
  • refinement function and extended function of the program can be described in the corresponding section above.
  • the embodiment of the present application may further provide a storage medium storing a program suitable for execution by a processor, the program being used to:
  • a target canvas drawn with a graphic is superimposed on the video image.
  • Embodiments of the present application also provide a computer program product comprising instructions that, when run on a computer, cause the computer to perform the video image processing method described herein.
  • the steps of a method or algorithm described in connection with the embodiments disclosed herein can be implemented directly in hardware, a software module executed by a processor, or a combination of both.
  • the software module can be placed in random access memory (RAM), memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or technical field. Any other form of storage medium known.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Graphics (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Databases & Information Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Business, Economics & Management (AREA)
  • Marketing (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Processing Or Creating Images (AREA)
  • Image Processing (AREA)

Abstract

一种视频图像处理方法、装置及终端，该方法包括：在屏幕显示所采集的视频图像；检测用户在屏幕的各操作点，将各操作点的屏幕坐标转换为在目标画布的目标画布坐标；根据各操作点的目标画布坐标，在所述目标画布上绘制图形；将绘制有图形的目标画布叠加在所述视频图像上。本申请实施例可降低特效素材的使用局限，提升对视频图像添加特效的灵活性。

Description

视频图像处理方法、装置及终端
本申请要求于2017年11月14日提交中国专利局、申请号为201711124135.9、申请名称为“一种视频图像处理方法、装置及终端”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理技术领域，具体涉及一种视频图像处理方法、装置及终端。
背景技术
目前大多数支持视频录制的应用(如视频摄影类应用，支持视频通话的即时通讯应用，视频直播应用等)，为实现用户的个性化需求，一般都提供了对视频图像添加特效的功能；对视频图像添加特效需要使用到特效素材，特效素材如虚拟人脸挂件(如可在视频图像的人脸上叠加的胡子，动物耳朵等虚拟人脸挂件)，滤镜(滤镜可调整视频图像背景主题，如的卡通、漫画等效果的滤镜)等。
目前特效素材的来源主要是制作者制作特效素材后上传到网页，或者，由服务商将特效素材预置在应用中；终端在使用特效素材时，需要预先下载或现场下载特效素材。然而，这些特效素材往往用于静态图片，对于动态的视频图像，终端在对其添加特效时，可使用的特效素材存在一定的局限，如何降低特效素材的使用局限，更为灵活的对视频图像添加特效成为了本领域技术人员需要考虑的问题。
发明内容
有鉴于此，本申请实施例提供一种视频图像处理方法、装置及终端，以降低特效素材的使用局限，提升对视频图像添加特效的灵活性。
为实现上述目的，本申请实施例提供如下技术方案：
一种视频图像处理方法，包括：
在屏幕显示所采集的视频图像；
检测用户在屏幕的各操作点，将各操作点的屏幕坐标转换为在目标画布的目标画布坐标；
根据各操作点的目标画布坐标，在所述目标画布上绘制图形；
将绘制有图形的目标画布叠加在所述视频图像上。
本申请实施例还提供一种视频图像处理装置，包括：
视频图像显示模块，用于在屏幕显示所采集的视频图像；
坐标转换模块，用于检测用户在屏幕的各操作点，将各操作点的屏幕坐标转换为在目标画布的目标画布坐标；
图形绘制模块，用于根据各操作点的目标画布坐标，在所述目标画布上绘制图形；
叠加模块，用于将绘制有图形的目标画布叠加在所述视频图像上。
本申请实施例还提供一种终端，包括：存储器，处理器和图形处理器；所述存储器存储有可执行的程序，所述程序可被所述处理器或所述图形处理器执行，所述程序用于：
在屏幕显示所采集的视频图像；
检测用户在屏幕的各操作点，将各操作点的屏幕坐标转换为在目标画布的目标画布坐标；
根据各操作点的目标画布坐标，在所述目标画布上绘制图形；
将绘制有图形的目标画布叠加在所述视频图像上。
本申请实施例还提供一种存储介质，所述存储介质用于存储可执行指令，所述可执行指令在计算机上运行时，使得所述计算机执行本申请所述的视频图像处理方法。
本申请实施例还提供一种包括指令的计算机程序产品，当其在计算机上运行时，使得所述计算机执行本申请所述的视频图像处理方法。
基于上述技术方案，本申请实施例提供的视频图像处理方法包括：在屏幕显示所采集的视频图像后，检测用户在屏幕上进行涂鸦所涉及的各操作点，将各操作点的屏幕坐标转换为在目标画布的目标画布坐标；进而可根据各操作点的目标画布坐标，在所述目标画布上绘制图形，将绘制有图形的目标画布叠加 在所述视频图像上，实现在视频图像上添加用户所涂鸦的图形的特效的效果。
本申请实施例可根据用户在屏幕上进行涂鸦所涉及的各操作点，在内存的目标画布上进行相应图形的绘制，绘制有图形的目标画布可作为特效素材使用，因此特效素材的内容可随着用户在屏幕上的涂鸦操作的不同(即用户在屏幕上进行涂鸦所涉及的操作点的不同)而调整，使得特效素材的内容可由用户个性化的调整，使得在视频图像上添加的视频特效的内容更为丰富多变，降低了特效素材的使用局限，提升了对视频图像添加特效的灵活性。例如，在用户自拍、视频通话等场景下，可基于本申请实施例提供的视频图像处理方法，在自拍图像、视频通话图像等图像上实现用户绘制的特效素材的添加；进一步，将所绘制的特效素材以贴纸形式添加到视频图像中，则可在自拍、视频通话等场景下相应采集的动态视频图像中，实现所绘制的特效素材跟随动态视频图像中的人脸移动的效果。
附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据提供的附图获得其他的附图。
图1为本申请实施例提供的视频图像处理方法的流程图；
图2为进入涂鸦模式的界面操作示意图；
图3为本申请实施例的应用示例图；
图4为将操作点的屏幕坐标转换为目标画布坐标的方法流程图；
图5为归一化后的虚拟画布的示意图；
图6为将操作点的屏幕坐标转换为目标画布坐标的另一方法流程图；
图7为将操作点的屏幕坐标转换为目标画布坐标的再一方法流程图；
图8为虚拟画布旋转后的示意图；
图9为加入画笔滤镜的应用程序框架示意图；
图10为本申请实施例提供的视频图像处理装置的结构框图；
图11为终端的硬件结构框图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本申请实施例可支持用户在视频图像上进行涂鸦操作，根据用户的涂鸦操作可在画布上绘制出相应的图形，从而将画布叠加到视频图像上，实现对视频图像添加特效的目的；本申请实施例可基于用户的涂鸦操作绘制出相应图形的画布，将该画布作为特效素材使用，可使得特效素材的内容可由用户不同的涂鸦操作灵活的改变，从而达到降低特效素材的使用局限，提升对视频图像添加特效的灵活性。
在此思路下，图1示出了本申请实施例提供的视频图像处理方法的流程图，该方法可应用于终端，终端可以如用户所使用的智能手机，平板电脑，笔记本电脑等用户设备；本申请实施例可通过在终端内装载所述视频图像处理方法相应的程序，由终端执行该程序实现本申请实施例提供的视频图像处理方法；
可选的，该程序可由终端的GPU(Graphics Processing Unit，图形处理器)执行；在一种可选实现上，本申请实施例可由GPU的OpenGL(Open Graphics Library，开放图形库)实现本申请实施例提供的视频图像处理方法，如通过GPU的OpenGL执行该程序，实现本申请实施例提供的视频图像处理方法。
参照图1，该视频图像处理方法可以包括：
步骤S100、在屏幕显示所采集的视频图像。
当用户在进行自拍、视频通话、视频直播等场景下，用户可启动终端内装载的支持视频录制的应用，用户可通过该应用调用终端的摄像头等图像采集装置，进行视频图像的采集，所采集的视频图像可回调给该应用，显示在终端的屏幕上。可以理解，本申请实施例所指的视频图像可以是视频录制时的图像，也可以是视频录制前的预览图像。该预览图像是指摄像头等图像采集装置在拍摄之前所采集的图像，预览图像可以随着摄像头的移动而发生变化。例如，用户在通过应用调用终端的摄像头执行视频录制操作时，可以移动终端的位置从 而移动摄像头的位置，在移动过程中，终端屏幕上显示对应位置的图像，该图像即为预览图像。还需要说明的是，本申请实施例的视频图像处理方法也可以用于其他场景，如用户在利用相机拍照前，摄像头等头像采集装置所采集的图像也可以作为视频图像。随着摄像头的移动，图像采集装置采集的图像也会发生变化，即屏幕显示的视频图像会发生变化。
可选的，支持视频录制的应用的形式可以根据实际情况而定；例如，在自拍场景下，可以是视频摄影类应用；在视频通话场景下，可以是即时通讯应用；在视频直播场景下，可以是视频直播应用。
步骤S110、检测用户在屏幕的各操作点，将各操作点的屏幕坐标转换为在所述目标画布的目标画布坐标，得到各操作点的目标画布坐标。
进入涂鸦模式后，用户可使用手指或触摸笔等作为画笔，在终端的屏幕上进行涂鸦；本申请实施例可检测用户在屏幕上的各操作点，基于操作点的操作先后顺序，记录各操作点的屏幕坐标，从而根据操作点的操作先后顺序，分别将各操作点的屏幕坐标转换为在所述目标画布上的目标画布坐标。
在具体实现时，终端可以响应于用户触发的进入涂鸦模式的操作，检测用户在屏幕的各操作点。其中，用户触发的进入涂鸦模式的操作有多种，一种操作可以是点击特效素材面板中涂鸦模式对应的图标，另一种操作可以是用户在终端的屏幕上的长按操作。
可选的，基于用户操作进入涂鸦模式的方式例如图2所示，可基于用户操作在屏幕展开显示特效素材面板，该特效素材面板可以显示有传统的虚拟人脸挂件、滤镜等特效素材，本申请实施例在该特效素材面板中加入了触发进入涂鸦模式的图标，当用户点击该图标则可触发进入涂鸦模式；进入涂鸦模式后，用户可在显示视频图像的屏幕上进行涂鸦操作；显然，触发进入涂鸦模式的图标并不一定加入到特效素材面板中，也可以在屏幕中单独显示；
进一步，当用户点击图标触发进入涂鸦模式后，屏幕可显示用户可进行涂鸦操作的提示信息，如屏幕可显示“可在屏幕上作画”的提示信息，以提示用户后续可进行涂鸦操作。
可选的，图2所示通过点击图标进入涂鸦模式的方式仅是可选的，本申请实施例也可在屏幕显示视频图像后，检测用户是否在屏幕进行长按操作等设定 操作(长按操作等设定操作可以设定为是用户涂鸦的起始操作，该设定操作也可能是双击屏幕等，形式并不固体)，若是，则认为用户具有在屏幕上进行涂鸦的意图，可触发进入涂鸦模式。
进入涂鸦模式时，本申请实施例可在内存中为目标画布申请内存空间；其中，目标画布的作用主要是基于用户在屏幕的操作在目标画布上绘制出相应的图形，以实现特效素材的获得；值得注意的是，此处所指的目标画布是在内存中申请了内存空间的画布。
为目标画布申请的内存空间的大小可以根据预先设定的目标画布大小(此处的目标画布大小指目标画布的分辨率大小)，以及每像素所占内存确定，如可根据目标画布大小*每像素所占内存(*表示乘以)的数值，为目标画布申请内存空间；即为目标画布申请的内存空间的大小可以是目标画布大小*每像素所占内存；例如，预先设定的目标画布分辨率大小为512*512(仅是一种可选示例)，则为目标画布申请的内存空间的大小可以是512*512*每像素所占内存。
这里需要留意的是，目标画布坐标是指操作点的屏幕坐标在申请了内存空间的目标画布上映射的画布坐标；由于本申请实施例是为目标画布申请的内存空间，因此将各操作点的屏幕坐标转换为在所述目标画布的目标画布坐标，可以认为是将各操作点映射到目标画布对应的内存空间。
步骤S120、根据各操作点的目标画布坐标，在所述目标画布上绘制图形。
在将各操作点的屏幕坐标转换为在目标画布的目标画布坐标后，本申请实施例可根据各操作点的目标画布坐标，确定出各操作点在目标画布上的映射点，从而可根据画笔算法(如直线画笔、纹理画笔等)，连接各操作点在目标画布上的映射点(如以各操作点的操作先后顺序进行连接)，绘制出相应的图形。
可见，本申请实施例所指的涂鸦可以认为是，用户可通过手指或触摸笔等作为画笔，在屏幕上进行作画操作。其中，通过作画操作涉及的各操作点在目标画布上的映射点，可在目标画布上实现图形绘制。本申请实施例所指的涂鸦模式，即认为是允许用户通过手指或触摸笔等在屏幕上进行作画操作的模式。
可以理解的是，用户通过手指或触摸笔等在屏幕上进行滑动等操作，可由 屏幕上的各操作点形成至少一个操作轨迹，基于各操作点在目标画布上的映射点进行图形绘制，则可在目标画布上绘制出形似所述操作轨迹的图形。
步骤S130、将绘制有图形的目标画布叠加在所述视频图像上。
绘制有图形的目标画布可以作为是贴纸类型的特效素材，可基于贴纸叠加到视频图像的原理，将绘制有图形的目标画布叠加在所述视频图像上。
可选的，目标画布上绘制的图形可实时的叠加在视频图像上，使得用户可基于实时叠加在视频图像上的目标画布的图形，调整涂鸦操作，进行图形的修改和调整。
需要说明的是，本申请实施例提供的视频图像处理方法，支持用户在视频图像上进行涂鸦，具体地，在屏幕显示所采集的视频图像，用户可以基于该视频图像触发进入涂鸦模式，基于此，检测用户在屏幕上进行涂鸦的各操作点，将各操作点的屏幕坐标转换为在目标画布的目标画布坐标后，以各操作点的目标画布坐标在目标画布上进行图形绘制，再将绘制有图形的目标画布作为特效素材叠加到视频图像上，实现在视频图像上进行涂鸦的效果。本申请实施例的这种视频图像处理方法，可使得基于动态视频图像的实时涂鸦成为可能，与传统的以静态图片形式的视频图像作为画布进行涂鸦的方式存在本质区别。
在一个可选应用示例中，如图3所示，用户打开终端的视频摄影类应用进行自拍，终端的前置摄像头可采集包含用户人脸的视频图像并显示到终端屏幕上；用户点击特效素材面板的“涂鸦”图标，可触发进入涂鸦模式，终端可初始化目标画布，为目标画布申请内存空间；并且，用户可参照屏幕显示的视频图像的用户人脸位置，使用手指在屏幕上进行涂鸦操作，如图3所示，通过用户手指在屏幕上的滑动来进行涂鸦操作，且手指在屏幕上的滑动轨迹形似“眼镜”；
在用户的手指在屏幕上滑动的过程中，终端可记录滑动过程中涉及的屏幕各操作点的屏幕坐标，将各操作点的屏幕坐标转换为在目标画布的目标画布坐标，得到各操作点在目标画布相应的映射点；从而根据各操作点的目标画布坐标，可将各操作点在目标画布相应的映射点进行连接，在目标画布上绘制出形似“眼镜”的图形；
绘制有形似“眼镜”的图形的目标画布可以贴纸的形式叠加到视频图像的 用户人脸上，并实现目标画布上的形似“眼镜”的图形随视频图像的人脸移动的效果(该效果如虚拟人脸挂件等动效贴纸随视频图像的人脸移动)，达成在视频图像的用户人脸上进行涂鸦的目的；
需要说明的是，图3中用户手指实际上是在屏幕上进行移动涂鸦，屏幕所显示的视频图像添加图形的效果，实际上是基于用户手指在屏幕上的移动涂鸦，在目标画布进行图形绘制后，将目标画布叠加在视频图像上所呈现的效果。
图3相应部分内容所示的触发进入涂鸦模式的方式仅是可选的，可以认为是为便于理解的举例示意。
本申请实施例提供的视频图像处理方法包括：在屏幕显示所采集的视频图像后，若根据用户操作进入涂鸦模式，则可为目标画布申请内存空间，实现目标画布在内存中的缓存；进入涂鸦模式后，可检测用户在屏幕上进行涂鸦所涉及的各操作点，将各操作点的屏幕坐标转换为在所述目标画布的目标画布坐标，得到各操作点的目标画布坐标；进而可根据各操作点的目标画布坐标，在所述目标画布上绘制图形，将绘制有图形的目标画布叠加在所述视频图像上，实现在视频图像上添加用户所涂鸦的图形的特效的效果。
本申请实施例可根据用户在屏幕上进行涂鸦所涉及的各操作点，在内存的目标画布上进行相应图形的绘制，绘制有图形的目标画布可作为特效素材使用，因此特效素材的内容可随着用户在屏幕上的涂鸦操作的不同(即用户在屏幕上进行涂鸦所涉及的操作点的不同)而调整，使得特效素材的内容可由用户个性化的调整，使得在视频图像上添加的视频特效的内容更为丰富多变，降低了特效素材的使用局限，提升了对视频图像添加特效的灵活性。例如，在用户自拍、视频通话等场景下，可基于本申请实施例提供的视频图像处理方法，在自拍图像、视频通话图像等图像上实现用户绘制的特效素材的添加；进一步，将所绘制的特效素材以贴纸形式添加到视频图像中，则可在自拍、视频通话等场景相应采集的动态视频图像中，实现所绘制的特效素材跟随动态视频图像中的人脸移动的效果。
可选的，在一种将操作点的屏幕坐标转换为在目标画布的目标画布坐标的可选实现上，本申请实施例可先绘制虚拟画布(该虚拟画布与申请了内存空间的目标画布不同)，虚拟画布的尺寸是可调整的，具体地，虚拟画布的尺寸与 视频图像中的脸部特征区域尺寸成预设比例关系，其中，该预设比例关系可以根据实际需求而设置。作为本申请的一个示例，虚拟画布可以以脸部的鼻梁骨为中心，根据脸部大小进行缩放，在具体实现时，可以假定该虚拟画布为正方形，其宽度可以是脸部两眼间距的两倍。然后，将操作点的屏幕坐标映射至虚拟画布中，得到各操作点的虚拟画布坐标，再将各操作点的虚拟画布坐标映射至目标画布中，得到各操作点在目标画布上的目标画布坐标；
可选的，这种方式可适用于为视频图像上的人脸添加视频特效的场景，在目标画布申请的内存空间的大小是固定的情况下，目标画布的大小并不适用于人脸的多样性，因此可通过虚拟画布进行人脸的多样性的适用，通过虚拟画布作为操作点的屏幕坐标至目标画布坐标的坐标转换的中间态，可使得操作点的屏幕坐标至目标画布的目标画布坐标的转换更为准确。
可选的，图4示出了本申请实施例提供的将各操作点的屏幕坐标转换为目标画布坐标的方法流程图，该方法可应用于终端，如可由终端的GPU执行实现；参照图4，该方法可以包括：
步骤S200、根据视频图像中的脸部特征确定虚拟画布。
虚拟画布与上述所指的目标画布不同，虚拟画布的尺寸与视频图像中的脸部特征区域尺寸成预设比例关系。在本申请实施例中，尺寸大小可以通过分辨率来表征，目标画布的尺寸也即目标画布的分辨率大小是预先设定的，而虚拟画布的尺寸也即虚拟画布的分辨率大小可根据视频图像中脸部的分辨率大小进行调整，即针对脸部的分辨率大小不同的视频图像，所相应绘制的虚拟画布的分辨率大小也可能是不同的。例如，用户在自拍时，由于与摄像头的距离不同，导致视频图像中人脸的分辨率也不同，距离较远时，其分辨率相对较小。即本申请实施例可根据视频图像中的人脸特征绘制虚拟画布。
在一种可选实现上，本申请实施例可通过人脸检测技术，检测视频图像中脸部第一特征点的坐标(以人脸为例，第一特征点可以如人脸的某一设定的五官特征点，如人脸的鼻尖、眉心、鼻梁骨等，人脸第一特征点的形式可根据实际情况设定)，然后，根据预设比例关系和视频图像中脸部区域尺寸确定虚拟画布的尺寸，接着，以脸部第一特征点的坐标为虚拟画布的中心点，根据所述虚拟画布的尺寸确定虚拟画布。
可选的，预设比例关系可以根据实际情况设定，具体数值可自行调节，以虚拟画布能覆盖到大部分屏幕为主。
步骤S210、将各操作点的屏幕坐标映射至所述虚拟画布中，得到各操作点的虚拟画布坐标。
可选的，在确定虚拟画布后，对于任一操作点，本申请实施例可根据操作点的屏幕坐标和虚拟画布的中心点的屏幕坐标(即视频图像中人脸第一特征点的坐标)，将操作点的屏幕坐标转换为虚拟画布中的坐标，得到操作点的虚拟画布坐标。在具体实现时，针对各操作点，终端可以根据操作点的屏幕坐标和虚拟画布的中心点的屏幕坐标，计算得到坐标平移向量，然后根据坐标平移向量确定操作点的虚拟画布坐标。
显然，上述将操作点的屏幕坐标转换为在虚拟画布的坐标的方式仅是可选的，本申请实施例也可通过记录屏幕的坐标系与虚拟画布的坐标系的映射关系，根据该映射关系确定操作点的屏幕坐标在虚拟画布中转换得到的坐标。
步骤S220、将各操作点的虚拟画布的坐标进行归一化，得到各操作点的归一化坐标。
可选的，本申请实施例可设定归一化后的虚拟画布的尺寸，该为了方便表述，可以记作指定归一化尺寸。如虚拟画布归一化后，虚拟画布的x轴和y轴方向的范围均是-1到1，则可认为虚拟画布的尺寸为2*2，如图5所示。可选的，指定归一化尺寸可以根据实际情况自定义调节，除如图5所示的可设定归一化后的虚拟画布为正方性，宽度为2外，还可设置归一化后的虚拟画布的宽度为4等(如虚拟画布的x轴和y轴方向的范围-2到2)。
在确定指定归一化尺寸后，本申请实施例可基于虚拟画布的尺寸以及指定归一化尺寸，确定第一坐标缩放比例；如将归一化后的虚拟画布的宽度，除以，虚拟画布绘制时的宽度，得到第一坐标缩放比例；
示例的，设归一化后的虚拟画布的宽度为2，虚拟画布绘制时的宽度为680，则第一坐标缩放比例为scale＝2/680。
在得到第一坐标缩放比例后，可根据操作点在虚拟画布的坐标与第一坐标缩放比例，确定操作点在虚拟画布的初始归一化坐标；如将操作点在虚拟画布的坐标乘以该第一坐标缩放比例，得到操作点在虚拟画布的初始归一化坐标。
可选的，上述示出的确定操作点在虚拟画布的初始归一化坐标的方式是可选的，本申请实施例也可设定虚拟画布的坐标系与归一化后的虚拟画布的坐标系的映射关系，根据该映射关系，得到各操作点的初始归一化坐标。
在得到操作点在虚拟画布的初始归一化坐标后，可分视频图像中人脸的旋转角度为0和不为0的情况，进行后续得到各操作点的归一化坐标的处理；
具体的，若视频图像中人脸的旋转角度为0，则可直接将操作点在虚拟画布的初始归一化坐标，作为各操作点的归一化坐标，即得到操作点在虚拟画布的初始归一化坐标，则认为得到了各操作点的归一化坐标；
若视频图像中人脸的旋转角度不为0，则在得到各操作点的初始归一化坐标后，还需将各操作点的初始归一化坐标逆向旋转该旋转角度，得到各操作点的归一化坐标。
步骤S230、将各操作点的归一化坐标映射至目标画布中，得到各操作点的目标画布坐标。
在得到各操作点的归一化坐标后，本申请实施例可根据目标画布的尺寸和指定归一化尺寸，确定第二坐标缩放比例；如将目标画布的宽度，除以，归一化后的虚拟画布的宽度，得到第二坐标缩放比例；
从而对于任一操作点，根据操作点的归一化坐标和第二坐标缩放比例，得到操作点在目标画布上的目标画布坐标；如可将设定数值加上操作点的归一化x轴坐标后，乘以第二坐标缩放比例，得到操作点的x轴目标画布坐标，将设定数值减去操作点的归一化y轴坐标后，乘以第二坐标缩放比例，得到操作点的y轴目标画布坐标，结合操作点的x轴目标画布坐标和y轴目标画布坐标，得到操作点在目标画布上的目标画布坐标；
可选的，设定数值可根据实际情况自定义；具体可以根据虚拟画布的尺寸而确定。示例的，虚拟画布尺寸为2*2，其中心点为原点，而目标画布是以左上角的顶点在目标画布的映射点为原点，以x轴正方向朝右，y轴正方向朝下，则设定数值为1，则对于操作点的归一化坐标(x’，y’)，先将中心点平移，得到(1+x’，1-y’)，而第二坐标缩放比例为512/2(其中，512假设为目标画布的宽度，2假设为归一化后的虚拟画布的宽度)，则可通过如下方式得到操作点在目标画布上的目标画布坐标：(1+x’，1-y’)*512/2。
可选的，步骤S220至步骤S230仅是将各操作点在虚拟画布的坐标转换为，各操作点在目标画布上的目标画布坐标的一种可选方式，本申请实施例也可设置虚拟画布与目标画布的坐标映射关系，根据该坐标映射关系，实现各操作点在虚拟画布的坐标至在目标画布上的目标画布坐标的转换。
本申请实施例在对视频图像的人脸进行涂鸦时，可通过虚拟画布作为操作点的屏幕坐标至目标画布坐标的坐标转换的中间态，由基于视频图像的人脸绘制的虚拟画布进行人脸的多样性的适用，可使得操作点的屏幕坐标至目标画布的目标画布坐标的转换更为准确，提升后续绘制有图形的目标画布在视频图像的人脸上叠加的准确性。
可选的，在视频图像的人脸的旋转角度为0的情况下，图6示出了本申请实施例提供的将各操作点的屏幕坐标转换为目标画布坐标的另一方法流程图，参照图6，该方法可以包括：
步骤S300、检测视频图像中脸部第一特征点的屏幕坐标。
步骤S310、以所述脸部第一特征点的屏幕坐标为虚拟画布的中心点，根据所述预设比例关系和所述视频图像中脸部区域尺寸确定虚拟画布的尺寸，根据所述虚拟画布的尺寸确定虚拟画布。
可选的，步骤S300至步骤S310可以认为是图4所示步骤S200的一种可选实现，其中，视频图像中人脸的旋转角度为0。
步骤S320、对于各操作点，根据操作点的屏幕坐标和虚拟画布的中心点的屏幕坐标，计算得到坐标平移向量，然后根据该坐标平移向量确定操作点的虚拟画布坐标。
坐标平移向量可以通过对操作点的屏幕坐标和虚拟画布的中心点的屏幕坐标进行减法运算得到。由于虚拟画布是将中心点平移得到，因此，坐标平移向量中x方向的分量即可视为虚拟画布的x轴坐标，y方向的分量即可视为虚拟画布的y轴坐标，如此，将操作点的x轴屏幕坐标减去虚拟画布的中心点的x轴坐标，将操作点的y轴屏幕坐标减去虚拟画布的中心点的y轴坐标，得到操作点在虚拟画布的坐标。其中，操作点在虚拟画布的坐标即为操作点的虚拟画布坐标。可选的，步骤S320可以认为是图4所示步骤S210将各操作点的屏幕坐标映射至所述虚拟画布中，得到各操作点的虚拟画布坐标的一种实现。
在一种可选实现上，假设虚拟画布的中心点的坐标(即视频图像中人脸第一特征点的坐标)为P1(360，500)，一操作点的屏幕坐标为P0(120，660)，则该操作点转换后的在虚拟画布的坐标可以为：P2＝(120-360，660-500)＝(-240,160)；即可采用操作点的x轴屏幕坐标减去虚拟画布的中心点的x轴坐标，采用操作点的y轴屏幕坐标减去虚拟画布的中心点的y轴坐标，得到操作点转换后的在虚拟画布的坐标；
即假设虚拟画布的中心点的坐标P1(P1x，P1y)，操作点的屏幕坐标为P0(P0x,P0y)，则操作点转换后的在虚拟画布的坐标也即虚拟画布坐标P2(P2x，P2y)＝(P0x-P1x，P0y-P1y)。
可选的，步骤S320是基于二维坐标系下的处理，在三维坐标系下，人脸检测还支持欧拉角检测时(三维空间里人脸旋转角度与x、y、z轴的夹角)，则将操作点的x轴屏幕坐标减去虚拟画布的中心点的x轴坐标，将操作点的y轴屏幕坐标减去虚拟画布的中心点的y轴坐标后，还需将所得结果乘以三维旋转矩阵，得到操作点在虚拟画布的坐标；
三维旋转矩阵的形式如下：
Figure PCTCN2018112304-appb-000001
步骤S330、根据指定归一化尺寸和虚拟画布的尺寸，确定第一坐标缩放比例。
可选的，如可将归一化后的虚拟画布的宽度，除以，虚拟画布绘制时的宽度，得到第一坐标缩放比例。
步骤S340、对于各操作点，根据操作点的虚拟画布坐标和第一坐标缩放比例，确定各操作点的归一化坐标。
可选的，如可将操作点在虚拟画布的坐标乘以该第一坐标缩放比例，得到操作点的归一化坐标。
可选的，本申请实施例可将操作点的虚拟画布坐标P2乘以第一坐标缩放比例scale，得到操作点在虚拟画布的归一化坐标；示例的，设操作点在虚拟画布的坐标P2为(-240，160)，第一坐标缩放比例scale为2/680，则操作点在虚拟画布的归一化坐标为P2*scale＝(-240，160)*2/680＝(-0.706，0.471)，此 处为便于说明以小数点后三位的四舍五入方式示出(-0.706，0.471)的结果，在实际情况下，具体的结果给出方式可自定义调整。
可选的，步骤S330至步骤S340可以认为是图4步骤S220的一种实现。
步骤S350、根据目标画布的尺寸和指定归一化尺寸确定第二坐标缩放比例。
可选的，可将目标画布的宽度，除以，归一化后的虚拟画布的宽度，得到第二坐标缩放比例。
步骤S360、对于各操作点，根据操作点的归一化坐标和第二坐标缩放比例，得到操作点的目标画布坐标。
在具体实现时，将设定数值加上操作点的归一化x轴坐标后，乘以第二坐标缩放比例，得到操作点的x轴目标画布坐标，将设定数值减去操作点的归一化y轴坐标后，乘以第二坐标缩放比例，得到操作点的y轴目标画布坐标，结合操作点的x轴目标画布坐标和y轴目标画布坐标，得到操作点在目标画布上的目标画布坐标。
可选的，在视频图像的人脸的旋转角度不为0的情况下，图7示出了本申请实施例提供的将各操作点的屏幕坐标转换为目标画布坐标的再一方法流程图，参照图7，该方法可以包括：
步骤S400、检测视频图像中脸部第一特征点的坐标。
步骤S410、以所述脸部第一特征点的屏幕坐标为虚拟画布的中心点，根据所述预设比例关系和所述视频图像中脸部区域尺寸确定虚拟画布的尺寸，根据所述虚拟画布的尺寸确定虚拟画布。
在一种示例中，虚拟画布可以为正方形，本申请实施例可以视频图像中人脸鼻梁骨的坐标为虚拟画布的中心点，以人脸宽度的两倍为虚拟画布的宽度确定虚拟画布；
例如，假设屏幕的分辨率大小为720*1280，在确定视频图像中人脸鼻梁骨的坐标为(360，500)，人脸宽度的分辨率大小为340(可认为是人脸左右太阳穴坐标点之间的宽度距离)；则可以人脸鼻梁骨的坐标(360，500)为虚拟画布的中心，得到人脸宽度的两倍(680*680分辨率大小)的虚拟画布。可选的，步骤S400和步骤S410可以认为是图4所示步骤S200的一种可选实现，其中， 视频图像中人脸的旋转角度不为0。
需要说明的是，图6所示步骤S300至步骤S310，和图7所示步骤S400和步骤S410的区别在于：图6的视频图像中人脸的旋转角度为0，而图7的视频图像中人脸的旋转角度不为0。
步骤S420、对于各操作点，根据操作点的屏幕坐标和虚拟画布的中心点的屏幕坐标，计算得到坐标平移向量，根据所述坐标平移向量确定操作点的虚拟画布坐标。
在具体实现时，将操作点的x轴屏幕坐标减去虚拟画布的中心点的x轴坐标，将操作点的y轴屏幕坐标减去虚拟画布的中心点的y轴坐标，得到操作点在虚拟画布的坐标。
可选的，步骤S420可以认为是图4所示步骤S210的一种实现，可与图6所示步骤S320相参照。
可选的，如果人脸检测还支持欧拉角检测时(三维空间里人脸旋转角度与x、y、z轴的夹角)，则将操作点的x轴屏幕坐标减去虚拟画布的中心点的x轴坐标，将操作点的y轴屏幕坐标减去虚拟画布的中心点的y轴坐标后，还需乘以三维旋转矩阵，得到操作点在虚拟画布的坐标。
步骤S430、根据指定归一化尺寸和虚拟画布的尺寸，确定第一坐标缩放比例。
可选的，如可将归一化后的虚拟画布的宽度，除以，虚拟画布绘制时的宽度，得到第一坐标缩放比例。
步骤S440、对于各操作点，根据操作点在虚拟画布的坐标和第一坐标缩放比例，确定各操作点的初始归一化坐标。
可选的，如可将操作点在虚拟画布的坐标乘以该第一坐标缩放比例，得到操作点的初始归一化坐标。
步骤S450、将各操作点的初始归一化坐标分别逆向旋转该旋转角度，得到各操作点的归一化坐标。
可选的，本申请实施例可确定所述旋转角度相应的二维旋转矩阵，对于各操作点，可根据操作点的初始归一化坐标和该二维旋转矩阵，确定各操作点的归一化坐标；如可将操作点的初始归一化坐标乘以该二维旋转矩阵，得到操作 点的归一化坐标。
设定视频图像中人脸的旋转角度为θ，M(θ)表示相应的二维旋转矩阵，则二维旋转矩阵的公式示例可如下：
Figure PCTCN2018112304-appb-000002
示例的，设一操作点的初始归一化坐标为(-0.706，0.471)，则该操作点的归一化坐标的得到方式可以如下：
M(θ)*(-0.706，0.471)；如设定θ为30°，则归一化坐标的结果为(-0.376，0.761)。
如图8所示，由于视频图像中脸部存在旋转角度，因此，在对虚拟画布坐标进行归一化时，需要对虚拟画布坐标进行旋转，如此，相当于对虚拟画布进行旋转。其中，旋转角度的正负方向可以以视频图像的中心线为参照自定义，本申请实施例对此不作限定。
可选的，步骤S430至步骤S450可以认为是图4步骤S220的一种实现，与图6所示步骤S330至步骤S340的区别在于，图7所示方法是基于视频图像的脸部旋转角度不为0的情况，图6步骤S340所得结果可以认为是图7所示方法的初始归一化坐标，在此基础上，图7所示方法还需基于所述旋转角度进行各操作点的初始归一化坐标的逆向旋转。
步骤S460、根据目标画布的尺寸和指定归一化尺寸确定第二坐标缩放比例。
可选的，可将目标画布的宽度，除以，归一化后的虚拟画布的宽度，得到第二坐标缩放比例。
步骤S470、对于各操作点，根据操作点的归一化坐标和第二坐标缩放比例，得到操作点的目标画布坐标。
具体地，可以将设定数值加上操作点的归一化x轴坐标后，乘以第二坐标缩放比例，得到操作点的x轴目标画布坐标，将设定数值减去操作点的归一化y轴坐标后，乘以第二坐标缩放比例，得到操作点的y轴目标画布坐标，结合操作点的x轴目标画布坐标和y轴目标画布坐标，得到操作点在目标画布上的目标 画布坐标。
这里需要说明的是，本申请实施例支持二维坐标系下的处理，也支持三维坐标系下的处理，两者的区别之一在于：相比于二维坐标系下，三维坐标系下进行操作点的屏幕坐标至虚拟画布的坐标的转换时，还需在二维坐标系下的处理基础上乘以三维旋转矩阵，如步骤S320、S420部分所示。
上文示出了在对视频图像的人脸进行涂鸦的场景下，由虚拟画布作为操作点的屏幕坐标至目标画布坐标的坐标转换的中间态，实现人脸的多样性的适用的目的；需要说明的是，通过虚拟画布进行操作点的屏幕坐标至目标画布坐标的坐标转换仅是一种可选方案，作为一种可选替换方案：如在不是对视频图像的人脸进行涂鸦等情况下，本申请实施例也可在屏幕显示与目标画布大小相应的涂鸦区域(可选的，进一步，该涂鸦区域的位置可受用户操作移动调整，但大小不变)，使得用户在该涂鸦区域内进行涂鸦，实现操作点的屏幕坐标至目标画布坐标的直接转换(此处的直接是相对于以虚拟画布作为中间态的间接方式而言)；相应的，绘制有图形的目标画布可根据涂鸦区域在视频图像中的位置，进行至视频图像的叠加。
进一步，在确定屏幕的各操作点的目标画布坐标，并在所述目标画布上绘制出相应图形后，可将绘制有图形的目标画布以贴纸形式，添加到动态的视频图像中，实现绘制有图形的目标画布跟随视频图像中的人脸移动的效果；
可选的，在根据各操作点的目标画布坐标，在目标画布上绘制图形后，绘制有图形的目标画布可作为贴纸类型的特效素材，叠加在视频图像上；如采用openGL为例，本申请实施例可将绘制有图形的目标画布的顶点坐标及旋转角度(与视频图像中脸部的旋转角度相应)传入顶点着色器，由顶点着色器根据旋转角度及绘制有图形的目标画布的顶点坐标，计算得到绘制有图形的目标画布在视频图像上的叠加位置坐标，从而根据所述叠加位置坐标，将绘制有图形的目标画布叠加到视频图像上；进一步，在将绘制有图形的目标画布添加到视频图像的脸部时，若视频图像为动态视频图像，脸部存在移动情况，则可使得绘制有图形的目标画布跟随视频图像中的脸部移动。
可选的，若在三维坐标下绘制有图形的目标画布至视频图像的叠加，则可采用三维空间贴纸至视频图像的叠加原理，比如采用透视投影技术等。
可选的，本申请实施例提供的视频图像处理方法可以画笔滤镜(painter filter)的形式存在，可加入到支持视频录制的应用的滤镜链中，滤镜链中还存在的滤镜如磨皮滤镜，美白滤镜，人脸检测，形变滤镜等，本申请实施例提供的视频图像处理方法可以画笔滤镜的形式加入到滤镜链末尾，以滤镜的处理形式实现对视频图像的涂鸦，涂鸦后的叠加有绘制图形的目标画布的视频图像数据可输出到屏幕或者写入本地视频文件保存。加入画笔滤镜的应用程序框架可如图9所示，可参照。
本申请实施例提供的视频图像处理方法，可支持用户在视频图像上进行涂鸦，且将基于用户涂鸦得到的绘制有图形的目标画布可作为特效素材使用，特效素材的内容可随着用户在屏幕上的涂鸦操作的不同而调整，使得特效素材的内容可由用户个性化的调整，使得在视频图像上添加的视频特效的内容更为丰富多变，降低了特效素材的使用局限，提升了对视频图像添加特效的灵活性。
下面对本申请实施例提供的视频图像处理装置进行介绍，下文描述的视频图像处理装置可以认为是，终端为实现本申请实施例提供的视频图像处理方法所需设置的程序模块。下文描述的视频图像处理装置的内容可与上文描述的视频图像处理方法的内容相互对应参照。
图10为本申请实施例提供的视频图像处理装置的结构框图，该视频图像处理装置可应用于终端，具体可应用于终端的GPU，参照图10，该视频图像处理装置可以包括：
视频图像显示模块100，用于在屏幕显示所采集的视频图像；
坐标转换模块200，用于检测用户在屏幕的各操作点，将各操作点的屏幕坐标转换为在目标画布的目标画布坐标；
图形绘制模块300，用于根据各操作点的目标画布坐标，在所述目标画布上绘制图形；
叠加模块400，用于将绘制有图形的目标画布叠加在所述视频图像上。
可选的，坐标转换模块200，用于将各操作点的屏幕坐标转换为在所述目标画布的目标画布坐标，具体包括：
根据视频图像中的脸部特征确定虚拟画布，所述虚拟画布的尺寸与所述视频图像中的脸部特征区域尺寸成预设比例关系；
将各操作点的屏幕坐标映射至所述虚拟画布中，得到各操作点的虚拟画布坐标；
将各操作点在虚拟画布坐标映射至目标画布中，得到各操作点在目标画布上的目标画布坐标。
可选的，坐标转换模块200，用于根据视频图像中的脸部特征确定虚拟画布，具体包括：
检测视频图像中脸部第一特征点的屏幕坐标；
以脸部第一特征点的屏幕坐标为虚拟画布的中心点，根据所述预设比例关系和所述视频图像中脸部区域尺寸确定虚拟画布的尺寸，根据所述虚拟画布的尺寸确定虚拟画布。
可选的，坐标转换模块200，用于将各操作点的屏幕坐标映射至所述虚拟画布中，得到各操作点的虚拟画布坐标，具体包括：
对于各操作点，根据操作点的屏幕坐标和虚拟画布的中心点的屏幕坐标，将操作点的屏幕坐标转换为虚拟画布的坐标，得到操作点的虚拟画布坐标。
可选的，在二维坐标系下，坐标转换模块200，用于根据操作点的屏幕坐标和虚拟画布的中心点的坐标，将操作点的屏幕坐标转换为虚拟画布的坐标，具体包括：
对于各操作点，根据操作点的屏幕坐标和虚拟画布的中心点的屏幕坐标，计算得到坐标平移向量；
根据所述坐标平移向量确定操作点的虚拟画布坐标。
可选的，在三维坐标系下，坐标转换模块200，用于根据操作点的屏幕坐标和虚拟画布的中心点的坐标，将操作点的屏幕坐标转换为虚拟画布的坐标，具体包括：
对于各操作点，将操作点的x轴屏幕坐标减去虚拟画布的中心点的x轴坐标，将操作点的y轴屏幕坐标减去虚拟画布的中心点的y轴坐标后，将所得到结果乘以三维旋转矩阵，得到操作点在虚拟画布的坐标。
可选的，坐标转换模块200，用于将各操作点的虚拟画布坐标映射至所述目标画布中，得到各操作点的的目标画布坐标，具体包括：
将各操作点的虚拟画布坐标进行归一化，得到各操作点的归一化坐标；
将各操作点的归一化坐标映射至所述目标画布中，得到各操作点的目标画布坐标。
可选的，若视频图像中人脸的旋转角度不为0，则坐标转换模块200，用于将各操作点的虚拟画布坐标进行归一化，得到各操作点的归一化坐标，具体包括：
根据指定归一化尺寸和所述虚拟画布的尺寸，确定第一坐标缩放比例；
对于各操作点，根据操作点的虚拟画布坐标和第一坐标缩放比例，确定各操作点的初始归一化坐标；
将各操作点的初始归一化坐标分别逆向旋转所述旋转角度，得到各操作点的归一化坐标。
可选的，坐标转换模块200在根据指定归一化尺寸和虚拟画布的尺寸确定第一坐标缩放比例时，具体可用于：将归一化后的虚拟画布的宽度，除以，虚拟画布绘制时的宽度，得到第一坐标缩放比例。
可选的，坐标转换模块200在根据操作点的虚拟画布坐标和第一坐标缩放比例，确定各操作点的初始归一化坐标时，具体可用于：将操作点的虚拟画布坐标乘以该第一坐标缩放比例，得到操作点的初始归一化坐标。
可选的，坐标转换模块200在将各操作点的初始归一化坐标分别逆向旋转所述旋转角度，得到各操作点的归一化坐标时，具体可用于：将操作点的初始归一化坐标乘以所述旋转角度相应的二维旋转矩阵，得到操作点的归一化坐标。
可选的，若视频图像中人脸的旋转角度为0，则坐标转换模块200，用于将各操作点的虚拟画布坐标进行归一化，得到各操作点的归一化坐标，具体包括：
根据指定归一化尺寸和虚拟画布的尺寸，确定第一坐标缩放比例；
对于各操作点，根据操作点的虚拟画布坐标和第一坐标缩放比例，确定各操作点的归一化坐标。
可选的，在视频图像中人脸的旋转角度为0的情况下，坐标转换模块200在根据操作点的虚拟画布坐标和第一坐标缩放比例，确定各操作点的归一化坐标时，具体可用于：可将操作点的虚拟画布坐标乘以该第一坐标缩放比例，得到操作点的归一化坐标。
可选的，坐标转换模块200，用于将各操作点的归一化坐标映射至所述目标画布中，得到各操作点的目标画布坐标，具体包括：
根据目标画布的尺寸和指定归一化尺寸确定第二坐标缩放比例；
对于各操作点，根据操作点的归一化坐标和第二坐标缩放比例，得到操作点的目标画布坐标。
可选的，坐标转换模块200在根据目标画布的尺寸和指定归一化尺寸确定第二坐标缩放比例时，具体可用于：将目标画布的宽度，除以，归一化后的虚拟画布的宽度，得到第二坐标缩放比例。
可选的，坐标转换模块200在根据操作点的归一化坐标和第二坐标缩放比例，得到操作点的目标画布坐标时，具体可用于：
对于各操作点，将设定数值加上操作点的归一化x轴坐标后，乘以第二坐标缩放比例，得到操作点的x轴目标画布坐标；将设定数值减去操作点的归一化y轴坐标后，乘以第二坐标缩放比例，得到操作点的y轴目标画布坐标；结合操作点的x轴目标画布坐标和y轴目标画布坐标，得到操作点在目标画布上的目标画布坐标。
可选的，坐标转换模块200，用于检测用户在屏幕的各操作点，具体包括：
响应于用户触发的进入涂鸦模式的操作，检测用户在屏幕的各操作点。
可选的，叠加模块400，用于将绘制有图形的目标画布叠加在所述视频图像上，具体包括：
将绘制有图形的目标画布的顶点坐标及视频图像中脸部的旋转角度传入顶点着色器，由顶点着色器根据所述旋转角度及顶点坐标，计算得到绘制有图形的目标画布在视频图像上的叠加位置坐标；
根据叠加位置坐标，将绘制有图形的目标画布叠加到视频图像上，且使绘制有图形的目标画布跟随视频图像中的脸部移动。
可选的，在另一种实现上，坐标转换模块200，用于检测用户在屏幕的各操作点，将各操作点的屏幕坐标转换为在目标画布的目标画布坐标，可具体包括：
检测用户在屏幕显示的与目标画布大小相应的涂鸦区域的操作点，将所述涂鸦区域的操作点的屏幕坐标转换为在所述目标画布的目标画布坐标。
可选的，在三维坐标系下，叠加模块400，可使用透视投影技术，将绘制有图形的目标画布叠加在所述视频图像上。
可选的，图11示出了终端的硬件结构框图，参照图11，该终端可以包括：至少一个处理器1，至少一个通信接口2，至少一个存储器3，至少一个通信总线4和至少一个图形处理器5；
可选的，通信接口2可以为通信模块的接口，如GSM模块的接口；
处理器1可能是一个中央处理器CPU，或者是特定集成电路ASIC(Application Specific Integrated Circuit)，或者是被配置成实施本申请实施例的一个或多个集成电路。
存储器3可能包含高速RAM存储器，也可能还包括非易失性存储器(non-volatile memory)，例如至少一个磁盘存储器。
其中，存储器3存储有可执行的程序，该程序可被处理器或图形处理器执行(在一些特殊情况下，本申请实施例提供的视频图像处理方法相应的程序也可由处理器执行实现)，该程序具体用于：
在屏幕显示所采集的视频图像；
检测用户在屏幕的各操作点，将各操作点的屏幕坐标转换为在目标画布的目标画布坐标；
根据各操作点的目标画布坐标，在所述目标画布上绘制图形；
将绘制有图形的目标画布叠加在所述视频图像上。
可选的，该程序的细化功能和扩展功能可参照上文相应部分描述。
本申请实施例还可提供一种存储介质，该存储介质存储有适于处理器执行的程序，该程序用于：
在屏幕显示所采集的视频图像；
检测用户在屏幕的各操作点，将各操作点的屏幕坐标转换为在目标画布的目标画布坐标；
根据各操作点的目标画布坐标，在所述目标画布上绘制图形；
将绘制有图形的目标画布叠加在所述视频图像上。
本申请实施例还提供一种包括指令的计算机程序产品，当其在计算机上运行时，使得所述计算机执行本申请所述的视频图像处理方法。
本说明书中各个实施例采用递进的方式描述，每个实施例重点说明的都是与其他实施例的不同之处，各个实施例之间相同相似部分互相参见即可。对于实施例公开的装置而言，由于其与实施例公开的方法相对应，所以描述的比较简单，相关之处参见方法部分说明即可。
专业人员还可以进一步意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、计算机软件或者二者的结合来实现，为了清楚地说明硬件和软件的可互换性，在上述说明中已经按照功能一般性地描述了各示例的组成及步骤。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
结合本文中所公开的实施例描述的方法或算法的步骤可以直接用硬件、处理器执行的软件模块，或者二者的结合来实施。软件模块可以置于随机存储器(RAM)、内存、只读存储器(ROM)、电可编程ROM、电可擦除可编程ROM、寄存器、硬盘、可移动磁盘、CD-ROM、或技术领域内所公知的任意其它形式的存储介质中。
对所公开的实施例的上述说明，使本领域专业技术人员能够实现或使用本申请。对这些实施例的多种修改对本领域的专业技术人员来说将是显而易见的，本文中所定义的一般原理可以在不脱离本申请的核心思想或范围的情况下，在其它实施例中实现。因此，本申请将不会被限制于本文所示的这些实施例，而是要符合与本文所公开的原理和新颖特点相一致的最宽的范围。

Claims (16)

  1. 一种视频图像处理方法，应用于终端，包括：
    在屏幕显示所采集的视频图像；
    检测用户在屏幕的各操作点，将各操作点的屏幕坐标转换为在目标画布的目标画布坐标；
    根据各操作点的目标画布坐标，在所述目标画布上绘制图形；
    将绘制有图形的目标画布叠加在所述视频图像上。
  2. 根据权利要求1所述的视频图像处理方法，所述将各操作点的屏幕坐标转换为在目标画布的目标画布坐标包括：
    根据视频图像中的脸部特征确定虚拟画布，所述虚拟画布的尺寸与所述视频图像中的脸部特征区域尺寸成预设比例关系；将各操作点的屏幕坐标映射至所述虚拟画布中，得到各操作点的虚拟画布坐标；
    将各操作点的虚拟画布坐标映射至目标画布中，得到各操作点在所述目标画布上的目标画布坐标。
  3. 根据权利要求2所述的视频图像处理方法，所述根据视频图像中的脸部特征确定虚拟画布包括：
    检测视频图像中脸部第一特征点的屏幕坐标；
    以脸部第一特征点的屏幕坐标为虚拟画布的中心点，根据所述预设比例关系和所述视频图像中脸部区域尺寸确定虚拟画布的尺寸，
    根据所述虚拟画布的尺寸确定虚拟画布。
  4. 根据权利要求2所述的视频图像处理方法，所述将各操作点的屏幕坐标映射至所述虚拟画布中，得到各操作点的虚拟画布坐标包括：
    对于各操作点，根据操作点的屏幕坐标和虚拟画布的中心点的屏幕坐标，将操作点的屏幕坐标转换为虚拟画布的坐标，得到操作点的虚拟画布坐标。
  5. 根据权利要求4所述的视频图像处理方法，其特征在于，所述根据操作点的屏幕坐标和虚拟画布的中心点的坐标，将操作点的屏幕坐标转换为虚拟画布的坐标包括：
    对于各操作点，根据操作点的屏幕坐标和虚拟画布的中心点的屏幕坐标，计算得到坐标平移向量；
    根据所述坐标平移向量确定操作点的虚拟画布坐标。
  6. 根据权利要求2所述的视频图像处理方法，其特征在于，所述将各操作点的虚拟画布坐标映射至所述目标画布中，得到各操作点的目标画布坐标包括：
    将各操作点的虚拟画布坐标进行归一化，得到各操作点的归一化坐标；
    将各操作点的归一化坐标映射至所述目标画布中，得到各操作点的目标画布坐标。
  7. 根据权利要求6所述的视频图像处理方法，其特征在于，所述将各操作点的虚拟画布坐标进行归一化，得到各操作点的归一化坐标包括：
    若视频图像中脸部的旋转角度不为0，根据指定归一化尺寸和所述虚拟画布的尺寸，确定第一坐标缩放比例；
    对于各操作点，根据操作点的虚拟画布坐标和第一坐标缩放比例，确定各操作点的初始归一化坐标；
    将各操作点的初始归一化坐标分别逆向旋转所述旋转角度，得到各操作点的归一化坐标。
  8. 根据权利要求6所述的视频图像处理方法，其特征在于，所述将各操作点的虚拟画布坐标进行归一化，得到各操作点的归一化坐标包括：
    若视频图像中脸部的旋转角度为0，根据指定归一化尺寸和所述虚拟画布的尺寸，确定第一坐标缩放比例；
    对于各操作点，根据操作点的虚拟画布坐标和第一坐标缩放比例，确定各操作点的归一化坐标。
  9. 根据权利要求6-8任一项所述的视频图像处理方法，其特征在于，所述将各操作点的归一化坐标映射至所述目标画布中，得到各操作点的目标画布坐标包括：
    根据所述目标画布的尺寸和所述指定归一化尺寸，确定第二坐标缩放比例；
    对于各操作点，根据操作点的归一化坐标和第二坐标缩放比例，得到操作点的目标画布坐标。
  10. 根据权利要求1所述的视频图像处理方法，所述检测用户在屏幕的各 操作点包括：
    响应于用户触发的进入涂鸦模式的操作，检测用户在屏幕的各操作点。。
  11. 根据权利要求1所述的视频图像处理方法，所述将绘制有图形的目标画布叠加在所述视频图像上包括：
    将绘制有图形的目标画布的顶点坐标及视频图像中脸部的旋转角度传入顶点着色器，由顶点着色器根据所述旋转角度及所述顶点坐标，计算得到绘制有图形的目标画布在视频图像上的叠加位置坐标；
    根据所述叠加位置坐标，将绘制有图形的目标画布叠加到视频图像上，且使绘制有图形的目标画布跟随视频图像中的脸部移动。
  12. 一种视频图像处理装置，包括：
    视频图像显示模块，用于在屏幕显示所采集的视频图像；
    坐标转换模块，用于检测用户在屏幕的各操作点，将各操作点的屏幕坐标转换为在目标画布的目标画布坐标；
    图形绘制模块，用于根据各操作点的目标画布坐标，在所述目标画布上绘制图形；
    叠加模块，用于将绘制有图形的目标画布叠加在所述视频图像上。
  13. 根据权利要求12所述的视频图像处理装置，所述坐标转换模块用于将各操作点的屏幕坐标转换为在目标画布的目标画布坐标，具体包括：
    根据视频图像中的脸部特征确定虚拟画布，所述虚拟画布的尺寸与所述视频图像中的脸部特征区域尺寸成预设比例关系；
    将各操作点的屏幕坐标映射至所述虚拟画布中，得到各操作点的虚拟画布坐标；
    将各操作点的虚拟画布坐标映射至所述目标画布中，得到各操作点的目标画布坐标。
  14. 一种终端，包括：存储器，处理器和图形处理器；所述存储器存储有可执行的程序，所述程序可被所述处理器或所述图形处理器执行，所述程序用于：
    在屏幕显示所采集的视频图像；
    检测用户在屏幕的各操作点，将各操作点的屏幕坐标转换为在目标画布的 目标画布坐标；
    根据各操作点的目标画布坐标，在所述目标画布上绘制图形；
    将绘制有图形的目标画布叠加在所述视频图像上。
  15. 一种存储介质，所述存储介质用于存储可执行指令，所述可执行指令在计算机上运行时，使得所述计算机执行权利要求1至11任意一项所述的视频图像处理方法。
  16. 一种包括指令的计算机程序产品，当其在计算机上运行时，使得所述计算机执行权利要求1至11任意一项所述的视频图像处理方法。
PCT/CN2018/112304 2017-11-14 2018-10-29 视频图像处理方法、装置及终端 WO2019095979A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP18879128.9A EP3713220B1 (en) 2017-11-14 2018-10-29 Video image processing method and apparatus, and terminal
US16/854,807 US11140339B2 (en) 2017-11-14 2020-04-21 Video image processing method, apparatus and terminal

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201711124135.9A CN107888845B (zh) 2017-11-14 2017-11-14 一种视频图像处理方法、装置及终端
CN201711124135.9 2017-11-14

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/854,807 Continuation US11140339B2 (en) 2017-11-14 2020-04-21 Video image processing method, apparatus and terminal

Publications (1)

Publication Number Publication Date
WO2019095979A1 true WO2019095979A1 (zh) 2019-05-23

Family

ID=61776653

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/112304 WO2019095979A1 (zh) 2017-11-14 2018-10-29 视频图像处理方法、装置及终端

Country Status (4)

Country Link
US (1) US11140339B2 (zh)
EP (1) EP3713220B1 (zh)
CN (1) CN107888845B (zh)
WO (1) WO2019095979A1 (zh)

Families Citing this family (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107888845B (zh) 2017-11-14 2022-10-21 腾讯数码（天津）有限公司 一种视频图像处理方法、装置及终端
CN110661985B (zh) * 2018-06-28 2022-02-01 杭州海康威视数字技术股份有限公司 图像数据录制方法、装置、审讯主机及存储介质
CN108958610A (zh) 2018-07-27 2018-12-07 北京微播视界科技有限公司 基于人脸的特效生成方法、装置和电子设备
CN110795177B (zh) * 2018-08-03 2021-08-31 浙江宇视科技有限公司 图形绘制方法及装置
CN109618211A (zh) * 2018-12-04 2019-04-12 深圳市子瑜杰恩科技有限公司 短视频道具编辑方法及相关产品
CN111862349A (zh) 2019-04-26 2020-10-30 北京字节跳动网络技术有限公司 虚拟画笔实现方法、装置和计算机可读存储介质
CN111147880A (zh) * 2019-12-30 2020-05-12 广州华多网络科技有限公司 视频直播的互动方法、装置、***、电子设备及存储介质
CN111476873B (zh) * 2020-03-12 2023-11-10 浙江工业大学 一种基于增强现实的手机虚拟涂鸦方法
KR20210135683A (ko) 2020-05-06 2021-11-16 라인플러스 주식회사 인터넷 전화 기반 통화 중 리액션을 표시하는 방법, 시스템, 및 컴퓨터 프로그램
CN113709389A (zh) * 2020-05-21 2021-11-26 北京达佳互联信息技术有限公司 一种视频渲染方法、装置、电子设备及存储介质
CN111798379A (zh) * 2020-07-20 2020-10-20 广州炫视智能科技有限公司 实现图形高速预览的方法及***
CN113986407A (zh) * 2020-07-27 2022-01-28 华为技术有限公司 封面生成方法、装置及计算机存储介质
US11630639B2 (en) * 2020-12-08 2023-04-18 Samsung Electronics Co., Ltd. Control method of electronic device using a plurality of sensors and electronic device thereof
CN118301261A (zh) * 2021-02-04 2024-07-05 北京字跳网络技术有限公司 一种特效展示方法、装置、设备及介质
CN113160031B (zh) * 2021-03-26 2024-05-14 北京达佳互联信息技术有限公司 图像处理方法、装置、电子设备及存储介质
US11704626B2 (en) * 2021-07-09 2023-07-18 Prezi, Inc. Relocation of content item to motion picture sequences at multiple devices
CN113542872B (zh) * 2021-07-30 2023-03-24 联想(北京)有限公司 一种图像处理方法、装置及电子设备
CN114071228A (zh) * 2021-09-28 2022-02-18 浙江预策科技有限公司 实时信息叠加方法、装置、电子装置和存储介质
CN113961067B (zh) * 2021-09-28 2024-04-05 广东新王牌智能信息技术有限公司 基于深度学习的非接触式涂鸦绘制方法与识别交互***
CN115185435A (zh) * 2021-09-30 2022-10-14 红科网安(武汉)科技有限公司 要图标绘特殊要素显示控制方法
US11995757B2 (en) * 2021-10-29 2024-05-28 Snap Inc. Customized animation from video
US12020358B2 (en) 2021-10-29 2024-06-25 Snap Inc. Animated custom sticker creation
CN115379250B (zh) * 2022-07-22 2024-06-11 广州博冠信息科技有限公司 视频处理方法、装置、计算机设备及存储介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150253974A1 (en) * 2014-03-07 2015-09-10 Sony Corporation Control of large screen display using wireless portable computer interfacing with display controller
CN105635519A (zh) * 2015-06-15 2016-06-01 广州市动景计算机科技有限公司 视频处理方法、装置及***
CN105677107A (zh) * 2016-01-11 2016-06-15 福州瑞芯微电子股份有限公司 屏幕手写跟随性的优化方法及***
CN106385591A (zh) * 2016-10-17 2017-02-08 腾讯科技（上海）有限公司 视频处理方法及视频处理装置
US9601086B1 (en) * 2009-09-11 2017-03-21 Hewlett-Packard Development Company, L.P. Defining a projector display region
CN107291346A (zh) * 2016-04-12 2017-10-24 北京三星通信技术研究有限公司 终端设备的绘制内容处理方法、装置及终端设备
CN107888845A (zh) * 2017-11-14 2018-04-06 腾讯数码（天津）有限公司 一种视频图像处理方法、装置及终端

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20100064873A (ko) * 2008-12-05 2010-06-15 엘지전자 주식회사 단말기 및 그 제어 방법
JP5525923B2 (ja) * 2010-06-09 2014-06-18 任天堂株式会社 画像処理プログラム、画像処理装置、画像処理システム、および画像処理方法
JP2014123855A (ja) * 2012-12-20 2014-07-03 Panasonic Corp 合成画像生成装置、合成画像生成方法、および放送受信装置
US20140240343A1 (en) * 2013-02-22 2014-08-28 Corel Corporation Color adjustment control in a digital graphics system using a vision system
CN105164999B (zh) * 2013-04-17 2018-08-10 松下知识产权经营株式会社 图像处理方法及图像处理装置
GB2524473A (en) * 2014-02-28 2015-09-30 Microsoft Technology Licensing Llc Controlling a computing-based device using gestures
GB201419441D0 (en) * 2014-10-31 2014-12-17 Microsoft Corp Modifying video call data
CN106303690A (zh) * 2015-05-27 2017-01-04 腾讯科技（深圳）有限公司 一种视频处理方法及装置

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9601086B1 (en) * 2009-09-11 2017-03-21 Hewlett-Packard Development Company, L.P. Defining a projector display region
US20150253974A1 (en) * 2014-03-07 2015-09-10 Sony Corporation Control of large screen display using wireless portable computer interfacing with display controller
CN105635519A (zh) * 2015-06-15 2016-06-01 广州市动景计算机科技有限公司 视频处理方法、装置及***
CN105677107A (zh) * 2016-01-11 2016-06-15 福州瑞芯微电子股份有限公司 屏幕手写跟随性的优化方法及***
CN107291346A (zh) * 2016-04-12 2017-10-24 北京三星通信技术研究有限公司 终端设备的绘制内容处理方法、装置及终端设备
CN106385591A (zh) * 2016-10-17 2017-02-08 腾讯科技（上海）有限公司 视频处理方法及视频处理装置
CN107888845A (zh) * 2017-11-14 2018-04-06 腾讯数码（天津）有限公司 一种视频图像处理方法、装置及终端

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3713220A4

Also Published As

Publication number Publication date
EP3713220B1 (en) 2023-02-22
EP3713220A1 (en) 2020-09-23
EP3713220A4 (en) 2021-01-06
US11140339B2 (en) 2021-10-05
CN107888845A (zh) 2018-04-06
CN107888845B (zh) 2022-10-21
US20200252553A1 (en) 2020-08-06

Similar Documents

Publication Publication Date Title
WO2019095979A1 (zh) 视频图像处理方法、装置及终端
CN109565551B (zh) 对齐于参考帧合成图像
JP7378354B2 (ja) ライブ動画ストリームからの指の押下の検出
US9307148B1 (en) Video enhancement techniques
US10863077B2 (en) Image photographing method, apparatus, and terminal
WO2014045026A1 (en) Systems and methods for reducing noise in video streams
JP7101269B2 (ja) ポーズ補正
CA2918273A1 (en) Systems and methods for capturing high resolution images of rectangular objects using a mobile device
TWI545508B (zh) 用於執行臉部追蹤功能的方法及其電子裝置
JP2013106352A (ja) 画像中の投影領域の決定方法、及び装置
CN110084765B (zh) 一种图像处理方法、图像处理装置及终端设备
CN113469883B (zh) 动态分辨率的渲染方法、装置、电子设备及可读存储介质
US10460487B2 (en) Automatic image synthesis method
WO2018058476A1 (zh) 一种图像校正方法及装置
US9152874B2 (en) Motion blur aware visual pose tracking
WO2019042369A1 (zh) 一种自拍图片的处理方法、存储介质及移动终端
CN111583329B (zh) 增强现实眼镜显示方法、装置、电子设备和存储介质
KR20130103217A (ko) 블러화된 이미지의 제공 장치 및 블러화된 이미지의 제공 방법
JP2012073703A (ja) 画像ボケ量計算装置およびそのプログラム
US10567787B1 (en) Autofocusing images using motion detection
US20230020964A1 (en) Content based image processing
US20200053377A1 (en) Image statistics for motion detection
US11195247B1 (en) Camera motion aware local tone mapping
CN114429480A (zh) 图像处理方法与装置、芯片和电子设备
US11763509B2 (en) Frame calibration for robust video synthesis

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18879128

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2018879128

Country of ref document: EP

Effective date: 20200615