CN110688018A

CN110688018A - Virtual picture control method and device, terminal equipment and storage medium

Info

Publication number: CN110688018A
Application number: CN201911073061.XA
Authority: CN
Inventors: 胡永涛; 戴景文; 贺杰
Original assignee: Guangdong Virtual Reality Technology Co Ltd
Current assignee: Guangdong Virtual Reality Technology Co Ltd
Priority date: 2019-11-05
Filing date: 2019-11-05
Publication date: 2020-01-14
Anticipated expiration: 2039-11-05
Also published as: CN110688018B

Abstract

The application discloses a control method and device of a virtual picture, terminal equipment and a storage medium, and relates to the technical field of display. The control method of the virtual picture is applied to terminal equipment, and comprises the following steps: acquiring a target image containing a target object; acquiring the contour information of the target object according to the target image; determining first virtual content to be switched in a currently displayed virtual picture; and acquiring second virtual content according to the outline information, and switching and displaying the first virtual content into the second virtual content. The method can realize simple and quick switching of the virtual content by utilizing the contour information of the object.

Description

Virtual picture control method and device, terminal equipment and storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for controlling a virtual screen, a terminal device, and a storage medium.

Background

In recent years, with the progress of science and technology, technologies such as Augmented Reality (AR) and Virtual Reality (VR) have become hot spots of research at home and abroad. Taking augmented reality as an example, augmented reality is a technology for increasing the perception of a user to the real world through information provided by a computer system, and adds a computer-generated virtual object, a scene, or a content object such as system prompt information to a real scene to enhance or modify the perception of the real world environment or data representing the real world environment. In the AR/VR technology, the control of the virtual screen is complicated, and especially, in the aspect of switching the content of the virtual screen, how to switch the content of the virtual screen quickly and effectively is an urgent problem to be solved.

Disclosure of Invention

The embodiment of the application provides a method and a device for controlling a virtual picture, a terminal device and a storage medium, which can switch and display a first virtual content into a second virtual content quickly and effectively by using contour information of an object.

In a first aspect, an embodiment of the present application provides a method for controlling a virtual screen, which is applied to a terminal device, and the method includes: acquiring a target image containing a target object; acquiring contour information of a target object according to the target image; determining first virtual content to be switched in a currently displayed virtual picture; and acquiring second virtual content according to the outline information, and switching and displaying the first virtual content into the second virtual content.

In a second aspect, an embodiment of the present application provides a control apparatus for a virtual screen, which is applied to a terminal device, and the apparatus includes: the system comprises an image acquisition module, an information identification module, a content determination module and a content switching module, wherein the image acquisition module is used for acquiring a target image containing a target object; the information identification module is used for acquiring the contour information of the target object according to the target image; the content determining module is used for determining first virtual content to be switched in the currently displayed virtual picture; the content switching module is used for acquiring second virtual content according to the outline information and switching and displaying the first virtual content into the second virtual content.

In a third aspect, an embodiment of the present application provides a terminal device, including: one or more processors; a memory; one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, the one or more application programs being configured to execute the control method of the virtual screen provided by the first aspect described above.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where a program code is stored in the computer-readable storage medium, and the program code may be called by a processor to execute the control method for a virtual screen provided in the first aspect.

According to the scheme provided by the embodiment of the application, the terminal equipment can acquire the target image containing the target object so as to acquire the outline information of the target object according to the target image, then the terminal equipment can determine the first virtual content to be switched in the currently displayed virtual picture, and finally the terminal equipment can acquire the second virtual content according to the outline information and switch and display the first virtual content as the second virtual content.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a schematic diagram of an application environment suitable for use in embodiments of the present application;

fig. 2 is a flowchart illustrating a control method of a virtual screen according to an embodiment of the present application;

fig. 3a and fig. 3b are schematic diagrams illustrating a gesture in a control method of a virtual screen according to an embodiment of the present application;

FIG. 4 is a schematic diagram illustrating another gesture in a control method for a virtual screen according to an embodiment of the present application;

fig. 5 is a flowchart illustrating a control method of a virtual screen according to another embodiment of the present application;

fig. 6 is a flowchart illustrating step S240 in a method for controlling a virtual screen according to another embodiment of the present application;

fig. 7 is a flowchart illustrating a step S242 in a method for controlling a virtual screen according to another embodiment of the present application;

fig. 8 is another flowchart illustrating step S242 in a method for controlling a virtual screen according to another embodiment of the present application;

fig. 9 is a schematic diagram illustrating first virtual content in a control method of a virtual screen according to another embodiment of the present application;

fig. 10 is a schematic diagram illustrating virtual content switching in a control method for a virtual screen according to another embodiment of the present application;

fig. 11 is a flowchart illustrating a step S242 in a method for controlling a virtual screen according to another embodiment of the present application;

fig. 12 is a flowchart illustrating a step S242 of a method for controlling a virtual screen according to another embodiment of the present application;

fig. 13 is a flowchart illustrating a step S242 of a method for controlling a virtual screen according to another embodiment of the present application;

fig. 14 is a flowchart illustrating a control method of a virtual screen according to still another embodiment of the present application;

fig. 15 is a flowchart illustrating a step S330 in a method for controlling a virtual screen according to another embodiment of the present application;

fig. 16 is a block diagram illustrating a control apparatus for a virtual screen according to an embodiment of the present application;

fig. 17 is a block diagram of a terminal device for executing a control method of a virtual screen according to an embodiment of the present application;

fig. 18 is a storage unit for storing or carrying a program code implementing a control method of a virtual screen according to an embodiment of the present application.

Detailed Description

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

In the AR/VR technology, adjustment or switching of virtual content is an indispensable operation, for example, switching a different virtual interface according to a user's requirement, or switching some specific virtual objects in the virtual interface according to the user's requirement. The virtual content switching usually needs to display an independent switching window, and then the user selects the target virtual content for switching, obviously, the operation of the user is complex when the user switches the virtual content, and thus, the virtual content switching is very complicated.

Through research, the inventors propose a method, an apparatus, a terminal device and a storage medium for controlling a virtual screen in the embodiments of the present application, which can reduce the complexity of switching virtual contents.

An application scenario of the method for controlling a virtual screen according to the embodiment of the present application is described below.

Referring to fig. 1, a display system provided in an embodiment of the present application is shown, which includes a terminal device 100 and a handheld object 200.

In some embodiments, the terminal device 100 may be a head-mounted display device, a mobile phone, a tablet computer, a personal computer, or the like, where the head-mounted display device may be an integrated head-mounted display device, or a head-mounted display device connected to an external electronic device. The terminal device 100 may also be an intelligent terminal such as a mobile phone connected to an external/access head-mounted display device, that is, the terminal device 100 may be used as a processing and storage device of the head-mounted display device, inserted into or accessed to the external head-mounted display device, and display the virtual content 300 through the head-mounted display device.

The virtual content 300 may be a virtual interface, a window in the virtual interface, or some virtual object in the virtual interface, and the virtual object may be a virtual character or some virtual prop adapted to the virtual character, for example, a gun or ammunition in a gun game may be referred to as a virtual object, i.e., the virtual object may include a game accessory.

In some embodiments, the user may interact with the virtual content 300 displayed by the terminal device 100 through the handheld object 200. The specific shape, structure and size of the hand-held object 200 are not limited, and may be in various shapes, such as a mobile phone, a book, a cup, a bottle, a stick, etc. For example, referring again to FIG. 1, when the user holds the object 200 as a stick, the hand-held weapon of the virtual character in the virtual content 300 is also the virtual stick 303.

In other embodiments, the user may also interact with the virtual content 300 through a hand gesture, and at this time, the terminal device 100 may receive a gesture instruction of the user, and perform corresponding control on the virtual content 300 according to the gesture instruction.

Referring to fig. 2, an embodiment of the present application provides a method for controlling a virtual screen, which is applied to a terminal device, and the method may include:

step S110: a target image containing a target object is acquired.

Because the current virtual content switching usually needs to display an independent switching window, and then a user selects a target virtual content for switching, the virtual content switching is very complicated. Therefore, in the embodiment of the application, the terminal device can realize the fast switching of the virtual content through the outline information of the object, and the simplification of the switching operation is realized. Specifically, the terminal device may acquire a target image containing a target object to determine contour information of the target object.

The target object may be a handheld object or a hand of a user. When the target object is a hand of the user, the user can control the virtual content displayed in the terminal device by controlling different hand gestures.

In some embodiments, the terminal device may acquire a scene image including a hand area of the user in real time to intercept a target image including a target object from the scene image. As one mode, when the target object is a hand of a user, the terminal device may determine whether the acquired scene image includes a hand region of the user by analyzing information such as whether a characteristic shape of a finger and a characteristic shape of a palm appear in the scene image. When information such as the characteristic shape of the fingers and the characteristic shape of the palms appears in the scene image, the terminal device can determine that the scene image contains the hands of the user, so that the local image containing the information such as the characteristic shape of the fingers and the characteristic shape of the palms can be directly intercepted, and the target image containing the target object can be obtained. In other embodiments, the terminal device may also directly position the field of view of the image acquisition device to the hand area of the user, so that the target image including the target object may be acquired in real time.

The scene image or the target image may be acquired by the terminal device by using an image acquisition device configured for the terminal device, or may be acquired by using an external image acquisition device, that is, after the external image acquisition device acquires a plurality of scene images or target images, the scene images or the target images may be transmitted to the terminal device in a wired or wireless manner, so as to be used by the terminal device. In the embodiment of the present application, both the image capturing device configured for the terminal device itself and the external image capturing device may be used to capture an image of a real object and capture a scene image of a target scene, and the two may be an infrared camera or a visible camera, and the specific type is not limited in the embodiment of the present application.

It should be noted that, regardless of whether the target object is a handheld object or a hand of a user, the terminal device may track the target object, and the tracking manners may be the same or different, and obviously, the target image acquired by the terminal device for tracking the target object must include an image of the target object.

In some embodiments, the target image may include other objects besides the target object, and these objects may be related to the target object or unrelated to the target object. For example, when the target object is a stick, the stick may be provided with a marker, and the marker is related to the target object, in this embodiment, the marker may be a pattern having a topological structure or a light point type marker, and the terminal device may obtain spatial position information such as a relative position and a posture of the stick by tracking the marker, so as to realize positioning and tracking of the stick. An object unrelated to the target object may be referred to as an interfering object, and the interfering object may be a person or an object unrelated to the display of the virtual content in the real scene, and specific unrelated person and object are not described in detail in the embodiments of the present application.

Step S120: and acquiring the contour information of the target object according to the target image.

In one embodiment, after the terminal device acquires the target image, the terminal device may acquire the contour information of the target object in the target image by identifying and analyzing the target image. The contour information of the target object may include contour key points of the outline of the target object. The terminal equipment can extract contour key points of the contour of the target object in the target image so as to determine the contour information of the target object according to the position distribution of the contour key points. The contour key points of the target object shape may be arranged at equal intervals, or may be arranged correspondingly according to different feature positions, which is not limited herein. Further, when the terminal device acquires a plurality of target images including the target object, the terminal device may also analyze and identify the plurality of target images to accurately acquire the three-dimensional contour information of the target object.

In other embodiments, the contour information of the target object acquired by the terminal device may also be information such as a line and a size of the outline of the target object. The line may be a smooth curve or a simplified broken line.

Step S130: and determining first virtual content to be switched in the currently displayed virtual picture.

In the embodiment of the application, before the terminal device switches the virtual content according to the profile information of the target object, the first virtual content to be switched in the currently displayed virtual picture can be determined.

In some embodiments, the terminal device may determine the first virtual content to be switched from the display content corresponding to the default region in the virtual picture, that is, in the current application scenario, the terminal device may only switch the virtual content in the default region according to the profile information of the target object. For example, in a gunshot game scene, the handheld weapon area of the virtual character in the virtual game picture is a default area for switching, and a specific gun in the area is the first virtual content to be switched.

In other embodiments, the terminal device may also be an area to be switched determined according to the contour information of the target object, so that the first virtual content to be switched may be determined according to the display content corresponding to the area to be switched. As one mode, when the terminal device obtains the contour information of the target object, the contour information may be matched with preset contour information. If the contour information of the target object is matched with the preset contour information, the area to be switched in the currently displayed virtual picture can be determined, and therefore the first virtual content to be switched is determined according to the display content of the area to be switched. For example, in a shooting game scene, please refer to fig. 3a, when the target object is a hand of a user, profile information corresponding to the hand comparing the "gun" gesture matches with profile information of preset equipment, the terminal device may determine that the area to be switched is a handheld weapon area of a virtual character in the virtual game screen, and the first virtual content to be switched is a specific handheld weapon in the handheld weapon area. For another example, please refer to fig. 3b, when the hand compares the "phone" gesture, the corresponding profile information matches the preset application profile information, the terminal device may determine that the area to be switched is the display area of the virtual game screen, and the first virtual content to be switched is the whole virtual game screen, that is, the first virtual content to be switched is the game application.

In some embodiments, a plurality of types of preset profile information may be stored in the terminal device, and each type of preset profile information may include a plurality of specific preset profile information, and the preset profile information may be set according to actual needs. For example, the preset weapon profile information of the equipment type may include preset firearm profile information, preset fist profile information, preset grenade profile information, and the like. If the types of the preset profile information are different, the corresponding areas to be switched are also different, that is, the types of the preset profile information and the areas to be switched are stored in the mapping table through a certain corresponding relationship. In other words, when the contour information of the target object matches the preset contour information of the target type, the terminal device may obtain a mapping table between the type of the preset contour information and the region to be switched from the memory, and obtain the region to be switched corresponding to the target type based on the obtained mapping table, so as to determine the first virtual content according to the display content of the region to be switched. If the mapping table can be set as: the type of the preset profile information is A, and the corresponding area to be switched is a; the type of the preset profile information is B, and the corresponding area to be switched is B.

In some embodiments, the preset profile information may be preset and stored in the terminal device, or may be obtained by analyzing a behavior habit of the user, for example, when the user is used to perform a switching operation on virtual content by using a certain target object, the profile information of the target object may be recorded. It should be noted that the preset contour information may be fixed, and may also be continuously optimized according to the actual situation. For example, in the first time period, the terminal device is used by the user a, the first preset profile information a may be obtained according to the operation habit of the user a, and in the second time period, the terminal device is used by the user B, the second preset profile information B may be obtained according to the operation habit of the user B, where the preset profile information may be the second preset profile information B obtained newly, or may be the third preset profile information obtained by combining the first preset profile information a and the second preset profile information B. For example, when the first preset contour information is a substantially strip-shaped contour, and the second preset contour information is a rectangular parallelepiped contour with a certain width, the third preset contour information may be a strip-shaped contour with a certain width obtained by synthesis. It is to be understood that the specific preset profile information setting method is not limited in the embodiment of the present application.

Step S140: and acquiring second virtual content according to the outline information, and switching and displaying the first virtual content into the second virtual content.

Because the contour information of the target object can include contour key points, lines, dimensions and the like of the contour of the target object, when the terminal device acquires the contour information, second virtual content matched with the contour information can be determined according to the contour information, and then the first virtual content to be switched is switched and displayed as the second virtual content. Therefore, the virtual content can be switched quickly only through the contour information of the target object, and the switching operation is simplified.

In some embodiments, the second virtual content matched with the contour information may be that the contour information of the outline of the target object is matched with the contour information of the second virtual content. For example, when the target object is a long stick, and the profile information of the long stick matches the profile information of the virtual sword, it is determined that the second virtual content matching the profile information of the long stick is the virtual sword.

In other embodiments, the second virtual content matched with the outline information may be that the functional attribute represented by the outline information of the outline of the target object is matched with the functional attribute of the second virtual content. For example, referring to fig. 4, when the target object is a hand of a user, the functional attribute represented by the outline information corresponding to the gesture of "square frame" of the hand may be a picture frame, and the second virtual content matched with the outline information of the hand may be determined to be a virtual album application if the functional attribute is matched with the functional attribute of the virtual album application.

In some embodiments, the first virtual content and the second virtual content may have similar attributes and similar functions, but the key attributes and key functions of the two may be different. As one way, the first virtual content is located in the currently displayed virtual screen, and plays an important role in the virtual screen, that is, the state of the first virtual content in the terminal device is a "used state" or a "selected state". The second virtual content may be located in the currently displayed virtual screen, or may not be located in the currently displayed virtual screen, the state of the second virtual content in the terminal device is an "unused state" or a "to-be-selected state", and any one of the second virtual contents may replace the currently displayed first virtual content. For example, the first virtual content to be switched in the currently displayed virtual picture is gun a in the game equipment, and the second virtual content acquired according to the contour information of the target object is a fist. The gun a is being used by the game character a, that is, the gun a is in the "used" state. The fist can also be located in the virtual frame, but it is not used by the game character a, or it is not displayed in the virtual frame, i.e. the fist is in an "unused" state, and the fist can only be displayed in the virtual frame after receiving the corresponding call instruction. Obviously, the firearm A and the fist have similar properties and functions, for example, the fist and the firearm A both belong to attack properties and can be used for attacking enemies, but the key properties and the key functions of the two are different, for example, the shapes of the firearm A and the fist are different, or the force value of the firearm A is higher than that of the fist.

After obtaining the second virtual content, the terminal device may switch and display the first virtual content as the second virtual content. Specifically, the terminal device may obtain a target switching instruction, which may be an instruction automatically generated when the second virtual content matching the profile information is obtained. When the terminal device obtains the target switching instruction, the first virtual content in the virtual picture can be switched to the second virtual content for displaying. As a way, the terminal device respectively acquires a first identifier corresponding to the first virtual content and a second identifier corresponding to the second virtual content according to the target switching instruction; and modifying the first identifier corresponding to the first virtual content into a second identifier corresponding to the second virtual content, so that the first virtual content can be switched and displayed as the second virtual content.

In the method for controlling the virtual screen provided by the embodiment of the application, the terminal device obtains the contour information of the target object by analyzing the target image containing the target object, and then the terminal device can quickly obtain the first virtual content and the second virtual content by confirming the contour information of the target object. The first virtual content is in the currently displayed virtual frame, and the second virtual content is used for replacing the first virtual content. Through confirmation of the outline information, the first virtual content can be accurately and effectively switched into the second virtual content, the switching operation is simple, and the operation requirement of a user on the virtual content is easily met.

Referring to fig. 5, another embodiment of the present application provides a method for controlling a virtual screen, which is applied to a terminal device, and the method may include:

step S210: a target image containing a target object is acquired.

Step S220: and acquiring the contour information of the target object according to the target image.

Step S230: and determining first virtual content to be switched in the currently displayed virtual picture.

Step S240: and acquiring second virtual content according to the outline information, and switching and displaying the first virtual content into the second virtual content.

In some embodiments, the terminal device may perform fast switching of the virtual content according to the specific shape of the target object. Specifically, as shown in fig. 6, the obtaining of the second virtual content according to the outline information may include: step S241 to step S242.

Step S241: and acquiring the shape of the target object according to the contour information.

In some embodiments, when the contour information acquired by the terminal device is contour key points of the contour of the target object, the contour key points may be sequentially connected by a line to obtain a contour line of the contour of the target object, so that the terminal device may determine the shape of the target object according to the contour line. The line may be a smooth curve or a broken line, which is not limited herein. For example, when the target object is a long stick, the shape formed by the corresponding contour line may be a cylinder shape; when the target object is a finger, the shape formed by the contour lines corresponding to the gesture of the gun drawn by the finger can be a combined shape formed by two cuboids (similar to a pistol).

As one way, it is contemplated to create a shape database having a plurality of shapes stored therein. The terminal device can match the contour line of the target object with the contour line of the shape in the shape database to obtain the target shape successfully matched with the contour line of the target object, so that the terminal device can determine that the shape of the target object is the target shape. The target shape successfully matched with the contour line of the target object may be a shape with a matching degree greater than a matching threshold, for example, the matching degree is greater than 90%, or may be a shape corresponding to the highest matching degree, which is not limited herein.

Step S242: and acquiring second virtual content matched with the shape.

After the shape of the target object is determined, the terminal device can acquire second virtual content matched with the shape. The second virtual content matched with the shape may be matched with the shape of the target object in the shape of the second virtual content, for example, when the target object is a stick, the second virtual content may be a virtual stick; it may also be that the functional attribute of the second virtual content matches the functional attribute characterized by the shape of the target object, for example, when the target object is a finger, and when the finger is compared to a "phone" gesture, the second virtual content may be a phone call application, and both the functional attributes are phone call functions.

In some embodiments, a plurality of preset shapes may be stored in the terminal device, the preset trajectory may be set according to actual needs, and if the preset shapes are different, the corresponding first virtual contents may be different, that is, the preset shapes and the second virtual contents are stored in the mapping table through a certain corresponding relationship, when the shape of the target object matches the preset shape, the terminal device may obtain the mapping table between the preset shapes and the second virtual contents from the memory thereof, and obtain the corresponding second virtual contents according to the shape of the target object based on the obtained mapping table.

In some embodiments, when there are a plurality of virtual contents matching the shape, the terminal device may determine the second virtual content according to the usage habit of the user. Specifically, as shown in fig. 7, the obtaining of the second virtual content matching the shape may include: step S2421 to step S2423.

Step S2421: and acquiring the candidate object matched with the shape.

When determining the shape of the target object, the terminal device may obtain all candidate objects matching the shape according to the shape. Specifically, the terminal device may search from the memory whether a candidate object matching the shape of the target object is stored. If the candidate object matched with the shape of the target object is not found out, it is indicated that no virtual content matched with the shape of the target object exists, and further, it can be indicated that the terminal device cannot perform virtual content switching operation on the first virtual content. If the candidate object matched with the shape of the target object is stored in the terminal equipment, the terminal equipment can acquire the searched candidate object.

In some embodiments, when the candidate object found by the terminal device is one, the terminal device may directly switch and display the candidate object as the second virtual content, that is, switch and display the currently displayed first virtual content as the candidate object. And the candidate objects are displayed in the virtual screen for selection by the user,

in an embodiment, when there are multiple candidate objects found, the terminal device may display the multiple candidate objects in a virtual screen for selection by a user, or may automatically select an optimal candidate object. The best candidate object can be specifically determined according to a specific application scene. For example, in a game scene, the best candidate object may be the candidate object corresponding to the highest attack force, or may be the candidate object corresponding to the best composite attribute.

Step S2422: when the candidate is plural, the frequency of use of each candidate is acquired.

In some embodiments, when the number of candidate objects acquired by the terminal device is multiple, the terminal device may determine the second virtual content according to the frequency of use of each candidate object. As one mode, the terminal device may record the number of times that each candidate object is in the "use" state in real time, perform real-time statistics and ranking, and obtain a usage frequency ranking list of all candidate objects. Therefore, when the terminal equipment acquires a plurality of candidate objects, the use frequency of each candidate object can be determined according to the use frequency ranking list.

As another mode, the terminal device may also record the duration of each candidate object in the "use" state, perform real-time statistics and ranking, and obtain a use frequency ranking list of all candidate objects, so that when the terminal device obtains multiple candidate objects, the use frequency of each candidate object may be determined according to the use frequency ranking list.

For example, there are three candidate objects matching the shape of the target object in the terminal device, namely, the gun 301, the gun 302 and the gun 303, and the terminal device can acquire the gun 301 with the use frequency of 55%, the gun 302 with the use frequency of 12% and the gun 303 with the use frequency of 33%, and the guns 301, 303 and 302 are obtained by sorting the use frequencies of the three guns from high to low.

In some embodiments, the terminal device may display the obtained ranking result of the frequency of use of the candidate object, and the user may select the second virtual content in sequence from high to low according to the display result to perform switching display.

Step S2423: and acquiring the candidate object corresponding to the highest using frequency as the second virtual content.

In some embodiments, when acquiring the use frequency of each candidate object, the terminal device may directly acquire the candidate object corresponding to the highest use frequency as the second virtual content. Specifically, the terminal device may determine, as the second virtual content, a candidate object corresponding to the highest usage frequency from among the plurality of candidate objects that are matched in shape according to the usage frequency ranking list that implements the update. For example, in the above example, the terminal device may use the highest frequency of use for gun 301 as the second virtual content.

In other embodiments, when the target object is a hand, the terminal device may determine the second virtual content differently according to a different hand. Specifically, before the second virtual content matched with the shape is acquired, the method for controlling the virtual content may further include:

and determining a mode corresponding to the hand according to the target image, wherein the mode comprises any one of a left-hand mode, a right-hand mode and a two-hand mode.

Under this embodiment, as shown in fig. 8, the obtaining of the second virtual content matching the shape may include: step S2424 to step S2425.

Step S2424: and determining the corresponding gesture of the hand according to the shape.

Step S2425: and acquiring second virtual content matched with the mode and the gesture.

In some embodiments, the terminal device may determine whether the target image includes one hand or both hands of the user according to the number of fingers or palms in the target image. When the terminal device detects that the target image contains the hands of the user, it is determined that the target image is in a two-hand mode, and the terminal device may determine the second virtual content according to the gesture in the two-hand mode. When the terminal device detects that the target image contains a single hand of the user, it is determined that the target image is in the single-hand mode, and the terminal device may determine the second virtual content according to a gesture in the single-hand mode. The second virtual content determined in the two-hand mode and the second virtual content determined in the gesture determined in the one-hand mode may be the same or different, and are not limited herein.

In some embodiments, when the terminal device detects that the target image includes a single hand of the user, the terminal device may determine whether the hand is a left hand of the user or a right hand of the user according to information such as a specific gesture of the hand in the target image, an orientation of fingers, and a position arrangement between the fingers. When the terminal device detects that the left hand of the user is included in the target image, the terminal device may determine the second virtual content according to the gesture in the left-hand mode. When the terminal device detects that the right hand of the user is included in the target image, the terminal device may determine the second virtual content according to the gesture in the right-hand mode. The second virtual content determined in the left-hand mode and the second virtual content determined in the gesture determined in the right-hand mode may be the same or different, and are not limited herein.

For example, referring to fig. 9 and 10, in the game scenario, when the first virtual content is a fist of a weapon used by the game character a (fig. 9), and when the user compares the gesture of "gun" with the right hand, the terminal device may follow the gesture of the right hand mode and "gun", determine that the second virtual content is a default gun (203 in fig. 10), and switch the weapon used by the current game character a from the fist to the default gun for display. For another example, when the user compares the gesture of "gun" with the left hand, the terminal device may determine that the second virtual content is a bullet when the terminal device follows the left hand mode and the gesture of "gun", and perform bullet-clamping display on a default gun of a weapon used by the current game character a. For another example, when the user compares the gesture of "gun" with both hands, the terminal device may determine that the second virtual content is a machine gun (204 in fig. 10) when the terminal device is in the two-hand mode and the gesture of "gun", and switch the weapon used by the current game character a from the default gun to the machine gun for display.

In still other embodiments, the terminal device may also determine the second virtual content according to the specific size of the target object. Specifically, as shown in fig. 11, the obtaining of the second virtual content matching the shape may include: step S2426 to step S2428.

Step S2426: and acquiring the shape and contour parameters of the target object according to the contour information.

In some embodiments, when the contour information acquired by the terminal device is contour key points of the contour of the target object, the contour key points may be sequentially connected by lines to obtain a contour line of the contour of the target object, so that the terminal device may determine contour parameters of the target object according to the contour line. The contour parameter may be a length, a width, a height, or the like of the target object, or may be a side length of each side of the target object, a bending angle of the curved surface, or the like, which is not limited herein.

Step S2427: one or more candidate objects matching the shape are obtained.

Step S2428: and acquiring a candidate object matched with the contour parameters from the one or more candidate objects as second virtual content.

In some embodiments, when the terminal device acquires one or more candidate objects matching the shape of the target object, the terminal device may determine the second virtual content from the one or more candidate objects according to the contour parameters. As one mode, when the terminal device acquires a candidate object that matches the shape of the target object, the terminal device may directly switch and display the candidate object as the second virtual content. As another mode, the terminal device may also adjust the profile parameter of the candidate object according to the specific profile parameter to fit the profile parameter of the target object, and perform switching display on the adjusted candidate object as the second virtual content. For example, when the target object is a plastic stick and the matched candidate object is only a virtual sword, the terminal device may adjust the length of the virtual sword to be the length of the plastic stick and switch and display the adjusted virtual sword.

As still another way, when the terminal device acquires a plurality of candidate objects that match the shape of the target object, the terminal device may acquire, as the second virtual content, a candidate object that matches the contour parameter of the target object from among the plurality of candidate objects. The candidate object matched with the contour parameter of the target object may be the candidate object with the highest matching degree, or may be the candidate object with the matching degree greater than the matching threshold, which is not limited herein.

In still other embodiments, when the candidate object matching the shape belongs to different object categories, the terminal device may determine the object category to which the second virtual content belongs according to the specific size of the target object, and then determine the specific second virtual content. Specifically, as shown in fig. 12, the obtaining of the second virtual content matching the shape may include: step S2429 to step S2433.

Step S2429: and acquiring the shape and contour parameters of the target object according to the contour information.

Step S2430: one or more object categories matched with the shape are obtained, and parameter conditions corresponding to each object category are determined.

In some embodiments, when the shape of the target object is relatively generic, the candidate objects that match the shape may have different object classes. Therefore, when the terminal device determines the shape of the target object, one or more object categories matched with the shape may be determined first, and a parameter condition corresponding to each object category may be determined, so as to determine an appropriate object category according to the parameter condition subsequently. For example, when the target object is a stick, the long and thin shape of the stick may be a dagger, a stick, a sword, or the like, and the object type matching the shape may be a dagger, a stick, or a sword, where the dagger has a length parameter of 10cm to 30cm, the stick has a length parameter of 31cm to 45cm, and the sword has a length parameter of 46cm to 70 cm.

Step S2431: and matching the contour parameters with the parameter conditions corresponding to each object type to obtain the object type corresponding to the contour parameters.

In some embodiments, the terminal device may match the contour parameter of the target object with the parameter condition corresponding to each object category described above, so as to obtain an object category corresponding to the contour parameter. Specifically, it may be determined in which object class the contour parameter of the target object is within the parameter condition range, and when the contour parameter is within the parameter condition range of the target object class, it may be determined that the target object belongs to the target object class. For example, in the above example, when the stick has a length of 47cm, the corresponding object type is sword.

Step S2432: and acquiring one or more candidate objects contained in the corresponding object category.

Step S2433: and acquiring a candidate object matched with the contour parameters from the one or more candidate objects as second virtual content.

In some embodiments, when the terminal device determines the object class of the candidate object, one or more candidate objects under the object class may be acquired. When only one candidate object exists in the object category, the terminal device may directly switch and display the candidate object as the second virtual content, or may specifically adjust the contour parameter of the candidate object according to the contour parameter of the target object. When there are a plurality of candidate objects under the object category, the terminal device may acquire, as the second virtual content, a candidate object that matches the contour parameter of the target object from among the plurality of candidate objects. The candidate object matched with the contour parameter of the target object may be the candidate object with the highest matching degree, or may be the candidate object with the matching degree greater than the matching threshold, which is not limited herein.

In some further embodiments, the terminal device may further render the second virtual content according to the material information of the target object, so as to fit the visual effect of the target object. Specifically, as shown in fig. 13, the obtaining of the second virtual content matching the shape may include: step S2434 to step S2437.

Step S2434: and acquiring material information of the target object according to the target image.

In some embodiments, the material information may be material information corresponding to a maximum area in the target object. As one mode, the terminal device may identify and analyze colors, lines, patterns, and combinations of the surface of the target object to determine a feature region with the largest layout area, and then the terminal device may extract an image of a partial region from the feature region as material information of the target object.

Step S2435: and acquiring second virtual content according to the outline information.

Step S2436: and acquiring preset texture information matched with the material information.

In some embodiments, the terminal device may store a plurality of pieces of preset texture information, and after the material information of the target object is determined, the terminal device may obtain the preset texture information matched with the material information from the plurality of pieces of preset texture information by matching the material information with the preset texture information. The preset texture information matched with the material information may be the preset texture information with the highest matching degree, or the preset texture information with the matching degree greater than the matching threshold.

Step S2437: and rendering the second virtual content according to the preset texture information, and switching and displaying the first virtual content into the rendered second virtual content.

After the terminal device obtains the preset texture information, the second virtual content can be rendered according to the preset texture information, and the first virtual content is switched and displayed to be the rendered second virtual content, so that the switched virtual content can be attached to the target object, and meanwhile, the switched virtual content has a personalized effect.

In the method for controlling the virtual screen provided by the embodiment of the application, the terminal device obtains the contour information of the target object by analyzing the target image containing the target object, and then confirms the shape and contour parameters of the target object through the contour information, so that the terminal device can quickly obtain the first virtual content and the second virtual content. The first virtual content is in the currently displayed virtual frame, and the second virtual content is used for replacing the first virtual content. Through the confirmation of the shape and the contour parameters of the target object, the first virtual content can be accurately and effectively switched into the second virtual content, the switching operation is simple, and the operation requirement of a user on the virtual content is easily met.

Referring to fig. 14, another embodiment of the present application provides a method for controlling a virtual screen, which is applied to a terminal device, and the method may include:

step S310: a target image containing a target object is acquired.

Step S320: and acquiring the contour information of the target object according to the target image.

Step S330: and determining first virtual content to be switched in the currently displayed virtual picture.

Wherein, as shown in fig. 15, step S330 may include steps S331 to S332.

Step S331: and acquiring the user identity information, and reading the personal data of the user according to the user identity information.

In one embodiment, the identity information of the user may include a user identity identifier, a login password, and the like, and the terminal device may obtain the user identity information from a smart card or a chip, where the smart card may include a Subscriber Identity Module (SIM) card; user identity information and the like can also be acquired from the wireless network. The user identity information and the user personal data are stored in the terminal device or the cloud end in a one-to-one corresponding mode, after the user identity information is obtained, the user personal data corresponding to the user identity information can be found according to the identity information, and the user personal data can comprise basic data, operation data, state data and the like of the user. Wherein, the basic data refers to personal basic information of the user, such as sex, age, hobby, etc. of the user; the operation data refers to some operations which are used by the user when the user uses the terminal equipment, for example, when the user plays games by using the terminal equipment, the user is used to realize the switching of the WeChat application program through the hand speed of the telephone; the state data refers to the state of the terminal device and the time corresponding to the state, and common states include online, offline, stealth, do not disturb, and the like.

Step S332: and determining first virtual content to be switched in the currently displayed virtual picture according to the personal data of the user.

In some embodiments, after acquiring the personal data of the user corresponding to the terminal device through the identity information of the user, the terminal device may analyze the personal data of the user to obtain a personal data analysis result, and then may determine the first virtual content to be switched corresponding to the profile information in the currently displayed virtual image according to the personal data analysis result acquired by the terminal device. For the same profile information, the first virtual content switched by different users can be the same or different, and each user has own specific behavior habit, so the terminal device can make different virtual content switching schemes according to the behavior habits of different users. In an implementation mode, images including user behavior at different moments can be acquired by using an internal image acquisition device or an external image acquisition device of the terminal device, feature extraction, recognition and analysis can be performed on the images at the different moments, behavior habits of the user can be further obtained, and first virtual content corresponding to a motion track in a currently displayed virtual picture can be determined according to the behavior habits.

Step S340: and acquiring second virtual content according to the outline information, and switching and displaying the first virtual content into the second virtual content.

In the control method of the virtual picture provided by the embodiment of the application, the terminal device obtains the contour information of the target object by analyzing the target image containing the target object, and then the terminal device can quickly obtain the first virtual content and the second virtual content by confirming the contour information of the target object, wherein the first virtual content is in the currently displayed virtual picture, and the first virtual content can be obtained through the personal data of the user, wherein the personal data of the user comprises the behavior habit data of the user, and the behavior habit data can be obtained by means of an artificial intelligence algorithm.

Referring to fig. 16, a block diagram of a control apparatus 400 for a virtual frame according to an embodiment of the present application is shown, where the apparatus may include: the system comprises an image acquisition module 410, an information identification module 420, a content determination module 430 and a content switching module 440, wherein the image acquisition module 410 is used for acquiring a target image containing a target object; the information identification module 420 is used for acquiring the contour information of the target object according to the target image; the content determining module 430 is configured to determine a first virtual content to be switched in a currently displayed virtual picture; the content switching module 440 is configured to obtain the second virtual content according to the contour information, and switch and display the first virtual content as the second virtual content.

In some embodiments, the content switching module may include: a shape acquisition unit and a content acquisition unit. The shape acquisition unit is used for acquiring the shape of the target object according to the contour information; the content acquisition unit is used for acquiring second virtual content matched with the shape.

In some embodiments, the content obtaining unit may be specifically configured to: obtaining a candidate object matched with the shape; when a plurality of candidate objects are available, acquiring the use frequency of each candidate object; and acquiring the candidate object corresponding to the highest using frequency as the second virtual content.

In some embodiments, the target object is a hand, and the control device 400 of the virtual screen may further include: a mode determination module. The mode determining module is used for determining a mode corresponding to the hand according to the target image, wherein the mode comprises any one of a left-hand mode, a right-hand mode and a double-hand mode.

In this embodiment, the content acquiring unit may be specifically configured to: determining gestures corresponding to the hand according to the shape; and acquiring second virtual content matched with the mode and the gesture.

In some embodiments, the content switching module 440 may be specifically configured to: acquiring the shape and contour parameters of the target object according to the contour information; acquiring one or more candidate objects matched with the shape; and acquiring a candidate object matched with the contour parameters from the one or more candidate objects as second virtual content.

In some embodiments, the content switching module 440 may be further specifically configured to: acquiring the shape and contour parameters of the target object according to the contour information; acquiring one or more object categories matched with the shape, and determining a parameter condition corresponding to each object category; matching the contour parameters with the parameter conditions corresponding to each object type to obtain the object type corresponding to the contour parameters; acquiring one or more candidate objects contained in the corresponding object category; and acquiring a candidate object matched with the contour parameters from the one or more candidate objects as second virtual content.

In some embodiments, the content switching module 440 may be further specifically configured to: acquiring material information of a target object according to the target image; acquiring second virtual content according to the outline information; acquiring preset texture information matched with the material information; and rendering the second virtual content according to the preset texture information, and switching and displaying the first virtual content into the rendered second virtual content.

In the control apparatus for a virtual screen according to the embodiment of the present application, the contour information of the target object is acquired by analyzing the target image including the target object, and then the terminal device can acquire the first virtual content and the second virtual content quickly by confirming the contour information of the target object. The first virtual content is in the currently displayed virtual frame, and the second virtual content is used for replacing the first virtual content. Through confirmation of the outline information, the first virtual content can be accurately and effectively switched into the second virtual content, the switching operation is simple, and the operation requirement of a user on the virtual content is easily met.

Referring to fig. 17, a block diagram of a terminal device according to an embodiment of the present application is shown. The terminal device 100 may be a terminal device such as a head-mounted display device capable of running an application. The terminal device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, wherein the memory 120 has one or more applications stored therein, the one or more applications configured to be executed by the one or more processors 110, the one or more programs configured to perform the methods as described in the foregoing method embodiments.

Processor 110 may include one or more processing cores. The processor 110 connects various parts within the entire terminal device 100 using various interfaces and lines, and performs various functions of the terminal device 100 and processes data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 120 and calling data stored in the memory 120. Alternatively, the processor 110 may be implemented in hardware using at least one of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 110 may integrate one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a modem, and the like. Wherein, the CPU mainly processes an operating system, a user interface, an application program and the like; the GPU is used for rendering and drawing display content; the modem is used to handle wireless communications. It is understood that the modem may not be integrated into the processor 110, but may be implemented by a communication chip.

The Memory 120 may include a Random Access Memory (RAM) or a Read-Only Memory (Read-Only Memory). The memory 120 may be used to store instructions, programs, code sets, or instruction sets. The memory 120 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing various method embodiments described below, and the like. The storage data area may also store data created by the terminal device 100 in use, and the like.

In some embodiments, the terminal device 100 may further include an image sensor 130 for capturing images of real objects and capturing scene images of the target scene. The image sensor 130 may be an infrared camera or a visible light camera, and the specific type is not limited in the embodiment of the present application.

In one embodiment, the terminal device is a head-mounted display device, and may further include one or more of the following components in addition to the processor, the memory, and the image capturing device described above: image acquisition device, display module assembly, optical module assembly, communication module and power.

The image acquisition device is used for acquiring images of real objects and acquiring scene images of a target scene. The image capturing device may be an infrared camera or a visible light camera, and the specific type is not limited in the embodiments of the present application.

The display module may include a display control unit. The display control unit is used for receiving the display image of the virtual content rendered by the processor, and then displaying and projecting the display image onto the optical module, so that a user can view the virtual content through the optical module. The display module can be a display screen or a projection device and the like and can be used for displaying images.

The optical module can adopt an off-axis optical system or a waveguide optical system, and a display image displayed by the display module can be projected to eyes of a user after passing through the optical module. The user sees the display image that the display module assembly was thrown through optical module assembly simultaneously. In some embodiments, the user can also observe the real environment through the optical module, and experience the augmented reality effect after the virtual content and the real environment are superimposed.

The communication module can be a module such as bluetooth, WiFi (Wireless-Fidelity), ZigBEE (violet technology) and the like, and the head-mounted display device can establish communication connection with the terminal device through the communication module. The head-mounted display device in communication connection with the terminal equipment can perform information and instruction interaction with the terminal equipment. For example, the head-mounted display device may receive image data transmitted from the terminal device via the communication module, and generate and display virtual content of a virtual world from the received image data.

The power supply can supply power for the whole head-mounted display device, and the normal operation of each part of the head-mounted display device is ensured.

Referring to fig. 18, a block diagram of a computer-readable storage medium according to an embodiment of the present application is shown. The computer-readable storage medium 900 has stored therein program code that can be invoked by a processor to perform the methods described in the method embodiments above.

The computer-readable storage medium 900 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read only memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer-readable storage medium 900 includes a non-volatile computer-readable storage medium. The computer readable storage medium 900 has storage space for program code 910 to perform any of the method steps of the method described above. The program code can be read from or written to one or more computer program products. The program code 910 may be compressed, for example, in a suitable form.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not necessarily depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.

Claims

1. A control method of a virtual picture is applied to a terminal device, and the method comprises the following steps:

acquiring a target image containing a target object;

acquiring the contour information of the target object according to the target image;

determining first virtual content to be switched in a currently displayed virtual picture;

and acquiring second virtual content according to the outline information, and switching and displaying the first virtual content into the second virtual content.

2. The method of claim 1, wherein the obtaining the second virtual content according to the contour information comprises:

acquiring the shape of the target object according to the contour information;

and acquiring second virtual content matched with the shape.

3. The method of claim 2, wherein the obtaining the second virtual content matching the shape comprises:

acquiring a candidate object matched with the shape;

when the candidate objects are multiple, acquiring the use frequency of each candidate object;

and acquiring the candidate object corresponding to the highest using frequency as the second virtual content.

4. The method of claim 2, wherein the target object is a hand, and prior to the obtaining the second virtual content matching the shape, the method further comprises:

determining a mode corresponding to the hand according to the target image, wherein the mode comprises any one of a left-hand mode, a right-hand mode and a double-hand mode;

the obtaining of the second virtual content matched with the shape includes:

determining a gesture corresponding to the hand according to the shape;

and acquiring second virtual content matched with the mode and the gesture.

5. The method of claim 1, wherein the obtaining the second virtual content according to the contour information comprises:

acquiring the shape and contour parameters of the target object according to the contour information;

acquiring one or more candidate objects matched with the shape;

and acquiring a candidate object matched with the contour parameter from the one or more candidate objects as second virtual content.

6. The method of claim 1, wherein the obtaining the second virtual content according to the contour information comprises:

acquiring one or more object categories matched with the shape, and determining a parameter condition corresponding to each object category;

matching the contour parameters with the parameter conditions corresponding to each object type to obtain the object type corresponding to the contour parameters;

acquiring one or more candidate objects contained in the corresponding object category;

7. The method according to any one of claims 1 to 6, wherein the obtaining of the second virtual content according to the profile information and the switching display of the first virtual content as the second virtual content comprises:

acquiring material information of the target object according to the target image;

acquiring second virtual content according to the outline information;

acquiring preset texture information matched with the material information;

rendering the second virtual content according to the preset texture information, and switching and displaying the first virtual content into the rendered second virtual content.

8. A control device of a virtual picture is applied to a terminal device, and the device comprises:

the image acquisition module is used for acquiring a target image containing a target object;

the information identification module is used for acquiring the contour information of the target object according to the target image;

the content determining module is used for determining first virtual content to be switched in the currently displayed virtual picture;

and the content switching module is used for acquiring second virtual content according to the outline information and switching and displaying the first virtual content into the second virtual content.

9. A terminal device, comprising:

one or more processors;

a memory;

one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications configured to perform the method of any of claims 1-7.

10. A computer-readable storage medium, having stored thereon program code that can be invoked by a processor to perform the method according to any one of claims 1 to 7.