CN113283268A - Image processing method and device - Google Patents
Image processing method and device Download PDFInfo
- Publication number
- CN113283268A CN113283268A CN202010102091.5A CN202010102091A CN113283268A CN 113283268 A CN113283268 A CN 113283268A CN 202010102091 A CN202010102091 A CN 202010102091A CN 113283268 A CN113283268 A CN 113283268A
- Authority
- CN
- China
- Prior art keywords
- vehicle
- target component
- dimensional image
- dimensional
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/08—Detecting or categorising vehicles
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Multimedia (AREA)
- Processing Or Creating Images (AREA)
Abstract
The embodiment of the application provides an image processing method and device, relates to the technical field of automatic driving, and specifically comprises the following steps: the association relationship between the second two-dimensional image and the semantic information of the target component in the first state can be constructed according to the first two-dimensional image marked with the vehicle target component in the driving environment and the three-dimensional model of the target component and the usage rule of the target component in the vehicle. In other words, in the embodiment of the application, a large number of images do not need to be manually acquired and labeled, and the incidence relation between the second two-dimensional image and the semantic information of the target component in the first state can be obtained based on image processing by combining the three-dimensional model of the target component and the two-dimensional image of the vehicle in the driving environment, so that the efficiency and the flexibility are greatly improved.
Description
Technical Field
The present application relates to the field of automatic driving technologies for image processing, and in particular, to an image processing method and apparatus.
Background
In the field of automatic driving, environmental perception is an important part. For example, in route planning, route planning is required to be adapted by sensing the state of surrounding vehicles. The state of the surrounding vehicle may include, for example: turning, long-time parking, short-time parking, etc.
In the prior art, in order to identify the state of a surrounding vehicle, a large number of vehicle state images are generally collected, and sample labeling and the like are performed to obtain an image template for identifying the vehicle state.
However, the cost, efficiency, and inflexibility of acquiring images and performing manual annotation are high.
Disclosure of Invention
The embodiment of the application provides an image processing method and device, and aims to solve the technical problems of high cost and low efficiency of image acquisition and manual labeling in the prior art.
A first aspect of an embodiment of the present application provides an image processing method, including:
acquiring a first two-dimensional image of a vehicle in a driving environment; a target component of the vehicle is marked in the first two-dimensional image;
acquiring a three-dimensional model of the target component and semantic information of the vehicle of the target component in a first state;
generating a second two-dimensional image corresponding to the vehicle when the target component is in the first state by using the first two-dimensional image, the three-dimensional model of the target component and the usage rule of the target component in the vehicle;
and establishing an incidence relation between the second two-dimensional image and the semantic information.
In the embodiment of the application, a large number of images do not need to be acquired and labeled manually, the incidence relation between the second two-dimensional image and the semantic information of the target component in the first state can be obtained based on image processing by combining the three-dimensional model of the target component and the two-dimensional image of the vehicle in the driving environment, and the efficiency and the flexibility are greatly improved.
Optionally, the target component is a moving component;
the generating, by using the first two-dimensional image, the three-dimensional model of the target component, and the usage rule of the target component in the vehicle, a second two-dimensional image corresponding to the vehicle when the target component is in the first state includes:
rendering by using the three-dimensional model of the target component to obtain a depth map of the target component;
recovering three-dimensional point cloud data of the target component by using the depth map of the target component and the first two-dimensional image;
translating or rotating the target component by utilizing the three-dimensional point cloud data and the motion rule of the motion component in the vehicle use to generate a target three-dimensional image corresponding to the vehicle when the target component is in the first state;
and mapping the target three-dimensional image to obtain the second two-dimensional image.
Optionally, the moving part comprises: door, bonnet and trunk.
Optionally, if an invisible target area exists in the second two-dimensional image, the method further includes:
acquiring an environment map of the first two-dimensional image;
rendering the target area by using the environment map, and fusing the rendered target area and the second two-dimensional image.
Optionally, the target component is a lighting component;
the generating, by using the first two-dimensional image, the three-dimensional model of the target component, and the usage rule of the target component in the vehicle, a second two-dimensional image corresponding to the vehicle when the target component is in the first state includes:
performing two-dimensional projection by using the three-dimensional model of the target component to obtain a projection area of the target component in the first two-dimensional image;
and editing the color of the projection area by using a lighting rule of the lighting component in the use of the vehicle to generate a second two-dimensional image corresponding to the vehicle.
Optionally, the lighting part includes: a lamp for a vehicle.
Optionally, the method further includes:
filling a hole area of the second two-dimensional image; and performing smooth filtering on the filled two-dimensional image. Therefore, the effect of the second two-dimensional image is better, and the semantic information is more accurately identified by using the second two-dimensional image.
Optionally, the semantic information of the vehicle of the target component in the first state includes one or more of the following:
the semantic information of the vehicle with the door in the open state is as follows: the personnel in the vehicle need to get off the vehicle;
the semantic information of the vehicle with the trunk in the open state is as follows: the personnel in the vehicle need to pick up or load goods;
the semantic information of the vehicle with the hood in the open state is: a vehicle failure;
the semantic information of the vehicle under the bright yellow state of the left headlight of the vehicle is as follows: turning left the vehicle;
the semantic information of the vehicle under the bright yellow state of the vehicle right headlight is as follows: the vehicle turns to the right.
A second aspect of the embodiments of the present application provides an apparatus for image processing, including:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring a first two-dimensional image of a vehicle in a driving environment; a target component of the vehicle is marked in the first two-dimensional image;
the second acquisition module is used for acquiring the three-dimensional model of the target component and semantic information of the vehicle of the target component in the first state;
a generating module, configured to generate a second two-dimensional image corresponding to the vehicle when the target component is in the first state, by using the first two-dimensional image, the three-dimensional model of the target component, and a usage rule of the target component in the vehicle;
and the establishing module is used for establishing the incidence relation between the second two-dimensional image and the semantic information.
Optionally, the target component is a moving component;
the generation module is specifically configured to:
rendering by using the three-dimensional model of the target component to obtain a depth map of the target component;
recovering three-dimensional point cloud data of the target component by using the depth map of the target component and the first two-dimensional image;
translating or rotating the target component by utilizing the three-dimensional point cloud data and the motion rule of the motion component in the vehicle use to generate a target three-dimensional image corresponding to the vehicle when the target component is in the first state;
and mapping the target three-dimensional image to obtain the second two-dimensional image.
Optionally, the moving part comprises: door, bonnet and trunk.
Optionally, if an invisible target area in the first two-dimensional image exists in the second two-dimensional image, the generating module is further configured to:
acquiring an environment map of the first two-dimensional image;
rendering the target area by using the environment map, and fusing the rendered target area and the second two-dimensional image.
Optionally, the target component is a lighting component;
the generation module is specifically configured to:
performing two-dimensional projection by using the three-dimensional model of the target component to obtain a projection area of the target component in the first two-dimensional image;
and editing the color of the projection area by using a lighting rule of the lighting component in the use of the vehicle to generate a second two-dimensional image corresponding to the vehicle.
Optionally, the lighting part includes: a lamp for a vehicle.
Optionally, the method further includes:
the optimization module is used for filling the void area of the second two-dimensional image; and performing smooth filtering on the filled two-dimensional image.
Optionally, the semantic information of the vehicle of the target component in the first state includes one or more of the following:
the semantic information of the vehicle with the door in the open state is as follows: the personnel in the vehicle need to get off the vehicle;
the semantic information of the vehicle with the trunk in the open state is as follows: the personnel in the vehicle need to pick up or load goods;
the semantic information of the vehicle with the hood in the open state is: a vehicle failure;
the semantic information of the vehicle under the bright yellow state of the left headlight of the vehicle is as follows: turning left the vehicle;
the semantic information of the vehicle under the bright yellow state of the vehicle right headlight is as follows: the vehicle turns to the right.
A third aspect of the embodiments of the present application provides an electronic device, including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of the preceding first aspects.
A fourth aspect of embodiments of the present application provides a non-transitory computer-readable storage medium having stored thereon computer instructions for causing a computer to perform the method of any one of the preceding first aspects.
In summary, the embodiment of the present application has the following beneficial effects with respect to the prior art:
the embodiment of the application provides an image processing method and device, and the incidence relation between a second two-dimensional image and semantic information of a target component in a first state can be constructed according to a first two-dimensional image which is marked with a vehicle target component in a driving environment, a three-dimensional model of the target component and a use rule of the target component in a vehicle. In other words, in the embodiment of the application, a large number of images do not need to be manually acquired and labeled, and the incidence relation between the second two-dimensional image and the semantic information of the target component in the first state can be obtained based on image processing by combining the three-dimensional model of the target component and the two-dimensional image of the vehicle in the driving environment, so that the efficiency and the flexibility are greatly improved.
Drawings
Fig. 1 is a schematic flowchart of a method for image processing according to an embodiment of the present disclosure;
FIG. 2 is a schematic diagram of generating a second two-dimensional image according to an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application;
fig. 4 is a block diagram of an electronic device for implementing the method of image processing according to the embodiment of the present application.
Detailed Description
The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness. The embodiments described below and the features of the embodiments can be combined with each other without conflict.
The image processing method of the embodiment of the application can be applied to equipment with image processing capability, for example, the equipment can be a terminal or a server, and the terminal can include: electronic equipment such as a mobile phone, a tablet computer, a notebook computer, or a desktop computer. The embodiment of the present application does not specifically limit the specific device used.
The driving environment described in the embodiment of the present application may be a real environment in which the vehicle is driving, rather than a virtual scene similar to a game.
The three-dimensional model of the target component described in the embodiment of the present application may be a Computer Aided Design (CAD) model, and the three-dimensional model may include depth information of the target component.
The target component of the embodiment of the application can be a component capable of reflecting semantic information of different vehicles in different states in the vehicle. The semantic information of the vehicle described in the embodiments of the present application may be information reflecting a vehicle state. Illustratively, the target component may include: the semantic information of the vehicle with the opened door is as follows: the personnel in the vehicle need to get off the vehicle; the semantic information of the vehicle with the trunk in the open state is as follows: the personnel in the vehicle need to pick up or load goods; the semantic information of the vehicle with the hood in the open state is: a vehicle failure; the semantic information of the vehicle under the bright yellow state of the left headlight of the vehicle is as follows: turning left the vehicle; the semantic information of the vehicle under the bright yellow state of the vehicle right headlight is as follows: vehicle right turn, etc.
The usage rule of the target component in the vehicle described in the embodiment of the application may be only the usage rule to be followed when the target component is used in the vehicle. For example, a door in a vehicle may be rotated about an axis, etc.
As shown in fig. 1, fig. 1 is a schematic flowchart of an image processing method according to an embodiment of the present disclosure.
The method specifically comprises the following steps:
s101: acquiring a first two-dimensional image of a vehicle in a driving environment; a target component of the vehicle is marked in the first two-dimensional image.
In the embodiment of the application, a two-dimensional image containing a vehicle can be shot in the running environment of the vehicle, and a target component of the vehicle is marked in the image. When the target component is marked, the target component can be marked by adopting a six-degree-of-freedom pose or any other possible mode.
Although there may be a scene in which images of the vehicle in various states are obtained using a vehicle model or the like in a game, graphics production, or the like, if images obtained in the game or graphics production environment are used for recognition as semantic information of the vehicle in actual driving, recognition may be inaccurate or impossible because the environment in which the game or graphics production is carried out is significantly different from the actual driving environment.
Therefore, in the embodiment of the application, the two-dimensional image of the vehicle in the driving environment is obtained, when the second two-dimensional image is obtained based on the first two-dimensional image of the driving environment, the second two-dimensional image is in accordance with the driving environment, and when the semantic information of the vehicle is identified by using the second two-dimensional image, a more accurate identification result can be obtained.
S102: and acquiring a three-dimensional model of the target component and semantic information of the vehicle of the target component in a first state.
In this embodiment of the application, both the three-dimensional model of the target component and the semantic information of the vehicle of the target component in the first state may be obtained from a local or a network, which is not specifically limited in this embodiment of the application.
In a specific application, the number of the target components may be one or more. A three-dimensional model of the vehicle, which includes a three-dimensional model of the target component, may be obtained according to the model of the vehicle, and the like. Alternatively, a three-dimensional model of each target component may be acquired separately.
S103: and generating a second two-dimensional image corresponding to the vehicle when the target component is in the first state by using the first two-dimensional image, the three-dimensional model of the target component and the usage rule of the target component in the vehicle.
In the embodiment of the application, the target component may include a plurality of states in the vehicle, for example, when the target component is a door, the state of the door may include an open state or a closed state; when the target component is a trunk, the state of the trunk may include an open state or a closed state; when the target component is a hood, the state of the trunk may include an open state or a closed state; when the target component is a car lamp, the state of the car lamp can comprise that the left car lamp is in a bright yellow state, or the right car lamp is in a bright yellow state; and the like.
The different states of the target component may correspond to semantic information of different vehicles, so the first state may be specifically determined by combining states that may exist in the target component itself, and the first state is not limited in this embodiment of the application.
The complete structure of the target component (for example, the complete structure comprises an appearance part and an interior part of the target component) can be obtained by using the three-dimensional model of the target component, the target component in the vehicle can be in the state A in the first two-dimensional image, the target component is rotated, translated and the like according to the use rule of the target component in the vehicle, and the structure which can be newly added after the target component is rotated, translated and the like can be supplemented by combining the three-dimensional model of the target component, so that a second two-dimensional image of the target component in any state different from the state A can be synthesized.
In specific application, the first two-dimensional image, the three-dimensional model of the target component, and the usage rule of the target component in the vehicle may be used in any manner to generate the second two-dimensional image corresponding to the vehicle when the target component is in the first state, which is not specifically limited in this embodiment of the application.
S104: and establishing an incidence relation between the second two-dimensional image and the semantic information.
In the embodiment of the application, after the second two-dimensional image is obtained, the association relationship between the second two-dimensional image and the corresponding semantic information can be established. For example, the second two-dimensional image and the corresponding semantic information may be stored in association. Subsequently, in an automatic driving scene, if the automatic driving vehicle takes a picture of a surrounding vehicle, the picture of the surrounding vehicle may be matched with the second two-dimensional image to obtain semantic information of the surrounding vehicle, and then an executed automatic driving strategy is executed.
Optionally, a void region or the like may exist in the second two-dimensional image during the synthesis, so that the effect of the second two-dimensional image is not good, and inaccuracy may be caused when the semantic information is identified by using the second two-dimensional image, and therefore, the void region of the second two-dimensional image may be further filled; and performing smooth filtering on the filled two-dimensional image.
For example, the content in the target component in the first two-dimensional image and the content in the target component in the second two-dimensional image may be subjected to difference calculation, etc., the region with the larger difference is regarded as a cavity region, the content of the target component in the first two-dimensional image is combined to perform filling, and filtering algorithms such as bilateral filtering may be adopted to perform smooth filtering on the graph, so as to obtain the second two-dimensional image with a better effect.
In summary, the embodiment of the present application provides an image processing method and an image processing apparatus, which can construct an association relationship between a second two-dimensional image and semantic information of a target component in a first state according to a first two-dimensional image labeled with a vehicle target component in a driving environment and a three-dimensional model of the target component and a usage rule of the target component in a vehicle. In other words, in the embodiment of the application, a large number of images do not need to be manually acquired and labeled, and the incidence relation between the second two-dimensional image and the semantic information of the target component in the first state can be obtained based on image processing by combining the three-dimensional model of the target component and the two-dimensional image of the vehicle in the driving environment, so that the efficiency and the flexibility are greatly improved.
Optionally, the target component is a moving component; the step S103 of generating, by using the first two-dimensional image, the three-dimensional model of the target component, and the usage rule of the target component in the vehicle, a second two-dimensional image corresponding to the vehicle when the target component is in the first state includes:
rendering by using the three-dimensional model of the target component to obtain a depth map of the target component; recovering three-dimensional point cloud data of the target component by using the depth map of the target component and the first two-dimensional image; translating or rotating the target component by utilizing the three-dimensional point cloud data and the motion rule of the motion component in the vehicle use to generate a target three-dimensional image corresponding to the vehicle when the target component is in the first state; and mapping the target three-dimensional image to obtain the second two-dimensional image.
In an embodiment of the present application, the moving part may include: door, bonnet and trunk.
For the moving part, it may be divided into a visible region and an invisible region, the visible region may be a portion that can be displayed in the first two-dimensional image, and the invisible region may be a portion that cannot be displayed in the first two-dimensional image. For example, if the target member is a door, and the door is in a closed state in the first two-dimensional image, the door is visible as a visible region outside the vehicle when the door is closed, and is invisible as an invisible region outside the vehicle.
For the visible area, a depth map of the target component can be obtained by rendering the three-dimensional model of the target component, the three-dimensional point cloud data of the target component is recovered by using the depth map of the target component and the first two-dimensional image, and then the target component is translated or rotated by using the three-dimensional point cloud data and the motion rule of the motion component in the use of the vehicle, so that a target three-dimensional image corresponding to the vehicle when the target component is in the first state is generated. For example, in the first two-dimensional image, the doors, the hood, and the like of the vehicle are all in the closed state, and a three-dimensional image of the vehicle with the door open or the hood open may be generated based on the above steps.
And then mapping the target three-dimensional image onto the two-dimensional image by utilizing a camera imaging principle to obtain a second two-dimensional image.
For the invisible area, the image of the invisible area can be obtained by supplementing according to the three-dimensional model of the target component.
Optionally, if an invisible target area exists in the second two-dimensional image, the method further includes: acquiring an environment map of the first two-dimensional image; rendering the target area by using the environment map, and fusing the rendered target area and the second two-dimensional image.
Since the invisible target area is supplemented from the three-dimensional model of the target component, it may have a poor fusion with the environment, resulting in a poor image quality of the second two-dimensional image.
In the embodiment of the application, an environment map of a first two-dimensional image can be obtained; and rendering the target area by using the environment map, and fusing the rendered target area and the second two-dimensional image, so that the invisible target area is fused with the driving environment of the first two-dimensional image, and a more accurate second two-dimensional image is obtained.
Optionally, the target component is a lighting component; the step S103 of generating, by using the first two-dimensional image, the three-dimensional model of the target component, and the usage rule of the target component in the vehicle, a second two-dimensional image corresponding to the vehicle when the target component is in the first state includes:
performing two-dimensional projection by using the three-dimensional model of the target component to obtain a projection area of the target component in the first two-dimensional image; and editing the color of the projection area by using a lighting rule of the lighting component in the use of the vehicle to generate a second two-dimensional image corresponding to the vehicle.
In the embodiment of the present application, the lighting member may be a lamp of a vehicle. For the lighting component, two-dimensional projection can be directly carried out on the three-dimensional model of the target component in the first two-dimensional image to obtain a corresponding area in the first two-dimensional image, and then color editing is carried out on the area according to semantic information of the vehicle, for example, when the semantic information is left turn, the left headlight is edited to be yellow; when the semantic information is parking, the two tail lamps are edited to be red; when the semantic information is danger alarm, the two tail lamps are edited to be yellow; and the like.
Illustratively, FIG. 2 shows a schematic diagram of obtaining a second two-dimensional image.
As shown in fig. 2, a two-dimensional image of the target component (door and trunk) with a six-degree-of-freedom pose being marked, taken in the running environment, and a CAD three-dimensional model of the target component may be taken as inputs, and the two-dimensional image of the vehicle when the door is opened and the trunk is opened may be output. The method comprises the steps of obtaining a depth map of a vehicle door and a trunk by rendering a three-dimensional model of the vehicle door and the trunk in a visible area of the vehicle door and the trunk, recovering three-dimensional point cloud data of the vehicle door and the trunk by using the depth map of the vehicle door and the trunk and an input two-dimensional image, realizing part reconstruction, translating or rotating the vehicle door and the trunk by using the three-dimensional point cloud data and a motion rule of the vehicle door and the trunk in the use process of the vehicle to generate a three-dimensional image of the vehicle door and the trunk, further projecting the three-dimensional image into a two-dimensional image, and optimizing the two-dimensional image obtained by projection. For the invisible areas of the doors and trunk, an environment map and three-dimensional part rendering can be performed.
In the embodiment of the application, a large number of images do not need to be manually acquired and labeled, the incidence relation between the second two-dimensional image and the semantic information of the target component in the first state can be obtained based on image processing by combining the three-dimensional model of the target component and the two-dimensional image of the vehicle in the driving environment, the efficiency and the flexibility are greatly improved, and the cost is low.
Fig. 3 is a schematic structural diagram of an embodiment of an image processing apparatus provided in the present application. As shown in fig. 3, the image processing apparatus provided in the present embodiment includes:
a first obtaining module 31, configured to obtain a first two-dimensional image of the vehicle in a driving environment; a target component of the vehicle is marked in the first two-dimensional image;
a second obtaining module 32, configured to obtain a three-dimensional model of the target component and semantic information of the vehicle of the target component in the first state;
a generating module 33, configured to generate a second two-dimensional image corresponding to the vehicle when the target component is in the first state, by using the first two-dimensional image, the three-dimensional model of the target component, and a usage rule of the target component in the vehicle;
and the establishing module 34 is configured to establish an association relationship between the second two-dimensional image and the semantic information.
Optionally, the target component is a moving component;
the generation module is specifically configured to:
rendering by using the three-dimensional model of the target component to obtain a depth map of the target component;
recovering three-dimensional point cloud data of the target component by using the depth map of the target component and the first two-dimensional image;
translating or rotating the target component by utilizing the three-dimensional point cloud data and the motion rule of the motion component in the vehicle use to generate a target three-dimensional image corresponding to the vehicle when the target component is in the first state;
and mapping the target three-dimensional image to obtain the second two-dimensional image.
Optionally, the moving part comprises: door, bonnet and trunk.
Optionally, if an invisible target area in the first two-dimensional image exists in the second two-dimensional image, the generating module is further configured to:
acquiring an environment map of the first two-dimensional image;
rendering the target area by using the environment map, and fusing the rendered target area and the second two-dimensional image.
Optionally, the target component is a lighting component;
the generation module is specifically configured to:
performing two-dimensional projection by using the three-dimensional model of the target component to obtain a projection area of the target component in the first two-dimensional image;
and editing the color of the projection area by using a lighting rule of the lighting component in the use of the vehicle to generate a second two-dimensional image corresponding to the vehicle.
Optionally, the lighting part includes: a lamp for a vehicle.
Optionally, the method further includes:
the optimization module is used for filling the void area of the second two-dimensional image; and performing smooth filtering on the filled two-dimensional image.
Optionally, the semantic information of the vehicle of the target component in the first state includes one or more of the following:
the semantic information of the vehicle with the door in the open state is as follows: the personnel in the vehicle need to get off the vehicle;
the semantic information of the vehicle with the trunk in the open state is as follows: the personnel in the vehicle need to pick up or load goods;
the semantic information of the vehicle with the hood in the open state is: a vehicle failure;
the semantic information of the vehicle under the bright yellow state of the left headlight of the vehicle is as follows: turning left the vehicle;
the semantic information of the vehicle under the bright yellow state of the vehicle right headlight is as follows: the vehicle turns to the right.
The embodiment of the application provides an image processing method and device, and the incidence relation between a second two-dimensional image and semantic information of a target component in a first state can be constructed according to a first two-dimensional image which is marked with a vehicle target component in a driving environment, a three-dimensional model of the target component and a use rule of the target component in a vehicle. In other words, in the embodiment of the application, a large number of images do not need to be manually acquired and labeled, and the incidence relation between the second two-dimensional image and the semantic information of the target component in the first state can be obtained based on image processing by combining the three-dimensional model of the target component and the two-dimensional image of the vehicle in the driving environment, so that the efficiency and the flexibility are greatly improved.
The image processing apparatus provided in the embodiments of the present application can be used to execute the method shown in the corresponding embodiments, and the implementation manner and principle thereof are the same, and are not described again.
According to an embodiment of the present application, an electronic device and a readable storage medium are also provided.
As shown in fig. 4, is a block diagram of an electronic device according to a method of image processing of an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.
As shown in fig. 4, the electronic apparatus includes: one or more processors 401, memory 402, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions for execution within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input/output apparatus (such as a display device coupled to the interface). In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and multiple memories, as desired. Also, multiple electronic devices may be connected, with each device providing portions of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). In fig. 4, one processor 401 is taken as an example.
The memory 402, as a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the method of image processing in the embodiment of the present application (for example, the first acquiring module 31, the second acquiring module 32, the generating module 33, and the establishing module 34 shown in fig. 3). The processor 401 executes various functional applications of the server and data processing, i.e., a method of implementing image processing in the above-described method embodiments, by executing non-transitory software programs, instructions, and modules stored in the memory 402.
The memory 402 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the electronic device for image processing, and the like. Further, the memory 402 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 402 optionally includes memory located remotely from processor 401, which may be connected to image processing electronics over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The electronic device of the method of image processing may further include: an input device 403 and an output device 404. The processor 401, the memory 402, the input device 403 and the output device 404 may be connected by a bus or other means, and fig. 4 illustrates an example of a connection by a bus.
The input device 403 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the image processing electronic apparatus, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointing stick, one or more mouse buttons, a track ball, a joystick, or other input devices. The output devices 404 may include a display device, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibrating motors), among others. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device can be a touch screen.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
These computer programs (also known as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
According to the technical scheme of the embodiment of the application, the association relationship between the second two-dimensional image and the semantic information of the target component in the first state can be constructed according to the first two-dimensional image marked with the vehicle target component in the driving environment, the three-dimensional model of the target component and the use rule of the target component in the vehicle. In other words, in the embodiment of the application, a large number of images do not need to be manually acquired and labeled, and the incidence relation between the second two-dimensional image and the semantic information of the target component in the first state can be obtained based on image processing by combining the three-dimensional model of the target component and the two-dimensional image of the vehicle in the driving environment, so that the efficiency and the flexibility are greatly improved.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present application can be achieved, and the present invention is not limited herein.
The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (18)
1. A method of image processing, the method comprising:
acquiring a first two-dimensional image of a vehicle in a driving environment; a target component of the vehicle is marked in the first two-dimensional image;
acquiring a three-dimensional model of the target component and semantic information of the vehicle of the target component in a first state;
generating a second two-dimensional image corresponding to the vehicle when the target component is in the first state by using the first two-dimensional image, the three-dimensional model of the target component and the usage rule of the target component in the vehicle;
and establishing an incidence relation between the second two-dimensional image and the semantic information.
2. The method of claim 1, wherein the target component is a moving component;
the generating, by using the first two-dimensional image, the three-dimensional model of the target component, and the usage rule of the target component in the vehicle, a second two-dimensional image corresponding to the vehicle when the target component is in the first state includes:
rendering by using the three-dimensional model of the target component to obtain a depth map of the target component;
recovering three-dimensional point cloud data of the target component by using the depth map of the target component and the first two-dimensional image;
translating or rotating the target component by utilizing the three-dimensional point cloud data and the motion rule of the motion component in the vehicle use to generate a target three-dimensional image corresponding to the vehicle when the target component is in the first state;
and mapping the target three-dimensional image to obtain the second two-dimensional image.
3. The method of claim 2, wherein the moving component comprises: door, bonnet and trunk.
4. The method of claim 2, wherein if a target area not visible in the first two-dimensional image is present in the second two-dimensional image, the method further comprises:
acquiring an environment map of the first two-dimensional image;
rendering the target area by using the environment map, and fusing the rendered target area and the second two-dimensional image.
5. The method of claim 1, wherein the target component is a light component;
the generating, by using the first two-dimensional image, the three-dimensional model of the target component, and the usage rule of the target component in the vehicle, a second two-dimensional image corresponding to the vehicle when the target component is in the first state includes:
performing two-dimensional projection by using the three-dimensional model of the target component to obtain a projection area of the target component in the first two-dimensional image;
and editing the color of the projection area by using a lighting rule of the lighting component in the use of the vehicle to generate a second two-dimensional image corresponding to the vehicle.
6. The method of claim 5, wherein the lighting component comprises: a lamp for a vehicle.
7. The method of any one of claims 1-6, further comprising:
filling a hole area of the second two-dimensional image;
and performing smooth filtering on the filled two-dimensional image.
8. The method of any of claims 1-6, wherein the semantic information of the vehicle by the target component in the first state includes one or more of:
the semantic information of the vehicle with the door in the open state is as follows: the personnel in the vehicle need to get off the vehicle;
the semantic information of the vehicle with the trunk in the open state is as follows: the personnel in the vehicle need to pick up or load goods;
the semantic information of the vehicle with the hood in the open state is: a vehicle failure;
the semantic information of the vehicle under the bright yellow state of the left headlight of the vehicle is as follows: turning left the vehicle;
the semantic information of the vehicle under the bright yellow state of the vehicle right headlight is as follows: the vehicle turns to the right.
9. An apparatus for image processing, comprising:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring a first two-dimensional image of a vehicle in a driving environment; a target component of the vehicle is marked in the first two-dimensional image;
the second acquisition module is used for acquiring the three-dimensional model of the target component and semantic information of the vehicle of the target component in the first state;
a generating module, configured to generate a second two-dimensional image corresponding to the vehicle when the target component is in the first state, by using the first two-dimensional image, the three-dimensional model of the target component, and a usage rule of the target component in the vehicle;
and the establishing module is used for establishing the incidence relation between the second two-dimensional image and the semantic information.
10. The apparatus of claim 9, wherein the target component is a moving component;
the generation module is specifically configured to:
rendering by using the three-dimensional model of the target component to obtain a depth map of the target component;
recovering three-dimensional point cloud data of the target component by using the depth map of the target component and the first two-dimensional image;
translating or rotating the target component by utilizing the three-dimensional point cloud data and the motion rule of the motion component in the vehicle use to generate a target three-dimensional image corresponding to the vehicle when the target component is in the first state;
and mapping the target three-dimensional image to obtain the second two-dimensional image.
11. The apparatus of claim 10, wherein the moving member comprises: door, bonnet and trunk.
12. The apparatus of claim 10, wherein if a target area not visible in the first two-dimensional image exists in the second two-dimensional image, the generating module is further configured to:
acquiring an environment map of the first two-dimensional image;
rendering the target area by using the environment map, and fusing the rendered target area and the second two-dimensional image.
13. The apparatus of claim 9, wherein the target component is a light component;
the generation module is specifically configured to:
performing two-dimensional projection by using the three-dimensional model of the target component to obtain a projection area of the target component in the first two-dimensional image;
and editing the color of the projection area by using a lighting rule of the lighting component in the use of the vehicle to generate a second two-dimensional image corresponding to the vehicle.
14. The apparatus of claim 13, wherein the lighting member comprises: a lamp for a vehicle.
15. The apparatus of any one of claims 9-14, further comprising:
the optimization module is used for filling the void area of the second two-dimensional image; and performing smooth filtering on the filled two-dimensional image.
16. The apparatus of any of claims 9-14, wherein the semantic information of the vehicle by the target component in the first state comprises one or more of:
the semantic information of the vehicle with the door in the open state is as follows: the personnel in the vehicle need to get off the vehicle;
the semantic information of the vehicle with the trunk in the open state is as follows: the personnel in the vehicle need to pick up or load goods;
the semantic information of the vehicle with the hood in the open state is: a vehicle failure;
the semantic information of the vehicle under the bright yellow state of the left headlight of the vehicle is as follows: turning left the vehicle;
the semantic information of the vehicle under the bright yellow state of the vehicle right headlight is as follows: the vehicle turns to the right.
17. An electronic device, comprising:
at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.
18. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010102091.5A CN113283268B (en) | 2020-02-19 | 2020-02-19 | Image processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010102091.5A CN113283268B (en) | 2020-02-19 | 2020-02-19 | Image processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113283268A true CN113283268A (en) | 2021-08-20 |
| CN113283268B CN113283268B (en) | 2023-06-23 |
Family
ID=77274937
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010102091.5A Active CN113283268B (en) | 2020-02-19 | 2020-02-19 | Image processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113283268B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103020590A (en) * | 2012-11-20 | 2013-04-03 | 北京航空航天大学深圳研究院 | Vehicle identification system and method based on three-dimensional model and image matching |
| US20170220887A1 (en) * | 2016-01-29 | 2017-08-03 | Pointivo, Inc. | Systems and methods for extracting information about objects from scene information |
| CN107730503A (en) * | 2017-09-12 | 2018-02-23 | 北京航空航天大学 | The image object component-level semantic segmentation method and apparatus of three-dimensional feature insertion |
| CN108428254A (en) * | 2018-03-15 | 2018-08-21 | 斑马网络技术有限公司 | The construction method and device of three-dimensional map |
| CN110321877A (en) * | 2019-06-04 | 2019-10-11 | 中北大学 | Three mesh rearview mirrors of one kind and trinocular vision safe driving method and system |
| CN110458112A (en) * | 2019-08-14 | 2019-11-15 | 上海眼控科技股份有限公司 | Vehicle checking method, device, computer equipment and readable storage medium storing program for executing |
-
2020
- 2020-02-19 CN CN202010102091.5A patent/CN113283268B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103020590A (en) * | 2012-11-20 | 2013-04-03 | 北京航空航天大学深圳研究院 | Vehicle identification system and method based on three-dimensional model and image matching |
| US20170220887A1 (en) * | 2016-01-29 | 2017-08-03 | Pointivo, Inc. | Systems and methods for extracting information about objects from scene information |
| CN107730503A (en) * | 2017-09-12 | 2018-02-23 | 北京航空航天大学 | The image object component-level semantic segmentation method and apparatus of three-dimensional feature insertion |
| CN108428254A (en) * | 2018-03-15 | 2018-08-21 | 斑马网络技术有限公司 | The construction method and device of three-dimensional map |
| CN110321877A (en) * | 2019-06-04 | 2019-10-11 | 中北大学 | Three mesh rearview mirrors of one kind and trinocular vision safe driving method and system |
| CN110458112A (en) * | 2019-08-14 | 2019-11-15 | 上海眼控科技股份有限公司 | Vehicle checking method, device, computer equipment and readable storage medium storing program for executing |
Non-Patent Citations (2)
| Title |
|---|
| DO NHAT QUANG(全杜晄): "基于三维模型的自动语义标注研究与应用", 中国优秀硕士学位论文全文数据库信息科技辑, no. 3, pages 138 - 6864 * |
| 石磊: "自主式车辆环境感知技术研究 -道路环境理解方法研究", 中国博士学位论文全文数据库信息科技辑, no. 8, pages 138 - 20 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113283268B (en) | 2023-06-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112132829A (en) | Vehicle information detection method and device, electronic equipment and storage medium | |
| CN111931591B (en) | Method, device, electronic equipment and readable storage medium for constructing key point learning model | |
| KR20210040326A (en) | Cross-modality processing method and apparatus, and computer storage medium | |
| CN111860167B (en) | Face fusion model acquisition method, face fusion model acquisition device and storage medium | |
| CN111294665B (en) | Video generation method and device, electronic equipment and readable storage medium | |
| WO2019113510A1 (en) | Techniques for training machine learning | |
| CN111832745B (en) | Data augmentation method and device and electronic equipment | |
| CN111722245B (en) | Positioning method, positioning device and electronic equipment | |
| CN111739005B (en) | Image detection method, device, electronic equipment and storage medium | |
| CN112529073A (en) | Model training method, attitude estimation method and apparatus, and electronic device | |
| CN112270711B (en) | Model training and posture prediction method, device, equipment and storage medium | |
| CN111753961A (en) | Model training method and device, and prediction method and device | |
| CN111401251B (en) | Lane line extraction method, lane line extraction device, electronic equipment and computer readable storage medium | |
| CN110619312B (en) | Method, device and equipment for enhancing positioning element data and storage medium | |
| CN111860304A (en) | Image labeling method, electronic device, equipment and storage medium | |
| CN111861955A (en) | Method and device for constructing image editing model | |
| CN113408662B (en) | Image recognition and training method and device for image recognition model | |
| CN112527374A (en) | Marking tool generation method, marking method, device, equipment and storage medium | |
| CN111709873A (en) | Training method and device of image conversion model generator | |
| CN110866504B (en) | Method, device and equipment for acquiring annotation data | |
| CN112581533A (en) | Positioning method, positioning device, electronic equipment and storage medium | |
| US11697428B2 (en) | Method and apparatus for 3D modeling | |
| CN112016524B (en) | Model training method, face recognition device, equipment and medium | |
| CN112508027B (en) | Head model for instance segmentation, instance segmentation model, image segmentation method and device | |
| CN112508964A (en) | Image segmentation method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |