CN117173691A - Virtual article searching method and device, electronic equipment and storage medium - Google Patents
Virtual article searching method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN117173691A CN117173691A CN202310913175.0A CN202310913175A CN117173691A CN 117173691 A CN117173691 A CN 117173691A CN 202310913175 A CN202310913175 A CN 202310913175A CN 117173691 A CN117173691 A CN 117173691A
- Authority
- CN
- China
- Prior art keywords
- virtual
- virtual article
- feature vector
- feature
- photographer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 61
- 239000013598 vector Substances 0.000 claims abstract description 116
- 238000012549 training Methods 0.000 claims description 47
- 230000006870 function Effects 0.000 claims description 17
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 4
- 238000004891 communication Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000010276 construction Methods 0.000 description 4
- 238000012423 maintenance Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 239000008186 active pharmaceutical agent Substances 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 2
- 239000003795 chemical substances by application Substances 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Landscapes
- Processing Or Creating Images (AREA)
Abstract
The application provides a method, a device, electronic equipment and a storage medium for searching a virtual article, which comprise the following steps: acquiring a shooting position of a photographer and a physical picture shot by the photographer at the shooting position; inputting the physical photo to a first feature extractor to obtain a feature vector of the physical photo; calculating the similarity between the feature vector of the physical photo and the feature vector of the virtual article corresponding to the virtual article, and determining a candidate set from the virtual article based on the similarity, wherein the feature vector of the virtual article is input to a second feature extractor by adopting a two-dimensional view; a target virtual item is determined from the candidate set based on the photographing position and the candidate set.
Description
Technical Field
The present application relates to the field of virtual article identification, and in particular, to a method and apparatus for searching for a virtual article, an electronic device, and a storage medium.
Background
BIM is an abbreviation for building information model (Building Information Modelling) which contains three-dimensional models and locations of several building elements, such as walls, columns, beams, doors, windows, air conditioners, water pumps, furniture, etc., and the element properties in BIM have non-geometric information such as materials, suppliers, costs, etc., in addition to geometric information. BIM plays a great role in the processes of building design, construction, operation management and the like, for example, more accurate simulation and emulation calculation can be realized in the design stage; the coordination and cost control of multiple parties can be realized in the construction stage; the management efficiency can be improved, the service life of equipment can be prolonged, and the like in the operation stage. Among the application scenes of the BIM, some on-site operation application scenes need to search corresponding virtual objects in the BIM through real objects. For example, after the installation of the physical device is completed, the virtual article of the device needs to be queried in the BIM, and the serial number label of the virtual article is attached to the physical device; in the equipment maintenance process, if the equipment has no label, maintenance personnel need to search the virtual article in the BIM so as to acquire parameters of the equipment, fill in maintenance results and the like. The searching work is realized in a browsing searching mode in a massive building information model manually at present, and the working efficiency is very low. Currently, attempts are made to implement a search for virtual items in a graphical search. However, the traditional graph searching method has the problem that the accuracy of searching the virtual articles in the building information model is not high.
Disclosure of Invention
The application provides a virtual article searching method, a virtual article searching device, electronic equipment and a storage medium.
The application provides a method for searching a virtual article, which comprises the following steps:
acquiring a shooting position of a photographer and a physical picture shot by the photographer at the shooting position;
inputting the physical photo to a first feature extractor to obtain a feature vector of the physical photo;
calculating the similarity between the feature vector of the physical photo and the feature vector of the virtual article corresponding to the virtual article, and determining a candidate set from the virtual article based on the similarity, wherein the feature vector of the virtual article is input to a second feature extractor by adopting a two-dimensional view;
a target virtual item is determined from the candidate set based on the photographing position and the candidate set.
In some embodiments, the method further comprises:
constructing a training set, wherein the training set comprises: the sample physical photo set and the sample two-dimensional view set corresponding to the sample physical;
training an initial second feature extractor by adopting the sample two-dimensional view set to obtain an intermediate second feature extractor;
inputting the sample physical photo set to an initial first feature extractor to obtain a first sample feature vector, inputting a sample two-dimensional view set corresponding to the sample physical photo set to an intermediate second feature extractor to obtain a second sample feature vector, and training the intermediate second feature extractor and the initial first feature extractor by adopting an L2 norm based on the first sample feature vector and the second sample feature vector to obtain the first feature extractor and the second feature extractor.
In some embodiments, the sample two-dimensional view set includes: training an initial second feature extractor by using the sample two-dimensional view set to obtain an intermediate second feature extractor, wherein the two-dimensional view of the virtual article from different view angles and view points comprises:
taking a two-dimensional view of the same virtual article from different perspectives and viewpoints as a positive sample of the initial second feature extractor training;
taking a two-dimensional view of different perspectives and viewpoints of different virtual articles as a negative sample of the initial second feature extractor training;
training the initial second feature extractor using an InfoNCE loss function based on the positive and negative samples to obtain the intermediate second feature extractor.
In some embodiments, the calculating the similarity between the feature vector of the physical photo and the virtual article feature vector corresponding to the virtual article, and determining the candidate set from the virtual article based on the similarity includes:
carrying out similarity calculation on the feature vector of the physical photo and the feature vector of the virtual article corresponding to the virtual article one by one to obtain the similarity of the feature vector of the physical photo and the feature vector of the virtual article;
And taking the first N virtual articles with the similarity larger than a similarity threshold value as a candidate set, wherein N is a positive integer.
In some embodiments, the acquiring the shooting location of the photographer and the physical photograph of the photographer taken at the shooting location includes:
selecting a point as an origin in a real object space, and establishing a three-dimensional coordinate system xyz;
when the photographer shoots a physical photo, the shooting position of the photographer is obtained through an indoor positioning system, and the shooting position of the photographer and the physical photo shot by the photographer at the shooting position are obtained, wherein the indoor positioning system takes the three-dimensional coordinate system xyz as a reference coordinate.
In some embodiments, the determining a target virtual item from the candidate set based on the shooting location and the candidate set includes:
mapping the shooting position of the photographer in a three-dimensional coordinate system xyz to a three-dimensional coordinate system x 'y' z ', judging whether the virtual object in the candidate set and the shooting position of the photographer are in the same space, wherein the same space is a range defined by taking the virtual object as an origin, and the three-dimensional coordinate system x' y 'z' is established in the same direction of the three-dimensional coordinate system xyz by selecting a point which is the same as the real object space in the virtual space as the origin;
And if the virtual article and the shooting position of the photographer are in the same space, and the virtual article is one, determining that the virtual article is a target virtual article.
In some embodiments, the method further comprises:
if the virtual article and the shooting positions of the photographers are in the same space, and the virtual articles are a plurality of, comparing the distances between the shooting positions of the photographers and the virtual articles one by one;
and determining the virtual object closest to the shooting position of the photographer as a target virtual object.
In some embodiments, the method further comprises:
acquiring a two-dimensional view of the virtual article;
inputting the two-dimensional view of the virtual article to a first feature extractor to extract feature vectors of the virtual article;
associating the feature vector of each virtual article with the number of the virtual article to form a feature dictionary;
the calculating the similarity between the feature vector of the physical photo and the feature vector of the virtual article corresponding to the virtual article, and determining the candidate set from the virtual article based on the similarity comprises:
and calculating the similarity between the feature vector of the physical photo and the feature vector corresponding to the number of the virtual object in the feature dictionary, and determining a candidate number set from the numbers of the virtual object based on the similarity so as to determine a candidate set based on the candidate number set.
The embodiment of the application provides a device for searching a virtual article, which comprises the following steps:
the shooting device comprises an acquisition module, a control module and a control module, wherein the acquisition module is used for acquiring a shooting position of a photographer and a physical picture shot by the photographer at the shooting position;
the input module is used for inputting the physical photo to the first feature extractor to obtain a feature vector of the physical photo;
the first determining module is used for calculating the similarity between the feature vector of the physical photo and the feature vector of the virtual article corresponding to the virtual article, and determining a candidate set from the virtual article based on the similarity, wherein the feature vector of the virtual article is input to the second feature extractor by adopting a two-dimensional view;
and a second determining module for determining a target virtual article from the candidate set based on the shooting position and the candidate set.
An embodiment of the present application provides an electronic device, including: a memory and a processor, the memory having stored thereon a computer program which, when executed by the processor, performs the method of any of the above.
Embodiments of the present application provide a storage medium storing a computer program executable by one or more processors for implementing a method as described in any one of the above.
The embodiment of the application provides a method, a device, electronic equipment and a storage medium for searching a virtual article, which are used for extracting a feature vector of a physical photo through a first feature extractor and calculating the similarity of the feature vector of the physical photo and the feature vector of the virtual article, wherein the feature vector of the virtual article is input to a second feature extractor through a two-dimensional view, so that the problem of cross-modal of the physical photo and the virtual article is solved, a candidate set is determined from the virtual article based on the similarity, a target virtual article is determined from the candidate set through a shooting position corresponding to the physical photo, and the matching accuracy and efficiency are improved.
Drawings
The application will be described in more detail hereinafter on the basis of embodiments and with reference to the accompanying drawings.
Fig. 1 is a schematic implementation flow chart of a method for searching a virtual article according to an embodiment of the present application;
FIG. 2 is a schematic flow chart of training a first feature extractor and a second feature extractor according to an embodiment of the present application;
FIG. 3 is a schematic flow chart of training an initial second feature extractor according to an embodiment of the present application;
FIG. 4 is a schematic flow chart of determining a candidate set from the virtual articles based on similarity according to an embodiment of the present application;
Fig. 5 is a schematic flow chart of acquiring a shooting position and a physical photo according to an embodiment of the present application;
fig. 6 is a schematic flow chart of determining a target virtual object based on a shooting position and a candidate set according to an embodiment of the present application;
fig. 7 is another schematic flow chart of determining a target virtual object based on a shooting position and a candidate set according to an embodiment of the present application;
FIG. 8 is a schematic flow chart of determining a candidate set based on a feature dictionary according to an embodiment of the present application;
fig. 9 is a schematic structural diagram of a device for searching for a virtual article according to an embodiment of the present application;
fig. 10 is a schematic diagram of a composition structure of an electronic device according to an embodiment of the present application.
Detailed Description
The present application will be further described in detail with reference to the accompanying drawings, for the purpose of making the objects, technical solutions and advantages of the present application more apparent, and the described embodiments should not be construed as limiting the present application, and all other embodiments obtained by those skilled in the art without making any inventive effort are within the scope of the present application.
In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is to be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with one another without conflict.
If a similar description of "first\second\third" appears in the application document, the following description is added, in which the terms "first\second\third" are merely distinguishing between similar objects and do not represent a particular ordering of the objects, it being understood that the "first\second\third" may be interchanged in a particular order or precedence, where allowed, to enable embodiments of the application described herein to be practiced in an order other than that illustrated or described herein.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the application only and is not intended to be limiting of the application.
Based on the problems existing in the related art, the embodiment of the application provides a method for searching a virtual article, which is applied to electronic equipment, wherein the electronic equipment can be a mobile phone, a computer and other equipment. The functions realized by the virtual article searching method provided by the embodiment of the application can be realized by calling program codes by a processor of electronic equipment, wherein the program codes can be stored in a computer storage medium.
An embodiment of the present application provides a method for searching a virtual article, and fig. 1 is a schematic implementation flow diagram of the method for searching a virtual article provided by the embodiment of the present application, as shown in fig. 1, including:
step S101, acquiring a shooting position of a photographer and a physical picture shot by the photographer at the shooting position;
in the embodiment of the application, a photographer uses a photographing device to obtain a photographing position of the photographer and a physical picture photographed by the photographer at the photographing position, and the photographing device can be a mobile phone, a camera and the like with an indoor positioning system and a communication function. When a photographer shoots a physical photo through a shooting device, the shooting position of the photographer is acquired through an indoor positioning system of the shooting device, so that the shooting device can acquire the position of the photographer and the physical photo shot by the photographer at the shooting position. It is to be appreciated that the indoor positioning system can be bluetooth, beacons, etc.
Step S102, inputting the physical photo to a first feature extractor to obtain a feature vector of the physical photo;
in an embodiment of the present application, the first feature extractor may select ResNet as the feature extractor. The first feature extractor may be denoted as g (·) and the feature vector of the physical photograph as g (x).
Step S103, calculating the similarity between the feature vector of the physical photo and the feature vector of the virtual article corresponding to the virtual article, and determining a candidate set from the virtual article based on the similarity, wherein the feature vector of the virtual article is input to a second feature extractor by adopting a two-dimensional view;
in the embodiment of the application, the similarity between the feature vector of the physical photo and the feature vector of the virtual article corresponding to the virtual article can be calculated by adopting methods such as Euclidean distance or cosine similarity. Taking Euclidean distance as an example, the feature vector g (x) of the physical photo and the feature vector z of the virtual object are exemplified i Through the formula d= |g (x) -z i || 2 And carrying out Euclidean distance calculation to obtain a distance value, wherein the smaller the distance value is, the higher the similarity is, the larger the distance value is, the lower the similarity is, and finally, determining a candidate set from the virtual object according to the distance value. The number of virtual articles in the candidate set may be screened by a preset condition, for example, sorting from small to large distance values, and selecting the first N virtual articles to be added to the candidate set, where N is a positive integer. Wherein, the second feature extractor can select ResNet as the feature extractor, the second feature extractor can be marked as f (), the feature vector z of the virtual article i Can be recorded asWherein b is i Representing virtual articles, σ and θ j Respectively representing a projection function and a projection parameter, and K represents the number of views.
In the embodiment of the application, the key features of the virtual article are captured through the two-dimensional view of the virtual article, the two-dimensional view is matched with the physical photo, and the two-dimensional view and the physical photo belong to the same mode, namely are presented in the form of two-dimensional images, so that the alignment of the feature spaces of the two-dimensional view and the physical photo is facilitated, and the problem of cross-mode when the two-dimensional physical photo and the three-dimensional virtual article are matched is solved. Wherein the two-dimensional view may be generated by selecting different perspectives and viewpoints of the virtual article for projection. By way of example, taking building information modeling as an example, a two-dimensional view may be generated by building or importing existing building information modeling in the Revit software, and selecting different views and viewpoints of the virtual object to project by using plug-ins (plug-ins for developing related functions based on APIs provided by the Revit software), so as to generate the two-dimensional view. The number of two-dimensional views corresponding to one virtual article can be set as required, for example, 12, 15 and 20.
Step S104, determining a target virtual article from the candidate set based on the shooting position and the candidate set.
In the embodiment of the application, on the basis of the similarity of the feature vector of the physical photo and the feature vector of the virtual object, the target virtual object is determined according to the position relation between the shooting position of the photographer and each virtual object in the candidate set, and the accuracy and efficiency of the final matching result are determined according to the similarity of the feature vector and the shooting position of the photographer, so that the selected target virtual object is ensured to be closest to the physical photo.
In some embodiments, fig. 2 is a schematic flow chart of training the first feature extractor and the second feature extractor according to an embodiment of the present application, as shown in fig. 2, before step S101, the method further includes:
step S11, a training set is constructed, wherein the training set comprises: the sample physical photo set and the sample two-dimensional view set corresponding to the sample physical;
in the embodiment of the application, the training set is composed of a plurality of sample two-dimensional views of the virtual object and sample physical photo pairs, wherein each sample physical photo should have a sample two-dimensional view of a group of virtual objects corresponding to each sample physical photo. The training set D may be described as:
wherein x is i Representing a sample physical photograph, b i Representing virtual articles, σ and θ j Respectively representing a projection function and a projection parameter, K represents the number of views, and N represents the size of the data set.
In an embodiment of the present application, taking a building information model as an example, two-dimensional view data of virtual articles in the building information model is collected. Illustratively, existing building information models are built or imported in the Revit software. These building information models should cover all types of virtual articles to be found, such as walls, columns, beams, doors, windows, air conditioners, water pumps, furniture, etc. In the Revit software, plug-ins (plug-ins for developing related functions based on an API provided by the Revit software) can be used to select different viewing angles and viewpoints of the virtual object to project, so as to generate a two-dimensional view corresponding to the virtual object. And collecting a sample physical photo set, wherein the sample physical photo set is a physical photo corresponding to the virtual object shot on site. These physical photographs should contain all types of items found in the virtual item. When the physical photo is acquired, the actual use scene, such as the environment of equipment installation, maintenance and the like, is simulated as much as possible; the same object should be photographed from different angles and viewpoints as much as possible to capture its features as comprehensively as possible.
Step S12, training an initial second feature extractor by using the sample two-dimensional view set to obtain an intermediate second feature extractor;
Step S13, inputting the sample physical photo set to an initial first feature extractor to obtain a first sample feature vector, inputting a sample two-dimensional view set corresponding to the sample physical photo set to an intermediate second feature extractor to obtain a second sample feature vector, training the intermediate second feature extractor and the initial first feature extractor by adopting an L2 norm based on the first sample feature vector and the second sample feature vector to obtain the first feature extractor and the second feature extractor.
In the embodiment of the application, a sample physical photo set is input to an initial first feature extractor to obtain a first sample feature vector, and a sample two-dimensional view set corresponding to the sample physical photo set is input to an intermediate second feature extractor to obtain a second sample feature vector. Training an intermediate second feature extractor and an initial first feature extractor by adopting an L2 norm based on the first sample feature vector and the second sample feature vector, wherein the L2 norm is as follows:
in the formula, b i For a sample virtual article, σ is the projection function, θ j For projection parameters, x i Is a sample physical photo.
In the embodiment of the application, the second feature extractor and the initial first feature extractor in the middle of L2 norm training are used for aligning the space of the feature vector of the physical photo and the feature vector of the virtual object, so that the features of the physical photo and the corresponding virtual object are pulled up, and the cross-domain problem in the matching process is solved.
In some embodiments, fig. 3 is a schematic flow chart of training an initial second feature extractor according to an embodiment of the present application, as shown in fig. 3, where the sample two-dimensional view set includes: training an initial second feature extractor by using the sample two-dimensional view set to obtain an intermediate second feature extractor, wherein the two-dimensional view of the virtual article from different view angles and view points comprises:
step S121, taking two-dimensional views of different perspectives and viewpoints of the same virtual article as positive samples of the initial second feature extractor training;
step S122, taking two-dimensional views of different visual angles and viewpoints of different virtual articles as negative samples of the initial second feature extractor training;
step S123, training the initial second feature extractor using an InfoNCE loss function based on the positive and negative samples, to obtain the intermediate second feature extractor.
In the embodiment of the application, in order to ensure that the view angle of the virtual article is unchanged, taking two-dimensional views of different view angles and view points of the same virtual article as positive samples for training an initial second feature extractor, taking two-dimensional views of different view angles and view points of different virtual articles as negative samples for training the initial second feature extractor, and training the initial second feature extractor through an InfoNCE loss function to obtain an intermediate second feature extractor.
Wherein, the InfoNCE loss function is:
in the formula, whereinIs the temperature parameter of InfoNCE loss function, sigma is projection function, b i Is a virtual article; θ j And theta j For different projection parameters, applied to b i Obtaining a set of positive sample pairs, b i′ And theta k All negative pairs of samples, different from positive ones, are enumerated.
In the embodiment of the application, the agent task of individual discrimination is completed through contrast learning, unsupervised feature learning is realized, and a large amount of unlabeled data can be utilized in the contrast learning process, so that only a two-dimensional view of thousands of groups of virtual articles is acquired.
In some embodiments, fig. 4 is a schematic flow chart of determining a candidate set from the virtual articles based on the similarity, as shown in fig. 4, where the calculating the similarity between the feature vector of the physical photo and the feature vector of the virtual article corresponding to the virtual article, and determining the candidate set from the virtual article based on the similarity, includes:
step S1031, performing similarity calculation on the feature vector of the physical photo and the feature vector of the virtual article corresponding to the virtual article one by one, to obtain similarity between the feature vector of the physical photo and the feature vector of the virtual article;
Step S1032, wherein the first N virtual articles with similarity greater than the similarity threshold are used as candidate sets, and N is a positive integer.
In the embodiment of the application, the similarity between the feature vector of the physical photo and the feature vector of the virtual object is compared with the similarity threshold value, and most of the virtual objects are removed through the preset similarity threshold value. It can be understood that the similarity between the feature vector of the physical photo and the feature vector of the virtual article is compared with the similarity threshold, and only one virtual article larger than the similarity threshold may exist, and the virtual article is selected as the target virtual article at this time; when a plurality of virtual articles larger than the similarity threshold exist, the first N virtual articles are selected as candidate sets, and the number of the virtual articles in the candidate sets can be preset, for example, 3, 5 and 6. When the virtual article is selected, the similarity of the feature vector of the physical photo and the feature vector of the virtual article can be ordered, and the selection is performed from large to small according to the similarity.
In some embodiments, fig. 5 is a schematic flow chart of acquiring a shooting position and a physical photograph according to an embodiment of the present application, as shown in fig. 5, where the acquiring the shooting position of a photographer and the physical photograph of the photographer at the shooting position includes:
Step S1011, selecting a point in the real space as an origin, and establishing a three-dimensional coordinate system xyz;
step S1012, when the photographer takes a physical photo, obtaining a photographing position of the photographer through an indoor positioning system, and obtaining the photographing position of the photographer and the physical photo taken by the photographer at the photographing position, where the indoor positioning system uses the three-dimensional coordinate system xyz as a reference coordinate.
In the embodiment of the application, a point is selected in the real space as an origin, and the three-dimensional coordinate system xyz is established as the reference coordinate of the indoor positioning system, so that the specific position of a photographer in the three-dimensional coordinate system xyz can be conveniently obtained through the indoor positioning system.
In some embodiments, fig. 6 is a schematic flow chart of determining a target virtual article based on a shooting position and a candidate set according to an embodiment of the present application, as shown in fig. 6, where determining the target virtual article from the candidate set based on the shooting position and the candidate set includes:
step S1041, mapping the shooting position of the photographer in the three-dimensional coordinate system xyz to a three-dimensional coordinate system x 'y' z ', and determining whether the virtual object in the candidate set and the shooting position of the photographer are in the same space, wherein the same space is a range defined by taking the virtual object as an origin, and the three-dimensional coordinate system x' y 'z' selects a point identical to the real object space in the virtual space as the origin and is established according to the same direction of the three-dimensional coordinate system xyz;
In step S1042, if the virtual object is in the same space as the shooting position of the photographer and the virtual object is one, the virtual object is determined to be the target virtual object.
In the embodiment of the application, the photographer position is represented by a three-dimensional coordinate system xyz, for example, the photographer position s (x s y s z s ) The position of the virtual object is represented by a three-dimensional coordinate system x 'y' z ', e.g. the position s' (x ') of the virtual object' i y’ i z’ i ) The three-dimensional coordinate system x ' y ' z ' is established according to the same direction of the three-dimensional coordinate system xyz by selecting a point which is the same as the real space as the origin in the virtual space, so that the photographer position s (x s y s z s ) Mapping the virtual space, denoted s (x' s y’ s z’ s ) I.e. according to s (x' s y’ s z’ s )、s’(x’ i y’ i z’ i ) And judging whether the shooting position of the photographer and the virtual articles in the candidate set are in the same space, and determining that the virtual articles are target virtual articles when the shooting position of the photographer is in the same space and the virtual articles are one. The same space is a range defined by taking the virtual object as an origin, and the same space can refer to a room in the virtual space, and when the virtual object and the position of the photographer are in the same room, the virtual object and the position of the photographer are judged to be in the same space.
In some embodiments, fig. 7 is another schematic flow chart of determining a target virtual object based on a shooting location and a candidate set according to an embodiment of the present application, as shown in fig. 7, where the method further includes:
step S1043, if the virtual object and the shooting positions of the photographers are in the same space, and the virtual objects are plural, comparing the distances between the shooting positions of the photographers and the virtual objects one by one;
step S1044 of determining the virtual article closest to the shooting position of the photographer as a target virtual article.
In the embodiment of the application, a plurality of virtual articles which are in the same space with the position of the photographer may exist in the candidate set, and at the moment, the virtual article with the closest distance is selected as the target virtual article by comparing the distance between the photographing position and the virtual article, so that the matching accuracy is further improved.
In some embodiments, fig. 8 is a schematic flow chart of determining a candidate set based on a feature dictionary according to an embodiment of the present application, as shown in fig. 8, where the method further includes:
step S1001, obtaining a two-dimensional view of the virtual article;
step S1002, inputting a two-dimensional view of the virtual article to a first feature extractor to extract feature vectors of the virtual article;
Step S1003, associating the feature vector of each virtual article with the number of the virtual article to form a feature dictionary;
the calculating the similarity between the feature vector of the physical photo and the feature vector of the virtual article corresponding to the virtual article, and determining the candidate set from the virtual article based on the similarity comprises:
step S1004, calculating a similarity between the feature vector of the physical photo and the feature vector corresponding to the number of the virtual article in the feature dictionary, and determining a candidate number set from the numbers of the virtual article based on the similarity, so as to determine a candidate set based on the candidate number set.
In the embodiment of the application, after the construction of the feature dictionary is finished, each item in the feature dictionary contains the feature vector and the corresponding number of the virtual article, the construction of the feature dictionary is a key step for realizing the searching of the virtual article by using the physical photo, and the feature dictionary provides a quick and efficient data structure, so that the searching of the target virtual article can be finished in a shorter time.
An embodiment of the present application provides a device for searching a virtual article, and fig. 9 is a schematic structural diagram of the device for searching a virtual article provided in the embodiment of the present application, where as shown in fig. 9, the device for searching a virtual article includes:
The shooting device comprises an acquisition module, a control module and a control module, wherein the acquisition module is used for acquiring a shooting position of a photographer and a physical picture shot by the photographer at the shooting position;
the input module is used for inputting the physical photo to the first feature extractor to obtain a feature vector of the physical photo;
the first determining module is used for calculating the similarity between the feature vector of the physical photo and the feature vector of the virtual article corresponding to the virtual article, and determining a candidate set from the virtual article based on the similarity, wherein the feature vector of the virtual article is input to the second feature extractor by adopting a two-dimensional view;
and a second determining module for determining a target virtual article from the candidate set based on the shooting position and the candidate set.
In some embodiments, the virtual article searching apparatus further comprises:
a building module, configured to build a training set, where the training set includes: the sample physical photo set and the sample two-dimensional view set corresponding to the sample physical;
the first training module is used for training the initial second feature extractor by adopting the sample two-dimensional view set to obtain an intermediate second feature extractor;
the second training module is used for inputting the sample physical photo set to an initial first feature extractor to obtain a first sample feature vector, inputting a sample two-dimensional view set corresponding to the sample physical photo set to an intermediate second feature extractor to obtain a second sample feature vector, and training the intermediate second feature extractor and the initial first feature extractor by adopting an L2 norm based on the first sample feature vector and the second sample feature vector to obtain the first feature extractor and the second feature extractor.
In some embodiments, the sample two-dimensional view set includes: the first training module trains an initial second feature extractor by using the sample two-dimensional view set to obtain an intermediate second feature extractor, and the first training module comprises:
a first sample unit for taking two-dimensional views of different perspectives and viewpoints of the same virtual article as positive samples of the initial second feature extractor training;
a second sample unit for taking a two-dimensional view of different perspectives and viewpoints of different virtual articles as a negative sample of the initial second feature extractor training;
and the training unit is used for training the initial second feature extractor by using an InfoNCE loss function based on the positive sample and the negative sample to obtain the intermediate second feature extractor.
In some embodiments, the first determining module comprises:
the computing unit is used for carrying out similarity computation on the feature vector of the physical photo and the feature vector of the virtual article corresponding to the virtual article one by one to obtain the similarity of the feature vector of the physical photo and the feature vector of the virtual article;
And the screening unit is used for taking the first N virtual articles with the similarity larger than a similarity threshold value as a candidate set, wherein N is a positive integer.
In some embodiments, the acquisition module comprises:
the building unit is used for selecting a point in the real space as an origin and building a three-dimensional coordinate system xyz;
the device comprises an acquisition unit, an indoor positioning system and a three-dimensional coordinate system xyz, wherein the acquisition unit is used for acquiring the shooting position of the photographer through the indoor positioning system when the photographer shoots the physical photo, so as to obtain the shooting position of the photographer and the physical photo shot by the photographer at the shooting position, and the indoor positioning system takes the three-dimensional coordinate system xyz as a reference coordinate.
In some embodiments, the determining module comprises:
a judging unit, configured to map a shooting position of the photographer in a three-dimensional coordinate system xyz to a three-dimensional coordinate system x 'y' z ', and judge whether the virtual object in the candidate set and the shooting position of the photographer are in a same space, where the same space is a range defined by taking the virtual object as an origin, and the three-dimensional coordinate system x' y 'z' selects a point identical to a real object space in the virtual space as the origin, and is established according to a direction identical to the three-dimensional coordinate system xyz;
And the first determining unit is used for determining the virtual article as a target virtual article if the virtual article and the shooting position of the photographer are in the same space and the virtual article is one.
In some embodiments, the determining module comprises:
a comparison unit, configured to compare distances between the shooting positions of the photographers and the virtual articles one by one if the virtual articles are in the same space as the shooting positions of the photographers and the virtual articles are multiple;
and a second determination unit configured to determine, as a target virtual article, the virtual article closest to a shooting position of the photographer.
In some embodiments, the virtual article searching apparatus further comprises:
the second acquisition module is used for acquiring the two-dimensional view of the virtual article;
the second input module is used for inputting the two-dimensional view of the virtual article to the first feature extractor to extract the feature vector of the virtual article;
the association module is used for associating the feature vector of each virtual article with the number of the virtual article to form a feature dictionary;
the calculating the similarity between the feature vector of the physical photo and the feature vector of the virtual article corresponding to the virtual article, and determining the candidate set from the virtual article based on the similarity comprises:
And calculating the similarity between the feature vector of the physical photo and the feature vector corresponding to the number of the virtual object in the feature dictionary, and determining a candidate number set from the numbers of the virtual object based on the similarity so as to determine a candidate set based on the candidate number set.
It should be noted that, in the embodiment of the present application, if the above-mentioned method for searching for a virtual article is implemented in the form of a software functional module, and sold or used as an independent product, the method may also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be embodied in essence or a part contributing to the prior art in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read Only Memory (ROM), a magnetic disk, an optical disk, or other various media capable of storing program codes. Thus, embodiments of the application are not limited to any specific combination of hardware and software.
Accordingly, an embodiment of the present application provides a storage medium having stored thereon a computer program which, when executed by a processor, implements the steps in the method for searching for a virtual article provided in the above embodiment.
The embodiment of the application provides electronic equipment; fig. 10 is a schematic diagram of a composition structure of an electronic device according to an embodiment of the present application, as shown in fig. 10, the electronic device 100 includes: a processor 101, at least one communication bus 102, a user interface 103, at least one external communication interface 104, a memory 105. Wherein the communication bus 102 is configured to enable connected communication between these components. The user interface 103 may include a display screen, and the external communication interface 104 may include a standard wired interface and a wireless interface, among others. The processor 101 is configured to execute a program of a method for searching for a virtual article stored in a memory to implement the steps in the method for searching for a virtual article provided in the above-described embodiment.
It should be noted here that: the description of the storage medium and apparatus embodiments above is similar to that of the method embodiments described above, with similar benefits as the method embodiments. For technical details not disclosed in the embodiments of the storage medium and the apparatus of the present application, please refer to the description of the method embodiments of the present application.
It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in various embodiments of the present application, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present application. The foregoing embodiment numbers of the present application are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above described device embodiments are only illustrative, e.g. the division of the units is only one logical function division, and there may be other divisions in practice, such as: multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. In addition, the various components shown or discussed may be coupled or directly coupled or communicatively coupled to each other via some interface, whether indirectly coupled or communicatively coupled to devices or units, whether electrically, mechanically, or otherwise.
The units described above as separate components may or may not be physically separate, and components shown as units may or may not be physical units; can be located in one place or distributed to a plurality of network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may be separately used as one unit, or two or more units may be integrated in one unit; the integrated units may be implemented in hardware or in hardware plus software functional units.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the above method embodiments may be implemented by hardware related to program instructions, and the foregoing program may be stored in a computer readable storage medium, where the program, when executed, performs steps including the above method embodiments; and the aforementioned storage medium includes: a mobile storage device, a Read Only Memory (ROM), a magnetic disk or an optical disk, or the like, which can store program codes.
Alternatively, the above-described integrated units of the present application may be stored in a computer-readable storage medium if implemented in the form of software functional modules and sold or used as separate products. Based on such understanding, the technical solutions of the embodiments of the present application may be embodied essentially or in part in the form of a software product stored in a storage medium, including instructions for causing a controller to perform all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a removable storage device, a ROM, a magnetic disk, or an optical disk.
The foregoing is merely an embodiment of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (11)
1. A method for locating a virtual article, comprising:
acquiring a shooting position of a photographer and a physical picture shot by the photographer at the shooting position;
inputting the physical photo to a first feature extractor to obtain a feature vector of the physical photo;
calculating the similarity between the feature vector of the physical photo and the feature vector of the virtual article corresponding to the virtual article, and determining a candidate set from the virtual article based on the similarity, wherein the feature vector of the virtual article is input to a second feature extractor by adopting a two-dimensional view;
a target virtual item is determined from the candidate set based on the photographing position and the candidate set.
2. The method according to claim 1, wherein the method further comprises:
Constructing a training set, wherein the training set comprises: the sample physical photo set and the sample two-dimensional view set corresponding to the sample physical;
training an initial second feature extractor by adopting the sample two-dimensional view set to obtain an intermediate second feature extractor;
inputting the sample physical photo set to an initial first feature extractor to obtain a first sample feature vector, inputting a sample two-dimensional view set corresponding to the sample physical photo set to an intermediate second feature extractor to obtain a second sample feature vector, and training the intermediate second feature extractor and the initial first feature extractor by adopting an L2 norm based on the first sample feature vector and the second sample feature vector to obtain the first feature extractor and the second feature extractor.
3. The method of claim 2, wherein the sample two-dimensional view set comprises: training an initial second feature extractor by using the sample two-dimensional view set to obtain an intermediate second feature extractor, wherein the two-dimensional view of the virtual article from different view angles and view points comprises:
taking a two-dimensional view of the same virtual article from different perspectives and viewpoints as a positive sample of the initial second feature extractor training;
Taking a two-dimensional view of different perspectives and viewpoints of different virtual articles as a negative sample of the initial second feature extractor training;
training the initial second feature extractor using an InfoNCE loss function based on the positive and negative samples to obtain the intermediate second feature extractor.
4. The method of claim 1, wherein the calculating the similarity between the feature vector of the physical photograph and the virtual item feature vector corresponding to a virtual item, determining a candidate set from the virtual item based on the similarity, comprises:
carrying out similarity calculation on the feature vector of the physical photo and the feature vector of the virtual article corresponding to the virtual article one by one to obtain the similarity of the feature vector of the physical photo and the feature vector of the virtual article;
and taking the first N virtual articles with the similarity larger than a similarity threshold value as a candidate set, wherein N is a positive integer.
5. The method of claim 1, wherein the acquiring the photographing location of the photographer and the physical photograph taken by the photographer at the photographing location comprises:
selecting a point as an origin in a real object space, and establishing a three-dimensional coordinate system xyz;
When the photographer shoots a physical photo, the shooting position of the photographer is obtained through an indoor positioning system, and the shooting position of the photographer and the physical photo shot by the photographer at the shooting position are obtained, wherein the indoor positioning system takes the three-dimensional coordinate system xyz as a reference coordinate.
6. The method of claim 5, wherein the determining a target virtual item from the candidate set based on the photographing location and the candidate set comprises:
mapping the shooting position of the photographer in a three-dimensional coordinate system xyz to a three-dimensional coordinate system x 'y' z ', judging whether the virtual object in the candidate set and the shooting position of the photographer are in the same space, wherein the same space is a range defined by taking the virtual object as an origin, and the three-dimensional coordinate system x' y 'z' is established in the same direction of the three-dimensional coordinate system xyz by selecting a point which is the same as the real object space in the virtual space as the origin;
and if the virtual article and the shooting position of the photographer are in the same space, and the virtual article is one, determining that the virtual article is a target virtual article.
7. The method of claim 6, wherein the method further comprises:
if the virtual article and the shooting positions of the photographers are in the same space, and the virtual articles are a plurality of, comparing the distances between the shooting positions of the photographers and the virtual articles one by one;
and determining the virtual object closest to the shooting position of the photographer as a target virtual object.
8. The method according to claim 1, wherein the method further comprises:
acquiring a two-dimensional view of the virtual article;
inputting the two-dimensional view of the virtual article to a first feature extractor to extract feature vectors of the virtual article;
associating the feature vector of each virtual article with the number of the virtual article to form a feature dictionary;
the calculating the similarity between the feature vector of the physical photo and the feature vector of the virtual article corresponding to the virtual article, and determining the candidate set from the virtual article based on the similarity comprises:
and calculating the similarity between the feature vector of the physical photo and the feature vector corresponding to the number of the virtual object in the feature dictionary, and determining a candidate number set from the numbers of the virtual object based on the similarity so as to determine a candidate set based on the candidate number set.
9. A virtual article finding apparatus, comprising:
the shooting device comprises an acquisition module, a control module and a control module, wherein the acquisition module is used for acquiring a shooting position of a photographer and a physical picture shot by the photographer at the shooting position;
the input module is used for inputting the physical photo to the first feature extractor to obtain a feature vector of the physical photo;
the first determining module is used for calculating the similarity between the feature vector of the physical photo and the feature vector of the virtual article corresponding to the virtual article, and determining a candidate set from the virtual article based on the similarity, wherein the feature vector of the virtual article is input to the second feature extractor by adopting a two-dimensional view;
and a second determining module for determining a target virtual article from the candidate set based on the shooting position and the candidate set.
10. An electronic device, comprising: a memory and a processor, the memory having stored thereon a computer program which, when executed by the processor, performs the method of any of claims 1 to 8.
11. A storage medium storing a computer program executable by one or more processors for implementing a method as claimed in any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310913175.0A CN117173691A (en) | 2023-07-24 | 2023-07-24 | Virtual article searching method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310913175.0A CN117173691A (en) | 2023-07-24 | 2023-07-24 | Virtual article searching method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117173691A true CN117173691A (en) | 2023-12-05 |
Family
ID=88943858
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310913175.0A Pending CN117173691A (en) | 2023-07-24 | 2023-07-24 | Virtual article searching method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117173691A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107944020A (en) * | 2017-12-11 | 2018-04-20 | 深圳云天励飞技术有限公司 | Facial image lookup method and device, computer installation and storage medium |
| CN109978936A (en) * | 2019-03-28 | 2019-07-05 | 腾讯科技（深圳）有限公司 | Parallax picture capturing method, device, storage medium and equipment |
| CN110389703A (en) * | 2019-07-25 | 2019-10-29 | 腾讯数码（天津）有限公司 | Acquisition methods, device, terminal and the storage medium of virtual objects |
| CN110428449A (en) * | 2019-07-31 | 2019-11-08 | 腾讯科技（深圳）有限公司 | Target detection tracking method, device, equipment and storage medium |
| CN114820137A (en) * | 2022-05-25 | 2022-07-29 | 中国工商银行股份有限公司 | Product recommendation method and device, processor and electronic equipment |
| CN115203460A (en) * | 2022-07-05 | 2022-10-18 | 中山大学·深圳 | Deep learning-based pixel-level cross-view-angle image positioning method and system |
-
2023
- 2023-07-24 CN CN202310913175.0A patent/CN117173691A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107944020A (en) * | 2017-12-11 | 2018-04-20 | 深圳云天励飞技术有限公司 | Facial image lookup method and device, computer installation and storage medium |
| CN109978936A (en) * | 2019-03-28 | 2019-07-05 | 腾讯科技（深圳）有限公司 | Parallax picture capturing method, device, storage medium and equipment |
| CN110389703A (en) * | 2019-07-25 | 2019-10-29 | 腾讯数码（天津）有限公司 | Acquisition methods, device, terminal and the storage medium of virtual objects |
| CN110428449A (en) * | 2019-07-31 | 2019-11-08 | 腾讯科技（深圳）有限公司 | Target detection tracking method, device, equipment and storage medium |
| CN114820137A (en) * | 2022-05-25 | 2022-07-29 | 中国工商银行股份有限公司 | Product recommendation method and device, processor and electronic equipment |
| CN115203460A (en) * | 2022-07-05 | 2022-10-18 | 中山大学·深圳 | Deep learning-based pixel-level cross-view-angle image positioning method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110246163B (en) | Image processing method, image processing device, image processing apparatus, and computer storage medium | |
| KR101531618B1 (en) | Method and system for comparing images | |
| KR100651010B1 (en) | Image matching system using 3-dimensional object model, image matching method, and computer readable recording medium which records image matching program | |
| JP5848833B2 (en) | Method and system for comparing images | |
| CN111062871A (en) | Image processing method and device, computer equipment and readable storage medium | |
| US20150332117A1 (en) | Composition modeling for photo retrieval through geometric image segmentation | |
| CN110516707B (en) | Image labeling method and device and storage medium thereof | |
| US10796154B2 (en) | Method of image-based relationship analysis and system thereof | |
| JP2015504215A5 (en) | ||
| CN111291768A (en) | Image feature matching method and device, equipment and storage medium | |
| CN114694185B (en) | Cross-modal target re-identification method, device, equipment and medium | |
| CN112818995B (en) | Image classification method, device, electronic equipment and storage medium | |
| CN109815823B (en) | Data processing method and related product | |
| CN109902681A (en) | User group's relationship determines method, apparatus, equipment and storage medium | |
| CN111382791A (en) | Deep learning task processing method, image recognition task processing method and device | |
| CN113298871B (en) | Map generation method, positioning method, system thereof, and computer-readable storage medium | |
| CN106997366B (en) | Database construction method, augmented reality fusion tracking method and terminal equipment | |
| CN108875445B (en) | Pedestrian re-identification method and device | |
| US20150254527A1 (en) | Methods for 3d object recognition and registration | |
| CN110120090B (en) | Three-dimensional panoramic model construction method and device and readable storage medium | |
| CN117173691A (en) | Virtual article searching method and device, electronic equipment and storage medium | |
| JP6304815B2 (en) | Image processing apparatus and image feature detection method, program and apparatus thereof | |
| CN110414792A (en) | Component centralized procurement management system and Related product based on BIM and big data | |
| CN109961503A (en) | A kind of image processing method and device, terminal and computer readable storage medium | |
| CN115082999A (en) | Group photo image person analysis method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |