CN115906637A

CN115906637A - AR and AI-based outdoor photoelectric equipment regulation and control selection method, device and equipment

Info

Publication number: CN115906637A
Application number: CN202211476106.XA
Authority: CN
Inventors: 盛誉满; 黄顺辉; 林晨鸣; 唐聪
Original assignee: Fuhua Intelligent Shenzhen Co ltd
Current assignee: Fuhua Intelligent Shenzhen Co ltd
Priority date: 2022-11-23
Filing date: 2022-11-23
Publication date: 2023-04-04
Anticipated expiration: 2042-11-23
Also published as: CN115906637B

Abstract

The invention relates to an artificial intelligence technology, and discloses an outdoor photoelectric equipment regulation and selection method based on AR and AI, which comprises the following steps: the method comprises the steps of establishing an enhanced feature matching library based on regional image features and regional spatial position information of a target region, carrying out spatial modeling on target outdoor photoelectric equipment to obtain a photoelectric equipment model, carrying out spatial feature matching on a real-time video stream of the target region and the enhanced feature matching library to obtain an AR modeling space of the target region, leading the photoelectric equipment model into the AR modeling space, adjusting the photoelectric equipment model by using a photoelectric equipment adjusting instruction input by a user to obtain an adjusted photoelectric equipment model set, and selecting a model with the highest stability as standard photoelectric equipment according to an operation stability value and an equipment stability value of each model in the adjusted photoelectric equipment model set. The invention further provides an AR and AI based outdoor photoelectric equipment regulation and control selection device and equipment. The invention can improve the accuracy of regulation and control of the outdoor photoelectric equipment.

Description

AR and AI-based outdoor photoelectric equipment regulation and control selection method, device and equipment

Technical Field

The invention relates to the technical field of artificial intelligence, in particular to an outdoor photoelectric equipment regulation and control selection method and device based on AR and AI, electronic equipment and a computer readable storage medium.

Background

The outdoor photoelectric equipment is comprehensive equipment which is used in environments such as ground, ocean and air and consists of parts such as optics, mechanical structures, electronics (information processing), computer hardware and software (even artificial intelligence), a control system and the like. The basic function is to convert the received optical radiation into an electrical signal and use it for some practical purpose, such as measuring the photometric quantity, radiometric quantity or various apparent temperatures of the target; measuring the spatial distribution and the temperature distribution of target light radiation; the position or image of the three-dimensional space where the target is located is measured. Therefore, the stability of outdoor equipment during installation and operation is crucial, and the accuracy of equipment measurement can be seriously influenced, and in the prior art, the regulation and control of the photoelectric equipment are often based on manual experience and can be restricted by the outdoor environment, so that different equipment is difficult to accurately regulate and control.

Disclosure of Invention

The invention provides an AR and AI-based outdoor photoelectric equipment regulation selection method, an AR and AI-based outdoor photoelectric equipment regulation selection device, electronic equipment and a readable storage medium, and mainly aims to improve the accuracy of outdoor photoelectric equipment regulation.

In order to achieve the above object, the present invention provides an outdoor optoelectronic device regulation and selection method based on AR and AI, comprising:

acquiring a regional space image set and regional space position information of a target region;

extracting regional image features in the regional spatial image set, constructing a matching feature decision tree based on the regional image features, and constructing an enhanced feature matching library according to the matching feature decision tree and the regional spatial position information;

acquiring an equipment operation image set of target outdoor photoelectric equipment, and carrying out space modeling on the target outdoor photoelectric equipment based on the equipment operation image set to obtain a photoelectric equipment model;

acquiring a real-time video stream of the target area, performing spatial feature matching on the real-time video stream and an enhanced feature matching library based on a joint matching algorithm, obtaining an AR modeling space of the target area after matching is successful, and importing the photoelectric equipment model into the AR modeling space;

receiving a photoelectric equipment adjusting instruction input by a user, and adjusting a photoelectric equipment model in the AR modeling space by using the photoelectric equipment adjusting instruction to obtain an adjusted photoelectric equipment model set;

and calculating the operation stability value of each model in the adjusted photoelectric equipment model set by using an equipment operation identification model trained by the equipment operation image set and the historical outdoor photoelectric equipment operation image set, calculating the equipment stability value of each model according to the equipment inclination angle of each model in the adjusted photoelectric equipment model set, and selecting the model with the highest stability from the adjusted photoelectric equipment model set as the standard photoelectric equipment according to the operation stability value and the equipment stability value.

Optionally, the extracting the region image features in the region spatial image set, and constructing a matching feature decision tree based on the region image features includes:

traversing pixel points of the images in the regional space image set, taking the traversed pixel points as central pixel points, and summarizing the central pixel points and edge pixel points in a preset range near the central pixel points as pixel point subsets;

determining the edge state of edge pixels in the pixel point subset based on a preset state formula, and calculating split nodes based on the edge state;

taking the splitting node as a central pixel point, returning to the step of summarizing the central pixel point and the edge pixel point in the preset range near the central pixel point as a pixel point subset, and performing iterative splitting to obtain a pixel decision tree;

and calculating the loss value of the pixel decision tree by using a preset decision loss function, and stopping iteration to obtain the matching feature decision tree when the loss value is less than or equal to a preset loss threshold value.

Optionally, the constructing an enhanced feature matching library according to the matching feature decision tree and the region spatial location information includes:

performing k-means clustering on pixel points of each node in the matched feature decision tree to obtain a cluster set with a preset number, repeating the k-means clustering on the cluster set until the clustering frequency meets a preset clustering threshold to obtain a clustering result, calculating the weight of features in the clustering result based on a TF-IDF algorithm, and constructing a bag-of-words tree model based on the weight and the clustering result;

and summarizing the bag-of-words tree model, the matching feature decision tree and the region space position information to obtain the enhanced feature matching library.

Optionally, the performing spatial modeling on the target outdoor optoelectronic device based on the device operation image set to obtain an optoelectronic device model includes:

acquiring an original three-dimensional model and a replacement structure model of the target outdoor optical equipment, and constructing equipment operation coordinates based on the equipment operation image set by using a preset modeling engine;

acquiring vertex coordinates of the original three-dimensional model and the replacement structure model, and generating an equipment primitive by using the vertex coordinates and the equipment operation coordinates;

and performing rasterization processing and geometric transformation processing on the equipment primitive to obtain the photoelectric equipment model.

Optionally, the obtaining the real-time video stream of the target area, performing spatial feature matching on the real-time video stream and an enhanced feature matching library based on a joint matching algorithm, and obtaining an AR modeling space of the target area after matching is successful includes:

acquiring real-time position information and a real-time video stream of the target area by using a preset AR device;

positioning by utilizing the real-time position information and the area space position information;

when the positioning is successful, reading the real-time images in the real-time video stream frame by frame, extracting the image characteristics in the real-time images by using the matching characteristic decision tree, and constructing a query bag-of-words model based on the image characteristics;

performing feature matching on the image features in the query bag-of-words model and the image features in the bag-of-words tree model, performing initial feature screening on feature matching results based on a GMS algorithm to obtain original feature matching results, and performing outlier deletion on the original feature matching results based on a RANSAC algorithm to obtain standard feature matching results;

and constructing a coordinate space based on the AR equipment, and performing pose registration on the standard feature matching result in the space coordinate by utilizing an ICP (inductively coupled plasma) algorithm to obtain the AR modeling space.

Optionally, the adjusting the model of the optoelectronic device in the AR modeling space by using the optoelectronic device adjusting instruction to obtain an adjusted model set of the optoelectronic device, calculating an operation stability value of each model in the model set of the optoelectronic device by using a device operation recognition model trained by the device operation image set and a historical outdoor optoelectronic device operation image set, calculating a device stability value of each model according to a device inclination angle of each model in the model set of the optoelectronic device, and selecting a model with the highest stability as a standard optoelectronic device according to the operation stability value and the device stability value includes:

analyzing a device replacement instruction and a device adjustment instruction in the photoelectric device adjustment instruction;

randomly combining the equipment replacement instruction and the equipment adjustment instruction to obtain a plurality of random combination instructions, and performing equipment adjustment on the photoelectric equipment model by using the plurality of random combination instructions to obtain an adjusted photoelectric equipment model set;

outputting a predicted operation label of each model in the adjusted photoelectric equipment model set in operation by using the equipment operation identification model, and taking an operation quantization value corresponding to the predicted operation label as an operation stability value;

calculating the equipment inclination angles of each model and a target area in the adjusted photoelectric equipment model set, and determining an inclination angle quantized value corresponding to the equipment inclination angle as an equipment stability value;

and performing weighted calculation on the operation stability value and the equipment stability value to obtain a weighted stability value, and determining the model with the maximum weighted stability value in the adjusted photoelectric equipment model set as the standard photoelectric equipment.

Optionally, the split node is calculated by the following formula:

H(g)＝H(P)-H(d)-H(s)-H(b)

h (g) represents the information entropy of the split node, H (P) represents the information entropy of the central pixel point, H (d) represents the information entropy of the edge pixel point with the darker edge state, H(s) represents the information entropy of the edge pixel point with the similar edge state, and H (b) represents the information entropy of the edge pixel point with the brighter edge state.

Optionally, the loss value of the pixel decision tree is calculated by using the following decision loss function:

wherein L represents a loss value, r represents a repetition rate of pixel points of an image in the regional space image set, and N represents an image in the regional space image setNumber of (d) _i The number of pixel points of the ith regional space image is represented, s represents the number of the node points in the pixel decision tree, and w _r 、w _n And w _s Is a preset fixed parameter.

In order to solve the above problems, the present invention further provides an outdoor optoelectronic device adjusting and selecting apparatus based on AR and AI, the apparatus comprising:

the feature matching library construction module is used for acquiring a region space image set and region space position information of a target region, extracting region image features in the region space image set, constructing a matching feature decision tree based on the region image features, and constructing an enhanced feature matching library according to the matching feature decision tree and the region space position information;

the device modeling module is used for acquiring a device operation image set of target outdoor photoelectric equipment, and carrying out space modeling on the target outdoor photoelectric equipment based on the device operation image set to obtain a photoelectric equipment model;

the space modeling module is used for acquiring a real-time video stream of the target area, performing space feature matching on the real-time video stream and an enhanced feature matching library based on a joint matching algorithm, obtaining an AR modeling space of the target area after matching is successful, and importing the photoelectric equipment model into the AR modeling space;

the device adjusting module is used for receiving a photoelectric device adjusting instruction input by a user, and adjusting the photoelectric device model in the AR modeling space by using the photoelectric device adjusting instruction to obtain an adjusted photoelectric device model set;

and the equipment selection module is used for calculating the operation stability value of each model in the adjustment photoelectric equipment model set by using an equipment operation recognition model trained by the equipment operation image set and the historical outdoor photoelectric equipment operation image set, calculating the equipment stability value of each model according to the equipment inclination angle of each model in the adjustment photoelectric equipment model set, and selecting the model with the highest stability from the adjustment photoelectric equipment model set as the standard photoelectric equipment according to the operation stability value and the equipment stability value.

In order to solve the above problem, the present invention also provides an electronic device, including:

a memory storing at least one computer program; and

and the processor is used for executing the computer program stored in the memory so as to realize the regulation and selection method of the outdoor photoelectric equipment based on the AR and the AI.

In the embodiment, the regional image features in the regional spatial image set of the target region are extracted, the matching feature decision tree is constructed based on the regional image features, the enhanced feature matching library is constructed according to the matching feature decision tree and the regional spatial position information, when AR spatial modeling is performed, the image features can be rapidly extracted through the matching feature decision tree, and feature matching is performed on the real-time video stream and the enhanced feature matching library based on a joint matching algorithm, so that the influence of the outdoor environment on the AR modeling can be reduced, and the modeling accuracy is improved. Meanwhile, space modeling is carried out on target outdoor photoelectric equipment based on an equipment operation image set to obtain a photoelectric equipment model, the photoelectric equipment model in an AR modeling space is adjusted by using a photoelectric equipment adjusting instruction, individualized adjustment can be carried out on the equipment to obtain an adjusted photoelectric equipment model set, the operation stability value of each model is calculated through an equipment operation identification model, and the equipment stability value of each model is calculated through an equipment inclination angle. Therefore, the method, the device, the electronic equipment and the computer readable storage medium for controlling and selecting the outdoor photoelectric equipment based on the AR and the AI can improve the accuracy of controlling the outdoor photoelectric equipment.

Drawings

Fig. 1 is a schematic flow chart of an AR and AI based outdoor optoelectronic device regulation and selection method according to an embodiment of the present invention;

fig. 2 is a functional block diagram of an outdoor optoelectronic device regulation and selection apparatus based on AR and AI according to an embodiment of the present invention;

fig. 3 is a schematic structural diagram of an electronic device for implementing the method for regulating and selecting an outdoor optoelectronic device based on AR and AI according to an embodiment of the present invention.

The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.

Detailed Description

It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the invention.

The embodiment of the application provides an outdoor photoelectric equipment regulation and control selecting method based on AR and AI. The execution subject of the outdoor photoelectric device regulation and selection method based on the AR and the AI includes, but is not limited to, at least one of electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the method for adjusting and selecting the outdoor optoelectronic device based on the AR and the AI may be executed by software or hardware installed in the terminal device or the server device, where the software may be a block chain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like. The server may be an independent server, or may be a cloud server that provides basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, a middleware service, a domain name service, a security service, a Content Delivery Network (CDN), a big data and artificial intelligence platform, and the like.

Fig. 1 is a schematic flow chart of an AR and AI-based outdoor optoelectronic device regulation and selection method according to an embodiment of the present invention. In this embodiment, the method for regulating and selecting outdoor optoelectronic devices based on AR and AI includes:

s1, acquiring a region space image set and region space position information of a target region.

In the embodiment of the invention, the target area is an area to be subjected to virtual reality augmentation, namely, a virtual object is drawn on the target area, and visualization operation is performed, for example, a virtual model of outdoor distance measurement photoelectric equipment is constructed on a street (target area). The regional spatial image set refers to images of different view angles taken at the target region, such as images taken by a ZED camera. The area spatial position information refers to spatial position information of a target area measured by a GPS position system.

S2, extracting the regional image features in the regional spatial image set, constructing a matching feature decision tree based on the regional image features, and constructing an enhanced feature matching library according to the matching feature decision tree and the regional spatial position information.

In the embodiment of the present invention, the matching feature decision tree may be an ID3 decision tree. The enhanced feature matching library is a prior large-scene enhanced target database, the database comprises spatial position information obtained through geographic positions, feature points of images and a bag-of-words model, targets which are possibly present around the current position are screened mainly by converting local map coordinates with GPS signals, and proper bag-of-words model data and proper matching feature decision trees are selected for positioning, so that spatial matching with a target area is realized.

In detail, the extracting the region image features in the region space image set and constructing a matching feature decision tree based on the region image features includes:

In an optional embodiment of the present invention, for example, a radius r (which may be 3) is set by traversing pixel points p of an image in a regional spatial image set, 16 pixel points on a circle with the radius r as a center pixel point p are determined as edge pixel points, and the edge pixel points are divided into three states by comparing pixel grayscales of the edge pixel points and the center pixel points: and (d) making the nodes darker, similar and brighter, and (b) selecting the position with the largest information gain to split until the loss value of the split node meets a preset loss threshold value to obtain a matching feature decision tree.

In detail, the preset state formula is as follows:

wherein S is _x Express the state of marginal pixel point x, baker expresses that the marginal state is darker, and simila expresses that the marginal state is similar, and broighter expresses that the marginal state is brighter, I _p Expressing the pixel gray level, I, of the central pixel point p _x And expressing the pixel gray of the edge pixel point x, and t expressing a preset gray parameter.

In an optional embodiment of the present invention, the split node is calculated by the following formula:

H(g)＝H(P)-H(d)-H(s)-H(b)

In an optional embodiment of the invention, the accuracy and the speed of feature matching can be improved by calculating the information entropy and splitting from the position with the maximum information entropy, wherein the node selected by each splitting is the position with the maximum information quantity.

Specifically, in the embodiment of the present invention, the loss value of the pixel decision tree is calculated by using the following decision loss function:

wherein L represents a loss value, r represents a repetition rate of pixel points of images in the regional space image set, N represents the number of images in the regional space image set, and d _i Expressing the number of pixel points of the ith regional space image, s expressing the number of the nodes in the pixel decision tree, w _r 、w _n And w _s Is preset fixed parameters.

Further, the constructing an enhanced feature matching library according to the matching feature decision tree and the region spatial location information includes:

performing k-means clustering on pixel points of each node in the matched feature decision tree to obtain a cluster set with a preset number, repeating the k-means clustering on the cluster set until the clustering frequency meets a preset clustering threshold to obtain a clustering result, calculating the weight of features in the clustering result based on a TF-IDF algorithm, and constructing a bag-of-word tree model based on the weight and the clustering result;

In an optional embodiment of the invention, the bag-of-words tree model is a key technology for AR loop detection, and features of an image are used as visual words (words) to describe the image and perform large-scale image retrieval. For example, clustering nodes in the matching feature decision tree into k cluster sets, then performing k-means clustering on each cluster set to generate k sets, repeating the process for V times, and finally generating k sets ^V And finally, calculating the weight of the leaf node through a TF-IDF algorithm to obtain the bag-of-words tree model.

And S3, acquiring an equipment operation image set of the target outdoor photoelectric equipment, and performing spatial modeling on the target outdoor photoelectric equipment based on the equipment operation image set to obtain a photoelectric equipment model.

In the embodiment of the invention, the target outdoor photoelectric equipment refers to photoelectric equipment to be installed in a target area, and the equipment running image set comprises images of the target outdoor photoelectric equipment with different visual angles in running.

Specifically, the performing spatial modeling on the target outdoor optoelectronic device based on the device operation image set to obtain an optoelectronic device model includes:

acquiring an original three-dimensional model and a replacement structure model of the target outdoor optical equipment, and constructing an equipment operation coordinate based on the equipment operation image set by using a preset modeling engine;

In an optional embodiment of the invention, an original three-dimensional model and a replacement structure model (such as a bracket, a base and the like) of the target outdoor optical equipment can be established by using modeling software such as SolidWorks, CAD and the like, and the preset modeling engine can be virtual reality software 3D-quest and the like. Meanwhile, the vertex coordinates are converted into primitives (namely triangles) through a vertex shader (written by opengles, defined by java in a character string form and transmitted), so that the three-dimensional world coordinates are converted into screen coordinates, after the primitives are generated, the model is colored, the texture (color and the like), light and the like of the model are changed through the vertex shader, and finally, a level of Detail (LOD) model is adopted for geometric conversion to obtain the photoelectric equipment model.

S4, acquiring a real-time video stream of the target area, carrying out spatial feature matching on the real-time video stream and an enhanced feature matching library based on a joint matching algorithm, obtaining an AR modeling space of the target area after matching is successful, and importing the photoelectric equipment model into the AR modeling space.

In the embodiment of the invention, the joint matching algorithm comprises a RANSAC (Random Sample Consensus) algorithm and a GMS (Grid-based Motion Statistics) algorithm, wherein the GMS algorithm is used for carrying out mismatching deletion on the features, namely, the GMS algorithm is used as a prefilter for carrying out feature screening and matching, the RANSAC algorithm is used for matching the screened features, and the speed of feature matching can be improved through the joint matching algorithm.

In detail, the obtaining of the real-time video stream of the target area, performing spatial feature matching on the real-time video stream and an enhanced feature matching library based on a joint matching algorithm, and obtaining an AR modeling space of the target area after matching is successful includes:

In an optional embodiment of the invention, the current real-time position information and the regional spatial position information of the target region are used for positioning, when the positioning is determined to be near the target region, the trained matching feature decision tree is used for feature extraction, the feature extraction rate is improved, meanwhile, feature matching is performed based on the bag-of-words model, for the matched features, the GMS algorithm is firstly used for mismatching deletion, and then the RANSAC algorithm is used for deleting irrelevant outliers, so that the most relevant standard feature matching result (namely the most relevant features in the regional spatial image set of the target region and the real-time video stream) is obtained. And finally, performing feature pose fusion by using ICP (Iterative Closest Point), so as to obtain a more accurate AR modeling space.

In an optional embodiment of the present invention, the GMS algorithm, the RANSAC algorithm, and the ICP algorithm are well known in the art and are not described herein again.

In the embodiment of the invention, the image features can be extracted more quickly by matching the feature decision tree, and the most relevant image features can be obtained based on the bag-of-words model and the joint matching algorithm, so that the inaccuracy of feature matching when the environment texture is less in the outdoor environment is avoided, and the accuracy of AR modeling is improved.

And S5, receiving a photoelectric equipment adjusting instruction input by a user, and adjusting the photoelectric equipment model in the AR modeling space by using the photoelectric equipment adjusting instruction to obtain an adjusted photoelectric equipment model set.

In the embodiment of the present invention, the optoelectronic device adjusting instruction refers to an instruction for adjusting and replacing an optoelectronic device structure, for example, adjusting a height of a bracket, and replacing brackets (hydraulic type and mechanical type) of different types.

In detail, the adjusting the optoelectronic device model in the AR modeling space by using the optoelectronic device adjusting instruction to obtain an adjusted optoelectronic device model set includes:

and randomly combining the equipment replacement instruction and the equipment adjustment instruction to obtain a plurality of random combination instructions, and performing equipment adjustment on the photoelectric equipment model by using the plurality of random combination instructions to obtain the adjusted photoelectric equipment model set.

In an optional embodiment of the present invention, the optoelectronic device adjustment instruction includes a device replacement instruction and a device adjustment instruction, where the device replacement instruction is used to replace different structural models, and the device adjustment instruction is used to adjust each structural parameter in the model, for example, the optoelectronic device adjustment instruction includes two different replacement scaffolds, and three model scaffold parameters, so that six adjustment optoelectronic device models are generated.

S6, calculating the operation stability value of each model in the adjustment photoelectric equipment model set by using an equipment operation recognition model trained by the equipment operation image set and the historical outdoor photoelectric equipment operation image set, calculating the equipment stability value of each model according to the equipment inclination angle of each model in the adjustment photoelectric equipment model set, and selecting the model with the highest stability from the adjustment photoelectric equipment model set as the standard photoelectric equipment according to the operation stability value and the equipment stability value.

The historical outdoor photoelectric equipment operation image set refers to videos and image sets of outdoor photoelectric equipment which is shot manually and operates in different environments, and is marked according to operation stability, such as stable, stable and unstable. The equipment operation recognition model can be a deep learning model such as a 2DCNN +LSTM model and a 3DCNN deep learning model, for example, video image features are extracted frame by frame through 2DCNN, then the LSTM model is input for prediction, and after training is completed, the equipment operation recognition model outputs a prediction operation label.

In detail, the calculating an operation stability value of each model in the adjusted photovoltaic device model set by using the device operation recognition model trained by the device operation image set and the historical outdoor photovoltaic device operation image set, calculating a device stability value of each model according to a device tilt angle of each model in the adjusted photovoltaic device model set, and selecting a model with the highest stability from the adjusted photovoltaic device model set as a standard photovoltaic device according to the operation stability value and the device stability value includes:

In an optional embodiment of the present invention, for example, six types of predicted operation labels of the adjusted optoelectronic device model during operation are output through the device operation identification model, a specific operation quantization value quantized by each predicted operation label (for example, "stable" corresponds to 100, "more stable" corresponds to 80, "unstable" corresponds to 50, and the like) is obtained as an operation stability value, then a device tilt angle (a bracket and ground tilt angle, an optoelectronic device reference plane and ground tilt angle, and the like) of each model is calculated, similarly, a tilt angle quantization value is set for different device tilt angles as a device stability value, a weighted stability value is calculated by setting weights α and β (α + β = 1), and finally, the most stable optoelectronic device is selected, and meanwhile, regulation and control of outdoor optoelectronic devices can be shared based on AR devices, so that accuracy of device regulation and control is further improved.

In the embodiment, the image features of the region in the region space image set of the target region are extracted, the matching feature decision tree is constructed based on the image features of the region, the enhanced feature matching library is constructed based on the matching feature decision tree and the region space position information, when AR space modeling is carried out, the image features can be rapidly extracted through the matching feature decision tree, feature matching is carried out on the real-time video stream and the enhanced feature matching library based on a joint matching algorithm, the influence of the outdoor environment on the AR modeling can be reduced, and the modeling accuracy is improved. Meanwhile, space modeling is carried out on target outdoor photoelectric equipment based on the equipment operation image set to obtain a photoelectric equipment model, the photoelectric equipment model in the AR modeling space is adjusted by using a photoelectric equipment adjusting instruction, individualized adjustment can be carried out on the equipment to obtain an adjusted photoelectric equipment model set, the operation stability value of each model is calculated through the equipment operation identification model, and the equipment stability value of each model is calculated through the equipment inclination angle. Therefore, the method for regulating and selecting the outdoor photoelectric equipment based on the AR and the AI can improve the accuracy of regulating and controlling the outdoor photoelectric equipment.

Fig. 2 is a functional block diagram of an outdoor optoelectronic device regulation and selection apparatus based on AR and AI according to an embodiment of the present invention.

The outdoor photoelectric device regulating and selecting device 100 based on AR and AI can be installed in electronic equipment. According to the realized functions, the outdoor photoelectric device regulation and selection device 100 based on AR and AI may include a feature matching library construction module 101, a device modeling module 102, a space modeling module 103, a device adjustment module 104, and a device selection module 105. The module of the present invention, which may also be referred to as a unit, refers to a series of computer program segments that can be executed by a processor of an electronic device and that can perform a fixed function, and that are stored in a memory of the electronic device.

In the present embodiment, the functions regarding the respective modules/units are as follows:

the feature matching library construction module 101 is configured to obtain a region space image set and region space position information of a target region, extract region image features in the region space image set, construct a matching feature decision tree based on the region image features, and construct an enhanced feature matching library according to the matching feature decision tree and the region space position information;

the device modeling module 102 is configured to obtain a device operation image set of a target outdoor photoelectric device, and perform spatial modeling on the target outdoor photoelectric device based on the device operation image set to obtain a photoelectric device model;

the spatial modeling module 103 is configured to obtain a real-time video stream of the target area, perform spatial feature matching on the real-time video stream and an enhanced feature matching library based on a joint matching algorithm, obtain an AR modeling space of the target area after matching is successful, and import the optoelectronic device model into the AR modeling space;

the device adjusting module 104 is configured to receive a photoelectric device adjusting instruction input by a user, and adjust a photoelectric device model in the AR modeling space by using the photoelectric device adjusting instruction to obtain an adjusted photoelectric device model set;

the device selection module 105 is configured to calculate an operation stability value of each model in the adjusted optoelectronic device model set by using a device operation recognition model trained by the device operation image set and the historical outdoor optoelectronic device operation image set, calculate a device stability value of each model according to a device tilt angle of each model in the adjusted optoelectronic device model set, and select a model with the highest stability from the adjusted optoelectronic device model set as a standard optoelectronic device according to the operation stability value and the device stability value.

In detail, the specific implementation manner of each module of the outdoor photoelectric device control and selection device 100 based on AR and AI is as follows:

step one, acquiring a regional space image set and regional space position information of a target region.

And step two, extracting the regional image features in the regional spatial image set, constructing a matching feature decision tree based on the regional image features, and constructing an enhanced feature matching library according to the matching feature decision tree and the regional spatial position information.

In detail, the preset state formula is as follows:

wherein S is _x Representing the state of an edge pixel point x, the marker representing the edge state as darker, the simila representing the edge state as similar, the broighter representing the edge state as brighter, and I _p Expressing the pixel gray level of the center pixel p, I _x And expressing the pixel gray of the edge pixel point x, and t expressing a preset gray parameter.

H(g)＝H(P)-H(d)-H(s)-H(b)

wherein L represents a loss value, r represents a repetition rate of pixel points of images in the regional space image set, N represents the number of images in the regional space image set, and d _i Expressing the number of pixel points of the ith regional space image, s expressing the number of the nodes in the pixel decision tree, w _r 、w _n And w _s Is a preset fixed parameter.

In an optional embodiment of the invention, the bag-of-words tree model is a key technology for AR loop detection, and features of an image are used as visual words (words) to describe the image and perform large-scale image retrieval. For example, clustering nodes in the matching feature decision tree into k cluster sets, then performing k-means clustering on each cluster set to generate k sets, repeating the process for V times, and finally generating k sets ^V And finally, calculating the weight of the leaf node through a TF-IDF algorithm to obtain the bag-of-word tree model.

And thirdly, acquiring an equipment operation image set of the target outdoor photoelectric equipment, and carrying out space modeling on the target outdoor photoelectric equipment based on the equipment operation image set to obtain a photoelectric equipment model.

and carrying out rasterization processing and geometric transformation processing on the equipment primitive to obtain the photoelectric equipment model.

In an optional embodiment of the invention, an original three-dimensional model and a replacement structure model (such as a bracket, a base and the like) of the target outdoor optical equipment can be established by using modeling software such as SolidWorks, CAD and the like, and the preset modeling engine can be virtual reality software 3D-quest and the like. Meanwhile, converting the vertex coordinates into primitives (namely triangles) through a vertex shader (written by opengles and defined and transmitted by java in the form of character strings), converting the three-dimensional world coordinates into screen coordinates, coloring the model after the generation of the primitives is finished, changing the texture (color and the like), light and the like of the model through the fragment shader, and finally performing geometric conversion by adopting a LOD (level of Detail) model to obtain the photoelectric equipment model.

And step four, acquiring a real-time video stream of the target area, carrying out spatial feature matching on the real-time video stream and an enhanced feature matching library based on a joint matching algorithm, obtaining an AR (augmented reality) modeling space of the target area after successful matching, and importing the photoelectric equipment model into the AR modeling space.

In the embodiment of the invention, the joint matching algorithm comprises a Random Sample Consensus (RANSAC) algorithm and a Grid-based Motion Statistics (GMS) algorithm, wherein the GMS algorithm is used for carrying out mismatching deletion on the features, namely, the GMS algorithm is used as a pre-filter for carrying out feature screening and matching, the RANSAC algorithm is used for matching the screened features, and the speed of feature matching can be improved through the joint matching algorithm.

In an optional embodiment of the invention, the current real-time position information and the regional spatial position information of the target region are used for positioning, when the current real-time position information and the regional spatial position information of the target region are determined to be positioned near the target region, the trained matching feature decision tree is used for feature extraction, the feature extraction rate is improved, meanwhile, feature matching is carried out based on the bag-of-words model, for the matched features, a GMS algorithm is firstly adopted for carrying out mismatching deletion, and then an RANSAC algorithm is used for deleting irrelevant outliers, so that the most relevant standard feature matching result (namely the most relevant features in the regional spatial image set of the target region and the real-time video stream) is obtained. And finally, performing characteristic pose fusion by using ICP (Iterative Closest Point) to obtain a more accurate AR modeling space.

In an optional embodiment of the present invention, the GMS algorithm, the RANSAC algorithm, and the ICP algorithm are well known in the art, and are not described herein again.

And fifthly, receiving a photoelectric equipment adjusting instruction input by a user, and adjusting the photoelectric equipment model in the AR modeling space by using the photoelectric equipment adjusting instruction to obtain an adjusted photoelectric equipment model set.

In an optional embodiment of the present invention, the optoelectronic device adjustment instruction includes a device replacement instruction and a device adjustment instruction, where the device replacement instruction is used to replace different structural models, and the device adjustment instruction is used to adjust each structural parameter in the model, for example, the optoelectronic device adjustment instruction includes two different replacement supports, and three model support parameters generate six adjustment optoelectronic device models.

And sixthly, calculating the operation stability value of each model in the adjustment photoelectric equipment model set by using an equipment operation recognition model trained by the equipment operation image set and the historical outdoor photoelectric equipment operation image set, calculating the equipment stability value of each model according to the equipment inclination angle of each model in the adjustment photoelectric equipment model set, and selecting the model with the highest stability from the adjustment photoelectric equipment model set as the standard photoelectric equipment according to the operation stability value and the equipment stability value.

The historical outdoor photoelectric equipment operation image set refers to videos and image sets of outdoor photoelectric equipment which is shot manually and operates in different environments, and the videos and the image sets are labeled according to operation stability, such as stable, stable and unstable. The equipment operation recognition model can be a deep learning model such as a 2DCNN + LSTM model and a 3DCNN model, for example, video image features are extracted frame by frame through 2DCNN, then the LSTM model is input for prediction, and after training is completed, the equipment operation recognition model outputs a prediction operation label.

In an optional embodiment of the present invention, for example, six types of predicted operation labels of an adjusted optoelectronic device model during operation are output through a device operation identification model, a specific operation quantized value quantized for each predicted operation label (for example, "stable" corresponds to 100, "more stable" corresponds to 80, "unstable" corresponds to 50, and the like) is obtained as an operation stability value, then a device tilt angle (a bracket and ground tilt angle, an optoelectronic device reference plane and ground tilt angle, and the like) of each model is calculated, similarly, a tilt angle quantized value is set for different device tilt angles as a device stability value, a weighted stability value representing stability is calculated by setting weights α and β (α + β = 1), and finally, the most stable optoelectronic device is selected, and meanwhile, regulation and control of outdoor optoelectronic devices can be shared based on AR devices, and accuracy of device regulation and control is further improved.

In the embodiment, the regional image features in the regional spatial image set of the target region are extracted, the matching feature decision tree is constructed based on the regional image features, the enhanced feature matching library is constructed according to the matching feature decision tree and the regional spatial position information, when AR spatial modeling is performed, the image features can be rapidly extracted through the matching feature decision tree, and feature matching is performed on the real-time video stream and the enhanced feature matching library based on a joint matching algorithm, so that the influence of the outdoor environment on the AR modeling can be reduced, and the modeling accuracy is improved. Meanwhile, space modeling is carried out on target outdoor photoelectric equipment based on the equipment operation image set to obtain a photoelectric equipment model, the photoelectric equipment model in the AR modeling space is adjusted by using a photoelectric equipment adjusting instruction, individualized adjustment can be carried out on the equipment to obtain an adjusted photoelectric equipment model set, the operation stability value of each model is calculated through the equipment operation identification model, and the equipment stability value of each model is calculated through the equipment inclination angle. Therefore, the AR and AI based outdoor photoelectric equipment regulation and control selection device provided by the invention can improve the accuracy of outdoor photoelectric equipment regulation and control.

The electronic device may include a processor 10, a memory 11, a communication interface 12 and a bus 13, and may further include a computer program stored in the memory 11 and executable on the processor 10, such as an outdoor optoelectronic device regulation selection program based on AR and AI.

The memory 11 includes at least one type of readable storage medium, which includes flash memory, removable hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 11 may in some embodiments be an internal storage unit of the electronic device, for example a removable hard disk of the electronic device. The memory 11 may also be an external storage device of the electronic device in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the electronic device. Further, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device. The memory 11 may be used to store not only application software installed in the electronic device and various data, such as codes of an outdoor optoelectronic device regulation selection program based on AR and AI, but also temporarily store data that has been output or will be output.

The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The processor 10 is a Control Unit of the electronic device, and is connected to each component of the whole electronic device by using various interfaces and lines, and executes various functions and processes data of the electronic device by running or executing programs or modules (for example, an outdoor optoelectronic device Control selection program based on AR and AI, etc.) stored in the memory 11 and calling data stored in the memory 11.

The communication interface 12 is used for communication between the electronic device and other devices, and includes a network interface and a user interface. Optionally, the network interface may include a wired interface and/or a wireless interface (e.g., WI-FI interface, bluetooth interface, etc.), which are typically used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a Display (Display), an input unit such as a Keyboard (Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable, among other things, for displaying information processed in the electronic device and for displaying a visualized user interface.

The bus 13 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus 13 may be divided into an address bus, a data bus, a control bus, etc. The bus 13 is arranged to enable connection communication between the memory 11 and at least one processor 10 or the like.

Fig. 3 shows only an electronic device having components, and those skilled in the art will appreciate that the structure shown in fig. 3 does not constitute a limitation of the electronic device, and may include fewer or more components than those shown, or some components may be combined, or a different arrangement of components.

For example, although not shown, the electronic device may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor 10 through a power management device, so that functions of charge management, discharge management, power consumption management and the like are realized through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.

Further, the electronic device may further include a network interface, and optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a bluetooth interface, etc.), which are generally used to establish a communication connection between the electronic device and other electronic devices.

Optionally, the electronic device may further comprise a user interface, which may be a Display (Display), an input unit (such as a Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable, among other things, for displaying information processed in the electronic device and for displaying a visualized user interface.

It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.

The AR and AI based outdoor optoelectronic device regulation and selection program stored in the memory 11 of the electronic device is a combination of a plurality of instructions, and when running in the processor 10, it can implement:

acquiring a region space image set and region space position information of a target region;

acquiring an equipment operation image set of target outdoor photoelectric equipment, and carrying out spatial modeling on the target outdoor photoelectric equipment based on the equipment operation image set to obtain a photoelectric equipment model;

and calculating the operation stability value of each model in the adjustment photoelectric equipment model set by using an equipment operation identification model trained by the equipment operation image set and the historical outdoor photoelectric equipment operation image set, calculating the equipment stability value of each model according to the equipment inclination angle of each model in the adjustment photoelectric equipment model set, and selecting the model with the highest stability from the adjustment photoelectric equipment model set as the standard photoelectric equipment according to the operation stability value and the equipment stability value.

Specifically, the specific implementation method of the instruction by the processor 10 may refer to the description of the relevant steps in the embodiment corresponding to the drawings, which is not described herein again.

Further, the electronic device integrated module/unit, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in a computer readable storage medium. The computer readable storage medium may be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, read-Only Memory (ROM).

The present invention also provides a computer-readable storage medium, storing a computer program which, when executed by a processor of an electronic device, may implement:

In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.

The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.

In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.

It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.

The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.

The embodiment of the application can acquire and process related data based on an artificial intelligence technology. Among them, artificial Intelligence (AI) is a theory, method, technique and application system that simulates, extends and expands human Intelligence using a digital computer or a machine controlled by a digital computer, senses the environment, acquires knowledge and uses the knowledge to obtain the best result.

The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not any particular order.

Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. An outdoor photoelectric device regulation and selection method based on AR and AI is characterized by comprising the following steps:

acquiring a real-time video stream of the target area, performing spatial feature matching on the real-time video stream and an enhanced feature matching library based on a joint matching algorithm, obtaining an AR (augmented reality) modeling space of the target area after the spatial feature matching is successful, and importing the photoelectric equipment model into the AR modeling space;

2. The method for regulation and control selection of outdoor optoelectronic devices based on AR and AI as claimed in claim 1, wherein said extracting regional image features in said regional spatial image set and constructing a matching feature decision tree based on said regional image features comprises:

and calculating the loss value of the pixel decision tree by using a preset decision loss function, and stopping iteration when the loss value is less than or equal to a preset loss threshold value to obtain the matching feature decision tree.

3. The method of claim 1, wherein the constructing an enhanced feature matching library according to the matching feature decision tree and the regional spatial location information comprises:

4. The AR and AI based outdoor optoelectronic device regulation selection method of claim 1, wherein the spatially modeling the target outdoor optoelectronic device based on the device operation image set to obtain an optoelectronic device model comprises:

5. The method for regulation and control selection of outdoor optoelectronic devices based on AR and AI according to claim 1, wherein the obtaining of the real-time video stream of the target area, the spatial feature matching of the real-time video stream and the enhanced feature matching library based on a joint matching algorithm, and the obtaining of the AR modeling space of the target area after successful matching comprises:

6. The method as claimed in claim 1, wherein the calculating the operation stability value of each model in the adjusted model set of photovoltaic devices by using the device operation recognition model trained by the device operation image set and the historical outdoor photovoltaic device operation image set, calculating the device stability value of each model according to the device tilt angle of each model in the adjusted model set of photovoltaic devices, and selecting the model with the highest stability from the adjusted model set of photovoltaic devices as the standard photovoltaic device according to the operation stability value and the device stability value comprises:

7. The AR and AI based outdoor optoelectronic device regulation selection method of claim 2, wherein the split node is calculated by the following formula:

H(g)＝H(P)-H(d)-H(s)-H(b)

8. The AR and AI based outdoor optoelectronic device regulatory selection method of claim 2, wherein the loss values of the pixel decision tree are calculated using the following decision loss function:

/>

9. An outdoor photoelectric device regulation and control selecting device based on AR and AI, characterized in that the device includes:

the feature matching library construction module is used for acquiring a regional space image set and regional space position information of a target region, extracting regional image features in the regional space image set, constructing a matching feature decision tree based on the regional image features, and constructing an enhanced feature matching library according to the matching feature decision tree and the regional space position information;

10. An electronic device, characterized in that the electronic device comprises:

at least one processor; and the number of the first and second groups,

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the AR and AI based outdoor photovoltaic device regulation election method of any one of claims 1 to 7.