CN117939262A - Underwater image acquisition system and method based on polarized light field - Google Patents

Abstract

The invention discloses an underwater image acquisition system and method based on a polarized light field, relates to the field of computer vision and optical engineering, and is difficult to obtain clear underwater images at present. The invention comprises an optical lens assembly, a polarization assembly, a mechanical connection assembly for connecting the optical lens and the polarization assembly, and an image processor for registering and fusing images to obtain clear underwater images; the optical lens assembly comprises a large-size convex lens and a concave lens array which is placed close to the rear surface of the convex lens; the polarization component comprises a polarization array, and the polarization array is positioned between the convex lens and the concave lens array. According to the technical scheme, more image information can be acquired by using a polarization imaging technology, so that the definition and resolution of an image are improved, the anti-interference capability to interference factors such as scattering and refraction in an underwater environment is high, the underwater environment can be better adapted, the processing efficiency is high, and the system cost is effectively reduced.

Description

Underwater image acquisition system and method based on polarized light field

Technical Field

The invention relates to the field of computer vision and optical engineering, in particular to an underwater image acquisition system and method based on polarized light fields.

Background

After entering the water, natural light can interact with particles in the water, and components contained in the water, turbidity of the water body and background environment can influence the imaging effect of a target object in the underwater turbid medium. The seawater contains a large amount of organic matters and inorganic matters, particles such as sediment suspended in the water can scatter and absorb light, light wave energy is seriously attenuated, and partial light propagation direction is changed, so that light cannot reach an optical imaging system or background scattered light enters the optical imaging system, thereby changing imaging quality in the underwater optical imaging process, reducing image contrast and resolution and blurring images, extracting effective characteristic information of underwater scene targets, and seriously affecting recognition and analysis of the underwater targets.

The absorption and scattering characteristics of the water body on the light waves are analyzed, so that the difficulty of improving the imaging quality by improving the traditional photoelectric detection technology is high, and the imaging detection distance is limited. Natural light is unpolarized before entering the atmosphere, but after entering the atmosphere, is scattered by various suspended particles, gas molecules, and the like in the atmosphere, thereby forming polarized light. The water body environment is similar to the atmospheric environment, contains more suspended particles and water body particles, has a scattering effect on light waves entering the water body environment, and generates polarized light. The polarized light is generated by reflection or scattering of light by various scattering particles in the water body, so that the polarized light contains important information of the environment of the water body and the target object, and has important significance for researching the polarized information of the target object.

Gary d. Gilbert and John c. Pernica performed a series of experiments on imaging circularly polarized light underwater on naval experimental bases for circularly polarized light imaging technology, but only circularly polarized light was used in the experiments, the polarization angles involved were relatively small, and image processing was unfavorable for later stage. Ming et al designed a camera array with 6 cameras, the noise patterns formed by the same object point in each sub-aperture were different, so the noise intensity could be suppressed by the light field refocusing algorithm, reducing the effect of forward scattering noise, but in this experiment a separate sensor was used for each camera, which resulted in some synchronization of the images on the different sensors, and some chromatic aberration of the different sensors, which affected the follow-up experiments. John G. Walker et al collected images with orthogonal polarization angles and processed the collected images using a weighted subtraction algorithm, found that the method improved the visible distance of the image under either active or passive illumination conditions, but did not verify or process the polarization at the remaining angles due to the fact that only two sets of orthogonal polarization were used in the experiment. Liu et al propose fusing an infrared image with a polarized image to obtain a high contrast image, but this approach can result in increased manufacturing costs for the imaging system. Chen et al performs wavelet transformation fusion on the polarization degree image and the polarization angle image with the middle-wavelength infrared image and the long-wavelength infrared image respectively, and the fusion result has a good target recognition rate, but the method can only finally obtain a two-dimensional RGB image, and three-dimensional information in a scene is lost.

Disclosure of Invention

The invention aims to solve the technical problems and provide the technical task of perfecting and improving the prior art scheme, and provides an underwater image acquisition system and method based on a polarized light field so as to achieve the purpose of acquiring a clear image by an underwater imaging system. For this purpose, the present invention adopts the following technical scheme.

The underwater image acquisition system based on the polarized light field comprises an optical lens assembly, a polarization assembly, a mechanical connection assembly for connecting the optical lens and the polarization assembly and an image processor for registering and fusing images to obtain clear underwater images;

the optical lens assembly comprises a large-size convex lens and a concave lens array which is placed close to the rear surface of the convex lens;

The polarization component comprises a polarization array, and the polarization array is positioned between the convex lens and the concave lens array.

According to the technical scheme, more image information can be obtained by using a polarization imaging technology, so that the definition and resolution of an image are improved, and the device has strong anti-interference capability on interference factors such as scattering, refraction and the like in an underwater environment and can be better adapted to the underwater environment. The polarization imaging technology can better highlight the outline and the characteristics of the target object, and the detection capability and the recognition accuracy of the target object are improved. By adopting the polarization array mode, the reliability and stability of the imaging system can be improved. In addition, through the design of connection structure, make imaging system more reliable and stable. The technical scheme can be flexibly adjusted according to actual requirements, for example, the positions and the number of the optical lens assemblies and the polarization assemblies can be adjusted to adapt to different underwater environments and task requirements. Through registering and cutting and image fusion of the acquired images, noise and interference factors in the images can be removed, the definition of the images is improved, and more accurate image information is provided for subsequent applications.

As a preferable technical means: the mechanical connection assembly comprises a front end lens fixing part, a polarization fixing part, a middle connecting cylinder part and a rear end lens switching part which are connected with each other in sequence;

The convex lens is fixed in the front lens fixing part through the front group pressing ring, and meanwhile, the assembly distance between the concave lens array and the large-size convex lens is reduced as far as possible through adopting the inner layer adjustable back group pressing ring; the middle connecting cylinder part is assembled to the front end lens fixing part in a connecting mode of internal and external threads; the rear lens transfer component for transferring the external standard lens is combined with the middle connecting cylinder component by adopting a telescopic sleeve.

Through connecting front end lens fixed part, polarization fixed part, intermediate junction barrel part and rear end lens changeover component each other in proper order, a compact structure has been formed for whole imaging system is more reliable and stable. The inner layer is adopted to adjust the back group pressing ring, so that the assembly distance between the concave lens array and the large-size convex lens is reduced as much as possible, and the design mode enables the installation process to be more convenient and rapid. After the intermediate connecting cylinder part is assembled to the front end lens fixing part by means of the connection of the internal and external threads, the stability of the whole imaging system can be improved. The telescopic mechanical structure can obtain the sub-viewpoint image with continuously variable spatial resolution on the final imaging plane through light path adjustment, and the characteristic change of the final imaging plane promotes the adaptability of the invention to camera main lenses and sensor target surfaces of different models. Better image registration and fusion can be achieved through continuously variable sub-view images, thereby improving the quality of the final image. The telescopic mechanical structure enables the system to be more flexibly adjusted and adapted to different underwater environments and working conditions. For example, the distance between the optical components can be adjusted according to actual requirements to adapt to different water depths and light conditions. Because of the adoption of the telescopic mechanical structure, the system can be self-adjusted and adapted in the working process, errors and faults caused by external factors (such as water flow, temperature change and the like) are reduced, and the reliability of the system is improved.

As a preferable technical means: the polarization array is provided with six polarization imaging channels with different polarization angles and two filtering channels with different filtering functions, each polarization imaging channel enables each channel to only project the light intensity of an imaging scene in one polarization state, and then polarized images in different polarization states in the same target scene can be received on a sensor area corresponding to each channel. By adding the polarized imaging channel and the filtering channel, the technical scheme can acquire more image information, and the sensor area corresponding to each channel can receive polarized images with different polarization states under the same target scene, so that the information quantity of the images is improved. Because each channel only projects the light intensity of the imaging scene under one polarization state, the sensor area corresponding to each channel can only receive the polarized image of one polarization state, and the contour and the characteristics of the target object can be better highlighted by registering and fusing the polarized images of different polarization states, so that the detection capability and the identification precision of the target object are improved. By acquiring polarized images with different polarization states, interference factors can be separated and suppressed, and the anti-interference capability of the images is further improved. By registering and fusing polarized images with different polarization states, noise and interference factors in the images can be removed, and the quality of the images is improved. In addition, by adding the filtering channel, the image can be further filtered, and the imaging quality is further improved. The polarization array can flexibly adjust the polarization angle and the filtering function, so that different underwater environments and task requirements are met, and the system is more flexible and reliable.

As a preferable technical means: the polarization array comprises a plurality of polarization components and an imaging detector; four linear polarizers in the polarizing components are aperture linear polarization imaging channels, two circular polarizers in the aperture circular polarization imaging channels, two filters and one plate glass; the front end of the imaging detector is respectively provided with a polaroid, a filter and plate glass in corresponding polarization states. According to the technical scheme, polarized images with different polarization states can be obtained, more abundant polarized information is obtained, and the accuracy and reliability of target detection are improved. By using the filter and the plate glass, the image can be further filtered to remove noise and interference factors, so that the device is better suitable for complex interference factors in an underwater environment and the anti-interference capability of the image is improved.

As a preferable technical means: the polarization angles of the four linear polarizers rotate to the included angles of 0 degree, 45 degree, 90 degree and 135 degree with the optical axis respectively, the two circular polarizations are 45 degree+1/4 wave plates and-45 degree+1/4 wave plates, and the two wave plates are a green light wave plate and a blue light wave plate respectively. The polarization angles of the four linear polarizers rotate to the included angles of 0 degrees, 45 degrees, 90 degrees and 135 degrees with the optical axis respectively, so that polarized images with different polarization states can be obtained, the four angles can cover the range of orthogonal polarization states, and more comprehensive polarization information can be obtained. The two circularly polarized wave plates are 45 degrees plus 1/4 wave plates and-45 degrees plus 1/4 wave plates, so that circularly polarized light in different rotation directions is provided, the circularly polarized light has the characteristic of rotation directions, additional information can be provided, and the target detection capability is further enhanced. The two filters are respectively a green filter and a blue filter, so that the images are filtered on different wave bands, the green light and the blue light are colors sensitive to human eyes, and the characteristics of a target object can be better highlighted by filtering the green light and the blue light, so that the accuracy of target detection is improved.

As a preferable technical means: the image processor processes the image based on an end-to-end network model guided by an unsupervised learning and attention mechanism, so that the polarization and light intensity images are fused; the end-to-end network model is provided with a feature extraction module, a feature fusion module and an image reconstruction module; the feature extraction module integrates the attention mechanism and constructs related loss functions and weight parameters. By adopting the end-to-end network model, the whole processing flow can be integrated into a complete model, so that the problems of incoherence and redundancy brought by the traditional step-by-step processing method are avoided, the inherent characteristics and rules of the image are better captured, and the processing efficiency and accuracy are improved. The non-supervision learning can utilize unlabeled data to learn, so that a large amount of manpower and material resources required by labeling the data are avoided, the model can better discover the inherent characteristics and rules in the data through the non-supervision learning, and the generalization capability and robustness of the model are improved. Attention can be paid to important parts of the image automatically in the processing process, so that the characteristics of the image are extracted better, the perceptibility and the robustness of the model are improved, and complex image contents can be better dealt with. The feature extraction module integrates an attention mechanism, constructs a related loss function and weight parameters, does not need to manually adjust the weight parameters, has high operation speed, strong robustness and adaptability, and is more accurate and efficient in extracting image features. The polarization and light intensity images can be effectively fused through the feature fusion module and the image reconstruction module, so that richer and more accurate image information is obtained, and the accuracy and reliability of target detection and identification are improved.

As a preferable technical means: in the feature extraction module, a light intensity image and a polarized image are input through two channels, wherein a first layer is a convolution layer containing a3×3 convolution kernel and an activation function ReLU and is used for extracting low-layer features; the second layer is a Dense Block module containing 3 convolution layers, each also using a3×3 convolution kernel, for extracting higher layer features; the operation step length of the convolution kernel is 1, and before the convolution operation, a BN layer and a ReLU activation function are arranged so as to accelerate the training speed of the network; the two input channels of the light intensity image and the polarized image share the same weight, so that the computational complexity of the network is reduced; the feature extraction module is provided with an attention unit, the attention unit takes the feature diagram of the upper layer as input, can capture the global relation in the data, and guides the distribution of the network learning feature diagram; and in the feature fusion module, the feature graphs output by the feature extraction module are overlapped. The convolutional neural network is adopted to extract the characteristics, so that the spatial information of the image can be effectively utilized, and the characteristic representation of the image can be automatically learned. The convolution layer can capture local features, and the Dense Block module can extract higher-layer features. The two input channels of the light intensity image and the polarized image share the same weight, which reduces the computational complexity of the network and improves the generalization capability of the network. The BN layer and the ReLU activation functions are used before the convolutional layer, which can increase the training speed of the network and improve the nonlinear expression capability of the model.

As a preferable technical means: the attention unit combines the channel attention and the spatial attention; the channel attention enables the network to learn the importance of the features in the channel domain and give different weights to the feature images, so that the selective combination of the light intensity images and the polarized images in the channel domain is realized; spatial attention is focused on the effective information distribution of each layer of the learning feature map so as to improve the transmission of the salient features; the attention unit comprises a global average pooling layer, a convolution layer, an activation layer and a splicing layer, and given X epsilon R ^H×W×C and X epsilon R ^H×W×C as input and output of the attention unit, the calculation process of the attention unit is as follows:

Where σ is the Sigmoid activation function, F _C is the channel attention branch, F _S is the spatial attention branch, Is a broadcast addition operation,/>Is a per-element multiplication operation;

when the input feature diagram X epsilon R ^H×W×C passes through the channel attention branch, the channel feature X _C∈R^1×1×C is obtained through the global average pooling layer, and then the channel feature size is obtained through the PWConv ₁, BN layer and ReLU activation function point-by-point convolution The channel attention profile X _C;F_C, which is 1×1×c in size, is obtained by point-wise convolution of PWConv ₂ and BN layers, is expressed as:

F_C(X)＝BN(PWConv₂(δ(BN(PWConv₁(GAP(X))))))

Wherein: delta is a ReLU activation function, GAP is global average pooling; similar to the channel attention branch, the size is obtained by first using the 3×3 convolution Conv ₁, BN layer and ReLU activation functions when passing through the spatial attention branch Is a feature map of (1); to obtain a spatial attention profile of size H W C1X 1, convolution PWConv ₂ and BN layers are used; f _S is expressed as:

F_S(X)＝BN(PWConv₂(δ(BN(Conv₁(X)))))

The loss function employs a globally weighted SSIM (structural similarity) loss function, which is a multi-scale weighted SSIM (MSW-SSIM):

Loss ^SSIM (x, y; ω) is a SSIM-based Loss function that represents the structural similarity of images x and y over window ω:

Omega _x is the area of the image within window omega, Is the mean of omega _x; variable/>And/>The variance of ω _x and the covariance of ω _xω_y, respectively; the remainder omega _y,/>Represents the corresponding meaning; when/>And/>Very near zero, C ₁ and C ₂ are constants that avoid destabilization;

Multi-window SSIM is provided in the loss function, so that the problem of image details under different scales is solved; window sizes used include 3, 5, 7, 9, and 11; different windows can extract features of different scales; in addition, in the case of the optical fiber, And Loss ^SSIM(I_DoLP,I_f; ω) use weight coefficients based on/>And/>The definition is shown as a formula (11); when the window ω of the variance of intensity image S ₀ is larger than the corresponding DoLP image, it is indicated that the local area of S ₀ has more image detail; that is, the weight coefficient γ _ω corresponding to the S ₀ image is larger;

Is the variance of the intensity image S ₀ within the window ω; /(I) Is the variance of the DoLP image within the window ω, g (x) =max (x, 0.0001) is a correction function to increase the robustness of the solution.

The technical scheme adopts a high-efficiency, accurate and flexible feature extraction and fusion method, and realizes high-quality fusion of the light intensity image and the polarized image by combining the technologies of channel attention and space attention, unsupervised learning, an end-to-end network model, an attention mechanism, a globally weighted SSIM loss function and the like.

Another object of the present invention is to provide an underwater image acquisition method based on polarized light field, the underwater image acquisition method comprising the steps of:

Disposing an optical lens assembly and a polarizing assembly; the large-size convex lens, the polarization array and the concave lens array are sequentially arranged; assuming that the focal length of the convex lens is F, a real image surface formed by the object plane passing through the convex lens is a real image surface of the positive focal length optical element, and the image distance of the real image surface from the convex lens is l ', and the height of the image surface is h'; then, the real image surface realizes mutual separation of information of different angles through a concave lens array and a polarization array with the unit size d and the negative focal length-f, forms a sub-viewpoint virtual image surface under different angles, and the height of the virtual image surface is h ', and the image distance between the virtual image surface and the concave lens array is l'; finally, the virtual image surface is taken as a virtual object surface of which the rear end is externally connected with an optical camera and is captured by the optical camera;

After images with different polarization angles are obtained, inputting the images into an image processor, and registering and fusing the images by the image processor so as to obtain clear underwater images; the image processor processes the image based on an end-to-end network model guided by an unsupervised learning and attention mechanism, and the end-to-end network model performs feature extraction, feature fusion and image reconstruction on the image; when the characteristics are extracted, the light intensity image and the polarized image are input through two channels, wherein the first layer is a convolution layer containing a3 multiplied by 3 convolution kernel and an activation function ReLU and is used for extracting the characteristics of the lower layer; the second layer is a Dense Block module containing 3 convolution layers, each also using a3×3 convolution kernel, for extracting higher layer features; the operation step length of the convolution kernel is 1, and before the convolution operation, a BN layer and a ReLU activation function are arranged so as to accelerate the training speed of the network; the two input channels of the light intensity image and the polarized image share the same weight so as to reduce the computational complexity of the network; during attention calculation, taking the feature diagram of the upper layer as input, capturing global relation in data, and guiding the network to learn the distribution of the feature diagram; when the features are fused, the feature graphs output after feature extraction are overlapped; attention calculations combine channel attention and spatial attention; the channel attention enables the network to learn the importance of the features in the channel domain and give different weights to the feature images, so that the selective combination of the light intensity images and the polarized images in the channel domain is realized; spatial attention is focused on the effective information distribution of each layer of the learning feature map so as to improve the transmission of the salient features;

According to the method, the optical lens component, the polarization component, the large-size convex lens, the polarization array and the concave lens array are sequentially arranged, so that clear acquisition of underwater images is realized, and meanwhile, the resolution and the definition of the images are further improved through registering and fusing the images with different polarization angles. Due to the complexity and interference factors of the underwater environment, it is often difficult to acquire high-quality underwater images, and scattered light of some interference factors in the underwater environment can be restrained by using polarized light fields, so that the contrast of the images is improved, and the acquired images are clearer and easier to identify. When the characteristics are extracted, the light intensity image and the polarized image are input through two channels, the same weight is shared, the calculation complexity of the network is reduced, meanwhile, the image is processed by using an end-to-end network model guided by an unsupervised learning and attention mechanism, complex characteristic engineering and parameter adjustment are avoided, and the processing process is more efficient and flexible. By using the attention mechanism, important parts of the image can be automatically focused, and global relations in the data are captured, so that the characteristics of the image are better extracted; meanwhile, the channel attention and the space attention are combined, so that the extracted features are more comprehensive and accurate, and the accuracy and the robustness of applications such as target detection, identification, enhancement and reconstruction are improved.

As a preferable technical means: the object plane height used for imaging in the real scene is h, the distance from the convex lens in the optical accessory is l, the field angle of the rear-end external optical camera is assumed to be FOV, and the entrance pupil of the main lens is positioned on the position of the real image plane, so as to ensure that the whole virtual image plane 6 area can be completely positioned within the field range of the rear-end external optical camera, the following inequality is satisfied:

n is the number of concave lens units in the longitudinal dimension of the concave lens array 2. The optical parameters F of the convex lens satisfy the following constraint:

Based on the above, on the premise of meeting the shortest focusing distance of the main lens of the rear-end external optical camera, the assembly distance of the rear-end external optical camera is adjusted, so that a sufficient quantity of light rays are ensured to enter the rear-end external optical camera.

The beneficial effects are that: according to the technical scheme, a set of underwater image acquisition system based on a polarized light field is obtained by combining a polarized imaging technology and a light field camera system; the polarization component adopts the polarization array, and a plurality of camera arrays are distributed, can carry out the collection of polarized image and visible light image to same imaging scene simultaneously, adopts a sensor to avoid synchronous problem and the colour difference problem that exists between the different sensors.

After the obtained images with different polarization angles are subjected to feature extraction, low-layer features and high-layer features of the images can be obtained, the low-layer features and the high-layer features are used as input of a feature fusion module, the extracted two different types of features can be fused, and then the fused features are reconstructed to obtain a complete underwater image.

Drawings

FIG. 1 is a schematic diagram of a light field capturing system with an external polarizer array according to the present invention.

Fig. 2 is a diagram of an image registration and fusion network of the present invention.

Fig. 3 is a structural diagram of an attention unit of the present invention.

In the figure: 1-a convex lens; a 2-concave lens array; a 3-polarization array; 4-object plane; 5-virtual image plane; 6-real image surface; 7-the rear end is externally connected with an optical camera.

Detailed Description

The technical scheme of the invention is further described in detail below with reference to the attached drawings.

As shown in fig. 1, the polarized light field-based underwater image acquisition system comprises an optical lens assembly, a polarization assembly, a mechanical connection assembly for connecting the optical lens and the polarization assembly, and an image processor for registering and fusing images to obtain clear underwater images;

The optical lens assembly comprises a large-sized convex lens and a concave lens array placed near the rear surface of the convex lens;

the polarizing assembly includes a polarizing array positioned between the convex lens and the concave lens array.

Compared with the traditional visible light imaging, the polarization imaging technology expands the information quantity from three dimensions (spectrum, light intensity and space) to seven dimensions (spectrum, light intensity, space, polarization degree, polarization angle, polarization ellipsometry and rotation direction), has the advantage incomparable with the intensity information, and the light field has the unique advantage, and the depth information of a target object can be obtained through the light field information in the environment. The polarization imaging technology can better highlight the outline and the characteristics of the target object, and the detection capability and the recognition accuracy of the target object are improved. By adopting the polarization array mode, the reliability and stability of the imaging system can be improved. In addition, through the design of connection structure, make imaging system more reliable and stable. The technical scheme can be flexibly adjusted according to actual requirements, for example, the positions and the number of the optical lens assemblies and the polarization assemblies can be adjusted to adapt to different underwater environments and task requirements. Through registering and cutting and image fusion of the acquired images, noise and interference factors in the images can be removed, the definition of the images is improved, and more accurate image information is provided for subsequent applications.

In this embodiment, the concave lens array is a regular polygon concave lens array, and the regular polygon concave lens array includes concave lens array units with identical optical parameters, and the focal length of each concave lens unit passes through the formula: And determining, wherein f is an absolute value of a focal length of the concave lens array unit, d is a size of the concave lens array unit, l is an object distance of a real scene, and h is an object plane size of the real scene. The regular polygon concave lens units are sequentially adhered on a round substrate made of the same material and are arranged into a regular polygon concave lens array, and meanwhile, a covering area of a non-concave lens array in the round substrate is required to be covered with a shading material so as to isolate extra light interference.

The polarization array consists of polarization components and imaging detectors, the imaging system is provided with nine channels, and in order to reduce imaging errors and registration difficulties and increase imaging overlapping areas, four apertures are selected as linear polarization imaging channels, two apertures are circularly polarized, two apertures are filter plates, and one aperture is plate glass. The front end of the optical imaging detector is respectively provided with a polaroid, a filter and plate glass in corresponding polarization states, the polarization angles of the four linear polaroids are rotated to be respectively 0 DEG, 45 DEG, 90 DEG and 135 DEG with the optical axis, the two circular polarizations are 45 DEG+1/4 wave plates (left-hand circular polarization) and-45 DEG+1/4 wave plates (right-hand circular polarization), and the two filter plates are respectively a green light filter and a blue light filter.

The sub-components of the mechanical connection assembly are all of a telescopic connection structure, and the assembly positions of the external standard lenses are required to satisfy the following conditions: the optical entrance pupil plane of the external standard lens is required to be positioned at the real image plane of the front large-size convex lens. The telescopic sleeve connection design of the rear-end lens adapting part makes it very easy to determine the actual optimal installation distance of the external standard lens; meanwhile, the telescopic mechanical structure can obtain the sub-viewpoint image with continuously variable spatial resolution on the final imaging plane through light path adjustment, and the characteristic change of the final imaging plane promotes the adaptability of the invention to camera main lenses and sensor target surfaces of different models.

The external lens array light field acquisition device is characterized in that a specific optical element accessory is added at the front end of a main lens of a traditional camera, so that light field sampling based on the traditional camera, namely an external optical camera, is realized. The optical element accessory separates different angle information from the same scene point in the real world, and enables light rays carrying the different angle information to be captured by a traditional camera at the rear end, so that sparse sampling of a real scene light field is realized. In order to separate information of different angles of the same scene point, the optical element accessory must comprise an array structure; meanwhile, considering the limitation of the physical size of the system arrangement and the inherent shortest focusing distance constraint of the main lens of the external optical camera at the rear end, in practice, the preferred scheme of the array structure is as follows: the positive focal length optical elements combine with the negative focal length optical element arrays.

Specifically, an optical path diagram of a typical convex lens combined concave lens array structure in fig. 1 is described:

The object plane 4 for imaging in a real scene has a height h and a distance l from the convex lens 1 in the optical attachment. First, assuming that the focal length of the convex lens 1 is F, a real image plane 6 of the object plane 4 passing through the convex lens 1 is a real image plane 6 of a positive focal length optical element, and an image distance of the real image plane 6 from the convex lens 1 is l ', and an image plane height is h'. Next, the real image plane 6 is separated from each other by the concave lens array 2 and the polarization array 3 having the cell size d and the negative focal length-f, and forms the sub-viewpoint virtual image plane 5 at different angles, and the height of the virtual image plane 5 is h″ and the image distance from the concave lens array 2 is l″. Finally, the virtual image plane 5 is captured as a virtual object plane with the rear end externally connected to the optical camera 7.

In the whole imaging process, the concave lens array 2 with the negative focal length is introduced, so that the imaging distance of the rear-end external optical camera is prolonged, and the limit of the inherent shortest focusing distance of the main lens of the rear-end external optical camera 7 can be met while the integral physical size of the system is reduced. With the two broken lines of light in fig. 1, the following equivalent relationship exists between the two broken lines of light and a similar triangle formed by the concave lens array 2 and the real image surface 6 in sequence:

The "object-image" relationship combining the principle of convex lens 1 thin lens imaging: And gaussian imaging formula for concave lens array 2 unit: /(I) The method can obtain:

Wherein: f is the absolute value of the focal length of the concave lens array unit, d is the size of the concave lens array unit, l is the object distance of the real scene, and h is the object plane 4 size of the real scene. As can be seen from the formula (2), the focal length of the concave lens array unit is independent of the optical parameters of the convex lens 1, and when the working parameters of the system (i.e. the current imaging working distance l of the system and the real scene object plane height h) are determined, the focal length of the concave lens array 2 unit is uniquely determined by the size thereof.

Since the optical parameters of the convex lens 1 do not affect the determination of the optical parameters of the concave lens array 2, when the assembly distance of the rear-end external optical camera 7 is determined, the field angle FOV becomes the only constraint that the focal length F of the convex lens 1 needs to satisfy. By the working parameters of the system, the optical parameters of the concave lens array can be uniquely determined under the condition that the sub-images of the adjacent concave lens array units are not overlapped, so that the size h' of the sub-viewpoint image in the virtual image plane 5 is only determined by the optical parameters of the convex lens 1. Specifically, the "object-image" relationship according to the imaging principle of the convex lens 1: And gaussian formula: /(I) And the "object-image" relationship of the concave lens array 2-cell thin lens imaging principle: /(I)The method can obtain:

Meanwhile, assuming that the field angle of the rear-end external optical camera 7 is FOV and the entrance pupil of the main lens is located at the position of the real image surface 6, in order to ensure that the entire virtual image surface 5 area can be completely located within the field range of the rear-end external optical camera 7, the following inequality relationship must be satisfied:

n is the number of concave lens units in the longitudinal dimension of the concave lens array 2. In connection with equation (3), the optical parameter F of the convex lens 1 needs to satisfy the following constraint:

based on the conclusion of the formula (5), on the premise of meeting the shortest focusing distance of the main lens of the rear-end external optical camera 7, by properly adjusting the assembly distance of the rear-end external optical camera 7, a sufficient quantity of light rays can be ensured to enter the rear-end external optical camera 7.

After images with different polarization angles are obtained, the images need to be subjected to feature extraction, feature fusion and image reconstruction, and the images consist of 3 modules, as shown in fig. 2. First, in the feature extraction module, an optical intensity image and a polarization image are input through two channels, and a first layer is a convolution layer including a 3×3 convolution kernel and an activation function ReLU (linear unit after rectification) for extracting low-layer features. The second layer is a Dense Block module containing 3 convolutional layers, each also using a 3×3 convolutional kernel, for extracting higher layer features. The operation step length of the convolution kernel is 1, and before the convolution operation, a BN (batch normalization) layer and a ReLU activation function are arranged, so that the ordering is beneficial to accelerating the training speed of the network. The two input channels of the light intensity image and the polarized image share the same weight, so that the computational complexity of the network can be reduced. The next is the attention unit, which takes as input the feature map of the previous layer. The method can capture global relations in the data and guide the distribution of the network learning feature map. And secondly, in the feature fusion module, the feature graphs output by the feature extraction module are overlapped. The channel size of the two feature graphs is 128, and the channel size of the fusion feature graph after superposition is 256.

The attention unit combines channel attention and spatial attention. Channel attention enables the network to learn the importance of features in the channel domain and to assign different weights to the feature maps, thereby achieving selective combination of light intensity images and polarized images in the channel domain. Spatial attention is focused on learning the effective information distribution of each layer of the feature map to enhance the transfer of salient features. The attention unit includes a global averaging layer, a convolution layer, an activation layer and a stitching layer, the detailed structure of which is shown in fig. 3. Given X ε R ^H×W×C and X' εR ^H×W×C as input and output of attention unit, the calculation process of the attention unit is:

Where σ is the Sigmoid activation function, F _C is the channel attention branch, F _S is the spatial attention branch, Is a broadcast addition operation,/>Is a per-element multiplication operation.

When the input feature diagram X epsilon R ^H×W×C passes through the channel attention branch, the channel feature X _C∈R^1×1×C is obtained through the global average pooling layer, and then the channel feature size is obtained through the PWConv ₁, BN layer and ReLU activation function point-by-point convolutionThe channel attention profile X _C.F_C, which is 1×1×c in size, is obtained by point-wise convolution of PWConv ₂ and BN layers, is expressed as:

F_C(X)＝BN(PWConv₂(δ(BN(PWConv₁(GAP(X)))))) (7)

In the formula (7): delta is the ReLU activation function and GAP is global average pooling. Similar to the channel attention branch, the size is obtained by first using the 3×3 convolution Conv1, BN layer and ReLU activation functions when passing through the spatial attention branch Is a feature map of (1). To obtain a spatial attention profile of size H W C1X 1, convolution PWConv and BN layers are used. F _S can be expressed as:

F_S(X)＝BN(PWConv₂(δ(BN(Conv₁(X))))) (8)

The loss function employs a globally weighted SSIM (structural similarity) loss function, which is a multi-scale weighted SSIM (MSW-SSIM):

Loss ^SSIM (x, y; ω) is a SSIM-based Loss function that represents the structural similarity of images x and y over window ω:

Omega _x is the area of the image within window omega, Is the mean of ω _x. Variable/>And/>The variance of ω _x and the covariance of ω _xω_y, respectively. The remainder omega _y,/>Representing the corresponding meaning. When/>And/>Very near zero, C ₁ and C ₂ are constants that avoid destabilization.

The multi-window SSIM is provided in the loss function, so that the problem of image details under different scales is solved. The window sizes used include 3, 5, 7, 9 and 11. Different windows may extract features of different scales. In addition, in the case of the optical fiber,And Loss ^SSIM(I_DoLP,I_f; ω) use weight coefficients based on/>And/>The definition is shown as a formula (11). When the window ω of the variance of intensity image S ₀ is larger than the corresponding DoLP image, it is indicated that the local area of S ₀ has more image detail; that is, the weight coefficient γ _ω corresponding to the S ₀ image is larger.

Is the variance of the intensity image S ₀ within the window ω; /(I)Is the variance of the DoLP image within the window ω, g (x) =max (x, 0.0001) is a correction function to increase the robustness of the solution.

The above underwater image acquisition system and method based on polarized light field shown in fig. 1,2 and 3 are specific embodiments of the present invention, and have already shown substantial features and improvements of the present invention, and can be modified in terms of shape, structure, etc. according to practical use requirements, in the light of the present invention, all of which are within the scope of protection of the present invention.

Claims (10)

1. The underwater image acquisition system based on the polarized light field is characterized in that: the device comprises an optical lens assembly, a polarization assembly, a mechanical connection assembly for connecting the optical lens and the polarization assembly, and an image processor for registering and fusing images to obtain clear underwater images;

the optical lens assembly comprises a large-size convex lens and a concave lens array which is placed close to the rear surface of the convex lens;

The polarization component comprises a polarization array, and the polarization array is positioned between the convex lens and the concave lens array.

2. The polarized light field based underwater image acquisition system of claim 1 wherein: the mechanical connection assembly comprises a front end lens fixing part, a polarization fixing part, a middle connecting cylinder part and a rear end lens switching part which are connected with each other in sequence;

The convex lens is fixed in the front lens fixing part through the front group pressing ring, and meanwhile, the assembly distance between the concave lens array and the large-size convex lens is reduced as far as possible through adopting the inner layer adjustable back group pressing ring; the middle connecting cylinder part is assembled to the front end lens fixing part in a connecting mode of internal and external threads; the rear lens transfer component for transferring the external standard lens is combined with the middle connecting cylinder component by adopting a telescopic sleeve.

3. The polarized light field based underwater image acquisition system of claim 1 wherein: the polarization array is provided with six polarization imaging channels with different polarization angles and two filtering channels with different filtering functions, each polarization imaging channel enables each channel to only project the light intensity of an imaging scene in one polarization state, and then polarized images in different polarization states in the same target scene can be received on a sensor area corresponding to each channel.

4. A polarized light field based underwater image acquisition system as claimed in claim 3 wherein: the polarization array comprises a plurality of polarization components and an imaging detector; four of the polarizing components are linear polarizers with the aperture of a linear polarization imaging channel, two of the polarizing components are circular polarizers with the aperture of a circular polarization imaging channel, two of the polarizing components are filter plates, and one of the polarizing components is plate glass; the front end of the imaging detector is respectively provided with a polaroid, a filter and plate glass in corresponding polarization states.

5. The polarized light field based underwater image acquisition system of claim 4 wherein: the polarization angles of the four linear polarizers rotate to the included angles of 0 degree, 45 degree, 90 degree and 135 degree with the optical axis respectively, the two circular polarizations are 45 degree+1/4 wave plates and-45 degree+1/4 wave plates, and the two wave plates are a green light wave plate and a blue light wave plate respectively.

6. The polarized light field based underwater image acquisition system of claim 1 wherein: the image processor processes the image based on an end-to-end network model guided by an unsupervised learning and attention mechanism, so that the polarization and light intensity images are fused; the end-to-end network model is provided with a feature extraction module, a feature fusion module and an image reconstruction module; the feature extraction module integrates the attention mechanism and constructs related loss functions and weight parameters.

7. The polarized light field based underwater image acquisition system of claim 6 wherein: in the feature extraction module, a light intensity image and a polarized image are input through two channels, wherein a first layer is a convolution layer containing a3×3 convolution kernel and an activation function ReLU and is used for extracting low-layer features; the second layer is a Dense Block module containing 3 convolution layers, each also using a3×3 convolution kernel, for extracting higher layer features; the operation step length of the convolution kernel is 1, and before the convolution operation, a BN layer and a ReLU activation function are arranged so as to accelerate the training speed of the network; the two input channels of the light intensity image and the polarized image share the same weight, so that the computational complexity of the network is reduced; the feature extraction module is provided with an attention unit, the attention unit takes the feature diagram of the upper layer as input, can capture the global relation in the data, and guides the distribution of the network learning feature diagram; and in the feature fusion module, the feature graphs output by the feature extraction module are overlapped.

8. The polarized light field based underwater image acquisition system of claim 7 wherein: the attention unit combines the channel attention and the spatial attention; the channel attention enables the network to learn the importance of the features in the channel domain and give different weights to the feature images, so that the selective combination of the light intensity images and the polarized images in the channel domain is realized; spatial attention is focused on the effective information distribution of each layer of the learning feature map so as to improve the transmission of the salient features; the attention unit comprises a global average pooling layer, a convolution layer, an activation layer and a splicing layer, and given X epsilon R ^H×W×C and X epsilon R ^H×W×C as input and output of the attention unit, the calculation process of the attention unit is as follows:

Where σ is the Sigmoid activation function, F _C is the channel attention branch, F _S is the spatial attention branch, Is a broadcast addition operation,/>Is a per-element multiplication operation;

when the input feature diagram X epsilon R ^H×W×C passes through the channel attention branch, the channel feature X _C∈R^1×1×C is obtained through the global average pooling layer, and then the channel feature size is obtained through the PWConv ₁, BN layer and ReLU activation function point-by-point convolution The channel attention profile X _C;F_C, which is 1×1×c in size, is obtained by point-wise convolution of PWConv ₂ and BN layers, is expressed as:

F_C(X)＝BN(PWConv₂(δ(BN(PWConv₁(GAP(X))))))

Wherein: delta is a ReLU activation function, GAP is global average pooling; similar to the channel attention branch, the size is obtained by first using the 3×3 convolution Conv ₁, BN layer and ReLU activation functions when passing through the spatial attention branch Is a feature map of (1); to obtain a spatial attention profile of size H W C1X 1, convolution PWConv ₂ and BN layers are used; f _S is expressed as:

F_S(X)＝BN(PWConv₂(δ(BN(Conv₁(X)))))

The loss function employs a globally weighted SSIM loss function, which is a multi-scale weighted SSIM (MSW-SSIM):

Loss ^SSIM (x, y; ω) is a SSIM-based Loss function that represents the structural similarity of images x and y over window ω:

Omega _x is the area of the image within window omega, Is the mean of omega _x; variable/>And/>The variance of ω _x and the covariance of ω _xω_y, respectively; the remainder omega _y,/>Represents the corresponding meaning; when/>And/>Very near zero, C ₁ and C ₂ are constants that avoid destabilization;

Multi-window SSIM is provided in the loss function, so that the problem of image details under different scales is solved; window sizes used include 3, 5, 7, 9, and 11; different windows can extract features of different scales; furthermore, loss ^SSIM(I_S0,I_f; ω) and Loss ^SSIM(I_DoLP,I_f; ω) use weight coefficients, which are based on And/>The definition is shown as a formula (11); when the window ω of the variance of intensity image S ₀ is larger than the corresponding DoLP image, it is indicated that the local area of S ₀ has more image detail; that is, the weight coefficient γ _ω corresponding to the S ₀ image is larger;

Is the variance of the intensity image S ₀ within the window ω; /(I) Is the variance of the DoLP image within the window ω, g (x) =max (x, 0.0001) is a correction function to increase the robustness of the solution.

9. A method of underwater image acquisition employing the polarized light field based underwater image acquisition system as claimed in any one of claims 1 to 8, characterized by comprising the steps of:

Disposing an optical lens assembly and a polarizing assembly; the large-size convex lens, the polarization array and the concave lens array are sequentially arranged; assuming that the focal length of the convex lens is F, a real image surface formed by the object plane passing through the convex lens is a real image surface of the positive focal length optical element, and the image distance of the real image surface from the convex lens is l ', and the height of the image surface is h'; then, the real image surface realizes mutual separation of information of different angles through a concave lens array and a polarization array with the unit size d and the negative focal length-f, forms a sub-viewpoint virtual image surface under different angles, and the height of the virtual image surface is h ', and the image distance between the virtual image surface and the concave lens array is l'; finally, the virtual image surface is taken as a virtual object surface of which the rear end is externally connected with an optical camera and is captured by the optical camera;

After images with different polarization angles are obtained, inputting the images into an image processor, and registering and fusing the images by the image processor so as to obtain clear underwater images; the image processor processes the image based on an end-to-end network model guided by an unsupervised learning and attention mechanism, and the end-to-end network model performs feature extraction, feature fusion and image reconstruction on the image; when the characteristics are extracted, the light intensity image and the polarized image are input through two channels, wherein the first layer is a convolution layer containing a 3 multiplied by 3 convolution kernel and an activation function ReLU and is used for extracting the characteristics of the lower layer; the second layer is a Dense Block module containing 3 convolution layers, each also using a 3×3 convolution kernel, for extracting higher layer features; the operation step length of the convolution kernel is 1, and before the convolution operation, a BN layer and a ReLU activation function are arranged so as to accelerate the training speed of the network; the two input channels of the light intensity image and the polarized image share the same weight so as to reduce the computational complexity of the network; during attention calculation, taking the feature diagram of the upper layer as input, capturing global relation in data, and guiding the network to learn the distribution of the feature diagram; when the features are fused, the feature graphs output after feature extraction are overlapped; attention calculations combine channel attention and spatial attention; the channel attention enables the network to learn the importance of the features in the channel domain and give different weights to the feature images, so that the selective combination of the light intensity images and the polarized images in the channel domain is realized; spatial attention is focused on learning the effective information distribution of each layer of the feature map to enhance the transfer of salient features.

10. The underwater image acquisition method according to claim 9, characterized in that: the object plane height used for imaging in the real scene is h, the distance from the convex lens 1 in the optical accessory is l, the field angle of the rear-end external optical camera is assumed to be FOV, and the entrance pupil of the main lens is located at the position of the real image plane ⑧, so as to ensure that the whole virtual image plane 6 area can be completely located within the field range of the rear-end external optical camera 7, and the following inequality is satisfied:

n is the number of concave lens units in the longitudinal dimension of the concave lens array 2; the optical parameters F of the convex lens satisfy the following constraint:

Based on the above, on the premise of meeting the shortest focusing distance of the main lens of the rear-end external optical camera, the assembly distance of the rear-end external optical camera is adjusted, so that a sufficient quantity of light rays are ensured to enter the rear-end external optical camera.