CN115242966A - Anti-shake method and device for camera equipment and computer readable storage medium - Google Patents
Anti-shake method and device for camera equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN115242966A CN115242966A CN202210576152.0A CN202210576152A CN115242966A CN 115242966 A CN115242966 A CN 115242966A CN 202210576152 A CN202210576152 A CN 202210576152A CN 115242966 A CN115242966 A CN 115242966A
- Authority
- CN
- China
- Prior art keywords
- jitter
- data
- shake
- sub
- prediction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000012549 training Methods 0.000 claims abstract description 47
- 238000012545 processing Methods 0.000 claims description 35
- 238000004590 computer program Methods 0.000 claims description 13
- 230000015654 memory Effects 0.000 claims description 8
- 238000013508 migration Methods 0.000 claims description 4
- 238000012217 deletion Methods 0.000 claims description 2
- 230000037430 deletion Effects 0.000 claims description 2
- 238000006073 displacement reaction Methods 0.000 description 14
- 238000013528 artificial neural network Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 7
- 238000004422 calculation algorithm Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 210000002569 neuron Anatomy 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 230000000306 recurrent effect Effects 0.000 description 4
- 230000001360 synchronised effect Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000001931 thermography Methods 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000006403 short-term memory Effects 0.000 description 1
Images
Landscapes
- Studio Devices (AREA)
Abstract
The application discloses an anti-shake method, an anti-shake device and a computer-readable storage medium of a camera device, wherein the method comprises the following steps: acquiring a jitter data sample list of the camera equipment, wherein the jitter data sample list comprises actual jitter data at historical time and actual jitter data at current time; training a current jitter prediction model based on the jitter data sample list; predicting the jitter data of the camera equipment at the next moment by adopting the trained jitter prediction model to obtain jitter prediction data; performing shake compensation on the image pickup apparatus at the next time based on the shake prediction data; acquiring actual jitter data of the camera equipment at the next moment to obtain equipment jitter data; and updating a jitter data sample list based on the equipment jitter data, updating the current jitter prediction model, and returning to the step of training the current jitter prediction model based on the jitter data sample list. By means of the mode, the anti-shaking device can achieve anti-shaking and reduce the shaking condition of the picture.
Description
Technical Field
The present application relates to the field of image processing technologies, and in particular, to an anti-shake method and apparatus for an image capturing device, and a computer-readable storage medium.
Background
The installation environment of the thermal imaging camera has certain mechanical shaking, and the shaking problem is more prominent particularly when the device is installed on a moving object (such as an airplane, a ship or a car); when the long-focus thermal imager is used for observing objects beyond a distance of several kilometers, even slight mechanical shake can cause severe shaking of images, so that the quality of shot contents is poor, and therefore, how to improve the anti-shake performance of the thermal imager becomes a problem to be solved urgently.
Disclosure of Invention
The application provides an anti-shake method and device for an image pickup device and a computer readable storage medium, which can reduce the image shaking condition.
In order to solve the technical problem, the technical scheme adopted by the application is as follows: there is provided an anti-shake method of an image pickup apparatus, the method including: acquiring a jitter data sample list of the camera equipment, wherein the jitter data sample list comprises actual jitter data at historical time and actual jitter data at current time; training a current jitter prediction model based on a jitter data sample list to obtain a trained jitter prediction model; predicting the jitter data of the camera equipment at the next moment by adopting the trained jitter prediction model to obtain jitter prediction data; performing shake compensation on the image pickup apparatus at the next timing based on the shake prediction data; acquiring actual jitter data of the camera equipment at the next moment to obtain equipment jitter data; and updating a jitter data sample list based on the equipment jitter data, updating the current jitter prediction model into a trained jitter prediction model, returning to the jitter data sample list based on the equipment jitter data, and training the current jitter prediction model.
In order to solve the above technical problem, another technical solution adopted by the present application is: there is provided an image pickup apparatus including a memory and a processor connected to each other, wherein the memory is configured to store a computer program, and the computer program, when executed by the processor, is configured to implement the anti-shake method of the image pickup apparatus in the above-described technical solution.
In order to solve the above technical problem, another technical solution adopted by the present application is: there is provided a computer-readable storage medium for storing a computer program for implementing the anti-shake method of an image pickup apparatus in the above-described technical solution when the computer program is executed by a processor.
Through above-mentioned scheme, this application's beneficial effect is: firstly, acquiring a jitter data sample list of the camera equipment, wherein the jitter data sample list comprises actual jitter data at historical time and actual jitter data at current time; then, training a current jitter prediction model by using a jitter data sample list to obtain a trained jitter prediction model; predicting the jitter data of the camera equipment at the next moment by adopting the trained jitter prediction model to obtain jitter prediction data; carrying out shake compensation on the camera shooting equipment at the next moment by using the shake prediction data, and acquiring actual shake data of the camera shooting equipment at the next moment to obtain equipment shake data; then updating a jitter data sample list by using equipment jitter data, updating the current jitter prediction model into a trained jitter prediction model, returning to execute the jitter-based data sample list, training the current jitter prediction model, and continuously updating the jitter prediction model by using the latest actual jitter data so as to improve the accuracy of prediction; according to the scheme, the camera equipment is subjected to displacement compensation when the camera equipment is displaced due to shaking, and the camera equipment is not compensated after the camera equipment is subjected to larger displacement, so that the compensation response speed of the camera equipment can be increased, the shaking problem of the camera equipment is relieved, more stable video pictures are provided, and the probability of back-and-forth shaking of the shot pictures is reduced.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts. Wherein:
fig. 1 is a schematic flowchart of an embodiment of an anti-shake method for an image capturing apparatus provided in the present application;
fig. 2 is a schematic flowchart of another embodiment of an anti-shake method for an image capturing apparatus provided in the present application;
FIG. 3 is a schematic diagram of a structure of a jitter prediction model provided herein;
fig. 4 is a schematic structural diagram of an embodiment of an image pickup apparatus provided by the present application;
fig. 5 is a schematic structural diagram of another embodiment of an image pickup apparatus provided by the present application;
FIG. 6 is a schematic structural diagram of an embodiment of a computer-readable storage medium provided in the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be noted that the following examples are only illustrative of the present application, and do not limit the scope of the present application. Likewise, the following examples are only some examples and not all examples of the present application, and all other examples obtained by a person of ordinary skill in the art without any inventive work are within the scope of the present application.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
It should be noted that the terms "first", "second" and "third" in this application are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or to imply that the number of indicated technical features is high. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of the feature. In the description of the present application, "plurality" means at least two, e.g., two, three, etc., unless explicitly specifically limited otherwise. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
In the anti-shake algorithm in the related art, displacement is generally detected through a built-in gyroscope, and then displacement fine adjustment is performed on a lens group or an image sensor in the opposite direction, so that the problem of image blur caused by camera shake is solved. However, this solution is not well applicable to long focus thermal imagers because: the jitter cannot be predicted in advance, and the opposite displacement compensation is required after the displacement is obviously deviated, so that although the passive correction method can better stabilize the video picture on a visible light camera with a short observation distance, the problem of picture shaking back and forth on a long-focus thermal imager with an observation distance of several kilometers is caused due to the hysteresis in control.
Based on this, the present application provides an active lens anti-shake method based on a time sequence prediction algorithm, that is, a mechanical shake trend at the next moment is predicted in advance through the time sequence prediction algorithm, and displacement compensation in the opposite direction is performed synchronously instead of performing displacement compensation in the opposite direction after the displacement is significantly offset, so as to provide a more stable video picture for a far-focus thermal imager, which is described in detail below.
Referring to fig. 1, fig. 1 is a schematic flowchart illustrating an embodiment of an anti-shake method for an image capturing apparatus according to the present disclosure, where the method includes:
s11: a list of shake data samples of an image pickup apparatus is acquired.
The mechanical jitter data of the long-focus thermal imager is analyzed, and the change of a jitter oscillogram is not completely random but shows a certain regularity in an adjacent period of time, so that a jitter prediction model is established, the trained jitter prediction model is obtained by training the jitter prediction model, and the trained jitter prediction model is used for predicting the jitter situation of the future time, so that the jitter compensation is carried out when the future time arrives, and the jitter situation of equipment is improved.
Further, a shake data sample list for storing actual shake data acquired by an offset sensor in the image pickup apparatus at each time period may be created first. Specifically, in the initial state, the list of jittered data samples is empty; the camera equipment also comprises an image sensor, and when the camera equipment is in a working state, the offset sensor is adopted to detect the shaking condition of the image sensor in real time to obtain the actual shaking data at each detection moment; after the jitter data collection is completed, the jitter data sample list includes actual jitter data at historical time and actual jitter data at current time, and the time interval between the earliest historical time and the current time may be a preset time interval, such as: 3s in the sequence.
S12: and training the current jitter prediction model based on the jitter data sample list to obtain the trained jitter prediction model.
After the actual jitter data of the time period from the historical time to the current time is collected, the actual jitter data (recorded as training samples) of a plurality of times in the jitter data sample list are utilized to train the current jitter prediction model, and the trained jitter prediction model is obtained. It is understood that, during the first training, the current jitter prediction model is a pre-established jitter prediction model.
In a specific embodiment, a set number of training samples can be selected from a jitter data sample list according to the sequence of the corresponding moments of the training samples to form batch sample data, and the batch sample data is input into a current jitter prediction model, so that the batch sample data is processed by the current jitter prediction model to obtain sample prediction data of a predicted moment; then calculating the loss between the sample prediction data and the training sample at the prediction moment to obtain a current loss value; then judging whether the preset training end condition is met or not at present, and if yes, obtaining a trained jitter prediction model; if not, returning to the step of selecting a set number of training samples from the jitter data sample list to form batch sample data according to the sequence of the corresponding moments of the training samples until a preset training end condition is met. As can be understood, the set number is the number of training samples in the batch of sample data, which is smaller than the number of all training samples in the jitter data sample list; for example, assuming that the jitter data sample list includes training samples at times T1 to T1000, and the set number is 50, the training samples at times T1 to T50 are selected from the jitter data sample list to form a first batch of sample data; and after the first batch of sample data is processed, selecting training samples at the time of T51-T100 to form a second batch of sample data, and repeating the steps until the training of the current jitter prediction model is finished.
Further, the preset training end condition includes: the loss value is converged, namely the difference value between the last loss value and the current loss value is smaller than a set value; judging whether the current loss value is smaller than a preset loss value, wherein the preset loss value is a preset loss threshold value, and if the current loss value is smaller than the preset loss value, determining that a preset training end condition is reached; training times reach a set value (for example: 10000 times of training); or the accuracy obtained when the test set is used for testing reaches a set condition (for example, the preset accuracy is exceeded), and the like.
S13: and predicting the jitter data of the camera equipment at the next moment by adopting the trained jitter prediction model to obtain jitter prediction data.
When the trained jitter prediction model is acquired, a plurality of data in the jitter data sample list may be input into the trained jitter prediction model to predict the jitter condition of the image capturing apparatus at the next time of the current time, and generate jitter prediction data at the next time. For example, assume that the current time is t m N actual jitter data are needed to match t m+1 Predicting the jitter situation of the time, inputting the actual jitter data of (n-1) times before the current time and the actual jitter data of the current time into the trained jitter prediction model to obtain t m+1 Jitter prediction data of time of day.
S14: the image pickup apparatus is subjected to shake compensation at the next timing based on the shake prediction data.
After the jitter prediction data at the next moment is acquired, the jitter condition which possibly occurs at the next moment can be compensated when the next moment comes so as to reduce the jitter degree of the camera equipment at the next moment; specifically, the image sensor may be driven to move in a direction opposite to a direction corresponding to the shake prediction data.
S15: and acquiring actual shaking data of the camera equipment at the next moment to obtain equipment shaking data.
Because the prediction of the jitter data at the next moment by adopting the trained jitter prediction model may not be accurate, the actual jitter data (namely the equipment jitter data) of the camera equipment at the moment can be obtained when the next moment is reached, so that the equipment jitter data is used for updating the jitter prediction model in the following process, and the prediction accuracy of the model is improved.
S16: and updating a jitter data sample list based on the equipment jitter data, and updating the current jitter prediction model into a trained jitter prediction model.
After equipment jitter data corresponding to the next moment is obtained, the equipment jitter data can be stored in a jitter data sample list so as to update the data in the jitter data sample list; and then updating the current jitter prediction model into a trained jitter prediction model, returning to the step of training the current jitter prediction model based on the jitter data sample list, namely returning to the step of executing S12 to predict the actual jitter data of the camera equipment at the subsequent moment by using the equipment jitter data at the next moment, and continuously improving the prediction effect of the model by continuously updating the jitter data sample list and training the jitter prediction model by using the updated jitter data sample list. For example, assuming that a jitter prediction model established initially is recorded as model 1, during first training, the current training model is model 1, and after the training of model 1 is completed, model 2 is obtained; taking the model 2 as a current shaking training model, and training the model 2 to obtain a model 3; and taking the model 3 as a current jitter prediction model, and so on, and gradually improving the prediction accuracy of the model.
The embodiment provides an active anti-shake scheme based on time sequence prediction, in the running process of a camera device, a shake trend at the next moment is predicted in advance according to actual shake data acquired in a recent period of time, then synchronous displacement compensation is carried out, namely, an image sensor in the camera device is driven to move in the opposite direction simultaneously when being displaced due to shake so as to carry out displacement compensation, and the image sensor is not driven to move in the opposite direction after having larger displacement, so that the shake problem of the camera device is alleviated, the back-and-forth shaking probability of a picture is reduced, and a more stable video picture is provided; in addition, this scheme can be applied to camera anti-shake technical field, especially is arranged in the anti-shake control of the thermal imaging system of super long observation distance.
Referring to fig. 2, fig. 2 is a schematic flowchart illustrating another embodiment of an anti-shake method for an image capturing apparatus according to the present application, the method includes:
s21: a list of jittered data samples is created and the number of samples is initialized to a preset value.
A list of jitter data samples and the number of samples are created, and then the number of samples is initialized, such as: the preset value is 0, namely the number of samples is initialized to 0, and the number of the samples represents the number of training samples acquired by the jitter prediction model; then, the following scheme is adopted to simultaneously perform anti-shake control and model training.
S22: and generating first sub-jitter data based on the actual jitter data of the camera equipment acquired by the offset sensor, and adding the number of samples to a preset step to update the number of samples.
The jitter data sample list comprises a preset number of first sub-jitter data, the camera equipment comprises an image sensor and an offset sensor, actual jitter data of the camera equipment, which is acquired by the offset sensor, can be acquired first, and second sub-jitter data is obtained, wherein the offset sensor can be a three-axis gyroscope; and then carrying out differential processing on the second sub-jitter data to obtain first sub-jitter data.
In a specific embodiment, the second sub-shake data includes an offset value in a target direction, and the image sensor is controlled to move the offset value in a direction opposite to the target direction to perform shake compensation for the image pickup apparatus. Specifically, the target direction includes a first preset direction, a second preset direction and a third preset direction, the offset value includes a first sub offset value, a second sub offset value and a third sub offset value, and the image sensor is controlled to move the first sub offset value in a direction opposite to the first preset direction; controlling the image sensor to move the second sub offset value in a direction opposite to the second preset direction; and controlling the image sensor to move a third sub offset value in a direction opposite to the third preset direction.
S23: and judging whether the number of the samples is less than a preset number.
If the number of the detected samples is greater than or equal to the preset number, the number of the training samples is sufficient, and the collection of the training samples can be finished. If the number of the samples is smaller than the preset number, returning to actual jitter data of the camera equipment acquired based on the offset sensor after a first preset time, and generating first sub-jitter data until the number of the samples is equal to the preset number; the first preset time is a preset time interval, and is generally 1ms.
In one embodiment, the preset step size may be 1, and the preset number may be 100; the first preset direction is an X-axis direction, the second preset direction is a Y-axis direction, and the third preset direction is a Z-axis direction, and the following scheme can be adopted:
1) Acquiring actual shaking data of the image sensor from the three-axis gyroscope to obtain a shaking offset vector (namely an offset value): [ X, Y, Z ], where X is a jitter offset value in the X-axis direction (i.e., a first sub-offset value), Y is a jitter offset value in the Y-axis direction (i.e., a second sub-offset value), and Z is a jitter offset value in the Z-axis direction (i.e., a third sub-offset value).
2) The image sensor is shifted by X in the opposite direction to the X axis (i.e., the direction opposite to the X axis), by Y in the opposite direction to the Y axis (i.e., the direction opposite to the Y axis), and by Z in the opposite direction to the Z axis (i.e., the direction opposite to the Z axis).
3) And storing the jitter offset vector [ x, y, z ] into a jitter data sample list, increasing the number of samples by 1, and judging whether the number of samples is less than 100.
4) If the number of samples is less than 100, skipping to the step 1) after timing a first preset time; if the number of samples is greater than or equal to 100, step 5) is entered.
5) And carrying out differential operation on the jitter offset vector, and storing an operation result into a jitter data sample list.
Because the jitter oscillogram has a certain regularity, when data is processed, the original data (namely the jitter offset vector [ x, y, z ]) is firstly subjected to a differential operation, the difference operation is converted into a stable sequence, and then the processed result is stored in a jitter data sample list to replace the original data.
S24: and training the current jitter prediction model based on the jitter data sample list to obtain the trained jitter prediction model.
The jitter prediction model learns the jitter rule of a recent period of time by using a time sequence neural network, as shown in fig. 3, the jitter prediction model comprises a first prediction module and a second prediction module, and the first prediction module is used for predicting the jitter data of the camera equipment at the next moment based on input data to obtain a sample prediction characteristic; the second prediction module is used for processing the sample prediction characteristics to obtain a jitter prediction value; calculating the loss between the jitter prediction value and the actual jitter data at the next moment to obtain a current loss value; whether the current loss value is converged or is smaller than a set value can be judged, so that whether a preset training end condition is reached is judged, and if yes, a trained jitter prediction model is obtained.
Further, the weight of the neural network can be optimized by adopting an Adaptive Moment Estimation (Adam) algorithm based on a Root Mean Square Error (MSE) loss function, so that the loss value is gradually reduced, and the continuous improvement of the prediction accuracy is realized, wherein the calculation formula of the RMSE is as follows:
wherein x is t For the actual jitter data at time t,
for the jitter prediction value at time t, N is the number of samples.
In a specific embodiment, the first prediction module includes an input layer and a first hidden layer, the first hidden layer includes a plurality of first sub-processing layers, and the input layer inputs data from a three-axis gyroscope, i.e., X-axis data, Y-axis data, and Z-axis data; the second prediction module comprises a second hidden layer, a third hidden layer and an output layer, the second hidden layer comprises a plurality of second sub-processing layers, and the first sub-processing layers correspond to the second sub-processing layers one to one.
Further, the first sub-processing layer is a Long Short-Term Memory (LSTM) recurrent neural network, that is, the first hidden layer includes 3 independent LSTM recurrent neural networks, and each network is responsible for predicting jitter offset data on one channel (there are 3 channels of x, y, and z in total); the second sub-processing layer may employ a drop (dropout) structure that drops neurons with a probability p and leaves other neurons with a probability q (q = 1-p) that is the same for each neuron being closed; because some nodes of the LSTM recurrent neural network are abandoned, the overfitting of the deep neural network can be well avoided, the robustness of the neural network is improved, and p can be 0.2 in the scheme; the third hidden layer is a fully-connected neural network, the number of the neural units in the fully-connected neural network can be 9, and each neuron is connected with the output of all neurons in the previous layer; the output layer outputs jitter prediction values on three axes (X-axis, Y-axis, Z-axis) at the next time.
In actual use, the actual jitter data at the current moment and the actual jitter data (i.e. N historical data) at (N-1) moments before the current moment can be input into the input layer for model training; and after the N historical data are subjected to iterative calculation through an LSTM recurrent neural network, the state of the memory unit is updated, and the jitter trend of the next moment is predicted according to the N historical data.
It can be understood that, at the moment when the image capturing apparatus is just running, the actual shake prediction model cannot output an accurate shake trend due to the lack of actual shake data from the three-axis gyroscope, and therefore, at this stage, the actual shake data is stored in the shake data sample list while shake compensation is performed by using a conventional method, so that the prediction accuracy of the shake prediction model is improved.
After the first stage (i.e., the model training stage), the current jitter prediction model completes pre-training, and at this time, the jitter trend of the next moment predicted by the model can be used to perform synchronous compensation on the impending jitter offset, so the second stage is also referred to as a synchronous anti-jitter stage, and the specific scheme is as follows.
S25: and predicting the jitter data of the camera equipment at the next moment by adopting the trained jitter prediction model to obtain jitter prediction data.
The jitter prediction data comprises a plurality of jitter prediction values, and a first prediction module is adopted to predict the jitter data of the camera equipment at the next moment to obtain jitter offset characteristics; and processing the jitter offset characteristics by adopting a second prediction module to obtain a plurality of jitter prediction values.
Further, the jitter offset characteristic comprises a plurality of first sub-offset characteristics, the first sub-jitter data comprises a plurality of sub-jitter data, and a plurality of first sub-processing layers are adopted to process the plurality of sub-jitter data in the first sub-jitter data respectively to obtain corresponding first sub-offset characteristics; the second sub-processing layer is adopted to carry out random deletion processing on the first sub-offset features output by the corresponding first sub-processing layer, and second sub-offset features are obtained; carrying out full connection processing on the second sub-migration features output by all the second sub-processing layers by adopting a third hidden layer to obtain third sub-migration features; and processing the third sub-offset characteristic by using an output layer to obtain a plurality of jitter predicted values, wherein the plurality of jitter predicted values comprise a first jitter predicted value in a first preset direction, a second jitter predicted value in a second preset direction and a third jitter predicted value in a third preset direction. For example, the jitter offset vector at the next time output by the current jitter prediction module is [ X ', Y', Z '], where X' is the jitter offset value to be generated in the X-axis direction (i.e., the first jitter prediction value) predicted by the current jitter prediction module, Y 'is the jitter offset value to be generated in the Y-axis direction (i.e., the second jitter prediction value), and Z' is the jitter offset value to be generated in the Z-axis direction (i.e., the third jitter prediction value).
S26: and controlling the image sensor to move in the direction opposite to the target direction at a speed corresponding to the jitter prediction value within a second preset time.
Controlling the image sensor to move in a direction opposite to the first preset direction at a first preset speed within a second preset time, wherein the first preset speed is the ratio of the first jitter prediction value to the second preset time; controlling the image sensor to move in a direction opposite to a second preset direction at a second preset speed within a second preset time, wherein the second preset speed is the ratio of the second jitter prediction value to the second preset time; and controlling the image sensor to move in a direction opposite to a third preset direction at a third preset speed within a second preset time, wherein the third preset speed is the ratio of the third jitter predicted value to the second preset time.
For example, if the second preset time is denoted as t, the image sensor is moved to the X axis at a speed (-X '/t), the Y axis at a speed (-Y '/t), and the Z axis at a speed (-Z '/t) while the mechanical shake is generated outside at the time t; the offset compensation is terminated after the time t has elapsed. It will be appreciated that a negative velocity indicates that the displacement compensation is performed in the opposite direction.
S27: and acquiring actual shaking data of the camera shooting equipment within second preset time by adopting the offset sensor to obtain equipment shaking data.
The equipment shaking data comprises shaking deviation values in the X-Z axis direction, actual shaking data in second preset time can be collected from the three-axis gyroscope, and an actual shaking deviation vector [ X ] is obtained c ,y c ,z c ]Wherein x is c Is the actual jitter offset value, y, in the X-axis direction within the second preset time c For the Y-axis square in the second preset timeUpward actual jitter offset value, z c The actual jitter offset value in the Z-axis direction within the second preset time is obtained.
S28: and updating a jitter data sample list based on the equipment jitter data, and updating the current jitter prediction model into a trained jitter prediction model.
Carrying out differential processing on the equipment jitter data to obtain processed equipment jitter data; storing the processed equipment jitter data into a jitter data sample list, deleting the data with the earliest storage time from the jitter data sample list, updating the current jitter prediction model into a trained jitter prediction model, returning to the jitter data sample list, and training the current jitter prediction model, namely optimizing the trained jitter prediction model by adopting the jitter data sample list.
Further, whether the time difference with the next moment reaches a third preset time or not can be judged; if yes, predicting the shake data of the camera equipment at the next moment by adopting the optimized shake prediction model to obtain shake prediction data, namely executing S25. Specifically, data in a latest jitter data sample list is input into a current jitter prediction model, based on a mean square error loss function, the weight of a neural network is optimized by adopting an Adam algorithm, so that a loss function value is gradually reduced, the prediction precision is continuously improved, and after each step of prediction is realized, the latest actual jitter value is used for replacing a jitter prediction value to perform iterative calculation of the next step of prediction.
The method provided by the embodiment predicts the shaking trend of the next moment in advance through the historical data by means of the time sequence prediction capability of the machine learning algorithm, and generates compensation in the opposite direction when external shaking occurs at the next moment, so that shaking is better inhibited, finer-grained anti-shaking control is provided, the harsh requirement of the long-focus thermal imager on anti-shaking is met, and the applicability is improved.
Referring to fig. 4, fig. 4 is a schematic structural diagram of an embodiment of an image capturing apparatus provided in the present application, where the image capturing apparatus 40 includes a memory 41 and a processor 42 that are connected to each other, the memory 41 is used for storing a computer program, and the computer program is used for implementing an anti-shake method of the image capturing apparatus in the foregoing embodiment when being executed by the processor 42.
Referring to fig. 5, fig. 5 is a schematic structural diagram of another embodiment of the image capturing apparatus provided in the present application, and the image capturing apparatus includes an offset sensor module 51, a shake timing prediction module 52, a shake compensation calculation module 53, a shake compensator module 54, and an image sensor module 55.
The offset sensor module 51 is connected to the image sensor module 55, and is configured to acquire a shake data sample list of the image capturing apparatus, where the shake data sample list includes actual shake data at a historical time and actual shake data at a current time; and acquiring actual shaking data of the camera equipment at the next moment to obtain equipment shaking data.
The jitter timing sequence prediction module 52 is connected with the offset sensor module 51, and is used for training a current jitter prediction model based on a jitter data sample list to obtain a trained jitter prediction model; and predicting the jitter data of the camera equipment at the next moment by adopting the trained jitter prediction model to obtain jitter prediction data.
The jitter compensation calculating module 53 is connected to the offset sensor module 51 and the jitter timing predicting module 52, and is configured to calculate a compensation value of the image sensor module 55 at the next time based on the jitter prediction data.
The shake compensator module 54 is connected to the shake compensation calculating module 53, and is configured to generate a control command based on the compensation value output by the shake compensation calculating module 53 and send the control command to the image sensor module 55, so as to adjust the displacement of the image sensor module 55 and achieve shake compensation. Specifically, assuming that the offset value at the next time is [ X, Y, Z ], the shake compensator module 54 shifts the image sensor module 55 by X in the opposite direction to the X axis, Y in the opposite direction to the Y axis, and Y in the opposite direction to the Z axis.
Further, the jitter timing prediction module 52 is further configured to update the jitter data sample list based on the device jitter data, update the current jitter prediction model to the trained jitter prediction model, and return to perform an operation of training the current jitter prediction model based on the jitter data sample list.
The embodiment provides a jitter prediction scheme based on machine learning, which predicts the jitter trend at the next moment according to historical data so as to be used for active lens anti-jitter; because the jitter trend of the next moment is predicted in advance, synchronous compensation is carried out in the reverse direction while jitter occurs, and the jitter can be relieved better.
Referring to fig. 6, fig. 6 is a schematic structural diagram of an embodiment of a computer-readable storage medium 60 provided in the present application, where the computer-readable storage medium is used for storing a computer program 61, and when the computer program 61 is executed by a processor, the computer program is used for implementing an anti-shake method of an image capturing apparatus in the foregoing embodiment.
The computer-readable storage medium 60 may be a server, a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various media capable of storing program codes.
In the several embodiments provided in the present application, it should be understood that the disclosed method and apparatus may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of modules or units is merely a logical division, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed.
Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units may be integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The above description is only an example of the present application, and is not intended to limit the scope of the present application, and all equivalent structures or equivalent processes performed by the present application and the contents of the attached drawings, which are directly or indirectly applied to other related technical fields, are also included in the scope of the present application.
Claims (13)
1. An anti-shake method for an image pickup apparatus, comprising:
acquiring a jitter data sample list of the camera equipment, wherein the jitter data sample list comprises actual jitter data at historical time and actual jitter data at current time;
training a current jitter prediction model based on the jitter data sample list to obtain a trained jitter prediction model;
predicting the jitter data of the camera equipment at the next moment by adopting the trained jitter prediction model to obtain jitter prediction data;
performing shake compensation on the image pickup apparatus at the next timing based on the shake prediction data;
acquiring actual jitter data of the camera equipment at the next moment to obtain equipment jitter data;
updating the jitter data sample list based on the equipment jitter data, updating the current jitter prediction model into the trained jitter prediction model, and returning to the jitter data sample list to train the current jitter prediction model.
2. The anti-shake method according to claim 1, wherein the shake data sample list includes a preset number of first sub-shake data, the image capturing apparatus includes an offset sensor, and the step of acquiring the shake data sample list of the image capturing apparatus includes:
initializing the number of samples to a preset value;
generating the first sub-jitter data based on the actual jitter data of the camera equipment acquired by the offset sensor, and adding the number of samples to a preset step length to update the number of samples;
judging whether the number of the samples is less than a preset number;
if yes, returning to the step of generating the first sub-jitter data based on the actual jitter data of the camera equipment acquired by the offset sensor after a first preset time until the number of samples is equal to the preset number.
3. The anti-shake method for an image pickup apparatus according to claim 2, wherein the step of generating the first sub-shake data based on actual shake data of the image pickup apparatus acquired by the offset sensor includes:
acquiring actual jitter data of the camera equipment acquired by the offset sensor to obtain second sub-jitter data;
and carrying out differential processing on the second sub-jitter data to obtain the first sub-jitter data.
4. The anti-shake method for an image capturing apparatus according to claim 3, wherein the image capturing apparatus further includes an image sensor, the second sub-shake data includes an offset value in a target direction, the method further comprising:
controlling the image sensor to move the offset value in a direction opposite to the target direction to perform shake compensation on the image pickup apparatus.
5. The anti-shake method for an image pickup apparatus according to claim 4, wherein the target direction includes a first preset direction, a second preset direction, and a third preset direction, the offset value includes a first sub-offset value, a second sub-offset value, and a third sub-offset value, and the step of controlling the image sensor to move the offset value in a direction opposite to the target direction includes:
controlling the image sensor to move the first sub offset value in a direction opposite to the first preset direction;
controlling the image sensor to move the second sub offset value in a direction opposite to the second preset direction;
controlling the image sensor to move the third sub offset value in a direction opposite to the third preset direction.
6. The anti-shake method for an image capturing apparatus according to claim 1, wherein the shake prediction data includes a plurality of shake prediction values, the image capturing apparatus further comprising an image sensor, and wherein the step of performing shake compensation on the image capturing apparatus at the next timing based on the shake prediction data includes:
and controlling the image sensor to move in a direction opposite to the target direction at a speed corresponding to the jitter prediction value within a second preset time.
7. The anti-shake method for an image capturing apparatus according to claim 6, wherein the target direction includes a first preset direction, a second preset direction, and a third preset direction, wherein the plurality of shake prediction values include a first shake prediction value in the first preset direction, a second shake prediction value in the second preset direction, and a third shake prediction value in the third preset direction, and wherein the step of controlling the image sensor to move in a direction opposite to the target direction at a speed corresponding to the shake prediction value for a second preset time includes:
controlling the image sensor to move in a direction opposite to the first preset direction at a first preset speed within the second preset time, wherein the first preset speed is the ratio of the first jitter predicted value to the second preset time;
controlling the image sensor to move in a direction opposite to the second preset direction at a second preset speed within the second preset time, wherein the second preset speed is a ratio of the second jitter predicted value to the second preset time;
and controlling the image sensor to move in a direction opposite to the third preset direction at a third preset speed within the second preset time, wherein the third preset speed is the ratio of the third jitter predicted value to the second preset time.
8. The anti-shake method for an image pickup apparatus according to claim 1, wherein the image pickup apparatus includes an offset sensor, and the step of acquiring actual shake data of the image pickup apparatus at the next time to obtain apparatus shake data includes:
acquiring actual shaking data of the camera shooting equipment within second preset time by adopting the offset sensor to obtain shaking data of the equipment;
the step of updating the list of jittered data samples based on the device jittering data comprises:
carrying out differential processing on the equipment jitter data to obtain processed equipment jitter data;
and storing the processed equipment jitter data into the jitter data sample list, and deleting the data with the earliest storage time from the jitter data sample list.
9. The anti-shake method for an image pickup apparatus according to claim 1, further comprising:
judging whether the time difference with the next moment reaches a third preset time or not;
and if so, returning to the trained jitter prediction model, and predicting the jitter data of the camera equipment at the next moment to obtain jitter prediction data.
10. The anti-shake method for an image capturing apparatus according to claim 1, wherein the shake prediction data includes a plurality of shake prediction values, wherein the shake prediction model includes a first prediction module and a second prediction module, and wherein the step of predicting shake data of the image capturing apparatus at a next time using the trained shake prediction model to obtain shake prediction data includes:
predicting the jitter data of the camera equipment at the next moment by adopting the first prediction module to obtain jitter offset characteristics;
and processing the jitter offset characteristics by adopting the second prediction module to obtain a plurality of jitter prediction values.
11. The anti-shake method according to claim 10, wherein the shake offset feature includes a plurality of first sub-shake features, the first sub-shake data includes a plurality of sub-shake data, the first prediction module includes a first hidden layer including a plurality of first sub-processing layers, the second prediction module includes a second hidden layer, an output layer, and a third hidden layer, the second hidden layer includes a plurality of second sub-processing layers, and the first sub-processing layers correspond to the second sub-processing layers one-to-one; the step of predicting the shake data of the image pickup apparatus at the next time by using the first prediction module to obtain the shake deviation characteristic includes:
processing the sub-jitter data by adopting the first sub-processing layers respectively to obtain corresponding first sub-offset characteristics;
the step of processing the jitter offset characteristic by using the second prediction module to obtain the plurality of jitter prediction values includes:
the second sub-processing layer is adopted to carry out random deletion processing on the first sub-offset features output by the corresponding first sub-processing layer, and second sub-offset features are obtained;
carrying out full connection processing on the second sub-migration features output by all the second sub-processing layers by adopting the third hidden layer to obtain third sub-migration features;
and processing the third sub-offset characteristic by adopting the output layer to obtain a plurality of jitter predicted values.
12. An image pickup apparatus comprising a memory and a processor connected to each other, wherein the memory is configured to store a computer program, and the computer program is configured to implement the anti-shake method of the image pickup apparatus according to any one of claims 1 to 11 when executed by the processor.
13. A computer-readable storage medium storing a computer program, wherein the computer program is configured to implement the anti-shake method of the image pickup apparatus according to any one of claims 1 to 11 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210576152.0A CN115242966A (en) | 2022-05-24 | 2022-05-24 | Anti-shake method and device for camera equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210576152.0A CN115242966A (en) | 2022-05-24 | 2022-05-24 | Anti-shake method and device for camera equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115242966A true CN115242966A (en) | 2022-10-25 |
Family
ID=83667802
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210576152.0A Pending CN115242966A (en) | 2022-05-24 | 2022-05-24 | Anti-shake method and device for camera equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115242966A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109671061A (en) * | 2018-12-07 | 2019-04-23 | 深圳美图创新科技有限公司 | A kind of image analysis method, calculates equipment and storage medium at device |
| CN111147757A (en) * | 2020-01-03 | 2020-05-12 | 浙江大华技术股份有限公司 | Optical anti-shake method and device for image pickup equipment |
| CN111371983A (en) * | 2018-12-26 | 2020-07-03 | 清华大学 | Video online stabilization method and system |
| CN111901532A (en) * | 2020-09-30 | 2020-11-06 | 南京理工大学 | Video stabilization method based on recurrent neural network iteration strategy |
| CN112308000A (en) * | 2020-11-06 | 2021-02-02 | 安徽清新互联信息科技有限公司 | High-altitude parabolic detection method based on space-time information |
| CN113163121A (en) * | 2021-04-21 | 2021-07-23 | 安徽清新互联信息科技有限公司 | Video anti-shake method and readable storage medium |
-
2022
- 2022-05-24 CN CN202210576152.0A patent/CN115242966A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109671061A (en) * | 2018-12-07 | 2019-04-23 | 深圳美图创新科技有限公司 | A kind of image analysis method, calculates equipment and storage medium at device |
| CN111371983A (en) * | 2018-12-26 | 2020-07-03 | 清华大学 | Video online stabilization method and system |
| CN111147757A (en) * | 2020-01-03 | 2020-05-12 | 浙江大华技术股份有限公司 | Optical anti-shake method and device for image pickup equipment |
| CN111901532A (en) * | 2020-09-30 | 2020-11-06 | 南京理工大学 | Video stabilization method based on recurrent neural network iteration strategy |
| CN112308000A (en) * | 2020-11-06 | 2021-02-02 | 安徽清新互联信息科技有限公司 | High-altitude parabolic detection method based on space-time information |
| CN113163121A (en) * | 2021-04-21 | 2021-07-23 | 安徽清新互联信息科技有限公司 | Video anti-shake method and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9979889B2 (en) | Combined optical and electronic image stabilization | |
| CN106164730B (en) | The focus adjusting method of focus-regulating device, camera arrangement and photographic device | |
| CN101742098B (en) | Image pickup apparatus and control method thereof | |
| KR101856947B1 (en) | Photographing apparatus, motion estimation apparatus, method for image compensation, method for motion estimation, computer-readable recording medium | |
| CN107026970B (en) | Image stabilizing apparatus and image stabilizing method | |
| US8682040B2 (en) | Image processing apparatus, image processing method, and image capturing apparatus | |
| CN101485199B (en) | Methods and devices for video correction of still camera motion | |
| CN101359147B (en) | Image-blur compensating device and image pickup apparatus | |
| CN105939454B (en) | Picture pick-up device and its control method | |
| CN107135349A (en) | Picture pick-up device, lens unit, camera system and its control method | |
| CN105430245A (en) | Camera device and image shake correction method | |
| CN107888821A (en) | Camera controller, image processing module and semiconductor system | |
| CN101656883B (en) | Real-time compensation method based on motion prediction of least squares support vector machine (LS-SVM) | |
| CN102792673B (en) | Image shake compensation device | |
| CN107231513A (en) | Camera system, picture pick-up device, lens assembly and its control method | |
| WO2022242569A1 (en) | Delay calibration method and apparatus, and computer device and storage medium | |
| JP2021033571A (en) | Information processor, control method and program | |
| CN114338994A (en) | Optical anti-shake method, optical anti-shake apparatus, electronic device, and computer-readable storage medium | |
| Chang et al. | Optical image stabilizing system using multirate fuzzy PID controller for mobile device camera | |
| CN115242966A (en) | Anti-shake method and device for camera equipment and computer readable storage medium | |
| CN116017158A (en) | Optical anti-shake calibration method and device | |
| JPH0720547A (en) | Camera-shake correcting device | |
| JP7256683B2 (en) | Error prediction device and method, anti-vibration control device and method, and imaging device | |
| JP2022142980A (en) | Electronic apparatus, control method, and program | |
| JP2021192498A (en) | Imaging apparatus and control method for the same, program, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |