WO2020207192A1 - Image processor, image processing method, photography apparatus, and electronic device - Google Patents

Abstract

An image processor (10), an image processing method, a photography apparatus (100), and an electronic device (1000). The image processor (10) comprises a hardware abstract module (12) and an algorithm post-processing module (16). The hardware abstract module (12) is used for transmitting image data and metadata corresponding thereto. The algorithm post-processing module (16) is connected to the hardware abstract module (12). At least one image processing algorithm is stored in the algorithm post-processing module (16). The algorithm post-processing module (16) is used for using an image processing algorithm to process the image data according to the metadata to achieve photography post-processing.

Description

图像处理器、图像处理方法、拍摄装置和电子设备Image processor, image processing method, photographing device and electronic equipment

优先权信息Priority information

本申请请求2019年04月10日向中国国家知识产权局提交的、专利申请号为201910285809.6的专利申请的优先权和权益，并且通过参照将其全文并入此处。This application requests the priority and rights of the patent application with the patent application number 201910285809.6 filed with the State Intellectual Property Office of China on April 10, 2019, and the full text is incorporated herein by reference.

技术领域Technical field

本申请涉及图像处理技术领域，更具体而言，涉及一种图像处理器、图像处理方法、拍摄装置和电子设备。This application relates to the field of image processing technology, and more specifically, to an image processor, an image processing method, a photographing device, and electronic equipment.

背景技术Background technique

目前，拍照后处理，例如美颜处理、滤镜处理、旋转处理、水印处理、虚化处理、高动态范围(High-Dynamic Range，HDR)处理、及多帧处理等都是在硬件抽象模块(Hardware Abstract Layer，HAL)中处理，其中，硬件抽象模块通常是由一家厂商提供，进行拍照后处理的图像处理算法则由另一家厂商提供，二者在做兼容设计时，图像处理算法要在硬件抽象模块本身的算法架构上做流程截断，二者耦合导致设计难度大。Currently, post-photographic processing, such as beauty processing, filter processing, rotation processing, watermark processing, blurring processing, High-Dynamic Range (HDR) processing, and multi-frame processing are all performed in the hardware abstraction module ( Hardware Abstract Layer, HAL), where the hardware abstraction module is usually provided by one manufacturer, and the image processing algorithm for post-photographing is provided by another manufacturer. When the two are designed to be compatible, the image processing algorithm must be in the hardware The algorithm architecture of the abstract module itself is truncated, and the coupling of the two makes the design difficult.

发明内容Summary of the invention

本申请实施方式提供一种图像处理器、图像处理方法、拍摄装置和电子设备。The embodiments of the present application provide an image processor, an image processing method, a photographing device, and electronic equipment.

本申请实施方式的图像处理器包括硬件抽象模块和算法后处理模块。所述硬件抽象模块用于传输图像数据和与所述图像数据对应的元数据。所述算法后处理模块与所述硬件抽象模块连接，所述算法后处理模块内存储有至少一种图像处理算法，所述算法后处理模块用于采用所述图像处理算法并根据所述元数据处理所述图像数据以实现拍照后处理。The image processor in the embodiment of the application includes a hardware abstraction module and an algorithm post-processing module. The hardware abstraction module is used to transmit image data and metadata corresponding to the image data. The algorithm post-processing module is connected to the hardware abstraction module, at least one image processing algorithm is stored in the algorithm post-processing module, and the algorithm post-processing module is configured to adopt the image processing algorithm and based on the metadata The image data is processed to achieve post-photographic processing.

本申请实施方式的图像处理方法包括：硬件抽象模块传输图像数据和与所述图像数据对应的元数据；及算法后处理模块采用图像处理算法并根据所述元数据处理所述图像数据以实现拍照后处理。The image processing method of the embodiment of the present application includes: a hardware abstraction module transmits image data and metadata corresponding to the image data; and an algorithm post-processing module adopts an image processing algorithm and processes the image data according to the metadata to realize photographing Post-processing.

本申请实施方式的拍摄装置包括上述图像处理器及图像传感器，所述图像传感器与所述图像处理器连接。The photographing device of the embodiment of the present application includes the above-mentioned image processor and image sensor, and the image sensor is connected to the image processor.

本申请实施方式的电子设备包括上述拍摄装置及壳体，所述拍摄装置与所述壳体结合。The electronic device of the embodiment of the present application includes the aforementioned imaging device and a housing, and the imaging device is combined with the housing.

本申请的实施方式的附加方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本申请的实施方式的实践了解到。The additional aspects and advantages of the embodiments of the present application will be partly given in the following description, and part of them will become obvious from the following description, or be understood through the practice of the embodiments of the present application.

附图说明Description of the drawings

本申请的上述和/或附加的方面和优点从结合下面附图对实施方式的描述中将变得明显和容易理解，其中：The above and/or additional aspects and advantages of the present application will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, in which:

图1和图2是本申请某些实施方式的拍摄装置的示意图；1 and 2 are schematic diagrams of photographing devices according to some embodiments of the present application;

图3是本申请某些实施方式的算法后处理模块的示意图；Fig. 3 is a schematic diagram of an algorithm post-processing module of some embodiments of the present application;

图4是本申请某些实施方式的拍摄装置的示意图；4 is a schematic diagram of a photographing device according to some embodiments of the present application;

图5和图6是本申请某些实施方式的电子设备的结构示意图；5 and 6 are schematic diagrams of the structure of an electronic device according to some embodiments of the present application;

图7至图12是本申请某些实施方式的图像处理方法的流程示意图。7 to 12 are schematic flowcharts of image processing methods in some embodiments of the present application.

具体实施方式detailed description

下面详细描述本申请的实施方式，实施方式的示例在附图中示出，相同或类似的标号自始至终表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施方式是示例性的，仅用于解释本申请，而不能理解为对本申请的限制。The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the drawings, and the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions throughout. The following embodiments described with reference to the accompanying drawings are exemplary, and are only used to explain the present application, and cannot be understood as a limitation to the present application.

下文的公开提供了许多不同的实施方式或例子用来实现本申请的实施方式的不同结构。为了简化本申请的实施方式的公开，下文中对特定例子的部件和设置进行描述。当然，它们仅仅为示例，并且目的不在于限制本申请。The following disclosure provides many different embodiments or examples for realizing different structures of the embodiments of the present application. In order to simplify the disclosure of the embodiments of the present application, the components and settings of specific examples are described below. Of course, they are only examples and are not intended to limit the application.

请参阅图1，拍摄装置100包括图像处理器10和图像传感器20。图像处理器10和图像传感器20连接。图像传感器20包括图像采集单元(sensor)22和RAW图像数据单元(Image Front-end，IFE)24，图像采集单元22用于接收光线以采集获得RAW图像，RAW图像数据单元24用于将图像采集单元22采集的RAW图像传输至图像处理器10，其中，RAW图像数据单元24可以对图像采集单元22采集获得的RAW图像进行处理并输出处理后的RAW图像至图像处理器10。Please refer to FIG. 1, the photographing device 100 includes an image processor 10 and an image sensor 20. The image processor 10 and the image sensor 20 are connected. The image sensor 20 includes an image acquisition unit (sensor) 22 and a RAW image data unit (Image Front-end, IFE) 24. The image acquisition unit 22 is used to receive light to acquire the RAW image, and the RAW image data unit 24 is used to capture the image. The RAW image collected by the unit 22 is transmitted to the image processor 10, where the RAW image data unit 24 can process the RAW image collected by the image collection unit 22 and output the processed RAW image to the image processor 10.

图像处理器10包括硬件抽象模块12和算法后处理模块(Algo process service，APS)16。The image processor 10 includes a hardware abstraction module 12 and an algorithm post-processing module (Algo process service, APS) 16.

硬件抽象模块12用于传输图像数据和与图像数据对应的元数据(metadata)。其中，图像数据可以包括RAW图像和/或YUV图像。具体地，硬件抽象模块12可以用于接收RAW图像、将RAW图像转换为YUV图像、及传输RAW图像和/或YUV图像。硬件抽象模块12可以与图像传感器20连接。硬件抽象模块12可以包括与图像传感器20连接的缓存单元(buffer queue)122、RAW转RGB处理单元(Bayer Process Segment，BPS)124和与应用程序模块14连接的降噪及YUV后处理单元(Image Process Engine，IPE)126。缓存单元122用于缓存来自图像传感器20的RAW图像并传输给算法后处理模块16。RAW转RGB处理单元124用于将来自缓存单元122的RAW图像转换为RGB图像。降噪及YUV后处理单元126用于处理RGB图像得到YUV图像并将YUV图像传输给算法后处理模块16。元数据包括3a(自动曝光控制AE、自动聚焦控制AF、自动白平衡控制AWB)信息、图片信息(例如图像宽度、高度)、曝光参数(光圈大小、快门速度和感光度光圈值)等，可以利用元数据辅助实现对RAW图像和/或YUV图像的拍照后处理(例如包括美颜处理、滤镜处理、旋转处理、水印处理、虚化处理、HDR处理、及多帧处理中的至少一种)。在一个实施例中，元数据包括感光度(ISO)信息，根据感光度信息可以辅助调节RAW图像和/或YUV图像的亮度，从而实现与调节亮度相关的拍照后处理。The hardware abstraction module 12 is used to transmit image data and metadata corresponding to the image data. Wherein, the image data may include RAW images and/or YUV images. Specifically, the hardware abstraction module 12 may be used to receive RAW images, convert RAW images into YUV images, and transmit RAW images and/or YUV images. The hardware abstraction module 12 may be connected with the image sensor 20. The hardware abstraction module 12 may include a buffer queue 122 connected to the image sensor 20, a RAW-to-RGB processing unit (Bayer Process Segment, BPS) 124, and a noise reduction and YUV post-processing unit connected to the application module 14 (Image Process Engine, IPE) 126. The buffer unit 122 is used to buffer the RAW image from the image sensor 20 and transmit it to the algorithm post-processing module 16. The RAW-to-RGB processing unit 124 is used to convert the RAW image from the buffer unit 122 into an RGB image. The noise reduction and YUV post-processing unit 126 is used to process the RGB image to obtain a YUV image and transmit the YUV image to the algorithm post-processing module 16. Metadata includes 3a (automatic exposure control AE, automatic focus control AF, automatic white balance control AWB) information, picture information (such as image width, height), exposure parameters (aperture size, shutter speed, and sensitivity aperture value), etc. Use metadata to assist in post-photographing processing of RAW images and/or YUV images (for example, including at least one of beauty processing, filter processing, rotation processing, watermark processing, blur processing, HDR processing, and multi-frame processing ). In one embodiment, the metadata includes sensitivity (ISO) information, and the brightness of the RAW image and/or YUV image can be assisted to adjust the brightness of the RAW image and/or YUV image according to the sensitivity information, so as to achieve post-photographing processing related to adjusting the brightness.

由于硬件抽象模块12不对图像数据进行拍照后处理，拍照后处理的图像处理算法无需在硬件抽象模块12本身的算法架构上做流程截断，只需在外部做兼容，设计难度减小。Since the hardware abstraction module 12 does not perform post-photographic processing on image data, the image processing algorithm for post-photographic processing does not need to be truncated on the algorithm architecture of the hardware abstraction module 12 itself, and only needs to be externally compatible, which reduces the design difficulty.

在相关技术中，应用程序接口(API)将硬件抽象模块建立成管道(pipeline)的方式，由于管道的创建需要大量的时间和内存，因此在相机启动时需要将相机对应的工作模式用到的所有管道均先创建好，而为了实现各种图像处理算法，一般需要创建大量管道(例如超过三条管道)，这会导致相机的启动需要耗费大量的时间，并且占用大量的内存。本申请实施方式的硬件抽象模块12不对RAW图像和/或YUV图像进行拍照后处理，因此，硬件抽象模块12只需建立少量(例如一条或两条)管道即可，无需建立大量的管道，从而能够节约内存，并且可以使得相机的启动速度变快。In related technologies, the application programming interface (API) builds the hardware abstraction module into a pipeline. Since the creation of the pipeline requires a lot of time and memory, it is necessary to use the camera's corresponding working mode when the camera is started. All pipelines are created first, and in order to implement various image processing algorithms, it is generally necessary to create a large number of pipelines (for example, more than three pipelines), which will cause the camera startup to take a lot of time and take up a lot of memory. The hardware abstraction module 12 in the embodiment of the present application does not post-process the RAW image and/or YUV image. Therefore, the hardware abstraction module 12 only needs to establish a small number (for example, one or two) pipelines, and there is no need to establish a large number of pipelines. Can save memory, and can make the camera start faster.

算法后处理模块16与硬件抽象模块12连接，算法后处理模块16内 存储有至少一种图像处理算法(例如包括美颜处理算法、滤镜处理算法、旋转处理算法、水印处理算法、虚化处理算法、HDR处理算法、及多帧处理算法中的至少一种)，算法后处理模块16用于采用图像处理算法并根据元数据处理图像数据(RAW图像和/或YUV图像)以实现拍照后处理。由于对RAW图像和/或YUV图像进行拍照后处理可由算法后处理模块16实现，从而无需在硬件抽象模块12本身的算法架构上做流程截断，只需在外部做兼容，设计难度减小。并且由于拍照后处理由算法后处理模块16实现，因此算法后处理模块16的功能更单一、更聚焦，从而可以达到移植快，扩展新的图像处理算法简单等效果。The algorithm post-processing module 16 is connected to the hardware abstraction module 12. The algorithm post-processing module 16 stores at least one image processing algorithm (for example, including beauty processing algorithm, filter processing algorithm, rotation processing algorithm, watermark processing algorithm, blurring processing). At least one of algorithms, HDR processing algorithms, and multi-frame processing algorithms), the algorithm post-processing module 16 is configured to adopt image processing algorithms and process image data (RAW images and/or YUV images) according to metadata to implement post-photographing processing . Since the post-processing of the RAW image and/or YUV image can be implemented by the algorithm post-processing module 16, there is no need to perform process truncation on the algorithm architecture of the hardware abstraction module 12 itself, and only need to be externally compatible, which reduces the design difficulty. In addition, since the post-photographing process is implemented by the algorithm post-processing module 16, the function of the algorithm post-processing module 16 is simpler and more focused, which can achieve effects such as faster transplantation and simple extension of new image processing algorithms.

在算法后处理模块16只处理RAW图像(例如图像处理算法针对RAW图像进行处理)时，硬件抽象模块12可以只传输RAW图像(此时可以不需要将RAW图像转换为YUV图像)；在算法后处理模块16只处理YUV图像(例如图像处理算法针对YUV图像进行处理)时，硬件抽象模块12可以只传输YUV图像；在算法后处理模块16处理RAW图像和YUV图像时，硬件抽象模块12可以传输RAW图像和YUV图像。When the post-algorithm processing module 16 only processes RAW images (for example, the image processing algorithm processes RAW images), the hardware abstraction module 12 may only transmit RAW images (there is no need to convert RAW images to YUV images); after the algorithm When the processing module 16 only processes YUV images (for example, when the image processing algorithm processes YUV images), the hardware abstraction module 12 can only transmit YUV images; when the algorithm post-processing module 16 processes RAW images and YUV images, the hardware abstraction module 12 can transmit RAW image and YUV image.

图像处理器10还包括应用程序模块(APP)14。应用程序模块14用于分别连接硬件抽象模块12和算法后处理模块16。本申请实施方式的图像处理器10中，硬件抽象模块12将图像数据和元数据传输至算法后处理模块16时，不需要通过应用程序模块14，使得数据传输更加实时和高效，因此，能够在单位时间内传输更多帧的图像数据，从而可以使得图像预览和录像等功能(需要较高帧率的功能)也无需在硬件抽象模块12本身的算法架构上做流程截断，可以采用算法后处理模块16进行拍照后处理以实现图像预览和录像等功能，进而便于通过图像预览和录像实现场景检测、视频防抖等效果。The image processor 10 also includes an application program module (APP) 14. The application module 14 is used to connect the hardware abstraction module 12 and the algorithm post-processing module 16 respectively. In the image processor 10 of the embodiment of the present application, when the hardware abstraction module 12 transmits image data and metadata to the algorithm post-processing module 16, it does not need to pass through the application module 14, so that the data transmission is more real-time and efficient. More frames of image data are transmitted per unit time, so that functions such as image preview and video recording (functions that require a higher frame rate) do not need to be truncated on the algorithm architecture of the hardware abstraction module 12 itself, and post-processing algorithms can be used The module 16 performs post-photographic processing to realize functions such as image preview and video recording, thereby facilitating scene detection and video anti-shake effects through image preview and video recording.

在某些实施方式中，图像处理器10还包括相机服务模块18，算法后处理模块16通过相机服务模块18与硬件抽象模块12连接。相机服务模块18对图像数据和元数据进行封装并将封装后的图像数据和元数据传输至算法后处理模块16。如此，通过相机服务模块18对图像进行封装，可以提高图像传输的效率，并且能够提高图像传输的安全性。具体地，可以在相机服务模块18的进程中创建两个队列，其中一个是图像数据队列，另一个是元数据队列，通过两个队列将图像数据和同一帧的元数据传输至算法后处理模块16。算法后处理模块16也同样存在两个对应的队列，在 相机启动后就创建这两个队列并用于分别存储图像数据和元数据。另外，应用程序模块14也可以建立接收队列，接收队列用于接收算法后处理模块16处理后的图像数据。In some embodiments, the image processor 10 further includes a camera service module 18, and the algorithm post-processing module 16 is connected to the hardware abstraction module 12 through the camera service module 18. The camera service module 18 encapsulates the image data and metadata and transmits the encapsulated image data and metadata to the algorithm post-processing module 16. In this way, by encapsulating the image through the camera service module 18, the efficiency of image transmission can be improved, and the security of image transmission can be improved. Specifically, two queues can be created in the process of the camera service module 18, one of which is an image data queue and the other is a metadata queue. The image data and metadata of the same frame are transmitted to the algorithm post-processing module through the two queues. 16. The algorithm post-processing module 16 also has two corresponding queues, which are created after the camera is started and used to store image data and metadata respectively. In addition, the application module 14 may also establish a receiving queue, which is used to receive the image data processed by the algorithm post-processing module 16.

应用程序模块14可以通过相机服务模块18与硬件抽象模块12连接，即应用程序模块14可以与相机服务模块18连接。应用程序模块14可以用于向相机服务模块18发起数据请求。具体地，应用程序模块14可以用于根据用户的输入产生数据请求并将该数据请求通过相机服务模块18发送给硬件抽象模块12，硬件抽象模块12再将数据请求发送给图像传感器20，从而以对图像传感器20的工作进行相应的控制。图像传感器20产生与数据请求相对应的RAW图像和元数据后，将RAW图像和元数据传输至硬件抽象模块12，硬件抽象模块12再根据数据请求传输对应的图像数据(RAW图像和/或YUV图像)和元数据至相机服务模块18，相机服务模块18根据数据请求对图像数据和元数据进行封装并将封装后的图像数据和元数据传输至算法后处理模块16。其中，应用程序模块14可以以64比特位(bit)运行，并且拍照后处理的图像处理算法的静态数据连接库(lib)可以配置为64比特位，以提高运算速度。相机服务模块18还可以用于向应用程序模块14反馈数据请求的结果。The application program module 14 can be connected to the hardware abstraction module 12 through the camera service module 18, that is, the application program module 14 can be connected to the camera service module 18. The application module 14 may be used to initiate a data request to the camera service module 18. Specifically, the application module 14 may be used to generate a data request according to the user's input and send the data request to the hardware abstraction module 12 through the camera service module 18, and the hardware abstraction module 12 sends the data request to the image sensor 20, thereby The operation of the image sensor 20 is controlled accordingly. After the image sensor 20 generates the RAW image and metadata corresponding to the data request, it transmits the RAW image and metadata to the hardware abstraction module 12. The hardware abstraction module 12 then transmits the corresponding image data (RAW image and/or YUV) according to the data request. The image) and metadata are sent to the camera service module 18, and the camera service module 18 encapsulates the image data and metadata according to the data request and transmits the encapsulated image data and metadata to the algorithm post-processing module 16. Among them, the application module 14 can run in 64 bits, and the static data connection library (lib) of the image processing algorithm for post-photographing processing can be configured in 64 bits to improve the calculation speed. The camera service module 18 can also be used to feed back the result of the data request to the application module 14.

在某些实施方式中，相机服务模块18可以根据感光度信息、陀螺仪的抖动情况、AR场景检测结果(检测场景类型，例如人物、动物、风景等)等向应用程序模块14发送帧数建议，例如，当陀螺仪检测到的抖动较大时，相机服务模块18向应用程序模块14发送的帧数建议可以是：建议较多帧，以更好地实现拍照后处理；当陀螺仪检测到的抖动较小时，相机服务模块18向应用程序模块14发送的帧数建议可以是：建议较少帧，以减少数据传输量。也即是说，相机服务模块18向应用程序模块14建议的帧数可以与陀螺仪检测到的抖动程度正相关。相机服务模块18还可以根据感光度信息、陀螺仪的抖动情况、AR场景检测结果等向应用程序模块14发送算法建议，例如，当陀螺仪检测到的抖动较大时，相机服务模块18向应用程序模块14发送的算法建议可以是多帧处理，以根据多帧处理消除抖动；当AR场景检测结果检测的场景类型为人物时，相机服务模块18向应用程序模块14发送的算法建议可以是美颜处理，以对人物进行美颜；当AR场景检测结果检测的场景类型为风景时，相机服务模块18向应用程序模块14发送的算法建议可以是HDR处理，以形成高动态范围 的风景图像。应用程序模块14根据帧数建议和算法建议向相机服务模块18发出数据请求。当然，也可以是硬件抽象模块12根据感光度信息、陀螺仪的抖动情况、AR场景检测结果获得帧数建议和/或算法建议，再将帧数建议和/或算法建议通过相机服务模块18传输至应用程序模块14。In some embodiments, the camera service module 18 may send frame recommendations to the application module 14 based on the sensitivity information, the jitter of the gyroscope, the AR scene detection result (detection of scene types, such as people, animals, scenery, etc.), etc. For example, when the jitter detected by the gyroscope is large, the recommended number of frames sent by the camera service module 18 to the application module 14 can be: suggest more frames to better achieve post-photographic processing; when the gyroscope detects When the jitter of is small, the recommended number of frames sent by the camera service module 18 to the application module 14 may be: suggest fewer frames to reduce the amount of data transmission. In other words, the number of frames suggested by the camera service module 18 to the application module 14 can be positively correlated with the degree of jitter detected by the gyroscope. The camera service module 18 can also send algorithm suggestions to the application module 14 according to the sensitivity information, the jitter of the gyroscope, the AR scene detection result, etc., for example, when the jitter detected by the gyroscope is large, the camera service module 18 sends The algorithm suggestion sent by the program module 14 can be multi-frame processing to eliminate jitter according to the multi-frame processing; when the scene type detected by the AR scene detection result is a person, the algorithm suggestion sent by the camera service module 18 to the application module 14 can be beautiful. Color processing to beautify people; when the scene type detected by the AR scene detection result is landscape, the algorithm proposal sent by the camera service module 18 to the application module 14 may be HDR processing to form a landscape image with a high dynamic range. The application module 14 sends a data request to the camera service module 18 according to the frame number recommendation and algorithm recommendation. Of course, the hardware abstraction module 12 can also obtain frame number recommendations and/or algorithm recommendations based on the sensitivity information, gyroscope jitter conditions, and AR scene detection results, and then transmit the frame number recommendations and/or algorithm recommendations through the camera service module 18. To application module 14.

请参阅图2，应用程序模块14包括算法后处理客户端142，算法后处理模块16将处理后的图像数据传输至算法后处理客户端142。算法后处理模块16可以用于对经过算法后处理模块16处理后的图像数据进行进一步的拍照后处理，例如算法后处理模块16进行一些拍照后处理(例如HDR处理、多帧处理等)，算法后处理客户端142进行另外一些拍照后处理(例如美颜处理、滤镜处理、旋转处理、水印处理、虚化处理等)，在算法后处理客户端142对图像数据进行拍照后处理后，可以将图像数据传输至显示模块(SurfaceFlinger)30进行显示。当然，在算法后处理客户端142不需要对图像进行拍照后处理时，算法后处理客户端142也可以直接将经过算法后处理模块16处理后的图像数据传输至显示模块30进行显示。在图像数据为动态图像(例如视频)时，显示模块30包括视频编码单元(media server)32，视频编码单元32用于对图像数据进行编码以形成视频。Referring to FIG. 2, the application module 14 includes an algorithm post-processing client 142, and the algorithm post-processing module 16 transmits the processed image data to the algorithm post-processing client 142. The algorithm post-processing module 16 can be used to perform further post-photographing processing on the image data processed by the algorithm post-processing module 16. For example, the algorithm post-processing module 16 performs some post-photographing processing (such as HDR processing, multi-frame processing, etc.). The post-processing client 142 performs other post-photographing processing (such as beauty processing, filter processing, rotation processing, watermark processing, blurring processing, etc.). After the algorithm post-processing client 142 performs photographing post-processing on the image data, it can The image data is transmitted to the display module (SurfaceFlinger) 30 for display. Of course, when the post-algorithm processing client 142 does not need to post-process the image, the post-algorithm processing client 142 can also directly transmit the image data processed by the algorithm post-processing module 16 to the display module 30 for display. When the image data is a dynamic image (such as a video), the display module 30 includes a video encoding unit (media server) 32, and the video encoding unit 32 is configured to encode the image data to form a video.

若是在算法后处理模块16进行一些拍照后处理(例如HDR处理、多帧处理等)，算法后处理客户端142进行另外一些拍照后处理(例如美颜处理、滤镜处理、旋转处理、水印处理、虚化处理等)，算法后处理客户端142内也可存储有至少一种图像处理算法(例如包括美颜处理算法、滤镜处理算法、旋转处理算法、水印处理算法、虚化处理算法、HDR处理算法、及多帧处理算法中的至少一种)，算法后处理客户端142还用于采用图像处理算法处理图像数据以实现拍照后处理。由于对图像数据进行拍照后处理由算法后处理客户端142与算法后处理模块16实现，从而无需在硬件抽象模块12本身的算法架构上做流程截断，只需在外部做兼容，设计难度同样也是大大减小。If the algorithm post-processing module 16 performs some post-photo processing (such as HDR processing, multi-frame processing, etc.), the algorithm post-processing client 142 performs other post-photo processing (such as beauty processing, filter processing, rotation processing, and watermark processing). , Blur processing, etc.), the algorithm post-processing client 142 may also store at least one image processing algorithm (for example, including beauty processing algorithm, filter processing algorithm, rotation processing algorithm, watermark processing algorithm, blur processing algorithm, At least one of HDR processing algorithm and multi-frame processing algorithm), the algorithm post-processing client 142 is also used to process image data by using an image processing algorithm to implement post-photographing processing. Since the post-processing of the image data is implemented by the algorithm post-processing client 142 and the algorithm post-processing module 16, there is no need to perform process truncation on the algorithm architecture of the hardware abstraction module 12 itself. It only needs to be externally compatible, and the design difficulty is also Greatly reduced.

为了便于用户使用算法后处理模块16中的图像处理算法，可以将算法后处理客户端142做成一个软件开发工具包(SDK)，从而便于第三方开发。In order to facilitate the user to use the image processing algorithm in the algorithm post-processing module 16, the algorithm post-processing client 142 can be made into a software development kit (SDK) to facilitate third-party development.

应用程序模块14和算法后处理模块16之间通信的数据除了图像数据外，还可以包括元数据、交互信息、图像内容等。例如用户点击了应用程序模块14的显示界面的某个区域，应用程序模块14可以将该区域的位置、 被点击的方式(比如单击或者双击)等交互信息传输至算法后处理模块16，从而算法后处理模块16可以根据该交互信息进行运行。又例如算法后处理模块16通过处理图像数据获得图像内容(例如检测到人脸)，则可以将图像数据中存在人脸的信息传输给应用程序模块14，从而应用程序模块14可以将信息告知用户(例如通过显示界面显示“存在人脸”)。In addition to image data, the data communicated between the application module 14 and the algorithm post-processing module 16 may also include metadata, interactive information, image content, and the like. For example, when the user clicks on a certain area of the display interface of the application module 14, the application module 14 can transmit interactive information such as the location of the area and the way the area was clicked (such as a single click or double click) to the algorithm post-processing module 16, thereby The algorithm post-processing module 16 can run according to the interactive information. For another example, the algorithm post-processing module 16 obtains the image content by processing the image data (for example, a face is detected), it can transmit the information of the human face in the image data to the application module 14, so that the application module 14 can inform the user of the information (For example, "faces exist" displayed on the display interface).

在某些实施方式中，算法后处理模块16可以直接将处理后的图像数据传输至显示模块30进行显示。具体地，应用程序模块14可以发送控制指令给算法后处理模块16，算法后处理模块16根据控制指令选择是直接将处理后的图像数据传输至显示模块30进行显示、或是将处理后的图像数据传输至应用程序模块14。如此，算法后处理模块16对图像数据的传输方式更加多样化。In some embodiments, the algorithm post-processing module 16 may directly transmit the processed image data to the display module 30 for display. Specifically, the application module 14 may send a control instruction to the algorithm post-processing module 16, and the algorithm post-processing module 16 selects according to the control instruction whether to directly transmit the processed image data to the display module 30 for display, or to transfer the processed image The data is transferred to the application module 14. In this way, the image data transmission mode of the algorithm post-processing module 16 is more diversified.

图像传感器20进行一次拍摄(曝光成像)后，将RAW图像传输给硬件抽象模块12，在算法后处理模块16接收到与RAW对应的图像数据(RAW图像和/或YUV图像)后，图像传感器20能够进行下一次拍摄、或图像传感器20能够关闭、或应用程序模块14能够关闭、或应用程序模块14能够退出应用界面。由于拍照后处理由算法后处理模块16实现，因此在将拍摄数据传输给算法后处理模块16后，只需要算法后处理模块16就可实现拍照后处理，此时图像传感器20和应用程序模块14可以不参与拍照后处理，因此，图像传感器20能够关闭或执行下一次拍摄，应用程序模块14能够关闭或退出应用界面。如此，拍摄装置100能够实现快拍，并且在算法后处理模块16进行拍照后处理时可以关闭应用程序模块14或退出应用程序界面，从而在电子设备上进行一些其他操作(例如与拍摄装置100无关的操作，比如浏览网页、看视频、打电话等)，从而用户无需花费大量的时间等待拍照后处理的完成，便于用户使用电子设备。After the image sensor 20 takes one shot (exposure imaging), it transmits the RAW image to the hardware abstraction module 12. After the algorithm post-processing module 16 receives the image data (RAW image and/or YUV image) corresponding to the RAW, the image sensor 20 The next shooting can be performed, or the image sensor 20 can be closed, or the application module 14 can be closed, or the application module 14 can exit the application interface. Since the post-photographing processing is implemented by the algorithm post-processing module 16, after the photographing data is transmitted to the algorithm post-processing module 16, only the algorithm post-processing module 16 is required to implement the post-photo processing. At this time, the image sensor 20 and the application module 14 It may not participate in post-photographing processing. Therefore, the image sensor 20 can be closed or the next photographing can be performed, and the application module 14 can be closed or exit the application interface. In this way, the photographing device 100 can realize snapshots, and when the algorithm post-processing module 16 performs photographing post-processing, the application module 14 can be closed or the application program interface can be exited, so as to perform some other operations on the electronic device (for example, irrelevant to the photographing device 100). Operations such as browsing web pages, watching videos, making phone calls, etc.), so that the user does not need to spend a lot of time waiting for the completion of the post-photograph processing, which is convenient for the user to use the electronic device.

算法后处理模块16可以包括编码单元162，编码单元162用于将YUV图像转换为JPG图像(或者JPEG图像等)。具体地，在算法后处理模块14处理的是YUV图像时，编码单元162可以直接对YUV图像进行编码以形成JPG图像，从而提高图像的输出速度。在算法后处理模块14处理的是RAW图像时，算法后处理模块14可以将处理实现拍照后处理的RAW图像经应用程序模块14回传至相机服务模块18，再由相机服务模块18回传至硬件抽象模块12，例如回传至RAW转RGB处理单元124，RAW转RGB处理单元124可以用于将算法后处理模块14处理实现拍照后处理 并经应用程序模块14回传的RAW图像转换为RGB图像，降噪及YUV后处理单元126可以将RGB图像转换为YUV图像，该YUV图像可以再次传输至算法后处理模块16的编码单元162中以将该YUV图像转换为JPG图像。在某些实施方式中，算法后处理模块14也可以将处理实现拍照后处理的RAW图像经应用程序模块14回传至相机服务模块18，再由相机服务模块18回传至缓存单元122，回传的RAW图像经过RAW转RGB处理单元124和降噪及YUV后处理单元126形成YUV图像，再传输至编码单元162以形成JPG图像。另外，算法后处理模块14还可以直接将处理实现拍照后处理的RAW图像经相机服务模块18(不经过应用程序模块14)回传至硬件抽象模块12。在形成JPG图像后，算法后处理模块16可以用于将JPG图像传输至存储器中保存。The algorithm post-processing module 16 may include an encoding unit 162 for converting the YUV image into a JPG image (or JPEG image, etc.). Specifically, when the algorithm post-processing module 14 processes a YUV image, the encoding unit 162 may directly encode the YUV image to form a JPG image, thereby improving the output speed of the image. When the post-algorithm processing module 14 processes RAW images, the post-algorithm processing module 14 can return the RAW images processed to achieve post-photographing processing to the camera service module 18 via the application module 14, and then the camera service module 18 returns to the camera service module 18 The hardware abstraction module 12, for example, returns to the RAW-to-RGB processing unit 124, and the RAW-to-RGB processing unit 124 can be used to process the algorithm post-processing module 14 to realize post-photographing processing and convert the RAW image returned by the application module 14 into RGB The image, noise reduction and YUV post-processing unit 126 can convert the RGB image into a YUV image, and the YUV image can be transmitted to the encoding unit 162 of the algorithm post-processing module 16 again to convert the YUV image into a JPG image. In some embodiments, the algorithm post-processing module 14 may also send the RAW images processed to achieve post-photographing processing to the camera service module 18 via the application module 14, and then the camera service module 18 sends back to the cache unit 122, The transmitted RAW image passes through the RAW-to-RGB processing unit 124 and the noise reduction and YUV post-processing unit 126 to form a YUV image, and then is transmitted to the encoding unit 162 to form a JPG image. In addition, the algorithm post-processing module 14 can also directly send back the RAW images processed for post-photographing to the hardware abstraction module 12 via the camera service module 18 (without the application module 14). After the JPG image is formed, the algorithm post-processing module 16 can be used to transfer the JPG image to the memory for storage.

请参阅图3，算法后处理模块16包括逻辑处理调用层164、算法模块接口层166和算法处理层168。逻辑处理调用层164用于与应用程序模块14通信。算法模块接口层166用于维护算法接口。算法处理层168包括至少一种图像处理算法。算法模块接口层166用于通过算法接口对算法处理层168的图像处理算法进行注册、注销、调用和回调中的至少一种操作。Referring to FIG. 3, the algorithm post-processing module 16 includes a logic processing calling layer 164, an algorithm module interface layer 166, and an algorithm processing layer 168. The logic processing call layer 164 is used to communicate with the application program module 14. The algorithm module interface layer 166 is used to maintain the algorithm interface. The algorithm processing layer 168 includes at least one image processing algorithm. The algorithm module interface layer 166 is used to perform at least one operation of registration, deregistration, invocation, and callback on the image processing algorithm of the algorithm processing layer 168 through the algorithm interface.

逻辑处理调用层164可以包括线程队列，算法后处理模块16接收到RAW图像和/或YUV图像的拍照后处理任务后，可以将拍照后处理任务缓存在线程队列中处理，其中，线程队列可以缓存多个拍照后处理任务，如此，可以通过逻辑处理调用层164实现快拍(即快拍机制)。另外，逻辑处理调用层164也可以接收应用程序模块14发送的初始化(init)、进程(process)等指令，并将对应的指令和数据保存到线程队列中。逻辑处理调用层164根据线程队列中的任务进行具体逻辑的调用(即具体逻辑调用组合)。逻辑处理调用层164还可以将处理获得的缩略图(thumbnail)回传给应用程序模块14进行显示(即缩略图回显)。在本申请的实施方式的描述中，“多个”的含义是两个或两个以上，除非另有明确具体的限定。The logic processing call layer 164 may include a thread queue. After the algorithm post-processing module 16 receives the post-photographing task of the RAW image and/or YUV image, it can buffer the post-photograph processing task in the thread queue for processing, where the thread queue can be cached Multiple post-photographing processing tasks, in this way, a snapshot (ie a snapshot mechanism) can be implemented through the logic processing call layer 164. In addition, the logic processing call layer 164 may also receive instructions such as initialization (init) and process (process) sent by the application module 14, and save the corresponding instructions and data in the thread queue. The logic processing call layer 164 performs specific logic calls (ie, specific logic call combinations) according to the tasks in the thread queue. The logic processing calling layer 164 may also return the processed thumbnail to the application module 14 for display (ie, thumbnail echo). In the description of the embodiments of the present application, "plurality" means two or more, unless otherwise clearly defined.

算法模块接口层166用于调用算法接口，调用命令也可以保存到线程队列中，算法处理层168在接收到线程队列的调用命令时，即可解析调用命令的参数获得需要调用的图像处理算法。算法模块接口层166对图像处理算法进行注册时，可以在算法处理层168中新增一种图像处理算法；算法模块接口层166对图像处理算法进行注销时，可以删除算法处理层168中一种图像处理算法；算法模块接口层166对图像处理算法进行调用时，可以调用算法 处理层168中一种图像处理算法；算法模块接口层166对图像处理算法进行回调时，可以将算法处理后的数据和状态回传给应用程序模块14。其中，可以采用统一接口实现图像处理算法的注册、注销、调用、回调等操作。算法处理层168中的每一种图像处理算法都是独立的，如此可以方便对图像处理算法实现注册、注销、调用、回调等操作。The algorithm module interface layer 166 is used to call the algorithm interface, and the call command can also be saved in the thread queue. When the algorithm processing layer 168 receives the call command of the thread queue, it can parse the parameters of the call command to obtain the image processing algorithm to be called. When the algorithm module interface layer 166 registers an image processing algorithm, it can add an image processing algorithm to the algorithm processing layer 168; when the algorithm module interface layer 166 cancels the image processing algorithm, it can delete one of the algorithm processing layers 168 Image processing algorithm; when the algorithm module interface layer 166 calls the image processing algorithm, it can call an image processing algorithm in the algorithm processing layer 168; when the algorithm module interface layer 166 calls back the image processing algorithm, it can use the data processed by the algorithm And the status is sent back to the application module 14. Among them, a unified interface can be used to implement operations such as registration, cancellation, invocation, and callback of image processing algorithms. Each image processing algorithm in the algorithm processing layer 168 is independent, so that operations such as registration, deregistration, calling, and callback can be conveniently implemented for the image processing algorithm.

请参阅图4，在某些实施方式中，硬件抽象模块12与算法后处理模块16直接连接，硬件抽象模块12直接将图像数据和元数据传输至算法后处理模块16。如此，能够以最短的传输路径传输图像数据和元数据，从而保证数据传输的实时性和高效性。此时，应用程序模块14可以与硬件抽象模块12直接连接。在图像处理器10不包括相机服务模块18时，图像处理器10中的数据(图像数据、元数据等)传输的路径可以进行适应性调整，即硬件抽象模块12与算法后处理模块16可以直接通信，硬件抽象模块12与应用程序模块14也可以直接通信。例如硬件抽象模块12直接将图像数据和元数据传输至算法后处理模块16。又例如硬件抽象模块12直接将帧数建议和/或算法建议发送给应用程序模块14，应用程序模块14直接将数据请求发送至硬件抽象模块12。又例如硬件抽象模块还可以用于向应用程序模块14反馈数据请求的结果。4, in some embodiments, the hardware abstraction module 12 is directly connected to the algorithm post-processing module 16, and the hardware abstraction module 12 directly transmits image data and metadata to the algorithm post-processing module 16. In this way, image data and metadata can be transmitted with the shortest transmission path, thereby ensuring the real-time and high efficiency of data transmission. At this time, the application module 14 can be directly connected to the hardware abstraction module 12. When the image processor 10 does not include the camera service module 18, the data (image data, metadata, etc.) transmission path in the image processor 10 can be adaptively adjusted, that is, the hardware abstraction module 12 and the algorithm post-processing module 16 can be directly For communication, the hardware abstraction module 12 and the application module 14 can also communicate directly. For example, the hardware abstraction module 12 directly transmits image data and metadata to the algorithm post-processing module 16. For another example, the hardware abstraction module 12 directly sends the frame number recommendation and/or algorithm recommendation to the application module 14, and the application module 14 directly sends the data request to the hardware abstraction module 12. For another example, the hardware abstraction module can also be used to feed back the result of the data request to the application module 14.

请结合图2，硬件抽象模块12和算法后处理模块16可以设置在用户(vendor)分区，应用程序模块14、相机服务模块18和显示模块30可以设置在***(system)分区。其中，用户分区和***分区可以是指存储元件的分区。由于算法后处理模块16设置在用户分区，因此能够使得***分区不做大的改动和变化，并且在***分区中的模块(例如应用程序模块14、相机服务模块18或显示模块30等)进行升级或其他变动时，算法后处理模块16不会影响升级的正常进行，升级后的模块也不会对算法后处理模块16造成干扰。在硬件抽象模块12直接将图像数据和元数据传输至算法后处理模块16时，由于硬件抽象模块12和算法后处理模块16设置在同一分区，因此通信相对方便，能够提高数据传输的效率。Referring to FIG. 2, the hardware abstraction module 12 and the algorithm post-processing module 16 can be set in the vendor partition, and the application module 14, the camera service module 18 and the display module 30 can be set in the system partition. Among them, user partition and system partition may refer to partitions of storage elements. Since the algorithm post-processing module 16 is set in the user partition, it can make the system partition not make major changes and changes, and the modules in the system partition (such as the application module 14, the camera service module 18 or the display module 30, etc.) are upgraded Or other changes, the algorithm post-processing module 16 will not affect the normal progress of the upgrade, and the upgraded module will not interfere with the algorithm post-processing module 16. When the hardware abstraction module 12 directly transmits image data and metadata to the algorithm post-processing module 16, since the hardware abstraction module 12 and the algorithm post-processing module 16 are arranged in the same partition, the communication is relatively convenient and the efficiency of data transmission can be improved.

请参阅图5和图6，电子设备1000包括上述任意一种实施方式的拍摄装置100和壳体200，拍摄装置100与壳体200结合。壳体200可以作为电子设备1000的功能元件的安装载体。壳体200可以为功能元件提供防尘、防摔、防水等保护，功能元件可以是显示屏、拍摄装置100、受话器等。其中，在一个实施方式中，壳体200包括主体210和可动支架220， 可动支架220在驱动装置的驱动下可以相对于主体210运动，例如可动支架220可以相对于主210体滑动，以滑入主体210(例如图5的状态)或从主体210滑出(例如图6的状态)。部分功能元件可以安装在主体210上，另一部分功能元件(例如拍摄装置100)可以安装在可动支架220上，可动支架220运动可带动该另一部分功能元件缩回主体210内或从主体210中伸出。在另一个实施方式中，壳体200上开设有采集窗口，拍摄装置100与采集窗口对准安装以使拍摄装置100能够通过采集窗口接收外界光线以形成图像。Referring to FIGS. 5 and 6, the electronic device 1000 includes the photographing device 100 and the housing 200 of any one of the above embodiments, and the photographing device 100 is combined with the housing 200. The housing 200 can be used as a mounting carrier for the functional elements of the electronic device 1000. The housing 200 may provide protections such as dustproof, anti-drop, and waterproof for the functional elements, and the functional elements may be a display screen, the camera 100, a receiver, and the like. Wherein, in one embodiment, the housing 200 includes a main body 210 and a movable bracket 220. The movable bracket 220 can move relative to the main body 210 under the driving of the driving device. For example, the movable bracket 220 can slide relative to the main body 210. To slide into the main body 210 (for example, the state of FIG. 5) or slide out from the main body 210 (for example, the state of FIG. 6). Part of the functional element can be installed on the main body 210, and another part of the functional element (for example, the camera 100) can be installed on the movable bracket 220, and the movement of the movable bracket 220 can drive the other part of the functional element to retract into or from the main body 210. Stretched out. In another embodiment, a collection window is opened on the housing 200, and the shooting device 100 is aligned and installed with the collection window so that the shooting device 100 can receive external light through the collection window to form an image.

在本申请的实施方式的描述中，需要说明的是，除非另有明确的规定和限定，术语“安装”应做广义理解，例如，可以是固定连接，也可以是可拆卸连接，或一体地连接；可以是机械连接，也可以是电连接或可以相互通讯；可以是直接相连，也可以通过中间媒介间接相连，可以是两个元件内部的连通或两个元件的相互作用关系。对于本领域的普通技术人员而言，可以根据具体情况理解上述术语在本申请的实施方式中的具体含义。In the description of the embodiments of this application, it should be noted that, unless otherwise clearly defined and defined, the term “installation” should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral Connection; It can be mechanical connection, it can be electrical connection or it can communicate with each other; it can be directly connected or indirectly connected through an intermediate medium, it can be the internal communication of two components or the interaction relationship between two components. For those of ordinary skill in the art, the specific meanings of the above-mentioned terms in the embodiments of the present application can be understood according to specific circumstances.

请参阅图1和图7，图像处理方法包括：Please refer to Figure 1 and Figure 7, the image processing methods include:

01：硬件抽象模块12传输图像数据和与图像数据对应的元数据；及02：算法后处理模块16采用图像处理算法并根据元数据处理图像数据以实现拍照后处理。01: The hardware abstraction module 12 transmits image data and metadata corresponding to the image data; and 02: The algorithm post-processing module 16 adopts an image processing algorithm and processes the image data according to the metadata to implement post-photographing processing.

本发明实施方式的图像处理方法可以用于本发明实施方式的图像处理器10，或者说，本发明实施方式的图像处理方法可以由本发明实施方式的图像处理器10实现，其中，步骤01可以由硬件抽象模块12实现，步骤02可以由算法后处理模块16实现。The image processing method of the embodiment of the present invention can be used in the image processor 10 of the embodiment of the present invention. In other words, the image processing method of the embodiment of the present invention can be implemented by the image processor 10 of the embodiment of the present invention. The hardware abstraction module 12 is implemented, and step 02 can be implemented by the algorithm post-processing module 16.

请参阅图1和图8，图像处理方法还包括：Please refer to Figure 1 and Figure 8, the image processing method also includes:

03：相机服务模块18对图像数据和元数据进行封装并将封装后的图像数据和元数据传输至算法后处理模块16。03: The camera service module 18 encapsulates the image data and metadata and transmits the encapsulated image data and metadata to the algorithm post-processing module 16.

其中，步骤03可以由相机服务模块18实现。Among them, step 03 can be implemented by the camera service module 18.

请参阅图1和图9，图像处理方法还包括：Please refer to Figure 1 and Figure 9, the image processing method also includes:

04：应用程序模块14向相机服务模块18发起数据请求；04: The application module 14 initiates a data request to the camera service module 18;

步骤03包括： Step 03 includes:

032：相机服务模块18根据数据请求对图像数据和元数据进行封装并将封装后的图像数据和元数据传输至算法后处理模块16。032: The camera service module 18 encapsulates the image data and metadata according to the data request, and transmits the encapsulated image data and metadata to the algorithm post-processing module 16.

其中，步骤04可以由应用程序模块14实现，步骤032可以由相机服 务模块18实现。Among them, step 04 can be implemented by the application module 14, and step 032 can be implemented by the camera service module 18.

请参阅图1和图10，图像处理方法还包括：Please refer to Figure 1 and Figure 10, the image processing method also includes:

05：算法后处理模块16将处理后的图像数据传输至算法后处理客户端142；05: The algorithm post-processing module 16 transmits the processed image data to the algorithm post-processing client 142;

06：算法后处理客户端142对处理后的图像数据进行处理。06: The algorithm post-processing client 142 processes the processed image data.

其中，步骤05可以由算法后处理模块16实现，步骤06可以由算法后处理客户端142实现。Among them, step 05 can be implemented by the algorithm post-processing module 16, and step 06 can be implemented by the algorithm post-processing client 142.

请参阅图2和图11，图像处理方法还包括：Please refer to Figure 2 and Figure 11, the image processing method also includes:

05：算法后处理模块16将处理后的图像数据传输至算法后处理客户端142；05: The algorithm post-processing module 16 transmits the processed image data to the algorithm post-processing client 142;

07：算法后处理客户端142将处理后的图像数据传输至显示模块30进行显示。07: The algorithm post-processing client 142 transmits the processed image data to the display module 30 for display.

其中，步骤05可以由算法后处理模块16实现，步骤07可以由算法后处理客户端142实现。Among them, step 05 can be implemented by the algorithm post-processing module 16, and step 07 can be implemented by the algorithm post-processing client 142.

请参阅图2和图12，图像处理方法还包括：Please refer to Figure 2 and Figure 12, the image processing method also includes:

08：算法后处理模块16将处理后的图像数据传输至显示模块30进行显示。08: The algorithm post-processing module 16 transmits the processed image data to the display module 30 for display.

其中，步骤08可以由算法后处理模块16实现。Among them, step 08 can be implemented by the algorithm post-processing module 16.

上述实施方式中对图像处理器10的解释，也适用于本发明实施方式的图像处理方法，在此不再赘述。The explanation of the image processor 10 in the foregoing embodiment is also applicable to the image processing method of the embodiment of the present invention, and will not be repeated here.

流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为，表示包括一个或更多个用于实现特定逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分，并且本申请的优选实施方式的范围包括另外的实现，其中可以不按所示出或讨论的顺序，包括根据所涉及的功能按基本同时的方式或按相反的顺序，来执行功能，这应被本申请的实施例所属技术领域的技术人员所理解。Any process or method description in the flowchart or described in other ways herein can be understood as a module, segment, or part of code that includes one or more executable instructions for implementing specific logical functions or steps of the process , And the scope of the preferred embodiments of the present application includes additional implementations, which may not be in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order according to the functions involved. This should It is understood by those skilled in the art to which the embodiments of this application belong.

在流程图中表示或在此以其他方式描述的逻辑和/或步骤，例如，可以被认为是用于实现逻辑功能的可执行指令的定序列表，可以具体实现在任何计算机可读介质中，以供指令执行***、装置或设备(如基于计算机的***、包括处理模块的***或其他可以从指令执行***、装置或设备取指令并执行指令的***)使用，或结合这些指令执行***、装置或设备而使用。就本说明书而言，"计算机可读介质"可以是任何可以包含、存储、通信、 传播或传输程序以供指令执行***、装置或设备或结合这些指令执行***、装置或设备而使用的装置。计算机可读介质的更具体的示例(非穷尽性列表)包括以下：具有一个或多个布线的电连接部(控制方法)，便携式计算机盘盒(磁装置)，随机存取存储器(RAM)，只读存储器(ROM)，可擦除可编辑只读存储器(EPROM或闪速存储器)，光纤装置，以及便携式光盘只读存储器(CDROM)。另外，计算机可读介质甚至可以是可在其上打印所述程序的纸或其他合适的介质，因为可以例如通过对纸或其他介质进行光学扫描，接着进行编辑、解译或必要时以其他合适方式进行处理来以电子方式获得所述程序，然后将其存储在计算机存储器中。The logic and/or steps represented in the flowchart or described in other ways herein, for example, can be considered as a sequenced list of executable instructions for implementing logic functions, and can be embodied in any computer-readable medium, For use by instruction execution systems, devices, or equipment (such as computer-based systems, systems including processing modules, or other systems that can fetch and execute instructions from instruction execution systems, devices, or equipment), or combine these instruction execution systems, devices Or equipment. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, device, or device or in combination with these instruction execution systems, devices, or devices. More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connections with one or more wiring (control method), portable computer disk cases (magnetic devices), random access memory (RAM), Read only memory (ROM), erasable and editable read only memory (EPROM or flash memory), fiber optic devices, and portable compact disk read only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable media on which the program can be printed, because it can be used, for example, by optically scanning the paper or other media, and then editing, interpreting, or other suitable media if necessary. The program is processed in a manner to obtain the program electronically and then stored in the computer memory.

应当理解，本申请的实施方式的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中，多个步骤或方法可以用存储在存储器中且由合适的指令执行***执行的软件或固件来实现。例如，如果用硬件来实现，和在另一实施方式中一样，可用本领域公知的下列技术中的任一项或他们的组合来实现：具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路，具有合适的组合逻辑门电路的专用集成电路，可编程门阵列(PGA)，现场可编程门阵列(FPGA)等。It should be understood that each part of the embodiments of the present application may be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if it is implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of the following technologies known in the art: a logic gate circuit for implementing logic functions on data signals Discrete logic circuits, application-specific integrated circuits with suitable combinational logic gates, programmable gate array (PGA), field programmable gate array (FPGA), etc.

本技术领域的普通技术人员可以理解实现上述实施例方法携带的全部或部分步骤是可以通过程序来指令相关的硬件完成，所述的程序可以存储于一种计算机可读存储介质中，该程序在执行时，包括方法实施例的步骤之一或其组合。Those of ordinary skill in the art can understand that all or part of the steps carried in the method of the foregoing embodiments can be implemented by a program instructing relevant hardware to complete. The program can be stored in a computer-readable storage medium. When executed, it includes one of the steps of the method embodiment or a combination thereof.

此外，在本申请的各个实施例中的各功能单元可以集成在一个处理模块中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。所述集成的模块如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or software functional modules. If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it may also be stored in a computer readable storage medium.

上述提到的存储介质可以是只读存储器，磁盘或光盘等。The aforementioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

在本说明书的描述中，参考术语“某些实施方式”等的描述意指结合所述实施方式或示例描述的具体特征、结构或者特点包含于本申请的至少一个实施方式中。在本说明书中，对上述术语的示意性表述不一定指的是相同的实施方式。而且，描述的具体特征、结构或者特点可以在任何的一个或多个实施方式中以合适的方式结合。In the description of this specification, the description with reference to the term "certain embodiments" and the like means that a specific feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment of the application. In this specification, the schematic representation of the above-mentioned terms does not necessarily refer to the same embodiment. Moreover, the described specific features, structures or characteristics can be combined in any one or more embodiments in an appropriate manner.

尽管上面已经示出和描述了本申请的实施方式，可以理解的是，上述实施方式是示例性的，不能理解为对本申请的限制，本领域的普通技术人员在本申请的范围内可以对上述实施方式进行变化、修改、替换和变型。Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limitations on the present application. Those of ordinary skill in the art can comment on the above within the scope of the present application. The implementation is subject to changes, modifications, replacements and modifications.

Claims (20)

1. 一种图像处理器，其特征在于，所述图像处理器包括：An image processor, characterized in that the image processor includes:

   硬件抽象模块，所述硬件抽象模块用于传输图像数据和与所述图像数据对应的元数据；及A hardware abstraction module, which is used to transmit image data and metadata corresponding to the image data; and

   算法后处理模块，所述算法后处理模块与所述硬件抽象模块连接，所述算法后处理模块内存储有至少一种图像处理算法，所述算法后处理模块用于采用所述图像处理算法并根据所述元数据处理所述图像数据以实现拍照后处理。An algorithm post-processing module, the algorithm post-processing module is connected to the hardware abstraction module, at least one image processing algorithm is stored in the algorithm post-processing module, and the algorithm post-processing module is used to adopt the image processing algorithm and The image data is processed according to the metadata to implement post-photographing processing.
2. 根据权利要求1所述的图像处理器，其特征在于，所述图像处理器还包括应用程序模块，所述应用程序模块用于分别连接所述硬件抽象模块和所述算法后处理模块。The image processor according to claim 1, wherein the image processor further comprises an application program module, and the application program module is used to connect the hardware abstraction module and the algorithm post-processing module respectively.
3. 根据权利要求2所述的图像处理器，其特征在于，所述图像处理器还包括相机服务模块，所述算法后处理模块通过所述相机服务模块与所述硬件抽象模块连接，所述相机服务模块用于对所述图像数据和所述元数据进行封装并将封装后的所述图像数据和所述元数据传输至所述算法后处理模块。The image processor according to claim 2, wherein the image processor further comprises a camera service module, the algorithm post-processing module is connected to the hardware abstraction module through the camera service module, and the camera service The module is used to encapsulate the image data and the metadata and transmit the encapsulated image data and the metadata to the algorithm post-processing module.
4. 根据权利要求3所述的图像处理器，其特征在于，所述应用程序模块与所述相机服务模块连接，所述应用程序模块用于向所述相机服务模块发起数据请求，所述相机服务模块用于根据所述数据请求对所述图像数据和所述元数据进行封装并将封装后的所述图像数据和所述元数据传输至所述算法后处理模块。The image processor according to claim 3, wherein the application program module is connected to the camera service module, and the application program module is used to initiate a data request to the camera service module, and the camera service module It is used for encapsulating the image data and the metadata according to the data request and transmitting the encapsulated image data and the metadata to the algorithm post-processing module.
5. 根据权利要求2或3所述的图像处理器，其特征在于，所述应用程序模块包括算法后处理客户端，所述算法后处理模块用于将处理后的所述图像数据传输至所述算法后处理客户端；The image processor according to claim 2 or 3, wherein the application module comprises an algorithm post-processing client, and the algorithm post-processing module is used to transmit the processed image data to the algorithm Post-processing client;

   所述算法后处理客户端用于对处理后的所述图像数据进行处理，或所述算法后处理客户端用于将处理后的所述图像数据传输至显示模块进行显示。The algorithm post-processing client is used to process the processed image data, or the algorithm post-processing client is used to transmit the processed image data to the display module for display.
6. 根据权利要求1至3中任意一项所述的图像处理器，其特征在于，所述硬件抽象模块和所述算法后处理模块设置在用户分区。The image processor according to any one of claims 1 to 3, wherein the hardware abstraction module and the algorithm post-processing module are arranged in a user partition.
7. 根据权利要求1至3中任意一项所述的图像处理器，其特征在于，所述算法后处理模块用于将处理后的所述图像数据传输至显示模块进行显示。The image processor according to any one of claims 1 to 3, wherein the algorithm post-processing module is used to transmit the processed image data to a display module for display.
8. 一种图像处理方法，其特征在于，所述图像处理方法包括：An image processing method, characterized in that the image processing method includes:

   硬件抽象模块传输图像数据和与所述图像数据对应的元数据；及The hardware abstraction module transmits image data and metadata corresponding to the image data; and

   算法后处理模块采用图像处理算法并根据所述元数据处理所述图像数据以实现拍照后处理。The algorithm post-processing module adopts an image processing algorithm and processes the image data according to the metadata to implement post-photographing processing.
9. 根据权利要求8所述的图像处理方法，其特征在于，所述图像处理方法还包括：8. The image processing method of claim 8, wherein the image processing method further comprises:

   相机服务模块对所述图像数据和所述元数据进行封装并将封装后的所述图像数据和所述元数据传输至算法后处理模块。The camera service module encapsulates the image data and the metadata and transmits the encapsulated image data and the metadata to the algorithm post-processing module.
10. 根据权利要求9所述的图像处理方法，其特征在于，所述图像处理方法还包括：9. The image processing method according to claim 9, wherein the image processing method further comprises:

    应用程序模块向所述相机服务模块发起数据请求；The application program module initiates a data request to the camera service module;

    所述相机服务模块对所述图像数据和所述元数据进行封装并将封装后的所述图像数据和所述元数据传输至算法后处理模块的步骤，包括：The step of the camera service module encapsulating the image data and the metadata and transmitting the encapsulated image data and the metadata to the algorithm post-processing module includes:

    所述相机服务模块根据所述数据请求对所述图像数据和所述元数据进行封装并将封装后的所述图像数据和所述元数据传输至所述算法后处理模块。The camera service module encapsulates the image data and the metadata according to the data request and transmits the encapsulated image data and the metadata to the algorithm post-processing module.
11. 根据权利要求8所述的图像处理方法，其特征在于，所述图像处理方法还包括：8. The image processing method of claim 8, wherein the image processing method further comprises:

    所述算法后处理模块将处理后的所述图像数据传输至算法后处理客户端；The algorithm post-processing module transmits the processed image data to the algorithm post-processing client;

    所述算法后处理客户端对处理后的所述图像数据进行处理，或所述算 法后处理客户端将处理后的所述图像数据传输至显示模块进行显示。The algorithm post-processing client processes the processed image data, or the algorithm post-processing client transmits the processed image data to the display module for display.
12. 根据权利要求8所述的图像处理方法，其特征在于，所述图像处理方法还包括：8. The image processing method of claim 8, wherein the image processing method further comprises:

    所述算法后处理模块将处理后的所述图像数据传输至显示模块进行显示。The algorithm post-processing module transmits the processed image data to the display module for display.
13. 一种拍摄装置，其特征在于，所述拍摄装置包括：A photographing device, characterized in that the photographing device comprises:

    图像处理器；及Image processor; and

    图像传感器，所述图像传感器与所述图像处理器连接；An image sensor, the image sensor is connected to the image processor;

    所述图像处理器包括：The image processor includes:

    硬件抽象模块，所述硬件抽象模块用于传输图像数据和与所述图像数据对应的元数据；及A hardware abstraction module, which is used to transmit image data and metadata corresponding to the image data; and

    算法后处理模块，所述算法后处理模块与所述硬件抽象模块连接，所述算法后处理模块内存储有至少一种图像处理算法，所述算法后处理模块用于采用所述图像处理算法并根据所述元数据处理所述图像数据以实现拍照后处理。An algorithm post-processing module, the algorithm post-processing module is connected to the hardware abstraction module, at least one image processing algorithm is stored in the algorithm post-processing module, and the algorithm post-processing module is used to adopt the image processing algorithm and The image data is processed according to the metadata to implement post-photographing processing.
14. 根据权利要求13所述的拍摄装置，其特征在于，所述图像处理器还包括应用程序模块，所述应用程序模块用于分别连接所述硬件抽象模块和所述算法后处理模块。The photographing device according to claim 13, wherein the image processor further comprises an application program module, and the application program module is used to connect the hardware abstraction module and the algorithm post-processing module respectively.
15. 根据权利要求14所述的拍摄装置，其特征在于，所述图像处理器还包括相机服务模块，所述算法后处理模块通过所述相机服务模块与所述硬件抽象模块连接，所述相机服务模块用于对所述图像数据和所述元数据进行封装并将封装后的所述图像数据和所述元数据传输至所述算法后处理模块。The photographing device according to claim 14, wherein the image processor further comprises a camera service module, the algorithm post-processing module is connected to the hardware abstraction module through the camera service module, and the camera service module It is used to encapsulate the image data and the metadata and transmit the encapsulated image data and the metadata to the algorithm post-processing module.
16. 根据权利要求15所述的拍摄装置，其特征在于，所述应用程序模块与所述相机服务模块连接，所述应用程序模块用于向所述相机服务模块发起数据请求，所述相机服务模块用于根据所述数据请求对所述图像数据和所述元数据进行封装并将封装后的所述图像数据和所述元数据传输至 所述算法后处理模块。The photographing device according to claim 15, wherein the application module is connected to the camera service module, and the application module is used to initiate a data request to the camera service module, and the camera service module uses According to the data request, the image data and the metadata are packaged, and the packaged image data and the metadata are transmitted to the algorithm post-processing module.
17. 根据权利要求14或15所述的拍摄装置，其特征在于，所述应用程序模块包括算法后处理客户端，所述算法后处理模块用于将处理后的所述图像数据传输至所述算法后处理客户端；The photographing device according to claim 14 or 15, wherein the application module comprises an algorithm post-processing client, and the algorithm post-processing module is used to transmit the processed image data to the algorithm post Handle the client;

    所述算法后处理客户端用于对处理后的所述图像数据进行处理，或所述算法后处理客户端用于将处理后的所述图像数据传输至显示模块进行显示。The algorithm post-processing client is used to process the processed image data, or the algorithm post-processing client is used to transmit the processed image data to the display module for display.
18. 根据权利要求13至15中任意一项所述的拍摄装置，其特征在于，所述硬件抽象模块和所述算法后处理模块设置在用户分区。The photographing device according to any one of claims 13 to 15, wherein the hardware abstraction module and the algorithm post-processing module are arranged in a user partition.
19. 根据权利要求13至15中任意一项所述的拍摄装置，其特征在于，所述算法后处理模块用于将处理后的所述图像数据传输至显示模块进行显示。The photographing device according to any one of claims 13 to 15, wherein the algorithm post-processing module is used to transmit the processed image data to a display module for display.
20. 一种电子设备，其特征在于，所述电子设备包括：An electronic device, characterized in that, the electronic device includes:

    权利要求13至19中任意一项所述的所述的拍摄装置；及The photographing device according to any one of claims 13 to 19; and

    壳体，所述拍摄装置与所述壳体结合。A housing, the imaging device is combined with the housing.

Images