EP3671722A2 - Schaltvorrichtung, elektronische vorrichtung und fahrzeug - Google Patents
Schaltvorrichtung, elektronische vorrichtung und fahrzeug Download PDFInfo
- Publication number
- EP3671722A2 EP3671722A2 EP19217505.7A EP19217505A EP3671722A2 EP 3671722 A2 EP3671722 A2 EP 3671722A2 EP 19217505 A EP19217505 A EP 19217505A EP 3671722 A2 EP3671722 A2 EP 3671722A2
- Authority
- EP
- European Patent Office
- Prior art keywords
- image
- rotation
- mapping
- performs
- circuit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
- G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
- G09G5/38—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory with means for controlling the display position
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2340/00—Aspects of display data processing
- G09G2340/04—Changes in size, position or resolution of an image
- G09G2340/0407—Resolution change, inclusive of the use of different resolutions for different screen areas
- G09G2340/0435—Change or adaptation of the frame rate of the video stream
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2340/00—Aspects of display data processing
- G09G2340/10—Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2360/00—Aspects of the architecture of display systems
- G09G2360/02—Graphics controller able to handle multiple formats, e.g. input or output formats
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2360/00—Aspects of the architecture of display systems
- G09G2360/06—Use of more than one graphics processor to process data before displaying to one or more screens
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2360/00—Aspects of the architecture of display systems
- G09G2360/08—Power processing, i.e. workload management for processors involved in display operations, such as CPUs or GPUs
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2380/00—Specific applications
- G09G2380/10—Automotive applications
Definitions
- a head up display that displays information to a user's field of view in a superimposed manner by displaying an image on a transparent screen or the like is known.
- US-A-2016-0307346 discloses a method for verifying image data output to a head up display.
- a plurality pieces of input image data is integrated, a mapping process is performed on the integrated image data, the image data after the mapping process is output to a head up display, and validity of the image data output to the head up display is checked.
- the head up display is mounted on, for example, a dashboard of an automobile, but there is a tolerance in a mounting angle of the head up display with respect to the dashboard or the like.
- the mounting angle is a rotation angle in a direction in which a screen rotates around the line of sight as an axis, that is, a rotation angle with an axis intersecting the screen of the head up display as a rotation axis. Since a display image is inclined when the mounting angle of the head up display changes, the inclination of the display image is canceled by rotating the image in accordance with the inclination of the display image.
- US-A-2016-0307346 does not disclose a method for verifying image data when such an image rotation is performed.
- An aspect of the present disclosure relates to a circuit device including an image processing circuit and a comparison circuit.
- the image processing circuit performs a first mapping process and a first rotation process on an input first image to generate a second image for a head up display and performs, on the second image, a second mapping process that is a reverse mapping process of the first mapping process and a second rotation process that is a reverse rotation process of the first rotation process to generate a third image.
- the comparison circuit performs a comparison between the first image and the third image and outputs a result of the comparison as information for detecting an error in the second image.
- the interface 110 receives image data transmitted from a processing device 200 or the like to the circuit device 100, for example.
- the interface 110 converts the received image data into a format used inside the circuit device 100, and outputs the converted image data as an image IMA1.
- the image IMA1 is a first image.
- the interface 110 is an open LVDS display interface (OpenLDI), and converts a serial signal received by low voltage differential signaling (LVDS) into an RGB parallel signal.
- the processing device 200 is, for example, a system on a chip (SoC), a micro control unit (MCU), or a central processing unit (CPU).
- SoC system on a chip
- MCU micro control unit
- CPU central processing unit
- the image processing circuit 135 includes a first processing circuit 131 that maps an image in accordance with the surface shape of a display portion of the head up display, and a second processing circuit 132 that performs a reverse mapping on the image.
- the first processing circuit 131 is also referred to as a first warp engine
- the second processing circuit 132 is also referred to as a second warp engine.
- the display portion is a screen or display in the head up display.
- the screen is also referred to as a body to be projected.
- the head up display includes a projection device that projects the image IMA2 onto the screen.
- the projection device includes, for example, a liquid crystal display panel, a display driver that drives the liquid crystal display panel, a light source, and a lens.
- the display driver displays an image on the liquid crystal display panel based on the received image data, the light source outputs light to the liquid crystal display panel, and the light passing through the liquid crystal display panel is projected onto the screen by the lens.
- the screen is a transparent object and has a reflecting surface that reflects the projected light.
- the screen is a transparent screen mounted on a dashboard or a windshield of an automobile.
- the display portion is a display
- the head up display displays the image IMA2 on the display.
- the display is a transparent display using an organic EL panel, for example.
- the second processing circuit 132 performs, on the image IMA2, a second mapping process using a second mapping information MPA2 and a second rotation process using a second rotation information RT2, and outputs an image IMA3 after the processes.
- the image IMA3 is a third image.
- the image IMA1' is the image IMA1
- the image IMA3 is an image obtained by performing a reverse mapping on the entire image IMA2.
- the image IMA1' is an image of the region of interest extracted from the image IMA1
- the image IMA3 is an image obtained by performing a reverse mapping on the image of the region of interest extracted from the image IMA2.
- a case where the image IMA1' is an image of the region of interest extracted from the image IMA1 will be described as an example.
- the storage unit 133 is a first storage unit.
- the first processing circuit 131 stores the image IMA1' of the region of interest extracted from the image IMA1 in the storage unit 133.
- the storage unit 160 is a memory.
- the memory is a semiconductor memory such as a RAM or a nonvolatile memory.
- the storage unit 133 and the storage unit 160 may each be configured with an individual memory, and may be configured with one memory.
- the comparison circuit 145 performs a comparison process between the image IMA1' stored in the storage unit 133 and the image IMA3, and outputs a result of the comparison.
- the result of the comparison is used to detect an error in the image IMA2. That is, the result of the comparison is used to verify whether or not the first mapping process and the first rotation process performed by the first processing circuit 131 are normal.
- the comparison circuit 145 obtains an index indicating a similarity between the image IMA1 and the image IMA3.
- the index is a shape index or a visibility index to be described below.
- the comparison circuit 145 may obtain sum of squared difference (SSD), sum of absolute difference (SAD), normalized cross correlation (NCC), or the like as an index.
- the error detection circuit 150 detects an error in the second image IMA2 by comparing the index with a threshold.
- the threshold is a threshold indicating the allowable degree of similarity that the image IMA1' and the image IMA3 can have.
- the image processing circuit 135 stops outputting the image IMA2 to the interface 140.
- the interface 140 stops outputting the image IMA2.
- the interface 140 may output the image IMA2 together with the error information, and the display driver that has received the error information may perform an operation based on the error information.
- the interface 190 may output the error information to the processing device 200, and the processing device 200 that has received the error information may perform an operation based on the error information.
- the error information is, for example, an error determination flag or an index.
- the operation based on the error information is, for example, a display stop of the head up display.
- the register circuit 170 is configured to be accessible from the processing device 200 via the interface 190.
- the register circuit 170 includes an error detection result register 176 and a threshold register 178.
- the error detection result register 176 stores an error detection result output from the error detection circuit 150.
- the error detection result is, for example, an error determination flag indicating whether or not the display image is determined to be an error.
- the processing device 200 can determine whether or not an error has occurred by reading the error detection result from the error detection result register 176 via the interface 190.
- a threshold is set in the threshold register 178 from the processing device 200 via the interface 190.
- the error detection circuit 150 compares the index and the threshold set in the threshold register 178 to detect an error.
- the storage unit 160 is a second storage unit.
- the storage unit 160 stores the first mapping information MPA1, the second mapping information MPA2, the first rotation information RT1, and the second rotation information RT2.
- the processing device 200 transmits MPA1, MPA2, RT1, and RT2 to the interface 190, and the storage unit 160 stores the MPA1, MPA2, RT1, and RT2 received by the interface 190.
- the image processing circuit 135 performs a mapping process and a rotation process based on the MPA1, MPA2, RT1, and RT2 read from the storage unit 160.
- the storage unit 160 is, for example, a memory or a register.
- the memory is a semiconductor memory such as a RAM or a nonvolatile memory.
- the image processing circuit 135, the comparison circuit 145, and the error detection circuit 150 are logic circuits.
- the image processing circuit 135, the comparison circuit 145, and the error detection circuit 150 may be configured as individual circuits, or may be configured as an integrated circuit by automatic placement and routing.
- some or all of these logic circuits may be implemented by a processor such as a digital signal processor (DSP).
- DSP digital signal processor
- a program or instruction set describing the function of each circuit is stored in the memory, and the function of each circuit is realized by the processor executing the program or instruction set.
- the circuit device 100 may not include the error detection circuit 150 and the error detection result register 176 in FIG. 1 , the circuit device 100 may not include the error detection circuit 150 and the error detection result register 176.
- the interface 190 may output the index obtained by the comparison circuit 145 to the processing device 200, and the processing device 200 that has received the index may detect an error by comparing the index with the threshold.
- the processing device 200 may perform an error handling operation such as display stop of the head up display.
- FIG. 2 is a diagram for describing a first operation example of the circuit device 100 according to the present embodiment.
- a first mapping process S12 the first processing circuit 131 maps the image IMA1 to the image IMA2 by using the mapping information RMPA1.
- the mapping process is also referred to as a warp process. Since the rotation-converted mapping information RMPA1 is used, an image rotation is performed together with mapping.
- the first mapping process S12 performs coordinate conversion that associates a position on the image IMA1 with a position on the display portion to convert the image IMA1 into the image IMA2.
- the coordinate conversion is defined by a map table.
- the first processing circuit 131 is a forward warp engine or an inverse warp engine.
- the first mapping information MPA1 is a forward map table
- the first mapping information MPA1 is an inverse map table.
- the inverse warp engine is also referred to as a reverse warp engine
- the inverse map table is also referred to as a reverse map table.
- the forward warp is a conversion that moves each pixel of an input image to the warp engine to any position in an output image.
- the forward warp engine is a warp engine having a forward warp function.
- the forward map table is a table in which each pixel of the input image is associated with a destination coordinate corresponding to each pixel.
- the forward map table is a table in which each pixel of the input image is associated with a relative movement amount to the destination coordinate corresponding to each pixel.
- the inverse warp is a conversion that obtains each pixel of the output image of the warp engine from a pixel at any position in the input image.
- the inverse warp engine is a warp engine having an inverse warp function.
- the first mapping process S12 is a process of deforming the image IMA1 so that the image displayed on the display portion of the head up display appears undistorted when viewed from the user.
- the display portion has a curved surface
- the image when an undistorted image is displayed on the display portion, the image appears distorted to the user due to the curved surface.
- the image is deformed so as to cancel the distortion.
- the image may be distorted when the image is projected on the display portion.
- the image is deformed so as to cancel the distortion.
- the image IMA2 is an image obtained by deforming the undistorted image IMA1
- the image IMA2 is a distorted image.
- the image IMA1 is not normally converted into the image IMA2.
- an error cannot be accurately detected even if the deformed image IMA2 and the original image IMA1 are simply compared. Therefore, in the present embodiment, the image comparison is performed after the image IMA2 is reverse-mapped. The reverse mapping and image comparison will be described.
- the first processing circuit 131 extracts the ROI from the image IMA1 and outputs the image IMA1' after the extraction. For example, a region to be verified whether the display is normal in the image displayed on the head up display is defined as ROI.
- the ROI is a region including an icon such as a warning light.
- the second processing circuit 132 extracts the ROI from the image IMA2 and outputs an image IMA2' after the extraction.
- the ROI in the image IMA2 is a region corresponding to the ROI in the image IMA1.
- the second processing circuit 132 In a second rotation process S21, the second processing circuit 132 generates mapping information RMPA2 by rotationally converting the second mapping information MPA2 by using the second rotation information RT2.
- the second mapping information MPA2 is a map table in which source coordinates and target coordinates are associated with each other.
- the second rotation information RT2 is information that defines reverse rotation conversion of rotation conversion defined by the first rotation information RT1. That is, the rotation conversion defined by the second rotation information RT2 is the same as the rotation conversion defined by the first rotation information RT1 in terms of the center coordinate and an absolute value of a rotation angle, and is opposite to the rotation conversion defined by the first rotation information RT1 in terms of the rotation direction.
- the second processing circuit 132 rotationally converts the source coordinates or the target coordinates based on the second rotation information RT2.
- the converted map table is the mapping information RMPA2.
- the second processing circuit 132 maps the image IMA2' to the image IMA3 by using the mapping information RMPA2.
- the image IMA3 is the third image. Since the rotation-converted mapping information RMPA2 is used, an image rotation is performed together with mapping.
- the second mapping process S22 performs coordinate conversion that associates a position on the display portion with a position on the image IMA3 to convert the image IMA2' into the image IMA3.
- the coordinate conversion is defined by a map table and is the reverse conversion of the coordinate conversion in the first mapping process S12.
- the second processing circuit 132 is a forward warp engine or an inverse warp engine.
- the second mapping information MPA2 is an inverse map table
- the second mapping information MPA2 is a forward map table.
- the second mapping process S22 is a process of returning an image deformed in accordance with the display portion of the head up display to the undistorted image.
- the image IMA1' before mapping and the image IMA3 after mapping and reverse mapping are to be the same image.
- the comparison circuit 145 obtains an index indicating a similarity between the image IMA1' and the image IMA3. When the similarity indicated by the index is equal to or less than a threshold, the error detection circuit 150 determines that an error has occurred.
- the error detection circuit 150 determines that no error has occurred.
- the allowable degree of similarity is set by a threshold compared with the index.
- the first processing circuit 131 performs the rotation process on the first mapping information MPA1 based on the first rotation information RT1 and performs the mapping process on the image IMA1 by using the processed mapping information RMPA1 has been described as an example.
- the rotation process and the mapping process are not limited to this.
- the rotation process and the mapping process performed by the second processing circuit 132 are also not limited to FIG. 2 .
- the first processing circuit 131 may perform the mapping process on the first image IMA1 by using the first mapping information MPA1, and perform the rotation process on the processed image based on the first rotation information RT1.
- the second processing circuit 132 may perform the rotation process on the image IMA2' based on the second rotation information RT2, and perform the mapping process on the processed image by using the second mapping information MPA2.
- FIG. 3 is a diagram for describing a relationship between a mounting tolerance of the head up display and the image rotation.
- a case where the head up display is installed on a dashboard of an automobile will be described as an example, but the installation place of the head up display is not limited to this.
- a direction DZ shown in FIG. 3 is a direction orthogonal to a display portion DSP of the head up display.
- the direction DZ and the display portion DSP are orthogonal to each other at any position of the display portion DSP.
- the direction DZ and the display portion DSP are orthogonal to each other at the center of the display portion DSP.
- a direction DX is a direction orthogonal to the direction DZ
- a direction DY is a direction orthogonal to the direction DX and the direction DZ.
- the direction DX corresponds to the horizontal direction. That is, the direction DX is parallel to the horizontal plane when the automobile is in a horizontal posture.
- the direction DY corresponds to the vertical direction, and the direction DY is perpendicular to the horizontal plane.
- the direction DY is not limited to the vertical direction, and the direction DY may be inclined with respect to the horizontal plane. That is, the head up display may be installed on the dashboard inclined in the depth direction.
- Rotation RZ indicates rotation of the display portion DSP about the axis parallel to the direction DZ as a rotation axis. Due to the tolerance when the head up display is mounted on the dashboard, variation occurs in the rotation angle of the rotation RZ. It is assumed that the rotation angle when the tolerance is zero is 0 degrees, the clockwise rotation is positive, and the counterclockwise rotation is negative. When the rotation RZ of the display portion DSP is a positive rotation angle, the display image is rotated clockwise when viewed from the user. The first processing circuit 131 performs the rotation process on the image in the negative direction, that is, counterclockwise. As a result, even when the display portion DSP is inclined due to the tolerance, an image that is not inclined can be displayed when viewed from the user.
- FIG. 4 is a diagram for describing details of the first rotation process.
- the first processing circuit 131 is a forward warp engine will be described as an example.
- the map table is a table in which source coordinates (XSi, YSi) are associated with target coordinates (XTi, YTi).
- i is an integer of 1 or more and 4 or less.
- coordinate conversion of four points is defined in the map table, but coordinate conversion of two or more points may be defined in the map table.
- the map table is a forward map table. That is, the source coordinates (XSi, YSi) indicate the position in the image IMA1, and the target coordinates (XTi, YTi) indicate the position in the image IMA2.
- the first processing circuit 131 rotationally converts the target coordinates (XTi, YTi) in the map table in the first rotation process. That is, the coordinates indicating the position in the image IMA2 are rotationally converted. Even when the first processing circuit 131 is an inverse map engine, the coordinates indicating the position in the image IMA2 in the inverse map table are rotationally converted.
- the rotation conversion is a conversion that rotates by a predetermined angle around the center coordinate in the XY coordinates.
- the processing device 200 writes the first rotation information RT1 indicating a predetermined angle into the register circuit 170 via the interface 190.
- the first processing circuit 131 sets the predetermined angle by reading the first rotation information RT1 from the register circuit 170.
- the predetermined angle includes information on the absolute value of the rotation angle and the rotation direction.
- the second processing circuit 132 rotationally converts coordinates indicating the position in the image IMA2 in the map table.
- the image processing circuit 135 performs the first mapping process and the first rotation process on the input image IMA1 to generate the image IMA2 for the head up display.
- the image processing circuit 135 performs, on the image IMA2', the second mapping process that is the reverse mapping process of the first mapping process and the second rotation process that is the reverse rotation process of the first rotation process to generate the image IMA3.
- the comparison circuit 145 performs the comparison between the image IMA1' and the image IMA3 and outputs the result of the comparison as information for detecting an error in the image IMA2.
- the comparison circuit 145 compares the image IMA1 and the image IMA3 in the region of interest. That is, as described with reference to FIG. 2 , the first processing circuit 131 extracts the image IMA1' of the ROI from the image IMA1. In addition, the second processing circuit 132 extracts the image IMA2' of the ROI from the image IMA2, and performs the second mapping process and the second rotation process on the image IMA2' to output the image IMA3. That is, the image IMA3 is the image of the ROI. The comparison circuit 145 compares the image IMA1' of the ROI and the image IMA3 of the ROI.
- error detection can be performed only for the region of interest in the image displayed on the head up display.
- the region of interest is a region in which content with high importance is displayed in the display image, for example.
- the processing load on the image processing circuit 135 and the comparison circuit 145 can be reduced. That is, by comparing the image IMA1' and the image IMA3 in the region of interest, the processing load on the comparison circuit 145 can be reduced as compared to the case where the entire image is compared.
- the second processing circuit 132 performs the second mapping process and the second rotation process on the image IMA2' in the region of interest, such that the processing load on the second processing circuit 132 can be reduced as compared to the case where the entire image is processed.
- the image processing circuit 135 does not perform a reduction process and an enlargement process on the image. However, as will be described with reference to FIGS. 5 to 7 below, the image processing circuit 135 may perform the reduction process on the image, or the reduction process and the enlargement process on the image. Hereinafter, description of the operation already described in FIG. 2 and the like will not be repeated.
- FIG. 5 is a diagram for describing a second operation example of the circuit device 100 according to the present embodiment.
- the first processing circuit 131 extracts the image of the ROI from the image IMA1 as shown in S13, performs the reduction process on the image of the ROI to 1/2 size as shown in S14, and outputs the image IMA1' after the reduction process.
- the reduction process is a process of reducing the number of pixels of an image, that is, a process of downsampling the image. Note that, the order of reduction and ROI extraction may be reversed. That is, the first processing circuit 131 may perform the reduction process on the image IMA1, extract the image of the ROI from the image after the reduction process, and output the extracted image IMA1' of the ROI.
- the second processing circuit 132 extracts the image of the ROI from the image IMA2 as shown in S23, performs the reduction process on the image of the ROI to 1/2 size as shown in S24, and outputs the image IMA2' after the reduction process. Since both the image IMA1' and the image IMA3 are reduced to 1/2 size, the image IMA1' and the image IMA3 have the same size. As in the case of the first processing circuit 131, the order of reduction and ROI extraction may be reversed.
- FIG. 6 is a diagram for describing a third operation example of the circuit device 100 according to the present embodiment.
- the first processing circuit 131 performs only ROI extraction in S13 and does not perform 1/2 reduction.
- the second processing circuit 132 performs the ROI extraction process in S23 and the 1/2 reduction process in S24.
- the second processing circuit 132 performs an enlargement process of doubling the image size of the image IMA3, and outputs an image IMA3' after the enlargement process.
- the comparison circuit 145 compares the image IMA1' and the image IMA3'.
- the image IMA3' has the same size as the image IMA1' by the 1/2 reduction process in S24 and the double enlargement process in S26.
- FIG. 7 is a diagram for describing a fourth operation example of the circuit device 100 according to the present embodiment.
- the first processing circuit 131 performs the ROI extraction process in S13 and the 1/2 reduction process in S14.
- the second processing circuit 132 extracts the image of the ROI from the image IMA2 as shown in S23, performs the reduction process on the image of the ROI to 1/4 size as shown in S25, and outputs the image IMA2' after the reduction process.
- the second processing circuit 132 performs the double enlargement process in S26. Since the image IMA1' is an image reduced to 1/2 size and the image IMA3 is an image reduced to 1/4 size and doubled, the image IMA1' and the image IMA3 have the same size.
- the second processing circuit 132 performs the second mapping process and the second rotation process on the image IMA2' reduced to 1/2 size or 1/4 size. Therefore, the processing load on the second processing circuit 132 can be reduced as compared to the case where the reduction process is not performed.
- the comparison circuit 145 since the comparison circuit 145 compares 1/2 reduced size images, the processing load on the comparison circuit 145 can be reduced as compared to the case of comparing original size images.
- a process of interpolating a map table according to an inclination angle of a display surface of the head up display will be described. First, the inclination angle of the display surface in the depth direction will be described with reference to FIG. 3 .
- a surface on which an image is displayed in the display portion DSP is a display surface.
- An angle formed by the display surface and a reference plane is an inclination angle ⁇ X of the display surface in the depth direction.
- the reference plane is, for example, a horizontal plane, but is not limited thereto.
- the inclination angle ⁇ X may be fixed or variable.
- the user may be configured to be able to adjust the inclination angle ⁇ X.
- FIG. 8 is a diagram for describing a process of interpolating a map table.
- the first mapping information MPA1 used by the first processing circuit 131 includes a first map table TB1 corresponding to a first inclination angle and a second map table TB2 corresponding to a second inclination angle.
- the first inclination angle is 5 degrees and the second inclination angle is 10 degrees.
- the first processing circuit 131 performs an interpolation process S50 based on the first map table TB1 and the second map table TB2 to generate a third map table TB3 corresponding to a third inclination angle and performs the first mapping process based on the third map table TB3.
- the processing device 200 writes an inclination information SLP indicating the third inclination angle into the register circuit 170 via the interface 190.
- the first processing circuit 131 performs the interpolation process S50 based on the inclination information SLP written in the register circuit 170.
- the interpolation process S50 is a process of obtaining the third map table from the first map table TB1 and the second map table TB2 by linear interpolation.
- the target coordinates (XTi, YTi) of the first map table TB1 and the target coordinates (XTi, YTi) of the second map table TB2 are different depending on the inclination angle.
- the first processing circuit 131 obtains target coordinates (XTi, YTi) of the third map table from the target coordinates (XTi, YTi) of the first map table TB1 and the target coordinates (XTi, YTi) of the second map table TB2 by linear interpolation.
- the first processing circuit 131 may perform higher-order interpolation such as cubic spline interpolation by using three or more map tables.
- the second processing circuit 132 also performs a similar interpolation process. That is, the second mapping information MPA2 includes a fourth map table corresponding to the first inclination angle and a fifth map table corresponding to the second inclination angle. The second processing circuit 132 performs an interpolation process based on the fourth map table and the fifth map table to generate a sixth map table corresponding to the third inclination angle and performs the second mapping process based on the sixth map table.
- the comparison circuit 145 performs image comparison in one or a plurality of frames.
- each frame refers to a frame for image comparison.
- the comparison circuit 145 may compare all the color components of the image in each frame to be compared, or may compare one color component of the image in each frame for each color.
- FIG. 9 is a diagram for describing a process when comparing one color component of the image in each frame for each color.
- an R component, a G component, and a B component of the image are a first color component, a second color component, and a third color component, respectively, but the color components are not limited to this.
- the first color component, the second color component, and the third color component may be a Y component, a Cr component, and a Cb component.
- image comparison is performed for each frame, and processes in frames F1 to F3 are repeated after a frame F4.
- the first processing circuit 131 performs the first mapping process and the first rotation process on the RGB components of the image IMA1, and outputs the RGB components of the image IMA2.
- the above processes are the same for the frames F2 and F3.
- the first processing circuit 131 extracts the ROI from the R component of the image IMA1 and outputs the R component of the image IMA1'.
- the second processing circuit 132 extracts the ROI from the R component of the image IMA2, outputs the R component of the image IMA2', performs the second mapping process and the second rotation process on the R component of the image IMA2', and outputs the R component of IMA3.
- the comparison circuit 145 compares the R component of the image IMA1' and the R component of the image IMA3.
- the first processing circuit 131, the second processing circuit 132, and the comparison circuit 145 perform processes similar to the above on the G component.
- the first processing circuit 131, the second processing circuit 132, and the comparison circuit 145 perform processes similar to the above on the B component.
- FIG. 10 shows an example of the image IMA2 displayed on the head up display.
- an icon ICA is overlapped on a meter image DIM.
- the icon ICA is blended into the meter image DIM with a certain transmittance.
- the circuit device 100 verifies whether or not the icon ICA is appropriately displayed. In this case, as indicated by the dotted rectangle in FIG. 10 , the region including the icon ICA is set as the ROI.
- FIG. 11 is an image IMA1' of the ROI extracted from the image IMA1 before the mapping process.
- the ROI includes the icon ICA and the meter image DIM that is a background image of the icon ICA.
- FIG. 12 is an image IMA3 obtained by performing a reverse mapping process on the image IMA2' of the ROI extracted from the image IMA2.
- FIG. 12 shows an example in which the icon ICA is not correctly displayed. In this case, the image IMA1' and the image IMA3 have the same background image only, and different icon ICA portions. Therefore, the similarity indicated by the index as the comparison result is low. In the image IMA3, when the icon ICA is correctly displayed, the similarity is high.
- the index indicating the similarity can take a continuous or stepwise value.
- the error detection circuit 150 compares the index with a threshold, it is possible to adjust the allowable degree of similarity by adjusting the threshold.
- the comparison circuit 145 obtains a shape index, a visibility index, or both as an index indicating the similarity between the image IMA1' and the image IMA3. As described above, the comparison circuit 145 may obtain SSD, SAD, NCC, or the like as an index. Note that, whether the index increases as the degree of similarity increases, or whether the index decreases as the degree of similarity increases, depends on the index calculation method.
- the threshold is set for each of the shape index and the visibility index.
- the comparison circuit 145 obtains an inter-image distance between the image IMA1' and the image IMA3 in the color space.
- the color space is, for example, RGB or YCrCb.
- the comparison circuit 145 obtains a square value of the distance between a pixel of the image IMA1' and a pixel of the image IMA3 corresponding to the pixel of the image IMA1' in the color space.
- the comparison circuit 145 accumulates the square value in the image and sets the accumulated value as the inter-image distance.
- the inter-image distance corresponds to the shape index.
- the comparison circuit 145 obtains an edge image EIMA1' by extracting an edge of the image IMA1'.
- the comparison circuit 145 obtains an edge image of the image IMA3 by extracting an edge of the image IMA3.
- the edge image of the image IMA3 is referred to as EIMA3.
- the comparison circuit 145 compares the edge image EIMA1' and the edge image EIMA3.
- the comparison circuit 145 extracts edges from the image IMA1' and the image IMA3 by using a Sobel filter or the like, and obtains a correlation value between the edge image EIMA1' and the edge image EIMA3.
- the correlation value between the edge images corresponds to the shape index.
- the comparison circuit 145 obtains a histogram from a Y channel of the image IMA1'. Similarly, the comparison circuit 145 obtains a histogram from a Cr channel and Cb channel of the image IMA1', and obtains a histogram from a Y channel, Cr channel, and Cb channel of the image IMA3.
- the comparison circuit 145 performs a cross-correlation operation on the histograms of the image IMA1' and the image IMA3 in the Y channel.
- the cross-correlation operation is an operation in which two histograms are shifted by a lag to obtain a correlation value, and the correlation value is obtained while changing the lag. When the lag is changed and the correlation value between the two histograms becomes high, the lag has a peak. There may be a plurality of peaks.
- the comparison circuit 145 performs a cross-correlation operation on the histograms of the image IMA1' and the image IMA3 in the Cr channel and the Cb channel.
- the comparison circuit 145 checks lag values having a peak in cross-correlation signals of all channels, and obtains the maximum lag value among the lag values .
- the maximum lag value corresponds to the visibility index.
- the visibility index indicates the color contrast between the icon and the background image. It is considered that the higher the color contrast is, the higher the visibility is. Therefore, it is determined that the similarity is higher as the visibility index is larger.
- the similarity between the image IMA1' and the image IMA3 of the ROI can be determined.
- the similarity is the highest when the image IMA1' and the image IMA3 match with each other. That is, the similarity can be paraphrased as the degree of coincidence. For example, when the icon ICA is rotated and displayed, the similarity is lower than when the icon ICA is not rotated. Meanwhile, since the visibility index indicates the color contrast, the similarity does not change much even when the icon ICA is rotated. Therefore, the visibility index may be used when allowing the case where the icon ICA is rotated. In addition, since the calculation method is different between the shape index and the visibility index, the accuracy of error detection can be improved by using the shape index and the visibility index.
- FIG. 14 is a configuration example of an electronic device including the circuit device according to the present embodiment.
- An electronic device 300 includes a processing device 310, a circuit device 320, a storage device 350, an operation device 360, a communication device 370, and a head up display 400.
- the circuit device 320 corresponds to the circuit device 100 in FIG. 1
- the processing device 310 corresponds to the processing device 200 in FIG. 1 .
- the head up display 400 includes a display driver 330 and a display panel 340.
- the processing device 310 is, for example, an MCU.
- the circuit device 320 corresponds to a display controller.
- the error detection method of the present embodiment is applicable not only to a display controller but also to a circuit device that generates a display image for a head up display.
- the processing device 310 transfers the image data stored in the storage device 350 or the image data received by the communication device 370 to the circuit device 320.
- the circuit device 320 performs an image process on image data, display timing control, generation of image data to be transferred to a display driver, error detection of the image data, and the like.
- the display driver 330 drives the display panel 340 based on the image data transferred from the circuit device 320 and the display timing control by the circuit device 320 to display an image.
- the display panel 340 is, for example, a liquid crystal display panel or an EL display panel.
- the storage device 350 is, for example, a memory, a hard disk drive, an optical disk drive, or the like.
- the operation device 360 is a device for the user to operate the electronic device 300, and is, for example, a button, a touch panel, a keyboard, or the like.
- the communication device 370 is, for example, a device that performs wired communication or a device that performs wireless communication.
- the wired communication is, for example, a LAN or USB.
- the wireless communication is, for example, a wireless LAN or wireless proximity communication.
- an in-vehicle electronic device As the electronic device including the circuit device according to the present embodiment, various devices such as an in-vehicle electronic device, a display terminal such as a factory facility, a display device mounted on a robot, or an information processing device can be assumed.
- the in-vehicle electronic device is, for example, a meter panel.
- the information processing device is, for example, a PC.
- FIG. 15 is an example of a vehicle including the circuit device 320 according to the present embodiment.
- the vehicle includes the circuit device 320 according to the present embodiment and the processing device 310 that transmits image data to the circuit device 320.
- the processing device 310 may perform an error handling process based on the error detection result of the display image for the head up display from the circuit device 320.
- the vehicle includes the head up display 400 and a control device 208.
- the control device 208 is an electronic control unit (ECU), and the circuit device 320 and the processing device 310 are incorporated in the ECU.
- the circuit device 320 may be incorporated in the head up display 400.
- the circuit device 320 according to the present embodiment can be incorporated in various vehicles such as a car, an airplane, a motorcycle, a bicycle, or a ship.
- the vehicle is, for example, a device or apparatus that moves on the ground, in the sky, or on the sea, including a driving mechanism such as an engine or motor, a steering mechanism such as a steering wheel or rudder, and various electronic devices.
- FIG. 15 schematically shows an automobile 206 as a specific example of the vehicle.
- the head up display 400 has a transparent screen, and the transparent screen is installed between the driver's seat and the windshield. Alternatively, the head up display may use a windshield as a transparent screen and project an image on the windshield.
- the head up display 400 functions as a meter panel of the automobile 206, for example.
- the circuit device includes the image processing circuit and the comparison circuit.
- the image processing circuit performs the first mapping process and the first rotation process on the input first image to generate the second image for the head up display.
- the image processing circuit performs, on the second image, the second mapping process that is the reverse mapping process of the first mapping process and the second rotation process that is the reverse rotation process of the first rotation process to generate the third image.
- the comparison circuit performs a comparison between the first image and the third image and outputs a result of the comparison as information for detecting an error in the second image.
- the image processing circuit performs, on the second image, the reverse mapping process of the first mapping process and the reverse rotation process of the first rotation process, when the second image is normal, the third image is to return to the same image as the first image.
- the comparison circuit can output information for detecting the error of the second image by comparing the third image and the first image.
- the comparison circuit may perform the comparison between the first image and the third image in a region of interest.
- error detection can be performed only for the region of interest in the image displayed on the head up display.
- the number of pixels of the image handled in the second mapping process and the second rotation process is reduced, so that the processing load on the image processing circuit is reduced.
- the number of pixels of the image handled in the comparison process is reduced, so that the processing load on the comparison circuit is reduced.
- the circuit device may include a first storage unit.
- the image processing circuit may extract an image of the region of interest in the first image to store the extracted image in the first storage unit.
- the comparison circuit may perform a comparison between the image of the region of interest in the first image, which is stored in the first storage unit, and the third image.
- the first storage unit can temporarily store the first image to be compared with the third image during the operation time.
- the image processing circuit may have a first processing circuit that performs the first mapping process and the first rotation process, and a second processing circuit that performs the second mapping process and the second rotation process.
- an error is detected when there is an abnormality in the first processing circuit or the second processing circuit. That is, when mapping and reverse mapping are performed in the same processing circuit, an abnormality that has occurred in mapping may be restored in reverse mapping, and the first image and the third image may be similar to each other. According to the present embodiment, an error can be reliably detected by independent processing circuits executing mapping and reverse mapping, respectively.
- an image rotation in the second rotation process may be the same as an image rotation in the first rotation process in terms of a rotation center and an absolute value of a rotation angle, and be opposite to the image rotation in the first rotation process in terms of a rotation direction.
- the image rotated in the first rotation process is reversely rotated in the second rotation process, thereby returning to the original image.
- An error can be detected by comparing the original image with the image before the first rotation process.
- image rotations in the first rotation process and the second rotation process may have an axis intersecting a display surface of the head up display as a rotation axis.
- the mounting angle of the head up display varies with respect to the mounting position of the head up display. Due to the variations in the mounting angle, the display image appears inclined to the user. The inclination corresponds to rotation about the axis intersecting the display surface as the rotation axis. In the present embodiment, it is possible to correct the inclination of the display image due to the variations in the mounting angle by rotating the image by the image process. In addition, an error can be detected even when rotation correction is performed by comparing the third image restored using reverse rotation with the first image that is the input image.
- the circuit device may include a storage unit.
- the storage unit may store first rotation information defining an image rotation in the first rotation process and second rotation information defining an image rotation in the second rotation process.
- the image processing circuit may perform the first rotation process based on the first rotation information stored in the storage unit and perform the second rotation process based on the second rotation information stored in the storage unit.
- the storage unit stores the first rotation information and the second rotation information as independent rotation information, so that an error can be reliably detected.
- the storage unit may store first mapping information defining mapping in the first mapping process and second mapping information defining mapping in the second mapping process.
- the image processing circuit may perform the first mapping process based on the first mapping information and perform the second mapping process based on the second mapping information.
- the storage unit stores the first mapping information and the second mapping information as independent mapping information, so that an error can be reliably detected.
- the first mapping information may include a first map table corresponding to a first inclination angle of a display surface of the head up display and a second map table corresponding to a second inclination angle of the display surface.
- the image processing circuit may perform an interpolation process based on the first map table and the second map table to generate a third map table corresponding to a third inclination angle of the display surface and may perform the first mapping process based on the third map table.
- the distortion of the display image changes according to the inclination angle of the display surface in the depth direction.
- mapping process mapping including the distortion according to the inclination angle is performed.
- the interpolation process based on the first map table corresponding to the first inclination angle and the second map table corresponding to the second inclination angle is performed, and thus the third map table corresponding to the third inclination angle of the display surface is generated, it is possible to correspond to various inclination angles.
- the map table is generated by the interpolation process, the number of map tables stored in the storage unit can be reduced.
- the second mapping information may include a fourth map table corresponding to the first inclination angle and a fifth map table corresponding to the second inclination angle.
- the image processing circuit may perform an interpolation process based on the fourth map table and the fifth map table to generate a sixth map table corresponding to the third inclination angle and may perform the second mapping process based on the sixth map table.
- both the third map table and the sixth map table corresponding to the reverse mapping correspond to the third inclination angle.
- the third image returns to the same image as the first image by the reverse mapping using the sixth map table.
- the image processing circuit may perform a reduction process on the second image, and perform the second mapping process and the second rotation process on the second image after the reduction process.
- the comparison circuit may perform the comparison between a first color component of the first image and a first color component of the third image in a first frame, and perform the comparison between a second color component of the first image and a second color component of the third image in a second frame.
- the image processing circuit may perform the second mapping process and the second rotation process on a first color component of the second image in the first frame to generate the first color component of the third image.
- the image processing circuit may perform the second mapping process and the second rotation process on a second color component of the second image in the second frame to generate the second color component of the third image.
- the load on the second processing circuit can be reduced compared to the case where all the color components of the image per frame are subjected to the second mapping process and the second rotation process.
- the comparison circuit may obtain an index indicating a similarity between the first image and the third image as a result of the comparison based on a pixel value of the first image and a pixel value of the third image or based on a pixel value of an edge image of the first image and a pixel value of an edge image of the third image.
- the pixel value of the first image and the pixel value of the third image is similar to each other, or the pixel value of the edge image of the first image and the pixel value of the edge image of the third image are similar to each other.
- the index since the index is obtained based on the pixel value of the first image and the pixel value of the third image or based on the pixel value of the edge image of the first image and the pixel value of the edge image of the third image, the index indicates the similarity between the first image and the third image.
- the circuit device may include an error detection circuit that detects the error in the second image by a comparison between the index and a threshold.
- the circuit device detects an error in the second image based on the index, so that the circuit device can perform an error handling process based on the error detection result, or output the error detection result to the outside of the circuit device.
- the electronic device according to the present embodiment includes the circuit device described in any of the above aspects.
- the vehicle according to the present embodiment includes the circuit device described in any of the above aspects.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
- Controls And Circuits For Display Device (AREA)
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201862782640P | 2018-12-20 | 2018-12-20 | |
JP2019173837A JP7419721B2 (ja) | 2018-12-20 | 2019-09-25 | 回路装置、電子機器及び移動体 |
Publications (2)
Publication Number | Publication Date |
---|---|
EP3671722A2 true EP3671722A2 (de) | 2020-06-24 |
EP3671722A3 EP3671722A3 (de) | 2020-07-22 |
Family
ID=68944634
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP19217505.7A Pending EP3671722A3 (de) | 2018-12-20 | 2019-12-18 | Schaltvorrichtung, elektronische vorrichtung und fahrzeug |
Country Status (1)
Country | Link |
---|---|
EP (1) | EP3671722A3 (de) |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20160307346A1 (en) | 2015-04-17 | 2016-10-20 | Freescale Semiconductor, Inc. | Display controller, heads-up image display system and method thereof |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
FR2788365B1 (fr) * | 1999-01-07 | 2003-12-05 | Bernard Boisdequin | Dispositif d'affichage a cristaux liquides, notamment afficheur de vitesse pour vehicule |
US20080088527A1 (en) * | 2006-10-17 | 2008-04-17 | Keitaro Fujimori | Heads Up Display System |
FR2948472B1 (fr) * | 2009-07-21 | 2012-02-10 | Alstom Transport Sa | Dispositif de verification de l'integrite d'une donnee affichee et procede associe |
-
2019
- 2019-12-18 EP EP19217505.7A patent/EP3671722A3/de active Pending
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20160307346A1 (en) | 2015-04-17 | 2016-10-20 | Freescale Semiconductor, Inc. | Display controller, heads-up image display system and method thereof |
Also Published As
Publication number | Publication date |
---|---|
EP3671722A3 (de) | 2020-07-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11180026B2 (en) | Circuit device, electronic device, and vehicle | |
US11210534B2 (en) | Method for position detection, device, and storage medium | |
TWI657409B (zh) | 虛擬導引圖示與真實影像之疊合裝置及其相關疊合方法 | |
US8754961B2 (en) | Apparatus and method for generating image data from overlapping regions of images | |
US10950004B2 (en) | Method, apparatus, device and medium for calibrating intelligent roadside unit | |
US9025029B2 (en) | Apparatus and method for removing a reflected light from an imaging device image | |
US20080309763A1 (en) | Driving Support System And Vehicle | |
CN111800589B (zh) | 图像处理方法、装置和***,以及机器人 | |
JP2019149760A (ja) | 回路装置及び電子機器 | |
US10412308B2 (en) | Display controller and display control method | |
US10764510B2 (en) | Image conversion device | |
CN111354285A (zh) | 电路装置、电子设备以及移动体 | |
US11010866B2 (en) | Circuit device, electronic apparatus, and mobile body | |
EP3671722A2 (de) | Schaltvorrichtung, elektronische vorrichtung und fahrzeug | |
US10354148B2 (en) | Object detection apparatus, vehicle provided with object detection apparatus, and non-transitory recording medium | |
US8503789B2 (en) | Method and system for measuring deflection angle of a camera lens | |
JP7467883B2 (ja) | 回路装置、電子機器及び移動体 | |
CN113519150A (zh) | 相机组件和方法 | |
US20230070936A1 (en) | Circuit Device and Error Detection Method | |
US20240193958A1 (en) | Processing method for vehicle surround view, vehicle surround view system and in-vehicle processing device | |
WO2019131926A1 (ja) | 対象選択装置 | |
US20150169046A1 (en) | Line scan camera eye tracking system and method | |
JP2023037738A (ja) | 制御システム及び電子機器 | |
CN117252932A (zh) | 一种ar-hud标定方法、***、设备及汽车 | |
KR20170010869A (ko) | Avm 시스템의 공차 보정 장치 및 방법 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
PUAL | Search report despatched |
Free format text: ORIGINAL CODE: 0009013 |
|
17P | Request for examination filed |
Effective date: 20200422 |
|
AK | Designated contracting states |
Kind code of ref document: A2 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
AK | Designated contracting states |
Kind code of ref document: A3 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G09G 5/38 20060101AFI20200618BHEP |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
17Q | First examination report despatched |
Effective date: 20211223 |