CN101793562A - Face detection and tracking algorithm of infrared thermal image sequence - Google Patents
Face detection and tracking algorithm of infrared thermal image sequence Download PDFInfo
- Publication number
- CN101793562A CN101793562A CN201010105381A CN201010105381A CN101793562A CN 101793562 A CN101793562 A CN 101793562A CN 201010105381 A CN201010105381 A CN 201010105381A CN 201010105381 A CN201010105381 A CN 201010105381A CN 101793562 A CN101793562 A CN 101793562A
- Authority
- CN
- China
- Prior art keywords
- face
- people
- thermal
- temperature
- visible images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention provides face detection and tracking algorithm of the infrared thermal image sequence, which is characterized by carrying out face detection on the infrared thermal images in combination with the face detection algorithm of the visible-light images and carrying out face tracking on the infrared thermal image sequence in combination with the face tracking algorithm of the visible-light image sequence. The algorithm provided by the invention can realize selective target (face) temperature measuring and selective target (face reaching the warning temperature) tracking and temperature measuring. The algorithm provided by the invention avoids the difficulty of whole registration of the infrared thermal images and the visible-light images with different properties. The algorithm is applied to the thermal infrared imagers and can organically combine the infrared thermal image information systems and the visible-light image information systems of the thermal infrared imagers, construct an image information system with certain intelligent information processing capability and expand the application field of the thermal infrared imagers.
Description
Technical field
The present invention relates to mode identification technology, the people's face that is specifically related to a kind of infrared thermal image sequence detects and the algorithm of following the tracks of.
Background technology
Since it is found that infrared radiation, people just begin constantly to use the whole bag of tricks that infrared radiation is detected, and are applied according to the characteristics of infrared radiation, have made various infrared eyes in succession.On using, infrared eye can be divided into two classes, and a class is an infrared viewing device, and another kind of is infrared thermometer.At civil area, using maximum infrared thermometers at present is non-refrigeration type micro-metering bolometer (also claiming thermal infrared imager), it can pass through means such as opto-electronic conversion, electric signal processing, converts the temperature distribution image of target object to video image, obtains object temperature by Temperature Scaling simultaneously.Popular along with epidemics such as SARS, bird flus, traditional inspection and quarantine work faces more and more stern challenge, functions such as that thermal infrared imager has is untouchable, volume is little, highly sensitive, continuous work and warning, the fine thermometric task of having finished in various places.
At present, two cover graphic information systems that most of thermal infrared imager is all integrated: thermal-induced imagery infosystem and visible light image information system.The thermal-induced imagery infosystem is the primary image infosystem of thermal infrared imager.The temperature of object in the thermal-induced imagery reflection visual field, still, the resolution of thermal-induced imagery is very low, and contrast is less, does not contain texture information.Therefore, utilize thermal-induced imagery to carry out pattern-recognition bigger difficulty is arranged.This causes in actual applications, even the temperature of certain object reaches the temperature of warning in the visual field, still, whether we are difficult to allow thermal infrared imager differentiate this object automatically is the object that we need monitor.
The visible light image information system is the auxiliary image information system of thermal infrared imager.Visible images can not reflect the temperature of object in the visual field, and still, the resolution of visible images is higher, contains abundant texture information, and is also fairly perfect based on the mode identification technology of visible images.Therefore, this two covers graphic information system of thermal-induced imagery infosystem and visible light image information system can be learnt from other's strong points to offset one's weaknesses fully, has complementary advantages, and constitutes cover graphic information system complete, that certain Intelligent Information Processing ability is arranged.Regrettably, in the application of at present most thermal infrared imagers, except the fusion of simple coarse shows, the visible light image information system in thermal infrared imager, almost become a kind of ornaments of cheapness.
The algorithm that the present invention detects in conjunction with visible images people face, the people's face that carries out thermal-induced imagery detects, for the people's face that reaches the temperature of warning, algorithm in conjunction with visible images sequence face tracking, carry out the face tracking of infrared thermal image sequence, and be background with the visible images, merge the thermal-induced imagery and the visible images that show people's face.
Summary of the invention
People's face that the present invention proposes a kind of infrared thermal image sequence detects and the algorithm of following the tracks of.This algorithm realizes that in conjunction with people's face detection algorithm of visible images people's face of thermal-induced imagery detects, and on this basis, realizes the face tracking of infrared thermal image sequence in conjunction with the face tracking algorithm of visible images sequence.
(1) realizes that in conjunction with visible images people face detection algorithm people's face of thermal-induced imagery detects
The detection of the human face region of A, visible images
People's face detection algorithm of ripe visible images is a lot of at present, and these algorithms all can be applicable to the present invention in principle.When carrying out the detection of people's face, use a certain size search window that visible images is pursued picture element scan earlier, whether comprise people's face in the detection window.Needs for the people's face that adapts to different size dimensions detects can use a scaling parameter, are used to change the size of scanning window, and do not need visible images is carried out convergent-divergent.Each size that changes window all will scan again and detect visible images.After search is finished, keep detected human face region, as the extraction of face template.
The extraction of B, face template
After finishing the detection of human face region, in each human face region, extract corresponding face template.Face template is the zone of people's face shape, only contains the people face part, does not comprise any background composition.The main process that face template extracts is as follows: at first, utilize the algorithm of skin color segmentation, face template is separated from human face region.Then, utilize the length and width of people's face to have these characteristics of certain ratio to get rid of pseudo-face template.These pseudo-face templates may since class area of skin color or area of skin color but non-face zone cause.At last, utilize the disposal route of mathematical morphologies such as corrosion and expansion that face template is done further processing.Erosion algorithm can be removed some little non-area of skin color, and these non-area of skin color cause mainly due to people's face.Expansion algorithm can be removed some " corner angle " on the face template edge, makes the profile of face template become clear and complete.
People's face of C, thermal-induced imagery detects
Use face template seeker's face in thermal-induced imagery.Before the search beginning, elder generation as the initial position of face template in thermal-induced imagery, is the position of face template in visible images a step-length with the unit picture element then, is the center with the initial position, moves as spiral fashion up and down in thermal-induced imagery.Face template whenever moves to a new position, and whether mate in the thermal-induced imagery zone that all will detect in face template and the template, and the thermal-induced imagery zone is human face region if coupling is then thought current face template place.
Thermal-induced imagery grey scale pixel value distribution range is little, and the gray level of most pixels concentrates in some zone, and its histogram has significantly unimodal or bimodal existence.Human body temperature (has comprised low temperature and high fever) greatly between 35 ℃ to 42 ℃, this temperature range is the elementary heat information of people's face, can not determine can not comprise people's face in the thermal-induced imagery part of this temperature range.When comprising a plurality of people's faces or other objects in the infrared thermal imaging system field range, can there be the situation that a plurality of people's faces block mutually or groups of people's face is blocked by other objects in thermal-induced imagery.People's face and people's neck temperature is roughly the same, and therefore, when searching the match people face in thermal-induced imagery when, the portion temperature that people's face and neck are bordered on is more or less the same.According to the characteristics of people's face in the above thermal-induced imagery, the step that people's face detects in the thermal-induced imagery is as follows:
1) detect in the thermal-induced imagery temperature in the face template region whether within the scope of human body temperature, if 90% and more than the temperature value of pixel satisfy this requirement, then the face template region may be people's face, forwards step 2 to); Otherwise, forward step 4) to.The human body temperature scope is decided to be 35 ℃-42 ℃, comprises low temperature and high fever.
In actual applications,, also can calculate in the thermal-induced imagery average of temperature and variance in the face template region in order to keep the robustness of algorithm to noise, if average within the scope of human body temperature and variance enough little, then forward step 2 to); Otherwise, forward step 4) to.
2) detect edge, face template region internal-external temperature difference and reach 5 ℃ and above part and whether account for more than 3/4 of face template region girth, if forward step 3) to, otherwise forward step 4) to.The environment temperature of considering the application scenario is generally below 30 ℃, and human body temperature is more than 35 ℃, so determine that temperature threshold is 5 ℃.This temperature threshold can be according to concrete application adjustment.
Similarly, in actual applications, in order to keep the robustness of algorithm to noise, also can calculate the temperature average of the pixel and the pixel outside the fringe region of inside, face template region in the thermal-induced imagery, if the absolute value of the difference of two averages is enough big, then forward step 3) to, otherwise forward step 4) to.
3) stop search, the match is successful.People's face is thought in the face template region in the thermal-induced imagery, thereby finishes the detection of people's face in the thermal-induced imagery, and is background with the visible images, merges the thermal-induced imagery and the visible images that show people's face.
4) detect the border whether face template has moved to thermal-induced imagery, if forward step 5) to, otherwise the mobile face template in thermal-induced imagery according to step-length and mobile rule forwards step 1) to.
5) abandon this person's face template, extract new face template from the next frame visible images, whether the region of ultra-red that detects in face template and the template in the next frame thermal-induced imagery mates, and forwards step 1) to.
The main flow chart of thermal-induced imagery people face detection algorithm is seen Fig. 1.
The sub-process figure of thermal-induced imagery people face detection algorithm sees Fig. 2.
(2) realize the face tracking of infrared thermal image sequence in conjunction with the face tracking algorithm of visible images sequence
On the basis that thermal-induced imagery people face detects, for the people's face that reaches the temperature of warning, face tracking algorithm in conjunction with the visible images sequence, carry out the face tracking of infrared thermal image sequence, and be background with the visible images, merge the thermal-induced imagery and the visible images that show tracked people's face, concrete steps are as follows:
A, the concrete application scenario of basis are provided with the temperature of warning.The human body normal body temperature on average between 36～37 ℃, exceeds this scope and generates heat exactly, is low-heat below 38 ℃, is high heat more than 39 ℃, and low 35 ℃ is hypothermia.When being applied to crowded public places such as airport or port when infrared thermal imaging system and carrying out the large tracts of land quarantine and examination, it is 39 ℃ that temperature threshold can be set.
B, for searched to people's face in the infrared thermal image sequence present frame, check whether its temperature reaches the temperature of warning.For the people's face that does not reach the temperature of warning, will can in infrared thermal image sequence, not follow the tracks of.For the people's face that reaches the temperature of warning, will with the visible images background, merge the thermal-induced imagery and the visible images that show these people's faces.
C, in the visible images sequence, the people's face that reaches the temperature of warning is followed the tracks of.In people's face of thermal-induced imagery detected, the position of people's face in thermal-induced imagery and the position in visible images were one to one.Therefore, in case in thermal-induced imagery, determine to reach people's face of the temperature of warning, also just in visible images, determined to reach simultaneously people's face of the temperature of warning.In the visible images sequence, utilize the algorithm of visible images sequence face tracking, people's face that search reaches the temperature of warning in the next frame of visible images sequence the position and extract face template.
D, the face template that in the next frame of visible images sequence, extracts according to the people's face that reaches the temperature of warning, seeker's face in the next frame of infrared thermal image sequence, thus be implemented in the infrared thermal image sequence tracking to the people's face that reaches the temperature of warning.
The process flow diagram of thermal-induced imagery face tracking algorithm is seen Fig. 3.
Characteristics of the present invention
(1) algorithm of the present invention's proposition can be realized selectable target thermometric in thermal-induced imagery.The present invention detects people's face earlier in visible images, utilize the result who detects to detect people's face again in thermal-induced imagery.The algorithm that the present invention proposes can detect the temperature of people's face in the thermal infrared imager visual field separately, effectively avoids the influence of non-face object to people's face temperature detection.At present, thermal infrared imager all is to carry out thermometric alike to the object in its visual field, can not distinguish target object and non-target object effectively, thereby can not carry out selectable target thermometric effectively.
(2) algorithm of the present invention's proposition can be realized selectable target following thermometric in infrared thermal image sequence.The present invention is on the basis that thermal-induced imagery people face detects, according to concrete application scenario the temperature of warning is set, for the people's face that reaches the temperature of warning, utilize the algorithm of visible images sequence face tracking, be implemented on the infrared thermal image sequence tracking and temperature testing to the people's face that reaches the temperature of warning.
(3) though thermal-induced imagery in the thermal infrared imager and visible images all are the images of same visual field, but, these two images independently absorb imaging by diverse location and camera lens of different nature, and therefore, these two images are all inequality at aspects such as the depth of field, resolution and quantisation depth.In addition, thermal-induced imagery has only the thermal information of scenery, do not have the texture information of scenery, and visible images does not have the thermal information of scenery, has only the texture information of scenery.Therefore, the whole registration of these two images is very difficult things.The algorithm that the present invention proposes has been avoided the whole registration problem of thermal-induced imagery and visible images by the mode of only target (people's face) being carried out match search and only target (people's face) is merged demonstration in thermal-induced imagery in visible images.
(4) algorithm of the present invention's proposition organically combines thermal-induced imagery infosystem in the thermal infrared imager and visible light image information system, constituted a graphic information system, expanded the application of thermal infrared imager with certain Intelligent Information Processing function.
Description of drawings
Fig. 1, thermal-induced imagery people face detection algorithm main flow chart
Fig. 2, thermal-induced imagery people face detection algorithm sub-process figure
Fig. 3, infrared thermal image sequence face tracking algorithm flow chart
Specific embodiments
Step 1: people's face of realizing thermal-induced imagery in conjunction with visible images people face detection algorithm detects
The people's face detection that realizes thermal-induced imagery in conjunction with visible images people face detection algorithm comprises following step: people's face of the detection of the human face region of visible images, the extraction of face template and thermal-induced imagery detects.Explanation respectively below.
(1) detection of the human face region of visible images
People's face detection algorithm of ripe visible images is a lot of at present, and these algorithms all can be applicable to the present invention in principle.This specific embodiments adopts the Adaboost algorithm.
The Adaboost algorithm is a kind of classifier algorithm, and its basic thought is to utilize the class Harr feature of sample to carry out the sorter training, finally obtains the boosted sorter of a cascade.After sorter has been trained, just can be applied to the detection of the area-of-interest (size identical) in the input picture with training sample.Detect the target area sorter and be output as 1, otherwise be output as 0.When detecting entire image, can be in image the mobile search window, detect each position and determine possible target.
When carrying out the detection of people's face, need precondition to be used on the cascade type Haar sorter that people's face detects, use a certain size search window that input picture is pursued picture element scan then, and use sorter to remove whether to comprise in the detection window people's face.In order to adapt to the detection needs of different big person of low position's faces, used a scaling parameter, be used to change the size (then not needing input picture is carried out convergent-divergent) of scanning window, change window size at every turn and all can scan again and detect input picture.
(2) extraction of face template
Take the skin color segmentation algorithm that the human face region that previous step obtains is suddenly handled in this specific embodiments, be partitioned into people face position.Consider that the method processing descendant face edge smoothly reaches the face place inadequately and can form " cavity ", and can be subjected to the influence of class area of skin color, so need doing mathematics morphology to handle and the filtering of class area of skin color again.Several committed steps are as follows:
A, skin color segmentation
In this specific embodiments, at first finish from the conversion of RGB → HSV color space for the pixel in the human face region of determining after detecting.Judge according to the value of H whether certain pixel belongs to the colour of skin, take in this specific embodiments when 0.02＜H＜0.08, to think that this pixel belongs to the colour of skin.
V＝max(R，G，B)
Wherein
Delta=V-min (R, G, B)
B, mathematical morphology are handled
The mathematical morphology disposal route of having taked corrosion in this specific embodiments and having expanded is handled the zone that previous step obtains.
Corrosion is a kind of process of eliminating all frontier points of object, and consequently Sheng Xia object is along the area of its periphery than the little pixel of the original.If the width of object any point is less than 3 pixels, it is named a person for a particular job at this and becomes unconnected (becoming two objects) so.The object that width in any direction is not more than 2 pixels will be removed.Corrosion can be to removing little and insignificant object in the width of cloth split image.
Expansion is the process that all background dots that contact with object is merged to this object.The result of process makes the area of object increase the point of respective numbers.Be less than 3 pixels if two objects a bit are separated by at certain, they will be communicated with to get up (being merged into an object) at this point.Expansion can be filled up the cavity of cutting apart the back object.
C, the filtering of class area of skin color
After handling based on the filtering method of mathematical morphology, the fritter noise great majority in the image are eliminated, but some less class area of skin color is still deposited greatly in the background.In order to delete dummy's face zone, we must analyze and calculate these zones.At first class area of skin color mark is come out, and then utilize the length breadth ratio of people's face to meet these characteristics of certain proportion and eliminate, get rid of those wide or long or excessive too small zones of length breadth ratio.
In order to determine the length breadth ratio in a certain zone, length L and width W that must the zone be obtained respectively.But, groups of people's face tilts because may depositing bigger rotations, this make left and right, upper and lower 4 summits can't directly utilize this zone coordinate figure (here, the coordinate system that adopts is to be initial point with the lower left corner of image, level is to the right the positive dirction of X-axis, is the positive dirction of Y-axis vertically upward) judge.Its detailed process is: add up the coordinate figure of being had a few on this zone boundary, seek the coordinate (X that has minimum, maximum X component on the X-axis
Min, X
Max), reach the minimum on the Y-axis, the coordinate (Y of maximum Y component
Min, Y
Max), L=X
Max-X
Min, W=Y
Max-Y
MinValue promptly is length and width (wide length) parameter value of people's face.The ratio of L and W
Be regional length and width (or the wide length) ratio of being asked.If people's face is a vertical front side, then this ratio should approach 1.2, but because there is rotation in people's face, and colour of skin similarity is cut apart and may be caused people's face incidence as same Region Segmentation, so the upper limit of r can suitably be amplified, to prevent the judgement of correct cut zone as mistake.In this specific embodiments, the span of r is (0.5,2), does not belong to the then directly deletion of candidate region of this scope.
(3) people's face of thermal-induced imagery detects
A, determine the initial position of face template in thermal-induced imagery
Visible images and infrared chart similarly are the image of same visual field, the pixel level unanimity, and pixel separation is identical.In order full out to find corresponding people's face, set the position of face template in visible images as the initial position of face template in thermal-induced imagery at thermal-induced imagery.
B, determine face template moving step length and rule
Human face region in the Infrared Thermogram varies, and in order to improve matching precision and accuracy rate as far as possible, determines with the unit picture element to be step-length.The mobile rule of face template is to be the center with the initial position, with spiral helicine shape to moving away from central spot.
C, use face template seeker's face in thermal-induced imagery
Before the search beginning, elder generation as the initial position of face template in thermal-induced imagery, is the position of face template in visible images a step-length with the unit picture element then, is the center with the initial position, moves as spiral fashion up and down in thermal-induced imagery.Face template whenever moves to a new position, and whether all will detect the face template region is human face region.
Thermal-induced imagery grey scale pixel value distribution range is little, and the gray level of most pixels concentrates in some zone, and its histogram has significantly unimodal or bimodal existence.Human body temperature (has comprised low temperature and high fever) greatly between 35 ℃ to 42 ℃, this temperature range is the elementary heat information of people's face, can not determine can not comprise people's face in the thermal-induced imagery part of this temperature range.When comprising a plurality of people's faces or other objects in the infrared thermal imaging system field range, can there be the situation that a plurality of people's faces block mutually or groups of people's face is blocked by other objects in thermal-induced imagery.People's face and people's neck temperature is roughly the same, and therefore, when searching the match people face in thermal-induced imagery when, the temperature that people's face and neck are bordered on part is more or less the same.According to the characteristics of people's face in the above thermal-induced imagery, the step that people's face detects in the thermal-induced imagery is as follows:
1) detect in the thermal-induced imagery temperature in the face template region whether within the scope of human body temperature, if 90% and more than the temperature value of pixel satisfy this requirement, then may there be people's face in the face template region, forwards step 2 to); Otherwise, forward step 4) to.The human body temperature scope is decided to be 35 ℃-42 ℃, comprises low temperature and high fever.
In actual applications, in order to keep the robustness of algorithm, also can calculate the temperature average T in the face template region in the thermal-induced imagery to noise
AvgWith the temperature variances sigma
Wherein N is the number of face template place region of ultra-red interior pixel point, T
iBe the temperature value of i pixel of template place region of ultra-red, i=1,2 ..., N.
If T
AvgNot in the human body temperature scope, think that then the face template region is not people's face, forwards step 4) to; Otherwise, with the variance threshold values σ of σ and setting
ThRelatively, if σ＜σ
Th, think that then the face template region may be people's face, forwards step 2 to), otherwise forward step 4) to.
2) detect edge, face template region internal-external temperature difference and reach 5 ℃ and above part and whether account for more than 3/4 of face template region girth, if forward step 3) to, otherwise forward step 4) to.The environment temperature of considering the application scenario is generally below 30 ℃, and human body temperature is more than 35 ℃, so determine that temperature threshold is 5 ℃.This temperature threshold can be according to concrete application adjustment.
Similarly, in actual applications, in order to keep the robustness of algorithm, also can investigate the temperature average outside region of ultra-red inside, face template place in the thermal-induced imagery and the edge thereof to noise, if the difference of two averages is enough big, think that then the template region may be people's face.Specific practice is: the template place region of ultra-red of at first finding out before front template place region of ultra-red and Moving Unit pixel is compared the pixel of increase and the pixel of minimizing, the pixel number that increases is identical with the pixel number of minimizing, be designated as M, again the temperature average T of the pixel of pixel that calculate to increase respectively and minimizing
Avg1And T
Avg2
α wherein
iBe the temperature value of i pixel increasing, β
iBe the temperature value of i pixel reducing, i=1,2 ..., M.If working as front template place region of ultra-red is human face region, then the temperature average T of the pixel of Zeng Jiaing
Avg1Should be in the temperature range of human body, the temperature average T of the pixel of minimizing (the outer pixel of half fringe region of promptly working as the region of ultra-red at front template place)
Avg2With temperature average T when front template place region of ultra-red pixel
AvgThe absolute value of difference should be greater than temperature threshold T
ThTherefore, if T
Avg1In the human body temperature scope, do not forward step 4) to; Otherwise, calculate | T
Avg-T
Avg2|, if | T
Avg-T
Avg2|＜T
Th, forward step 4) to.
If | T
Avg-T
Avg2| 〉=T
Th, promptly the temperature value when the outer pixel of half fringe region of the region of ultra-red at front template place meets the demands, and investigates when the outer pixel of other half fringe region of front template place region of ultra-red again.In like manner, the template place region of ultra-red of finding out earlier after front template place region of ultra-red and Moving Unit pixel is compared the pixel of increase and the pixel of minimizing, the pixel number that increases is identical with the pixel number of minimizing, be designated as L, again the temperature average T of the pixel of pixel that calculate to increase respectively and minimizing
Avg3And T
Avg4
χ wherein
iBe the temperature value of the pixel that increases, δ
iBe the temperature value of the pixel that reduces, i=1,2 ..., L.If working as front template place region of ultra-red is human face region, then the temperature average T of the pixel of Jian Shaoing
Avg4Should be in the temperature range of human body, the temperature average T of the pixel of the increase pixel of other half fringe region of the region of ultra-red at front template place (promptly when)
Avg3With temperature average T when front template place region of ultra-red pixel
AvgThe absolute value of difference should be greater than temperature threshold T
ThTherefore, if T
Avg4In the human body temperature scope, do not forward step 4) to; Otherwise, calculate | T
Avg-T
Avg3|, if | T
Avg-T
Avg3|＜T
Th, forward step 4) to; Otherwise, forward step 3) to.
So far, the fringe region of the region of ultra-red at current face template place is all investigated and is finished.Keep the region of ultra-red at the template place before the Moving Unit pixel before this substep carries out, when the region of ultra-red at front template place and the region of ultra-red at the template place after the Moving Unit pixel.
3) stop search, the match is successful.People's face is thought in the face template region in the thermal-induced imagery, thereby finishes the detection of people's face in the thermal-induced imagery, and is background with the visible images, merges the thermal-induced imagery and the visible images that show people's face.
4) detect the border whether face template has moved to thermal-induced imagery, if forward step 5) to, otherwise the mobile face template in thermal-induced imagery according to step-length and mobile rule forwards step 1) to.
5) abandon this person's face template, from the next frame visible images, extract new face template, in the next frame thermal-induced imagery, detect whether the face template region is human face region, forward step 1) to.
Step 2: the face tracking of realizing infrared thermal image sequence in conjunction with the face tracking algorithm of visible images sequence
This infrared face track algorithm is based on the applied infrared face detection algorithm of step 1, on the basis that this infrared face detects, from the infrared face that has detected, find the infrared face that reaches the temperature of warning,, obtain the visible light people face of this infrared face correspondence according to the corresponding relation that has obtained.Then, using visible light people face algorithm is followed the tracks of this visible light people face.At last, detect the infrared face of every frame visible light facial image correspondence again, can realize tracking this infrared face.Concrete steps are as follows:
(1) according to concrete application scenario the temperature of warning is set
The human body normal body temperature on average between 36～37 ℃, exceeds this scope and generates heat exactly, is low-heat below 38 ℃, is high heat more than 39 ℃, and low 35 ℃ is hypothermia.Be applied to crowded public places such as airport or port when infrared thermal imaging system and carry out large tracts of land quarantine and examination when work, it is 39 ℃ that temperature threshold can be set.
(2) in the infrared face that has detected, search for the infrared face that reaches the temperature of warning
For searched to people's face in the infrared thermal image sequence present frame, check whether its temperature reaches the temperature of warning.For the people's face that does not reach the temperature of warning, will can in infrared thermal image sequence, not follow the tracks of.For the people's face that reaches the temperature of warning, will with the visible images background, merge the thermal-induced imagery and the visible images that show these people's faces.
(3) the visible light people face of the infrared face correspondence that reaches the temperature of warning is followed the tracks of.
In people's face of thermal-induced imagery detected, the position of people's face in thermal-induced imagery and the position in visible images were one to one.Therefore, in case in thermal-induced imagery, determine to reach people's face of the temperature of warning, also just in visible images, determined to reach simultaneously people's face of the temperature of warning.In the visible images sequence, utilize the algorithm of visible images sequence face tracking, people's face that search reaches the temperature of warning in the next frame of visible images sequence the position and extract face template.In principle, any effective visible light face tracking algorithm all can be applicable in this step.
(4) detect the infrared face of every frame visible light people face correspondence again, realize tracking infrared face
Use the infrared face detection algorithm to following the tracks of each the frame visible light facial image that obtains, extract face template, carry out the infrared face coupling, promptly carry out infrared face and detect, determine the position of people's face at visible images sequence next frame at corresponding thermal-induced imagery.From whole video stream, as long as this people's face that reaches the temperature of warning does not leave the thermal infrared imager field range as yet, the visible light facial image that tracking is obtained carries out the infrared face detection, can be implemented in track human faces in the infrared thermal image sequence.
Claims (3)
1. the present invention proposes a kind of detection of people's face and the algorithm of following the tracks of of infrared thermal image sequence, it is characterized in that
A, the people's face that carries out thermal-induced imagery in conjunction with people's face detection algorithm of visible images detect;
B, carry out the face tracking of infrared thermal image sequence in conjunction with the face tracking algorithm of visible images sequence.
2. people's face of infrared thermal image sequence as claimed in claim 1 detects and track algorithm, it is characterized in that described steps A specifically comprises: for thermal-induced imagery that same visual field is taken simultaneously and visible images, at first people's face detection algorithm of using visible light image detects at the enterprising pedestrian's face of visible images, obtain the shape and the position of people's face on the visible images, the template of the shape of people's face on the visible images as people's face detection on the thermal-induced imagery, the initial position of the position of people's face on the visible images as people's face detection on the thermal-induced imagery, calculate the average of temperature in the thermal-induced imagery cope match-plate pattern and the average of variance and template environment temperature, if the average of temperature is in the tolerance band of body temperature in the template, the difference of the average of the average of the enough little and interior temperature of template of the variance of temperature and template environment temperature is enough big in the template, judge that then the template position is the position of people's face on the thermal-induced imagery, otherwise, on thermal-induced imagery, calculate again and differentiate, till detecting people's face according to the position of certain regular movable platen.
3. people's face of infrared thermal image sequence as claimed in claim 1 detects and track algorithm, it is characterized in that described step B specifically comprises: for infrared thermal image sequence that Same Scene is taken simultaneously and visible images sequence, at first people's face detection algorithm of using visible light image is determined the position of people's face at visible images sequence present frame, application rights requires 2 described methods then, determine the position of people's face at the infrared thermal image sequence present frame, the method of using visible light image sequence face tracking then, determine the position of people's face at visible images sequence next frame, at last, application rights requires 2 described methods once more, determine the position of people's face, thereby be implemented in track human faces in the infrared thermal image sequence at the infrared thermal image sequence next frame.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010105381 CN101793562B (en) | 2010-01-29 | 2010-01-29 | Face detection and tracking algorithm of infrared thermal image sequence |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010105381 CN101793562B (en) | 2010-01-29 | 2010-01-29 | Face detection and tracking algorithm of infrared thermal image sequence |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101793562A true CN101793562A (en) | 2010-08-04 |
| CN101793562B CN101793562B (en) | 2013-04-24 |
Family
ID=42586378
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201010105381 Expired - Fee Related CN101793562B (en) | 2010-01-29 | 2010-01-29 | Face detection and tracking algorithm of infrared thermal image sequence |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101793562B (en) |
Cited By (44)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102004906A (en) * | 2010-11-18 | 2011-04-06 | 无锡中星微电子有限公司 | Face identification system and method |
| CN103678968A (en) * | 2012-09-11 | 2014-03-26 | 索尼公司 | Gesture- and expression-based authentication |
| CN103996182A (en) * | 2013-02-15 | 2014-08-20 | 欧姆龙株式会社 | Image processing device, image processing method, and image processing program |
| CN105447857A (en) * | 2015-11-17 | 2016-03-30 | 电子科技大学 | Feature extraction method of pulsed eddy-current infrared thermal image |
| CN105760816A (en) * | 2016-01-27 | 2016-07-13 | 四川长虹电器股份有限公司 | Method of intelligently recognizing human body sleep gesture under strong wind cooling |
| CN105844240A (en) * | 2016-03-23 | 2016-08-10 | 深圳云天励飞技术有限公司 | Method and device for detecting human faces in infrared temperature measurement system |
| CN105912908A (en) * | 2016-04-14 | 2016-08-31 | 苏州优化智能科技有限公司 | Infrared-based real person living body identity verification method |
| TWI547896B (en) * | 2014-01-28 | 2016-09-01 | 姜崇義 | Intelligent monitoring system |
| CN105931254A (en) * | 2016-05-17 | 2016-09-07 | 北京市检验检疫科学技术研究院 | Low-temperature inspection method and system for entry and exit inspection and quarantine of frontier port |
| CN106327461A (en) * | 2015-06-16 | 2017-01-11 | 浙江大华技术股份有限公司 | Image processing method and device used for monitoring |
| CN106991688A (en) * | 2017-03-09 | 2017-07-28 | 广东欧珀移动通信有限公司 | Human body tracing method, human body tracking device and electronic installation |
| CN107016348A (en) * | 2017-03-09 | 2017-08-04 | 广东欧珀移动通信有限公司 | With reference to the method for detecting human face of depth information, detection means and electronic installation |
| CN107133592A (en) * | 2017-05-05 | 2017-09-05 | 国网江苏省电力公司无锡供电公司 | Electricity substation is using infrared thermal imaging and the human body target feature detection algorithm of visual light imaging technological incorporation |
| CN107451543A (en) * | 2017-07-14 | 2017-12-08 | 广东欧珀移动通信有限公司 | Event-handling method and Related product |
| CN107595254A (en) * | 2017-10-17 | 2018-01-19 | 黄晶 | A kind of infrared ray health monitor method and system |
| CN107844773A (en) * | 2017-11-10 | 2018-03-27 | 广东日月潭电源科技有限公司 | A kind of Three-Dimensional Dynamic Intelligent human-face recognition methods and system |
| CN107871324A (en) * | 2017-11-02 | 2018-04-03 | 中国南方电网有限责任公司超高压输电公司检修试验中心 | One kind is based on twin-channel method for tracking target and device |
| CN108010051A (en) * | 2017-11-29 | 2018-05-08 | 广西师范大学 | Multisource video subject fusion tracking based on AdaBoost algorithms |
| CN108073891A (en) * | 2017-11-10 | 2018-05-25 | 广东日月潭电源科技有限公司 | A kind of 3 D intelligent face identification system |
| CN108268839A (en) * | 2018-01-05 | 2018-07-10 | 北京万相融通科技股份有限公司 | A kind of live body verification method and its system |
| CN108419130A (en) * | 2018-04-12 | 2018-08-17 | 四川斐讯信息技术有限公司 | A kind of method and system that multimedia equipment volume is intelligently adjusted |
| CN108764058A (en) * | 2018-05-04 | 2018-11-06 | 吉林大学 | A kind of dual camera human face in-vivo detection method based on thermal imaging effect |
| CN109063701A (en) * | 2018-08-08 | 2018-12-21 | 合肥英睿***技术有限公司 | Labeling method, device, equipment and the storage medium of target in a kind of infrared image |
| CN109060143A (en) * | 2018-08-16 | 2018-12-21 | 南京农业大学 | Boar Thermometer System and method based on thermal infrared technology |
| CN109580005A (en) * | 2018-12-24 | 2019-04-05 | 国网山西省电力公司检修分公司 | A kind of failure analysis methods, device, equipment and the storage medium of wire-connection point |
| CN109857175A (en) * | 2019-01-23 | 2019-06-07 | 南京邮电大学 | A kind of AI cognitive method of non-intrusion type human thermal comfort |
| CN110044486A (en) * | 2019-03-11 | 2019-07-23 | 武汉高德智感科技有限公司 | Method, apparatus, the equipment of repetition of alarms are avoided for human body inspection and quarantine system |
| CN110060272A (en) * | 2018-01-18 | 2019-07-26 | 杭州海康威视数字技术股份有限公司 | Determination method, apparatus, electronic equipment and the storage medium of human face region |
| CN110411570A (en) * | 2019-06-28 | 2019-11-05 | 武汉高德智感科技有限公司 | Infrared human body temperature screening method based on human testing and human body tracking technology |
| CN111191574A (en) * | 2019-12-26 | 2020-05-22 | 新绎健康科技有限公司 | Method and device for acquiring viscera partition temperature of facial examination |
| CN111222425A (en) * | 2019-12-26 | 2020-06-02 | 新绎健康科技有限公司 | Method and device for positioning facial features |
| CN111310692A (en) * | 2020-02-25 | 2020-06-19 | 云从科技集团股份有限公司 | Detection object management method, system, machine readable medium and equipment |
| CN111317452A (en) * | 2020-03-02 | 2020-06-23 | 清华大学 | Automatic method for rapidly detecting body temperature of person in passenger car |
| CN111339951A (en) * | 2020-02-26 | 2020-06-26 | 北京迈格威科技有限公司 | Body temperature measuring method, device and system |
| CN111353438A (en) * | 2020-03-02 | 2020-06-30 | 成都三零凯天通信实业有限公司 | AI-based public place emergency medical person searching method |
| CN111426388A (en) * | 2020-03-31 | 2020-07-17 | 高新兴科技集团股份有限公司 | Personnel body temperature measuring method, system, computer storage medium and electronic equipment |
| CN111854963A (en) * | 2020-06-11 | 2020-10-30 | 浙江大华技术股份有限公司 | Temperature detection method, device, equipment and computer equipment |
| CN111951444A (en) * | 2020-03-27 | 2020-11-17 | 武汉小码联城科技有限公司 | Subway trip epidemic prevention and control method and system, station entry gate and server |
| CN112380938A (en) * | 2020-11-04 | 2021-02-19 | 浙江大华技术股份有限公司 | Face recognition and temperature measurement method, device, equipment and medium |
| US10943092B2 (en) | 2018-05-23 | 2021-03-09 | ClairLabs Ltd. | Monitoring system |
| CN112507783A (en) * | 2020-10-29 | 2021-03-16 | 上海交通大学 | Mask face detection, identification, tracking and temperature measurement method based on attention mechanism |
| CN113379772A (en) * | 2021-07-06 | 2021-09-10 | 新疆爱华盈通信息技术有限公司 | Mobile temperature measurement method based on background elimination and tracking algorithm in complex environment |
| CN113494964A (en) * | 2020-03-20 | 2021-10-12 | 深圳云天励飞技术有限公司 | Personnel temperature measurement method and device, electronic equipment and storage medium |
| TWI768641B (en) * | 2020-03-13 | 2022-06-21 | 大陸商深圳市商湯科技有限公司 | Monitoring method, electronic equipment and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101303724A (en) * | 2007-05-10 | 2008-11-12 | ***股份有限公司 | Authentication authorization method and system |
| CN101404060B (en) * | 2008-11-10 | 2010-06-30 | 北京航空航天大学 | Human face recognition method based on visible light and near-infrared Gabor information amalgamation |
-
2010
- 2010-01-29 CN CN 201010105381 patent/CN101793562B/en not_active Expired - Fee Related
Cited By (56)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102004906A (en) * | 2010-11-18 | 2011-04-06 | 无锡中星微电子有限公司 | Face identification system and method |
| CN103678968A (en) * | 2012-09-11 | 2014-03-26 | 索尼公司 | Gesture- and expression-based authentication |
| CN103996182A (en) * | 2013-02-15 | 2014-08-20 | 欧姆龙株式会社 | Image processing device, image processing method, and image processing program |
| CN103996182B (en) * | 2013-02-15 | 2017-12-12 | 欧姆龙株式会社 | Image processing apparatus, image processing method |
| TWI547896B (en) * | 2014-01-28 | 2016-09-01 | 姜崇義 | Intelligent monitoring system |
| CN106327461B (en) * | 2015-06-16 | 2019-11-15 | 浙江大华技术股份有限公司 | A kind of image processing method and device for monitoring |
| CN106327461A (en) * | 2015-06-16 | 2017-01-11 | 浙江大华技术股份有限公司 | Image processing method and device used for monitoring |
| CN105447857A (en) * | 2015-11-17 | 2016-03-30 | 电子科技大学 | Feature extraction method of pulsed eddy-current infrared thermal image |
| CN105447857B (en) * | 2015-11-17 | 2018-05-04 | 电子科技大学 | The feature extracting method of impulse eddy current thermal-induced imagery |
| CN105760816A (en) * | 2016-01-27 | 2016-07-13 | 四川长虹电器股份有限公司 | Method of intelligently recognizing human body sleep gesture under strong wind cooling |
| CN105844240A (en) * | 2016-03-23 | 2016-08-10 | 深圳云天励飞技术有限公司 | Method and device for detecting human faces in infrared temperature measurement system |
| CN105912908A (en) * | 2016-04-14 | 2016-08-31 | 苏州优化智能科技有限公司 | Infrared-based real person living body identity verification method |
| CN105931254A (en) * | 2016-05-17 | 2016-09-07 | 北京市检验检疫科学技术研究院 | Low-temperature inspection method and system for entry and exit inspection and quarantine of frontier port |
| CN105931254B (en) * | 2016-05-17 | 2019-03-26 | 北京市检验检疫科学技术研究院 | A kind of frontier port inspection and quarantining for import/export low temperature checking method and system |
| CN106991688A (en) * | 2017-03-09 | 2017-07-28 | 广东欧珀移动通信有限公司 | Human body tracing method, human body tracking device and electronic installation |
| CN107016348A (en) * | 2017-03-09 | 2017-08-04 | 广东欧珀移动通信有限公司 | With reference to the method for detecting human face of depth information, detection means and electronic installation |
| CN107133592A (en) * | 2017-05-05 | 2017-09-05 | 国网江苏省电力公司无锡供电公司 | Electricity substation is using infrared thermal imaging and the human body target feature detection algorithm of visual light imaging technological incorporation |
| CN107451543A (en) * | 2017-07-14 | 2017-12-08 | 广东欧珀移动通信有限公司 | Event-handling method and Related product |
| CN107595254A (en) * | 2017-10-17 | 2018-01-19 | 黄晶 | A kind of infrared ray health monitor method and system |
| CN107871324B (en) * | 2017-11-02 | 2020-01-24 | 中国南方电网有限责任公司超高压输电公司检修试验中心 | Target tracking method and device based on double channels |
| CN107871324A (en) * | 2017-11-02 | 2018-04-03 | 中国南方电网有限责任公司超高压输电公司检修试验中心 | One kind is based on twin-channel method for tracking target and device |
| CN107844773A (en) * | 2017-11-10 | 2018-03-27 | 广东日月潭电源科技有限公司 | A kind of Three-Dimensional Dynamic Intelligent human-face recognition methods and system |
| CN108073891A (en) * | 2017-11-10 | 2018-05-25 | 广东日月潭电源科技有限公司 | A kind of 3 D intelligent face identification system |
| CN108010051A (en) * | 2017-11-29 | 2018-05-08 | 广西师范大学 | Multisource video subject fusion tracking based on AdaBoost algorithms |
| CN108268839A (en) * | 2018-01-05 | 2018-07-10 | 北京万相融通科技股份有限公司 | A kind of live body verification method and its system |
| CN110060272A (en) * | 2018-01-18 | 2019-07-26 | 杭州海康威视数字技术股份有限公司 | Determination method, apparatus, electronic equipment and the storage medium of human face region |
| CN108419130A (en) * | 2018-04-12 | 2018-08-17 | 四川斐讯信息技术有限公司 | A kind of method and system that multimedia equipment volume is intelligently adjusted |
| CN108419130B (en) * | 2018-04-12 | 2022-04-08 | 深圳市维度印象科技有限公司 | Method and system for intelligently adjusting volume of multimedia equipment |
| CN108764058A (en) * | 2018-05-04 | 2018-11-06 | 吉林大学 | A kind of dual camera human face in-vivo detection method based on thermal imaging effect |
| CN108764058B (en) * | 2018-05-04 | 2021-05-25 | 吉林大学 | Double-camera face in-vivo detection method based on thermal imaging effect |
| US10943092B2 (en) | 2018-05-23 | 2021-03-09 | ClairLabs Ltd. | Monitoring system |
| CN109063701A (en) * | 2018-08-08 | 2018-12-21 | 合肥英睿***技术有限公司 | Labeling method, device, equipment and the storage medium of target in a kind of infrared image |
| CN109060143A (en) * | 2018-08-16 | 2018-12-21 | 南京农业大学 | Boar Thermometer System and method based on thermal infrared technology |
| CN109580005A (en) * | 2018-12-24 | 2019-04-05 | 国网山西省电力公司检修分公司 | A kind of failure analysis methods, device, equipment and the storage medium of wire-connection point |
| CN109857175A (en) * | 2019-01-23 | 2019-06-07 | 南京邮电大学 | A kind of AI cognitive method of non-intrusion type human thermal comfort |
| CN109857175B (en) * | 2019-01-23 | 2020-11-03 | 南京邮电大学 | Non-invasive AI (artificial intelligence) sensing method for human body thermal comfort |
| CN110044486B (en) * | 2019-03-11 | 2020-05-22 | 武汉高德智感科技有限公司 | Method, device and equipment for avoiding repeated alarm of human body inspection and quarantine system |
| CN110044486A (en) * | 2019-03-11 | 2019-07-23 | 武汉高德智感科技有限公司 | Method, apparatus, the equipment of repetition of alarms are avoided for human body inspection and quarantine system |
| CN110411570A (en) * | 2019-06-28 | 2019-11-05 | 武汉高德智感科技有限公司 | Infrared human body temperature screening method based on human testing and human body tracking technology |
| CN110411570B (en) * | 2019-06-28 | 2020-08-28 | 武汉高德智感科技有限公司 | Infrared human body temperature screening method based on human body detection and human body tracking technology |
| CN111222425A (en) * | 2019-12-26 | 2020-06-02 | 新绎健康科技有限公司 | Method and device for positioning facial features |
| CN111191574A (en) * | 2019-12-26 | 2020-05-22 | 新绎健康科技有限公司 | Method and device for acquiring viscera partition temperature of facial examination |
| CN111310692A (en) * | 2020-02-25 | 2020-06-19 | 云从科技集团股份有限公司 | Detection object management method, system, machine readable medium and equipment |
| CN111339951A (en) * | 2020-02-26 | 2020-06-26 | 北京迈格威科技有限公司 | Body temperature measuring method, device and system |
| CN111317452A (en) * | 2020-03-02 | 2020-06-23 | 清华大学 | Automatic method for rapidly detecting body temperature of person in passenger car |
| CN111353438A (en) * | 2020-03-02 | 2020-06-30 | 成都三零凯天通信实业有限公司 | AI-based public place emergency medical person searching method |
| TWI768641B (en) * | 2020-03-13 | 2022-06-21 | 大陸商深圳市商湯科技有限公司 | Monitoring method, electronic equipment and storage medium |
| CN113494964A (en) * | 2020-03-20 | 2021-10-12 | 深圳云天励飞技术有限公司 | Personnel temperature measurement method and device, electronic equipment and storage medium |
| CN111951444A (en) * | 2020-03-27 | 2020-11-17 | 武汉小码联城科技有限公司 | Subway trip epidemic prevention and control method and system, station entry gate and server |
| CN111426388A (en) * | 2020-03-31 | 2020-07-17 | 高新兴科技集团股份有限公司 | Personnel body temperature measuring method, system, computer storage medium and electronic equipment |
| CN111854963A (en) * | 2020-06-11 | 2020-10-30 | 浙江大华技术股份有限公司 | Temperature detection method, device, equipment and computer equipment |
| CN112507783A (en) * | 2020-10-29 | 2021-03-16 | 上海交通大学 | Mask face detection, identification, tracking and temperature measurement method based on attention mechanism |
| CN112380938A (en) * | 2020-11-04 | 2021-02-19 | 浙江大华技术股份有限公司 | Face recognition and temperature measurement method, device, equipment and medium |
| CN112380938B (en) * | 2020-11-04 | 2024-05-03 | 浙江大华技术股份有限公司 | Face recognition and temperature measurement method, device, equipment and medium |
| CN113379772A (en) * | 2021-07-06 | 2021-09-10 | 新疆爱华盈通信息技术有限公司 | Mobile temperature measurement method based on background elimination and tracking algorithm in complex environment |
| CN113379772B (en) * | 2021-07-06 | 2022-10-11 | 新疆爱华盈通信息技术有限公司 | Mobile temperature measurement method based on background elimination and tracking algorithm in complex environment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101793562B (en) | 2013-04-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101793562B (en) | Face detection and tracking algorithm of infrared thermal image sequence | |
| Setjo et al. | Thermal image human detection using Haar-cascade classifier | |
| Davis et al. | Fusion-based background-subtraction using contour saliency | |
| Kwak et al. | Pedestrian tracking using online boosted random ferns learning in far-infrared imagery for safe driving at night | |
| CN100504910C (en) | Detection method and apparatus of human | |
| Davis et al. | Background-subtraction in thermal imagery using contour saliency | |
| Davis et al. | Robust Background-Subtraction for Person Detection in Thermal Imagery. | |
| US9274204B2 (en) | Camera tracing and surveillance system and method for security using thermal image coordinate | |
| CN103824070B (en) | A kind of rapid pedestrian detection method based on computer vision | |
| Yasuno et al. | Pedestrian detection and tracking in far infrared images | |
| CN109598242B (en) | Living body detection method | |
| Kumar et al. | Fusion of thermal infrared and visible spectrum video for robust surveillance | |
| CN106991654A (en) | Human body beautification method and apparatus and electronic installation based on depth | |
| Han et al. | Detecting moving humans using color and infrared video | |
| Mu | Ear detection based on skin-color and contour information | |
| Labati et al. | Weight estimation from frame sequences using computational intelligence techniques | |
| Ramezani et al. | A new DSWTS algorithm for real-time pedestrian detection in autonomous agricultural tractors as a computer vision system | |
| CN103456123B (en) | A kind of video smoke detection method based on flowing with diffusion characteristic | |
| Miller et al. | Person tracking in UAV video | |
| KR101146417B1 (en) | Apparatus and method for tracking salient human face in robot surveillance | |
| CN107145820A (en) | Eyes localization method based on HOG features and FAST algorithms | |
| Chen et al. | Head-shoulder detection using joint HOG features for people counting and video surveillance in library | |
| Hadi et al. | Fusion of thermal and depth images for occlusion handling for human detection from mobile robot | |
| Bertozzi et al. | Multi stereo-based pedestrian detection by daylight and far-infrared cameras | |
| Shimizu et al. | Development of a person-searching algorithm using an omnidirectional camera and LiDAR for the Tsukuba challenge |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C17 | Cessation of patent right | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20130424 Termination date: 20140129 |