AU2021103468A4 - Smart Bus with AI Based Face Mask Detection System in Pandemic Situations Using Raspberry PI - Google Patents
Smart Bus with AI Based Face Mask Detection System in Pandemic Situations Using Raspberry PI Download PDFInfo
- Publication number
- AU2021103468A4 AU2021103468A4 AU2021103468A AU2021103468A AU2021103468A4 AU 2021103468 A4 AU2021103468 A4 AU 2021103468A4 AU 2021103468 A AU2021103468 A AU 2021103468A AU 2021103468 A AU2021103468 A AU 2021103468A AU 2021103468 A4 AU2021103468 A4 AU 2021103468A4
- Authority
- AU
- Australia
- Prior art keywords
- wearing
- mask
- face
- face mask
- person
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 24
- 241000711573 Coronaviridae Species 0.000 claims abstract description 8
- 230000000903 blocking effect Effects 0.000 claims description 2
- 238000013461 design Methods 0.000 claims description 2
- 238000013135 deep learning Methods 0.000 abstract description 15
- 238000010801 machine learning Methods 0.000 abstract description 9
- 241000700605 Viruses Species 0.000 abstract description 4
- 238000013473 artificial intelligence Methods 0.000 abstract description 4
- 230000005540 biological transmission Effects 0.000 abstract description 4
- 201000010099 disease Diseases 0.000 abstract description 2
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 abstract description 2
- 230000008520 organization Effects 0.000 abstract description 2
- 230000009466 transformation Effects 0.000 abstract 1
- 238000013527 convolutional neural network Methods 0.000 description 11
- 238000000034 method Methods 0.000 description 10
- 238000013528 artificial neural network Methods 0.000 description 6
- 230000001815 facial effect Effects 0.000 description 6
- 239000004606 Fillers/Extenders Substances 0.000 description 3
- 238000003709 image segmentation Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000006399 behavior Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 210000004556 brain Anatomy 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000011897 real-time detection Methods 0.000 description 2
- 238000005406 washing Methods 0.000 description 2
- 208000025721 COVID-19 Diseases 0.000 description 1
- 241000206607 Porphyra umbilicalis Species 0.000 description 1
- 235000011034 Rubus glaucus Nutrition 0.000 description 1
- 244000235659 Rubus idaeus Species 0.000 description 1
- 235000009122 Rubus idaeus Nutrition 0.000 description 1
- 230000000845 anti-microbial effect Effects 0.000 description 1
- 239000004599 antimicrobial Substances 0.000 description 1
- 230000004888 barrier function Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000009189 diving Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000010859 live-cell imaging Methods 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 230000035755 proliferation Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 229960005486 vaccine Drugs 0.000 description 1
- 210000000216 zygoma Anatomy 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
- G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/245—Classification techniques relating to the decision surface
- G06F18/2453—Classification techniques relating to the decision surface non-linear, e.g. polynomial classifier
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Oral & Maxillofacial Surgery (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Human Computer Interaction (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Nonlinear Science (AREA)
- Multimedia (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
Smart Bus with AI Based Face Mask Detection System in Pandemic Situations Using
Raspberry PI
The global pandemic of the corona virus is precluded by the recommendations of the
World Health Organization (WHO), so wearing a face mask in the workplace has been declared to
be the only effective way to avoid getting infected. The pandemic made governments across the
world to stay under Lock downs to prevent from virus transmissions. Reports show that wearing
facemasks would clearly reduce the risk of transmission. With the rise in population in cities,
there is a greater need for efficient city management in today's world for reducing the impact of
Corona disease. For Smart Cities to prosper, major improvements to occur in public
transportation, roads, businesses, houses, and the city streets, as well as other facets of city life
will have to be developed. The current public bus transportation system, such as it is, should be
expanded with Artificial Intelligence. The autonomous mask detection and alert system is needed
to find whether the person is wearing face mask or not.
This system has almost complete face-identification capabilities with respect to people's
presence in the case where they are wearing masks, with an error rate of only 1.1 %. A
transformation of CNN's classifiers has better efficiency over the DNN's classifier algorithm. We
are also added the face-recognition security system as well, which would allow the person to
enter the building only if they were wearing a face mask Deep learning and modern machine
learning concepts workhorse concepts enable the artificial intelligence to achieve the greatest
accuracy possible.
Description
Smart Bus with Al Based Face Mask Detection System in Pandemic Situations Using Raspberry PI
[001]. The present disclosure is generally related to a Smart bus with Al based Face Mask detection system in Pandemic situations using Raspberry PI
[002]. The global pandemic of the corona virus is precluded by the recommendations of the World Health Organization (WHO), so wearing a face mask in the workplace has been declared to be the only effective way to avoid getting infected. The pandemic made governments across the world to stay under Lock downs to prevent from virus transmissions. Reports show that wearing facemasks would clearly reduce the risk of transmission. With the rise in population in cities, there is a greater need for efficient city management in today's world for reducing the impact of Corona disease. For Smart Cities to prosper, major improvements to occur in public transportation, roads, businesses, houses, and the city streets, as well as other facets of city life will have to be developed. The current public bus transportation system, such as it is, should be expanded with Artificial Intelligence. The autonomous mask detection and alert system is needed to find whether the person is wearing face mask or not.
[003]. The objective of this invention is to design an Al based low-cost device for Smart Bus aimed at reducing the impact of corona virus by detecting and alerting whether an individual wear a mask or not. It is a real-time system to detect face-masks in video streaming using camera. Also it is designed to be a low-cost system to sense whether an individual wear a mask or not. This will act as an autonomous system designed for automatic gateway for blocking the person, if he/she doesn't wear a mask in the Smart Bus.
[004]. The proposed Face Mask detection system recognizes whether or not a person wears a mask. People without masks can be detected and alerted using the app with any existing or IP cameras that have been connected to the system through network extenders. In addition, users can also have faces and phone numbers added to their expandable to send them an alert when they are missing a mask. If a user is not adequately identified in the camera, a notification can be sent to the administrator.
[005]. The deep learning technique and convolution algorithm is combined to implement a real-time face Mask detection system with an alert system. The Deep Learning and machine learning algorithms to help create a CNN system (Convolution Neural Networks). The image segmentation method produces efficient and accurate results for face detection. Face-reading showed that over half of the participants could accurately determine if the individual was wearing a mask or not.
[0061. The trained model with the implementation of tenser-flow and the VGG16 Convolution system could display accurate and efficient behavior. By analyzing who regularly whether or doesn't wear face masks to screen the Corona virus, we can provide an effective way to stop the spread of the virus.
[007]. The system supports real-time processing of the inputs and makes sure to force the people to wear the face mask as per the guidelines. The proposed CNN model, which is extracted from RNN model helped us to achieve the highest possible accuracy, throughput, etc. The dataset " Without and with face mask " is created using images of person not wearing face mask. This dataset is also trained using thousands of images. The attributes that can be used to define the images of person not wearing the face mask are visibility of nose, mouth, etc. The dataset also had an accuracy of 97%.
[0081. Figure 2 shows the model of Proposed device for monitoring the people in the Smart Bus. The prototype model has the following kits: The model starts to execute when connected to the AC supply and it is developed considering its cost efficiency, size and durability. The gate-way opens or closes depending on the person crossing it, with or without mask along with an alarm sound. The proposing model is constructed considering the environment and the place it's going to used. It is developed with cheaper materials to make it accessible to everyone.
[009]. The proposed system produces the output as seen in Figure 3, which can look into a person's appearance whether he/she wore a mask on the bus. If the system detects that someone has not wore a mask on the entrance of the Smart Bus, the output is in figure 4 will be of them.
[0010]. Using deep neural networks and machine learning techniques, the scientists concluded that the deep learning facemask and model were particularly beneficial for real-time detection (Convolutional Neural Networks). This model serves our needs when facemasks are found since it is an effective and precise representation of facial detection in all locations. For further research, a separate population was created to identify persons who are concealing facemasks. The trained model would give satisfactory results with the precision at 97% using the VGG-16 CNN model. Also, If the person is detected without mask, the door will not open for entering inside the bus.
[0011]. COVID-19(CORONA Virus) has currently circulated the world without an effective vaccine. However, wearing a face mask is an effective method of anti-microbial barrier protection along with: as well as numerous others, including washing the hands, practicing good hygiene, and frequent hand washing (WHO). The idea of wearing a face mask in public places has now entered the realm of the general public consciousness. This raises the additional concerns of Al (artificial intelligence) and deep learning to propose for face detection that automatically relies on principles to be automated. A face matching model using a deep-learning/classical machine learning model and traditional machine learning will be proposed. The main challenge is to create the dataset composed of face masks and non- and from open-face images in real-time, and we will use OpenCV to do real-time face detection.
[0012]. We have to build a computer vision-based face detector using our dataset before diving into OpenCV and Python with Tensor Flow, with our custom machine learning framework, using the pre-defined algorithm. We will be using computer vision and deep learning concepts to identify whether the person is using a mask in the face or not. This is sure to expedite the proliferation of computer vision in currently nascent areas such as digital signage, autonomous driving, video recognition, customer service, language translation, and mobile apps. The main element of deep learning is DNNs is the whole brain emulation, which offers object recognition, segmentation, and even is applicable in the brain structure image classification.
[0013]. Generally used in tasks related to computer vision, they are also seen as an effective tool in increasing the resolution of a classifier by increasing the computational length in complexity. After applying CNN and the advanced feature extraction and classification methods, the models can identify and classify facial images with minimal features and store fine details. A deep learning classifier is used to collect photos of a person wearing a face mask, but not from a database, and then uses features such as people, expressions, content, and spatial information to distinguish between images of facemasks and photos of people.
[0014]. While unprocessed data sets have shown to be very effective for finding features in artificial neural networks, researchers still have not proven their utility in the clinical setting. The primary purpose of the Raspberry Pi circuit board is to carry out these critical tasks, such as the CPU, the GPU, input, and output. Raspberry Pi board features GPIO pins essential to using hardware programming to enable the Raspberry for control of electronic circuits and data processing devices on input and output. When it comes to the Raspberry Pi, having both an HDMI cable and a keyboard, mouse, and a monitor that matches the specification, make sure to include a power adapter, too. It is possible to install and run the Raspberry Pi under the Raspbian OS. It is shipped with a pre programmed version of Python.
[0015]. Thus, it is simple to identify a person at the bus's entrance in an image/video stream, whether or not they are wearing a face mask, using computer vision and deep learning concepts. If a person wearing a mask enters the area, an automatic gate will open; if the person does not wear a mask, the gate will remain closed. The block diagram of the proposed work is as shown in figure 1.
[0016]. Face Mask Detection Model: It is first necessary to collect suitable examples of faces to feed the model for the face classifier so that it may be able to determine if the individual in question is wearing a mask. Once the classifier has been trained, face detection is needed to check for possible face-covering before proceeding with the classifying the individual, since the SSDMNV2 evaluates whether or not the individual has a mask. The purpose of this research is to improve the discrimination capability of masks without wasting significant computational resources. The DNN module from OpenCV used the 'Single Shot MultiBox' (SSD) object detection framework with ResNet-10 as its base. A frame works extend the features of the Raspberry Pi, such as live imaging, to occur in real time. This trained model's classifier expands on a trained model and independent network models to enable it to distinguish whether a person is wearing a mask or not
[0017]. A number of single-use, fixed image datasets are available for face detection only almost all of the datasets can be considered fake in the absence of real world information and the majority of the ones suffer from inclusion of incorrect information, either as well as inclusion of noise. This meant some effort was required in order to identify the best possible dataset for the SSDMNV2 model. Kaggle and Witkowski's Medical Mask dataset were utilized to expand the model's training set. And, in addition, data gathering was performed using the masked dataset, which involved blind application.
[0018]. The Kaggle dataset contains a large number of individuals with faces blurred out to protect their privacy, as well as relevant XML files that describe their anonymity protection devices. It holds a total of 678 photographs in this dataset. PySearch for the expansion of the Pylmage dataset in the Natural masking settings, and return it as 'Prajary B' The dataset consists of 1,376 photos which are grouped into two groups: those that have masks (as well as ones that do not) (686 images). Prajna Bhand created a dataset by utilizing standard photographs of facial points to artificial points of additional to them.
[0019]. The appearance of facial features can be defined by facial landmarks such as the eyes, brows, nose, mouth, and cheekbones This dataset was constructed by merging two layers of images-one of the original, one on top of the other. The development team, however, rejected the reused images and did not employ them for the purpose of producing artificial life. To an extent not previously considered, non-face measurements were included in the model, which increased the chances of a significant distortion. The usage of images from outside of data sets of different origin carries the risk of inaccuracy.
[0020]. Proposed Model: The proposed Face Mask detection system recognizes whether or not a person wears a mask. People without masks can be detected and alerted using the app with any existing or IP cameras that have been connected to the system through network extenders. In addition, users can also have faces and phone numbers added to their expandable to send them an alert when they are missing a mask. If a user is not adequately identified in the camera, a notification can be sent to the administrator. The deep learning technique and convolution algorithm is combined to implement a real time face Mask detection system with an alert system. The Deep Learning and machine learning algorithms to help create a CNN system (Convolution Neural Networks). The image segmentation method produces efficient and accurate results for face detection.
[0021]. The proposed Face Mask detection system recognizes whether or not a person wears a mask. People without masks can be detected and alerted using the app with any existing or IP cameras that have been connected to the system through network extenders. In addition, users can also have faces and phone numbers added to their expandable to send them an alert when they are missing a mask. If a user is not adequately identified in the camera, a notification can be sent to the administrator.
[0022]. The deep learning technique and convolution algorithm is combined to implement a real-time face Mask detection system with an alert system. The Deep Learning and machine learning algorithms to help create a CNN system (Convolution Neural Networks). The image segmentation method produces efficient and accurate results for face detection. Face-reading showed that over half of the participants could accurately determine if the individual was wearing a mask or not.
[0023]. Algorithml:FaceMaskDetection Algorithm
Input: Dataset includingfaces with and without masks
Output: Categorized image depicting the presence offacemask
For each image in the dataset do
Visualize the image in two categories and label them
Convert the RGB image to Gray-scale image Fh sizethegray-scaleimageintol00x1005 Normalizetheimageandconvertitinto4dimensionalarray end for building the CNN model do Add a Convolution layer of 200filters Add the second Convolution laver of100filters Insert a Flatten layer to the network classifier Add a Dense layer of 64 neurons Add thefinalDense layer with 2 outputsfor2 categories end
[0024]. The trained model with the implementation of tenser-flow and the VGG16 Convolution system could display accurate and efficient behavior. By analyzing who regularly whether or doesn't wear face masks to screen the Coronavirus, we can provide an effective way to stop the spread of the virus. The system supports real-time processing of the inputs and makes sure to force the people to wear the face mask as per the guidelines. The proposed CNN model, which is extracted from RNN model helped us to achieve the highest possible accuracy, throughput, etc.The dataset " Without and with face mask "is created using images of person not wearing face mask. This dataset is also trained using thousands of images. The attributes that can be used to define the images of person not wearing the face mask are visibility of nose, mouth, etc. The dataset also had an accuracy of 97%.
[0025]. Figure 2 shows the model of Proposed device for monitoring the people in the Smart Bus. The prototype model has the following kits: The model starts to execute when connected to the AC supply and it is developed considering its cost efficiency, size and durability. The gate-way opens or closes depending on the person crossing it, with or without mask along with an alarm sound. The proposing model is constructed considering the environment and the place it's going to used. It is developed with cheaper materials to make it accessible to everyone.
[0026]. The proposed system produces the output as seen in Figure 3, which can look into a person's appearance whether he/she wore a mask on the bus. If the system detects that someone has not wore a mask on the entrance of the Smart Bus, the output is in figure 4 will be of them.
[0027]. Using deep neural networks and machine learning techniques, the scientists concluded that the deep learning facemask and model were particularly beneficial for real-time detection (Convolutional Neural Networks). This model serves our needs when facemasks are found since it is an effective and precise representation of facial detection in all locations. For further research, a separate population was created to identify persons who are concealing facemasks. The trained model would give satisfactory results with the precision at 97% using the VGG-16 CNN model. Also, If the person is detected without mask, the door will not open for entering inside the bus.
Claims (6)
1. We claim that this invention is generally related to a Smart bus with Al based Face Mask detection system in Pandemic situations using Raspberry PI.
2. As claimed in 1, the proposed invention offers the design of Al based low-cost device for Smart Bus aimed to reduce the spreading of corona virus.
3. The proposed invention will detect and alert if the any one of the individual is not wearing the mask.
4. The proposed invention is a real-time system to detect the face-masks in video streaming using camera.
5. The proposed invention will act as an autonomous system designed for automatic gateway for blocking the person, if he/she doesn't wear a mask in the Smart Bus.
6. As we claimed in 1, 2, 3, 4 and 5, the proposed invention will help the people in preventing the spread of corona virus.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| AU2021103468A AU2021103468A4 (en) | 2021-06-18 | 2021-06-18 | Smart Bus with AI Based Face Mask Detection System in Pandemic Situations Using Raspberry PI |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| AU2021103468A AU2021103468A4 (en) | 2021-06-18 | 2021-06-18 | Smart Bus with AI Based Face Mask Detection System in Pandemic Situations Using Raspberry PI |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| AU2021103468A4 true AU2021103468A4 (en) | 2021-08-12 |
Family
ID=77195564
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| AU2021103468A Ceased AU2021103468A4 (en) | 2021-06-18 | 2021-06-18 | Smart Bus with AI Based Face Mask Detection System in Pandemic Situations Using Raspberry PI |
Country Status (1)
| Country | Link |
|---|---|
| AU (1) | AU2021103468A4 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113723308A (en) * | 2021-08-31 | 2021-11-30 | 上海西井信息科技有限公司 | Detection method, system, equipment and storage medium of epidemic prevention suite based on image |
-
2021
- 2021-06-18 AU AU2021103468A patent/AU2021103468A4/en not_active Ceased
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113723308A (en) * | 2021-08-31 | 2021-11-30 | 上海西井信息科技有限公司 | Detection method, system, equipment and storage medium of epidemic prevention suite based on image |
| CN113723308B (en) * | 2021-08-31 | 2023-08-22 | 上海西井科技股份有限公司 | Image-based epidemic prevention kit detection method, system, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Suresh et al. | Face mask detection by using optimistic convolutional neural network | |
| Negi et al. | Face mask detection classifier and model pruning with keras-surgeon | |
| WO2021063056A1 (en) | Facial attribute recognition method and apparatus, and electronic device and storage medium | |
| Mohammed Ali et al. | Face mask detection methods and techniques: A review | |
| Shukla et al. | Masked face recognition using mobilenet v2 with transfer learning. | |
| Amin et al. | Deep learning based face mask detection and crowd counting | |
| Prasad et al. | COVID vision: An integrated face mask detector and social distancing tracker | |
| AU2021103468A4 (en) | Smart Bus with AI Based Face Mask Detection System in Pandemic Situations Using Raspberry PI | |
| Kumar et al. | Deep learning model for face mask based attendance system in the era of the COVID-19 pandemic | |
| Kaliappan et al. | Real-time face mask position recognition system using YOLO models for preventing COVID-19 disease spread in public places | |
| Sadanand et al. | Social distance and face mask detector system exploiting transfer learning. | |
| Maurya et al. | COVID-19 face mask detection | |
| Banik et al. | Automatic approach for mask detection: effective for COVID-19 | |
| Abdurrasyid et al. | Face detection and global positioning system on a walking aid for blind people | |
| Nahar et al. | Twins and Similar Faces Recognition Using Geometric and Photometric Features with Transfer Learning | |
| Machiraju et al. | Face mask detection using keras, opencv and tensorflow by implementing mobilenetv2 | |
| Kale et al. | Real Time Face Mask Detection-A Survey | |
| Sharma et al. | Face mask detection analysis for covid19 using cnn and deep learning | |
| Jeny et al. | Deep Learning Framework for Face Mask Detection | |
| Saminathan et al. | Face Mask Detection Using Raspberry Pi | |
| Nainan et al. | Real Time Face Mask Detection Using MobileNetV2 and InceptionV3 Models | |
| Moyo et al. | COVID-19 Face Mask Detection Alert System | |
| Abiodun et al. | An Access Control System for Covid19 Using Computer Vision and Deep Learning Techniques: A Aystematic Review | |
| Alani et al. | Convolutional neural network-based Face Mask Detection | |
| Bhattacharjee et al. | Facial Landmark and Mask Detection System |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| FGI | Letters patent sealed or granted (innovation patent) | ||
| MK22 | Patent ceased section 143a(d), or expired - non payment of renewal fee or expiry |