WO2020176045A1

WO2020176045A1 - Interactive artificial intelligence controlled education system.

Info

Publication number: WO2020176045A1
Application number: PCT/TR2019/000059
Authority: WO
Inventors: Ulaş ÇELENK
Original assignee: Celenk Ulas
Priority date: 2019-02-27
Filing date: 2019-09-16
Publication date: 2020-09-03
Also published as: EP3867894A1; TR201902975A2

Abstract

An interactive artificial intelligence controlled education system, characterized in that it consists of a cartoon character (1), a skeleton (2), a model (3), a child (4), an artificial intelligence (5), a scenario (6), a speech detection (7), a movement detection (8) and a device (9) such as computer, tablet, telephone, etc. providing education fitted with microphone, camera and loudspeaker allowing acting of cartoon characters according to scenarios provided to them depending on detection of speeches and movements of children with artificial intelligence as a support for children with autism, attention deficit, hyperactivity disorder, development retardation or psychological and neurological problems requiring special support for normal development.

Description

INTERACTIVE ARTIFICIAL INTELLIGENCE CONTROLLED EDUCATION

SYSTEM

TECHNICAL FIELD

This invention relates to an interactive artificial intelligence controlled education system based on acting of cartoon characters according to scenarios provided to them depending on detection of speeches and movements of children with artificial intelligence as a support for children with autism, attention deficit, hyperactivity disorder, development retardation or psychological and neurological problems requiring special support for normal development.

STATE OF ART

In our day, children with autism, attention deficit, hyperactivity disorder, development retardation or psychological and neurological problems requiring special support for normal development are educated individually or as a part of a group by instructors in private educational institutions. There are applications detecting their attention towards an instructor or educational material by following eye movements of these children by means of an artificial intelligence. However, domestic education cannot be realized besides educational institution or an instructor.

OBJECT OF THE INVENTION

The invention relates to cartoon characters liked by children with attention deficit or concentration problem. In cases of inadequate education or in order to enhance the quality of education, this invention was devised to provide the children with the related education under control of an interactive artificial intelligence with cartoon characters liked by children.

Children are highly interested in liked cartoon characters. They rapidly internalize movements and actions performed by these cartoon characters. The children with no concentration for present instructors may concentrate on the related cartoon characters. Artificial intelligence measuring the attention of children by means of their eye movements can merely produce indications exhibiting the efficiency of the education. They do not provide an education for the children. In the related study, it was aimed to provide the children with the education directly by an artificial intelligence, or a combination of instructor with an artificial intelligence.

Autism is a communication problem.

Children generally maintain their physical development by the age 2.

After the age of 2, the children realize nice advancements for their mental development by following and observing their environments. They observe their parents and environment, and educate themselves. Children with autism have rare communication with the external world. These children are isolated from the external world, seem to live within a bell jar (mental isolation from the external world), and do not observe their surroundings. A normal child would reach a level by the age of 3. However, children with autism cannot reach this level as they cannot educate themselves.

3 years of age is an age in which autism problem is manifested clearly. An education initiated at the age of 3 is a nonnal treatment procedure. An education initiated before the age of 3 is an early treatment procedure, and is highly successful. An education initiated after the age of

4 is a delayed education, and is pretty challenging. Unfortunately, there are children rejected due to the inability to do anything in case of delayed education.

This is a tough situation for families. The child has the physical strength of 5 years old, but is mentally far behind.

They cannot express themselves due to the inability to talk, and are profoundly aggressive. As they have physical strength, their care is challenging for families thereof.

Mothers attempt to tolerate as much as they can. However, they sometimes have to give up. The children are sent to nursing homes. They are detached from their families.

There are not various treatments in nursing homes as well. Children are given sedatives, and they spend their lives in a semi-asleep situation. This situation is different in case of a normal or an early treatment. Instructors initially attempt to break the bell jar around the child. They gather around the child, and play games such as clapping, and sing. Child is initially not interested. However, instructors continue with patience.

Afterwards, the child claps or reacts. Instructors become pleased, and cheerful. They show the children their happiness. The child likes it, and starts to observe his/her environment even if just a bit. For experiencing the same cheer again. This is a crack on the bell jar, and is valuable. With patience, instructors attempt to break up the bell jar and enable the child to communicate with the external world. A year passes by, and the child is no longer isolated from the external world, and is capable of communication. However, despite physically being 4 years old, he/she is mentally 3 years old.

The education thereafter is for catching up peers of the child. There are provided a number of educations relating to the concepts of inside, outside, long, short and the like.

In the children with autism, communication with the external world is challenging particularly for children with delayed education. They generally live within a bell jar. Nevertheless, there are some moments in which there is no bell jar, and they communicate with the external world. Children concentrate while watching liked cartoons. They have open perceptions during that time. They smile as cartoon characters smile.

They feel sad as cartoon characters cry. There is no bell jar.

In our invention, we have provided an interactive artificial intelligence to provide children having a communication difficulty with the related education by means of cartoon characters.

We have enabled that cartoon characters may perceive the movements and speeches of the children, and thusly provide the related education in accordance with the reactions of the children by abiding by the scenarios provided.

These educations may be used for advanced and beginner level autism. Sphere of influence of the education is not only limited to autism. It may be easily used for children with development retardation, attention deficit, hyperactivity and similar yet different problems. Moreover, it may also be used for educations of healthy children with no problem with ease as well.

DESCRIPTION OF THE DRAWINGS

Fig. 1 Various cartoon characters consisting of models placed onto a skeleton.

Fig. 2 Visual of face and eye detection of a child with OpenCv.

Fig. 3 Visual of the detection of finger and hand movement and movements such as clapping of a child with OpenCv.

Fig. 4 Scenario flow diagram in which a cartoon character waits for watching by a child while singing, and then attendance of the child with his/her movements.

Fig. 5 Scenario flow diagram in which a child is expected to follow when a cartoon character speaks or moves, and in case of not following the child is stimulated by calling for his/her name.

Fig. 6 Scenario flow diagram in which a child is expected to look and finger-point an object named by a cartoon character.

Fig. 7 Scenario flow diagram in which a cartoon character tells a story, and a child is expected to answer questions such who, what, where, how, why and what happened relating to the story.

Fig. 8 Scenario flow diagram in which a cartoon character asks questions relating to self-care, social skills and various skills, and a child is expected to answer the same.

Fig. 9 Scenario flow diagram in which a child is expected to recognize concepts, and to answer questions asked by a cartoon character relating to the larger one or shorter one with respect to two glass shown.

Fig. 10 Scenario flow diagram relating to action recognition skill.

Fig. 11 Scenario flow diagram relating to action recognition and imitation skill.

Fig. 12 Scenario flow diagram in which a child is expected to react to absurdities.

Fig. 13 Scenario flow diagram in which a cartoon character says hallo or claps and expects correspondence from a child by saying hallo back and clapping back.

DESCRIPTION OF REFERENCES:

1 - Cartoon character

2 - Skeleton 3 - Model

4 - Child

5 - Artificial intelligence

6 - Scenario

7 - Speech detection

8 - Movement detection

9 - Device such as computer, tablet, telephone, etc. providing education fitted with microphone, camera and loudspeaker.

DESCRIPTION OF THE INVENTION

The components such as movements of the cartoon character (1), environment of the cartoon and background are controlled with the systems of the invention.

These cartoon characters (1) consists of 2 basic components: the skeleton (2) and the model (3). The skeleton (2) consists of joints just like a human skeleton, and animations of the cartoon characters (1) are enabled with movements of these joints. There are dozens of joint points in the skeleton (2) of this cartoon character (1). Knees and feet are some of these joints. For example, when it is desired for the cartoon character (1) to wave, joint points in the hand region are moved rightwards and leftwards, thereby performing the related movement.

2^nd component of the cartoon character (1) is the model (3). The model (3) is like a clothed cartoon character (1) placed onto the skeleton (2), and comprises body and clothing components of the skeleton (2). The model (3) technically consists of polygons. Thanks to this polygon structure, a 3 -dimensional visuality skill is enabled.

The cartoon characters (1) (such as Pepe, Spiderman) are each a model (3), and placed onto the skeleton (2) (Fig. 1). Movements are designed on the skeleton (2). The model (3) on the skeleton (2) moves upon the movement of the skeleton (2). Different models (3) can be placed onto a skeleton (2). For example, if a waving skeleton (2) movement is designed and a Pepe model is placed thereon, Pepe waves; and if a Spiderman model is placed thereon, Spiderman waves. The cartoon characters (1) in the application is in an interactive communication with the child (4). The cartoon characters determine their movements in accordance with speeches and movements of the child. The speeches and movements of the child (4) are detected with the artificial intelligence (5). The voice processing, voice recognition and similar libraries of the system and the cloud and similar utilities providing this service are used for the speech detection (7). The related voice processing, voice recognition libraries and the cloud and similar utilities providing this service interact with the microphone of the device (9) such as computer, tablet, telephone, etc. providing communication fitted with microphone, camera and loudspeaker executing the program, and interpret the incoming voices. The words desired to be caught are entered into the voice processing, voice recognition libraries and the cloud and similar utilities as a data.

There is a code piece listening to the words such as "Hallo", "How Are You?" and "Let's Play".

The program executes an operation depending on the word upon articulation of any of the related words.

The application detects the movements of the children along with the speech. The image processing, image recognition and similar libraries of the system and the cloud and similar utilities providing this service are used for the movement detection (8) (Fig. 2). The system comprises a number of libraries based on image processing. Face and eye detection may be performed with the image processing, image recognition and similar libraries and the cloud and similar utilities providing this service, thusly it is detected that the child is in front of the application and in communication with the same.

Like face and eye detection, the detection of objects such as hand and finger as well as detection of movements (8) such as bouncing and clapping may be realized with the image processing, image recognition and similar libraries and the cloud and similar utilities providing this service (Fig. 3).

The application is based on acting of the cartoon characters (1) according to scenarios (6) provided to them depending on the speech detection (7) and the movement detection (8) of children (4) with the artificial intelligence (5). Scenario (6) flow diagram in which the cartoon character (1) says hallo or claps, and expects correspondence from the child (4) by saying hallo back and clapping back (Fig. 13).

1. The application says hallo to the child (4).

2. The application expect the child (4) to say hallo back.

3. It says "Let's play clapping" to the child (4).

4. The cartoon character (1) in the application claps.

5. It says "Now you clap" to the child (4).

6. The child (4) is expected to clap.

7. If the child (4) does not clap, it persists by telling the child to clap.

8. If the child (4) claps, the scenario (6) is terminated by saying "Yes, he/she has clapped".

Education of children with autism (4) may be divided into various levels from the beginner level to the treatment completion level (Each level comprises harder practices and exercises depending on the basis of a previous level).

- Communicating

In this level, it is aimed that the child (4) can communicate and fulfill basic commands. Looking of the child (4) upon calling for his/her name is one of the first operations (Fig. 5). The child (4) is given simple commands such as "Come here", and the child (4) is expected to fulfill these commands. It is expected to sing songs, play games and participation of the child (4) into these games (Fig. 4). Games such as quoits, finger plays and games such as Ring a ring of roses are played; and the child (4) is expected to initially watch these games and then participate in them. In this level, the aim to provide the child (4) with the ability to communicate with the external world. It is highly important to make eye contact with the child (4) while talking. It is attempted to make eye contact while playing. The game is paused in case the child (4) loses eye contact, and is resumed after enabling the eye contact again. It is aimed for the child to speak simple words such as mom, dad and food.

- Noticing distant objects It is aimed to increase communication of the child (4) with the external world by enabling his/her attention to distant objects. When a distant object (1-2 meters) is wanted from the child (4), the child (4) is expected to bring or show the same (Fig. 6). The child (4) is expected to follow games such as quoits played farther. In this level, it is expected that the game can be continued for 3-4 rounds, and also alternating games (as one instructor and then one child (4)) are played as well. The child is expected to imitate other players (Fig. 11). In this level, the child (4) is expected to eat on his/her own, unclothe without help, wash his/her hands and face with help, brush his/her teeth with help, meet toilet requirements with help, be able to walk on a flat line, be able to bounce on a single foot, and hit a ball onto the ground and hold it back. The child can discharge liquids from a vessel into another vessel. The child can form various shapes with a soft material.

- Making choices.

The child (4) is expected to distinguish and choose the desired one among a number of cards or objects. Objects in the house or around are recognized. Concepts are learned such as big, small, and long and short (Fig. 9). The concept of contrast is learned. Colors are learned. Classifications are learned such as "Which one is a food?" or "Which one is a toy?". The child is expected to answer questions such as "Is it eatable?" or "Is it sweet?". Distinguishing is learned with questions such as "Which ones are eatable among the pictures on the table?" The child is expected to count from 1 to 10, and be able to choose pictures with the same or different components. The child completes a 4-piece puzzle. Fie/she recognizes body parts such as nose and mouth. He/she recognizes emotions such as sadness and happiness. He/she answers questions such as "who, what, when" relating to a story. He/she uses subjects and verbs during speech. He/she uses pronouns during speech. He/she knows the concept of plurality. He/she answers questions asked relating to his/her daily routine. He/she can explain 2 successive events in the order of occurrence. He/she distinguishes front and rear portions of clothes. He/she collects his/her belongings. He/she participates in group plays. He/she gets in line, and waits for his/her turn. He/she shares his/her belongings. He/she asks for permission from elderlies.

- Forming a sentence The child is expected to answer questions relating to the functions of objects. The children (4) are expected to explain a scenario with sentences on showing pictured cards ranked in line with the scenario (6). The child answers questions such as "why, how, who"; he/she can fulfill 3- or 4-consecutive directives (go to the closet, get the pencil, leave it onto the table, etc.). He/she answers questions relating to short and simple stories, speaks with minimal voice mistakes, talks about his/her experiences. He/she knows materials of some objects. He/she distinguishes objects with different textures. He/she forms a link between objects and numbers. He/she matches objects groups with the same number. He/she forms a whole by combining two halves. He/she fulfills directives relating to a location in a room. He/she explains events in the order of occurrence. He/she answers questions relating to an object, a person or objects seen. He/she forms a story from the pictures shown and answers questions comprising cause and effect relation. He/she knows the differences (high, deep, high-pitched) in voices.

- Answering the question of "What do you want?"

The child (4) is expected to express his/her wishes. He/she shows their emotions. He/she understands emotions of other people. He/she abides by the rules of ruled games. He/she explains rules of a game to another person. He/she directs games and his/her friends. He/she fulfills an undertaken responsibility. He/she adopts a target, and directs his/her behaviors accordingly. He/she animates a certain event or situation. He/she uses unique ways to express himself/herself. He/she stands up for his/her rights. He/she explains his/her daily experiences. He/she uses a verb complying with a subject in a sentence. He/she asks for meanings of new and unknown words. He/she knowns synonyms and antonyms of some words. He/she answers questions such as "when, why, how, etc." (Fig. 7) (Fig. 8). He makes simple jokes. He/she chooses clothes appropriate for weather conditions. He/she uses conjunctions such as because, after, etc.

- Presenting an opinion

The children (4) are expected to use their reasoning skills (Fig. 10). They are expected to share their opinions about stories. The children (4) notice contrasts and absurdities embedded into the stories (like going swimming in winter) (Fig. 12). They answer questions such as "What do you see, what do you hear, or what is here?" Questions such as "How do you think it will happen? are answered, and commented. Details not provided in the stories are predicted with reasoning (it is cold if he is overdressed). They are expected to go by circumstances (such as asking for an umbrella if it is rainy).

Claims

1. An interactive artificial intelligence controlled education system, characterized in that it consists of a cartoon character (1), a skeleton (2), a model (3), a child (4), an artificial intelligence (5), a scenario (6), a speech detection (7), a movement detection (8) and a device (9) such as computer, tablet, telephone, etc. providing education fitted with microphone, camera and loudspeaker allowing acting of cartoon characters according to scenarios provided to them depending on detection of speeches and movement of children with artificial intelligence as a support for children with autism, attention deficit, hyperactivity disorder, development retardation or psychological and neurological problems requiring special support for normal development.

2. The interactive artificial intelligence controlled education system according to claim 1 , characterized in that the education of children with autism (4) has various scenarios (6) such as communicating, noticing distant objects, making choices, forming sentences, answering a question of "What do you want?" and presenting an opinion from a beginner level to an education completion level, each level comprising harder practices and exercises depending on the basis of a previous level.

3. The interactive artificial intelligence controlled education system, characterized in that it comprises the steps of:

- controlling of components such as movements of the cartoon character (1), environment of the cartoon and background with the systems of the invention;

- designing of the movements on the skeleton (2) by placing the cartoon characters (1) onto the skeleton (2);

- enabling the movement of the model (3) on the skeleton (2) upon the movement of the skeleton (2);

- enabling the interactive communication of the child (1) with the cartoon characters

(1);

- determining the movements of the cartoon character (1) by himself/herself in accordance with the speeches and movements of the child (4);

- detecting speeches and movements of the child (4) with the artificial intelligence (5); - using voice processing, voice recognition and similar libraries of the system and the cloud and similar utilities providing this service for the speech detection (7);

- enabling entry, as a data, of words desired to be caught into the voice processing, voice recognition and similar libraries and the cloud and similar utilities providing this service;

- enabling comparison of the voices being obtained with the interaction of the related voice processing, voice recognition and similar libraries and the cloud and similar utilities providing this service with the device (9) such as computer, tablet, telephone, etc. providing communication fitted with microphone, camera and loudspeaker executing the program and with the interpretation of the incoming voices, with the words desired to be caught;

- executing of an operation depending on the word upon articulation of the related words,

- enabling face and eye detection with the image processing, image recognition and similar libraries of the system comprising a number of libraries based on image processing for movement detection (8) of the child (4), and the cloud and similar utilities providing this service;

- detecting the presence of the child (4) in front of the application, and in communication with the application;

- enabling the movement detection (8) such as bouncing and clapping along with face and eye detection of the child with the image processing, image recognition and similar libraries of the system and the cloud and similar utilities providing this service;

- acting of the cartoon characters (1) according to scenarios (6) provided to them depending on the speech detection (7) and the movement detection (8) of children (4) with the artificial intelligence (5).