WO2013103947A2 - Student observation platform - Google Patents

Abstract

A system for recording and presenting a playback of a student's scratch work comprises a question, an answer area and a scratch area that may be presented on a digital tablet computer. A stylus is provided and the motions of the stylus are recorded while a student answers the questions. Both writing and erasing are recorded. The recorded motions can be played back by a teacher or parent to determine not only if a question was answered correctly based on the writing in the answer area, but how a question was answered based on writing in the scratch area. Intermediate answers can be seen since all of the writing and erasing is played back. Erasing can be presented as a light color so that the existence completely erased intermediate work can be identified from a final image of the student's answers.

Description

Title

Student Observation Platform Cross-Reference to Related Application

This application claims priority to US provisional patent application "Student Observation Platform", serial number 61583650, filed 6 January 2012. Said application is incorporated in its entirety herein by reference. Copyright Notice

A portion of the disclosure of this patent document contains material to which a claim for copyright is made. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but reserves all other copyright rights whatsoever.

Technical field

The inventions described herein are generally in the field of computer assisted education technologies.

Background of invention

Computerized writing tablets are useful tools in education. Figure 1 , for example, presents a copy of Figure 8 from US patent application publication US2008/0070221A1 , "System and Method for Administering Learning Session Over a Network" by John J. Stuppy (Stuppy). Said application is incorporated herein by reference with specific reference to the computer implemented technologies described therein.

Stuppy describes a computerized tablet where a worksheet 100 is presented to a student using a tablet screen. The student's handwritten responses are recorded 102 by the tablet. Icons 104 are presented to change pages. The student may also select 106 a "pencil" or "eraser" mode for the stylus used to write on the tablet. Data from the student's tablet may be transferred to a teacher's workstation for review.

US published patent application US 2004/0121298, "System and Method of Capturing and Processing Hand-Written Responses in the Administration of Assessments", by Creamer et al. (Creamer), describes a system where a student may submit handwritten constructed responses to a computer system using a digital recording stylus. Said application is incorporated herein by reference with specific reference to the stylus and other computer

implemented technologies described therein.

The stylus records each pen stroke and labels it with a time stamp. The stylus mays also record erasure strokes. A human scorer can determine the intended response of the student by reviewing each pen and erasure stroke in the order it was made.

Neither Stuppy nor Creamer provides any guidance on how to determine a student's mastery of a problem. They are only concerned with determining what a student's intended answer was. Some educational systems, however, require the determination of a student's mastery of a problem. The Kumon® system is one such system. The basics of the Kumon system are described in "The Kumon Approach to Teaching and Learning", Journal of Japanese Studies, Vol. 20, No. 1 Winter, 1994, pp. 87-1 13 (Ukai). Said reference is incorporated herein by reference with particular reference to the description of the Kumon approach. This approach is summarized as follows:

"To its sequential presentation, the Kumon method adds the principle of automaticity, or "over/earning, " which is the measure of whether material has been mastered. Children must practice computation until finding solutions becomes automatic. They progress to a higher level of work only after they show the ability to complete sheets accurately within prescribed time and mistake limits. If either of the limits is exceeded, additional drilling is assigned. The method is put into practice as follows:

1. The newly enrolled child takes a 20-minute diagnostic test. After the score is evaluated, the child is placed at an extremely low skill level in order to enhance his or her early performance and thereby build confidence and motivation.

2. The child is presented with a new plastic Kumon box that contains several stapled packets of 3 to 10 small-sized worksheets. One packet is to be completed each day, requiring 15-30 minutes' study.

3. Twice a week, the child attends a Kumon classroom (kyoshitsu).

The completed homework is turned in and that day's packet is done at the classroom.

4. The child receives back previous worksheets and corrects the mistakes until a perfect score is returned. The process of correcting one's own mistakes is seen as an important opportunity for self-teaching.

5. The instructor charts the child's progress in a detailed record book and, according to the most recent results, assigns more difficult work or repetition of previous pages.

6. Kumon is practiced every day of the year" (Ukai p 91 ).

Thus one of the goals of the Kumon system is to be able to determine if a child has overlearned a technique to the point were finding a solution becomes automatic. This is termed "automaticity".

110

Traditionally, automaticity in the Kumon approach has been measured by a combination of the time it takes a student to complete work, the accuracy of the student's work, direct observation of the student doing the work and a review of the scratch work in a given workbook or diagnostic test. Figure 2 l is presents a page of math problems 200 from a Kumon diagnostic test. The page is shown partially completed. The first problem on the page, (21 ), provides visual clues 202 and 204 to the student on how the problem is to be solved. The student's scratch work 206 is available for the teacher to review. As used herein, "scratch work" refers to writing that is not to be considered in 120 determining if a student answered a question correctly or not. A work page may have specific areas indicated for answers. Any writing outside of a predefined answer area would be defined as scratch work. In figure 2, the answer area would be above and adjacent to a division problem. Scratch work would be anywhere else, such as below a division problem. The scratch 125 work in figure 2 would indicate to a teacher or instructor that the student is solving the problems with an acceptable degree of automaticity. The scratch work shows intermediate subtractions used to arrive at the answers. Each subtraction appears to have been done correctly with one try. If the student completed this page in the required time, then the student would be judged to 130 have mastered this level of mathematics.

Area 208, however, shows that there is a great deal of information missing on how the student solved the problem. Area 208 is the area where the student erased other scratch work. An observant teacher might have noticed the

135 erasing, but would not necessarily have known what the student had initially written or why it was being erased. A replay of what a student writes and when the student writes in the scratch work area, therefore, would be very helpful to the instructor. If the student was doing multiple subtractions using multiple guesses on the digits for the quotients, then that would indicate that

140 the student needed more repetition of these sorts of problems before moving on to the next degree of difficulty. If the student was doing something unrelated to automaticity, then advancement might be appropriate. An unrelated reason might be that the student was erasing sloppy writing to replace it with neater writing.

145

One could apply the electronic recording technologies described herein to record and play back a student's intermediate scratch work for a teacher. A teacher, however, needs to review the work of a large number of students in a short amount of time. A large Kumon classroom, for example, might have 150 800 or more students in it at a given time. Additional tools, therefore, must be provided so that the teacher can quickly identify which particular sections of scratch work need to be reviewed for which students. Tools must also be provided to accelerate the rate at which any particular review is done. There is also need for tools that curriculum developers can use to visualize how a 155 very large number of students are doing on a set of worksheets so that the design of the worksheets themselves can be improved. There is also need for alternative productive activities that a student may do while waiting for corrections to be made to worksheets.

160 Disclosure of Invention

The Summary of the Invention is provided as a guide to understanding the invention. It does not necessarily describe the most generic embodiment of the invention or all species of the invention disclosed herein.

165 Figure 3 illustrates a computerized tablet technology 300 that will record the information necessary for a teacher to quickly identify which portions of a student's scratch work should be reviewed in order to determine the student's automaticity with respect to the problems presented. An exemplary

computerized tablet comprises one or more microprocessors, fixed memory no comprising computer code, input devices, and output devices. The input

devices may comprise a touch screen 302 and a video camera 306. Output devices may comprise a monitor, a radio transmitter for Wi-Fi communications and one or more ports for wired communications. An Apple® iPad® is an exemplary tablet computer.

175

A stylus 310 may be provided with a writing end 312 and an eraser end 314. The touch screen of the tablet will sense when the writing end is down, record its position and output dark pixels to the screen in the vicinity of the writing end. The pixels will be viewed by the student as writing. The touch screen 180 will also sense when the erasure end is down, record its position and output lighter toned pixels in perhaps a larger radius around its location. These will be viewed by the student as erasures. The touch screen may be tuned to the stylus so that only contact by the stylus will be recorded. Thus if a student touches the screen with a hand, finger or other object, said touch will not be 185 recorded. The moment when each pixel makes a transition will also be recorded so that a playback of the student's writing and erasure may be done.

The camera 306 may record pictures of the student at a given interval, such as once per 10 seconds. The camera may also continuously record video.

190

The student may change pages by selecting one or more navigation icons 304 that are provided. Other means for page changes may also be provided.

Technologies suitable for recording a student's writing and erasures also 195 include laptop and desktop computers with digitizing tablets, recording pens, such as those described in Creamer, video cameras observing a student writing, or any other technology that records what a student writes and erases and when the writing and erasing is done.

200 Once a student's writing is recorded, the data files are transferred to a teacher workstation for review. Figure 4 illustrates how the data of a student's writing and erasures can be presented 400 to an instructor, grader or teacher for review. The data is presented on a computerized tablet, laptop computer, desktop computer or other viewing device. The presentation comprises

205 images of the student 402, the worksheet the student was working on 404 and a time bar 410. The student's facial image and work page image are shown at a time indicated by an arrow 416 on the time bar. The particular image presented in figure 4, therefore, represents that state of the student's work at about 19 minutes into the assignment.

210

The time bar comprises a work bar 412 and a time scale 414. The work bar comprises darkly shaded areas 422 that indicate when a student was either writing or erasing. The lighter shaded areas 424 show when the student was not writing or erasing. The time bar also comprises indications 426 of what 215 page a student was working on in a given workbook at a given time. Page numbers are shown in this illustration. Different colors or shades may be used to indicate different activities or external conditions relative to the student's assignment. Writing activity may

220 be one color and erasing activity a different color. The colors may be

changed for different time periods in an assignment. If the standard time for completing an assignment is 20 minutes, for example, then the shade for writing or erasing may be green for the initial 20 minutes of the assignment and red 428 for work done after the initial 20 minutes. In this case, the

225 assignment is a placement test and 20 minutes represents the time to

complete the placement test for an acceptable level of automaticity.

Teachers can use the time bar to identify which portions of the student's recorded activities should be reviewed more closely. Large areas of non- 230 writing might indicate that a student is either confused, thinking or distracted.

A teacher might thus review the images of the student's face during those periods to determine which of those three it might be. The teacher might also observe something new or unexpected that would account for a long period of non-writing.

235

Large areas of writing and erasing might indicate that a student is either correcting penmanship, performing excessive scratch work or perhaps just doodling. A teacher might thus review a playback of the writing and erasures during those periods to determine which of those three it might be. The 240 teacher might also observe something new or unexpected.

A "minimal pause time" selector 434 may be provided to the teacher. The minimal pause time is the minimum time interval between pixel transitions that indicate that a student is still writing. When writing a multi-digit number, for

245 example, there is a short period of non-writing between the writing of one digit and the writing of the next digit. A teacher might select a minimum pause time greater than the inter-digit period so that that period did not show up as non- writing and needlessly complicate the time bar. A minimum pause time of 0.5 seconds would be suitable for this purpose. Another teacher might consider a

250 student still to be writing even if the student were to pause for short thoughts. This teacher might set the minimum pause time to something longer, such as 5 seconds. This would further simplify the display and highlight periods of long inactivity. Thus each teacher can optimize the utility of the time bar for his/her purposes.

A "replay speed" selector 436 may be provided. A teacher may wish to quickly review a section at relatively high speed, such as 15x, to scan for problem areas. A slower speed may be selected for areas where careful analysis is required.

A "skip gap" selector 438 may be provided. In the particular example shown in figure 4, about 50% of the student's time is spent not writing. By skipping these periods in the playback of the writing, substantial teacher time can be saved.

Various navigation buttons 432 may be provided. These include system settings, a summary, a place to edit comments and exit. A pause/play button 408 may be provided.

Brief Description of Drawings

Fig. 1 is an illustration of prior art computer tablet pen recording technology. Fig. 2 is an illustration of a prior art paper based placement test.

275 Fig. 3 is an illustration of a tablet computer pen recording and face recording technology.

Fig. 4 is an illustration of an exemplary embodiment of student work presentation technology.

Figures 5A through 5G illustrate sequential presentations of a student's writing 280 and erasing while solving a problem.

Fig. 6 is an illustration of a student's partially completed workbook page and associated time bar.

Fig. 7 is an illustration of two time bars for a student's first time and second time doing the same workbook page.

285 Fig. 8 is an illustration of summary data related to a student's homework that is presented to a teacher or other reviewer.

Figures 9A and 9B illustrate an arithmetic exercise.

Figures 10A and 10B illustrate a writing exercise.

Fig. 1 1 illustrates a computer based system for storing and distributing data 290 on how a student solved a problem.

Fig. 12 illustrates how summary data from a large number of student's can be presented to a curricula developer or other interested party.

Fig. 13 illustrates how thumbnail figures of completed student worksheets can be presented.

295 Figures 14A and 14B illustrate how worksheets from multiple students can be compared to identify consistent errors.

Figures 15A and 15B illustrate how worksheets from different students can be compared to identify different consistent errors.

Figures 16A and 16B illustrate how a student wrote a first answer and then 300 erased it to write a second answer.

Figure 17 presents a series of time bars to show how a student's work progressed over a several month period. Figure 18 presents time bars for three different students working on the same worksheets to show which students are at the right level of worksheet difficulty and which students should be advanced to more difficult worksheets.

Modes for carrying out invention

The following detailed description discloses various embodiments and features of the invention. These embodiments and features are meant to be 310 exemplary and not limiting.

The tools described herein help a teacher determine the level of automaticity that a student has achieved in a particular field. Referring to figure 4 as described above, a teacher has observed that a student has spent an

315 excessively long amount of time on page 6c of a placement test in the Kumon method. The student started the page at 13 minutes into the placement test and didn't complete the page until 26 minutes into the test. The student spent most of his/her time working on the problems as indicated by the long periods of writing and erasing. The teacher decided to review at least one problem on

320 the page to see exactly what the student was writing and erasing. The

teacher set the time bar pointer 416 to the writing area corresponding to problem (23) and then replayed the student's work. The replay speed was set at 15x, so the playback took only a few seconds. Nonetheless, the problem the student was having was immediately apparent. The teacher also

325 immediately realized what had to be done to improve the student's

automaticity.

Figures 5A through 5G present sequential screen shots of the student's writing and erasures while solving problem 23. Three digit numbers beginning 330 in "5" refer to elements in figures 5A through 5G.

5A shows the problem before the student began work on it. 5B shows that the student's initial guess for the quotient was "6". The student then tested the guess 502 by multiplying "6" times the divisor, "77". The result, "462", was too small.

335 5C shows that the student erased 503 the initial guess.

5D shows the student guessed again at "8" for the quotient and

multiplied it out 504. The result, "616", was too large. 5E shows the student erased 505 the second guess but immediately knew by process of elimination that the correct quotient was "7". 340 The student wrote it down and performed additional calculation.

5F shows the student completing 506 the problem.

5G shows the student erasing 507 the scratch multiplication but leaving the scratch subtraction.

After the playback, the teacher realized that the student needed additional 345 repetition with "single digit times double digit" multiplication. The student knew how to do it correctly, but the student's automaticity was not high enough so that he/she could see the correct quotient in "three digit divided by two digit" math problems without guessing. If the student were to be pushed beyond this level of math ability without the required automaticity, then the 350 student would have undue difficulty and frustration at the higher levels.

With the combination of a time bar and selectable writing playback, the teacher was able to make this diagnosis within a few minutes of total review time. Without this combination it would have taken much longer or not been 355 possible at all. The combination of the time bar and playback also facilitated discussions with the student's parents. Once the parents saw how the student was solving the problems, they readily agreed with the teacher that additional practice at multi-digit multiplication was called for, even though this skill level was below grade level for the child.

360

Figure 6 illustrates when a review of the camera photos of a student is appropriate. A worksheet 600, camera image 602, and time bar 610 are shown for a student working on a Kumon workbook. The problems presented are addition of mixed fractions/whole numbers and subtraction of mixed 365 fractions/whole numbers. During in-class work, the student fills out the work book and then presents the completed workbook to a grader. The grader then identifies any problems with an incorrect answer and returns the workbook to the student. The student then corrects the answers and returns the workbook to the grader. The grader checks the corrected answers and returns the 370 workbook to the student with any additional incorrect answers identified. The process is repeated until all of the problems are answered correctly.

The grader or instructor may be provided with a special pen for digitally writing on the student's work pages. The grader's pen might have a different

375 impedance than the student's pen so that the system software can distinguish between a grader and student's writing. Other forms of distinguishing between pens may be used. The grader's writing may be presented as a different color, such as red, to distinguish it from the student's writing.

380 With the systems described herein, the grader need not be present. The

information can be transmitted between a student and a remote grader.

Grading can also be done on a page by page basis. A completed page can be transmitted to a grader and the corrections returned to the student's computerized tablet for review once the student has completed the workbook.

385 Thus, the student need not wait for grading once the workbook is complete.

The student can immediately return to the first page to see which problems need correction.

Grading may be outsourced to grading firms located in remote or centralized 390 locations. Grading for students in North America, for example, might be done by graders located in India. The problems being graded may be sorted according to ease of grading. Problems with clear answers, for example, may be graded automatically by handwriting recognition programs. Problems with less clear answers may be graded manually. Problems with ambiguous 395 answers might be graded by a team of people that use consensus, voting or other form of collective decision making to determine whether or not a given answer is correct.

After the student completes the workbook, the data files are transferred to a 400 teacher or instructor for review. The time bar presented to the teacher or instructor in figure 6 indicates when a student was writing or erasing, the pages the student was working on during any given time period and the percent of correct answers 612 on any given 405 page. Times spent on grading 614 are indicated by yellow bands. They do not count towards a student's overall completion time. Writing and erasing periods during initial workbook completion are shown as green. Other colors may be used. Writing and erasing periods when a student corrects previous errors are shown as blue.

410

Figure 6 shows the results of an actual student's work. When a teacher reviews this student's work, the teacher might notice the relatively large non- writing periods 616 during the correction period. This might indicate

confusion, fatigue, inattention, and/or boredom on the student's part.

415 Reviewing the images of the student 602 during these periods can help

determine what the cause of non-writing was.

Figure 7 illustrates how presenting time bars for repeated work can help a teacher see how quickly a student is mastering a given workbook. When the 420 student first did the workbook, the time bar 700 indicated that the student went over the allotted time of 30 minutes. The color of the writing and erasing periods during the overtime period 702 was red. The student then had to go through 4 rounds of corrections 704 before all problems were answered correctly. The total time of completion was about 50 minutes.

425

The second time the student did the workbook 710, the student completed the work book within the allotted 30 minutes 712 and only required two rounds 714 of correction. Thus the student was making visible progress in mastering the material. If the student made little or no progress, then that would have 430 indicated that additional review of the details of the student's work would be appropriate.

A student's tablet computer may be used by the student to do homework. The tablet computer can record how the student did the homework. The data from 435 completed homework may be uploaded to a teacher's computer when the student comes to class if, for example, the student does not have internet access from home. The teacher can then review the student's homework at the beginning of a class.

440 Figure 8 presents a screen view 800 of the information that may be presented to a teacher related to a student's homework. A "completed" 802 presents summary data on how the student did his/her homework. An "assign" area 804 indicates the future work that will be assigned to the student. The completed area presents summary data for each day in rows 812. The

445 summary data may include the date and day homework was assigned, an indication of the worksheet set and time to complete the worksheet set, the status of the assignment (e.g. "F" means "finished"), and the score for how many of the problems were completed correctly on each worksheet of a given worksheet set. Comments 814 provided by the student may be presented. If

450 a teacher notices a problem with a particular day, then by clicking or other indication, a time bar 816 may be presented. In this case, the teacher noticed that the assignment from 7/6 took an exceptionally long time to complete (5 hours). A review of the time bar showed that the student spent very little time writing with long periods of inaction between each period of writing. A teacher

455 might wish to have a discussion with the student and/or the student's parents to determine why the homework was done in this manner. A teacher may also wish to pull up the recording of the student's actual writing and erasures as well as images of the student to help determine why this particular set of worksheets was taking so long.

460

If a teacher is occupied during class time with reviewing the students' homework assignments, then it may be desirable to provide drill exercises to the students during that period. Figures 9A and 9B illustrated a drill exercise called a "blue card". A blue card exercise helps student's master basic 465 arithmetic skills such as multiplying. Figure 9A is the initial screen presented to a student. A multiplication table 902 is presented with the answers filled in. An option to play a song 904 may be presented. The song may be a musical recitation of the multiplication table. This will help the student to remember each multiplication. A slider bar 906 or other means may be provided for a

470 student or teacher to select a "timer interval" for the subsequent exercise of the student filling in the multiplication table. Figure 9B presents said subsequent exercise. A highlight bar 922 begins at the top of the exercise and moves down one problem at a time according to the timer interval. In this case, the timer interval is set to 2 seconds. The student attempts to write in

475 the correct answer in synch with the highlight bar. Automatic character

recognition may be provided to convert the student's written answer into a typed answer 924. If the student did not write the correct answer, then the tablet computer will write in the correct answer and provide an indication, such as background color, that the answer the student wrote was missing or

480 incorrect 926. The student can repeat the exercise until the multiplication table is mastered and then the next table may be presented.

Figures 10A and 10B illustrate a handwriting exercise that may be presented to a student. The goal of the exercise is to have the student write numerals,

485 letters, symbols or other characters with the proper size. Figure 10A shows the first screen 1000 that may be presented to student. An example 1002 is shown of a character being written. Thus the student can see how the character is written out. The student then writes the character in a small box 1004. A larger external area 1006 is presented which the student is not to

490 write in. Referring to figure 10B, if the student writes a character outside 1012 of the small box, then the portion outside is indicated by, for example, a red highlight. Another box 1014 is then presented for the student to try again. Boxes for additional tries are presented until the student writes the character the correct size.

495

Curricula development

The systems described above may be used to improve a given curricula. In the Kumon system, for example, the contents of the worksheets provided to students is continually improved through personal observations by curricula 500 professionals of how students solve problems in worksheets. The student observation platform can be used to more thoroughly capture how a large number of students solve problems so that curricula developers can make additional improvements to the worksheets. This is illustrated in figure 1 1 . Students work on worksheets 1 102 which are corrected by graders.

505 Recordings of the students' work is uploaded to a central data base 1 104.

The data may be made available to teachers 1 106 and parents 1 108 with appropriate privacy and access controls. The data from a relatively larger number of students may be made available to curricula developers 1 1 10 with appropriate access control.

510

Figure 12 illustrates how data from a relatively large number of students may be presented 1200 to curricula developers. Each row 1202 presents how a student did on a given worksheet. Demographic information, such as the center that the student is registered with, may also be presented. The

515 worksheet is indicated. In this case, worksheet C1 13a from the Kumon

system is shown. Worksheet C1 13a is the first worksheet in the Kumon system that presents to students division problems formatted with the symbol "÷". Prior to this, students had been drilled in division using alternative symbols, so all students should be skilled in division. They merely have to get

520 to know what the symbol "÷" means. Each column in the area "questions" indicates whether or not a particular student answered a particular question correctly. A green symbol indicates a correct answer. A red symbol indicates an incorrect answer. Thus curricula developers can scroll through a large number of students to see if there are any consistencies in how students

525 make errors. If a consistency is discovered, the curricular developers can then modify the earlier worksheets accordingly to try and make sure that students are fully prepared when they get to this worksheet to get all questions answered correctly. Even with the small set of students presented in figure 12, it is apparent that there is a potential pattern in the errors.

530 Student's seem to get question #6 (1204) and #16 (1206) wrong more than other questions. Figure 13 is an illustration 1300 of a drill down view of the data presented in figure 12. In this case, thumbnail sketches 1302 are presented for the

535 worksheets summarized in figure 12. This is possible since the system

described in figure 10 stores all of the worksheet information from all of the students. Sheets that are all correct have large circles on them. Sheets with one or more errors 1304 are presented with the incorrectly answered questions indicated by a smaller circle. Other indicia of correct sheets and

540 incorrectly answered questions may be used. Review of the thumbnail

sketches indicated that when questions 6 and 16 were answered incorrectly, they were answered incorrectly in a consistent manner.

Figures 14A and 14B illustrate one manner of consistent errors in questions 6 545 and 16. Question 6 posed the problem "2÷2". Question 16 posed the

problem "3÷3". A fair number of students would answer these both with a "0". See 1402, 1404, 1412 and 1414. This indicates that the operation of dividing a number by itself was not as effectively presented as it could have been to these students in the earlier worksheets. It's possible that this sort of error is 550 more prevalent in students of certain demographics. Worksheets from

students of different demographics could be analyzed to explore this hypothesis. If a consistent pattern is found, worksheets might be customized for these students to more appropriately take into account their demographics or cultural background.

555

Figures 15A and 15B illustrate another manner in which students are consistently incorrectly answering question 6. They are answering the problem "2÷2" with 4. Compare 1502 and 1512. They are answering the problem "3÷3" correctly, however. See 1504 and 1514. This indicates that 560 these particular students understand the concept "a number divided by itself is 1 ", but that there is something else about question 6 that is misleading them. It's possible that the question immediately prior to question 6 might be causing the problem. Question 5 is "4÷2 =2". This may be causing the student's to reflexively put "4" in the answer for question 6. This hypothesis could be 565 tested by rearranging the questions in column 1 to see if the incidence of this error is reduced.

Provision can be provided to present a playback of a student's work to further diagnose consistent errors that may be attributable to worksheet design. For

570 example, referring to the placement test page presented in figure 4, it is

intended that the student answer the questions by proceeding down the first column and then down the second column. The questions in a column are in order of increasing difficulty proceeding down. Thus the questions in the second column are significantly harder than the questions in the first column.

575 Some students, however, are strongly conditioned to work from left to right.

These students might get immediately stuck if they do an easy problem at the top of the left column and then proceed to a much harder problem at the top of the right column. If the overall data showed that this was widespread problem, then the placement tests might be reorganized so that the hardness

580 progressed from left to right instead of top to bottom.

Erasures can be used in combination with playback to determine if students are confused by a particular problem. In figure 15A and 15B, for example, students did not erase initial answers to question 6, "2÷2=". This suggests

585 that students were not confused but were confident that their answers were correct, even though they were wrong. This could have implications on how the earlier worksheets are adjusted. One student did erase 1516 an initial answer to question 7. Figures 16A and 16B illustrate what a curricula developer would see if he/she played back how that particular student

590 answered the questions. Figure 16A shows what the student initially wrote

1602. Figure 16B shows what the student finally wrote 1612. It was the same correct answer, "6". The student simply wished to write it more clearly. Thus this question does not seem to be a source of confusion, even though an initial answer was erased.

595

Example 1 A large Kumon classroom of 800 students was initially set up to run with manual grading of paper worksheets. When students arrived, they provided their completed paper homework books to one or more clerks who checked 600 them in. The homework books were given to one or more graders who

corrected the homework. While the graders were correcting homework, the students retrieved their paper worksheets for the session and began to work on them. The class had 10 graders, 4 clerks, 3 assistants to monitor the class room, 1 instructor and a worksheet helper to handle worksheets.

605

Each student received 10 worksheets. Different students received different worksheets depending upon their level of proficiency. The students then completed the paper worksheets and provided them to the graders and retrieved their corrected homework. The graders graded the worksheets while

610 the students corrected any errors in the homework. Corrected homework was provided back to the graders and the graders provided corrected worksheets to the students. The process continued to cycle until the students answer all of the questions in the homework and worksheets correctly. The students then retrieved paper homework and left the classroom. Students spent

615 between 10 to 60 minutes or more in the classroom, depending upon how long it took to get all of the questions correct. Students also consulted with graders when they were confused about one or more questions.

A subset of students, clerks and graders in the Kumon classroom were then 620 provided with computerized tablets with Student Observation Platform (SOP) software loaded on them. A special SOP assistant helped insure that the SOP systems operated properly. The tablets presented worksheets to the students and recorded all of the students' answers and scratch work so that it could be played back. Data was uploaded to a centralized database for future 625 retrieval. When the students arrived at class, their homework was uploaded wirelessly to the database and then downloaded to the clerks' and graders' tablets. The clerks checked the students in and the students' worksheets for the session were downloaded to their tablets. The graders graded the homework and the graded homework was downloaded to the students' 630 tablets. As the students completed each worksheet, the completed worksheets were transferred to the graders who graded them and transferred the graded worksheets back to the students. Thus the waiting time for both students and graders for each other's completed work was substantially reduced relative to the paper system. Furthermore, the classroom was much

635 quieter since there was less manual transfer of paper worksheets and less need for students and staff to walk around the classroom. When the students were finished, their homework worksheets were automatically transferred to their tablets.

640 Example 2

A student in a Kumon classroom was provided with a Student Observation Platform and used it for a period of several months. The student completed two series of mathematical worksheets, series D and series E. An instructor and the student's parents reviewed several time bars of the student's work as

645 shown in figure 17. Each time bar shows summary data 1702 with the date the work was done, the worksheet pages (e.g. D26-30), the time to complete the work and the number of corrections required. A time bar 1704 is presented below each day's summary data. The time bar is color coded as described above with writing and erasing time shown as a dark shade and

650 non-writing or erasing time shown as a light shade (e.g. white).

The parents were initially concerned about the relatively large non-writing times 1706 and 1708 shown for worksheet pages D151 -155. The instructor explained, however, that these particular worksheets introduce new concepts

655 and that the non-writing times imply that the student was thinking about the examples provided. A review of the associated video of the student while working on the problems confirmed that the students was focused on the examples and not otherwise distracted. The instructor also indicated to the parents that the short time 1712 that the student took for the more advanced

660 worksheet pages E 21 -25 supported the conclusion that the student achieved an appropriate degree of mastery for the material first presented in the earlier worksheets D151 -155. Example 3

665 An instructor compared the time bars for three different students, A, B and C for a set of Kumon math worksheets E26-30 presented on Student

Observation Platforms. The instructor observed that students A and B wrote answers almost continuously as indicated by the large bands of writing 1802, with minimal non-writing areas 1804. Students A and B did show significant

670 periods of non-writing 1806, 1808, but when the instructor reviewed the

student images it was clear that the students were bored and not confused. The instructor concluded that these two students were ready to advance to the next level. Student C, on the other hand had a relatively even ratio of writing 1812 and non-writing time 1814. A review of student C's images

675 confirmed that the student was thinking about the problems during the non- writing time. The time student C took to complete each subsequent workbook page decreased. This indicated to the instructor that student C was at the appropriate level of worksheets for the student's skills and was learning the material rapidly.

680

The instructor also showed several students how their time bars compared to the time bar of a very good student. This motived the students to improve their work habits.

685 Conclusion

Whereas various particular embodiments of the present invention have been described with respect to different applications, a person of ordinary skill in the art will appreciate how various elements of the different embodiments can be applied to a variety of applications in accordance with the requirements of said 690 applications.

Claims

Wherein I claim:

1 . A method for recording and presenting a playback of a student's

scratch work, said method comprising:

a) presenting a question, an answer area and a scratch area to a student;

b) recording a plurality of positions and times of a stylus used by said student to answer said question during an answer period wherein at least a portion of said positions are in said scratch area and outside of said answer area;

c) playing back said recording to a person such that the motions of said stylus in said scratch area are reproduced.

2. The method of claim 1 which further comprises the step of presenting to said person a time bar, said time bar indicating when said stylus was writing and when it was not writing.

3. The method of claim 2 wherein a first time bar is presented for the first time said student answered said question and a second time bar is presented for the second time said student answered said question.

4. The method of claim 1 which further comprising recording when said stylus positions are writings or erasures and wherein said playback displays said writings as one color or shade and said erasures as a different color or shade.

5. The method of claim 4 wherein said playback is a single image of said writing at a point in time.

6. The method of claim 1 which further comprises recording at least one image of said student taken during said answer period and wherein said playback comprises said at least one image synchronized with said motion of said stylus.

7. The method of claim 1 which further comprises presenting summary data to said person, said summary data comprising an indication of whether or not said student answered said question correctly.

8. The method of claim 7 which further comprises the step of

automatically determining if said question is answered correctly by using a handwriting recognition algorithm.

9. The method of claim 1 which further comprises the steps of:

a) presenting a first question to said student;

b) presenting a second question to said student;

c) determining by said person if said first question is answered correctly while said student is answering said second question; and

d) transmitting said determination to said student such that said student may answer said first question again immediately after answering said second question if said first answer to said first question was found to be incorrect.

10. The method of claim 1 which comprises the steps of providing a

plurality of questions to a plurality of students and recording the stylus motions of said students answering said questions in a data base.

1 1 . The method of claiml O wherein summary data of said answers are presented to said person, said summary data comprising an array of icons corresponding to the correctness of the answers given by said students to said questions.

12. The method of claim 1 1 which further comprises presenting to said person the handwritten answers of said questions by at least two of said students.

13. The method of claim 12 which further comprises modifying said

questions based on consistencies in incorrect answers in said handwritten answers.

14. A method of drilling a student in arithmetic, said method comprising: a) presenting a plurality of arithmetic problems to said student on a digital presentation device wherein each of said problems comprises an answer area;

b) sequentially highlighting each of said plurality of problems for a fixed period of time;

c) recognizing the handwritten answer to each problem using an automated handwriting analysis performed on each answer area at the end of the period each answer area was highlighted; d) presenting the correct answer in an answer area if the

handwritten answer was incorrect; and

e) providing a visual indication that an answer was incorrect.

15. The method of claim 14 which further comprises playing a song that recites the answers to said plurality of arithmetic problems.

16. A method of drilling a student to write characters a proper size, said method comprising:

a) presenting a first correct area for said student to write said

character;

b) providing a first incorrect area where said student should not write said character, said first incorrect area being outside said first correct area;

c) recording the motions said student makes to write said character a first time in said first correct area; and

d) automatically presenting a second correct area for said student to write said character a second time if a portion of said first character said student wrote was in said first incorrect area.

17. The method of claim 16 wherein said portion of said first character that is in said first incorrect area is highlighted.

18. A system for recording and presenting a playback of a student's

scratch work, said system comprising:

a) a question, an answer area and a scratch area;

b) a stylus;

c) a recording device for recording a plurality of positions and times of said stylus used by said student to answer said question during an answer period wherein said device records if said stylus motions are in said answer area or in said scratch area; d) a playback device for playing back said recording to a person such that the motions of said stylus in said scratch area are reproduced.

19. The system of claim 18 which further comprises a student's stylus and a grader's stylus and wherein said playback device presents the positions of said student's stylus in a first color that the positions of said grader's stylus in a second color.

20. The system of claim 18 wherein said stylus comprises a writing end and an erasing end and wherein said display device displays recorded motions of said writing end in a writing color and said erasing end in an erasing color.

21 . The system of claim 18 which further comprises a clerical device, and wherein said recording device comprises computer readable

instructions on a fixed memory to forward the answers to said questions to said clerical device such that said clerical device can confirm that said questions have been answered.

22. The system of claim 21 that comprises 100 or more recording devices that comprise computer readable instructions on a fixed memory to transmit answers on said recording devices to one or more clerical devices and recorded stylus movements on said recording devices to one or more display devices such that clerks in possession of said clerical devices can confirm said questions are answered while at the same time graders in possession of said display devices can correct said answers to said questions.