WO2022269801A1 - Video analysis system - Google Patents

Video analysis system Download PDF

Info

Publication number
WO2022269801A1
WO2022269801A1 PCT/JP2021/023776 JP2021023776W WO2022269801A1 WO 2022269801 A1 WO2022269801 A1 WO 2022269801A1 JP 2021023776 W JP2021023776 W JP 2021023776W WO 2022269801 A1 WO2022269801 A1 WO 2022269801A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
moving image
unit
analysis
biological reaction
Prior art date
Application number
PCT/JP2021/023776
Other languages
French (fr)
Japanese (ja)
Inventor
渉三 神谷
Original Assignee
株式会社I’mbesideyou
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社I’mbesideyou filed Critical 株式会社I’mbesideyou
Priority to JP2023529316A priority Critical patent/JPWO2022269801A1/ja
Priority to PCT/JP2021/023776 priority patent/WO2022269801A1/en
Publication of WO2022269801A1 publication Critical patent/WO2022269801A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/15—Conference systems

Definitions

  • the present invention relates to a moving image analysis system that analyzes participants' biological reactions based on moving images obtained from online sessions conducted by multiple participants.
  • Patent Document 1 A technique for analyzing the emotions others receive in response to a speaker's remarks (see Patent Document 1, for example).
  • Patent Document 2 For example.
  • Patent Document 3 A technique for chronologically analyzing changes in a subject's facial expression over a long period of time and estimating the emotions held during that period.
  • Patent Documents 3 to 5 There are also known techniques for identifying the factors that most affected changes in emotions (see Patent Documents 3 to 5, for example).
  • Patent Documents 3 to 5 There is also known a technique that compares the subject's usual facial expression with the current facial expression and issues an alert when the facial expression is dark (see Patent Document 6, for example).
  • Patent Document 6 There is also known a technique for determining the degree of emotion of a subject by comparing the subject's normal (expressionless) facial expression with the current facial expression (see, for example, Patent Documents 7 to 9).
  • Techniques for analyzing the emotions of an organization and the atmosphere felt by individuals within a group are also known (see Patent Documents 10 and 11, for example).
  • the purpose of the present invention is to objectively evaluate these communications in order to conduct more efficient communication in situations where online communication is the main focus, such as meetings and lectures.
  • a moving image analysis system that analyzes reactions of users, comprising: a moving image acquiring unit that acquires a moving image obtained by photographing the user during the online session for each of the plurality of users; an analysis unit that analyzes changes in biological reactions of the user based on the moving image acquired by the acquisition unit; a specifying unit that specifies a section of a moving image in which another user different from the user is speaking and acting toward the user; and an evaluation information generation unit that generates a moving image analysis system.
  • exchanged communication can be objectively evaluated in order to conduct more efficient communication in situations where online communication is the main activity.
  • FIG. 1 is an example of a functional block diagram of an evaluation terminal according to an embodiment of the present invention
  • FIG. 3 is a diagram showing functional configuration example 1 of the evaluation terminal according to the embodiment of the present invention
  • FIG. 8 is a diagram showing functional configuration example 2 of the evaluation terminal according to the embodiment of the present invention
  • FIG. 10 is a diagram showing a functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention
  • 7 is a screen display example according to the functional configuration example 3 of FIG. 6.
  • FIG. FIG. 7 is another screen display example according to the functional configuration example 3 of FIG. 6.
  • FIG. FIG. 12 is a diagram showing another configuration of functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention
  • FIG. 12 is a diagram showing another configuration of functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention. It is a figure showing an example of functional composition of a system concerning this embodiment.
  • FIG. 4 is a diagram for explaining a specific example of an evaluation target section according to the embodiment; It is a figure which shows an example of the output mode by the evaluation output part which concerns on this embodiment.
  • the contents of the embodiments of the present disclosure are listed and described.
  • the present disclosure has the following configurations. (Item 1) In an environment where an online session is held by a plurality of users, the reaction of the user is analyzed based on a moving image obtained by photographing the user regardless of whether or not the user is displayed on a screen during the online session.
  • a moving image analysis system a moving image acquisition unit that acquires a moving image obtained by photographing the user during the online session for each of the plurality of users; an analysis unit that analyzes changes in biological reactions of the user based on the moving image acquired by the moving image acquisition unit; A specifying unit that specifies a section of a moving image in which a user different from the user is speaking and acting toward the user based on the timing at which the analysis result of the biological reaction obtained by the analysis unit satisfies a predetermined condition.
  • an evaluation information generation unit that generates evaluation information for the speech and behavior of the other user based on the moving image included in the section;
  • a moving image analysis system A moving image analysis system.
  • (Item 2) The moving image analysis system according to item 1, The moving image analysis system, wherein the evaluation information generation unit generates the evaluation information based on the moving image of the other user.
  • (Item 3) The moving image analysis system according to item 1 or 2, The moving image analysis system, wherein the evaluation information generation unit generates the evaluation information for the behavior of the user based on an analysis result of the biological reaction of the user.
  • a video session in an environment where a video session (hereinafter referred to as an online session including one-way and two-way sessions) is held by a plurality of people, the person to be analyzed among the plurality of people is different from the others. It is a system that analyzes and evaluates specific emotions (feelings that occur in response to one's own or others' words and actions. pleasant/unpleasant, or their degree).
  • Online sessions are, for example, online meetings, online classes, online chats, etc. Terminals installed in multiple locations are connected to a server via a communication network such as the Internet, and moving images are transmitted between multiple terminals through the server. It's made to be interactable.
  • Moving images handled in online sessions include facial images and voices of users using terminals.
  • Moving images also include images such as materials that are shared and viewed by a plurality of users. It is possible to switch between the face image and the document image on the screen of each terminal to display only one of them, or to divide the display area and display the face image and the document image at the same time. In addition, it is possible to display the image of one user out of a plurality of users on the full screen, or divide the images of some or all of the users into small screens and display them. It is possible to designate one or a plurality of users among a plurality of users participating in an online session using terminals as analysis subjects.
  • an online session leader, moderator, or manager designates any user as an analysis subject.
  • Hosts of online sessions are, for example, instructors of online classes, chairpersons and facilitators of online meetings, coaches of sessions for coaching purposes, and the like.
  • the host of an online session is typically one of multiple users participating in the online session, but may be another person who does not participate in the online session. It should be noted that all participants may be analyzed without designating the analysis subject.
  • an online session leader, moderator, or administrator hereinafter collectively referred to as the organizer to designate any user as an analysis subject.
  • Hosts of online sessions are, for example, instructors of online classes, chairpersons and facilitators of online meetings, coaches of sessions for coaching purposes, and the like.
  • the host of an online session is typically one of multiple users participating in the online session, but may be another person who does not participate in the online session.
  • the video session evaluation system displays at least moving images obtained from a video session established between a plurality of terminals.
  • the displayed moving image is acquired by the terminal, and at least a face image included in the moving image is identified for each predetermined frame unit. An evaluation value for the identified face image is then calculated.
  • the evaluation value is shared as necessary.
  • the acquired moving image is stored in the terminal, analyzed and evaluated on the terminal, and the result is provided to the user of the terminal. Therefore, for example, even a video session containing personal information or a video session containing confidential information can be analyzed and evaluated without providing the moving image itself to an external evaluation agency or the like.
  • the evaluation result evaluation value
  • the video session evaluation system includes user terminals 10 and 20 each having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display, and an output unit such as a speaker. , a video session service terminal 30 for providing an interactive video session to the user terminals 10, 20, and an evaluation terminal 40 for performing part of the evaluation of the video session.
  • Each functional block, functional unit, and functional module described below can be configured by any of hardware, DSP (Digital Signal Processor), and software provided in a computer, for example.
  • DSP Digital Signal Processor
  • a computer CPU random access memory
  • RAM random access memory
  • ROM read-only memory
  • a series of processes by the systems and terminals described herein may be implemented using software, hardware, or a combination of software and hardware. It is possible to create a computer program for realizing each function of the information sharing support device 10 according to the present embodiment and implement it in a PC or the like. It is also possible to provide a computer-readable recording medium storing such a computer program.
  • the recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like.
  • the above computer program may be distributed, for example, via a network without using a recording medium.
  • the evaluation terminal acquires a moving image from a video session service terminal, identifies at least a face image included in the moving image for each predetermined frame unit, and calculates an evaluation value for the face image ( will be described in detail later).
  • the video session service provided by the video session service terminal (hereinafter sometimes simply referred to as "this service") provides user terminals 10 and 20 with two-way images and voice. Communication is possible.
  • this service a moving image captured by the camera of the other user's terminal is displayed on the display of the user's terminal, and audio captured by the microphone of the other's user's terminal can be output from the speaker.
  • this service allows both or either of the user terminals to record moving images and sounds (collectively referred to as "moving images, etc.") in the storage unit of at least one of the user terminals. configured as possible.
  • the recorded moving image information Vs (hereinafter referred to as “recorded information”) is cached in the user terminal that started recording and is locally recorded only in one of the user terminals. If necessary, the user can view the recorded information by himself or share it with others within the scope of using this service.
  • FIG. 4 is a block diagram showing a configuration example according to this embodiment.
  • the video session evaluation system of this embodiment is implemented as a functional configuration of the user terminal 10.
  • the user terminal 10 has, as its functions, a moving image acquisition unit 11, a biological reaction analysis unit 12, a peculiar determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16.
  • the moving image acquisition unit 11 acquires from each terminal a moving image obtained by photographing a plurality of people (a plurality of users) with a camera provided in each terminal during an online session. It does not matter whether the moving image acquired from each terminal is set to be displayed on the screen of each terminal. That is, the moving image acquisition unit 11 acquires moving images from each terminal, including moving images being displayed and moving images not being displayed on each terminal.
  • the biological reaction analysis unit 12 analyzes changes in the biological reaction of each of a plurality of people based on the moving images (whether or not they are being displayed on the screen) acquired by the moving image acquiring unit 11.
  • the biological reaction analysis unit 12 separates the moving image acquired by the moving image acquisition unit 11 into a set of images (collection of frame images) and voice, and analyzes changes in the biological reaction from each.
  • the biological reaction analysis unit 12 analyzes the user's facial image using a frame image separated from the moving image acquired by the moving image acquisition unit 11 to obtain at least one of facial expression, gaze, pulse, and facial movement. Analyze changes in biological reactions related to Further, the biological reaction analysis unit 12 analyzes the voice separated from the moving image acquired by the moving image acquisition unit 11 to analyze changes in the biological reaction related to at least one of the user's utterance content and voice quality.
  • the biological reaction analysis unit 12 calculates a biological reaction index value reflecting the change in biological reaction by quantifying the change in biological reaction according to a predetermined standard.
  • the analysis of changes in facial expressions is performed as follows. That is, for each frame image, a facial region is identified from the frame image, and the identified facial expressions are classified into a plurality of types according to an image analysis model machine-learned in advance. Then, based on the classification results, it analyzes whether positive facial expression changes occur between consecutive frame images, whether negative facial expression changes occur, and to what extent the facial expression changes occur, A facial expression change index value corresponding to the analysis result is output.
  • the analysis of changes in line of sight is performed as follows. That is, for each frame image, the eye region is specified in the frame image, and the orientation of both eyes is analyzed to analyze where the user is looking. For example, it analyzes whether the user is looking at the face of the speaker being displayed, whether the user is looking at the shared material being displayed, or whether the user is looking outside the screen. Also, it may be analyzed whether the eye movement is large or small, or whether the movement is frequent or infrequent. A change in line of sight is also related to the user's degree of concentration.
  • the biological reaction analysis unit 12 outputs a line-of-sight change index value according to the analysis result of the line-of-sight change.
  • the analysis of pulse changes is performed, for example, as follows. That is, for each frame image, the face area is specified in the frame image. Then, using a trained image analysis model that captures numerical values of face color information (G of RGB), changes in the G color of the face surface are analyzed. By arranging the results along the time axis, a waveform representing changes in color information is formed, and the pulse is identified from this waveform. When a person is tense, the pulse speeds up, and when the person is calm, the pulse slows down. The biological reaction analysis unit 12 outputs a pulse change index value according to the analysis result of the pulse change.
  • G of RGB face color information
  • analysis of changes in facial movement is performed as follows. That is, for each frame image, the face area is specified in the frame image, and the direction of the face is analyzed to analyze where the user is looking. For example, it analyzes whether the user is looking at the face of the speaker being displayed, whether the user is looking at the shared material being displayed, or whether the user is looking outside the screen. Further, it may be analyzed whether the movement of the face is large or small, or whether the movement is frequent or infrequent. The movement of the face and the movement of the line of sight may be analyzed together. For example, it may be analyzed whether the face of the speaker being displayed is viewed straight, whether the face is viewed with upward or downward gaze, or whether the face is viewed obliquely.
  • the biological reaction analysis unit 12 outputs a face orientation change index value according to the analysis result of the face orientation change.
  • Analysis of the contents of the statement is performed, for example, as follows. That is, the biological reaction analysis unit 12 converts the voice into a character string by performing known voice recognition processing on the voice for a specified time (for example, about 30 to 150 seconds), and morphologically analyzes the character string. By doing so, words such as particles and articles that are unnecessary for expressing conversation are removed. Then, vectorize the remaining words, analyze whether a positive emotional change has occurred, whether a negative emotional change has occurred, and to what extent the emotional change has occurred. Outputs the statement content index value.
  • Voice quality analysis is performed, for example, as follows. That is, the biological reaction analysis unit 12 identifies the acoustic features of the voice by performing known voice analysis processing on the voice for a specified time (for example, about 30 to 150 seconds). Then, based on the acoustic features, it analyzes whether a positive change in voice quality has occurred, whether a negative change in voice quality has occurred, and to what extent the change in voice quality has occurred, and according to the analysis results, output the voice quality change index value.
  • a specified time for example, about 30 to 150 seconds
  • the biological reaction analysis unit 12 uses at least one of the facial expression change index value, eye line change index value, pulse change index value, face direction change index value, statement content index value, and voice quality change index value calculated as described above. to calculate the biological reaction index value.
  • the biological reaction index value is calculated by weighting the facial expression change index value, eye line change index value, pulse change index value, face direction change index value, statement content index value, and voice quality change index value.
  • the peculiarity determination unit 13 determines whether or not the change in the analyzed biological reaction of the person to be analyzed is more specific than the change in the analyzed biological reaction of the person other than the person to be analyzed. In the present embodiment, the peculiarity determination unit 13 compares changes in the biological reaction of the person to be analyzed with those of others based on the biological reaction index values calculated for each of the plurality of users by the biological reaction analysis unit 12. is specific or not.
  • the peculiar determination unit 13 calculates the variance of the biological reaction index values calculated for each of the plurality of persons by the biological reaction analysis unit 12, and compares the biological reaction index values calculated for the analysis subject with the variance, It is determined whether or not the change in the analyzed biological reaction of the person to be analyzed is specific compared to others.
  • the following three patterns are conceivable as cases where the changes in biological reactions analyzed for the subject of analysis are more specific than those of others.
  • the first is a case where a relatively large change in biological reaction occurs in the subject of analysis, although no particularly large change in biological reaction has occurred in the other person.
  • the second is a case where a particularly large change in biological reaction has not occurred in the subject of analysis, but a relatively large change in biological reaction has occurred in the other person.
  • the third is a case where a relatively large change in biological reaction occurs in both the subject of analysis and the other person, but the content of the change differs between the subject of analysis and the other person.
  • the related event identification unit 14 identifies an event occurring in relation to at least one of the person to be analyzed, the other person, and the environment when the change in the biological reaction determined to be peculiar by the peculiarity determination unit 13 occurs. .
  • the related event identification unit 14 identifies from the moving image the speech and behavior of the person to be analyzed when a specific change in biological reaction occurs in the person to be analyzed.
  • the related event identifying unit 14 identifies, from the moving image, the speech and behavior of the other person when a specific change in the biological reaction of the person to be analyzed occurs.
  • the related event identification unit 14 identifies from the moving image the environment in which a specific change in the biological reaction of the person to be analyzed occurs.
  • the environment is, for example, the shared material being displayed on the screen, the background image of the person to be analyzed, and the like.
  • the clustering unit 15 clusters the change in the biological reaction determined to be specific by the peculiarity determination unit 13 (for example, one or a combination of eye gaze, pulse, facial movement, statement content, and voice quality), and the peculiarity Analyzing the degree of correlation with an event (event identified by the related event identification unit 14) that occurs when a change in biological reaction occurs, and if it is determined that the correlation is at a certain level or more , to cluster the subjects or events based on the correlation analysis results.
  • the peculiarity determination unit 13 for example, one or a combination of eye gaze, pulse, facial movement, statement content, and voice quality
  • the clustering unit 15 clusters the person to be analyzed or the event into one of a plurality of pre-segmented categories according to the content of the event, the degree of negativity, the magnitude of the correlation, and the like.
  • the clustering unit 15 clusters the person to be analyzed or the event into one of a plurality of pre-segmented classifications according to the content of the event, the degree of positivity, the degree of correlation, and the like.
  • the analysis result notification unit 16 reports at least one of the changes in the biological reaction determined to be specific by the peculiar determination unit 13, the event identified by the related event identification unit 14, and the classification clustered by the clustering unit 15. , to notify the designator of the subject of analysis (the subject of analysis or the organizer of the online session).
  • the analysis result notification unit 16 recognizes that when a change in a specific biological reaction that is different from that of the other person occurs in the person to be analyzed (one of the three patterns described above; the same applies hereinafter), the analysis target is Notifies the person to be analyzed of his/her own behavior. This allows the person to be analyzed to understand that he/she has a different feeling from others when he or she performs a certain behavior. At this time, the person to be analyzed may also be notified of the change in the specific biological reaction identified for the person to be analyzed. Furthermore, the person to be analyzed may be further notified of the change in the biological reaction of the other person to be compared.
  • the words and deeds of the person to be analyzed performed without being particularly conscious of their usual emotions, or the words and deeds of the person to be analyzed consciously accompanied by certain emotions, and the emotions and behaviors that others received
  • the emotion held by the person to be analyzed is different from the feeling held by the person to be analyzed at the time
  • the person to be analyzed is notified of the speech and behavior of the person to be analyzed at that time.
  • the analysis result notification unit 16 notifies the organizer of the online session of the event occurring when the person to be analyzed undergoes a specific change in biological reaction that is different from that of the other person, together with the change in the specific biological reaction. to notify.
  • the organizer of the online session can know what kind of event affects what kind of emotional change as a phenomenon specific to the specified analysis subject. Then, it becomes possible to perform appropriate treatment on the person to be analyzed according to the grasped contents.
  • the analysis result notification unit 16 notifies the organizer of the online session of the event occurring when a specific change in biological reaction occurs in the analysis subject, which is different from that of others, or the clustering result of the analysis subject. do.
  • online session organizers can grasp behavioral tendencies peculiar to analysis subjects and predict possible future behaviors and situations, depending on which classification the specified analysis subjects have been clustered into. be able to. Then, it becomes possible to take appropriate measures for the person to be analyzed.
  • the biological reaction index value is calculated by quantifying the change in biological reaction according to a predetermined standard, and the analysis subject is analyzed based on the biological reaction index value calculated for each of the plurality of people.
  • the biological reaction analysis unit 12 analyzes the movement of the line of sight for each of a plurality of people and generates a heat map indicating the direction of the line of sight.
  • the peculiar determination unit 13 compares the heat map generated for the person to be analyzed by the biological reaction analysis unit 12 with the heat map generated for the other person, so that the change in the biological reaction analyzed for the person to be analyzed It is determined whether it is specific compared with the change in biological response analyzed for.
  • moving images of a video session are stored in the local storage of the user terminal 10, and the above analysis is performed on the user terminal 10.
  • the machine specs of the user terminal 10 it is possible to analyze the moving image information without providing it to the outside.
  • the video session evaluation system of this embodiment may include a moving image acquisition unit 11, a biological reaction analysis unit 12, and a reaction information presentation unit 13a as functional configurations.
  • the reaction information presentation unit 13a presents information indicating changes in biological reactions analyzed by the biological reaction analysis unit 12a, including participants not displayed on the screen.
  • the reaction information presenting unit 13a presents information indicating changes in biological reactions to an online session leader, moderator, or administrator (hereinafter collectively referred to as the organizer).
  • Hosts of online sessions are, for example, instructors of online classes, chairpersons and facilitators of online meetings, coaches of sessions for coaching purposes, and the like.
  • An online session host is typically one of the users participating in the online session, but may be another person who does not participate in the online session.
  • the organizer of the online session can also grasp the state of the participants who are not displayed on the screen in an environment where the online session is held by multiple people.
  • FIG. 6 is a block diagram showing a configuration example according to this embodiment. As shown in FIG. 6, in the video session evaluation system of the present embodiment, functions similar to those of the above-described first embodiment are given the same reference numerals, and explanations thereof may be omitted.
  • the system includes a camera unit that acquires images of a video session, a microphone unit that acquires audio, an analysis unit that analyzes and evaluates moving images, and information obtained by evaluating the acquired moving images.
  • an object generator for generating a display object (described below) based on the display; and a display for displaying both the moving image of the video session and the display object during execution of the video session.
  • the analysis unit includes the moving image acquisition unit 11, the biological reaction analysis unit 12, the peculiar determination unit 13, the related event identification unit 14, the clustering unit 15, and the analysis result notification unit 16, as described above.
  • the function of each element is as described above.
  • the object generation unit generates an object 50 representing the recognized face part and the above-mentioned Information 100 indicating the content of the analysis/evaluation performed is superimposed on the moving image and displayed.
  • the object 50 may identify and display all faces of a plurality of persons when the faces of the plurality of persons are moved in the moving image.
  • the object 50 is, for example, when the camera function of the video session is stopped at the other party's terminal (that is, it is stopped by software within the application of the video session instead of physically covering the camera). If the other party's face is recognized by the other party's camera, the object 50 or the object 100 may be displayed in the part where the other party's face is located. This makes it possible for both parties to confirm that the other party is in front of the terminal even if the camera function is turned off. In this case, for example, in a video session application, the information obtained from the camera may be hidden while only the object 50 or object 100 corresponding to the face recognized by the analysis unit is displayed. Also, the video information acquired from the video session and the information recognized by the analysis unit may be divided into different display layers, and the layer relating to the former information may be hidden.
  • the objects 50 and 100 may be displayed in all areas or only in some areas. For example, as shown in FIG. 8, it may be displayed only on the moving image on the guest side.
  • the embodiments of the invention described in Basic Configuration Example 1 to Basic Configuration Example 3 described above may be implemented as a single device, or a plurality of devices (for example, cloud servers) partially or entirely connected by a network. and the like.
  • the control unit 110 and the storage 130 of each terminal 10 may be realized by different servers connected to each other via a network. That is, the system includes user terminals 10, 20, a video session service terminal 30 for providing an interactive video session to the user terminals 10, 20, and an evaluation terminal 40 for evaluating the video session, Variation combinations of the following configurations are conceivable. (1) Processing everything only on the user terminal As shown in FIG. 8, by performing the processing by the analysis unit on the terminal that is performing the video session (although a certain processing capacity is required), the video session can be performed.
  • an analysis unit may be provided in an evaluation terminal connected via a network or the like.
  • the moving images acquired by the user terminal are shared with the evaluation terminal at the same time as or after the video session, and are analyzed and evaluated by the analysis unit in the evaluation terminal.
  • the moving image data that is, information including at least analysis data
  • a moving image analysis system (hereinafter simply referred to as "system") according to an embodiment of the present disclosure shoots all participants or only a specific participant in an environment where an online session is held with a plurality of participants. Participants' reactions are analyzed based on the moving images obtained by this process. The analysis may occur whether or not participants are on screen during the online session.
  • the system analysis unit
  • the analysis unit statistically analyzes and outputs the content such as the amount and frequency of communication between users and their emotions at that time by analyzing moving images.
  • the analysis unit described above analyzes the content of the utterance based on not only the user's emotion but also the moving image described above.
  • Such analysis of the content of the utterance can be performed, for example, by a known speech analysis technique or natural language processing technique for moving images.
  • the target of such analysis may be, for example, the behavior of a single user.
  • One user's words and actions cause other users to react, and such reactions can be analyzed. This reaction is easy to miss in online sessions and not easy to feedback.
  • a system is realized that enables users to receive more accurate feedback on their own behavior.
  • FIG. 10 is a diagram showing an example of the functional configuration of the system according to this embodiment.
  • the system shown in FIG. 10 includes an analysis result DB 21, a specifying unit 22, an evaluation information generating unit 23, and an output control unit 24.
  • the analysis result DB 21 can be realized by the above-described storage medium or the like.
  • the identification unit 22, the evaluation information generation unit 23, and the output control unit 24 read a program stored in a storage medium or the like provided in, for example, the user terminals 10, 20 or the evaluation terminal 40 into a memory or the like and It can be realized by execution by a processor such as.
  • the analysis result DB 21 is, for example, a database that stores analysis result data obtained by the various functional units described above.
  • the analysis result data may be, for example, the analysis result data obtained from the change in the user's biological reaction described above, or the analysis result data relating to the user's utterance.
  • These analysis results are obtained by analysis of moving images of online sessions.
  • these analysis result data include a user ID that identifies the user, analysis information obtained as a result of analysis of movement on a moving image caused by the user, or an input generated by the user's input to the user terminal Information and the like may be included as user information.
  • the identification unit 22 Based on the timing at which the analysis result of the biological reaction obtained by the analysis unit satisfies a predetermined condition, the identification unit 22 allows a user different from the one user to be analyzed to perform behavior toward the one user. It has a function to identify the section of the moving image being performed. Specifically, the identification unit 22 acquires the analysis result of the online session from the analysis result DB 21, and detects such a change in the biological reaction at the timing when the analysis result of the biological reaction of one user exceeds a predetermined standard. Identify a segment of the moving image in which the other user's speech and behavior that is thought to have occurred is performed.
  • the specifying unit 22 specifies not only the timing at which the analysis result of the biological reaction satisfies the predetermined standard, but also the timing at which the change in the biological reaction exceeds (or falls below) the predetermined standard. may be used as information for Also, such a section of the moving image may be, for example, the same section as the timing at which the biological reaction analysis result satisfies a predetermined criterion, or a section earlier than that.
  • the timing of the start or end of the interval can be determined according to, for example, the timing at which a change in biological reaction occurs, the timing of the biological reaction, or the like.
  • Such an interval may be specified based on context information obtained from analysis result data of speech and behavior before and/or after a time-series interval corresponding to the speech and behavior of another user.
  • the context information may be not only the analysis result of the information on the speech and behavior contained in the preceding and succeeding sections, but also the analysis result of changes in other users' biological reactions, for example.
  • the predetermined criterion may be, for example, a criterion for analysis results regarding positive reactions when it is desired to evaluate speech and behavior corresponding to positive biological reactions of the user.
  • the predetermined condition can be a condition based on one or more types of biological response (eg, positive, negative, joyful, sad, angry, etc.).
  • the evaluation information generation unit 23 has a function of generating evaluation information on the speech and behavior of other users based on the moving images included in the specified speech period. Specifically, the evaluation information generation unit 23 generates evaluation information for the speech and behavior of other users from the moving image included in the section.
  • the user's behavior may include, for example, speech content based on audio information obtained from moving images, and content related to biological reactions obtained from analysis results of moving images of other users.
  • the contents of the utterance can be obtained, for example, by analysis performed on speech information by a known speech analysis technique.
  • the evaluation information generation unit 23 may generate evaluation information based on the analysis result of the moving image of the one user who received the speech and behavior.
  • the evaluation information is, for example, information specifying what kind of behavior the biological reaction of one user has received, or what attribute the behavior belongs to (for example, positive, negative, fun, etc.). (e.g. sadness, anger, etc.) and information on feedback such as appropriateness of said behavior.
  • FIG. 11 is a diagram for explaining a specific example of the evaluation target section according to this embodiment.
  • a graph 1000 shown in FIG. 11 is a graph (reaction graph) showing an analysis result of the biological reaction of user B when user A and user B are having a conversation in an online session, and a graph (reaction graph) of user A and user B. shows the utterance interval of
  • the identification unit 22 first determines whether the value of the reaction graph of user B satisfies a predetermined standard when user A is speaking, or shows a change that satisfies (falls below) a predetermined standard.
  • the sections 1001 and 1002 in which the Next, the identifying unit 22 identifies utterance sections 1005 and 1006 of user A corresponding to sections 1001 and 1002 .
  • the evaluation information generator 23 can generate evaluation information based on the behavior of the user A in the utterance sections 1005 and 1006 .
  • the utterance segment 1006 includes a segment in which the value of the biological reaction analysis result is low before the start timing of the segment 1002 in which the value of the reaction graph of the user B satisfies a predetermined criterion.
  • a speech segment corresponding to segment 1004 is also included. As a result, it is possible to know in more detail which speech or behavior triggered a change in the reaction of the user B.
  • the specifying unit 22 also specifies the user B's utterance segment 1007 corresponding to the segment 1003 in which the user B's reaction graph value satisfies a predetermined criterion, and the evaluation information generating unit 23
  • the evaluation information may be generated based on the speech and behavior of the user B in the utterance section 1007 . As a result, it is possible to grasp what kind of influence is exerted on the user B's mind when what kind of behavior the user B performs.
  • the output control unit 24 may have a function of outputting evaluation information in the identified section.
  • the output control unit 24 may output the evaluation information, for example, by changing the display mode according to the evaluation result. For example, in the example of the present embodiment, when user A's behavior toward user B in an online session has a positive influence on user B with respect to a predetermined criterion, information regarding such behavior may be displayed as a color heat map or an object shape. You may output to screens, such as user terminal 10, 20, by changing display modes, such as. Thereby, the result of the feedback to the user A can be grasped intuitively.
  • the output control unit 24 may output the evaluation information related to the behavior in association with the section corresponding to the behavior. This makes it possible to easily grasp whether the behavior in which section was good (or bad) for the user. Note that the output mode of the evaluation information by the output control unit 24 is not particularly limited.
  • FIG. 12 is a flowchart showing an example of the flow of processing by the system according to this embodiment.
  • the specifying unit 22 specifies an evaluation target section of the user's behavior based on the timing when the analysis result satisfies a predetermined criterion (step S101).
  • the evaluation information generation unit 23 analyzes the behavior of the user in the section (step S103), and generates evaluation information based on the analysis result (step S105).
  • the output control unit 24 outputs the generated evaluation information to the user terminals 10, 20, etc. (step S107).

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

[Problem] To objectively evaluate online communications, which have become mainstream, in order to carry out communication more efficiently. [Solution] A system according to the present disclosure comprising: a video acquisition unit that acquires video obtained by imaging each of a plurality of users during an online session; an analysis unit that analyzes changes in biological reactions of the users on the basis of the video acquired by the video acquisition unit; an identification unit that identifies, on the basis of timing at which the biological reaction analysis results obtained from the analysis unit satisfy prescribed conditions, video segments in which a user is spoken to and interacted with by an other user different to the user; and an evaluation information generation unit that generates evaluation information for the speech and interaction of the other user on the basis of the video in the segments.

Description

動画像分析システムVideo image analysis system
　本発明は、複数人の参加者で行われるオンラインセッションによって得られる動画像をもとに参加者の生体反応を解析する動画像分析システムに関する。 The present invention relates to a moving image analysis system that analyzes participants' biological reactions based on moving images obtained from online sessions conducted by multiple participants.
　発言者の発言に対して他者が受ける感情を解析する技術が知られている（例えば、特許文献１参照）。対象者の表情の変化を長期間にわたり時系列的に解析し、その間に抱いた感情を推定する技術も知られている（例えば、特許文献２参照）。感情の変化に最も影響を与えた要素を特定する技術も知られている（例えば、特許文献３～５参照）。対象者の普段の表情と現在の表情とを比較して、表情が暗い場合にアラートを発する技術も知られている（例えば、特許文献６参照）。対象者の平常時（無表情時）の表情と現在の表情とを比較して、対象者の感情の度合いを判定するようにした技術も知られている（例えば、特許文献７～９参照）。組織としての感情や、個人が感じるグループ内の雰囲気を分析する技術も知られている（例えば、特許文献１０、１１参照）。 A technique is known for analyzing the emotions others receive in response to a speaker's remarks (see Patent Document 1, for example). There is also known a technique for chronologically analyzing changes in a subject's facial expression over a long period of time and estimating the emotions held during that period (see Patent Document 2, for example). There are also known techniques for identifying the factors that most affected changes in emotions (see Patent Documents 3 to 5, for example). There is also known a technique that compares the subject's usual facial expression with the current facial expression and issues an alert when the facial expression is dark (see Patent Document 6, for example). There is also known a technique for determining the degree of emotion of a subject by comparing the subject's normal (expressionless) facial expression with the current facial expression (see, for example, Patent Documents 7 to 9). . Techniques for analyzing the emotions of an organization and the atmosphere felt by individuals within a group are also known (see Patent Documents 10 and 11, for example).
特開２０１９－５８６２５号公報JP 2019-58625 A 特開２０１６－１４９０６３号公報JP 2016-149063 A 特開２０２０－８６５５９号公報JP 2020-86559 A 特開２０００－７６４２１号公報JP-A-2000-76421 特開２０１７－２０１４９９号公報JP 2017-201499 A 特開２０１８－１１２８３１号公報JP 2018-112831 A 特開２０１１－１５４６６５号公報JP 2011-154665 A 特開２０１２－８９４９号公報JP-A-2012-8949 特開２０１３－３００号公報Japanese Unexamined Patent Application Publication No. 2013-300 特開２０１１－１８６５２１号公報JP 2011-186521 A ＷＯ１５／１７４４２６号公報WO15/174426
　上述したすべての技術は、現実空間におけるコミュニケーションが主である状況におけるサブ的な機能にすぎない。即ち、昨今の業務のＤＸ（Ｄｉｇｉｔａｌ　Ｔｒａｎｓｆｏｒｍａｔｉｏｎ）化や、世界的な感染症の流行等を受け、業務や授業等のコミュニケーションがオンラインで行われることが主とされる状況に生まれたものではない。 All the technologies mentioned above are only sub-functions in situations where communication in the real world is the main thing. In other words, due to the recent DX (Digital Transformation) of work and the global epidemic of infectious diseases, it is not a situation where communication such as work and classes is mainly conducted online.
　本発明は、会議や講義等、オンラインコミュニケーションが主となる状況において、より効率的なコミュニケーションを行うために、これらのコミュニケーションを客観的に評価することを目的とする。 The purpose of the present invention is to objectively evaluate these communications in order to conduct more efficient communication in situations where online communication is the main focus, such as meetings and lectures.
　本発明によれば、複数のユーザでオンラインセッションが行われる環境においてオンラインセッション中にユーザが画面に表示されているか否かによらず前記ユーザを撮影することによって得られる動画像をもとに前記ユーザの反応を分析する動画像分析システムであって、複数の前記ユーザの夫々について、前記オンラインセッション中に前記ユーザを撮影することによって得られる動画像を取得する動画像取得部と、前記動画像取得部により取得された動画像に基づいて、前記ユーザについて生体反応の変化を解析する解析部と、前記解析部により得られた生体反応の解析結果が所定の条件を満たしているタイミングに基づいて前記ユーザとは異なる他のユーザが前記ユーザに対する言動を行っている動画像の区間を特定する特定部と、該区間に含まれる動画像に基づいて、前記他のユーザの前記言動に対する評価情報を生成する評価情報生成部と、を備える動画像分析システムが得られる。 According to the present invention, in an environment where an online session is held by a plurality of users, regardless of whether or not the user is displayed on the screen during the online session, the above-mentioned A moving image analysis system that analyzes reactions of users, comprising: a moving image acquiring unit that acquires a moving image obtained by photographing the user during the online session for each of the plurality of users; an analysis unit that analyzes changes in biological reactions of the user based on the moving image acquired by the acquisition unit; a specifying unit that specifies a section of a moving image in which another user different from the user is speaking and acting toward the user; and an evaluation information generation unit that generates a moving image analysis system.
　本開示によれば、ビデオセッションの動画像を分析評価することにより、特に内容に関する評価を客観的に行うことができる。 According to the present disclosure, by analyzing and evaluating moving images of a video session, it is possible to objectively evaluate especially the content.
　特に、本発明によれば、オンラインコミュニケーションが主となる状況において、より効率的なコミュニケーションを行うために、交わされたコミュニケーションを客観的に評価することができる。 In particular, according to the present invention, exchanged communication can be objectively evaluated in order to conduct more efficient communication in situations where online communication is the main activity.
本発明の実施の形態によるシステム全体図を示す図である。It is a figure which shows the whole system diagram by embodiment of this invention. 本発明の実施の形態による評価端末の機能ブロック図の一例である。1 is an example of a functional block diagram of an evaluation terminal according to an embodiment of the present invention; FIG. 本発明の実施の形態による評価端末の機能構成例１を示す図である。FIG. 3 is a diagram showing functional configuration example 1 of the evaluation terminal according to the embodiment of the present invention; 本発明の実施の形態による評価端末の機能構成例２を示す図である。FIG. 8 is a diagram showing functional configuration example 2 of the evaluation terminal according to the embodiment of the present invention; 本発明の実施の形態による評価端末の機能構成例３を示す図である。FIG. 10 is a diagram showing a functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention; 図６の機能構成例３による画面表示例である。7 is a screen display example according to the functional configuration example 3 of FIG. 6. FIG. 図６の機能構成例３による他の画面表示例である。FIG. 7 is another screen display example according to the functional configuration example 3 of FIG. 6. FIG. 本発明の実施の形態による評価端末の機能構成例３の他の構成を示す図である。FIG. 12 is a diagram showing another configuration of functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention; 本発明の実施の形態による評価端末の機能構成例３の他の構成を示す図である。FIG. 12 is a diagram showing another configuration of functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention; 本実施形態に係るシステムの機能構成の一例を示す図である。It is a figure showing an example of functional composition of a system concerning this embodiment. 本実施形態に係る評価対象区間の特定の例について説明するための図である。FIG. 4 is a diagram for explaining a specific example of an evaluation target section according to the embodiment; 本実施形態に係る評価出力部による出力態様の一例を示す図である。It is a figure which shows an example of the output mode by the evaluation output part which concerns on this embodiment.
　本開示の実施形態の内容を列記して説明する。本開示は、以下のような構成を備える。
（項目１）
　複数のユーザでオンラインセッションが行われる環境においてオンラインセッション中にユーザが画面に表示されているか否かによらず前記ユーザを撮影することによって得られる動画像をもとに前記ユーザの反応を分析する動画像分析システムであって、
　複数の前記ユーザの夫々について、前記オンラインセッション中に前記ユーザを撮影することによって得られる動画像を取得する動画像取得部と、
　前記動画像取得部により取得された動画像に基づいて、前記ユーザについて生体反応の変化を解析する解析部と、
　前記解析部により得られた生体反応の解析結果が所定の条件を満たしているタイミングに基づいて前記ユーザとは異なる他のユーザが前記ユーザに対する言動を行っている動画像の区間を特定する特定部と、
　該区間に含まれる動画像に基づいて、前記他のユーザの前記言動に対する評価情報を生成する評価情報生成部と、
　を備える動画像分析システム。
（項目２）
　項目１に記載の動画像分析システムであって、
　前記評価情報生成部は、前記他のユーザについての前記動画像に基づいて前記評価情報を生成する、動画像分析システム。
（項目３）
　項目１または２に記載の動画像分析システムであって、
　前記評価情報生成部は、前記ユーザの前記生体反応の解析結果に基づいて、前記ユーザの言動に対する前記評価情報を生成する、動画像分析システム。 The contents of the embodiments of the present disclosure are listed and described. The present disclosure has the following configurations.
(Item 1)
In an environment where an online session is held by a plurality of users, the reaction of the user is analyzed based on a moving image obtained by photographing the user regardless of whether or not the user is displayed on a screen during the online session. A moving image analysis system,
a moving image acquisition unit that acquires a moving image obtained by photographing the user during the online session for each of the plurality of users;
an analysis unit that analyzes changes in biological reactions of the user based on the moving image acquired by the moving image acquisition unit;
A specifying unit that specifies a section of a moving image in which a user different from the user is speaking and acting toward the user based on the timing at which the analysis result of the biological reaction obtained by the analysis unit satisfies a predetermined condition. When,
an evaluation information generation unit that generates evaluation information for the speech and behavior of the other user based on the moving image included in the section;
A moving image analysis system.
(Item 2)
The moving image analysis system according to item 1,
The moving image analysis system, wherein the evaluation information generation unit generates the evaluation information based on the moving image of the other user.
(Item 3)
The moving image analysis system according to item 1 or 2,
The moving image analysis system, wherein the evaluation information generation unit generates the evaluation information for the behavior of the user based on an analysis result of the biological reaction of the user.
　以下に添付図面を参照しながら、本開示の好適な実施の形態について詳細に説明する。なお、本明細書及び図面において、実質的に同一の機能構成を有する構成要素については、同一の符号を付することにより重複説明を省略する。 Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.
　＜基本機能＞
　本実施形態のビデオセッション評価システムは、複数人でビデオセッション（以下、一方向及び双方向含めてオンラインセッションという）が行われる環境において、当該複数人の中の解析対象者について他者とは異なる特異的な感情（自分または他人の言動に対して起こる気持ち。快・不快またはその程度など）を解析し評価するシステムである。オンラインセッションは、例えばオンライン会議、オンライン授業、オンラインチャットなどであり、複数の場所に設置された端末をインターネットなどの通信ネットワークを介してサーバに接続し、当該サーバを通じて複数の端末間で動画像をやり取りできるようにしたものである。オンラインセッションで扱う動画像には、端末を使用するユーザの顔画像や音声が含まれる。また、動画像には、複数のユーザが共有して閲覧する資料などの画像も含まれる。各端末の画面上に顔画像と資料画像とを切り替えて何れか一方のみを表示させたり、表示領域を分けて顔画像と資料画像とを同時に表示させたりすることが可能である。また、複数人のうち１人の画像を全画面表示させたり、一部または全部のユーザの画像を小画面に分割して表示させたりすることが可能である。端末を使用してオンラインセッションに参加する複数のユーザのうち、何れか１人または複数人を解析対象者として指定することが可能である。例えば、オンラインセッションの主導者、進行者または管理者（以下、まとめて主催者という）が何れかのユーザを解析対象者として指定する。オンラインセッションの主催者は、例えばオンライン授業の講師、オンライン会議の議長やファシリテータ、コーチングを目的としたセッションのコーチなどである。オンラインセッションの主催者は、オンラインセッションに参加する複数のユーザの中の一人であるのが普通であるが、オンラインセッションに参加しない別人であってもよい。なお、解析対象者を指定せず全ての参加者を解析対象としてもよい。また、オンラインセッションの主導者、進行者または管理者（以下、まとめて主催者という）が何れかのユーザを解析対象者として指定することも可能である。オンラインセッションの主催者は、例えばオンライン授業の講師、オンライン会議の議長やファシリテータ、コーチングを目的としたセッションのコーチなどである。オンラインセッションの主催者は、オンラインセッションに参加する複数のユーザの中の一人であるのが普通であるが、オンラインセッションに参加しない別人であってもよい。 <Basic functions>
In the video session evaluation system of the present embodiment, in an environment where a video session (hereinafter referred to as an online session including one-way and two-way sessions) is held by a plurality of people, the person to be analyzed among the plurality of people is different from the others. It is a system that analyzes and evaluates specific emotions (feelings that occur in response to one's own or others' words and actions. Pleasant/unpleasant, or their degree). Online sessions are, for example, online meetings, online classes, online chats, etc. Terminals installed in multiple locations are connected to a server via a communication network such as the Internet, and moving images are transmitted between multiple terminals through the server. It's made to be interactable. Moving images handled in online sessions include facial images and voices of users using terminals. Moving images also include images such as materials that are shared and viewed by a plurality of users. It is possible to switch between the face image and the document image on the screen of each terminal to display only one of them, or to divide the display area and display the face image and the document image at the same time. In addition, it is possible to display the image of one user out of a plurality of users on the full screen, or divide the images of some or all of the users into small screens and display them. It is possible to designate one or a plurality of users among a plurality of users participating in an online session using terminals as analysis subjects. For example, an online session leader, moderator, or manager (hereinafter collectively referred to as the organizer) designates any user as an analysis subject. Hosts of online sessions are, for example, instructors of online classes, chairpersons and facilitators of online meetings, coaches of sessions for coaching purposes, and the like. The host of an online session is typically one of multiple users participating in the online session, but may be another person who does not participate in the online session. It should be noted that all participants may be analyzed without designating the analysis subject. In addition, it is also possible for an online session leader, moderator, or administrator (hereinafter collectively referred to as the organizer) to designate any user as an analysis subject. Hosts of online sessions are, for example, instructors of online classes, chairpersons and facilitators of online meetings, coaches of sessions for coaching purposes, and the like. The host of an online session is typically one of multiple users participating in the online session, but may be another person who does not participate in the online session.
　本実施の形態によるビデオセッション評価システムは、複数の端末間においてビデオセッションセッションが確立された場合に、当該ビデオセッションから取得される少なくとも動画像を表示される。表示された動画像は、端末によって取得され、動画像内に含まれる少なくとも顔画像を所定のフレーム単位ごとに識別される。その後、識別された顔画像に関する評価値が算出される。当該評価値は必要に応じて共有される。特に、本実施の形態においては、取得した動画像は当該端末に保存され、端末上で分析評価され、その結果が当該端末のユーザに提供される。従って、例えば個人情報を含むビデオセッションや機密情報を含むビデオセッションであっても、その動画自体を外部の評価機関等に提供することなく分析評価できる。また、必要に応じて、当該評価結果（評価値）だけを外部端末に提供することによって、結果を可視化したり、クロス分析等行うことができる。 The video session evaluation system according to the present embodiment displays at least moving images obtained from a video session established between a plurality of terminals. The displayed moving image is acquired by the terminal, and at least a face image included in the moving image is identified for each predetermined frame unit. An evaluation value for the identified face image is then calculated. The evaluation value is shared as necessary. In particular, in this embodiment, the acquired moving image is stored in the terminal, analyzed and evaluated on the terminal, and the result is provided to the user of the terminal. Therefore, for example, even a video session containing personal information or a video session containing confidential information can be analyzed and evaluated without providing the moving image itself to an external evaluation agency or the like. In addition, by providing only the evaluation result (evaluation value) to the external terminal as necessary, the result can be visualized and cross-analysis can be performed.
　図１に示されるように、本実施の形態によるビデオセッション評価システムは、少なくともカメラ部及びマイク部等の入力部と、ディスプレイ等の表示部とスピーカー等の出力部とを有するユーザ端末１０、２０と、ユーザ端末１０、２０に双方向のビデオセッションを提供するビデオセッションサービス端末３０と、ビデオセッションに関する評価の一部を行う評価端末４０とを備えている。 As shown in FIG. 1, the video session evaluation system according to the present embodiment includes user terminals 10 and 20 each having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display, and an output unit such as a speaker. , a video session service terminal 30 for providing an interactive video session to the user terminals 10, 20, and an evaluation terminal 40 for performing part of the evaluation of the video session.
＜ハードウェア構成例＞
　以下に説明する各機能ブロック、機能単位、機能モジュールは、例えばコンピュータに備えられたハードウェア、ＤＳＰ（Digital Signal Processor）、ソフトウェアの何れによっても構成することが可能である。例えばソフトウェアによって構成する場合、実際にはコンピュータのＣＰＵ、ＲＡＭ、ＲＯＭなどを備えて構成され、ＲＡＭやＲＯＭ、ハードディスクまたは半導体メモリ等の記録媒体に記憶されたプログラムが動作することによって実現される。本明細書において説明するシステム及び端末による一連の処理は、ソフトウェア、ハードウェア、及びソフトウェアとハードウェアとの組合せのいずれを用いて実現されてもよい。本実施形態に係る情報共有支援装置１０の各機能を実現するためのコンピュータプログラムを作製し、ＰＣ等に実装することが可能である。また、このようなコンピュータプログラムが格納された、コンピュータで読み取り可能な記録媒体も提供することが可能である。記録媒体は、例えば、磁気ディスク、光ディスク、光磁気ディスク、フラッシュメモリ等である。また、上記のコンピュータプログラムは、記録媒体を用いずに、例えばネットワークを介して配信されてもよい。 <Hardware configuration example>
Each functional block, functional unit, and functional module described below can be configured by any of hardware, DSP (Digital Signal Processor), and software provided in a computer, for example. For example, when configured by software, it is actually configured with a computer CPU, RAM, ROM, etc., and is realized by running a program stored in a recording medium such as RAM, ROM, hard disk, or semiconductor memory. A series of processes by the systems and terminals described herein may be implemented using software, hardware, or a combination of software and hardware. It is possible to create a computer program for realizing each function of the information sharing support device 10 according to the present embodiment and implement it in a PC or the like. It is also possible to provide a computer-readable recording medium storing such a computer program. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Also, the above computer program may be distributed, for example, via a network without using a recording medium.
　本実施の形態による評価端末は、ビデオセッションサービス端末から動画像を取得し、当該動画像内に含まれる少なくとも顔画像を所定のフレーム単位ごとに識別すると共に、顔画像に関する評価値を算出する（詳しくは後述する）。 The evaluation terminal according to the present embodiment acquires a moving image from a video session service terminal, identifies at least a face image included in the moving image for each predetermined frame unit, and calculates an evaluation value for the face image ( will be described in detail later).
＜動画の取得方法＞
　図２に示されるように、ビデオセッションサービス端末が提供するビデオセッションサービス（以下、単に「本サービス」と言うことがある」）は、ユーザ端末１０、２０に対して双方向に画像および音声によって通信が可能となるものである。本サービスは、ユーザ端末のディスプレイに相手のユーザ端末のカメラ部で取得した動画像を表示し、相手のユーザ端末のマイク部で取得した音声をスピーカーから出力可能となっている。また、本サービスは双方の又はいずれかのユーザ端末によって、動画像及び音声（これらを合わせて「動画像等」という）を少なくともいずれかのユーザ端末上の記憶部に記録（レコーディング）することが可能に構成されている。記録された動画像情報Ｖｓ（以下「記録情報」という）は、記録を開始したユーザ端末にキャッシュされつついずれかのユーザ端末のローカルのみに記録されることとなる。ユーザは、必要があれば当該記録情報を本サービスの利用の範囲内で自分で視聴、他者に共有等行うこともできる。 <How to get videos>
As shown in FIG. 2, the video session service provided by the video session service terminal (hereinafter sometimes simply referred to as "this service") provides user terminals 10 and 20 with two-way images and voice. Communication is possible. In this service, a moving image captured by the camera of the other user's terminal is displayed on the display of the user's terminal, and audio captured by the microphone of the other's user's terminal can be output from the speaker. In addition, this service allows both or either of the user terminals to record moving images and sounds (collectively referred to as "moving images, etc.") in the storage unit of at least one of the user terminals. configured as possible. The recorded moving image information Vs (hereinafter referred to as “recorded information”) is cached in the user terminal that started recording and is locally recorded only in one of the user terminals. If necessary, the user can view the recorded information by himself or share it with others within the scope of using this service.
＜機能構成例１＞
　図４は、本実施形態による構成例を示すブロック図である。図４に示すように、本実施形態のビデオセッション評価システムは、ユーザ端末１０が有する機能構成として実現される。すなわち、ユーザ端末１０はその機能として、動画像取得部１１、生体反応解析部１２、特異判定部１３、関連事象特定部１４、クラスタリング部１５および解析結果通知部１６を備えている。 <Functional configuration example 1>
FIG. 4 is a block diagram showing a configuration example according to this embodiment. As shown in FIG. 4, the video session evaluation system of this embodiment is implemented as a functional configuration of the user terminal 10. FIG. That is, the user terminal 10 has, as its functions, a moving image acquisition unit 11, a biological reaction analysis unit 12, a peculiar determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16.
　動画像取得部１１は、オンラインセッション中に各端末が備えるカメラにより複数人（複数のユーザ）を撮影することによって得られる動画像を各端末から取得する。各端末から取得する動画像は、各端末の画面上に表示されるように設定されているものか否かは問わない。すなわち、動画像取得部１１は、各端末に表示中の動画像および非表示中の動画像を含めて、動画像を各端末から取得する。 The moving image acquisition unit 11 acquires from each terminal a moving image obtained by photographing a plurality of people (a plurality of users) with a camera provided in each terminal during an online session. It does not matter whether the moving image acquired from each terminal is set to be displayed on the screen of each terminal. That is, the moving image acquisition unit 11 acquires moving images from each terminal, including moving images being displayed and moving images not being displayed on each terminal.
　生体反応解析部１２は、動画像取得部１１により取得された動画像（画面上に表示中のものか否かは問わない）に基づいて、複数人のそれぞれについて生体反応の変化を解析する。本実施形態において生体反応解析部１２は、動画像取得部１１により取得された動画像を画像のセット（フレーム画像の集まり）と音声とに分離し、それぞれから生体反応の変化を解析する。 The biological reaction analysis unit 12 analyzes changes in the biological reaction of each of a plurality of people based on the moving images (whether or not they are being displayed on the screen) acquired by the moving image acquiring unit 11. In the present embodiment, the biological reaction analysis unit 12 separates the moving image acquired by the moving image acquisition unit 11 into a set of images (collection of frame images) and voice, and analyzes changes in the biological reaction from each.
　例えば、生体反応解析部１２は、動画像取得部１１により取得された動画像から分離したフレーム画像を用いてユーザの顔画像を解析することにより、表情、目線、脈拍、顔の動きの少なくとも１つに関する生体反応の変化を解析する。また、生体反応解析部１２は、動画像取得部１１により取得された動画像から分離した音声を解析することにより、ユーザの発言内容、声質の少なくとも１つに関する生体反応の変化を解析する。 For example, the biological reaction analysis unit 12 analyzes the user's facial image using a frame image separated from the moving image acquired by the moving image acquisition unit 11 to obtain at least one of facial expression, gaze, pulse, and facial movement. Analyze changes in biological reactions related to Further, the biological reaction analysis unit 12 analyzes the voice separated from the moving image acquired by the moving image acquisition unit 11 to analyze changes in the biological reaction related to at least one of the user's utterance content and voice quality.
　人は感情が変化すると、それが表情、目線、脈拍、顔の動き、発言内容、声質などの生体反応の変化となって現れる。本実施形態では、ユーザの生体反応の変化を解析することを通じて、ユーザの感情の変化を解析する。本実施形態において解析する感情は、一例として、快／不快の程度である。本実施形態において生体反応解析部１２は、生体反応の変化を所定の基準に従って数値化することにより、生体反応の変化の内容を反映させた生体反応指標値を算出する。 When a person's emotions change, it manifests as a change in biological reactions such as facial expressions, eye gaze, pulse, facial movements, content of remarks, and voice quality. In this embodiment, changes in the user's emotions are analyzed through analysis of changes in the user's biological reactions. The emotion analyzed in this embodiment is, for example, the degree of comfort/discomfort. In the present embodiment, the biological reaction analysis unit 12 calculates a biological reaction index value reflecting the change in biological reaction by quantifying the change in biological reaction according to a predetermined standard.
　表情の変化の解析は、例えば以下のようにして行う。すなわち、フレーム画像ごとに、フレーム画像の中から顔の領域を特定し、事前に機械学習させた画像解析モデルに従って特定した顔の表情を複数に分類する。そして、その分類結果に基づいて、連続するフレーム画像間でポジティブな表情変化が起きているか、ネガティブな表情変化が起きているか、およびどの程度の大きさの表情変化が起きているかを解析し、その解析結果に応じた表情変化指標値を出力する。 For example, the analysis of changes in facial expressions is performed as follows. That is, for each frame image, a facial region is identified from the frame image, and the identified facial expressions are classified into a plurality of types according to an image analysis model machine-learned in advance. Then, based on the classification results, it analyzes whether positive facial expression changes occur between consecutive frame images, whether negative facial expression changes occur, and to what extent the facial expression changes occur, A facial expression change index value corresponding to the analysis result is output.
　目線の変化の解析は、例えば以下のようにして行う。すなわち、フレーム画像ごとに、フレーム画像の中から目の領域を特定し、両目の向きを解析することにより、ユーザがどこを見ているかを解析する。例えば、表示中の話者の顔を見ているか、表示中の共有資料を見ているか、画面の外を見ているかなどを解析する。また、目線の動きが大きいか小さいか、動きの頻度が多いか少ないかなどを解析するようにしてもよい。目線の変化はユーザの集中度にも関連する。生体反応解析部１２は、目線の変化の解析結果に応じた目線変化指標値を出力する。 For example, the analysis of changes in line of sight is performed as follows. That is, for each frame image, the eye region is specified in the frame image, and the orientation of both eyes is analyzed to analyze where the user is looking. For example, it analyzes whether the user is looking at the face of the speaker being displayed, whether the user is looking at the shared material being displayed, or whether the user is looking outside the screen. Also, it may be analyzed whether the eye movement is large or small, or whether the movement is frequent or infrequent. A change in line of sight is also related to the user's degree of concentration. The biological reaction analysis unit 12 outputs a line-of-sight change index value according to the analysis result of the line-of-sight change.
　脈拍の変化の解析は、例えば以下のようにして行う。すなわち、フレーム画像ごとに、フレーム画像の中から顔の領域を特定する。そして、顔の色情報（ＲＧＢのＧ）の数値を捉える学習済みの画像解析モデルを用いて、顔表面のＧ色の変化を解析する。その結果を時間軸に合わせて並べることによって色情報の変化を表した波形を形成し、この波形から脈拍を特定する。人は緊張すると脈拍が速くなり、気持ちが落ち着くと脈拍が遅くなる。生体反応解析部１２は、脈拍の変化の解析結果に応じた脈拍変化指標値を出力する。 The analysis of pulse changes is performed, for example, as follows. That is, for each frame image, the face area is specified in the frame image. Then, using a trained image analysis model that captures numerical values of face color information (G of RGB), changes in the G color of the face surface are analyzed. By arranging the results along the time axis, a waveform representing changes in color information is formed, and the pulse is identified from this waveform. When a person is tense, the pulse speeds up, and when the person is calm, the pulse slows down. The biological reaction analysis unit 12 outputs a pulse change index value according to the analysis result of the pulse change.
　顔の動きの変化の解析は、例えば以下のようにして行う。すなわち、フレーム画像ごとに、フレーム画像の中から顔の領域を特定し、顔の向きを解析することにより、ユーザがどこを見ているかを解析する。例えば、表示中の話者の顔を見ているか、表示中の共有資料を見ているか、画面の外を見ているかなどを解析する。また、顔の動きが大きいか小さいか、動きの頻度が多いか少ないかなどを解析するようにしてもよい。顔の動きと目線の動きとを合わせて解析するようにしてもよい。例えば、表示中の話者の顔をまっすぐ見ているか、上目遣いまたは下目使いに見ているか、斜めから見ているかなどを解析するようにしてもよい。生体反応解析部１２は、顔の向きの変化の解析結果に応じた顔向き変化指標値を出力する。 For example, analysis of changes in facial movement is performed as follows. That is, for each frame image, the face area is specified in the frame image, and the direction of the face is analyzed to analyze where the user is looking. For example, it analyzes whether the user is looking at the face of the speaker being displayed, whether the user is looking at the shared material being displayed, or whether the user is looking outside the screen. Further, it may be analyzed whether the movement of the face is large or small, or whether the movement is frequent or infrequent. The movement of the face and the movement of the line of sight may be analyzed together. For example, it may be analyzed whether the face of the speaker being displayed is viewed straight, whether the face is viewed with upward or downward gaze, or whether the face is viewed obliquely. The biological reaction analysis unit 12 outputs a face orientation change index value according to the analysis result of the face orientation change.
　発言内容の解析は、例えば以下のようにして行う。すなわち、生体反応解析部１２は、指定した時間（例えば、３０～１５０秒程度の時間）の音声について公知の音声認識処理を行うことによって音声を文字列に変換し、当該文字列を形態素解析することにより、助詞、冠詞などの会話を表す上で不要なワードを取り除く。そして、残ったワードをベクトル化し、ポジティブな感情変化が起きているか、ネガティブな感情変化が起きているか、およびどの程度の大きさの感情変化が起きているかを解析し、その解析結果に応じた発言内容指標値を出力する。 　Analysis of the contents of the statement is performed, for example, as follows. That is, the biological reaction analysis unit 12 converts the voice into a character string by performing known voice recognition processing on the voice for a specified time (for example, about 30 to 150 seconds), and morphologically analyzes the character string. By doing so, words such as particles and articles that are unnecessary for expressing conversation are removed. Then, vectorize the remaining words, analyze whether a positive emotional change has occurred, whether a negative emotional change has occurred, and to what extent the emotional change has occurred. Outputs the statement content index value.
　声質の解析は、例えば以下のようにして行う。すなわち、生体反応解析部１２は、指定した時間（例えば、３０～１５０秒程度の時間）の音声について公知の音声解析処理を行うことによって音声の音響的特徴を特定する。そして、その音響的特徴に基づいて、ポジティブな声質変化が起きているか、ネガティブな声質変化が起きているか、およびどの程度の大きさの声質変化が起きているかを解析し、その解析結果に応じた声質変化指標値を出力する。 Voice quality analysis is performed, for example, as follows. That is, the biological reaction analysis unit 12 identifies the acoustic features of the voice by performing known voice analysis processing on the voice for a specified time (for example, about 30 to 150 seconds). Then, based on the acoustic features, it analyzes whether a positive change in voice quality has occurred, whether a negative change in voice quality has occurred, and to what extent the change in voice quality has occurred, and according to the analysis results, output the voice quality change index value.
　生体反応解析部１２は、以上のようにして算出した表情変化指標値、目線変化指標値、脈拍変化指標値、顔向き変化指標値、発言内容指標値、声質変化指標値の少なくとも１つを用いて生体反応指標値を算出する。例えば、表情変化指標値、目線変化指標値、脈拍変化指標値、顔向き変化指標値、発言内容指標値および声質変化指標値を重み付け計算することにより、生体反応指標値を算出する。 The biological reaction analysis unit 12 uses at least one of the facial expression change index value, eye line change index value, pulse change index value, face direction change index value, statement content index value, and voice quality change index value calculated as described above. to calculate the biological reaction index value. For example, the biological reaction index value is calculated by weighting the facial expression change index value, eye line change index value, pulse change index value, face direction change index value, statement content index value, and voice quality change index value.
　特異判定部１３は、解析対象者について解析された生体反応の変化が、解析対象者以外の他者について解析された生体反応の変化と比べて特異的か否かを判定する。本実施形態において、特異判定部１３は、生体反応解析部１２により複数のユーザのそれぞれについて算出された生体反応指標値に基づいて、解析対象者について解析された生体反応の変化が他者と比べて特異的か否かを判定する。 The peculiarity determination unit 13 determines whether or not the change in the analyzed biological reaction of the person to be analyzed is more specific than the change in the analyzed biological reaction of the person other than the person to be analyzed. In the present embodiment, the peculiarity determination unit 13 compares changes in the biological reaction of the person to be analyzed with those of others based on the biological reaction index values calculated for each of the plurality of users by the biological reaction analysis unit 12. is specific or not.
　例えば、特異判定部１３は、生体反応解析部１２により複数人のそれぞれについて算出された生体反応指標値の分散を算出し、解析対象者について算出された生体反応指標値と分散との対比により、解析対象者について解析された生体反応の変化が他者と比べて特異的か否かを判定する。 For example, the peculiar determination unit 13 calculates the variance of the biological reaction index values calculated for each of the plurality of persons by the biological reaction analysis unit 12, and compares the biological reaction index values calculated for the analysis subject with the variance, It is determined whether or not the change in the analyzed biological reaction of the person to be analyzed is specific compared to others.
　解析対象者について解析された生体反応の変化が他者と比べて特異的である場合として、次の３パターンが考えられる。１つ目は、他者については特に大きな生体反応の変化が起きていないが、解析対象者について比較的大きな生体反応の変化が起きた場合である。２つ目は、解析対象者については特に大きな生体反応の変化が起きていないが、他者について比較的大きな生体反応の変化が起きた場合である。３つ目は、解析対象者についても他者についても比較的大きな生体反応の変化が起きているが、変化の内容が解析対象者と他者とで異なる場合である。 The following three patterns are conceivable as cases where the changes in biological reactions analyzed for the subject of analysis are more specific than those of others. The first is a case where a relatively large change in biological reaction occurs in the subject of analysis, although no particularly large change in biological reaction has occurred in the other person. The second is a case where a particularly large change in biological reaction has not occurred in the subject of analysis, but a relatively large change in biological reaction has occurred in the other person. The third is a case where a relatively large change in biological reaction occurs in both the subject of analysis and the other person, but the content of the change differs between the subject of analysis and the other person.
　関連事象特定部１４は、特異判定部１３により特異的であると判定された生体反応の変化が起きたときに解析対象者、他者および環境の少なくとも１つに関して発生している事象を特定する。例えば、関連事象特定部１４は、解析対象者について特異的な生体反応の変化が起きたときにおける解析対象者自身の言動を動画像から特定する。また、関連事象特定部１４は、解析対象者について特異的な生体反応の変化が起きたときにおける他者の言動を動画像から特定する。また、関連事象特定部１４は、解析対象者について特異的な生体反応の変化が起きたときにおける環境を動画像から特定する。環境は、例えば画面に表示中の共有資料、解析対象者の背景に写っているものなどである。 The related event identification unit 14 identifies an event occurring in relation to at least one of the person to be analyzed, the other person, and the environment when the change in the biological reaction determined to be peculiar by the peculiarity determination unit 13 occurs. . For example, the related event identification unit 14 identifies from the moving image the speech and behavior of the person to be analyzed when a specific change in biological reaction occurs in the person to be analyzed. In addition, the related event identifying unit 14 identifies, from the moving image, the speech and behavior of the other person when a specific change in the biological reaction of the person to be analyzed occurs. In addition, the related event identification unit 14 identifies from the moving image the environment in which a specific change in the biological reaction of the person to be analyzed occurs. The environment is, for example, the shared material being displayed on the screen, the background image of the person to be analyzed, and the like.
　クラスタリング部１５は、特異判定部１３により特異的であると判定された生体反応の変化（例えば、目線、脈拍、顔の動き、発言内容、声質のうち１つまたは複数の組み合わせ）と、当該特異的な生体反応の変化が起きたときに発生している事象（関連事象特定部１４により特定された事象）との相関の程度を解析し、相関が一定レベル以上であると判定された場合に、その相関の解析結果に基づいて解析対象者または事象をクラスタリングする。 The clustering unit 15 clusters the change in the biological reaction determined to be specific by the peculiarity determination unit 13 (for example, one or a combination of eye gaze, pulse, facial movement, statement content, and voice quality), and the peculiarity Analyzing the degree of correlation with an event (event identified by the related event identification unit 14) that occurs when a change in biological reaction occurs, and if it is determined that the correlation is at a certain level or more , to cluster the subjects or events based on the correlation analysis results.
　例えば、特異的な生体反応の変化がネガティブな感情変化に相当するものであり、当該特異的な生体反応の変化が起きたときに発生している事象もネガティブな事象である場合には一定レベル以上の相関が検出される。クラスタリング部１５は、その事象の内容やネガティブな度合い、相関の大きさなどに応じて、あらかじめセグメント化した複数の分類の何れかに解析対象者または事象をクラスタリングする。 For example, if a change in a specific biological reaction corresponds to a negative emotional change, and the event occurring when the specific change in biological reaction occurs is also a negative event, a certain level The above correlation is detected. The clustering unit 15 clusters the person to be analyzed or the event into one of a plurality of pre-segmented categories according to the content of the event, the degree of negativity, the magnitude of the correlation, and the like.
　同様に、特異的な生体反応の変化がポジティブな感情変化に相当するものであり、当該特異的な生体反応の変化が起きたときに発生している事象もポジティブな事象である場合には一定レベル以上の相関が検出される。クラスタリング部１５は、その事象の内容やポジティブな度合い、相関の大きさなどに応じて、あらかじめセグメント化した複数の分類の何れかに解析対象者または事象をクラスタリングする。 Similarly, if a specific change in biological reaction corresponds to a positive emotional change and the event occurring when the specific change in biological reaction occurs is also a positive event, Level or higher correlations are detected. The clustering unit 15 clusters the person to be analyzed or the event into one of a plurality of pre-segmented classifications according to the content of the event, the degree of positivity, the degree of correlation, and the like.
　解析結果通知部１６は、特異判定部１３により特異的であると判定された生体反応の変化、関連事象特定部１４により特定された事象、およびクラスタリング部１５によりクラスタリングされた分類の少なくとも１つを、解析対象者の指定者（解析対象者またはオンラインセッションの主催者）に通知する。 The analysis result notification unit 16 reports at least one of the changes in the biological reaction determined to be specific by the peculiar determination unit 13, the event identified by the related event identification unit 14, and the classification clustered by the clustering unit 15. , to notify the designator of the subject of analysis (the subject of analysis or the organizer of the online session).
　例えば、解析結果通知部１６は、解析対象者について他者とは異なる特異的な生体反応の変化が起きたとき（上述した３パターンの何れか。以下同様）に発生している事象として解析対象者自身の言動を解析対象者自身に通知する。これにより、解析対象者は、自分がある言動を行ったときに他者とは違う感情を持っていることを把握することができる。このとき、解析対象者について特定された特異的な生体反応の変化も併せて解析対象者に通知するようにしてもよい。さらに、対比される他者の生体反応の変化を更に解析対象者に通知するようにしてもよい。 For example, the analysis result notification unit 16 recognizes that when a change in a specific biological reaction that is different from that of the other person occurs in the person to be analyzed (one of the three patterns described above; the same applies hereinafter), the analysis target is Notifies the person to be analyzed of his/her own behavior. This allows the person to be analyzed to understand that he/she has a different feeling from others when he or she performs a certain behavior. At this time, the person to be analyzed may also be notified of the change in the specific biological reaction identified for the person to be analyzed. Furthermore, the person to be analyzed may be further notified of the change in the biological reaction of the other person to be compared.
　例えば、解析対象者が普段どおりの感情で特に意識せずに行った言動、または、解析対象者がある感情を伴って特に意識して行った言動に対して他者が受けた感情と、言動の際に解析対象者自身が抱いていた感情とが相違している場合に、そのときの解析対象者自身の言動が解析対象者に通知される。これにより、自分の意識に反して他者の受けが良い言動や他者の受けが良くない言動などを発見することも可能である。 For example, the words and deeds of the person to be analyzed performed without being particularly conscious of their usual emotions, or the words and deeds of the person to be analyzed consciously accompanied by certain emotions, and the emotions and behaviors that others received When the emotion held by the person to be analyzed is different from the feeling held by the person to be analyzed at the time, the person to be analyzed is notified of the speech and behavior of the person to be analyzed at that time. As a result, it is possible to discover behaviors that are well received by others or behaviors that are not well received by others, contrary to one's own consciousness.
　また、解析結果通知部１６は、解析対象者について他者とは異なる特異的な生体反応の変化が起きたときに発生している事象を、特異的な生体反応の変化と共にオンラインセッションの主催者に通知する。これにより、オンラインセッションの主催者は、指定した解析対象者に特有の現象として、どのような事象がどのような感情の変化に影響を与えているのかを知ることができる。そして、その把握した内容に応じて適切な処置を解析対象者に対して行うことが可能となる。 In addition, the analysis result notification unit 16 notifies the organizer of the online session of the event occurring when the person to be analyzed undergoes a specific change in biological reaction that is different from that of the other person, together with the change in the specific biological reaction. to notify. As a result, the organizer of the online session can know what kind of event affects what kind of emotional change as a phenomenon specific to the specified analysis subject. Then, it becomes possible to perform appropriate treatment on the person to be analyzed according to the grasped contents.
　また、解析結果通知部１６は、解析対象者について他者とは異なる特異的な生体反応の変化が起きたときに発生している事象または解析対象者のクラスタリング結果をオンラインセッションの主催者に通知する。これにより、オンラインセッションの主催者は、指定した解析対象者がどの分類にクラスタリングされたかによって、解析対象者に特有の行動の傾向を把握したり、今後起こり得る行動や状態などを予測したりすることができる。そして、それに対して適切な処置を解析対象者に対して行うことが可能となる。 In addition, the analysis result notification unit 16 notifies the organizer of the online session of the event occurring when a specific change in biological reaction occurs in the analysis subject, which is different from that of others, or the clustering result of the analysis subject. do. As a result, online session organizers can grasp behavioral tendencies peculiar to analysis subjects and predict possible future behaviors and situations, depending on which classification the specified analysis subjects have been clustered into. be able to. Then, it becomes possible to take appropriate measures for the person to be analyzed.
　なお、上記実施形態では、生体反応の変化を所定の基準に従って数値化することによって生体反応指標値を算出し、複数人のそれぞれについて算出された生体反応指標値に基づいて、解析対象者について解析された生体反応の変化が他者と比べて特異的か否かを判定する例について説明したが、この例に限定されない。例えば、以下のようにしてもよい。 In the above embodiment, the biological reaction index value is calculated by quantifying the change in biological reaction according to a predetermined standard, and the analysis subject is analyzed based on the biological reaction index value calculated for each of the plurality of people. Although the example of determining whether the change in the biological reaction received is specific compared to others has been described, the present invention is not limited to this example. For example, it may be as follows.
　すなわち、生体反応解析部１２は、複数人のそれぞれについて目線の動きを解析して目線の方向を示すヒートマップを生成する。特異判定部１３は、生体反応解析部１２により解析対象者について生成されたヒートマップと他者について生成されたヒートマップとの対比により、解析対象者について解析された生体反応の変化が、他者について解析された生体反応の変化と比べて特異的か否かを判定する。 That is, the biological reaction analysis unit 12 analyzes the movement of the line of sight for each of a plurality of people and generates a heat map indicating the direction of the line of sight. The peculiar determination unit 13 compares the heat map generated for the person to be analyzed by the biological reaction analysis unit 12 with the heat map generated for the other person, so that the change in the biological reaction analyzed for the person to be analyzed It is determined whether it is specific compared with the change in biological response analyzed for.
　このように、本実施の形態においては、ビデオセッションの動画像をユーザ端末１０のローカルストレージに保存し、ユーザ端末１０上で上述した分析を行うこととしている。ユーザ端末１０のマシンスペックに依存する可能性があるとはいえ、動画像の情報を外部に提供することなく分析することが可能となる。 Thus, in the present embodiment, moving images of a video session are stored in the local storage of the user terminal 10, and the above analysis is performed on the user terminal 10. Although it may depend on the machine specs of the user terminal 10, it is possible to analyze the moving image information without providing it to the outside.
＜機能構成例２＞
　図５に示すように、本実施形態のビデオセッション評価システムは、機能構成として、動画像取得部１１、生体反応解析部１２および反応情報提示部１３ａを備えていてもよい。 <Functional configuration example 2>
As shown in FIG. 5, the video session evaluation system of this embodiment may include a moving image acquisition unit 11, a biological reaction analysis unit 12, and a reaction information presentation unit 13a as functional configurations.
　反応情報提示部１３ａは、画面に表示されていない参加者を含めて生体反応解析部１２ａにより解析された生体反応の変化を示す情報を提示する。例えば、反応情報提示部１３ａは、生体反応の変化を示す情報をオンラインセッションの主導者、進行者または管理者（以下、まとめて主催者という）に提示する。オンラインセッションの主催者は、例えばオンライン授業の講師、オンライン会議の議長やファシリテータ、コーチングを目的としたセッションのコーチなどである。オンラインセッションの主催者は、オンラインセッションに参加する複数のユーザの中の一人であるのが普通であるが、オンラインセッションに参加しない別人であってもよい。 The reaction information presentation unit 13a presents information indicating changes in biological reactions analyzed by the biological reaction analysis unit 12a, including participants not displayed on the screen. For example, the reaction information presenting unit 13a presents information indicating changes in biological reactions to an online session leader, moderator, or administrator (hereinafter collectively referred to as the organizer). Hosts of online sessions are, for example, instructors of online classes, chairpersons and facilitators of online meetings, coaches of sessions for coaching purposes, and the like. An online session host is typically one of the users participating in the online session, but may be another person who does not participate in the online session.
　このようにすることにより、オンラインセッションの主催者は、複数人でオンラインセッションが行われる環境において、画面に表示されていない参加者の様子も把握することができる。 By doing so, the organizer of the online session can also grasp the state of the participants who are not displayed on the screen in an environment where the online session is held by multiple people.
＜機能構成例３＞
　図６は、本実施形態による構成例を示すブロック図である。図６に示すように、本実施形態のビデオセッション評価システムは、機能構成として、上述した実施の形態１と類似する機能については同一つの参照符号を付して説明を省略することがある。 <Functional configuration example 3>
FIG. 6 is a block diagram showing a configuration example according to this embodiment. As shown in FIG. 6, in the video session evaluation system of the present embodiment, functions similar to those of the above-described first embodiment are given the same reference numerals, and explanations thereof may be omitted.
　本実施の形態によるシステムは、ビデオセッションの映像を取得するカメラ部及び音声を取得するマイク部と、動画像を分析及び評価する解析部、取得した動画像を評価することによって得られた情報に基づいて表示オブジェクト（後述する）を生成するオブジェクト生成部、前記ビデオセッション実行中にビデオセッションの動画像と表示オブジェクトの両方を表示する表示部と、を備えている。 The system according to this embodiment includes a camera unit that acquires images of a video session, a microphone unit that acquires audio, an analysis unit that analyzes and evaluates moving images, and information obtained by evaluating the acquired moving images. an object generator for generating a display object (described below) based on the display; and a display for displaying both the moving image of the video session and the display object during execution of the video session.
　解析部は、上述した説明と同様に、動画像取得部１１、生体反応解析部１２、特異判定部１３、関連事象特定部１４、クラスタリング部１５および解析結果通知部１６を備えている。各要素の機能については上述したとおりである。 The analysis unit includes the moving image acquisition unit 11, the biological reaction analysis unit 12, the peculiar determination unit 13, the related event identification unit 14, the clustering unit 15, and the analysis result notification unit 16, as described above. The function of each element is as described above.
　図７に示されるように、オブジェクト生成部は、解析部によってビデオセッションから取得される動画像を解析した結果に基づいて、必要に応じて、当該認識した顔の部分を示すオブジェクト５０と、上述した分析・評価した内容を示す情報１００を当該動画像に重畳して表示する。当該オブジェクト５０は、複数人の顔が動画像内に移っている場合には、複数人全員の顔を識別し、表示することとしてもよい。 As shown in FIG. 7, the object generation unit generates an object 50 representing the recognized face part and the above-mentioned Information 100 indicating the content of the analysis/evaluation performed is superimposed on the moving image and displayed. The object 50 may identify and display all faces of a plurality of persons when the faces of the plurality of persons are moved in the moving image.
　また、オブジェクト５０は、例えば、相手側の端末において、ビデオセッションのカメラ機能を停止している場合（即ち、物理的にカメラを覆う等ではなく、ビデオセッションのアプリケーション内においてソフトウェア的に停止している場合）であっても、相手側のカメラで相手の顔を認識していた場合には、相手の顔が位置している部分にオブジェクト５０やオブジェクト１００を表示することとしてもよい。これにより、カメラ機能がオフになっていたとしても、相手側が端末の前にいることがお互い確認することが可能となる。この場合、例えば、ビデオセッションのアプリケーションにおいては、カメラから取得した情報を非表示にする一方、解析部によって認識された顔に対応するオブジェクト５０やオブジェクト１００のみを表示することとしてもよい。また、ビデオセッションから取得される映像情報と、解析部によって認識され得られた情報とを異なる表示レイヤーに分け、前者の情報に関するレイヤーを非表示にすることとしてもよい。 In addition, the object 50 is, for example, when the camera function of the video session is stopped at the other party's terminal (that is, it is stopped by software within the application of the video session instead of physically covering the camera). If the other party's face is recognized by the other party's camera, the object 50 or the object 100 may be displayed in the part where the other party's face is located. This makes it possible for both parties to confirm that the other party is in front of the terminal even if the camera function is turned off. In this case, for example, in a video session application, the information obtained from the camera may be hidden while only the object 50 or object 100 corresponding to the face recognized by the analysis unit is displayed. Also, the video information acquired from the video session and the information recognized by the analysis unit may be divided into different display layers, and the layer relating to the former information may be hidden.
　オブジェクト５０やオブジェクト１００は、複数の動画像を表示する領域がある場合には、すべての領域又は一部の領域のみに表示することとしてもよい。例えば、図８に示されるように、ゲスト側の動画像のみに表示することとしてもよい。 When there are multiple moving image display areas, the objects 50 and 100 may be displayed in all areas or only in some areas. For example, as shown in FIG. 8, it may be displayed only on the moving image on the guest side.
　以上説明した基本構成例１乃至基本構成例３において説明した発明の実施の形態は、単独の装置として実現されてもよく、一部または全部がネットワークで接続された複数の装置（例えばクラウドサーバ）等により実現されてもよい。例えば、各端末１０の制御部１１０およびストレージ１３０は、互いにネットワークで接続された異なるサーバにより実現されてもよい。即ち、本システムは、ユーザ端末１０、２０と、ユーザ端末１０、２０に双方向のビデオセッションを提供するビデオセッションサービス端末３０と、ビデオセッションに関する評価を行う評価端末４０とを含んでいるところ、以下のような構成のバリエーション組み合わせが考えられる。
（１）すべてをユーザ端末のみで処理
　図８に示されるように、解析部による処理をビデオセッションを行っている端末で行うことにより、（一定の処理能力は必要なものの）ビデオセッションを行っている時間と同時に（リアルタイムに）分析・評価結果を得ることができる。
（２）ユーザ端末と評価端末とで処理
　図９に示されるように、ネットワーク等で接続された評価端末に解析部を備えさせることとしてもよい。この場合、ユーザ端末で取得された動画像は、ビデオセッションと同時に又は事後的に評価端末に共有され、評価端末における解析部によって分析・評価されたのちに、オブジェクト５０及びオブジェクト１００の情報がユーザ端末に動画像データと共に又は別に（即ち、少なくとも解析データを含む情報が）共有され表示部に表示される。 The embodiments of the invention described in Basic Configuration Example 1 to Basic Configuration Example 3 described above may be implemented as a single device, or a plurality of devices (for example, cloud servers) partially or entirely connected by a network. and the like. For example, the control unit 110 and the storage 130 of each terminal 10 may be realized by different servers connected to each other via a network. That is, the system includes user terminals 10, 20, a video session service terminal 30 for providing an interactive video session to the user terminals 10, 20, and an evaluation terminal 40 for evaluating the video session, Variation combinations of the following configurations are conceivable.
(1) Processing everything only on the user terminal As shown in FIG. 8, by performing the processing by the analysis unit on the terminal that is performing the video session (although a certain processing capacity is required), the video session can be performed. Analysis/evaluation results can be obtained at the same time (in real time) as you are.
(2) Processing by User Terminal and Evaluation Terminal As shown in FIG. 9, an analysis unit may be provided in an evaluation terminal connected via a network or the like. In this case, the moving images acquired by the user terminal are shared with the evaluation terminal at the same time as or after the video session, and are analyzed and evaluated by the analysis unit in the evaluation terminal. Together with or separately from the moving image data (that is, information including at least analysis data) is shared with the terminal and displayed on the display unit.
　上述した機能構成例１乃至機能構成例３の各構成又はそれらの組み合わせを用いて、以下のシステムが実現する。 The following system is realized using each configuration of functional configuration example 1 to functional configuration example 3 or a combination thereof.
＜実施の形態＞
　本開示の一実施形態による動画像分析システム（以下、単に「システム」という）は、複数人の参加者でオンラインセッションが行われる環境において、当該参加者の全員又は特定の参加者のみを撮影することによって得られる動画像をもとに参加者の反応を解析・分析するものである。分析は、オンラインセッション中に参加者が画面に表示されているか否かによらず行われるものとしてもよい。例えば、本実施形態に係るシステム（解析部）は、動画像を分析することにより、ユーザ同士のコミュニケーションの量や頻度、そのときの感情といった内容を統計的に分析して出力する。また、上述した解析部は、ユーザの感情だけではなく、上記の動画像に基づいて発言の内容を解析する。かかる発言の内容の解析は、例えば動画像に対する公知の音声解析技術や自然言語処理技術により行われ得る。 <Embodiment>
A moving image analysis system (hereinafter simply referred to as "system") according to an embodiment of the present disclosure shoots all participants or only a specific participant in an environment where an online session is held with a plurality of participants. Participants' reactions are analyzed based on the moving images obtained by this process. The analysis may occur whether or not participants are on screen during the online session. For example, the system (analysis unit) according to the present embodiment statistically analyzes and outputs the content such as the amount and frequency of communication between users and their emotions at that time by analyzing moving images. Also, the analysis unit described above analyzes the content of the utterance based on not only the user's emotion but also the moving image described above. Such analysis of the content of the utterance can be performed, for example, by a known speech analysis technique or natural language processing technique for moving images.
　かかる解析の対象としては、例えば、一のユーザの言動であり得る。一のユーザの言動により、それを受けた他のユーザが反応し、かかる反応が解析され得る。この反応は、オンラインセッションにおいては見逃されやすく、フィードバックが容易ではない。 The target of such analysis may be, for example, the behavior of a single user. One user's words and actions cause other users to react, and such reactions can be analyzed. This reaction is easy to miss in online sessions and not easy to feedback.
　そこで、本実施形態では、自らの言動についてより的確にフィードバックを受けることが可能となるシステムを実現する。 Therefore, in the present embodiment, a system is realized that enables users to receive more accurate feedback on their own behavior.
　図１０は、本実施形態に係るシステムの機能構成の一例を示す図である。図１０に示すシステムは、解析結果ＤＢ２１と、特定部２２と、評価情報生成部２３と、出力制御部２４とを備える。解析結果ＤＢ２１は、上述する記憶媒体等により実現され得る。また、特定部２２と、評価情報生成部２３と、出力制御部２４とは、例えばユーザ端末１０、２０や、評価端末４０などに設けられる記憶媒体等に記憶されるプログラムをメモリ等に読み込みＣＰＵ等のプロセッサが実行することにより実現され得る。 FIG. 10 is a diagram showing an example of the functional configuration of the system according to this embodiment. The system shown in FIG. 10 includes an analysis result DB 21, a specifying unit 22, an evaluation information generating unit 23, and an output control unit 24. The analysis result DB 21 can be realized by the above-described storage medium or the like. The identification unit 22, the evaluation information generation unit 23, and the output control unit 24 read a program stored in a storage medium or the like provided in, for example, the user terminals 10, 20 or the evaluation terminal 40 into a memory or the like and It can be realized by execution by a processor such as.
　解析結果ＤＢ２１は、例えば上述する各種機能部により得られる解析結果のデータを格納するデータベースである。解析結果のデータは、例えば上述したユーザの生体反応の変化から得られる解析結果のデータであってもよいし、ユーザの発言に関する解析結果のデータであってもよい。これらの解析結果は、オンラインセッションの動画像の解析により得られる。また、これらの解析結果のデータには、ユーザを特定するユーザＩＤや、ユーザに起因する動画像上の動きの解析の結果得られた解析情報、またはユーザのユーザ端末に対する入力により生成される入力情報等が、ユーザ情報として含まれてよい。 The analysis result DB 21 is, for example, a database that stores analysis result data obtained by the various functional units described above. The analysis result data may be, for example, the analysis result data obtained from the change in the user's biological reaction described above, or the analysis result data relating to the user's utterance. These analysis results are obtained by analysis of moving images of online sessions. In addition, these analysis result data include a user ID that identifies the user, analysis information obtained as a result of analysis of movement on a moving image caused by the user, or an input generated by the user's input to the user terminal Information and the like may be included as user information.
　特定部２２は、解析部により得られた生体反応の解析結果が所定の条件を満たしているタイミングに基づいて、解析対象である一のユーザとは異なる他のユーザが、一のユーザに対する言動を行っている動画像の区間を特定する機能を有する。具体的には、特定部２２は、解析結果ＤＢ２１からオンラインセッションの解析結果を取得し、一のユーザの生体反応の解析結果が所定の基準を超えたタイミングにおいて、そのような生体反応の変化を生じさせたと思われる他のユーザの言動を行っている動画像の区間を特定する。なお、特定部２２は、生体反応の解析結果が所定の基準を満たしているタイミングだけではなく、所定の基準を超える（または下回る）ような生体反応の変化が生じているタイミングも、区間の特定のための情報として用いてもよい。また、かかる動画像の区間は、例えば、生体反応の解析結果が所定の基準を満たしたタイミングと同じ区間であるか、それよりも前の区間であり得る。区間の開始または終了のタイミングは、例えば生体反応の変化が生じたタイミングまたは生体反応タイミング等に応じて定められ得る。かかる区間は、他のユーザの言動に対応する時系列の区間の前および／または後ろの言動についての解析結果のデータから得られるコンテクスト情報に基づいて、特定されてもよい。なお、かかるコンテクスト情報は、前後の区間に含まれる言動に関する情報の解析結果だけではなく、例えば他のユーザの生体反応の変化の解析結果を用いてもよい。所定の基準とは、例えば、ユーザのポジティブな生体反応に対応する言動を評価したい場合は、ポジティブな反応に関する解析結果のための基準であり得る。所定の条件は、一または複数の生体反応の種類（例えば、ポジティブ、ネガティブ、楽しさ、悲しさ、怒り等）に基づく条件であり得る。 Based on the timing at which the analysis result of the biological reaction obtained by the analysis unit satisfies a predetermined condition, the identification unit 22 allows a user different from the one user to be analyzed to perform behavior toward the one user. It has a function to identify the section of the moving image being performed. Specifically, the identification unit 22 acquires the analysis result of the online session from the analysis result DB 21, and detects such a change in the biological reaction at the timing when the analysis result of the biological reaction of one user exceeds a predetermined standard. Identify a segment of the moving image in which the other user's speech and behavior that is thought to have occurred is performed. Note that the specifying unit 22 specifies not only the timing at which the analysis result of the biological reaction satisfies the predetermined standard, but also the timing at which the change in the biological reaction exceeds (or falls below) the predetermined standard. may be used as information for Also, such a section of the moving image may be, for example, the same section as the timing at which the biological reaction analysis result satisfies a predetermined criterion, or a section earlier than that. The timing of the start or end of the interval can be determined according to, for example, the timing at which a change in biological reaction occurs, the timing of the biological reaction, or the like. Such an interval may be specified based on context information obtained from analysis result data of speech and behavior before and/or after a time-series interval corresponding to the speech and behavior of another user. It should be noted that the context information may be not only the analysis result of the information on the speech and behavior contained in the preceding and succeeding sections, but also the analysis result of changes in other users' biological reactions, for example. The predetermined criterion may be, for example, a criterion for analysis results regarding positive reactions when it is desired to evaluate speech and behavior corresponding to positive biological reactions of the user. The predetermined condition can be a condition based on one or more types of biological response (eg, positive, negative, joyful, sad, angry, etc.).
　評価情報生成部２３は、特定された発話区間に含まれる動画像に基づいて、他のユーザの言動に対する評価情報を生成する機能を有する。具体的には、評価情報生成部２３は、上記区間に含まれる動画像から他のユーザの言動に対して、評価情報を生成する。ユーザの言動は、例えば、動画像から得られる音声情報に基づく発話内容や、他のユーザの動画像の解析結果から得られる生体反応に関する内容を含みうる。発話内容については、例えば公知の音声解析技術により音声情報に対して行われる解析により取得され得る。また、評価情報生成部２３は、言動を受けた一のユーザの動画像の解析結果に基づいて、評価情報を生成してもよい。評価情報は、例えば、一のユーザの生体反応がどのような言動を受けたものであるかを特定する情報や、かかる言動がどのような属性に属するものであるか（例えばポジティブ、ネガティブ、楽しさ、悲しさ、怒り等）の情報や、該言動の適正さ等のフィードバックに関する情報を含みうる。このような評価情報を生成することで、他のユーザが一のユーザに与えた言動の影響を把握することができ、話し相手に対する影響をより適切に認知することができる。 The evaluation information generation unit 23 has a function of generating evaluation information on the speech and behavior of other users based on the moving images included in the specified speech period. Specifically, the evaluation information generation unit 23 generates evaluation information for the speech and behavior of other users from the moving image included in the section. The user's behavior may include, for example, speech content based on audio information obtained from moving images, and content related to biological reactions obtained from analysis results of moving images of other users. The contents of the utterance can be obtained, for example, by analysis performed on speech information by a known speech analysis technique. Also, the evaluation information generation unit 23 may generate evaluation information based on the analysis result of the moving image of the one user who received the speech and behavior. The evaluation information is, for example, information specifying what kind of behavior the biological reaction of one user has received, or what attribute the behavior belongs to (for example, positive, negative, fun, etc.). (e.g. sadness, anger, etc.) and information on feedback such as appropriateness of said behavior. By generating such evaluation information, it is possible to comprehend the influence of speech and behavior given to one user by other users, and it is possible to more appropriately recognize the influence on the interlocutor.
　図１１は、本実施形態に係る評価対象区間の特定の例について説明するための図である。図１１に示すグラフ１０００は、オンラインセッションにおいてユーザＡとユーザＢが会話をしているときの、ユーザＢの生体反応の解析結果を示したグラフ（反応グラフ）と、ユーザＡとユーザＢのそれぞれの発話区間を示している。このとき、まず特定部２２は、例えばユーザＡが発話しているときにユーザＢの反応グラフの値が所定の基準を満たしているか、または所定の基準を満たす（下回る）ような変化を見せている区間１００１、１００２を特定する。次に、特定部２２は、区間１００１、１００２に対応するユーザＡの発話区間１００５、１００６を特定する。評価情報生成部２３は、かかる発話区間１００５、１００６におけるユーザＡの言動に基づいて評価情報を生成し得る。 FIG. 11 is a diagram for explaining a specific example of the evaluation target section according to this embodiment. A graph 1000 shown in FIG. 11 is a graph (reaction graph) showing an analysis result of the biological reaction of user B when user A and user B are having a conversation in an online session, and a graph (reaction graph) of user A and user B. shows the utterance interval of At this time, the identification unit 22 first determines whether the value of the reaction graph of user B satisfies a predetermined standard when user A is speaking, or shows a change that satisfies (falls below) a predetermined standard. The sections 1001 and 1002 in which the Next, the identifying unit 22 identifies utterance sections 1005 and 1006 of user A corresponding to sections 1001 and 1002 . The evaluation information generator 23 can generate evaluation information based on the behavior of the user A in the utterance sections 1005 and 1006 .
　なお、発話区間１００６は、発話区間１００５と異なり、ユーザＢの反応グラフの値が所定の基準を満たしている区間１００２の開始タイミングよりも前の、生体反応の解析結果の値が低い区間も含めた区間１００４に対応する発話区間も含めている。これにより、どの言動がきっかけでユーザＢの反応が変化したかをより詳細に知ることができる。 Note that, unlike the utterance segment 1005, the utterance segment 1006 includes a segment in which the value of the biological reaction analysis result is low before the start timing of the segment 1002 in which the value of the reaction graph of the user B satisfies a predetermined criterion. A speech segment corresponding to segment 1004 is also included. As a result, it is possible to know in more detail which speech or behavior triggered a change in the reaction of the user B.
　なお、特定部２２は、ユーザＢの発話区間においても、ユーザＢの反応グラフの値が所定の基準を満たしている区間１００３に対応するユーザＢの発話区間１００７を特定し、評価情報生成部２３が、かかる発話区間１００７におけるユーザＢの言動に基づいて評価情報を生成してもよい。これにより、ユーザＢがどのような言動を行っているときにユーザＢの心理にどのような影響を与えているかを把握することができる。 Note that the specifying unit 22 also specifies the user B's utterance segment 1007 corresponding to the segment 1003 in which the user B's reaction graph value satisfies a predetermined criterion, and the evaluation information generating unit 23 However, the evaluation information may be generated based on the speech and behavior of the user B in the utterance section 1007 . As a result, it is possible to grasp what kind of influence is exerted on the user B's mind when what kind of behavior the user B performs.
　出力制御部２４は、特定された区間における評価情報を出力する機能を有し得る。出力制御部２４は、例えば評価情報を、評価結果に応じて表示態様を変化させて出力してもよい。例えば、本実施形態の例示においては、オンラインセッションにおけるユーザＡによるユーザＢに対する言動についてユーザＢに所定の基準についていい影響を与えた場合は、かかる言動に関する情報について、色彩によるヒートマップやオブジェクトの形状等の表示態様を変化させて、ユーザ端末１０、２０等の画面に出力してもよい。これにより直感的にユーザＡへのフィードバックの結果を把握することができる。 The output control unit 24 may have a function of outputting evaluation information in the identified section. The output control unit 24 may output the evaluation information, for example, by changing the display mode according to the evaluation result. For example, in the example of the present embodiment, when user A's behavior toward user B in an online session has a positive influence on user B with respect to a predetermined criterion, information regarding such behavior may be displayed as a color heat map or an object shape. You may output to screens, such as user terminal 10, 20, by changing display modes, such as. Thereby, the result of the feedback to the user A can be grasped intuitively.
　また、出力制御部２４は、言動に関する評価情報を、言動に対応する区間と紐付けて出力してもよい。これにより、どの区間における言動がユーザにとって良い（または悪い）ものであったかどうかを容易に把握することができる。なお、出力制御部２４による評価情報の出力態様は特に限定されない。 Also, the output control unit 24 may output the evaluation information related to the behavior in association with the section corresponding to the behavior. This makes it possible to easily grasp whether the behavior in which section was good (or bad) for the user. Note that the output mode of the evaluation information by the output control unit 24 is not particularly limited.
　図１２は、本実施形態に係るシステムによる処理の流れの一例を示すフローチャートである。まず、特定部２２は、解析結果が所定の基準を満たしたタイミングに基づいて、ユーザの言動の評価対象の区間を特定する（ステップＳ１０１）。次に、評価情報生成部２３は、区間におけるユーザの言動の解析を行い（ステップＳ１０３）、解析結果に基づいて評価情報を生成する（ステップＳ１０５）。 FIG. 12 is a flowchart showing an example of the flow of processing by the system according to this embodiment. First, the specifying unit 22 specifies an evaluation target section of the user's behavior based on the timing when the analysis result satisfies a predetermined criterion (step S101). Next, the evaluation information generation unit 23 analyzes the behavior of the user in the section (step S103), and generates evaluation information based on the analysis result (step S105).
　次に、出力制御部２４は、生成された評価情報を、ユーザ端末１０、２０等に出力する（ステップＳ１０７）。 Next, the output control unit 24 outputs the generated evaluation information to the user terminals 10, 20, etc. (step S107).
　以上、本開示の一実施形態によれば、一のユーザの言動が他のユーザに対してどのような影響を与えたかを評価することができる。これによい、自らの言動についてより的確にフィードバックを受けることが可能となる。 As described above, according to an embodiment of the present disclosure, it is possible to evaluate how one user's behavior affects other users. This makes it possible to receive more accurate feedback on one's behavior.
　本明細書においてフローチャート図を用いて説明した処理は、必ずしも図示された順序で実行されなくてもよい。いくつかの処理ステップは、並列的に実行されてもよい。また、追加的な処理ステップが採用されてもよく、一部の処理ステップが省略されてもよい。 The processes described using the flowcharts in this specification do not necessarily have to be executed in the illustrated order. Some processing steps may be performed in parallel. Also, additional processing steps may be employed, and some processing steps may be omitted.
　以上説明した実施の形態を適宜組み合わせて実施することとしてもよい。また、本明細書に記載された効果は、あくまで説明的または例示的なものであって限定的ではない。つまり、本開示に係る技術は、上記の効果とともに、または上記の効果に代えて、本明細書の記載から当業者には明らかな他の効果を奏しうる。 The embodiments described above may be combined as appropriate and implemented. Also, the effects described herein are merely illustrative or exemplary, and are not limiting. In other words, the technology according to the present disclosure can produce other effects that are obvious to those skilled in the art from the description of this specification, in addition to or instead of the above effects.
　１０、２０　　　ユーザ端末
　２２　　　特定部
　２３　　　評価情報生成部
　２４　　　出力制御部
　３０　　　ビデオセッションサービス端末
　４０　　　評価端末 10, 20 user terminal 22 identification unit 23 evaluation information generation unit 24 output control unit 30 video session service terminal 40 evaluation terminal

Claims (3)

  1. 　複数のユーザでオンラインセッションが行われる環境においてオンラインセッション中にユーザが画面に表示されているか否かによらず前記ユーザを撮影することによって得られる動画像をもとに前記ユーザの反応を分析する動画像分析システムであって、
    　複数の前記ユーザの夫々について、前記オンラインセッション中に前記ユーザを撮影することによって得られる動画像を取得する動画像取得部と、
    　前記動画像取得部により取得された動画像に基づいて、前記ユーザについて生体反応の変化を解析する解析部と、
    　前記解析部により得られた生体反応の解析結果が所定の条件を満たしているタイミングに基づいて前記ユーザとは異なる他のユーザが前記ユーザに対する言動を行っている動画像の区間を特定する特定部と、
    　該区間に含まれる動画像に基づいて、前記他のユーザの前記言動に対する評価情報を生成する評価情報生成部と、
    　を備える動画像分析システム。     In an environment where an online session is held by a plurality of users, the reaction of the user is analyzed based on a moving image obtained by photographing the user regardless of whether or not the user is displayed on a screen during the online session. A moving image analysis system,
    a moving image acquisition unit that acquires a moving image obtained by photographing the user during the online session for each of the plurality of users;
    an analysis unit that analyzes changes in biological reactions of the user based on the moving image acquired by the moving image acquisition unit;
    A specifying unit that specifies a section of a moving image in which a user different from the user is speaking and acting toward the user based on the timing at which the analysis result of the biological reaction obtained by the analysis unit satisfies a predetermined condition. When,
    an evaluation information generation unit that generates evaluation information for the speech and behavior of the other user based on the moving image included in the section;
    A moving image analysis system.
  2. 　請求項１に記載の動画像分析システムであって、
    　前記評価情報生成部は、前記他のユーザについての前記動画像に基づいて前記評価情報を生成する、動画像分析システム。     The moving image analysis system according to claim 1,
    The moving image analysis system, wherein the evaluation information generation unit generates the evaluation information based on the moving image of the other user.
  3. 　請求項１または２に記載の動画像分析システムであって、
    　前記評価情報生成部は、前記ユーザの前記生体反応の解析結果に基づいて、前記ユーザの言動に対する前記評価情報を生成する、動画像分析システム。

     
     
          The moving image analysis system according to claim 1 or 2,
    The moving image analysis system, wherein the evaluation information generation unit generates the evaluation information for the behavior of the user based on an analysis result of the biological reaction of the user.
PCT/JP2021/023776 2021-06-23 2021-06-23 Video analysis system WO2022269801A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2023529316A JPWO2022269801A1 (en) 2021-06-23 2021-06-23
PCT/JP2021/023776 WO2022269801A1 (en) 2021-06-23 2021-06-23 Video analysis system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/023776 WO2022269801A1 (en) 2021-06-23 2021-06-23 Video analysis system

Publications (1)

Publication Number Publication Date
WO2022269801A1 true WO2022269801A1 (en) 2022-12-29

Family

ID=84545338

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/023776 WO2022269801A1 (en) 2021-06-23 2021-06-23 Video analysis system

Country Status (2)

Country Link
JP (1) JPWO2022269801A1 (en)
WO (1) WO2022269801A1 (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2020154093A (en) * 2019-03-19 2020-09-24 株式会社Ｗｉｔｈ Ｔｈｅ Ｗｏｒｌｄ Education support system, method, and program
JP2021022909A (en) * 2019-07-30 2021-02-18 株式会社リコー Information processing apparatus, information processing program, information processing system, and information processing method

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2020154093A (en) * 2019-03-19 2020-09-24 株式会社Ｗｉｔｈ Ｔｈｅ Ｗｏｒｌｄ Education support system, method, and program
JP2021022909A (en) * 2019-07-30 2021-02-18 株式会社リコー Information processing apparatus, information processing program, information processing system, and information processing method

Also Published As

Publication number Publication date
JPWO2022269801A1 (en) 2022-12-29

Similar Documents

Publication Publication Date Title
WO2022269801A1 (en) Video analysis system
WO2022269802A1 (en) Video analysis system
JP7121436B1 (en) Video analysis program
JP7197955B1 (en) Video meeting evaluation terminal
JP7121433B1 (en) Video analysis program
JP7121439B1 (en) Video image analysis system
WO2022201274A1 (en) Video analysis programme
WO2022230065A1 (en) Video analysis system
WO2022254489A1 (en) Video analysis system
WO2022264220A1 (en) Video analysis system
WO2022201275A1 (en) Video analysis programme
JP7152819B1 (en) Video analysis program
WO2022201265A1 (en) Video analysis program
WO2022201267A1 (en) Video analysis program
JP7138990B1 (en) Video image analysis system
WO2022201270A1 (en) Video analysis program
WO2022230051A1 (en) Video analysis system
WO2022264219A1 (en) Video analysis system
WO2022230049A1 (en) Video analysis system
WO2022230136A1 (en) Video analysis system
WO2022230137A1 (en) Video analysis system
WO2022230050A1 (en) Video analysis system
WO2023013026A1 (en) Video analysis system
WO2022201269A1 (en) Video analysis program
WO2022201268A1 (en) Video analysis programme

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21947086

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023529316

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21947086

Country of ref document: EP

Kind code of ref document: A1