WO2018025492A1 - Customer attendance monitoring device, customer attendance monitoring system, and customer attendance monitoring method - Google Patents

Customer attendance monitoring device, customer attendance monitoring system, and customer attendance monitoring method Download PDF

Info

Publication number
WO2018025492A1
WO2018025492A1 PCT/JP2017/020901 JP2017020901W WO2018025492A1 WO 2018025492 A1 WO2018025492 A1 WO 2018025492A1 JP 2017020901 W JP2017020901 W JP 2017020901W WO 2018025492 A1 WO2018025492 A1 WO 2018025492A1
Authority
WO
WIPO (PCT)
Prior art keywords
customer
information
customer service
monitoring device
unit
Prior art date
Application number
PCT/JP2017/020901
Other languages
French (fr)
Japanese (ja)
Inventor
林　裕昭
寿嗣 辻
鉄平 福田
亮太 藤井
Original Assignee
パナソニックＩｐマネジメント株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by パナソニックＩｐマネジメント株式会社 filed Critical パナソニックＩｐマネジメント株式会社
Publication of WO2018025492A1 publication Critical patent/WO2018025492A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/04—Segmentation; Word boundary detection

Definitions

  • This disclosure relates to a service monitoring device, a service monitoring system, and a service monitoring method.
  • a camera is installed in the store, and this camera is used for security purposes.
  • this camera may be used for improvement of store operation, for example, confirmation of the presence or absence of customer service by a store clerk.
  • the following activity status analysis system is known as a system for determining the presence or absence of customer service using a camera (see, for example, Patent Document 1).
  • This activity status analysis system acquires the staying information of a moving body from a captured image by a camera that captures a target area, and acquires the staying information of an ID moving body from a portable terminal held by a store clerk. Then, this activity state analysis system determines that there is a customer service when the ID mobile body and the mobile body stay at a nearby point. The determination result of this customer service is utilized for store management as customer service information.
  • the activity situation analysis system may determine that there is a customer service if the store clerk and the customer are nearby, and the customer service determination accuracy may decrease. For example, even when a store clerk and a customer are located close to each other, it may be determined that the customer is in service, and it is desirable that the determination accuracy of the customer service is improved.
  • This disclosure aims to improve customer service accuracy.
  • the customer service monitoring device acquires sound data related to sound collected by a sound collecting device that collects sound of a target area divided into a plurality of grids, and captures an image of the target area.
  • the utterance detection unit that detects the utterance based on the audio data, and based on the image data, the customer information or the residence information of the person including the customer is generated,
  • a residence information generation unit that generates customer residence information, and detection of person residence information, customer residence information, and utterances Based on the result, when the customer and the customer stay in the same grid for the first predetermined time and the utterance is detected for the second predetermined time or more in the same grid, the customer receives the customer in the same grid. Have And a customer determination unit determines that.
  • the customer service monitoring system of the present disclosure includes a sound collection device that collects sound of a target area divided into a plurality of grids, an imaging device that captures an image of the target area, and a customer service monitoring device.
  • the customer service monitoring device acquires sound data related to the sound collected by the sound collecting device, acquires image data related to the image picked up by the image pickup device, detects speech based on the sound data, and detects image data Based on the location information of the customer including the customer or the customer is generated, and the customer's residence information is generated based on the location information of the mobile terminal possessed by each customer in the target area.
  • the staying person and the customer stay in the same grid for a first predetermined time, and the utterance exceeds the second predetermined time in the same grid.
  • the customer service monitoring method of the present disclosure is a customer service monitoring method in the customer service monitoring device, and acquires voice data related to a sound collected by a sound collecting device that collects sound of a target area divided into a plurality of grids. , Including image data related to an image captured by an imaging device that captures an image of the target area, and an utterance detection unit that detects an utterance based on audio data, and a customer or a customer based on the image data Based on the location information of the mobile terminal possessed by each customer who is located in the target area, the customer's residence information is generated, and the person's residence information, the customer's residence information, and Based on the utterance detection result, if the customer and the customer stay in the same grid for the first predetermined time and the utterance is detected for the second predetermined time or more in the same grid, the same grid is detected. Determines that the customer who is in customer service in de.
  • FIG. 1 is a block diagram showing a customer service monitoring system according to the first embodiment.
  • FIG. 2 is a flowchart illustrating an operation example of the customer service monitoring apparatus according to the first embodiment.
  • FIG. 3 is a schematic diagram illustrating an example of a customer service viewed from the ceiling of the store.
  • FIG. 4 is a perspective view corresponding to FIG. 3 and showing an example of customer service in a store.
  • FIG. 5 is a block diagram illustrating a configuration example of the customer service monitoring system according to the second embodiment.
  • FIG. 6 is a flowchart illustrating an operation example of the customer service monitoring apparatus according to the second embodiment.
  • FIG. 7 is a schematic diagram for explaining a specific action performed by a customer.
  • FIG. 8A is a schematic diagram for explaining an example of peeping determination.
  • FIG. 8A is a schematic diagram for explaining an example of peeping determination.
  • FIG. 8B is a schematic diagram for explaining an example of the ginger determination.
  • FIG. 8C is a schematic diagram for explaining an example of tilt determination.
  • FIG. 9 is a schematic diagram for explaining a store clerk serving as a transmission destination of customer service instruction information.
  • FIG. 10 corresponds to FIG. 9 and is a schematic diagram for explaining a store clerk serving as a destination of customer service instruction information.
  • FIG. 11 is a block diagram illustrating a configuration example of the customer service monitoring system according to the third embodiment.
  • FIG. 12 is a flowchart illustrating an operation example of the customer service monitoring apparatus according to the third embodiment.
  • FIG. 13 is a schematic diagram for explaining the identity of the staying information.
  • FIG. 1 is a block diagram illustrating a configuration example of a customer service monitoring system 100 according to the first embodiment.
  • the customer service monitoring system 100 includes a mobile terminal TE, a microphone array device MA, a camera device CA, a recorder RC, and a customer service monitoring device 10.
  • the mobile terminal TE, the microphone array device MA, the camera device CA, the recorder RC, and the customer service monitoring device 10 are connected via a network NT so that data communication is possible.
  • the network NT is formed including a wired network or a wireless network.
  • the network NT widely includes various networks such as the Internet, a cellular network, a WAN (Wide Area Network), a LAN (Local Area Network), a public line, and a dedicated line.
  • the portable terminal TE (business terminal) is carried by a store clerk SA (for example, see FIGS. 3 and 4) working in the store ST, and can communicate with another communication device (for example, the customer service monitoring device 10) via the network NT. is there.
  • another communication device for example, the customer service monitoring device 10.
  • customers who enter the store ST do not have the mobile terminal TE.
  • the mobile terminal TE detects the position of the mobile terminal TE for the service determination by the service monitoring device 10.
  • Position detection technology using GPS (Global Positioning System), wireless LAN, RFID (Radio Frequency IDentifier), or the like may be used to detect the position of the mobile terminal TE in the store ST.
  • the mobile terminal TE communicates a beacon signal with each beacon terminal installed at a predetermined interval in the in-store ST, and based on the positional relationship with the beacon terminal as the transmission source of the received beacon signal, You may detect the position in ST.
  • the beacon signal may include, for example, information on coordinates indicating a relative position with respect to a reference point in the store ST (for example, the center position of the store ST). Further, the position of the mobile terminal TE in the store ST may be detected by a method other than the above.
  • the portable terminal TE transmits, to the customer service monitoring device 10, the position information where the portable terminal TE is located, for example, by an arbitrary communication method (for example, communication using wireless LAN communication or a public line). Further, the mobile terminal TE receives from the customer service monitoring device 10 information related to customer service (for example, customer service instruction information, information indicating the necessity of customer service, customer CT position information).
  • customer service instruction information for example, customer service instruction information, information indicating the necessity of customer service, customer CT position information.
  • the mobile terminal TE may include an acceleration sensor and detect in which direction the clerk SA carrying the mobile terminal TE has moved (movement information).
  • the customer service monitoring device 10 generates flow line information and staying information for the clerk SA and the customer CT based on the position information of the mobile terminal TE acquired from the mobile terminal TE or based on the position information and movement information of the mobile terminal TE. To do.
  • the camera device CA is a fixed camera with a fixed angle of view, for example, installed on the ceiling or wall of the store ST.
  • the camera device CA functions as a monitoring camera that can image an imaging area that is an imaging space in which the device itself is installed.
  • the camera device CA may temporarily capture a video (image), or may continuously capture and accumulate video (image).
  • the camera apparatus CA is not limited to a fixed camera, and may be an omnidirectional camera or a PTZ camera capable of pan / tilt / zoom operations.
  • the camera device CA stores the time (image capturing time) when the image is captured in association with the image data. Further, the camera device CA transmits the stored video data and imaging time data to the customer service monitoring device 10 via the network NW.
  • the video includes at least one of a moving image and a still image.
  • the microphone array device MA is an omnidirectional microphone array device installed on the ceiling of the store ST, for example.
  • the microphone array device MA collects sound in all directions in the sound collection space (sound collection area) where the device itself is installed.
  • the microphone array apparatus MA may temporarily collect voice, or may continuously collect voice and accumulate it.
  • the microphone array apparatus MA includes, for example, a housing having an opening formed at the center, and a plurality of microphone units arranged concentrically around the opening in the circumferential direction.
  • a high sound quality small electret condenser microphone ECM: Electric Condenser Microphone
  • ECM Electric Condenser Microphone
  • the imaging area and the sound collection area are substantially the same.
  • the imaging area and the sound collection area are service determination target areas.
  • the microphone array device MA stores the collected sound data in association with the time of sound collection (sound collection time). Further, the microphone array apparatus MA transmits the stored voice data and sound collection time data to the customer service monitoring apparatus 10 via the network NW.
  • Recorder RC stores various data.
  • the recorder RC may store the video data captured by the camera device CA and the video time information together with the customer service monitoring device 10 or instead of the customer service monitoring device 10.
  • the recorder RC may store the voice data collected by the microphone array device MA and the sound collection time together with the customer service monitoring device 10 or instead of the customer service monitoring device 10.
  • the customer service monitoring device 10 is installed in or outside the store ST where the microphone array device MA and the camera device CA are installed, for example, in the back yard of the store ST.
  • the customer service monitoring device 10 is, for example, a stationary PC (Personal Computer) or a server.
  • the customer service monitoring device 10 may be installed as a server (for example, a cloud server) in a remote place (for example, a monitoring center for collectively monitoring each store ST) away from the store ST.
  • the customer service monitoring device 10 is monitored, for example, by a monitor (for example, a manager of the store ST, a monitor of the monitoring center), and the display is confirmed and operated.
  • the customer service monitoring device 10 is used for voice detection function (speech detection function), voice detection result accumulation function, person flow line detection function (staying detection function), and voice detection and person flow line detection. Based on customer service determination function.
  • the customer service monitoring device 10 has a position detection function (positioning measurement function) for detecting a position where the mobile terminal TE is located (corresponding to a position of an ID mobile body described later).
  • the customer service monitoring device 10 may be a communication terminal such as a mobile phone, a tablet terminal, or a smartphone instead of the PC.
  • the customer service monitoring device 10 includes a communication unit 11, a control unit 12, a memory 13, an operation unit 14, a display device 15, and a speaker device 16.
  • the customer service monitoring device 10 includes a voice processing unit 20, a person detection unit 30, and a customer service determination unit 40.
  • the voice processing unit 20 includes a voice detection unit 21 and a detection result accumulation unit 22.
  • the person detecting unit 30 includes a moving body staying information generating unit 31 and an ID staying information generating unit 32.
  • the customer service determination unit 40 includes a customer service condition setting unit 41 and a customer service action extraction unit 42.
  • the communication unit 11 receives the video data including the imaging time transmitted from the camera device CA and the audio data including the sound collection time transmitted from the microphone array device MA, and outputs them to the control unit 12. Further, the communication unit 11 receives position information of the mobile terminal TE from the mobile terminal TE.
  • the operation unit 14 is a user interface (UI: User Interface) for notifying the control unit 12 of the content of the user's input operation, and includes a pointing device such as a mouse or a keyboard.
  • UI User Interface
  • the operation unit 14 may be configured using, for example, a touch panel or a touch pad that is arranged corresponding to the screen of the display device 15 and can be input with a user's finger or stylus pen.
  • the operation unit 14 receives an input operation for designating a range of the grid GR.
  • the operation unit 14 may receive an input operation for changing the range of the grid GR.
  • the grid GR is a unit area for determining customer service.
  • the grid GR may be set according to, for example, a new product area, a seasonable area, a standard area, or an area that is classified according to the characteristics of the store.
  • the size of the grid GR may be set arbitrarily.
  • the range of one grid GR may be defined in association with the range of the area where the products of the same concept are displayed, or without being related to the range of the area where the products of the similar concept are displayed, A range of one grid GR may be defined.
  • the division of the grid GR may be determined at the beginning of the opening of the store and may not be changed, or may be changed every time the layout is changed in the store ST.
  • the memory 13 may include, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory).
  • the memory 13 functions as a program memory, a data memory, and a work memory when the customer service monitoring device 10 operates.
  • the memory 13 may include, for example, an HDD (Hard Disk Drive) or an SDD (Solid State Drive).
  • the memory 13 stores, for example, sound data of the sound collected by the microphone array device MA together with the sound collection time.
  • the memory 13 stores, for example, video data of video captured by the camera device CA together with the imaging time. Further, the memory 13 holds information on the portable ID of the portable terminal TE possessed by each clerk SA.
  • the control unit 12 supervises each component of the customer service monitoring device 10.
  • the voice processing unit 20 performs various voice processes on voice data collected by the microphone array device MA.
  • the sound detection unit 21 detects the presence or absence of sound at each position (for example, each divided area described later) in the sound collection area in the store ST.
  • the voice detection by the voice detection unit 21 is performed, for example, constantly or periodically.
  • the voice detection unit 21 performs directivity processing and directs directivity to each divided area DA (see FIG. 3). That is, the voice detection unit 21 sequentially designates each divided area DA, and forms directivity in the direction of the designated divided area DA with the microphone array device MA as a base point based on the voice data. Thereby, the customer service monitoring apparatus 10 can detect the presence or absence of the sound in each divided area DA.
  • the sound collection area (for example, the entire store ST) is divided into a predetermined number in the x direction and the y direction (see FIG. 3), for example. That is, the sound collection area is divided into divided areas DA for determining the presence / absence (utterance presence / absence) of the clerk SA or the customer CT.
  • the sound detection unit 21 detects the presence or absence of sound for each divided area DA. Note that the size of one divided area DA is smaller than the size of one grid GR.
  • the presence or absence of sound for example, when the signal level of the sound data is equal to or higher than a predetermined level, it is detected that there is sound (there is an utterance), and there is no sound when the signal level of the sound data is lower than the predetermined level. (No utterance) may be detected.
  • the detection results are accumulated in time series and can be detected at a constant rate within a predetermined time, it may be detected that there is a voice (there is an utterance). In this case, the customer service monitoring device 10 can reduce false detection due to sudden noise in the store ST.
  • the detection result of the presence or absence of audio in each divided area DA may be visualized as an audio heat map by being displayed on the display device 15 under the control of the control unit 12. Further, in consideration of the size of the sound level, it may be visualized as a sound heat map indicating each position in the store ST and the sound level at that position.
  • the utterance may be detected in one or more divided areas DA.
  • utterances may be detected in all the divided areas DA in which utterances are detected, or utterances are detected in one divided area DA having the maximum detected voice level among the divided areas DA in which utterances are detected. May be.
  • the detection result accumulating unit 22 accumulates and stores in the memory 13 the detection result (sound heat map) of the presence or absence of sound detected by the sound detecting unit 21. For example, the detection result accumulating unit 22 accumulates information on the detection result of the presence / absence of sound in each divided area DA generated every predetermined time.
  • the person detecting unit 30 detects the presence / absence of the clerk SA and the customer CT at each position (for example, each grid GR) in the store ST. For example, the person detection unit 30 accumulates detection information of the positions of the clerk SA and the customer CT, and derives flow line information indicating a trajectory of the movement of the clerk SA and the customer CT over time. Based on the flow line information, the person detection unit 30 derives retention information indicating the retention of the clerk SA and the customer CT.
  • the moving body staying information generating unit 31 generates staying information on the moving body (person) using the video data from the camera device CA.
  • This moving body includes, for example, a customer CT or a store clerk SA. If the ID residence information (corresponding to the residence information of the clerk SA) acquired by the ID residence information generation unit 32 is subtracted from the residence information generated by the mobile body residence information generation unit 31, the residence information of the customer CT is derived. Is done.
  • the moving body staying information generating unit 31 acquires real-time video data imaged by the camera device CA or past video data recorded in the recorder RC or the memory 13.
  • the moving body staying information generating unit 31 detects a person (moving body) from the acquired video data, and generates tracking information (detected position information) for each person.
  • This moving body tracking process is performed using, for example, an image recognition technique such as a person detection technique or a person tracking technique.
  • the moving object staying information generating unit 31 stores the tracking information in the memory 13.
  • the tracking information includes information on the detection position for each person detected in each frame of the video data, the detection time for each person acquired from the imaging time of the frame in which the person was detected, and the like.
  • the moving body staying information generating unit 31 generates flow line information for each person detected from the video data based on the generated tracking information.
  • the moving body staying information generation unit 31 acquires the tracking information from the memory 13 when performing the flow line generation processing after the fact, and directly uses the generated tracking information when performing the flow line generation processing in real time. .
  • the moving body staying information generating unit 31 stores the flow line information generated for each person in the memory 13.
  • the flow line information includes information relating to a movement vector indicating the locus of movement of the person, the position and time of the change point of the movement vector, and the like.
  • the moving body staying information generating unit 31 may generate the flow line information for the peripheral part of each grid GR in order to determine the intersection between the outer periphery of each grid GR and the flow line in the target area.
  • the moving body staying information generating unit 31 generates staying information for each person in the target area (information indicating how much time is staying at which position) for each grid GR based on the flow line information for each person. .
  • the staying information includes, for example, the position of the staying grid GR, its staying time or staying time.
  • the moving object staying information generating unit 31 stores the generated staying information for each person in the memory 13.
  • the ID residence information generation unit 32 derives residence information (also referred to as ID residence information) of the mobile terminal TE using an ID (mobile ID) for identifying the mobile terminal TE. Since the portable terminal TE is possessed by the store clerk SA, the ID stay information can be said to be stay information related to the store clerk SA.
  • the ID retention information generation unit 32 acquires information (ID position information) on the position where the mobile terminal TE is detected by communication with the mobile terminal TE by the communication unit 11.
  • ID position information detection technique for example, the technique described above is used.
  • the ID residence information generation unit 32 accumulates the acquired ID position information in the memory 13.
  • the ID residence information generation unit 32 generates flow line information (ID flow line information) of the mobile terminal TE based on the acquired ID position information.
  • the ID staying information generation unit 32 acquires the ID position information once accumulated in the memory 13 when the flow line generation process is performed later, and the detected ID when the flow line generation process is performed in real time. Use location information directly.
  • the ID retention information generation unit 32 stores ID flow line information in the memory 13.
  • the ID flow line information includes information regarding a movement vector indicating the movement locus of the mobile terminal TE, the position of the change point, the time, and the like.
  • the ID staying information generation unit 32 may generate flow line information for the peripheral part of each grid GR in order to determine the intersection between the outer periphery of each grid GR and the flow line in the target area.
  • the ID staying information generating unit 32 generates ID staying information in the target area for each grid GR based on the generated ID flow line information for each person (each store clerk SA).
  • the ID staying information includes, for example, the position of the staying grid GR and the staying time and staying time thereof.
  • the ID staying information generating unit 32 stores the generated ID staying information in the memory 13.
  • the customer service determination unit 40 determines whether or not the clerk SA is serving the customer CT at the store ST.
  • the customer service condition setting unit 41 sets customer service conditions based on input information received via the operation unit 14, for example.
  • the customer service condition includes, for example, selection of an area (grid GR) for performing customer service determination, operation related to customer service (for example, operation related to customer service shown in FIG. 2 described later), and setting of various threshold values used in operations related to customer service. It is.
  • the customer service condition defines, for example, in which area the store clerk SA or customer CT determines what kind of behavior (customer service behavior) is determined to be in service.
  • the customer service behavior extraction unit 42 extracts the customer service behavior that satisfies the set customer service conditions. When the customer service condition satisfying the customer service is extracted, it indicates that the clerk SA is serving the customer CT. The customer service behavior extraction unit 42 extracts presence / absence of customer service behavior in the store ST.
  • the display device 15 displays real-time or past video data captured by the camera device CA on the screen.
  • the speaker device 16 outputs the sound data picked up by the microphone array device MA or the sound data picked up by the microphone array device MA having directivity formed at the directivity angles ( ⁇ MAh, ⁇ MAv).
  • the display device 15 and the speaker device 16 may be configured as separate devices from the customer service monitoring device 10.
  • the directivity angle ⁇ MAh indicates the horizontal angle from the microphone array apparatus MA to the sound source position (for example, the divided area DA)
  • the directivity angle ⁇ MAv indicates the vertical angle from the microphone array apparatus MA to the sound source position (for example, the divided area DA). Show.
  • each constituent unit for example, the control unit 12, the voice processing unit 20, the person detection unit 30, and the service determination unit 40
  • the processor includes a CPU (Central Processing Unit), a DSP (Digital Signal Processor), a GPU (Graphical Processing Unit), and the like.
  • FIG. 2 is a flowchart showing an operation example of the customer service monitoring device 10.
  • the customer service condition setting unit 41 sets customer service conditions based on, for example, input information obtained via the operation unit 14 (S11).
  • the customer service conditions here include, for example, the processes shown in S14 to S19.
  • the moving body staying information generating unit 31 acquires video data from the camera device CA and generates staying information on the moving body (S12). Further, the ID residence information generation unit 32 acquires information on the position of each clerk SA from the mobile terminal TE identified by the mobile ID, and generates ID residence information (S12).
  • the staying information and ID staying information of the moving body may be real-time information or past information.
  • the generation of the stay information of the moving body and the generation of the ID stay information may be performed constantly or periodically.
  • the voice processing unit 20 acquires voice data from the microphone array device MA via the communication unit 11. Moreover, the audio
  • the customer service condition setting unit 41 selects a grid GR to be analyzed (customer determination target) in the store ST based on, for example, input information (operation information for selecting a grid) obtained via the operation unit 14. (S14).
  • the customer service behavior extraction unit 42 determines whether or not the store clerk SA and the customer CT are located in the grid GR to be analyzed based on the stay information and ID stay information of the moving body (S15). For example, the customer service behavior extraction unit 42 detects a plurality of moving bodies in the staying information of the moving body, and the ID staying information has less than the number of moving bodies in the staying information of the moving body and one or more ID moving bodies. Is detected, it is determined that the clerk SA and the customer CT are located in the grid GR to be analyzed. In S15, it may be determined whether or not the clerk SA and the customer CT exist for a certain period of time.
  • the voice processing unit 20 determines whether or not voice is detected for a certain period of time in an arbitrary divided area DA in the grid GR. Is determined (S16).
  • the divided areas DA to be subjected to sound detection are sequentially changed, and the presence or absence of sound is determined in each divided area DA.
  • the customer service action extraction unit 42 determines that the clerk SA is serving the customer CT in the grid GR (S17).
  • This voice includes, for example, the voice of the clerk SA or the voice of the customer CT.
  • the speaker may or may not be identified by voice, but depending on the position of the divided area DA where the voice is detected, it may be possible to identify the voice of the clerk SA or the voice of the customer CT. .
  • the extraction unit 42 determines that the clerk SA is not serving the customer CT (S18).
  • the customer service determination unit 40 determines whether the determination of the presence / absence of customer service is completed in all the grids GR to be analyzed (S19). When the determination of the presence / absence of customer service in all grids GR is completed (Yes in S19), the customer service monitoring device 10 ends the process of FIG. When the determination of the presence / absence of customer service in at least one grid GR has not been completed (No in S19), the process proceeds to S14, and the customer service determination unit 40 selects the next grid GR as the analysis target grid.
  • the customer service monitoring device 10 determines the position of the store clerk SA at which position in the store ST and at which time zone based on the stay information of the person (mobile body) and the stay information of the portable ID. And whether the customer CT is located. Further, the customer service monitoring device 10 can use the information on the presence / absence of voice in the divided area DA to determine the presence / absence of customer service. Therefore, the customer service monitoring device 10 can suppress the determination that there is customer service by simply detecting the approach between the clerk SA and the customer CT in the same time zone.
  • FIG. 3 is a schematic diagram showing an example of a customer service viewed from the ceiling of the store ST.
  • FIG. 4 corresponds to FIG. 3 and is a perspective view showing an example of a state of customer service in the store ST.
  • the clerk SA is located at an arbitrary position in the store ST, for example.
  • the customer CT approaches the product shelf SS in order to purchase and confirm the product displayed on the product shelf SS in the store ST.
  • the stores ST are supermarkets, home centers, furniture stores, department stores, and the like. When the store ST is a relatively wide store, the customer service monitoring system 100 is more useful.
  • the periphery of the product shelf SS1 is included in the areas of grids GR1, GR2, GR5, GR6.
  • the grid GR1 has a store clerk SA1 overlooking the entire store ST.
  • the store clerk SA1 greets the customer CT or the like who has visited the store.
  • voice is detected in each divided area DA of the area D1 in GR1.
  • the person detection unit 30 detects that one person is present in the grid GR1 based on the video data captured by the camera device CA (S12 in FIG. 2).
  • the person detection unit 30 detects that the clerk SA1 is located in the grid GR1 based on the portable ID of the portable terminal TE possessed by the clerk SA1 (S12 in FIG. 2). Therefore, the person detection unit 30 detects that there is one clerk SA1 in the grid GR1 and no customer CT (No in S15 in FIG. 2).
  • the voice processing unit 20 detects the voice generated by the clerk SA1 in the surrounding divided area DA including the location of the clerk SA1 in the grid GR1.
  • the customer service determination unit 40 determines that the customer is not in service in the grid GR1 (S18 in FIG. 2).
  • the store clerk SA2 and the customer CT1 are located in the area of the grid GR7 between the product shelves SS2 and SS3. As shown in FIG. 4, the clerk SA2 faces the product shelf SS2, the customer CT1 faces the product shelf SS3, and the clerk SA2 and the customer CT1 are back to back. Here, the clerk SA2 and the customer CT1 are not talking.
  • the person detection unit 30 detects that there are two sights on the grid GR7 based on the video data captured by the camera device CA (S12 in FIG. 2).
  • the person detection unit 30 detects that the clerk SA2 is located in the grid GR7 based on the portable ID of the portable terminal TE possessed by the clerk SA2 (S12 in FIG. 2). Therefore, the person detection unit 30 detects that one clerk SA2 and one customer CT1 are present in the grid GR7 (Yes in S15 of FIG. 2).
  • the sound processing unit 20 does not detect any sound emitted by the store clerk SA2 and the customer CT1 in the surrounding divided area DA including the locations of the store clerk SA2 and the customer CT1 (No in S16 of FIG. 2). . Therefore, the customer service determination unit 40 determines that the customer is not in service in the grid GR1 (S18 in FIG. 2).
  • the store clerk SA3 and the customer CT2 are located in the area of the grid GR14 between the product shelves SS4 and SS5. As shown in FIG. 4, the clerk SA3 and the customer CT2 face the same product shelf SS5. For example, the clerk SA3 and the customer CT2 have a conversation related to the products displayed on the product shelf SS5. In FIG. 3, voice (speech) is detected in each divided area DA of the area D2 in the GR14.
  • the person detection unit 30 detects that there are two sights on the grid GR14 based on the video data captured by the camera device CA (S12 in FIG. 2).
  • the person detection unit 30 detects that the clerk SA3 is located in the grid GR14 based on the mobile ID of the mobile terminal TE possessed by the clerk SA3 (S12 in FIG. 2). Therefore, the person detection unit 30 detects that one clerk SA3 and one customer CT2 are located in the grid GR1 (Yes in S15 in FIG. 2).
  • the voice processing unit 20 detects a voice uttered by at least one of the clerk SA3 and the customer CT2 in the surrounding divided area DA including the location of the clerk SA3 and the customer CT2 (Yes in S16 of FIG. 2). ). Therefore, the customer service determination unit 40 determines that the clerk SA3 is serving the customer CT2 in the grid GR14 (S17 in FIG. 2).
  • the customer service monitoring system 100 includes the mobile terminal TE, the microphone array device MA, the camera device CA, and the customer service monitoring device 10.
  • the customer service monitoring device 10 includes a communication unit 11, a voice processing unit 20, a person detection unit 30, and a customer service determination unit 40.
  • the communication unit 11 acquires audio data related to the sound collected by the microphone array device MA that collects the sound of the target area divided into a plurality of grids GD.
  • the communication unit 11 acquires image data relating to an image captured by the camera device CA that captures an image of the target area.
  • the voice processing unit 20 detects an utterance based on the voice data.
  • the person detection unit 30 generates residence information of a person including the clerk SA or the customer CT based on the image data.
  • the person detecting unit 30 generates the stay information of the clerk SA based on the position information of the mobile terminal TE possessed by each clerk SA located in the target area.
  • the customer service determination unit 40 stores the clerk SA and the customer CT in the same grid GR for a predetermined time t1 based on the person's stay information, the stay information of the store clerk SA, and the utterance detection result. When the utterance is detected for the time t2 or more, it is determined that the store clerk SA is in contact with the same grid GR.
  • the voice processing unit 20 is an example of an utterance detection unit.
  • the person detection unit 30 is an example of a staying information generation unit.
  • the microphone array device MA is an example of a sound collection device.
  • the camera device CA is an example of an imaging device.
  • the salesclerk SA is an example of a customer service.
  • the predetermined time t1 is, for example, 5 seconds.
  • the predetermined time t2 is, for example, 1 second.
  • the customer monitoring system 100 and the customer monitoring device 10 do not determine that the customer is in service in response to the fact that the clerk SA and the customer CT are close to each other, and the sound collected by the microphone array device MA is collected. Data can be used for customer service judgment. Therefore, the customer service monitoring system 100 and the customer service monitoring device 10 determine that the customer is not in service when, for example, the store clerk SA and the customer CT accidentally approach but do not talk and serve, so the determination accuracy of the service is increased. Can be improved. Moreover, the customer service monitoring system 100 and the customer service monitoring apparatus 10 can perform customer service determination without requiring a special operation for customer service determination.
  • FIG. 5 is a block diagram illustrating a configuration example of the customer service monitoring system 100A according to the second embodiment. 5 is different from the customer monitoring system 100 shown in FIG. 1 in that a customer monitoring device 10A is provided instead of the customer monitoring device 10.
  • the same components as those of the customer service monitoring system 100 shown in FIG. 1 are denoted by the same reference numerals, and description thereof is omitted or simplified.
  • the customer service monitoring device 10A includes a voice processing unit 20A, a person detection unit 30, and a customer service determination unit 40A.
  • the voice processing unit 20A includes a voice detection unit 21, a detection result accumulation unit 22, a directivity synthesis unit 23, and a voice discrimination unit 24.
  • the service determination unit 40A includes a service condition setting unit 41A, a behavior determination unit 43, a service necessity determination unit 44, and a terminal notification unit 45.
  • the customer service determination unit 40A may have the function of the customer service determination unit 40 of the first embodiment. That is, the customer service determination unit 40A may determine the presence or absence of customer service at each position in the store ST.
  • the directivity synthesis unit 23 forms voice directivity in the direction where the specific customer CT in the store ST is located.
  • the specific customer CT is, for example, a customer who is not served by the store clerk SA, and includes a customer who needs an explanation about a product, a customer who is looking for a product, or another customer who needs support.
  • the voice discriminating unit 24 discriminates whether or not a specific keyword is included in the voice detected by the voice detecting unit 21 by a predetermined voice recognition process.
  • the specific keyword includes, for example, a word (including explicit or suggestion) indicating that customer service by the salesclerk SA is required. Specific examples of specific keywords will be described later.
  • the customer service condition setting unit 41A sets the customer service condition based on the input information received via the operation unit 14, for example.
  • the customer service conditions include, for example, selection of an area (grid GR) for determining customer service necessity, operation related to customer service necessity determination (for example, operation related to customer service necessity determination shown in FIG. 6 described later), customer service required.
  • the setting of various threshold values used in the negative determination is included.
  • the customer service condition here defines, for example, in which area the store clerk SA and the customer CT determine what kind of behavior the customer service is necessary.
  • the behavior determination unit 43 determines whether or not the customer CT located in the store ST has performed a specific behavior.
  • the specific behavior includes, for example, behavior indicating that the customer CT needs customer service by the store clerk SA (including explicit or suggestion). Specific examples of specific actions will be described later.
  • the customer service necessity determination unit 44 determines the necessity of customer service at each position (each grid GR) in the store ST. For example, when the customer CT located in the store ST issues a specific keyword or when the customer CT located in the store ST performs a specific action, the customer service necessity determination unit 44 needs to serve the customer. Is determined. Further, for example, the customer service necessity determination unit 44 determines that the customer service is unnecessary when the customer CT located in the store ST does not issue a specific keyword or does not perform a specific action.
  • the terminal notification unit 45 transmits, through the communication unit 11, customer service instruction information indicating that customer service is required to all or a part of the mobile terminal TE possessed by the store clerk SA.
  • the customer service instruction information includes, for example, information on the location of the customer CT that needs customer service and information that instructs the customer CT.
  • the information on the location of the customer CT may be, for example, the location information of the grid GR including the location of the customer CT, or the location information of the divided area DA including the location of the customer CT.
  • the information on the location of the customer CT may be intuitively shown using a map indicating the inside of the store ST.
  • FIG. 6 is a flowchart showing an operation example of the customer service monitoring device 10A.
  • the customer service condition setting unit 41 sets a customer service condition (customer service necessity determination condition) based on, for example, input information obtained through the operation unit 14 (S21).
  • the customer service conditions here include, for example, the processes of S23 to S31.
  • the moving body staying information generating unit 31 acquires video data from the camera device CA and generates staying information of the moving body (S22). Further, the ID staying information generating unit 32 acquires information on the position of each clerk SA from the mobile terminal TE identified by the mobile ID, and generates ID staying information (S22).
  • the staying information and ID staying information of the mobile object may be real-time information or past information. Note that the generation of the stay information of the moving body and the generation of the ID stay information may be performed constantly or periodically. That is, the process of S22 is the same as the process of S12.
  • the customer service condition setting unit 41A selects, for example, the grid GR to be analyzed (service request necessity determination target) in the store ST based on the input information obtained through the operation unit 14 (S23).
  • the customer service necessity determination unit 44 determines whether or not the customer CT is alone in the selected grid GR (S24). For example, the customer service necessity determination unit 44 can extract the stay information of the customer CT by subtracting the ID stay information from the stay information of the moving body (person), and can determine whether or not the customer CT is alone. .
  • the voice processing unit 20A acquires voice data from the microphone array device MA, the recorder RC, or the memory 13 via the communication unit 11 (S25). ).
  • the directivity synthesis unit 23 forms voice directivity in the direction in which the customer CT detected in S24 in the selected grid GR is located (S25).
  • the voice discriminating unit 24 determines whether or not a specific keyword (target keyword) issued by the customer CT in the selected grid GR has been detected (S26).
  • specific keywords include, for example, “I ’m sorry”, “Is there someone else”, “Care clerk”, “Well, what to do”, “I don't know”, “I don't know”, “ Words pronounced of the need for customer service, such as “Do n’t get lost” and “Keep me here”.
  • the specific keyword is not limited to these words.
  • the customer service determination unit 40A displays the flow line information or staying information of the customer CT in the selected grid GR as the person detection unit 30 or Read (acquire) from the memory 13 (S27).
  • the behavior determination unit 43 determines whether or not the customer CT has performed a specific behavior based on the flow line information or the stay information of the customer CT in the selected grid GR (S28). A specific example of the specific action will be described later.
  • the customer service necessity determination unit 44 determines that customer service is not required for this customer CT (S29).
  • the customer service necessity determination unit 44 needs to serve the customer CT. It is determined that there is (S30).
  • the terminal notification unit 45 transmits customer service instruction information to all or a part of the mobile terminal TE possessed by the store clerk SA via the communication unit 11 (S30). A specific example of the clerk SA that is the destination of the customer service instruction information will be described later.
  • the customer service determination unit 40A It is determined whether or not the determination of the necessity of customer service has been completed in all grids GR (S30A).
  • the customer service monitoring apparatus 10A ends the process of FIG.
  • the process proceeds to S23, and the customer service determination unit 40A selects the next grid GR as the analysis target grid.
  • the customer service monitoring apparatus 10A determines whether or not the customer needs to be served when the customer CT is alone and a specific keyword or a specific action is detected. Therefore, even if the customer CT happens to be alone in the store ST, issues a specific keyword, or performs a specific action, it is not determined that the customer service is necessary, so the accuracy of determining whether the customer service is necessary is improved. it can.
  • FIG. 7 is a schematic diagram for explaining a specific action performed by the customer CT.
  • six commodity shelves SS are installed in the store ST.
  • the area in the vicinity of the product shelf SS is assumed to be a product tour area A1 for the customer CT to tour the products on the product shelf SS.
  • the passage for moving the product shelf SS to be visited and the passage on the outer periphery in the store ST are assumed to be the store clerk search area A2 for the customer CT to search for the store clerk SA.
  • the product tour area A1 is an area where the customer CT may stay in order to tour the product.
  • the clerk search area A2 is an area where the customer CT may stay in order to search for the clerk SA.
  • the customer service determination unit 40A refers to the stay information of the customer CT, the stay time in the product tour area A1 is a predetermined time t11 (for example, 1 second) or more, then moves to the clerk search area A2, and then the product tour area again.
  • t11 for example, 1 second
  • the customer CT went to look for the clerk SA because the height of the merchandise shelf SS was high, but gave up without being found.
  • the customer service determination unit 40A May be determined to have performed a specific action. In this behavior, it can be considered that the customer CT has been thinking about the product displayed on the product shelf SS for a long time.
  • the customer service determination unit 40A determines that the customer CT has performed a specific action. May be. In this behavior, it is considered that the customer CT moves to the clerk search area A2 and searches for the clerk SA.
  • a predetermined time t13 for example, 600 seconds (10 minutes)
  • the customer service determination unit 40A When the customer service determination unit 40A is searching for a clerk SA or a product beyond the product shelf SS in the product tour area A1 (that is, the customer CT looks into the product shelf SS and extends, for example), the customer CT May be determined to have performed a specific action. In this case, the service determination unit 40A determines whether or not the position of the head HD1 of the customer CT is higher than the average position HDave in the vertical direction while staying in the product tour area A1 by a predetermined threshold th1 or more. To do. This determination can be made by, for example, image recognition processing or feature extraction processing on video data.
  • FIG. 8A is a schematic diagram for explaining an example of whether or not the customer CT has looked into from the upper side of the commodity shelf SS (looking-in determination).
  • the position of the head HD of the customer CT is at the position of the head HD1 by looking into the top of the product shelf SS, and is above the threshold th1 by the head th average position HDave.
  • the customer service determination unit 40A determines that the customer CT is looking in, and determines that the customer CT is performing a specific action.
  • the customer service determination unit 40A may determine that the customer CT has performed a specific action when the customer CT is in the product tour area A1 or the clerk search area A2. In this case, the service determination unit 40A determines that the position of the head HD or the shoulder SD of the customer CT is the left-right direction (a direction rotated 90 ° counterclockwise or right) as viewed from the top of the head (above the head HD). It is determined whether or not it has been moved a plurality of times at a certain angle (for example, a direction rotated at an arbitrary angle other than 90 ° counterclockwise or clockwise). This determination can be made by, for example, image recognition processing or feature extraction processing on video data.
  • FIG. 8B is a schematic diagram for explaining an example of determining whether or not the customer CT is in a dark state (a black and white determination).
  • the head HD is rotated counterclockwise at an angle ⁇ and rotated a plurality of times.
  • the customer service determination unit 40A determines that the customer is afraid and determines that the user is performing a specific action.
  • the fact that the number of rotations of the head HD or the shoulder SD is equal to or greater than a predetermined number may be added to a condition that satisfies the mentality determination.
  • the customer service determination unit 40A may determine that the customer CT has performed a specific action when the customer CT leans his / her head in the product tour area A1 or the clerk search area A2. In this case, the service determination unit 40A determines that the position of the head HD or shoulder SD of the customer CT is in the left-right direction (counterclockwise or clockwise) when viewed from the front direction of the head HD (that is, the front direction of the face). It is determined whether or not it is tilted at a predetermined angle), for a predetermined time t21 (for example, 5 seconds) or more or for a predetermined number of times (for example, 3 times), that is, whether the neck is tilted. This determination can be made by, for example, image recognition processing or feature extraction processing on video data.
  • FIG. 8C is a schematic diagram for explaining an example of whether or not the head is tilted (tilting determination).
  • the head HD of the customer CT is tilted leftward at an angle ⁇ 2 a predetermined number of times t22 or more.
  • the customer service determination unit 40A determines that the head is curled and determines that a specific action is being performed.
  • the customer service determination unit 40A may determine that the customer CT has performed a specific action when the customer CT has a different product in the product tour area A1 or the clerk search area A2. In this case, for example, the customer service determination unit 40A detects the area of the arm or hand of the customer CT from the video data, and determines whether or not an object (product) exists in the area around both of the areas. When the customer CT has a plurality of products, it may be possible to compare both products and want to know the characteristics of each product.
  • FIG. 9 and FIG. 10 are schematic diagrams for explaining the store clerk SA that is the transmission destination of the customer service instruction information.
  • the terminal notification unit 45 may transmit customer service instruction information with priority given to a single clerk SA (for example, clerk SA11).
  • the person detection unit 30 may detect that there is one clerk SA in the selected grid GR and that it is a single clerk SA when there is one ID residence information in the selected grid GR. Good.
  • the terminal notifying unit 45 excludes that the store clerk SA is otherwise incapable of serving customers, from the destination of the customer service instruction information. May be. In this case, the customer service instruction information is not transmitted to the portable terminal TE possessed by the clerk SA.
  • the person detection unit 30 may determine whether or not the customer CT is located near the store clerk SA, for example, by the same method as S15 in FIG. That is, in the selected grid GR, the person detection unit 30 detects a plurality of moving bodies in the staying information of the moving bodies, and the ID staying information has less than the number of moving bodies in the staying information of the moving bodies, and 1 When two or more ID moving bodies are detected, it may be determined that the customer CT is located near the clerk SA.
  • the terminal notification unit 45 transmits customer service instruction information to the mobile terminal TE owned by the clerk SA located in the backyard outside the store ST. Also good.
  • the person detection unit 30 analyzes each grid GR, and when there is no ID residence information in each grid GR, or even if there is ID residence information in each grid GR, the customer CT is located near the clerk SA. In this case, it may be detected that there is no single clerk SA.
  • the terminal notification unit 45 may transmit the customer service instruction information to the mobile terminal TE identified by the mobile ID for which the ID staying information is not detected by the person detection unit 30. In this case, the mobile ID information is held in advance in the memory 13 for each mobile terminal TE possessed by the store clerk SA located inside and outside the store ST.
  • the terminal notification unit 45 provides customer service instruction information to all the mobile terminals TE owned by the clerk SA (all clerk) located inside and outside the store ST. May be sent. In this case, the customer service instruction information is notified to all the clerk SA, so that the clerk SA can cooperate with each other as much as possible.
  • the person detection unit 30 detects one first customer CT that stays for a predetermined time t1 or longer in an arbitrary grid GR.
  • the customer service determination unit 40A needs to serve the first customer CT. Is determined.
  • the communication unit 11 transmits customer service instruction information for instructing customer service for the first customer CT to the mobile terminal TE possessed by at least a part of the store clerk SA when customer service for the first customer CT is required.
  • the customer service monitoring device 10A detects a mobile body (that is, the first customer CT) that is not the store clerk SA in an arbitrary grid GR, and determines whether the first customer CT needs to support the store clerk SA. Inferred from the word issued by the customer CT and the behavior of the first customer CT. Then, when the first customer CT needs the support of the store clerk SA, the customer service monitoring apparatus 10A notifies the mobile terminal TE held by the store clerk SA to that effect.
  • a mobile body that is, the first customer CT
  • the customer service monitoring apparatus 10A notifies the mobile terminal TE held by the store clerk SA to that effect.
  • the customer service monitoring system 100A and the customer service monitoring device 10A utilize the ward issued by the first customer CT that requires customer service and the behavior of the first customer CT to determine whether or not the customer needs to be served. Loss can be reduced. Further, regardless of whether or not the first customer CT directly talks to the clerk SA, the clerk SA can recognize that the customer service is required, and the satisfaction of the first customer CT can be improved.
  • the service determination unit 40A performs a specific action based on the detection position of the head HD or shoulder SD of the first customer CT and the detection time or the number of detections of the head HD or shoulder SD. It may be determined whether or not.
  • the customer service monitoring system 100A and the customer service monitoring device 10A can easily determine whether or not a specific action is being performed by detecting the movement of each part of the body of the first customer CT. It can be used for judgment.
  • the customer service determination unit 40A may determine that a specific action is being performed when the stay information of the first customer CT includes a predetermined stay pattern (for example, first to third stay patterns).
  • the customer service monitoring system 100A and the customer service monitoring device 10A can easily determine whether or not a specific action is being performed by detecting a staying pattern based on the trajectory of the movement of the first customer CT. It can be used to determine necessity.
  • the communication unit 11 may transmit the customer service instruction information to the clerk SA located by one person in an arbitrary grid based on the stay information of the clerk SA.
  • the clerk SA located by himself / herself can check the customer service instruction information and can quickly serve the customer 1 for the first customer CT.
  • the communication unit 11 may transmit customer service instruction information to a non-customer (for example, a customer clerk SA who is not in customer service) based on the customer clerk SA residence information and person residence information.
  • a non-customer for example, a customer clerk SA who is not in customer service
  • the clerk SA who has confirmed the customer service instruction information is not in service, and can quickly serve the first customer CT.
  • the communication unit 11 transmits the customer service instruction information to the mobile terminal TE possessed by the clerk SA for which no stay information of the clerk SA has been generated. Also good.
  • the store clerk SA who is not in the store ST and has not generated the stay information and is waiting in the backyard confirms the customer service instruction information with the mobile terminal TE, and quickly becomes the first customer CT. You can serve customers.
  • the communication unit 11 may transmit the customer service instruction information to the mobile terminals TE possessed by all the clerk SA when there is no clerk SA who is alone in any grid GR.
  • each clerk SA confirms the customer service instruction information received by the mobile terminal TE, and each clerk SA consults with each other. For example, according to the current customer service situation, any clerk SA serves the first customer CT. can do.
  • the customer service monitoring device 10A determines the behavior of the customer CT who is about to purchase the product but is expected to be confused, the behavior of the customer CT that is expected to explain the product, and the clerk SA.
  • the behavior of the customer CT expected to be searched is regarded as the behavior that the customer needs, and the customer clerk SA is notified of the customer service instruction.
  • the customer service instruction information can be notified to the clerk SA who is expected to be able to serve customers.
  • the notified clerk SA can recognize the necessity of customer service for the customer CT.
  • the customer service monitoring device 10A can promptly notify the appropriate store clerk SA of the necessity of customer service, and can suppress customer service loss.
  • the customer service monitoring device 10A is based on person detection and flow line detection based on an image captured by the camera device CA, position detection of the clerk SA using radio, sound detection based on sound collected by the microphone array device MA, and the like.
  • the customer CT that requires the store clerk SA can be detected, and an alert can be output to the store clerk SA and other monitoring personnel that the customer service is required.
  • the customer service monitoring device 10A can suppress the loss of the customer service opportunity because the store clerk SA cannot recognize the customer service requirement even though the customer CT needs the customer service.
  • the customer service monitoring device 10A can analyze the behavior of the customer CT without forcing the customer CT, and can realize customer service with higher accuracy by associating the customer CT with the clerk SA.
  • the customer service monitoring device in the customer service monitoring system uses voice recognition to determine whether or not the person is the same person when the person's flow line indicated by the person's flow line information is interrupted by an obstacle or the like. Explain that Judgment.
  • FIG. 11 is a block diagram illustrating a configuration example of the customer service monitoring system 100B according to the third embodiment. 11 is different from the customer monitoring system 100 shown in FIG. 1 in that a customer monitoring device 10B is provided instead of the customer monitoring device 10.
  • the same components as those of the customer service monitoring system 100 shown in FIG. 1 are denoted by the same reference numerals, and description thereof is omitted or simplified.
  • the customer service monitoring device 10B includes a voice processing unit 20B, a person detection unit 30B, and a customer service determination unit 40.
  • the voice processing unit 20B includes a voice detection unit 21, a detection result accumulation unit 22, and a speaker identification unit 25.
  • the person detection unit 30 ⁇ / b> B includes a moving body residence information generation unit 31, an ID residence information generation unit 32, and a residence information adjustment unit 33.
  • the speaker identification unit 25 identifies a speaker that utters the voice detected by the voice detection unit 21 by a predetermined speaker recognition process (for example, voiceprint identification).
  • the staying information adjusting unit 33 integrates staying information determined to be the same person by speaker identification or the like, and stores the staying information in the memory 13. For example, if it is determined that the same person as the person who has been generated and retained in the memory 13 has appeared again in the store ST, the staying information adjustment unit 33 continues the staying information ( As part of the information, information related to this person's stay is accumulated.
  • FIG. 12 is a flowchart showing an operation example of the customer service monitoring device 10B.
  • the customer service monitoring device 10B executes the processing of S11 to S14 shown in FIG.
  • the customer service behavior extraction unit 42 determines whether or not the store clerk SA or the customer CT is located in the grid GR to be analyzed (S31). For example, the customer service behavior extraction unit 42 can determine that the store clerk SA or the customer CT is located when the moving object is detected in the staying information of the moving object.
  • the speaker identification unit 25 acquires speaker identification information for identifying the speaker by speaker authentication processing (S32). ). For example, the speaker identification unit 25 acquires information on a spectrum of a speech waveform related to a salesclerk SA or a customer CT as a speaker as speaker identification information.
  • the speaker identification unit 25 determines whether or not speaker identification of this clerk SA or customer CT has been performed before (S33). Whether or not it has been performed before can be determined by whether or not the same speaker identification information is held in the memory 13.
  • the same here does not need to be completely the same, and when it exceeds a certain likelihood, for example, when the correlation value or similarity between the speech spectra as speaker identification information is a predetermined value or more May be included identically.
  • the position in the store ST where the speaker identification information was obtained in the past may be an arbitrary position, and may not be limited to the same grid as the grid GR selected in S14.
  • the speaker identification unit 25 Holds (registers) the acquired speaker identification information in the memory 13 (S34). By registering the speaker identification information in the memory 13 for the first time, speaker identification can be performed using the speaker identification information for the second and subsequent times.
  • the staying information adjustment unit 33 When speaker identification of this clerk SA or customer CT has been performed before, that is, when the same speaker identification information as the speaker identification information acquired in S32 is held in the memory 13, the staying information adjustment unit 33 The staying information related to the person with the same speaker identification information is integrated (merged) and held in the memory 13 as staying information of the same person (S35).
  • the stay information adjusting unit 33 integrates the stay information of the mobile body related to the store clerk SA, and further integrates the ID stay information related to the store clerk SA.
  • the staying information adjustment unit 33 integrates the staying information of the moving body related to the customer CT.
  • the customer service determination unit 40 performs the analysis in all the grids GR to be analyzed. It is determined whether the determination of the identity of the staying information has been completed (S36). When the determination of the identity of the staying information in all grids GR is completed (Yes in S36), the customer service monitoring device 10B ends the process of FIG. When the determination of the identity of the stay information in at least one grid GR has not been completed (No in S36), the process proceeds to S14, and the customer service determination unit 40 analyzes the next grid GR (the identity of the stay information). Select as the judgment target grid.
  • the service monitoring apparatus 10 ⁇ / b> B corresponds to the person staying information related to the speaker identification information when the speaker identification information related to the newly appearing person has existed in the past.
  • the existing staying information is integrated and stored in the memory 13.
  • the customer service monitoring device 10B uses the speaker identification processing even when the flow line information or staying information of the person is interrupted, for example, when the person moves through a shield such as the product shelf SS. Thus, it is possible to determine the identity of the sound produced by each person.
  • the customer service monitoring device 10B can be used for customer service activities as flow line information and staying information of the same person when a plurality of persons are identified as being the same person by exceeding the certain likelihood by the identification process using voice.
  • FIG. 13 is a schematic diagram for explaining the identity of the staying information.
  • the first person P1 is hidden behind the product shelf SS21 from the left side of the product shelf SS21, and the second person P2 appears from the right side of the product shelf SS21.
  • the flow line information and the staying information relating to the first person P1 and the second person P2 are derived by the person detection unit 30, the flow line is blocked by the product shelf SS21 as a shield. Therefore, the flow line information and the staying information are obtained separately for each of the first person P1 and the second person P2.
  • the voice processing unit 20B obtains voice data of the first person P1 and the second person P2, and performs voice identification processing on the voice data of the both persons, thereby generating voice generated by the same person. Or not.
  • the customer service monitoring apparatus 10B can handle the flow line information and the staying information as the same person information in an integrated manner even if the flow line information and the staying information of a plurality of persons are divided by the shield. Therefore, for example, when a customer CT appears from the back of the product shelf SS21 instead of a new customer, it can be linked to whether or not the customer is a customer from past residence information, and the customer service activities are made more efficient it can.
  • the speaker identification unit 25 identifies the first speaker related to the utterance detected by the voice detection unit 21 by the speaker identification process.
  • the first speaker identification information related to the first speaker is stored in the memory 13.
  • the staying information adjustment unit 33 includes first speaker identification information, speaker identification information related to the second speaker identified after the first speaker identification information is obtained by the speaker identification unit 25, and Are substantially the same, the retention information of the person related to the second speaker or the retention information of the clerk SA is integrated into the retention information of the person related to the first speaker or the retention information of the clerk SA.
  • the service monitoring system 100B and the service monitoring device 10B are provided separately if the flow line information and the stay information are once blocked and the stay information can be linked by speaker identification even if another person is detected. Can be handled as flow line information and staying information of the same person.
  • the customer service monitoring system 100B and the customer service monitoring device 10B determine whether or not they are the same person using the audio data, so compared with the case where it is determined whether or not they are the same person using face matching or the like using the image data. , Processing load can be reduced. Therefore, since the customer service monitoring system 100B and the customer service monitoring device 10B can improve the person identification accuracy related to the flow line information and the stay information, the customer service determination accuracy can be improved.
  • the processor may be physically configured in any manner. Further, if a programmable processor is used, the processing contents can be changed by changing the program, so that the degree of freedom in designing the processor can be increased.
  • the processor may be composed of one semiconductor chip or physically composed of a plurality of semiconductor chips. When configured by a plurality of semiconductor chips, each control of the first to third embodiments may be realized by different semiconductor chips. In this case, it can be considered that a plurality of semiconductor chips constitute one processor.
  • the processor may be configured by a member (capacitor or the like) having a function different from that of the semiconductor chip. Further, one semiconductor chip may be configured so as to realize the functions of the processor and other functions.
  • a plurality of processors may be configured by one processor.
  • each configuration may be realized by hardware or software. Also good.
  • This disclosure is useful for a customer service monitoring device, a customer service monitoring system, and a customer service monitoring method that can improve the accuracy of customer service determination.

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Accounting & Taxation (AREA)
  • Finance (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Marketing (AREA)
  • Human Computer Interaction (AREA)
  • Strategic Management (AREA)
  • Economics (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • General Business, Economics & Management (AREA)
  • Acoustics & Sound (AREA)
  • Development Economics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Alarm Systems (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Image Analysis (AREA)

Abstract

A customer attendance monitoring device provided with: a communication unit for acquiring speech data pertaining to a speech picked by a sound pickup device that picks up speech in a target area divided into a plurality of grids, and acquiring image data pertaining to an image captured by an imaging device that images the target area; a staying information generation unit for generating staying information about persons including an attendant or a customer and generating staying information about the attendant; and a customer attendance determination unit which, when the attendant and the customer stay for a first prescribed time in the same grid and an utterance of a duration of a second prescribed time or more is detected in the same grid, determines on the basis of the staying information about persons, the staying information about the attendant, and an utterance detection result, that the attendant is attending the customer in said same grid.

Description

接客モニタリング装置、接客モニタリングシステム、及び接客モニタリング方法Service monitoring device, service monitoring system, and service monitoring method
　本開示は、接客モニタリング装置、接客モニタリングシステム、及び接客モニタリング方法に関する。 This disclosure relates to a service monitoring device, a service monitoring system, and a service monitoring method.
　店舗内にカメラが設置され、このカメラがセキュリティ目的で使用されている。また、このカメラは、店舗運営の改善、例えば店員による接客の有無の確認に利用されることがある。 A camera is installed in the store, and this camera is used for security purposes. In addition, this camera may be used for improvement of store operation, for example, confirmation of the presence or absence of customer service by a store clerk.
　カメラを用いて接客の有無を判定するシステムとして、以下の活動状況分析システムが知られている（例えば、特許文献１参照）。この活動状況分析システムは、対象エリアを撮像するカメラによる撮像画像から移動体の滞留情報を取得し、店員が保持する携帯端末からＩＤ移動体の滞留情報を取得する。そして、この活動状況分析システムは、ＩＤ移動体と移動体とが近い地点で滞留した場合、接客ありと判定する。この接客の判定結果は、接客情報として店舗運営に活用される。 The following activity status analysis system is known as a system for determining the presence or absence of customer service using a camera (see, for example, Patent Document 1). This activity status analysis system acquires the staying information of a moving body from a captured image by a camera that captures a target area, and acquires the staying information of an ID moving body from a portable terminal held by a store clerk. Then, this activity state analysis system determines that there is a customer service when the ID mobile body and the mobile body stay at a nearby point. The determination result of this customer service is utilized for store management as customer service information.
　ところで、活動状況分析システムは、店員と客とが近くに所在すれば、接客ありと判定し、接客の判定精度が低下することがある。例えば、店員と客とが背中合わせで近くに所在する場合でも接客中と判定されることがあり、接客の判定精度が向上することが望ましい。 By the way, the activity situation analysis system may determine that there is a customer service if the store clerk and the customer are nearby, and the customer service determination accuracy may decrease. For example, even when a store clerk and a customer are located close to each other, it may be determined that the customer is in service, and it is desirable that the determination accuracy of the customer service is improved.
　本開示は、接客の判定精度を向上することを目的とする。 This disclosure aims to improve customer service accuracy.
特許第５８７９６１６号公報Japanese Patent No. 5879616
　本開示の接客モニタリング装置は、複数のグリッドに区分される対象エリアの音声を収音する収音装置により収音された音声に係る音声データを取得し、対象エリアの画像を撮像する撮像装置により撮像された画像に係る画像データを取得する通信部と、音声データを基に、発話を検出する発話検出部と、画像データを基に、接客者又は客を含む人物の滞留情報を生成し、対象エリア内に所在する各接客者が所持する携帯端末の位置情報を基に、接客者の滞留情報を生成する滞留情報生成部と、人物の滞留情報、接客者の滞留情報、及び発話の検出結果を基に、同一のグリッドに接客者と客とが第１の所定時間滞留し、且つ、同一のグリッドにおいて第２の所定時間以上発話が検出された場合、同一のグリッドにおいて接客者が接客していると判定する接客判定部と、を備える。 The customer service monitoring device according to the present disclosure acquires sound data related to sound collected by a sound collecting device that collects sound of a target area divided into a plurality of grids, and captures an image of the target area. Based on the communication unit that acquires the image data related to the captured image, the utterance detection unit that detects the utterance based on the audio data, and based on the image data, the customer information or the residence information of the person including the customer is generated, Based on the location information of the mobile terminal possessed by each customer located in the target area, a residence information generation unit that generates customer residence information, and detection of person residence information, customer residence information, and utterances Based on the result, when the customer and the customer stay in the same grid for the first predetermined time and the utterance is detected for the second predetermined time or more in the same grid, the customer receives the customer in the same grid. Have And a customer determination unit determines that.
　本開示の接客モニタリングシステムは、複数のグリッドに区分される対象エリアの音声を収音する収音装置と、対象エリアの画像を撮像する撮像装置と、接客モニタリング装置と、を備える。接客モニタリング装置は、収音装置により収音された音声に係る音声データを取得し、撮像装置により撮像された画像に係る画像データを取得し、音声データを基に、発話を検出し、画像データを基に、接客者又は客を含む人物の滞留情報を生成し、対象エリア内に所在する各接客者が所持する携帯端末の位置情報を基に、接客者の滞留情報を生成し、人物の滞留情報、接客者の滞留情報、及び発話の検出結果を基に、同一のグリッドに接客者と客とが第１の所定時間滞留し、且つ、同一のグリッドにおいて第２の所定時間以上発話が検出された場合、同一のグリッドにおいて接客者が接客していると判定する。 The customer service monitoring system of the present disclosure includes a sound collection device that collects sound of a target area divided into a plurality of grids, an imaging device that captures an image of the target area, and a customer service monitoring device. The customer service monitoring device acquires sound data related to the sound collected by the sound collecting device, acquires image data related to the image picked up by the image pickup device, detects speech based on the sound data, and detects image data Based on the location information of the customer including the customer or the customer is generated, and the customer's residence information is generated based on the location information of the mobile terminal possessed by each customer in the target area. Based on the staying information, the staying person's staying information, and the utterance detection result, the staying person and the customer stay in the same grid for a first predetermined time, and the utterance exceeds the second predetermined time in the same grid. When it is detected, it is determined that the customer is serving in the same grid.
　本開示の接客モニタリング方法は、接客モニタリング装置における接客モニタリング方法であって、複数のグリッドに区分される対象エリアの音声を収音する収音装置により収音された音声に係る音声データを取得し、対象エリアの画像を撮像する撮像装置により撮像された画像に係る画像データを取得し、音声データを基に、発話を検出する発話検出部と、画像データを基に、接客者又は客を含む人物の滞留情報を生成し、対象エリア内に所在する各接客者が所持する携帯端末の位置情報を基に、接客者の滞留情報を生成し、人物の滞留情報、接客者の滞留情報、及び発話の検出結果を基に、同一のグリッドに接客者と客とが第１の所定時間滞留し、且つ、同一のグリッドにおいて第２の所定時間以上発話が検出された場合、同一のグリッドにおいて接客者が接客していると判定する。 The customer service monitoring method of the present disclosure is a customer service monitoring method in the customer service monitoring device, and acquires voice data related to a sound collected by a sound collecting device that collects sound of a target area divided into a plurality of grids. , Including image data related to an image captured by an imaging device that captures an image of the target area, and an utterance detection unit that detects an utterance based on audio data, and a customer or a customer based on the image data Based on the location information of the mobile terminal possessed by each customer who is located in the target area, the customer's residence information is generated, and the person's residence information, the customer's residence information, and Based on the utterance detection result, if the customer and the customer stay in the same grid for the first predetermined time and the utterance is detected for the second predetermined time or more in the same grid, the same grid is detected. Determines that the customer who is in customer service in de.
　本開示によれば、接客の判定精度を向上できる。 According to the present disclosure, it is possible to improve the accuracy of customer service determination.
図１は、第１の実施形態における接客モニタリングシステムを示すブロック図である。FIG. 1 is a block diagram showing a customer service monitoring system according to the first embodiment. 図２は、第１の実施形態における接客モニタリング装置の動作例を示すフローチャートである。FIG. 2 is a flowchart illustrating an operation example of the customer service monitoring apparatus according to the first embodiment. 図３は、店舗の天井から見た接客の様子の一例を示す模式図である。FIG. 3 is a schematic diagram illustrating an example of a customer service viewed from the ceiling of the store. 図４は、図３に対応し、店舗における接客の様子の一例を示す斜視図である。FIG. 4 is a perspective view corresponding to FIG. 3 and showing an example of customer service in a store. 図５は、第２の実施形態における接客モニタリングシステムの構成例を示すブロック図である。FIG. 5 is a block diagram illustrating a configuration example of the customer service monitoring system according to the second embodiment. 図６は、第２の実施形態における接客モニタリング装置の動作例を示すフローチャートである。FIG. 6 is a flowchart illustrating an operation example of the customer service monitoring apparatus according to the second embodiment. 図７は、客が行う特定の行動を説明するための模式図である。FIG. 7 is a schematic diagram for explaining a specific action performed by a customer. 図８Ａは、覗き込み判定の一例を説明するための模式図である。FIG. 8A is a schematic diagram for explaining an example of peeping determination. 図８Ｂは、きょろきょろ判定の一例を説明するための模式図である。FIG. 8B is a schematic diagram for explaining an example of the ginger determination. 図８Ｃは、傾げ判定の一例を説明するための模式図である。FIG. 8C is a schematic diagram for explaining an example of tilt determination. 図９は、接客指示情報の送信先となる店員を説明するための模式図である。FIG. 9 is a schematic diagram for explaining a store clerk serving as a transmission destination of customer service instruction information. 図１０は、図９に対応し、接客指示情報の送信先となる店員を説明するための模式図である。FIG. 10 corresponds to FIG. 9 and is a schematic diagram for explaining a store clerk serving as a destination of customer service instruction information. 図１１は、第３の実施形態における接客モニタリングシステムの構成例を示すブロック図である。FIG. 11 is a block diagram illustrating a configuration example of the customer service monitoring system according to the third embodiment. 図１２は、第３の実施形態における接客モニタリング装置の動作例を示すフローチャートである。FIG. 12 is a flowchart illustrating an operation example of the customer service monitoring apparatus according to the third embodiment. 図１３は、滞留情報の同一性を説明するための模式図である。FIG. 13 is a schematic diagram for explaining the identity of the staying information.
　以下、適宜図面を参照しながら、実施形態を詳細に説明する。但し、必要以上に詳細な説明は省略する場合がある。例えば、既によく知られた事項の詳細説明や実質的に同一の構成に対する重複説明を省略する場合がある。これは、以下の説明が不必要に冗長になることを避け、当業者の理解を容易にするためである。尚、添付図面及び以下の説明は、当業者が本開示を十分に理解するために提供されるものであり、これらにより請求の範囲に記載の主題を限定することは意図されていない。 Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of already well-known matters and repeated descriptions for substantially the same configuration may be omitted. This is to avoid the following description from becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the claimed subject matter.
　（第１の実施形態）
　［接客モニタリングシステムの構成］
　図１は、第１の実施形態における接客モニタリングシステム１００の構成例を示すブロック図である。接客モニタリングシステム１００は、携帯端末ＴＥ、マイクアレイ装置ＭＡ、カメラ装置ＣＡ、レコーダＲＣ、及び接客モニタリング装置１０を備える。携帯端末ＴＥ、マイクアレイ装置ＭＡ、カメラ装置ＣＡ、レコーダＲＣ、及び接客モニタリング装置１０は、ネットワークＮＴを介してデータ通信可能に接続される。 (First embodiment)
[Configuration of customer service monitoring system]
FIG. 1 is a block diagram illustrating a configuration example of a customer service monitoring system 100 according to the first embodiment. The customer service monitoring system 100 includes a mobile terminal TE, a microphone array device MA, a camera device CA, a recorder RC, and a customer service monitoring device 10. The mobile terminal TE, the microphone array device MA, the camera device CA, the recorder RC, and the customer service monitoring device 10 are connected via a network NT so that data communication is possible.
　ネットワークＮＴは、有線ネットワーク又は無線ネットワークを含んで形成される。ネットワークＮＴは、インターネット、セルラーネットワーク、ＷＡＮ（Ｗｉｄｅ　Ａｒｅａ　Ｎｅｔｗｏｒｋ）、ＬＡＮ（Ｌｏｃａｌ　Ａｒｅａ　Ｎｅｔｗｏｒｋ）、公衆回線、専用回線など、各種ネットワークを広く含む。 The network NT is formed including a wired network or a wireless network. The network NT widely includes various networks such as the Internet, a cellular network, a WAN (Wide Area Network), a LAN (Local Area Network), a public line, and a dedicated line.
　携帯端末ＴＥ（業務用端末）は、店舗ＳＴ内で働く店員ＳＡ（例えば図３，図４参照）によって所持され、ネットワークＮＴを介して他の通信装置（例えば接客モニタリング装置１０）と通信可能である。一方、店舗ＳＴに入店する客（例えば図３，図４参照）は、携帯端末ＴＥを所持していない。 The portable terminal TE (business terminal) is carried by a store clerk SA (for example, see FIGS. 3 and 4) working in the store ST, and can communicate with another communication device (for example, the customer service monitoring device 10) via the network NT. is there. On the other hand, customers who enter the store ST (see, for example, FIGS. 3 and 4) do not have the mobile terminal TE.
　携帯端末ＴＥは、接客モニタリング装置１０による接客判定のために、携帯端末ＴＥの位置を検出する。携帯端末ＴＥの店舗ＳＴ内での位置検出には、ＧＰＳ（Ｇｌｏｂａｌ　Ｐｏｓｉｔｉｏｎｉｎｇ　Ｓｙｓｔｅｍ）、無線ＬＡＮ、ＲＦＩＤ（Ｒａｄｉｏ　Ｆｒｅｑｕｅｎｃｙ　ＩＤｅｎｔｉｆｉｅｒ）、等を用いた位置検出技術が用いられてもよい。 The mobile terminal TE detects the position of the mobile terminal TE for the service determination by the service monitoring device 10. Position detection technology using GPS (Global Positioning System), wireless LAN, RFID (Radio Frequency IDentifier), or the like may be used to detect the position of the mobile terminal TE in the store ST.
　携帯端末ＴＥは、店舗内ＳＴ内に所定間隔で設置された各ビーコン端末との間でビーコン信号の通信を行い、受信したビーコン信号の送信元としてのビーコン端末との位置関係に基づいて、店舗ＳＴ内での位置を検出してもよい。ビーコン信号には、例えば、店舗ＳＴ内の基準点（例えば店舗ＳＴの中心位置）に対する相対位置を示す座標の情報が含まれてもよい。また、上記以外の手法により、携帯端末ＴＥの店舗ＳＴ内での位置が検出されてもよい。 The mobile terminal TE communicates a beacon signal with each beacon terminal installed at a predetermined interval in the in-store ST, and based on the positional relationship with the beacon terminal as the transmission source of the received beacon signal, You may detect the position in ST. The beacon signal may include, for example, information on coordinates indicating a relative position with respect to a reference point in the store ST (for example, the center position of the store ST). Further, the position of the mobile terminal TE in the store ST may be detected by a method other than the above.
　携帯端末ＴＥは、例えば、任意の通信手法（例えば、無線ＬＡＮ通信、公衆回線を用いた通信）により、携帯端末ＴＥが位置する位置情報を、接客モニタリング装置１０へ送信する。また、携帯端末ＴＥは、接客モニタリング装置１０から、接客に係る情報（例えば、接客指示情報や接客の要否を示す情報、客ＣＴの位置情報）を受信する。 The portable terminal TE transmits, to the customer service monitoring device 10, the position information where the portable terminal TE is located, for example, by an arbitrary communication method (for example, communication using wireless LAN communication or a public line). Further, the mobile terminal TE receives from the customer service monitoring device 10 information related to customer service (for example, customer service instruction information, information indicating the necessity of customer service, customer CT position information).
　携帯端末ＴＥは、加速度センサを備え、携帯端末ＴＥを所持する店員ＳＡがどの方向に移動したか（移動情報）を検出してもよい。接客モニタリング装置１０は、携帯端末ＴＥから取得した携帯端末ＴＥの位置情報を基に、又は携帯端末ＴＥの位置情報及び移動情報を基に、店員ＳＡや客ＣＴの動線情報や滞留情報を生成する。 The mobile terminal TE may include an acceleration sensor and detect in which direction the clerk SA carrying the mobile terminal TE has moved (movement information). The customer service monitoring device 10 generates flow line information and staying information for the clerk SA and the customer CT based on the position information of the mobile terminal TE acquired from the mobile terminal TE or based on the position information and movement information of the mobile terminal TE. To do.
　カメラ装置ＣＡは、例えば、店舗ＳＴの天井や壁等に設置された、画角が固定された固定カメラである。カメラ装置ＣＡは、自装置が設置された撮像空間である撮像エリアを撮像可能な監視カメラとして機能する。カメラ装置ＣＡは、一時的に映像（画像）を撮像してもよいし、継続して映像（画像）を撮像し、累積してもよい。 The camera device CA is a fixed camera with a fixed angle of view, for example, installed on the ceiling or wall of the store ST. The camera device CA functions as a monitoring camera that can image an imaging area that is an imaging space in which the device itself is installed. The camera device CA may temporarily capture a video (image), or may continuously capture and accumulate video (image).
　なお、カメラ装置ＣＡは、固定カメラに限られず、全方位カメラ、パン・チルト・ズーム動作自在なＰＴＺカメラであってもよい。カメラ装置ＣＡは、映像を撮像した時刻（撮像時刻）を映像データと対応付けて記憶する。また、カメラ装置ＣＡは、記憶した映像データ及び撮像時刻のデータを、ネットワークＮＷを介して接客モニタリング装置１０に送信する。尚、映像には、動画像及び静止画像の少なくとも一方を含まれる。 Note that the camera apparatus CA is not limited to a fixed camera, and may be an omnidirectional camera or a PTZ camera capable of pan / tilt / zoom operations. The camera device CA stores the time (image capturing time) when the image is captured in association with the image data. Further, the camera device CA transmits the stored video data and imaging time data to the customer service monitoring device 10 via the network NW. Note that the video includes at least one of a moving image and a still image.
　マイクアレイ装置ＭＡは、例えば店舗ＳＴの天井に設置された全方位マイクアレイ装置である。マイクアレイ装置ＭＡは、自装置が設置された収音空間（収音エリア）における全方位の音声を収音する。マイクアレイ装置ＭＡは、一時的に音声を収音してもよいし、継続して音声を収音し、累積してもよい。 The microphone array device MA is an omnidirectional microphone array device installed on the ceiling of the store ST, for example. The microphone array device MA collects sound in all directions in the sound collection space (sound collection area) where the device itself is installed. The microphone array apparatus MA may temporarily collect voice, or may continuously collect voice and accumulate it.
　マイクアレイ装置ＭＡは、例えば、中央に開口部が形成された筐体、及びこの開口部の周囲に円周方向に沿って同心円状に配置された複数のマイクロホンユニットを有する。マイクロホンユニット（以下、単にマイクロホンと称する）には、例えば高音質小型エレクトレットコンデンサマイクロホン（ＥＣＭ：Ｅｌｅｃｔｒｅｔ　Ｃｏｎｄｅｎｓｅｒ　Ｍｉｃｒｏｐｈｏｎｅ）が用いられる。 The microphone array apparatus MA includes, for example, a housing having an opening formed at the center, and a plurality of microphone units arranged concentrically around the opening in the circumferential direction. For example, a high sound quality small electret condenser microphone (ECM: Electric Condenser Microphone) is used for the microphone unit (hereinafter simply referred to as a microphone).
　尚、カメラ装置ＣＡが、例えばマイクアレイ装置ＭＡの筐体に形成された開口部に収容される全方位カメラである場合、撮像エリアと収音エリアは略同一となる。本実施形態では、撮像エリアと収音エリアとは、接客判定の対象エリアとなる。 Note that when the camera apparatus CA is an omnidirectional camera accommodated in an opening formed in the housing of the microphone array apparatus MA, for example, the imaging area and the sound collection area are substantially the same. In the present embodiment, the imaging area and the sound collection area are service determination target areas.
　マイクアレイ装置ＭＡは、収音した音声データを、収音した時刻（収音時刻）と対応付けて記憶する。また、マイクアレイ装置ＭＡは、記憶した音声データ及び収音時刻のデータを、ネットワークＮＷを介して接客モニタリング装置１０に送信する。 The microphone array device MA stores the collected sound data in association with the time of sound collection (sound collection time). Further, the microphone array apparatus MA transmits the stored voice data and sound collection time data to the customer service monitoring apparatus 10 via the network NW.
　レコーダＲＣは、各種データを記憶する。例えば、レコーダＲＣは、カメラ装置ＣＡにより撮像された映像データ及び映像時刻の情報を、接客モニタリング装置１０と共に、又は接客モニタリング装置１０の代わりに記憶してもよい。また、例えば、レコーダＲＣは、マイクアレイ装置ＭＡにより収音された音声データ及び収音時刻を、接客モニタリング装置１０と共に、又は接客モニタリング装置１０の代わりに記憶してもよい。 Recorder RC stores various data. For example, the recorder RC may store the video data captured by the camera device CA and the video time information together with the customer service monitoring device 10 or instead of the customer service monitoring device 10. Further, for example, the recorder RC may store the voice data collected by the microphone array device MA and the sound collection time together with the customer service monitoring device 10 or instead of the customer service monitoring device 10.
　接客モニタリング装置１０は、例えばマイクアレイ装置ＭＡ及びカメラ装置ＣＡが設置された店舗ＳＴの内又は外に設置され、例えば店舗ＳＴのバックヤードに設置される。接客モニタリング装置１０は、例えば、据置型のＰＣ（Ｐｅｒｓｏｎａｌ　Ｃｏｍｐｕｔｅｒ）、サーバである。また、接客モニタリング装置１０は、サーバ（例えばクラウドサーバ）として、店舗ＳＴから離れた遠隔地（例えば、各店舗ＳＴをまとめて監視するための監視センタ）に設置されてもよい。接客モニタリング装置１０は、例えば、監視者（例えば店舗ＳＴの管理者、監視センタの監視者）により監視され、表示が確認され、操作される。 The customer service monitoring device 10 is installed in or outside the store ST where the microphone array device MA and the camera device CA are installed, for example, in the back yard of the store ST. The customer service monitoring device 10 is, for example, a stationary PC (Personal Computer) or a server. Moreover, the customer service monitoring device 10 may be installed as a server (for example, a cloud server) in a remote place (for example, a monitoring center for collectively monitoring each store ST) away from the store ST. The customer service monitoring device 10 is monitored, for example, by a monitor (for example, a manager of the store ST, a monitor of the monitoring center), and the display is confirmed and operated.
　接客モニタリング装置１０は、詳細は後述するが、音声検出機能（発話検出機能）、音声検出結果の累積機能、人物の動線検出機能（滞留検出機能）、及び音声検出と人物の動線検出に基づく接客判定機能を有する。また、接客モニタリング装置１０は、携帯端末ＴＥが所在する位置（後述するＩＤ移動体の位置に相当）を検出する位置検出機能（測位測定機能）を有する。 As will be described in detail later, the customer service monitoring device 10 is used for voice detection function (speech detection function), voice detection result accumulation function, person flow line detection function (staying detection function), and voice detection and person flow line detection. Based on customer service determination function. In addition, the customer service monitoring device 10 has a position detection function (positioning measurement function) for detecting a position where the mobile terminal TE is located (corresponding to a position of an ID mobile body described later).
　尚、接客モニタリング装置１０は、ＰＣの代わりに、携帯電話機、タブレット端末、スマートフォン等の通信端末でもよい。 The customer service monitoring device 10 may be a communication terminal such as a mobile phone, a tablet terminal, or a smartphone instead of the PC.
　［接客モニタリング装置の構成］
　接客モニタリング装置１０は、通信部１１、制御部１２、メモリ１３、操作部１４、ディスプレイ装置１５、及びスピーカ装置１６を備える。また、接客モニタリング装置１０は、音声処理部２０、人物検出部３０、及び接客判定部４０を備える。音声処理部２０は、音声検出部２１及び検出結果累積部２２を備える。人物検出部３０は、移動体滞留情報生成部３１及びＩＤ滞留情報生成部３２を備える。接客判定部４０は、接客条件設定部４１及び接客行動抽出部４２を備える。 [Configuration of customer service monitoring device]
The customer service monitoring device 10 includes a communication unit 11, a control unit 12, a memory 13, an operation unit 14, a display device 15, and a speaker device 16. In addition, the customer service monitoring device 10 includes a voice processing unit 20, a person detection unit 30, and a customer service determination unit 40. The voice processing unit 20 includes a voice detection unit 21 and a detection result accumulation unit 22. The person detecting unit 30 includes a moving body staying information generating unit 31 and an ID staying information generating unit 32. The customer service determination unit 40 includes a customer service condition setting unit 41 and a customer service action extraction unit 42.
　通信部１１は、カメラ装置ＣＡが送信した撮像時刻を含む映像データ、及びマイクアレイ装置ＭＡが送信した収音時刻を含む音声データ、を受信して、制御部１２に出力する。また、通信部１１は、携帯端末ＴＥから、携帯端末ＴＥの位置情報を受信する。 The communication unit 11 receives the video data including the imaging time transmitted from the camera device CA and the audio data including the sound collection time transmitted from the microphone array device MA, and outputs them to the control unit 12. Further, the communication unit 11 receives position information of the mobile terminal TE from the mobile terminal TE.
　操作部１４は、ユーザの入力操作の内容を制御部１２に通知するためのユーザインターフェース（ＵＩ：Ｕｓｅｒ　Ｉｎｔｅｒｆａｃｅ）であり、例えばマウス、キーボード等のポインティングデバイスを含んで構成される。また、操作部１４は、例えばディスプレイ装置１５の画面に対応して配置され、ユーザの指やスタイラスペンによって入力操作が可能なタッチパネル又はタッチパッドを用いて構成されてもよい。 The operation unit 14 is a user interface (UI: User Interface) for notifying the control unit 12 of the content of the user's input operation, and includes a pointing device such as a mouse or a keyboard. The operation unit 14 may be configured using, for example, a touch panel or a touch pad that is arranged corresponding to the screen of the display device 15 and can be input with a user's finger or stylus pen.
　例えば、操作部１４は、グリッドＧＲの範囲を指定するための入力操作を受け付ける。また、操作部１４は、グリッドＧＲの範囲を変更するための入力操作を受け付けてもよい。 For example, the operation unit 14 receives an input operation for designating a range of the grid GR. The operation unit 14 may receive an input operation for changing the range of the grid GR.
　尚、グリッドＧＲとは、接客判定するための単位領域である。グリッドＧＲは、例えば、新商品エリア、シーズナブルエリア、定番エリア、店舗の特性に応じた様々に区分されるエリア、に応じて設定されてもよい。グリッドＧＲの大きさは、任意に設定されてもよい。尚、同様のコンセプトの商品が陳列されたエリアの範囲と関連付けて、１つのグリッドＧＲの範囲が定められてもよいし、同様のコンセプトの商品が陳列されたエリアの範囲と関連付けられずに、１つのグリッドＧＲの範囲が定められてもよい。また、グリッドＧＲの区分は、例えば、開店当初に定められて変更されなくてもよいし、店舗ＳＴ内でレイアウト変更される度に変更されてもよい。 The grid GR is a unit area for determining customer service. The grid GR may be set according to, for example, a new product area, a seasonable area, a standard area, or an area that is classified according to the characteristics of the store. The size of the grid GR may be set arbitrarily. In addition, the range of one grid GR may be defined in association with the range of the area where the products of the same concept are displayed, or without being related to the range of the area where the products of the similar concept are displayed, A range of one grid GR may be defined. Moreover, the division of the grid GR may be determined at the beginning of the opening of the store and may not be changed, or may be changed every time the layout is changed in the store ST.
　メモリ１３は、例えばＲＯＭ（Ｒｅａｄ　Ｏｎｌｙ　Ｍｅｍｏｒｙ）やＲＡＭ（Ｒａｎｄｏｍ　Ａｃｃｅｓｓ　Ｍｅｍｏｒｙ）を含んで構成されてもよい。メモリ１３は、接客モニタリング装置１０が動作する際、プログラムメモリ、データメモリ、ワークメモリとして機能する。 The memory 13 may include, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory). The memory 13 functions as a program memory, a data memory, and a work memory when the customer service monitoring device 10 operates.
　メモリ１３は、例えばＨＤＤ（Ｈａｒｄ　Ｄｉｓｋ　Ｄｒｉｖｅ）やＳＤＤ（Ｓｏｌｉｄ　Ｓｔａｔｅ　Ｄｒｉｖｅ）を含んで構成されてもよい。メモリ１３は、例えば、マイクアレイ装置ＭＡで収音された音声の音声データを収音時刻とともに記憶する。メモリ１３は、例えば、カメラ装置ＣＡで撮像された映像の映像データを撮像時刻とともに記憶する。また、メモリ１３は、各店員ＳＡが所持する携帯端末ＴＥの携帯ＩＤの情報を保持する。 The memory 13 may include, for example, an HDD (Hard Disk Drive) or an SDD (Solid State Drive). The memory 13 stores, for example, sound data of the sound collected by the microphone array device MA together with the sound collection time. The memory 13 stores, for example, video data of video captured by the camera device CA together with the imaging time. Further, the memory 13 holds information on the portable ID of the portable terminal TE possessed by each clerk SA.
　制御部１２は、接客モニタリング装置１０が有する各構成部を統括する。 The control unit 12 supervises each component of the customer service monitoring device 10.
　音声処理部２０は、マイクアレイ装置ＭＡで収音された音声の音声データに対して各種音声処理する。 The voice processing unit 20 performs various voice processes on voice data collected by the microphone array device MA.
　音声検出部２１は、店舗ＳＴ内の収音エリアにおける各位置（例えば後述する各分割領域）での音声の有無を検出する。音声検出部２１による音声検出は、例えば常時又は定期的に行われる。 The sound detection unit 21 detects the presence or absence of sound at each position (for example, each divided area described later) in the sound collection area in the store ST. The voice detection by the voice detection unit 21 is performed, for example, constantly or periodically.
　また、音声検出部２１は、指向性処理を行い、各分割領域ＤＡ（図３参照）に対して指向性を向ける。つまり、音声検出部２１は、各分割領域ＤＡを順番に指定し、音声データを基に、マイクアレイ装置ＭＡを基点として、指定された分割領域ＤＡの方向に対して、指向性を形成する。これにより、接客モニタリング装置１０は、各分割領域ＤＡでの音声の有無を検出できる。 Also, the voice detection unit 21 performs directivity processing and directs directivity to each divided area DA (see FIG. 3). That is, the voice detection unit 21 sequentially designates each divided area DA, and forms directivity in the direction of the designated divided area DA with the microphone array device MA as a base point based on the voice data. Thereby, the customer service monitoring apparatus 10 can detect the presence or absence of the sound in each divided area DA.
　ここで、収音エリア（例えば店舗ＳＴ全体）は、例えば、ｘ方向及びｙ方向（図３参照）において所定数に分割される。つまり、収音エリアは、店員ＳＡ又は客ＣＴの音声の有無（発話の有無）を判定するための分割領域ＤＡに区分される。音声検出部２１は、分割領域ＤＡ毎に、音声の有無を検出する。尚、１つの分割領域ＤＡの大きさは、１つのグリッドＧＲの大きさよりも小さい。 Here, the sound collection area (for example, the entire store ST) is divided into a predetermined number in the x direction and the y direction (see FIG. 3), for example. That is, the sound collection area is divided into divided areas DA for determining the presence / absence (utterance presence / absence) of the clerk SA or the customer CT. The sound detection unit 21 detects the presence or absence of sound for each divided area DA. Note that the size of one divided area DA is smaller than the size of one grid GR.
　音声の有無については、例えば、音声データの信号レベルが所定レベル以上である場合に音声がある（発話がある）と検出され、音声データの信号レベルが上記所定レベル未満である場合に音声が無し（発話が無し）と検出されてもよい。また、検出結果を時系列に累積し、一定時間の中、一定の割合で検知できた場合、音声がある（発話がある）と検出されてもよい。この場合、接客モニタリング装置１０は、店舗ＳＴ内で突発的な物音による誤検知を削減できる。 As for the presence or absence of sound, for example, when the signal level of the sound data is equal to or higher than a predetermined level, it is detected that there is sound (there is an utterance), and there is no sound when the signal level of the sound data is lower than the predetermined level. (No utterance) may be detected. In addition, when the detection results are accumulated in time series and can be detected at a constant rate within a predetermined time, it may be detected that there is a voice (there is an utterance). In this case, the customer service monitoring device 10 can reduce false detection due to sudden noise in the store ST.
　各分割領域ＤＡにおける音声の有無の検出結果は、制御部１２による制御により、ディスプレイ装置１５に表示されることで、音声ヒートマップとして可視化されてもよい。また、音声レベルの大きさを加味して、店舗ＳＴ内の各位置とその位置での音声レベルとを示す音声ヒートマップとして可視化されてもよい。 The detection result of the presence or absence of audio in each divided area DA may be visualized as an audio heat map by being displayed on the display device 15 under the control of the control unit 12. Further, in consideration of the size of the sound level, it may be visualized as a sound heat map indicating each position in the store ST and the sound level at that position.
　また、店員ＳＡ又は客ＣＴが発話すると１つ以上の分割領域ＤＡで発話検出されることがある。この場合、発話検出された分割領域ＤＡの全てにおいて発話が検出されたとしてもよいし、発話検出された分割領域ＤＡのうち検出された音声レベルが最大の１つの分割領域ＤＡにおいて発話が検出されたとしてもよい。 Also, when a clerk SA or a customer CT utters, the utterance may be detected in one or more divided areas DA. In this case, utterances may be detected in all the divided areas DA in which utterances are detected, or utterances are detected in one divided area DA having the maximum detected voice level among the divided areas DA in which utterances are detected. May be.
　検出結果累積部２２は、音声検出部２１により検出された音声の有無の検出結果（音声ヒートマップ）を、メモリ１３に累積して記憶する。例えば、検出結果累積部２２は、所定時間毎に生成された各分割領域ＤＡでの音声の有無の検出結果の情報を蓄積する。 The detection result accumulating unit 22 accumulates and stores in the memory 13 the detection result (sound heat map) of the presence or absence of sound detected by the sound detecting unit 21. For example, the detection result accumulating unit 22 accumulates information on the detection result of the presence / absence of sound in each divided area DA generated every predetermined time.
　人物検出部３０は、店舗ＳＴ内の各位置（例えば各グリッドＧＲ）での店員ＳＡ及び客ＣＴの存否を検出する。人物検出部３０は、例えば、店員ＳＡ及び客ＣＴの位置の検出情報を累積して、店員ＳＡ及び客ＣＴの時間経過に伴う移動の軌跡を示す動線情報を導出する。人物検出部３０は、動線情報に基づいて、店員ＳＡ及び客ＣＴの滞留を示す滞留情報を導出する。 The person detecting unit 30 detects the presence / absence of the clerk SA and the customer CT at each position (for example, each grid GR) in the store ST. For example, the person detection unit 30 accumulates detection information of the positions of the clerk SA and the customer CT, and derives flow line information indicating a trajectory of the movement of the clerk SA and the customer CT over time. Based on the flow line information, the person detection unit 30 derives retention information indicating the retention of the clerk SA and the customer CT.
　移動体滞留情報生成部３１は、カメラ装置ＣＡからの映像データを用いて、移動体（人物）の滞留情報を生成する。この移動体は、例えば、客ＣＴ又は店員ＳＡを含む。尚、移動体滞留情報生成部３１により生成された滞留情報から、ＩＤ滞留情報生成部３２により取得されたＩＤ滞留情報（店員ＳＡの滞留情報に相当）を差し引くと、客ＣＴの滞留情報が導出される。 The moving body staying information generating unit 31 generates staying information on the moving body (person) using the video data from the camera device CA. This moving body includes, for example, a customer CT or a store clerk SA. If the ID residence information (corresponding to the residence information of the clerk SA) acquired by the ID residence information generation unit 32 is subtracted from the residence information generated by the mobile body residence information generation unit 31, the residence information of the customer CT is derived. Is done.
　移動体滞留情報生成部３１は、カメラ装置ＣＡにより撮像されたリアルタイムの映像データ、又は、レコーダＲＣ若しくはメモリ１３に録画された過去の映像データを取得する。移動体滞留情報生成部３１は、取得された映像データから人物（移動体）を検出し、人物毎の追跡情報（検出位置情報）を生成する。この移動体追跡処理は、例えば、人物検出技術又は人物追跡技術などの画像認識技術を利用して行われる。 The moving body staying information generating unit 31 acquires real-time video data imaged by the camera device CA or past video data recorded in the recorder RC or the memory 13. The moving body staying information generating unit 31 detects a person (moving body) from the acquired video data, and generates tracking information (detected position information) for each person. This moving body tracking process is performed using, for example, an image recognition technique such as a person detection technique or a person tracking technique.
　移動体滞留情報生成部３１は、上記の追跡情報を、メモリ１３に蓄積する。追跡情報には、映像データの各フレームで検出された人物毎の検出位置、人物が検出されたフレームの撮像時刻から取得した人物毎の検出時刻、等に関する情報が含まれる。 The moving object staying information generating unit 31 stores the tracking information in the memory 13. The tracking information includes information on the detection position for each person detected in each frame of the video data, the detection time for each person acquired from the imaging time of the frame in which the person was detected, and the like.
　移動体滞留情報生成部３１は、生成された追跡情報に基づいて、映像データから検出された人物毎に動線情報を生成する。移動体滞留情報生成部３１は、動線生成処理を事後に行う場合には、メモリ１３から追跡情報を取得し、動線生成処理をリアルタイムで行う場合には、生成された追跡情報を直接用いる。 The moving body staying information generating unit 31 generates flow line information for each person detected from the video data based on the generated tracking information. The moving body staying information generation unit 31 acquires the tracking information from the memory 13 when performing the flow line generation processing after the fact, and directly uses the generated tracking information when performing the flow line generation processing in real time. .
　移動体滞留情報生成部３１は、人物毎に生成された動線情報を、メモリ１３に蓄積する。動線情報には、人物の移動の軌跡を示す移動ベクトル、移動ベクトルの変化点の位置及び時刻、等に関する情報が含まれる。移動体滞留情報生成部３１は、対象エリアにおける各グリッドＧＲの外周と動線との交差状況を判定するために、各グリッドＧＲの周辺部についても動線情報を生成してもよい。 The moving body staying information generating unit 31 stores the flow line information generated for each person in the memory 13. The flow line information includes information relating to a movement vector indicating the locus of movement of the person, the position and time of the change point of the movement vector, and the like. The moving body staying information generating unit 31 may generate the flow line information for the peripheral part of each grid GR in order to determine the intersection between the outer periphery of each grid GR and the flow line in the target area.
　移動体滞留情報生成部３１は、人物毎の動線情報に基づいて、対象エリア内における人物毎の滞留情報（どの位置にどの程度の時間留まっているかを示す情報）をグリッドＧＲ毎に生成する。滞留情報には、例えば、滞留しているグリッドＧＲの位置、その滞留時刻又は滞留時間、が含まれる。移動体滞留情報生成部３１は、生成された人物毎の滞留情報を、メモリ１３に蓄積する。 The moving body staying information generating unit 31 generates staying information for each person in the target area (information indicating how much time is staying at which position) for each grid GR based on the flow line information for each person. . The staying information includes, for example, the position of the staying grid GR, its staying time or staying time. The moving object staying information generating unit 31 stores the generated staying information for each person in the memory 13.
　ＩＤ滞留情報生成部３２は、携帯端末ＴＥを識別するためのＩＤ（携帯ＩＤ）を用いて、携帯端末ＴＥの滞留情報（ＩＤ滞留情報ともいう）を導出する。携帯端末ＴＥは、店員ＳＡにより所持されるので、ＩＤ滞留情報は、店員ＳＡに係る滞留情報とも言える。 The ID residence information generation unit 32 derives residence information (also referred to as ID residence information) of the mobile terminal TE using an ID (mobile ID) for identifying the mobile terminal TE. Since the portable terminal TE is possessed by the store clerk SA, the ID stay information can be said to be stay information related to the store clerk SA.
　ＩＤ滞留情報生成部３２は、通信部１１による携帯端末ＴＥとの通信によって、携帯端末ＴＥが検出された位置の情報（ＩＤ位置情報）を取得する。このＩＤ位置情報の検出技術には、例えば上述の技術が用いられる。ＩＤ滞留情報生成部３２は、取得されたＩＤ位置情報を、メモリ１３に蓄積する。 The ID retention information generation unit 32 acquires information (ID position information) on the position where the mobile terminal TE is detected by communication with the mobile terminal TE by the communication unit 11. As the ID position information detection technique, for example, the technique described above is used. The ID residence information generation unit 32 accumulates the acquired ID position information in the memory 13.
　ＩＤ滞留情報生成部３２は、取得されたＩＤ位置情報に基づいて、携帯端末ＴＥの動線の情報（ＩＤ動線情報）を生成する。ＩＤ滞留情報生成部３２は、動線生成処理を事後に行う場合には、一旦メモリ１３に蓄積されたＩＤ位置情報を取得し、動線生成処理をリアルタイムで行う場合には、検出されたＩＤ位置情報を直接用いる。 The ID residence information generation unit 32 generates flow line information (ID flow line information) of the mobile terminal TE based on the acquired ID position information. The ID staying information generation unit 32 acquires the ID position information once accumulated in the memory 13 when the flow line generation process is performed later, and the detected ID when the flow line generation process is performed in real time. Use location information directly.
　ＩＤ滞留情報生成部３２は、ＩＤ動線情報をメモリ１３に蓄積する。ＩＤ動線情報には、携帯端末ＴＥの移動の軌跡を示す移動ベクトル、その変化点の位置及び時刻などに関する情報が含まれる。ＩＤ滞留情報生成部３２は、対象エリアにおける各グリッドＧＲの外周と動線との交差状況を判定するために、各グリッドＧＲの周辺部についても動線情報を生成してもよい。 The ID retention information generation unit 32 stores ID flow line information in the memory 13. The ID flow line information includes information regarding a movement vector indicating the movement locus of the mobile terminal TE, the position of the change point, the time, and the like. The ID staying information generation unit 32 may generate flow line information for the peripheral part of each grid GR in order to determine the intersection between the outer periphery of each grid GR and the flow line in the target area.
　ＩＤ滞留情報生成部３２は、生成された人物毎（店員ＳＡ毎）のＩＤ動線情報に基づいて、対象エリア内におけるＩＤ滞留情報をグリッドＧＲ毎に生成する。ＩＤ滞留情報には、例えば、滞留しているグリッドＧＲの位置並びにその滞留時刻及び滞留時間などが含まれる。ＩＤ滞留情報生成部３２は、生成されたＩＤ滞留情報をメモリ１３に蓄積する。 The ID staying information generating unit 32 generates ID staying information in the target area for each grid GR based on the generated ID flow line information for each person (each store clerk SA). The ID staying information includes, for example, the position of the staying grid GR and the staying time and staying time thereof. The ID staying information generating unit 32 stores the generated ID staying information in the memory 13.
　接客判定部４０は、店舗ＳＴにおいて店員ＳＡが客ＣＴを接客しているか否かを判定する。 The customer service determination unit 40 determines whether or not the clerk SA is serving the customer CT at the store ST.
　接客条件設定部４１は、例えば操作部１４を介して受け付けた入力情報に基づいて、接客条件を設定する。接客条件には、例えば、接客判定を行うエリア（グリッドＧＲ）の選定、接客に係る動作（例えば後述する図２に示す接客に係る動作）、接客に係る動作で用いる各種閾値の設定、が含まれる。つまり、接客条件は、例えば、どのエリアで店員ＳＡや客ＣＴがどのような行動（接客行動）をしたら接客していると判定するかを定めたものである。 The customer service condition setting unit 41 sets customer service conditions based on input information received via the operation unit 14, for example. The customer service condition includes, for example, selection of an area (grid GR) for performing customer service determination, operation related to customer service (for example, operation related to customer service shown in FIG. 2 described later), and setting of various threshold values used in operations related to customer service. It is. In other words, the customer service condition defines, for example, in which area the store clerk SA or customer CT determines what kind of behavior (customer service behavior) is determined to be in service.
　接客行動抽出部４２は、設定された接客条件を満たす接客行動を抽出する。接客条件を満たす接客行動が抽出された場合、店員ＳＡが客ＣＴを接客していることを示す。接客行動抽出部４２は、店舗ＳＴ内での接客行動の有無を抽出する。 The customer service behavior extraction unit 42 extracts the customer service behavior that satisfies the set customer service conditions. When the customer service condition satisfying the customer service is extracted, it indicates that the clerk SA is serving the customer CT. The customer service behavior extraction unit 42 extracts presence / absence of customer service behavior in the store ST.
　ディスプレイ装置１５は、カメラ装置ＣＡが撮像したリアルタイム又は過去の映像データを画面に表示する。 The display device 15 displays real-time or past video data captured by the camera device CA on the screen.
　スピーカ装置１６は、マイクアレイ装置ＭＡが収音した音声データ、又は指向角（θＭＡｈ，θＭＡｖ）に指向性が形成されたマイクアレイ装置ＭＡが収音した音声データ、を音声出力する。尚、ディスプレイ装置１５及びスピーカ装置１６は、接客モニタリング装置１０とは別体の装置として構成されてもよい。尚、指向角θＭＡｈは、マイクアレイ装置ＭＡから音源位置（例えば分割領域ＤＡ）への水平角を示し、指向角θＭＡｖは、マイクアレイ装置ＭＡから音源位置（例えば分割領域ＤＡ）への垂直角を示す。 The speaker device 16 outputs the sound data picked up by the microphone array device MA or the sound data picked up by the microphone array device MA having directivity formed at the directivity angles (θMAh, θMAv). The display device 15 and the speaker device 16 may be configured as separate devices from the customer service monitoring device 10. The directivity angle θMAh indicates the horizontal angle from the microphone array apparatus MA to the sound source position (for example, the divided area DA), and the directivity angle θMAv indicates the vertical angle from the microphone array apparatus MA to the sound source position (for example, the divided area DA). Show.
　尚、接客モニタリング装置１０は、プロセッサがメモリ１３に保持されたプログラムを実行することで、各構成部（例えば、制御部１２、音声処理部２０、人物検出部３０、接客判定部４０）の機能を実現する。プロセッサは、ＣＰＵ（Ｃｅｎｔｒａｌ　Ｐｒｏｃｅｓｓｉｎｇ　Ｕｎｉｔ）、ＤＳＰ（Ｄｉｇｉｔａｌ　Ｓｉｇｎａｌ　Ｐｒｏｃｅｓｓｏｒ）、ＧＰＵ（Ｇｒａｐｈｉｃａｌ　Ｐｒｏｃｅｓｓｉｎｇ　Ｕｎｉｔ）、等を含む。 In the service monitoring apparatus 10, the function of each constituent unit (for example, the control unit 12, the voice processing unit 20, the person detection unit 30, and the service determination unit 40) is executed by the processor executing a program stored in the memory 13. To realize. The processor includes a CPU (Central Processing Unit), a DSP (Digital Signal Processor), a GPU (Graphical Processing Unit), and the like.
　［接客モニタリングシステムの動作］
　次に、接客モニタリングシステム１００の動作例について説明する。 [Operation of customer service monitoring system]
Next, an operation example of the customer service monitoring system 100 will be described.
　図２は、接客モニタリング装置１０の動作例を示すフローチャートである。 FIG. 2 is a flowchart showing an operation example of the customer service monitoring device 10.
　接客条件設定部４１は、例えば操作部１４を介して得られた入力情報を基に、接客条件を設定する（Ｓ１１）。ここでの接客条件は、例えば、Ｓ１４～Ｓ１９に示す処理を含む。 The customer service condition setting unit 41 sets customer service conditions based on, for example, input information obtained via the operation unit 14 (S11). The customer service conditions here include, for example, the processes shown in S14 to S19.
　移動体滞留情報生成部３１は、カメラ装置ＣＡから映像データを取得し、移動体の滞留情報を生成する（Ｓ１２）。また、ＩＤ滞留情報生成部３２は、携帯ＩＤで識別される携帯端末ＴＥから各店員ＳＡの位置の情報を取得し、ＩＤ滞留情報を生成する（Ｓ１２）。 The moving body staying information generating unit 31 acquires video data from the camera device CA and generates staying information on the moving body (S12). Further, the ID residence information generation unit 32 acquires information on the position of each clerk SA from the mobile terminal TE identified by the mobile ID, and generates ID residence information (S12).
　尚、移動体の滞留情報及びＩＤ滞留情報は、リアルタイムの情報でもよいし、過去の情報でもよい。尚、移動体の滞留情報の生成及びＩＤ滞留情報の生成は、常時行われてもよいし、定期的に行われてもよい。 Note that the staying information and ID staying information of the moving body may be real-time information or past information. Note that the generation of the stay information of the moving body and the generation of the ID stay information may be performed constantly or periodically.
　音声処理部２０は、通信部１１を介して、マイクアレイ装置ＭＡから音声データを取得する。また、音声処理部２０は、取得された音声データに基づいて、店舗ＳＴ内の分割領域ＤＡ毎の音声の有無を検出する（Ｓ１３）。尚、分割領域ＤＡ毎の音声の有無を検出は、常時行われてもよいし、定期的に行われてもよい。 The voice processing unit 20 acquires voice data from the microphone array device MA via the communication unit 11. Moreover, the audio | voice process part 20 detects the presence or absence of the audio | voice for every division area DA in the shop ST based on the acquired audio | voice data (S13). The detection of the presence or absence of sound for each divided area DA may be performed constantly or periodically.
　接客条件設定部４１は、例えば操作部１４を介して得られた入力情報（グリッドを選択するための操作情報）を基に、店舗ＳＴにおける、分析対象（接客判定対象）のグリッドＧＲを選択する（Ｓ１４）。 The customer service condition setting unit 41 selects a grid GR to be analyzed (customer determination target) in the store ST based on, for example, input information (operation information for selecting a grid) obtained via the operation unit 14. (S14).
　接客行動抽出部４２は、移動体の滞留情報及びＩＤ滞留情報を基に、分析対象のグリッドＧＲ内に、店員ＳＡと客ＣＴとが所在するか否かを判定する（Ｓ１５）。例えば、接客行動抽出部４２は、移動体の滞留情報において複数の移動体が検出され、ＩＤ滞留情報において、移動体の滞留情報における移動体の数よりも少なく、且つ１つ以上のＩＤ移動体が検出された場合、分析対象のグリッドＧＲ内に店員ＳＡと客ＣＴとが所在すると判定する。尚、Ｓ１５では、一定時間、店員ＳＡと客ＣＴとが所在するか否かが判定されてもよい。 The customer service behavior extraction unit 42 determines whether or not the store clerk SA and the customer CT are located in the grid GR to be analyzed based on the stay information and ID stay information of the moving body (S15). For example, the customer service behavior extraction unit 42 detects a plurality of moving bodies in the staying information of the moving body, and the ID staying information has less than the number of moving bodies in the staying information of the moving body and one or more ID moving bodies. Is detected, it is determined that the clerk SA and the customer CT are located in the grid GR to be analyzed. In S15, it may be determined whether or not the clerk SA and the customer CT exist for a certain period of time.
　Ｓ１５において分析対象のグリッドＧＲ内に店員ＳＡと客ＣＴとが所在すると判定された場合、音声処理部２０は、グリッドＧＲ内の任意の分割領域ＤＡにおいて、一定時間、音声が検出されたか否かを判定する（Ｓ１６）。ここでは、音声検出の対象となる分割領域ＤＡが順次変更されて、それぞれの分割領域ＤＡで音声の有無が判定される。尚、Ｓ１６では、一定時間、音声が検出されたか否かが判定されてもよい。 When it is determined in S15 that the clerk SA and the customer CT are located in the analysis target grid GR, the voice processing unit 20 determines whether or not voice is detected for a certain period of time in an arbitrary divided area DA in the grid GR. Is determined (S16). Here, the divided areas DA to be subjected to sound detection are sequentially changed, and the presence or absence of sound is determined in each divided area DA. In S16, it may be determined whether or not a voice is detected for a certain time.
　Ｓ１６において、グリッドＧＲ内の任意の分割領域ＤＡにおいて一定時間音声が検出された場合、接客行動抽出部４２は、グリッドＧＲにおいて店員ＳＡが客ＣＴを接客していると判定する（Ｓ１７）。この音声には、例えば、店員ＳＡの声又は客ＣＴの声が含まれる。尚、ここでは音声によって話者識別してもしなくてもよいが、音声が検出された分割領域ＤＡの位置に応じて、店員ＳＡの声か客ＣＴの声かが識別可能であってもよい。 In S16, when voice is detected for a certain period of time in an arbitrary divided area DA in the grid GR, the customer service action extraction unit 42 determines that the clerk SA is serving the customer CT in the grid GR (S17). This voice includes, for example, the voice of the clerk SA or the voice of the customer CT. Here, the speaker may or may not be identified by voice, but depending on the position of the divided area DA where the voice is detected, it may be possible to identify the voice of the clerk SA or the voice of the customer CT. .
　Ｓ１５において、分析対象のグリッドＧＲ内に店員ＳＡと客ＣＴとの少なくとも一方が所在しない場合、又は、Ｓ１６において、グリッドＧＲ内の全分割領域ＤＡにおいて一定時間音声が検出されなかった場合、接客行動抽出部４２は、店員ＳＡが客ＣＴを接客していないと判定する（Ｓ１８）。 If at least one of the clerk SA and the customer CT is not present in the grid GR to be analyzed in S15, or if no voice is detected in all the divided areas DA in the grid GR in S16, The extraction unit 42 determines that the clerk SA is not serving the customer CT (S18).
　Ｓ１７又はＳ１８による接客の有無の判定後、接客判定部４０は、分析対象の全グリッドＧＲにおいて接客の有無の判定が終了したか否かを判定する（Ｓ１９）。全グリッドＧＲでの接客の有無の判定が終了した場合（Ｓ１９のＹｅｓ）、接客モニタリング装置１０は、図２の処理を終了する。少なくとも１つのグリッドＧＲでの接客の有無の判定が終了していない場合（Ｓ１９のＮｏ）、Ｓ１４に進み、接客判定部４０は、次のグリッドＧＲを分析対象のグリッドとして選択する。 After the determination of the presence / absence of customer service in S17 or S18, the customer service determination unit 40 determines whether the determination of the presence / absence of customer service is completed in all the grids GR to be analyzed (S19). When the determination of the presence / absence of customer service in all grids GR is completed (Yes in S19), the customer service monitoring device 10 ends the process of FIG. When the determination of the presence / absence of customer service in at least one grid GR has not been completed (No in S19), the process proceeds to S14, and the customer service determination unit 40 selects the next grid GR as the analysis target grid.
　このような図２の処理によれば、接客モニタリング装置１０は、人物（移動体）の滞留情報や携帯ＩＤの滞留情報に基づいて、店舗ＳＴ内のどの位置に、どの時間帯に、店員ＳＡや客ＣＴが所在するかを把握できる。また、接客モニタリング装置１０は、分割領域ＤＡでの音声の有無の情報を、接客の有無の判定に用いることができる。よって、接客モニタリング装置１０は、単に同じ時間帯に店員ＳＡと客ＣＴとの接近を検出することで接客ありと判定することを抑制できる。 According to the process of FIG. 2, the customer service monitoring device 10 determines the position of the store clerk SA at which position in the store ST and at which time zone based on the stay information of the person (mobile body) and the stay information of the portable ID. And whether the customer CT is located. Further, the customer service monitoring device 10 can use the information on the presence / absence of voice in the divided area DA to determine the presence / absence of customer service. Therefore, the customer service monitoring device 10 can suppress the determination that there is customer service by simply detecting the approach between the clerk SA and the customer CT in the same time zone.
　次に、店舗ＳＴにおける接客の様子について説明する。 Next, the state of customer service at the store ST will be described.
　図３は、店舗ＳＴの天井から見た接客の様子の一例を示す模式図である。図４は、図３と対応しており、店舗ＳＴにおける接客の様子の一例を示す斜視図である。店員ＳＡは、例えば店舗ＳＴ内の任意の位置に所在する。客ＣＴは、店舗ＳＴ内の商品棚ＳＳに陳列された商品を購入や確認するために、商品棚ＳＳへ近寄る。 FIG. 3 is a schematic diagram showing an example of a customer service viewed from the ceiling of the store ST. FIG. 4 corresponds to FIG. 3 and is a perspective view showing an example of a state of customer service in the store ST. The clerk SA is located at an arbitrary position in the store ST, for example. The customer CT approaches the product shelf SS in order to purchase and confirm the product displayed on the product shelf SS in the store ST.
　尚、店舗ＳＴは、スーパーマーケット、ホームセンター、家具店、百貨店、等である。店舗ＳＴが比較的広い店舗である場合、接客モニタリングシステム１００は、一層役立つ。 The stores ST are supermarkets, home centers, furniture stores, department stores, and the like. When the store ST is a relatively wide store, the customer service monitoring system 100 is more useful.
　図３，図４では、店舗ＳＴ内に商品棚ＳＳ（ＳＳ１～ＳＳ６）が６個設置されている。また、店舗ＳＴには、均等に区分されたグリッドＧＲ（ＧＲ１～ＧＲ１６）が、４×４＝１６個含まれている。 3 and 4, six product shelves SS (SS1 to SS6) are installed in the store ST. Further, the store ST includes 4 × 4 = 16 grids GR (GR1 to GR16) equally divided.
　商品棚ＳＳ１の周囲は、グリッドＧＲ１，ＧＲ２，ＧＲ５，ＧＲ６の領域に含まれる。グリッドＧＲ１には、店員ＳＡ１が所在し、店舗ＳＴ全体を見渡している。店員ＳＡ１は、来店された客ＣＴ等に対して、挨拶等をしている。図３では、ＧＲ１における領域Ｄ１の各分割領域ＤＡにおいて、音声（発話）が検出されている。 The periphery of the product shelf SS1 is included in the areas of grids GR1, GR2, GR5, GR6. The grid GR1 has a store clerk SA1 overlooking the entire store ST. The store clerk SA1 greets the customer CT or the like who has visited the store. In FIG. 3, voice (utterance) is detected in each divided area DA of the area D1 in GR1.
　この場合、人物検出部３０は、カメラ装置ＣＡにより撮像された映像データを基に、グリッドＧＲ１に人物が１名所在することを検出する（図２のＳ１２）。人物検出部３０は、店員ＳＡ１が所持する携帯端末ＴＥの携帯ＩＤを基に、店員ＳＡ１がグリッドＧＲ１内に所在することを検出する（図２のＳ１２）。よって、人物検出部３０は、グリッドＧＲ１内に店員ＳＡ１が１人おり、客ＣＴがいないことを検出する（図２のＳ１５のＮｏ）。また、音声処理部２０は、グリッドＧＲ１において、店員ＳＡ１の所在位置を含む周囲の分割領域ＤＡにおいて、店員ＳＡ１が発する音声を検出する。この場合、接客判定部４０は、グリッドＧＲ１において、接客中でないと判定する（図２のＳ１８）。 In this case, the person detection unit 30 detects that one person is present in the grid GR1 based on the video data captured by the camera device CA (S12 in FIG. 2). The person detection unit 30 detects that the clerk SA1 is located in the grid GR1 based on the portable ID of the portable terminal TE possessed by the clerk SA1 (S12 in FIG. 2). Therefore, the person detection unit 30 detects that there is one clerk SA1 in the grid GR1 and no customer CT (No in S15 in FIG. 2). In addition, the voice processing unit 20 detects the voice generated by the clerk SA1 in the surrounding divided area DA including the location of the clerk SA1 in the grid GR1. In this case, the customer service determination unit 40 determines that the customer is not in service in the grid GR1 (S18 in FIG. 2).
　また、商品棚ＳＳ２，ＳＳ３の間でありグリッドＧＲ７の領域に、店員ＳＡ２と客ＣＴ１とが所在する。図４に示すように、店員ＳＡ２は商品棚ＳＳ２の方を向いており、客ＣＴ１は商品棚ＳＳ３の方を向いており、店員ＳＡ２と客ＣＴ１とが背中合わせとなっている。ここでは、店員ＳＡ２と客ＣＴ１とは会話していない。 In addition, the store clerk SA2 and the customer CT1 are located in the area of the grid GR7 between the product shelves SS2 and SS3. As shown in FIG. 4, the clerk SA2 faces the product shelf SS2, the customer CT1 faces the product shelf SS3, and the clerk SA2 and the customer CT1 are back to back. Here, the clerk SA2 and the customer CT1 are not talking.
　この場合、人物検出部３０は、カメラ装置ＣＡにより撮像された映像データを基に、グリッドＧＲ７に人物が２名所在することを検出する（図２のＳ１２）。人物検出部３０は、店員ＳＡ２が所持する携帯端末ＴＥの携帯ＩＤを基に、店員ＳＡ２がグリッドＧＲ７内に所在することを検出する（図２のＳ１２）。よって、人物検出部３０は、店員ＳＡ２が１人と客ＣＴ１が１人とがグリッドＧＲ７内に所在することを検出する（図２のＳ１５のＹｅｓ）。また、音声処理部２０は、グリッドＧＲ７において、店員ＳＡ２及び客ＣＴ１の所在位置を含む周囲の分割領域ＤＡにおいて、店員ＳＡ２及び客ＣＴ１が発する音声をいずれも検出しない（図２のＳ１６のＮｏ）。よって、接客判定部４０は、グリッドＧＲ１において、接客中でないと判定する（図２のＳ１８）。 In this case, the person detection unit 30 detects that there are two sights on the grid GR7 based on the video data captured by the camera device CA (S12 in FIG. 2). The person detection unit 30 detects that the clerk SA2 is located in the grid GR7 based on the portable ID of the portable terminal TE possessed by the clerk SA2 (S12 in FIG. 2). Therefore, the person detection unit 30 detects that one clerk SA2 and one customer CT1 are present in the grid GR7 (Yes in S15 of FIG. 2). Further, in the grid GR7, the sound processing unit 20 does not detect any sound emitted by the store clerk SA2 and the customer CT1 in the surrounding divided area DA including the locations of the store clerk SA2 and the customer CT1 (No in S16 of FIG. 2). . Therefore, the customer service determination unit 40 determines that the customer is not in service in the grid GR1 (S18 in FIG. 2).
　また、商品棚ＳＳ４，ＳＳ５の間でありグリッドＧＲ１４の領域に、店員ＳＡ３と客ＣＴ２とが所在する。図４に示すように、店員ＳＡ３及び客ＣＴ２は同じ商品棚ＳＳ５の方を向いている。店員ＳＡ３と客ＣＴ２とは、例えば、商品棚ＳＳ５に陳列された商品に係る会話をしている。図３では、ＧＲ１４における領域Ｄ２の各分割領域ＤＡにおいて、音声（発話）が検出されている。 In addition, the store clerk SA3 and the customer CT2 are located in the area of the grid GR14 between the product shelves SS4 and SS5. As shown in FIG. 4, the clerk SA3 and the customer CT2 face the same product shelf SS5. For example, the clerk SA3 and the customer CT2 have a conversation related to the products displayed on the product shelf SS5. In FIG. 3, voice (speech) is detected in each divided area DA of the area D2 in the GR14.
　この場合、人物検出部３０は、カメラ装置ＣＡにより撮像された映像データを基に、グリッドＧＲ１４に人物が２名所在することを検出する（図２のＳ１２）。人物検出部３０は、店員ＳＡ３が所持する携帯端末ＴＥの携帯ＩＤを基に、店員ＳＡ３がグリッドＧＲ１４内に所在することを検出する（図２のＳ１２）。よって、人物検出部３０は、店員ＳＡ３が１人と客ＣＴ２が１人とがグリッドＧＲ１内に所在することを検出する（図２のＳ１５のＹｅｓ）。また、音声処理部２０は、グリッドＧＲ７において、店員ＳＡ３及び客ＣＴ２の所在位置を含む周囲の分割領域ＤＡにおいて、店員ＳＡ３及び客ＣＴ２の少なくとも一方が発する音声を検出する（図２のＳ１６のＹｅｓ）。よって、接客判定部４０は、グリッドＧＲ１４において、店員ＳＡ３が客ＣＴ２を接客中であると判定する（図２のＳ１７）。 In this case, the person detection unit 30 detects that there are two sights on the grid GR14 based on the video data captured by the camera device CA (S12 in FIG. 2). The person detection unit 30 detects that the clerk SA3 is located in the grid GR14 based on the mobile ID of the mobile terminal TE possessed by the clerk SA3 (S12 in FIG. 2). Therefore, the person detection unit 30 detects that one clerk SA3 and one customer CT2 are located in the grid GR1 (Yes in S15 in FIG. 2). Further, in the grid GR7, the voice processing unit 20 detects a voice uttered by at least one of the clerk SA3 and the customer CT2 in the surrounding divided area DA including the location of the clerk SA3 and the customer CT2 (Yes in S16 of FIG. 2). ). Therefore, the customer service determination unit 40 determines that the clerk SA3 is serving the customer CT2 in the grid GR14 (S17 in FIG. 2).
　［効果等］
　このように、本実施形態の接客モニタリングシステム１００は、携帯端末ＴＥ、マイクアレイ装置ＭＡ、カメラ装置ＣＡ、及び接客モニタリング装置１０を備える。接客モニタリング装置１０は、通信部１１、音声処理部２０、人物検出部３０、及び接客判定部４０を備える。 [Effects]
As described above, the customer service monitoring system 100 according to this embodiment includes the mobile terminal TE, the microphone array device MA, the camera device CA, and the customer service monitoring device 10. The customer service monitoring device 10 includes a communication unit 11, a voice processing unit 20, a person detection unit 30, and a customer service determination unit 40.
　通信部１１は、複数のグリッドＧＤに区分される対象エリアの音声を収音するマイクアレイ装置ＭＡにより収音された音声に係る音声データを取得する。通信部１１は、対象エリアの画像を撮像するカメラ装置ＣＡにより撮像された画像に係る画像データを取得する。音声処理部２０は、音声データを基に、発話を検出する。人物検出部３０は、画像データを基に、店員ＳＡ又は客ＣＴを含む人物の滞留情報を生成する。人物検出部３０は、対象エリア内に所在する各店員ＳＡが所持する携帯端末ＴＥの位置情報を基に、店員ＳＡの滞留情報を生成する。接客判定部４０は、人物の滞留情報、店員ＳＡの滞留情報、発話検出結果を基に、同一のグリッドＧＲに店員ＳＡと客ＣＴとが所定時間ｔ１滞留し、且つ、同一のグリッドＧＲにおいて所定時間ｔ２以上発話が検出された場合、同一のグリッドＧＲにおいて店員ＳＡが接客していると判定する。 The communication unit 11 acquires audio data related to the sound collected by the microphone array device MA that collects the sound of the target area divided into a plurality of grids GD. The communication unit 11 acquires image data relating to an image captured by the camera device CA that captures an image of the target area. The voice processing unit 20 detects an utterance based on the voice data. The person detection unit 30 generates residence information of a person including the clerk SA or the customer CT based on the image data. The person detecting unit 30 generates the stay information of the clerk SA based on the position information of the mobile terminal TE possessed by each clerk SA located in the target area. The customer service determination unit 40 stores the clerk SA and the customer CT in the same grid GR for a predetermined time t1 based on the person's stay information, the stay information of the store clerk SA, and the utterance detection result. When the utterance is detected for the time t2 or more, it is determined that the store clerk SA is in contact with the same grid GR.
　音声処理部２０は、発話検出部の一例である。人物検出部３０は、滞留情報生成部の一例である。マイクアレイ装置ＭＡは、収音装置の一例である。カメラ装置ＣＡは、撮像装置の一例である。店員ＳＡは、接客者の一例である。所定時間ｔ１は、例えば５秒である。所定時間ｔ２は、例えば１秒である。 The voice processing unit 20 is an example of an utterance detection unit. The person detection unit 30 is an example of a staying information generation unit. The microphone array device MA is an example of a sound collection device. The camera device CA is an example of an imaging device. The salesclerk SA is an example of a customer service. The predetermined time t1 is, for example, 5 seconds. The predetermined time t2 is, for example, 1 second.
　これにより、接客モニタリングシステム１００及び接客モニタリング装置１０は、単に店員ＳＡと客ＣＴとが接近していることに応じて接客中であると判断せずに、マイクアレイ装置ＭＡにより収音された音声データを接客判定に活用できる。よって、接客モニタリングシステム１００及び接客モニタリング装置１０は、例えば、店員ＳＡと客ＣＴとが偶然接近したが会話せず接客していない場合には接客していないと判定するので、接客の判定精度を向上できる。また、接客モニタリングシステム１００及び接客モニタリング装置１０は、接客判定のために特別な操作を必要とせずに、接客判定できる。 As a result, the customer monitoring system 100 and the customer monitoring device 10 do not determine that the customer is in service in response to the fact that the clerk SA and the customer CT are close to each other, and the sound collected by the microphone array device MA is collected. Data can be used for customer service judgment. Therefore, the customer service monitoring system 100 and the customer service monitoring device 10 determine that the customer is not in service when, for example, the store clerk SA and the customer CT accidentally approach but do not talk and serve, so the determination accuracy of the service is increased. Can be improved. Moreover, the customer service monitoring system 100 and the customer service monitoring apparatus 10 can perform customer service determination without requiring a special operation for customer service determination.
　（第２の実施形態）
　第１の実施形態では、店舗ＳＴ内の各位置での接客の有無を判定することを説明した。第２の実施形態では、店舗ＳＴ内の各位置での接客の要否を判定し、接客が必要な場合に店員へ通知することを説明する。 (Second Embodiment)
In the first embodiment, it has been described that the presence / absence of customer service at each position in the store ST is determined. In the second embodiment, it will be described that the necessity of customer service at each position in the store ST is determined and the customer clerk is notified when customer service is required.
　［接客モニタリングシステムの構成］
　図５は、第２の実施形態における接客モニタリングシステム１００Ａの構成例を示すブロック図である。図５の接客モニタリングシステム１００Ａは、図１に示した接客モニタリングシステム１００と比較すると、接客モニタリング装置１０の代わりに接客モニタリング装置１０Ａを備える点が異なる。図５の接客モニタリングシステム１００Ａにおいて、図１に示した接客モニタリングシステム１００と同様の構成については、同一の符号を付し、説明を省略又は簡略化する。 [Configuration of customer service monitoring system]
FIG. 5 is a block diagram illustrating a configuration example of the customer service monitoring system 100A according to the second embodiment. 5 is different from the customer monitoring system 100 shown in FIG. 1 in that a customer monitoring device 10A is provided instead of the customer monitoring device 10. In the customer service monitoring system 100A of FIG. 5, the same components as those of the customer service monitoring system 100 shown in FIG. 1 are denoted by the same reference numerals, and description thereof is omitted or simplified.
　接客モニタリング装置１０Ａは、音声処理部２０Ａ、人物検出部３０、及び接客判定部４０Ａを備える。音声処理部２０Ａは、音声検出部２１、検出結果累積部２２、指向性合成部２３、及び音声判別部２４を備える。接客判定部４０Ａは、接客条件設定部４１Ａ、行動判定部４３、接客要否判定部４４、及び端末通知部４５を備える。 The customer service monitoring device 10A includes a voice processing unit 20A, a person detection unit 30, and a customer service determination unit 40A. The voice processing unit 20A includes a voice detection unit 21, a detection result accumulation unit 22, a directivity synthesis unit 23, and a voice discrimination unit 24. The service determination unit 40A includes a service condition setting unit 41A, a behavior determination unit 43, a service necessity determination unit 44, and a terminal notification unit 45.
　尚、接客判定部４０Ａは、第１の実施形態の接客判定部４０の機能を有してもよい。つまり、接客判定部４０Ａは、店舗ＳＴ内の各位置での接客の有無を判定してもよい。 The customer service determination unit 40A may have the function of the customer service determination unit 40 of the first embodiment. That is, the customer service determination unit 40A may determine the presence or absence of customer service at each position in the store ST.
　指向性合成部２３は、店舗ＳＴ内の特定の客ＣＴが所在する方向に音声の指向性を形成する。特定の客ＣＴとは、例えば、店員ＳＡにより接客されていない客であり、商品に関する説明を必要としている客、商品を探している客、又はその他サポートを必要としている客を含む。 The directivity synthesis unit 23 forms voice directivity in the direction where the specific customer CT in the store ST is located. The specific customer CT is, for example, a customer who is not served by the store clerk SA, and includes a customer who needs an explanation about a product, a customer who is looking for a product, or another customer who needs support.
　音声判別部２４は、所定の音声認識処理により、音声検出部２１により検出された音声に、特定のキーワードが含まれるか否かを判別する。特定のキーワードは、例えば、店員ＳＡによる接客を必要とすることを示す（明示又は示唆を含む）ワードを含む。特定のキーワードの具体例については後述する。 The voice discriminating unit 24 discriminates whether or not a specific keyword is included in the voice detected by the voice detecting unit 21 by a predetermined voice recognition process. The specific keyword includes, for example, a word (including explicit or suggestion) indicating that customer service by the salesclerk SA is required. Specific examples of specific keywords will be described later.
　接客条件設定部４１Ａは、例えば操作部１４を介して受け付けた入力情報に基づいて、接客条件を設定する。ここでの接客条件には、例えば、接客要否判定を行うエリア（グリッドＧＲ）の選定、接客要否判定に係る動作（例えば後述する図６に示す接客要否判定に係る動作）、接客要否判定で用いる各種閾値の設定、が含まれる。つまり、ここでの接客条件は、例えば、どのエリアで店員ＳＡや客ＣＴがどのような行動をしたら接客が必要と判定するかを定めたものである。 The customer service condition setting unit 41A sets the customer service condition based on the input information received via the operation unit 14, for example. The customer service conditions include, for example, selection of an area (grid GR) for determining customer service necessity, operation related to customer service necessity determination (for example, operation related to customer service necessity determination shown in FIG. 6 described later), customer service required. The setting of various threshold values used in the negative determination is included. In other words, the customer service condition here defines, for example, in which area the store clerk SA and the customer CT determine what kind of behavior the customer service is necessary.
　行動判定部４３は、店舗ＳＴ内に所在する客ＣＴが特定の行動をしたか否かを判定する。特定の行動は、例えば、客ＣＴが店員ＳＡによる接客を必要とすることを示す（明示又は示唆を含む）行動を含む。特定の行動の具体例については後述する。 The behavior determination unit 43 determines whether or not the customer CT located in the store ST has performed a specific behavior. The specific behavior includes, for example, behavior indicating that the customer CT needs customer service by the store clerk SA (including explicit or suggestion). Specific examples of specific actions will be described later.
　接客要否判定部４４は、店舗ＳＴ内の各位置（各グリッドＧＲ）での接客の要否を判定する。接客要否判定部４４は、例えば、店舗ＳＴ内に所在する客ＣＴが特定のキーワードを発した場合、又は、店舗ＳＴ内に所在する客ＣＴが特定の行動をした場合、接客が必要であると判定する。また、例えば、接客要否判定部４４は、店舗ＳＴ内に所在する客ＣＴが特定のキーワードを発しない場合、又は特定の行動をしなかった場合、接客が不要であると判定する。 The customer service necessity determination unit 44 determines the necessity of customer service at each position (each grid GR) in the store ST. For example, when the customer CT located in the store ST issues a specific keyword or when the customer CT located in the store ST performs a specific action, the customer service necessity determination unit 44 needs to serve the customer. Is determined. Further, for example, the customer service necessity determination unit 44 determines that the customer service is unnecessary when the customer CT located in the store ST does not issue a specific keyword or does not perform a specific action.
　端末通知部４５は、通信部１１を介して、接客が必要である旨を示す接客指示情報を、店員ＳＡが所持する携帯端末ＴＥの全部又は一部へ送信する。接客指示情報は、例えば、接客が必要な客ＣＴの所在位置の情報や客ＣＴに対して接客を指示する情報を含む。客ＣＴの所在位置の情報は、例えば、客ＣＴの所在位置を含むグリッドＧＲの位置情報であってもよいし、客ＣＴの所在位置を含む分割領域ＤＡの位置情報であってもよい。客ＣＴの所在位置の情報は、店舗ＳＴ内を示すマップを用いて直感的に示されてもよい。 The terminal notification unit 45 transmits, through the communication unit 11, customer service instruction information indicating that customer service is required to all or a part of the mobile terminal TE possessed by the store clerk SA. The customer service instruction information includes, for example, information on the location of the customer CT that needs customer service and information that instructs the customer CT. The information on the location of the customer CT may be, for example, the location information of the grid GR including the location of the customer CT, or the location information of the divided area DA including the location of the customer CT. The information on the location of the customer CT may be intuitively shown using a map indicating the inside of the store ST.
　［接客モニタリングシステムの動作］
　次に、接客モニタリングシステム１００Ａの動作について説明する。 [Operation of customer service monitoring system]
Next, the operation of the customer service monitoring system 100A will be described.
　図６は、接客モニタリング装置１０Ａの動作例を示すフローチャートである。 FIG. 6 is a flowchart showing an operation example of the customer service monitoring device 10A.
　接客条件設定部４１は、例えば操作部１４を介して得られた入力情報を基に、接客条件（接客要否判定条件）を設定する（Ｓ２１）。ここでの接客条件は、例えば、Ｓ２３～Ｓ３１の処理を含む。 The customer service condition setting unit 41 sets a customer service condition (customer service necessity determination condition) based on, for example, input information obtained through the operation unit 14 (S21). The customer service conditions here include, for example, the processes of S23 to S31.
　移動体滞留情報生成部３１は、カメラ装置ＣＡから映像データを取得し、移動体の滞留情報を生成する（Ｓ２２）。また、ＩＤ滞留情報生成部３２は、携帯ＩＤで識別される携帯端末ＴＥから各店員ＳＡの位置の情報を取得し、ＩＤ滞留情報を生成する（Ｓ２２）。尚、移動体の滞留情報及びＩＤ滞留情報は、リアルタイムの情報でもよいし、過去の情報でもよい。尚、移動体の滞留情報の生成及びＩＤ滞留情報の生成は、常時行われてもよいし、定期的に行われてもよい。つまり、Ｓ２２の処理はＳ１２の処理と同じである。 The moving body staying information generating unit 31 acquires video data from the camera device CA and generates staying information of the moving body (S22). Further, the ID staying information generating unit 32 acquires information on the position of each clerk SA from the mobile terminal TE identified by the mobile ID, and generates ID staying information (S22). The staying information and ID staying information of the mobile object may be real-time information or past information. Note that the generation of the stay information of the moving body and the generation of the ID stay information may be performed constantly or periodically. That is, the process of S22 is the same as the process of S12.
　接客条件設定部４１Ａは、例えば操作部１４を介して得られた入力情報を基に、店舗ＳＴにおける、分析対象（接客要否判定対象）のグリッドＧＲを選択する（Ｓ２３）。 The customer service condition setting unit 41A selects, for example, the grid GR to be analyzed (service request necessity determination target) in the store ST based on the input information obtained through the operation unit 14 (S23).
　接客要否判定部４４は、選択されたグリッドＧＲ内に、客ＣＴが一人で所在しているか否かを判定する（Ｓ２４）。例えば、接客要否判定部４４は、移動体（人物）の滞留情報からＩＤ滞留情報を差し引くことで、客ＣＴの滞留情報を抽出でき、客ＣＴが一人であるか否かを判定可能である。 The customer service necessity determination unit 44 determines whether or not the customer CT is alone in the selected grid GR (S24). For example, the customer service necessity determination unit 44 can extract the stay information of the customer CT by subtracting the ID stay information from the stay information of the moving body (person), and can determine whether or not the customer CT is alone. .
　選択されたグリッドＧＲ内に客ＣＴが一人で所在している場合、音声処理部２０Ａは、通信部１１を介して、マイクアレイ装置ＭＡ、レコーダＲＣ、又はメモリ１３から音声データを取得する（Ｓ２５）。また、指向性合成部２３は、選択されたグリッドＧＲにおけるＳ２４で検出された客ＣＴが所在する方向に、音声の指向性を形成する（Ｓ２５）。 When the customer CT is alone in the selected grid GR, the voice processing unit 20A acquires voice data from the microphone array device MA, the recorder RC, or the memory 13 via the communication unit 11 (S25). ). In addition, the directivity synthesis unit 23 forms voice directivity in the direction in which the customer CT detected in S24 in the selected grid GR is located (S25).
　音声判別部２４は、選択されたグリッドＧＲ内の客ＣＴが発する特定のキーワード（対象キーワード）を検出したか否かを判定する（Ｓ２６）。ここでは、特定のキーワードとして、例えば、「すいません」、「どなたか居ませんか」、「店員さーん」、「うーん、どうしよう」、「困ったなあ」、「分からないなあ」、「迷うなあ」、「ちょっとこっちきて」、等の、接客が必要であることを連想させるワードが挙げられる。尚、特定のキーワードは、これらのワードに限られない。 The voice discriminating unit 24 determines whether or not a specific keyword (target keyword) issued by the customer CT in the selected grid GR has been detected (S26). Here, specific keywords include, for example, “I ’m sorry”, “Is there someone else”, “Care clerk”, “Well, what to do”, “I don't know”, “I don't know”, “ Words reminiscent of the need for customer service, such as “Do n’t get lost” and “Keep me here”. The specific keyword is not limited to these words.
　選択されたグリッドＧＲ内の客ＣＴが発する特定のキーワードが検出されなかった場合、接客判定部４０Ａは、選択されたグリッドＧＲ内の客ＣＴの動線情報又は滞留情報を、人物検出部３０又はメモリ１３から読み込む（取得する）（Ｓ２７）。 When a specific keyword emitted by the customer CT in the selected grid GR is not detected, the customer service determination unit 40A displays the flow line information or staying information of the customer CT in the selected grid GR as the person detection unit 30 or Read (acquire) from the memory 13 (S27).
　行動判定部４３は、選択されたグリッドＧＲ内の客ＣＴの動線情報又は滞留情報を基に、この客ＣＴが特定の行動をしたか否かを判定する（Ｓ２８）。尚、特定の行動の具体例については、後述する。 The behavior determination unit 43 determines whether or not the customer CT has performed a specific behavior based on the flow line information or the stay information of the customer CT in the selected grid GR (S28). A specific example of the specific action will be described later.
　客ＣＴが特定の行動をしなかった場合、接客要否判定部４４は、この客ＣＴに対して接客が不要であると判定する（Ｓ２９）。 When the customer CT does not perform a specific action, the customer service necessity determination unit 44 determines that customer service is not required for this customer CT (S29).
　客ＣＴが特定の行動をした場合、又は、選択されたグリッドＧＲ内の客ＣＴが発する特定のキーワードが検出された場合、接客要否判定部４４は、この客ＣＴに対して接客が必要であると判定する（Ｓ３０）。また、端末通知部４５は、通信部１１を介して、店員ＳＡが所持する携帯端末ＴＥの全部又は一部に対して、接客指示情報を送信する（Ｓ３０）。尚、接客指示情報の送信先となる店員ＳＡの具体例については、後述する。 When the customer CT performs a specific action, or when a specific keyword issued by the customer CT in the selected grid GR is detected, the customer service necessity determination unit 44 needs to serve the customer CT. It is determined that there is (S30). In addition, the terminal notification unit 45 transmits customer service instruction information to all or a part of the mobile terminal TE possessed by the store clerk SA via the communication unit 11 (S30). A specific example of the clerk SA that is the destination of the customer service instruction information will be described later.
　Ｓ２４で客ＣＴが１人でない場合（つまり客ＣＴがいない又は客ＣＴが複数である場合）（Ｓ２４のＮｏ）、Ｓ２９の処理後、又はＳ３０の処理後、接客判定部４０Ａは、分析対象の全グリッドＧＲにおいて接客の要否の判定が終了したか否かを判定する（Ｓ３０Ａ）。全グリッドＧＲでの接客の要否の判定が終了した場合（Ｓ３０ＡのＹｅｓ）、接客モニタリング装置１０Ａは、図６の処理を終了する。少なくとも１つのグリッドＧＲでの接客の要否の判定が終了していない場合（Ｓ３０ＡのＮｏ）、Ｓ２３に進み、接客判定部４０Ａは、次のグリッドＧＲを分析対象のグリッドとして選択する。 When the customer CT is not one in S24 (that is, when there is no customer CT or there are a plurality of customer CTs) (No in S24), after the process of S29 or after the process of S30, the customer service determination unit 40A It is determined whether or not the determination of the necessity of customer service has been completed in all grids GR (S30A). When the determination of the necessity of customer service in all grids GR is completed (Yes in S30A), the customer service monitoring apparatus 10A ends the process of FIG. When the determination of the necessity of customer service in at least one grid GR has not been completed (No in S30A), the process proceeds to S23, and the customer service determination unit 40A selects the next grid GR as the analysis target grid.
　このような図６の処理によれば、接客モニタリング装置１０Ａは、客ＣＴが一人でおり、特定のキーワードや特定の行動が検出されたことに応じて、接客の要否を判定する。よって、偶発的に、客ＣＴが店舗ＳＴ内で一人になったり、特定のキーワードを発したり、特定の行動を行ったとしても、接客が必要と判断されないので、接客要否の判定精度を向上できる。 According to the processing of FIG. 6 as described above, the customer service monitoring apparatus 10A determines whether or not the customer needs to be served when the customer CT is alone and a specific keyword or a specific action is detected. Therefore, even if the customer CT happens to be alone in the store ST, issues a specific keyword, or performs a specific action, it is not determined that the customer service is necessary, so the accuracy of determining whether the customer service is necessary is improved. it can.
　次に、客ＣＴによる特定の行動の具体例について説明する。 Next, a specific example of a specific action by the customer CT will be described.
　図７は、客ＣＴが行う特定の行動を説明するための模式図である。図７では、店舗ＳＴ内に商品棚ＳＳが６個設置されている。商品棚ＳＳの近傍のエリアは、客ＣＴが商品棚ＳＳの商品を見学するための商品見学エリアＡ１であると想定される。見学対象の商品棚ＳＳを移動するための通路や店舗ＳＴ内の外周の通路は、客ＣＴが店員ＳＡを探すための店員探索エリアＡ２であることが想定される。商品見学エリアＡ１は、商品を見学するために客ＣＴが滞在する可能性があるエリアである。店員探索エリアＡ２は、店員ＳＡを探すために客ＣＴが滞在する可能性があるエリアである。 FIG. 7 is a schematic diagram for explaining a specific action performed by the customer CT. In FIG. 7, six commodity shelves SS are installed in the store ST. The area in the vicinity of the product shelf SS is assumed to be a product tour area A1 for the customer CT to tour the products on the product shelf SS. The passage for moving the product shelf SS to be visited and the passage on the outer periphery in the store ST are assumed to be the store clerk search area A2 for the customer CT to search for the store clerk SA. The product tour area A1 is an area where the customer CT may stay in order to tour the product. The clerk search area A2 is an area where the customer CT may stay in order to search for the clerk SA.
　接客判定部４０Ａは、客ＣＴの滞留情報を参照し、商品見学エリアＡ１での滞在時間が所定時間ｔ１１（例えば１秒）以上であり、その後店員探索エリアＡ２へ移動し、その後再び商品見学エリアＡ１に戻った場合（第１の滞留パターン）、客ＣＴが特定の行動をしたと判定してもよい。この行動では、客ＣＴが、店員ＳＡを目視確認するために店員探索エリアＡ２に移動したが、見つからずに諦めたと考えられる。また、客ＣＴが、商品棚ＳＳの高さが高いために店員ＳＡを探しに行ったが、見つからずに諦めたと考えられる。 The customer service determination unit 40A refers to the stay information of the customer CT, the stay time in the product tour area A1 is a predetermined time t11 (for example, 1 second) or more, then moves to the clerk search area A2, and then the product tour area again. When returning to A1 (first staying pattern), it may be determined that the customer CT has performed a specific action. In this behavior, it is considered that the customer CT moved to the clerk search area A2 to visually confirm the clerk SA, but gave up without being found. In addition, it is considered that the customer CT went to look for the clerk SA because the height of the merchandise shelf SS was high, but gave up without being found.
　接客判定部４０Ａは、商品見学エリアＡ１での滞在時間が長時間としての所定時間ｔ１２（例えば３００秒（５分））（ｔ１２＞ｔ１１）以上である場合（第２の滞留パターン）、客ＣＴが特定の行動をしたと判定してもよい。この行動では、客ＣＴが、商品棚ＳＳに陳列された商品について長時間考え込んでいると考えらえる。 When the stay time in the product tour area A1 is longer than a predetermined time t12 (for example, 300 seconds (5 minutes)) (t12> t11) (second residence pattern), the customer service determination unit 40A May be determined to have performed a specific action. In this behavior, it can be considered that the customer CT has been thinking about the product displayed on the product shelf SS for a long time.
　接客判定部４０Ａは、店員探索エリアＡ２での滞在時間が所定時間ｔ１３（例えば６００秒（１０分））以上である場合（第３の滞留パターン）、客ＣＴが特定の行動をしたと判定してもよい。この行動では、客ＣＴが、店員探索エリアＡ２に移動して店員ＳＡを探していると考えられる。 When the stay time in the clerk search area A2 is equal to or longer than a predetermined time t13 (for example, 600 seconds (10 minutes)) (the third stay pattern), the customer service determination unit 40A determines that the customer CT has performed a specific action. May be. In this behavior, it is considered that the customer CT moves to the clerk search area A2 and searches for the clerk SA.
　接客判定部４０Ａは、商品見学エリアＡ１において商品棚ＳＳを越えて（つまり客ＣＴが商品棚ＳＳの上から覗き込んで、例えば背伸びして）、店員ＳＡや商品を探している場合、客ＣＴが特定の行動をしたと判定してもよい。この場合、接客判定部４０Ａは、客ＣＴの頭部ＨＤ１の位置が、商品見学エリアＡ１に滞在中における鉛直方向での平均位置ＨＤａｖｅよりも、所定閾値ｔｈ１以上上側に位置するか否かを判定する。この判定は、例えば、映像データにおける画像認識処理や特徴抽出処理により可能である。 When the customer service determination unit 40A is searching for a clerk SA or a product beyond the product shelf SS in the product tour area A1 (that is, the customer CT looks into the product shelf SS and extends, for example), the customer CT May be determined to have performed a specific action. In this case, the service determination unit 40A determines whether or not the position of the head HD1 of the customer CT is higher than the average position HDave in the vertical direction while staying in the product tour area A1 by a predetermined threshold th1 or more. To do. This determination can be made by, for example, image recognition processing or feature extraction processing on video data.
　図８Ａは、客ＣＴが商品棚ＳＳの上側から覗き込んだか否かの判定（覗き込み判定）の一例を説明するための模式図である。図８Ａでは、客ＣＴの頭部ＨＤの位置が、商品棚ＳＳの上部からを覗き込むことで頭部ＨＤ１の位置にあり、頭部の平均位置ＨＤａｖｅよりも閾値ｔｈ１以上に上部にあるとする。この場合、接客判定部４０Ａは、客ＣＴが覗き込んでいると判定し、特定の行動をしていると判定する。 FIG. 8A is a schematic diagram for explaining an example of whether or not the customer CT has looked into from the upper side of the commodity shelf SS (looking-in determination). In FIG. 8A, it is assumed that the position of the head HD of the customer CT is at the position of the head HD1 by looking into the top of the product shelf SS, and is above the threshold th1 by the head th average position HDave. . In this case, the customer service determination unit 40A determines that the customer CT is looking in, and determines that the customer CT is performing a specific action.
　接客判定部４０Ａは、商品見学エリアＡ１又は店員探索エリアＡ２において、客ＣＴがきょろきょろしている場合、客ＣＴが特定の行動をしたと判定してもよい。この場合、接客判定部４０Ａは、客ＣＴの頭部ＨＤ又は肩部ＳＤの位置が、頭頂方向（頭部ＨＤの上側）から見て、左右方向に（左回りに９０°回転した方向又は右回りに９０°回転した方向）、又は一定角度（例えば左回り又は右回りに９０°以外の任意の角度で回転した方向）で、複数回移動したか否かを判定する。この判定は、例えば、映像データにおける画像認識処理や特徴抽出処理により可能である。 The customer service determination unit 40A may determine that the customer CT has performed a specific action when the customer CT is in the product tour area A1 or the clerk search area A2. In this case, the service determination unit 40A determines that the position of the head HD or the shoulder SD of the customer CT is the left-right direction (a direction rotated 90 ° counterclockwise or right) as viewed from the top of the head (above the head HD). It is determined whether or not it has been moved a plurality of times at a certain angle (for example, a direction rotated at an arbitrary angle other than 90 ° counterclockwise or clockwise). This determination can be made by, for example, image recognition processing or feature extraction processing on video data.
　図８Ｂは、客ＣＴがきょろきょろしているか否かの判定（きょろきょろ判定）の一例を説明するための模式図である。図８Ｂでは、頭部ＨＤを角度θで左回転させ、且つ複数回回転させているとする。この場合、接客判定部４０Ａは、きょろきょろしていると判定し、特定の行動をしていると判定する。尚、頭部ＨＤ又は肩部ＳＤの回転回数が所定回数以上であることを、きょろきょろ判定を満たす条件に加えてもよい。 FIG. 8B is a schematic diagram for explaining an example of determining whether or not the customer CT is in a dark state (a black and white determination). In FIG. 8B, it is assumed that the head HD is rotated counterclockwise at an angle θ and rotated a plurality of times. In this case, the customer service determination unit 40A determines that the customer is afraid and determines that the user is performing a specific action. It should be noted that the fact that the number of rotations of the head HD or the shoulder SD is equal to or greater than a predetermined number may be added to a condition that satisfies the mentality determination.
　接客判定部４０Ａは、商品見学エリアＡ１又は店員探索エリアＡ２において、客ＣＴが首を傾げた場合、客ＣＴが特定の行動をしたと判定してもよい。この場合、接客判定部４０Ａは、客ＣＴの頭部ＨＤ又は肩部ＳＤの位置が、頭部ＨＤの正面方向（つまり顔の正面方向）から見て、左右方向に（左回り又は右回りに任意の角度で回転した方向）、所定時間ｔ２１（例えば５秒）以上又は所定回数（例えば３回）以上傾いている、つまり首を傾げているか否かを判定する。この判定は、例えば、映像データにおける画像認識処理や特徴抽出処理により可能である。 The customer service determination unit 40A may determine that the customer CT has performed a specific action when the customer CT leans his / her head in the product tour area A1 or the clerk search area A2. In this case, the service determination unit 40A determines that the position of the head HD or shoulder SD of the customer CT is in the left-right direction (counterclockwise or clockwise) when viewed from the front direction of the head HD (that is, the front direction of the face). It is determined whether or not it is tilted at a predetermined angle), for a predetermined time t21 (for example, 5 seconds) or more or for a predetermined number of times (for example, 3 times), that is, whether the neck is tilted. This determination can be made by, for example, image recognition processing or feature extraction processing on video data.
　図８Ｃは、首を傾げたか否かの判定（傾げ判定）の一例を説明するための模式図である。図８Ｃでは、客ＣＴの頭部ＨＤが、所定回数ｔ２２以上の回数、左方向に角度θ２で傾いているとする。この場合、接客判定部４０Ａは、首をかしげていると判定し、特定の行動をしていると判定する。 FIG. 8C is a schematic diagram for explaining an example of whether or not the head is tilted (tilting determination). In FIG. 8C, it is assumed that the head HD of the customer CT is tilted leftward at an angle θ2 a predetermined number of times t22 or more. In this case, the customer service determination unit 40A determines that the head is curled and determines that a specific action is being performed.
　接客判定部４０Ａは、商品見学エリアＡ１又は店員探索エリアＡ２において、客ＣＴが異なる商品を所持している場合、客ＣＴが特定の行動をしたと判定してもよい。この場合、接客判定部４０Ａは、例えば、映像データから客ＣＴの腕又は手の領域を検出し、その両方の領域の周辺の領域に、物体（商品）が存在するか否かを判定する。客ＣＴが複数の商品を持っている場合、両商品を見比べ、それぞれの商品の特徴を知りたいと考えていることが考えられる。 The customer service determination unit 40A may determine that the customer CT has performed a specific action when the customer CT has a different product in the product tour area A1 or the clerk search area A2. In this case, for example, the customer service determination unit 40A detects the area of the arm or hand of the customer CT from the video data, and determines whether or not an object (product) exists in the area around both of the areas. When the customer CT has a plurality of products, it may be possible to compare both products and want to know the characteristics of each product.
　尚、図８Ｂのように頭部ＨＤの上方から見て特定の行動か否か判断するか、図８Ａ，図８Ｃのように頭部ＨＤの正面方向から見て特定の行動か否か判断するかは、カメラ装置ＣＡに対する客ＣＴの位置に依存する。例えば、カメラ装置ＣＡの真下に近い位置に客ＣＴが所在する場合、映像データは頭部ＨＤの上方から見た画像に近くなり、カメラ装置ＣＡの真下から遠い位置（例えば店舗ＳＴの外周近く）に客ＣＴが所在する場合、映像データは頭部ＨＤの正面方向から見た画像に近くなる。 Note that it is determined whether or not a specific action is seen from above the head HD as shown in FIG. 8B, or whether or not a specific action is seen from the front direction of the head HD as shown in FIGS. 8A and 8C. This depends on the position of the customer CT with respect to the camera device CA. For example, when the customer CT is located at a position near directly below the camera apparatus CA, the video data is close to an image viewed from above the head HD, and is a position far below the camera apparatus CA (for example, near the outer periphery of the store ST). When the customer CT is located, the video data is close to an image viewed from the front direction of the head HD.
　次に、接客指示情報の送信先となる店員ＳＡの具体例について説明する。 Next, a specific example of the clerk SA that is the destination of the customer service instruction information will be described.
　図９及び図１０は、接客指示情報の送信先となる店員ＳＡを説明するための模式図である。 FIG. 9 and FIG. 10 are schematic diagrams for explaining the store clerk SA that is the transmission destination of the customer service instruction information.
　端末通知部４５は、単独の店員ＳＡ（例えば店員ＳＡ１１）を優先して、接客指示情報を送信してもよい。人物検出部３０は、選択されたグリッドＧＲ内にＩＤ滞留情報が１つ存在する場合、選択されたグリッドＧＲ内に店員ＳＡが１人であり、単独の店員ＳＡであることを検出してもよい。 The terminal notification unit 45 may transmit customer service instruction information with priority given to a single clerk SA (for example, clerk SA11). The person detection unit 30 may detect that there is one clerk SA in the selected grid GR and that it is a single clerk SA when there is one ID residence information in the selected grid GR. Good.
　端末通知部４５は、店員ＳＡ（例えば店員ＳＡ１２）の近くに客ＣＴ（例えば客ＣＴ１１）が所在する場合、その店員ＳＡは他に接客対応不可であるとして、接客指示情報の送信先から除外してもよい。この場合、この店員ＳＡが所持する携帯端末ＴＥに対しては、接客指示情報が送信されない。 When the customer CT (for example, customer CT11) is located near the store clerk SA (for example, store clerk SA12), the terminal notifying unit 45 excludes that the store clerk SA is otherwise incapable of serving customers, from the destination of the customer service instruction information. May be. In this case, the customer service instruction information is not transmitted to the portable terminal TE possessed by the clerk SA.
　尚、人物検出部３０は、例えば図２のＳ１５と同様の方法で、店員ＳＡの近くに客ＣＴが所在するか否かを判定してもよい。つまり、人物検出部３０は、選択されたグリッドＧＲにおいて、移動体の滞留情報において複数の移動体が検出され、ＩＤ滞留情報において、移動体の滞留情報における移動体の数よりも少なく、且つ１つ以上のＩＤ移動体が検出された場合、店員ＳＡの近くに客ＣＴが所在すると判定してもよい。 Note that the person detection unit 30 may determine whether or not the customer CT is located near the store clerk SA, for example, by the same method as S15 in FIG. That is, in the selected grid GR, the person detection unit 30 detects a plurality of moving bodies in the staying information of the moving bodies, and the ID staying information has less than the number of moving bodies in the staying information of the moving bodies, and 1 When two or more ID moving bodies are detected, it may be determined that the customer CT is located near the clerk SA.
　端末通知部４５は、店舗ＳＴ内の全グリッドＧＲにおいて単独の店員ＳＡが存在しない場合、店舗ＳＴの外のバックヤードに所在する店員ＳＡが所持する携帯端末ＴＥへ、接客指示情報を送信してもよい。人物検出部３０は、各グリッドＧＲを分析し、各グリッドＧＲにおいてＩＤ滞留情報が０個である場合、又は、各グリッドＧＲにおいてＩＤ滞留情報が存在しても店員ＳＡの近くに客ＣＴが所在する場合、単独の店員ＳＡが存在しないことを検出してもよい。この場合、端末通知部４５は、人物検出部３０によりＩＤ滞留情報が検出されていない携帯ＩＤで識別される携帯端末ＴＥに対して、接客指示情報を送信してもよい。尚、この場合、店舗ＳＴ内及び店舗ＳＴ外に所在する店員ＳＡが所持する各携帯端末ＴＥについて、携帯ＩＤの情報がメモリ１３に予め保持されている。 When there is no single clerk SA in all grids GR in the store ST, the terminal notification unit 45 transmits customer service instruction information to the mobile terminal TE owned by the clerk SA located in the backyard outside the store ST. Also good. The person detection unit 30 analyzes each grid GR, and when there is no ID residence information in each grid GR, or even if there is ID residence information in each grid GR, the customer CT is located near the clerk SA. In this case, it may be detected that there is no single clerk SA. In this case, the terminal notification unit 45 may transmit the customer service instruction information to the mobile terminal TE identified by the mobile ID for which the ID staying information is not detected by the person detection unit 30. In this case, the mobile ID information is held in advance in the memory 13 for each mobile terminal TE possessed by the store clerk SA located inside and outside the store ST.
　また、端末通知部４５は、店舗ＳＴ内の全グリッドＧＲにおいて単独の店員ＳＡが存在しない場合、店舗ＳＴの内外に所在する店員ＳＡ（全店員）が所持する全携帯端末ＴＥへ、接客指示情報を送信してもよい。この場合、全店員ＳＡに対して接客指示情報が通知されることで、店員ＳＡ間で相互に連携を取って可能な限り接客することが可能である。 In addition, when there is no single clerk SA in all grids GR in the store ST, the terminal notification unit 45 provides customer service instruction information to all the mobile terminals TE owned by the clerk SA (all clerk) located inside and outside the store ST. May be sent. In this case, the customer service instruction information is notified to all the clerk SA, so that the clerk SA can cooperate with each other as much as possible.
　［効果等］
　このように、本実施形態の接客モニタリングシステム１００Ａ及び接客モニタリング装置１０Ａでは、人物検出部３０は、任意のグリッドＧＲにおいて所定時間ｔ１以上滞留する第１の客ＣＴを１人検出する。接客判定部４０Ａは、第１の客ＣＴが発した特定のキーワードが検出された場合、又は第１の客ＣＴが特定の行動が検出された場合、第１の客ＣＴに対する接客が必要であると判定する。通信部１１は、第１の客ＣＴに対する接客が必要な場合、店員ＳＡの少なくとも一部が所持する携帯端末ＴＥへ、第１の客ＣＴに対する接客を指示するための接客指示情報を送信する。 [Effects]
Thus, in the customer service monitoring system 100A and the customer service monitoring device 10A of the present embodiment, the person detection unit 30 detects one first customer CT that stays for a predetermined time t1 or longer in an arbitrary grid GR. When a specific keyword issued by the first customer CT is detected, or when a specific action is detected by the first customer CT, the customer service determination unit 40A needs to serve the first customer CT. Is determined. The communication unit 11 transmits customer service instruction information for instructing customer service for the first customer CT to the mobile terminal TE possessed by at least a part of the store clerk SA when customer service for the first customer CT is required.
　つまり、接客モニタリング装置１０Ａは、任意のグリッドＧＲにおいて店員ＳＡでない移動体（つまり第１の客ＣＴ）を検出し、第１の客ＣＴが店員ＳＡのサポートが必要か否かを、第１の客ＣＴが発するワードや第１の客ＣＴの行動から推測する。そして、接客モニタリング装置１０Ａは、第１の客ＣＴが店員ＳＡのサポートを必要とする場合、店員ＳＡが保有する携帯端末ＴＥへその旨通知する。 That is, the customer service monitoring device 10A detects a mobile body (that is, the first customer CT) that is not the store clerk SA in an arbitrary grid GR, and determines whether the first customer CT needs to support the store clerk SA. Inferred from the word issued by the customer CT and the behavior of the first customer CT. Then, when the first customer CT needs the support of the store clerk SA, the customer service monitoring apparatus 10A notifies the mobile terminal TE held by the store clerk SA to that effect.
　これにより、接客モニタリングシステム１００Ａ及び接客モニタリング装置１０Ａは、接客を必要とする第１の客ＣＴが発するワードや第１の客ＣＴの行動を接客の要否の判断に活用し、接客の機会の逸失を低減できる。また、第１の客ＣＴが直接店員ＳＡに話しかけたか否かによらず、接客を必要としていることを店員ＳＡが気付くことができ、第１の客ＣＴの満足度も向上できる。 As a result, the customer service monitoring system 100A and the customer service monitoring device 10A utilize the ward issued by the first customer CT that requires customer service and the behavior of the first customer CT to determine whether or not the customer needs to be served. Loss can be reduced. Further, regardless of whether or not the first customer CT directly talks to the clerk SA, the clerk SA can recognize that the customer service is required, and the satisfaction of the first customer CT can be improved.
　また、接客判定部４０Ａは、第１の客ＣＴの頭部ＨＤ又は肩部ＳＤの検出位置と、頭部ＨＤ又は肩部ＳＤの検出時間又は検出回数と、に基づいて、特定の行動をしているか否かを判定してもよい。 In addition, the service determination unit 40A performs a specific action based on the detection position of the head HD or shoulder SD of the first customer CT and the detection time or the number of detections of the head HD or shoulder SD. It may be determined whether or not.
　これにより、接客モニタリングシステム１００Ａ及び接客モニタリング装置１０Ａは、第１の客ＣＴの体の各部の動きを検出することで、容易に特定の行動をしているか否かを判断でき、接客の要否の判定に活用できる。 Accordingly, the customer service monitoring system 100A and the customer service monitoring device 10A can easily determine whether or not a specific action is being performed by detecting the movement of each part of the body of the first customer CT. It can be used for judgment.
　また、接客判定部４０Ａは、第１の客ＣＴの滞留情報が所定の滞留パターン（例えば第１～第３の滞留パターン）を含む場合、特定の行動をしていると判定してもよい。 Further, the customer service determination unit 40A may determine that a specific action is being performed when the stay information of the first customer CT includes a predetermined stay pattern (for example, first to third stay patterns).
　これにより、接客モニタリングシステム１００Ａ及び接客モニタリング装置１０Ａは、第１の客ＣＴが移動した軌跡に基づく滞留パターンを検出することで、容易に特定の行動をしているか否かを判断でき、接客の要否の判定に活用できる。 Accordingly, the customer service monitoring system 100A and the customer service monitoring device 10A can easily determine whether or not a specific action is being performed by detecting a staying pattern based on the trajectory of the movement of the first customer CT. It can be used to determine necessity.
　また、通信部１１は、店員ＳＡの滞留情報に基づいて、任意のグリッド内に１人で所在する店員ＳＡへ接客指示情報を送信してもよい。 Further, the communication unit 11 may transmit the customer service instruction information to the clerk SA located by one person in an arbitrary grid based on the stay information of the clerk SA.
　これにより、１人で所在する店員ＳＡは、接客指示情報を確認し、迅速に第１の客ＣＴに対して接客できる。 Thereby, the clerk SA located by himself / herself can check the customer service instruction information and can quickly serve the customer 1 for the first customer CT.
　また、通信部１１は、店員ＳＡの滞留情報及び人物の滞留情報に基づいて、非接客者（例えば接客中でない店員ＳＡ）へ接客指示情報を送信してもよい。 Further, the communication unit 11 may transmit customer service instruction information to a non-customer (for example, a customer clerk SA who is not in customer service) based on the customer clerk SA residence information and person residence information.
　これにより、接客指示情報を確認した店員ＳＡは、接客中ではないので、迅速に第１の客ＣＴに対して接客できる。 Thus, the clerk SA who has confirmed the customer service instruction information is not in service, and can quickly serve the first customer CT.
　通信部１１は、任意のグリッドＧＲ内に１人で所在する店員ＳＡが存在しない場合、店員ＳＡの滞留情報が生成されなかった店員ＳＡが所持する携帯端末ＴＥへ、接客指示情報を送信してもよい。 When there is no clerk SA located alone in any grid GR, the communication unit 11 transmits the customer service instruction information to the mobile terminal TE possessed by the clerk SA for which no stay information of the clerk SA has been generated. Also good.
　これにより、例えば、店舗ＳＴ内に所在せずに滞留情報が生成されなかった、バックヤードで待機中の店員ＳＡが、接客指示情報を携帯端末ＴＥにより確認し、迅速に第１の客ＣＴに対して接客できる。 Thereby, for example, the store clerk SA who is not in the store ST and has not generated the stay information and is waiting in the backyard confirms the customer service instruction information with the mobile terminal TE, and quickly becomes the first customer CT. You can serve customers.
　通信部１１は、任意のグリッドＧＲ内に１人で所在する店員ＳＡが存在しない場合、全ての店員ＳＡが所持する携帯端末ＴＥへ、接客指示情報を送信してもよい。 The communication unit 11 may transmit the customer service instruction information to the mobile terminals TE possessed by all the clerk SA when there is no clerk SA who is alone in any grid GR.
　これにより、各店員ＳＡが携帯端末ＴＥにより受信した接客指示情報を確認し、各店員ＳＡが互いに相談し、例えば現在の接客状況に応じて、いずれかの店員ＳＡが第１の客ＣＴを接客することができる。 As a result, each clerk SA confirms the customer service instruction information received by the mobile terminal TE, and each clerk SA consults with each other. For example, according to the current customer service situation, any clerk SA serves the first customer CT. can do.
　また、接客モニタリング装置１０Ａは、商品を購入しようとしているが購入を迷っていると予想される客ＣＴの行動、商品の説明が必要と考えていると予想される客ＣＴの行動、店員ＳＡを探していると予想される客ＣＴの行動、等を接客が必要な行動として捉え、店員ＳＡに対して接客の指示を通知する。また、接客が可能と予想される店員ＳＡに対して、接客指示情報を通知するようにできる。これにより、通知された店員ＳＡは、客ＣＴに対する接客の必要性を認識できる。つまり、接客モニタリング装置１０Ａは、接客の必要性を迅速に的確な店員ＳＡに通知し、接客ロスの抑制を図ることができる。 In addition, the customer service monitoring device 10A determines the behavior of the customer CT who is about to purchase the product but is expected to be confused, the behavior of the customer CT that is expected to explain the product, and the clerk SA. The behavior of the customer CT expected to be searched is regarded as the behavior that the customer needs, and the customer clerk SA is notified of the customer service instruction. In addition, the customer service instruction information can be notified to the clerk SA who is expected to be able to serve customers. Thus, the notified clerk SA can recognize the necessity of customer service for the customer CT. In other words, the customer service monitoring device 10A can promptly notify the appropriate store clerk SA of the necessity of customer service, and can suppress customer service loss.
　また、接客モニタリング装置１０Ａは、カメラ装置ＣＡが撮像した画像による人物検出や動線検出、無線を用いた店員ＳＡの位置検出、マイクアレイ装置ＭＡが収音した音声による音声検出、等に基づいて、店員ＳＡを必要とする客ＣＴを検出し、店員ＳＡやその他の監視者に対して接客を必要とする旨のアラートを出力できる。これにより、接客モニタリング装置１０Ａは、客ＣＴが接客を必要としているにも関わらず、店員ＳＡが接客の必要性を認識できずに接客機会を失うことを抑制できる。また、接客モニタリング装置１０Ａは、客ＣＴの行動を客ＣＴに強いることなく分析でき、客ＣＴと店員ＳＡとを対応付けることで、より高精度の接客を実現できる。 In addition, the customer service monitoring device 10A is based on person detection and flow line detection based on an image captured by the camera device CA, position detection of the clerk SA using radio, sound detection based on sound collected by the microphone array device MA, and the like. The customer CT that requires the store clerk SA can be detected, and an alert can be output to the store clerk SA and other monitoring personnel that the customer service is required. Thereby, the customer service monitoring device 10A can suppress the loss of the customer service opportunity because the store clerk SA cannot recognize the customer service requirement even though the customer CT needs the customer service. In addition, the customer service monitoring device 10A can analyze the behavior of the customer CT without forcing the customer CT, and can realize customer service with higher accuracy by associating the customer CT with the clerk SA.
　（第３の実施形態）
　第３の実施形態では、接客モニタリングシステムにおける接客モニタリング装置は、人物の動線情報により示される人物の動線が遮蔽物等により途切れた場合に、音声認識を利用して、同一人物か否かを判断することを説明する。 (Third embodiment)
In the third embodiment, the customer service monitoring device in the customer service monitoring system uses voice recognition to determine whether or not the person is the same person when the person's flow line indicated by the person's flow line information is interrupted by an obstacle or the like. Explain that Judgment.
　［接客モニタリングシステムの構成］
　図１１は、第３の実施形態における接客モニタリングシステム１００Ｂの構成例を示すブロック図である。図１１の接客モニタリングシステム１００Ｂは、図１に示した接客モニタリングシステム１００と比較すると、接客モニタリング装置１０の代わりに接客モニタリング装置１０Ｂを備える点が異なる。図１１の接客モニタリングシステム１００Ｂにおいて、図１に示した接客モニタリングシステム１００と同様の構成については、同一の符号を付し、説明を省略又は簡略化する。 [Configuration of customer service monitoring system]
FIG. 11 is a block diagram illustrating a configuration example of the customer service monitoring system 100B according to the third embodiment. 11 is different from the customer monitoring system 100 shown in FIG. 1 in that a customer monitoring device 10B is provided instead of the customer monitoring device 10. In the customer service monitoring system 100B of FIG. 11, the same components as those of the customer service monitoring system 100 shown in FIG. 1 are denoted by the same reference numerals, and description thereof is omitted or simplified.
　接客モニタリング装置１０Ｂは、音声処理部２０Ｂ、人物検出部３０Ｂ、及び接客判定部４０を備える。音声処理部２０Ｂは、音声検出部２１、検出結果累積部２２、及び話者識別部２５を備える。人物検出部３０Ｂは、移動体滞留情報生成部３１、ＩＤ滞留情報生成部３２、及び滞留情報調整部３３を備える。 The customer service monitoring device 10B includes a voice processing unit 20B, a person detection unit 30B, and a customer service determination unit 40. The voice processing unit 20B includes a voice detection unit 21, a detection result accumulation unit 22, and a speaker identification unit 25. The person detection unit 30 </ b> B includes a moving body residence information generation unit 31, an ID residence information generation unit 32, and a residence information adjustment unit 33.
　話者識別部２５は、所定の話者認識処理（例えば声紋識別）により、音声検出部２１により検出された音声を発する話者を識別する。 The speaker identification unit 25 identifies a speaker that utters the voice detected by the voice detection unit 21 by a predetermined speaker recognition process (for example, voiceprint identification).
　滞留情報調整部３３は、話者識別等により同一人物と判定された滞留情報を、統合（Ｍｅｒｇｅ）してメモリ１３に蓄積する。例えば、滞留情報調整部３３は、滞留情報が生成されてメモリ１３に保持された人物と同一人物が再度店舗ＳＴ内に出現したと判定された場合、この滞留情報の続きとして（この滞留情報の一部の情報として）、この人物の滞留に係る情報を蓄積する。 The staying information adjusting unit 33 integrates staying information determined to be the same person by speaker identification or the like, and stores the staying information in the memory 13. For example, if it is determined that the same person as the person who has been generated and retained in the memory 13 has appeared again in the store ST, the staying information adjustment unit 33 continues the staying information ( As part of the information, information related to this person's stay is accumulated.
　［接客モニタリングシステムの動作］
　次に、接客モニタリングシステム１００Ｂの動作について説明する。 [Operation of customer service monitoring system]
Next, the operation of the customer service monitoring system 100B will be described.
　図１２は、接客モニタリング装置１０Ｂの動作例を示すフローチャートである。 FIG. 12 is a flowchart showing an operation example of the customer service monitoring device 10B.
　まず、接客モニタリング装置１０Ｂは、図２に示したＳ１１～Ｓ１４の処理を実行する。 First, the customer service monitoring device 10B executes the processing of S11 to S14 shown in FIG.
　続いて、接客行動抽出部４２は、分析対象のグリッドＧＲ内に、店員ＳＡ又は客ＣＴが所在するか否かを判定する（Ｓ３１）。例えば、接客行動抽出部４２は、移動体の滞留情報において移動体が検出された場合、店員ＳＡ又は客ＣＴが所在することを判別可能である。 Subsequently, the customer service behavior extraction unit 42 determines whether or not the store clerk SA or the customer CT is located in the grid GR to be analyzed (S31). For example, the customer service behavior extraction unit 42 can determine that the store clerk SA or the customer CT is located when the moving object is detected in the staying information of the moving object.
　選択されたグリッドＧＲ内に店員ＳＡ又は客ＣＴが所在する場合（Ｓ３１のＹｅｓ）、話者識別部２５は、話者認証処理により話者を識別するための話者識別情報を取得する（Ｓ３２）。例えば、話者識別部２５は、話者としての店員ＳＡ又は客ＣＴに係る音声波形のスペクトルの情報を、話者識別情報として取得する。 When the clerk SA or the customer CT is located in the selected grid GR (Yes in S31), the speaker identification unit 25 acquires speaker identification information for identifying the speaker by speaker authentication processing (S32). ). For example, the speaker identification unit 25 acquires information on a spectrum of a speech waveform related to a salesclerk SA or a customer CT as a speaker as speaker identification information.
　話者識別部２５は、この店員ＳＡ又は客ＣＴの話者識別を以前にも実施したことがあるか否かを判定する（Ｓ３３）。以前に実施しているか否かは、メモリ１３に同一の話者識別情報が保持されているか否かにより判別可能である。 The speaker identification unit 25 determines whether or not speaker identification of this clerk SA or customer CT has been performed before (S33). Whether or not it has been performed before can be determined by whether or not the same speaker identification information is held in the memory 13.
　尚、ここでの同一は、完全に同一でなくてもよく、一定の尤度を超えている場合、例えば話者識別情報としての音声スペクトル同志の相関値や類似度が所定値以上である場合には、同一に含まれてもよい。また、過去に話者識別情報が得られた店舗ＳＴ内での位置は、任意の位置でよく、Ｓ１４で選択されたグリッドＧＲと同一のグリッド内に限られなくてよい。 In addition, the same here does not need to be completely the same, and when it exceeds a certain likelihood, for example, when the correlation value or similarity between the speech spectra as speaker identification information is a predetermined value or more May be included identically. Further, the position in the store ST where the speaker identification information was obtained in the past may be an arbitrary position, and may not be limited to the same grid as the grid GR selected in S14.
　この店員ＳＡ又は客ＣＴの話者識別を以前に実施していない場合、つまりメモリ１３にＳ３２で取得された話者識別情報と同じ話者識別情報が保持されていない場合、話者識別部２５は、取得された話者識別情報をメモリ１３に保持（登録）する（Ｓ３４）。初回に話者識別情報をメモリ１３に登録することで、２回目以降において話者識別情報を用いて話者識別が可能となる。 When speaker identification of the store clerk SA or customer CT has not been performed before, that is, when the same speaker identification information as the speaker identification information acquired in S32 is not held in the memory 13, the speaker identification unit 25 Holds (registers) the acquired speaker identification information in the memory 13 (S34). By registering the speaker identification information in the memory 13 for the first time, speaker identification can be performed using the speaker identification information for the second and subsequent times.
　この店員ＳＡ又は客ＣＴの話者識別を以前に実施した場合、つまりメモリ１３にＳ３２で取得された話者識別情報と同じ話者識別情報が保持されている場合、滞留情報調整部３３は、話者識別情報が一致する人物に係る滞留情報を統合（マージ）し、同一人物の滞留情報としてメモリ１３に保持する（Ｓ３５）。 When speaker identification of this clerk SA or customer CT has been performed before, that is, when the same speaker identification information as the speaker identification information acquired in S32 is held in the memory 13, the staying information adjustment unit 33 The staying information related to the person with the same speaker identification information is integrated (merged) and held in the memory 13 as staying information of the same person (S35).
　ここでは、Ｓ３１で店員ＳＡが所在すると判定された場合、滞留情報調整部３３は、店員ＳＡに係る移動体の滞留情報を統合し、更に店員ＳＡに係るＩＤ滞留情報を統合する。Ｓ３１で客ＣＴが所在すると判定された場合、滞留情報調整部３３は、客ＣＴに係る移動体の滞留情報を統合する。 Here, when it is determined in S31 that the clerk SA is located, the stay information adjusting unit 33 integrates the stay information of the mobile body related to the store clerk SA, and further integrates the ID stay information related to the store clerk SA. When it is determined in S31 that the customer CT is located, the staying information adjustment unit 33 integrates the staying information of the moving body related to the customer CT.
　Ｓ３１で選択されたグリッドＧＲ内に店員ＳＡ及び客ＣＴのいずれも所在しない場合（Ｓ３１のＮｏ）、Ｓ３４の処理後、又はＳ３５の処理後、接客判定部４０は、分析対象の全グリッドＧＲにおいて滞留情報の同一性の判定が終了したか否かを判定する（Ｓ３６）。全グリッドＧＲでの滞留情報の同一性の判定が終了した場合（Ｓ３６のＹｅｓ）、接客モニタリング装置１０Ｂは、図１２の処理を終了する。少なくとも１つのグリッドＧＲでの滞留情報の同一性の判定が終了していない場合（Ｓ３６のＮｏ）、Ｓ１４に進み、接客判定部４０は、次のグリッドＧＲを分析対象（滞留情報の同一性の判定対象）のグリッドとして選択する。 If neither the clerk SA nor the customer CT is located in the grid GR selected in S31 (No in S31), after the process in S34 or after the process in S35, the customer service determination unit 40 performs the analysis in all the grids GR to be analyzed. It is determined whether the determination of the identity of the staying information has been completed (S36). When the determination of the identity of the staying information in all grids GR is completed (Yes in S36), the customer service monitoring device 10B ends the process of FIG. When the determination of the identity of the stay information in at least one grid GR has not been completed (No in S36), the process proceeds to S14, and the customer service determination unit 40 analyzes the next grid GR (the identity of the stay information). Select as the judgment target grid.
　このような図１２の処理によれば、接客モニタリング装置１０Ｂは、新たに出現した人物に係る話者識別情報が過去においても存在した場合、この話者識別情報に係る人物の滞留情報を、対応する既存の滞留情報に統合して、メモリ１３に記憶する。 According to the processing of FIG. 12, the service monitoring apparatus 10 </ b> B corresponds to the person staying information related to the speaker identification information when the speaker identification information related to the newly appearing person has existed in the past. The existing staying information is integrated and stored in the memory 13.
　これにより、接客モニタリング装置１０Ｂは、例えば、商品棚ＳＳ等の遮蔽物を通過して人物が移動した場合など、人物の動線情報や滞留情報が途切れた場合でも、話者識別処理を利用して各人物が発した音声の同一性を判定できる。接客モニタリング装置１０Ｂは、音声を用いた識別処理により複数の人物が一定の尤度を超えて同一人物であると識別した場合、同一人物の動線情報や滞留情報として、接客活動に活用できる。 Thereby, the customer service monitoring device 10B uses the speaker identification processing even when the flow line information or staying information of the person is interrupted, for example, when the person moves through a shield such as the product shelf SS. Thus, it is possible to determine the identity of the sound produced by each person. The customer service monitoring device 10B can be used for customer service activities as flow line information and staying information of the same person when a plurality of persons are identified as being the same person by exceeding the certain likelihood by the identification process using voice.
　次に、人物が移動する移動経路上に遮蔽物が存在する場合の滞留情報の扱いについて説明する。 Next, the handling of the staying information when there is a shield on the moving route along which the person moves will be described.
　図１３は、滞留情報の同一性を説明するための模式図である。図１３では、第１の人物Ｐ１が、商品棚ＳＳ２１の左側から商品棚ＳＳ２１の裏側に隠れ、第２の人物Ｐ２が、商品棚ＳＳ２１の右側から出現している。人物検出部３０により第１の人物Ｐ１及び第２の人物Ｐ２に係る動線情報や滞留情報を導出する場合、遮蔽物としての商品棚ＳＳ２１により動線が遮断される。そのため、動線情報及び滞留情報は、第１の人物Ｐ１及び第２の人物Ｐ２の各々で別個に得られることになる。 FIG. 13 is a schematic diagram for explaining the identity of the staying information. In FIG. 13, the first person P1 is hidden behind the product shelf SS21 from the left side of the product shelf SS21, and the second person P2 appears from the right side of the product shelf SS21. When the flow line information and the staying information relating to the first person P1 and the second person P2 are derived by the person detection unit 30, the flow line is blocked by the product shelf SS21 as a shield. Therefore, the flow line information and the staying information are obtained separately for each of the first person P1 and the second person P2.
　この場合、音声処理部２０Ｂは、第１の人物Ｐ１及び第２の人物Ｐ２の音声データを取得し、双方の音声データに対して話者識別処理することで、同一人物が発した音声であるか否かを識別する。これにより、接客モニタリング装置１０Ｂは、遮蔽物により複数の人物の動線情報や滞留情報が分断されても、同一人物の情報として動線情報や滞留情報を統合して扱うことができる。よって、例えば、新規に来店した客ではなく、商品棚ＳＳ２１の背後から客ＣＴが表れた場合に、接客済みの客であるか否かを過去の滞留情報から紐付けでき、接客活動を効率化できる。 In this case, the voice processing unit 20B obtains voice data of the first person P1 and the second person P2, and performs voice identification processing on the voice data of the both persons, thereby generating voice generated by the same person. Or not. Thereby, the customer service monitoring apparatus 10B can handle the flow line information and the staying information as the same person information in an integrated manner even if the flow line information and the staying information of a plurality of persons are divided by the shield. Therefore, for example, when a customer CT appears from the back of the product shelf SS21 instead of a new customer, it can be linked to whether or not the customer is a customer from past residence information, and the customer service activities are made more efficient it can.
　［効果等］
　このように、本実施形態の接客モニタリングシステム１００Ｂ及び接客モニタリング装置１０Ｂでは、話者識別部２５は、音声検出部２１により検出された発話に係る第１の話者を話者識別処理により識別し、第１の話者に係る第１の話者識別情報をメモリ１３に保持させる。滞留情報調整部３３は、第１の話者識別情報と、話者識別部２５により第１の話者識別情報が得られた後に識別された第２の話者に係る話者識別情報と、が略一致する場合、第１の話者に係る人物の滞留情報又は店員ＳＡの滞留情報に、第２の話者に係る人物の滞留情報又は店員ＳＡの滞留情報を統合する。 [Effects]
Thus, in the customer service monitoring system 100B and the customer service monitoring device 10B of the present embodiment, the speaker identification unit 25 identifies the first speaker related to the utterance detected by the voice detection unit 21 by the speaker identification process. The first speaker identification information related to the first speaker is stored in the memory 13. The staying information adjustment unit 33 includes first speaker identification information, speaker identification information related to the second speaker identified after the first speaker identification information is obtained by the speaker identification unit 25, and Are substantially the same, the retention information of the person related to the second speaker or the retention information of the clerk SA is integrated into the retention information of the person related to the first speaker or the retention information of the clerk SA.
　これにより、接客モニタリングシステム１００Ｂ及び接客モニタリング装置１０Ｂは、動線情報や滞留情報が一旦遮断され、別の人物が検出されても、話者識別により滞留情報を紐付け可能な場合には、別個の動線情報や滞留情報として取り扱わず、同一人物の動線情報や滞留情報として取扱いできる。また、接客モニタリングシステム１００Ｂ及び接客モニタリング装置１０Ｂは、音声データを用いて同一人物か否かを判定するので、画像データを用いて顔照合等により同一人物か否かが判定される場合と比較すると、処理負荷を軽減できる。よって、接客モニタリングシステム１００Ｂ及び接客モニタリング装置１０Ｂは、動線情報や滞留情報に係る人物の識別精度を向上できるので、接客の判定精度を向上できる。 As a result, the service monitoring system 100B and the service monitoring device 10B are provided separately if the flow line information and the stay information are once blocked and the stay information can be linked by speaker identification even if another person is detected. Can be handled as flow line information and staying information of the same person. In addition, the customer service monitoring system 100B and the customer service monitoring device 10B determine whether or not they are the same person using the audio data, so compared with the case where it is determined whether or not they are the same person using face matching or the like using the image data. , Processing load can be reduced. Therefore, since the customer service monitoring system 100B and the customer service monitoring device 10B can improve the person identification accuracy related to the flow line information and the stay information, the customer service determination accuracy can be improved.
　（他の実施形態）
　以上のように、本開示における技術の例示として、第１～第３の実施形態を説明した。しかし、本開示における技術は、これに限定されず、変更、置き換え、付加、省略などを行った実施形態にも適用できる。また、各実施形態を組み合わせてもよい。 (Other embodiments)
As described above, the first to third embodiments have been described as examples of the technology in the present disclosure. However, the technology in the present disclosure is not limited to this, and can also be applied to embodiments in which changes, replacements, additions, omissions, and the like are performed. Moreover, you may combine each embodiment.
　第１～第３の実施形態では、プロセッサは、物理的にどのように構成してもよい。また、プログラム可能なプロセッサを用いれば、プログラムの変更により処理内容を変更できるので、プロセッサの設計の自由度を高めることができる。プロセッサは、１つの半導体チップで構成してもよいし、物理的に複数の半導体チップで構成してもよい。複数の半導体チップで構成する場合、第１～第３の実施形態の各制御をそれぞれ別の半導体チップで実現してもよい。この場合、それらの複数の半導体チップで１つのプロセッサを構成すると考えることができる。また、プロセッサは、半導体チップと別の機能を有する部材（コンデンサ等）で構成してもよい。また、プロセッサが有する機能とそれ以外の機能とを実現するように、１つの半導体チップを構成してもよい。また、複数のプロセッサが１つのプロセッサで構成されてもよい。 In the first to third embodiments, the processor may be physically configured in any manner. Further, if a programmable processor is used, the processing contents can be changed by changing the program, so that the degree of freedom in designing the processor can be increased. The processor may be composed of one semiconductor chip or physically composed of a plurality of semiconductor chips. When configured by a plurality of semiconductor chips, each control of the first to third embodiments may be realized by different semiconductor chips. In this case, it can be considered that a plurality of semiconductor chips constitute one processor. Further, the processor may be configured by a member (capacitor or the like) having a function different from that of the semiconductor chip. Further, one semiconductor chip may be configured so as to realize the functions of the processor and other functions. In addition, a plurality of processors may be configured by one processor.
　第１～第３の実施形態では、図１，５，１１において接客モニタリングシステム及び接客モニタリング装置の構成を示したが、各構成は、ハードウェアにより実現されてもよいし、ソフトウェアにより実現されてもよい。 In the first to third embodiments, the configurations of the service monitoring system and the service monitoring device are shown in FIGS. 1, 5, and 11, but each configuration may be realized by hardware or software. Also good.
　本開示は、接客の判定精度を向上できる接客モニタリング装置、接客モニタリングシステム、及び接客モニタリング方法等に有用である。 This disclosure is useful for a customer service monitoring device, a customer service monitoring system, and a customer service monitoring method that can improve the accuracy of customer service determination.
　１００，１００Ａ，１００Ｂ　　接客モニタリングシステム
　１０，１０Ａ，１０Ｂ　　接客モニタリング装置
　１１　　通信部
　１２　　制御部
　１３　　メモリ
　１４　　操作部
　１５　　ディスプレイ装置
　１６　　スピーカ装置
　２０，２０Ａ，２０Ｂ　　音声処理部
　２１　　音声検出部
　２２　　検出結果累積部
　２３　　指向性合成部
　２４　　音声判別部
　２５　　話者識別部
　３０，３０Ｂ　　人物検出部
　３１　　移動体滞留情報生成部
　３２　　ＩＤ滞留情報生成部
　３３　　滞留情報調整部
　４０，４０Ａ　　接客判定部
　４１　　接客条件設定部
　４２　　接客行動抽出部
　４３　　行動判定部
　４４　　接客要否判定部
　４５　　端末通知部
　Ａ１　　商品見学エリア
　Ａ２　　店員探索エリア
　ＣＡ　　カメラ装置
　ＣＴ，ＣＴ１，ＣＴ２，ＣＴ１１　　客
　ＨＤ，ＨＤ１　　頭部
　ＭＡ　　マイクアレイ装置
　ＮＴ　　ネットワーク
　ＲＣ　　レコーダ
　ＳＡ，ＳＡ１，ＳＡ２，ＳＡ３，Ｓ１１，Ｓ１２　　店員
　ＳＤ　　肩部
　ＳＳ，ＳＳ１～ＳＳ６，ＳＳ２１　　商品棚
　ＴＥ　　携帯端末 100, 100A, 100B Service monitoring system 10, 10A, 10B Service monitoring device 11 Communication unit 12 Control unit 13 Memory 14 Operation unit 15 Display device 16 Speaker device 20, 20A, 20B Audio processing unit 21 Audio detection unit 22 Detection result accumulation unit DESCRIPTION OF SYMBOLS 23 Directionality synthetic | combination part 24 Voice discrimination | determination part 25 Speaker identification part 30, 30B Person detection part 31 Mobile body residence information generation part 32 ID residence information generation part 33 Residence information adjustment part 40, 40A Customer service determination part 41 Customer service condition setting part 42 Customer Service Extraction Unit 43 Behavior Judgment Unit 44 Customer Service Necessity Determination Unit 45 Terminal Notification Unit A1 Product Visit Area A2 Clerk Search Area CA Camera Device CT, CT1, CT2, CT11 Customer HD, HD1 Head MA Microphone Array Device NT Network RC Over da SA, SA1, SA2, SA3, S11, S12 clerk SD shoulder SS, SS1 ~ SS6, SS21 shelf TE mobile terminal

Claims (11)

  1. 　複数のグリッドに区分される対象エリアの音声を収音する収音装置により収音された音声に係る音声データを取得し、前記対象エリアの画像を撮像する撮像装置により撮像された画像に係る画像データを取得する通信部と、
    　前記音声データを基に、発話を検出する発話検出部と、
    　前記画像データを基に、接客者又は客を含む人物の滞留情報を生成し、前記対象エリア内に所在する各接客者が所持する携帯端末の位置情報を基に、前記接客者の滞留情報を生成する滞留情報生成部と、
    　前記人物の滞留情報、前記接客者の滞留情報、及び前記発話の検出結果を基に、同一のグリッドに前記接客者と客とが第１の所定時間滞留し、且つ、前記同一のグリッドにおいて第２の所定時間以上発話が検出された場合、前記同一のグリッドにおいて前記接客者が接客していると判定する接客判定部と、
    　を備える接客モニタリング装置。     An image related to an image picked up by an image pickup device for acquiring sound data related to sound picked up by a sound pickup device that picks up sound of a target area divided into a plurality of grids and picking up an image of the target area A communication unit for acquiring data;
    An utterance detection unit for detecting an utterance based on the voice data;
    Based on the image data, the staying information of the customer or the person including the customer is generated, and the staying information of the customer is obtained based on the position information of the mobile terminal possessed by each customer located in the target area. A retention information generation unit to generate,
    Based on the retention information of the person, the retention information of the customer, and the detection result of the utterance, the customer and the customer stay in the same grid for a first predetermined time, and in the same grid If the utterance is detected for a predetermined time of 2 or more, a service determination unit that determines that the service is in the same grid,
    A customer service monitoring device.
  2. 　請求項１に記載の接客モニタリング装置であって、
    　前記接客判定部は、任意のグリッドにおいて前記第１の所定時間以上滞留する第１の客を１人検出し、前記第１の客が発した特定のキーワードが検出された場合、又は前記第１の客が特定の行動が検出された場合、前記第１の客に対する接客が必要であると判定し、
    　前記通信部は、前記第１の客に対する接客が必要な場合、前記接客者の少なくとも一部が所持する携帯端末へ、前記第１の客に対する接客を指示するための接客指示情報を送信する、接客モニタリング装置。     The customer service monitoring device according to claim 1,
    The customer service determination unit detects one first customer who stays for at least the first predetermined time in an arbitrary grid, and when a specific keyword issued by the first customer is detected, or the first When a specific behavior is detected, the customer is determined to be in need of serving the first customer,
    The communication unit transmits customer service instruction information for instructing customer service to the first customer to a portable terminal possessed by at least a part of the customer when customer service to the first customer is required. Hospitality monitoring device.
  3. 　請求項２に記載の接客モニタリング装置であって、
    　前記接客判定部は、前記第１の客の頭部又は肩部の検出位置と、前記頭部又は前記肩部の検出時間又は検出回数と、に基づいて、前記特定の行動をしているか否かを判定する、
    接客モニタリング装置。     The customer service monitoring device according to claim 2,
    Whether the customer service determination unit performs the specific action based on the detection position of the head or shoulder of the first customer and the detection time or the number of detections of the head or shoulder. To determine,
    Hospitality monitoring device.
  4. 　請求項２に記載の接客モニタリング装置であって、
    　前記接客判定部は、前記第１の客の滞留情報が所定の滞留パターンを含む場合、前記特定の行動をしていると判定する、接客モニタリング装置。     The customer service monitoring device according to claim 2,
    The customer service monitoring device, wherein the customer service determination unit determines that the specific customer is performing the specific behavior when the first customer's stay information includes a predetermined stay pattern.
  5. 　請求項２に記載の接客モニタリング装置であって、
    　前記通信部は、前記接客者の滞留情報に基づいて、任意のグリッド内に１人で所在する接客者へ前記接客指示情報を送信する、接客モニタリング装置。     The customer service monitoring device according to claim 2,
    The said communication part is the customer service monitoring apparatus which transmits the said customer service instruction information to the customer service person who exists in arbitrary grids based on the said customer's stay information.
  6. 　請求項２に記載の接客モニタリング装置であって、
    　前記通信部は、前記接客者の滞留情報及び前記人物の滞留情報に基づいて、非接客者へ前記接客指示情報を送信する、接客モニタリング装置。     The customer service monitoring device according to claim 2,
    The said communication part is a customer service monitoring apparatus which transmits the said customer service instruction information to a non-customer based on the said customer's customer's stay information and the said person's stay information.
  7. 　請求項２に記載の接客モニタリング装置であって、
    　前記通信部は、任意のグリッド内に１人で所在する接客者が存在しない場合、前記接客者の滞留情報が生成されなかった接客者が所持する携帯端末へ、前記接客指示情報を送信する、接客モニタリング装置。     The customer service monitoring device according to claim 2,
    The communication unit transmits the customer service instruction information to a mobile terminal possessed by a customer who does not generate the staying information of the customer when there is no customer who is present alone in any grid. Hospitality monitoring device.
  8. 　請求項２に記載の接客モニタリング装置であって、
    　前記通信部は、任意のグリッド内に１人で所在する接客者が存在しない場合、全ての接客者が所持する携帯端末へ、前記接客指示情報を送信する、接客モニタリング装置。     The customer service monitoring device according to claim 2,
    The customer service monitoring device, wherein the communication unit transmits the customer service instruction information to portable terminals possessed by all customer customers when there is no customer customer located in any grid.
  9. 　請求項１に記載の接客モニタリング装置であって、更に、
    　前記発話検出部により検出された発話に係る第１の話者を話者識別処理により識別し、前記第１の話者に係る第１の話者識別情報をメモリに保持させる話者識別部と、
    　前記第１の話者識別情報と、前記話者識別部により前記第１の話者識別情報が得られた後に識別された第２の話者に係る話者識別情報と、が一致する場合、前記第１の話者に係る前記人物の滞留情報又は前記接客者の滞留情報に、前記第２の話者に係る前記人物の滞留情報又は前記接客者の滞留情報を統合する滞留情報調整部と、
    　を備える接客モニタリング装置。     The customer service monitoring device according to claim 1, further comprising:
    A speaker identification unit that identifies a first speaker related to the utterance detected by the utterance detection unit by a speaker identification process, and stores first speaker identification information related to the first speaker in a memory; ,
    When the first speaker identification information matches the speaker identification information related to the second speaker identified after the first speaker identification information is obtained by the speaker identification unit, A residence information adjustment unit that integrates the residence information of the person relating to the second speaker or the residence information of the customer with the residence information of the person relating to the first speaker or the residence information of the customer serving; ,
    A customer service monitoring device.
  10. 　複数のグリッドに区分される対象エリアの音声を収音する収音装置と、前記対象エリアの画像を撮像する撮像装置と、接客モニタリング装置と、を備える接客モニタリングシステムであって、
    　前記接客モニタリング装置は、
    　前記収音装置により収音された音声に係る音声データを取得し、
    　前記撮像装置により撮像された画像に係る画像データを取得し、
    　前記音声データを基に、発話を検出し、
    　前記画像データを基に、接客者又は客を含む人物の滞留情報を生成し、
    　前記対象エリア内に所在する各接客者が所持する携帯端末の位置情報を基に、前記接客者の滞留情報を生成し、
    　前記人物の滞留情報、前記接客者の滞留情報、及び前記発話の検出結果を基に、同一のグリッドに前記接客者と客とが第１の所定時間滞留し、且つ、前記同一のグリッドにおいて第２の所定時間以上発話が検出された場合、前記同一のグリッドにおいて前記接客者が接客していると判定する、接客モニタリングシステム。     A customer service monitoring system comprising: a sound collection device that collects sound of a target area divided into a plurality of grids; an imaging device that captures an image of the target area; and a customer service monitoring device.
    The customer monitoring device is:
    Obtaining voice data related to the voice collected by the sound collection device;
    Obtaining image data relating to an image captured by the imaging device;
    Based on the voice data, utterance is detected,
    Based on the image data, generate the stay information of the customer or the person including the customer,
    Based on the location information of the mobile terminal possessed by each customer who is located in the target area, to generate the stay information of the customer,
    Based on the retention information of the person, the retention information of the customer, and the detection result of the utterance, the customer and the customer stay in the same grid for a first predetermined time, and in the same grid A customer service monitoring system that determines that the customer service is in the same grid when an utterance is detected for a predetermined time of 2 or more.
  11. 　接客モニタリング装置における接客モニタリング方法であって、
    　複数のグリッドに区分される対象エリアの音声を収音する収音装置により収音された音声に係る音声データを取得し、
    　前記対象エリアの画像を撮像する撮像装置により撮像された画像に係る画像データを取得し、
    　前記音声データを基に、発話を検出する発話検出部と、
    　前記画像データを基に、接客者又は客を含む人物の滞留情報を生成し、
    　前記対象エリア内に所在する各接客者が所持する携帯端末の位置情報を基に、前記接客者の滞留情報を生成し、
    　前記人物の滞留情報、前記接客者の滞留情報、及び前記発話の検出結果を基に、同一のグリッドに前記接客者と客とが第１の所定時間滞留し、且つ、前記同一のグリッドにおいて第２の所定時間以上発話が検出された場合、前記同一のグリッドにおいて前記接客者が接客していると判定する、接客モニタリング方法。     A service monitoring method in a service monitoring device,
    Acquire audio data related to audio collected by a sound collection device that collects audio in a target area divided into a plurality of grids;
    Obtaining image data related to an image captured by an imaging device that captures an image of the target area;
    An utterance detection unit for detecting an utterance based on the voice data;
    Based on the image data, generate the stay information of the customer or the person including the customer,
    Based on the location information of the mobile terminal possessed by each customer who is located in the target area, to generate the stay information of the customer,
    Based on the retention information of the person, the retention information of the customer, and the detection result of the utterance, the customer and the customer stay in the same grid for a first predetermined time, and in the same grid A customer service monitoring method, in which, when an utterance is detected for two or more predetermined times, it is determined that the customer service is in the same grid.
PCT/JP2017/020901 2016-08-02 2017-06-06 Customer attendance monitoring device, customer attendance monitoring system, and customer attendance monitoring method WO2018025492A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2016-152135 2016-08-02
JP2016152135A JP2018022284A (en) 2016-08-02 2016-08-02 Customer service monitoring device, customer service monitoring system, and customer service monitoring method

Publications (1)

Publication Number Publication Date
WO2018025492A1 true WO2018025492A1 (en) 2018-02-08

Family

ID=61073861

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2017/020901 WO2018025492A1 (en) 2016-08-02 2017-06-06 Customer attendance monitoring device, customer attendance monitoring system, and customer attendance monitoring method

Country Status (2)

Country Link
JP (1) JP2018022284A (en)
WO (1) WO2018025492A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2021536046A (en) * 2019-08-14 2021-12-23 北京市商▲湯▼科技▲開▼▲發▼有限公司Beijing Sensetime Technology Development Co., Ltd. Data processing methods, equipment, and storage media

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2589025A (en) * 2018-07-05 2021-05-19 Panasonic Ip Man Co Ltd Sales promotion system and sales promotion method
JP7366547B2 (en) * 2019-01-10 2023-10-23 東芝テック株式会社 Information processing device and information processing program
JP7336674B2 (en) * 2019-02-07 2023-09-01 パナソニックＩｐマネジメント株式会社 Customer Service Situation Analysis Device, Customer Service Situation Analysis System, and Customer Service Situation Analysis Method
JPWO2021038814A1 (en) * 2019-08-29 2021-09-27 株式会社オプティム Computer systems, specific situation improvement methods and programs
JP6954687B1 (en) * 2020-06-17 2021-10-27 Ｎｅｃプラットフォームズ株式会社 Customer service management system, customer service management method, management server and program
CN116157833A (en) * 2020-09-23 2023-05-23 Jvc建伍株式会社 Image processing device and image processing program
JP7315049B1 (en) 2022-02-22 2023-07-26 富士通株式会社 Information processing program, information processing method, and information processing apparatus

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07287781A (en) * 1994-04-19 1995-10-31 Hitachi Ltd Device for identifying/tracking mobile object
JP2009238044A (en) * 2008-03-27 2009-10-15 Brother Ind Ltd Customer service support system
JP2011237966A (en) * 2010-05-10 2011-11-24 Seiko Epson Corp Customer service support device, customer service support method and program
JP5879616B1 (en) * 2014-10-07 2016-03-08 パナソニックＩｐマネジメント株式会社 Activity status analysis system, activity status analysis device, activity status analysis method, activity status analysis program, and storage medium for storing the program

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07287781A (en) * 1994-04-19 1995-10-31 Hitachi Ltd Device for identifying/tracking mobile object
JP2009238044A (en) * 2008-03-27 2009-10-15 Brother Ind Ltd Customer service support system
JP2011237966A (en) * 2010-05-10 2011-11-24 Seiko Epson Corp Customer service support device, customer service support method and program
JP5879616B1 (en) * 2014-10-07 2016-03-08 パナソニックＩｐマネジメント株式会社 Activity status analysis system, activity status analysis device, activity status analysis method, activity status analysis program, and storage medium for storing the program

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2021536046A (en) * 2019-08-14 2021-12-23 北京市商▲湯▼科技▲開▼▲發▼有限公司Beijing Sensetime Technology Development Co., Ltd. Data processing methods, equipment, and storage media
JP7082688B2 (en) 2019-08-14 2022-06-08 北京市商▲湯▼科技▲開▼▲發▼有限公司 Data processing methods, equipment, and storage media

Also Published As

Publication number Publication date
JP2018022284A (en) 2018-02-08

Similar Documents

Publication Publication Date Title
WO2018025492A1 (en) Customer attendance monitoring device, customer attendance monitoring system, and customer attendance monitoring method
JP5871296B1 (en) Smart security digital system, method and program
JP6135880B2 (en) Audio processing method, audio processing system, and storage medium
US10182280B2 (en) Sound processing apparatus, sound processing system and sound processing method
US8184154B2 (en) Video surveillance correlating detected moving objects and RF signals
EP3059733A2 (en) Automatic alerts for video surveillance systems
JP6674584B2 (en) Video surveillance system
US20170330208A1 (en) Customer service monitoring device, customer service monitoring system, and customer service monitoring method
US20150281832A1 (en) Sound processing apparatus, sound processing system and sound processing method
JP2012118838A (en) Apparatus and method for monitoring actions of a monitoring target person
JP6887102B2 (en) Audio processing equipment, image processing equipment, microphone array system, and audio processing method
US20120250581A1 (en) Ad-Hoc Surveillance Network
US20170262725A1 (en) Method and arrangement for receiving data about site traffic derived from imaging processing
EP3121799A1 (en) Digital loss-defense security system, method, and program
JP2018132803A (en) Person detection system
US11514600B2 (en) Information processing device and information processing method
JP2016152557A (en) Sound collection system and sound collection setting method
KR20120113014A (en) Image recognition apparatus and vison monitoring method thereof
WO2015151130A1 (en) Sound processing apparatus, sound processing system, and sound processing method
JP7138547B2 (en) store equipment
JP2018120345A (en) Customer service status analyzing apparatus and customer service status analyzing system
JP6773389B1 (en) Digital autofile security system, methods and programs
WO2021186751A1 (en) Digital auto-filing security system, method, and program
JPWO2018143341A1 (en) Monitoring system, monitoring method and program
EP2927885A1 (en) Sound processing apparatus, sound processing system and sound processing method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17836599

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17836599

Country of ref document: EP

Kind code of ref document: A1