CN106373456B - Based on AR augmented reality and play the violin assisted learning experiencing system evaluated - Google Patents
Based on AR augmented reality and play the violin assisted learning experiencing system evaluated Download PDFInfo
- Publication number
- CN106373456B CN106373456B CN201610985563.XA CN201610985563A CN106373456B CN 106373456 B CN106373456 B CN 106373456B CN 201610985563 A CN201610985563 A CN 201610985563A CN 106373456 B CN106373456 B CN 106373456B
- Authority
- CN
- China
- Prior art keywords
- violin
- unit
- animated element
- image
- processing unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B15/00—Teaching music
- G09B15/02—Boards or like means for providing an indication of notes
- G09B15/023—Electrically operated
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Business, Economics & Management (AREA)
- Physics & Mathematics (AREA)
- Educational Administration (AREA)
- Educational Technology (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Auxiliary Devices For Music (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention discloses a kind of based on AR augmented reality and plays the violin assisted learning experiencing system of evaluation, comprising: image acquisition units acquire the video image that currently used person holds violin fingerboard in real time, and video image also includes to hold qin hand finger nail；Image analyzing unit analyzes the image of current violin, identifies profile and the fingerboard region of current violin, four strings of identification positioning violin；Storage unit, the unit store the virtual animated element of music score and syllable and the corresponding push action finger of note and bow operation；And operating unit, processing unit and display unit；Profile and fingerboard region of the processing unit according to the current violin, four strings of the violin after positioning are shown on the display unit, and set the animated element display area；Processing unit transfers the corresponding music score in storage unit and the virtual animated element of corresponding push action and bow operation, shows in the animated element display area.
Description
Technical field
The present invention relates to the violin assisted learning experiencing systems based on AR augmented reality and performance evaluation.
Background technique
All the time, a large amount of repeatability practice is required, it is very big that violin this performance difficulty can be grasped
Musical instrument.And in learning process, need music teachers to teach the practice of supervision student in real time, meanwhile, need student from basic skill
Method starts continual a large amount of repetition trainings, until forming muscle memory, just calculates real acquistion violin skill and technique, entire to learn
It is very long and uninteresting to practise process exception, needs great willpower and habit that could complete.This high threshold, by many music, especially
It is that violin fan hangs back, greatly hinders the universal of violin, affects the universal of comprehensive musical quality.
AR enhances display technology, by Internet technology, can obtain peripheral object within the scope of the normal visual field of human eye
Information, and by the information of the target on periphery display on specific display interface, increase particular range in crowd interaction.
In conjunction with technologies such as big data analysis, Artificial intelligent algorithms, specific content can be shown in the display interface, greatly
The visual field of the mankind is expanded.
Since the performance of the musical instruments such as violin all has a relatively-stationary method, mainly fortune bow method and hold qin finger by
The movement of string two parts is pressed, and although different player's hand-types and fortune bow mode has certain difference, generally all
A possibility that finger for needing to complete core presses and string run action, this just brings action criteria, if utilizing AR
Enhance display technology, provides the corresponding prompt of music score, will be expected to get rid of traditional performance learning process, reduce violin
Threshold.
Summary of the invention
The it is proposed of the present invention in view of the above problems, and a kind of violin based on AR augmented reality and performance evaluation developed
Assisted learning experiencing system, comprising:
Image acquisition units, acquire the video image that currently used person holds violin fingerboard in real time, and video image also wraps
Containing holding qin hand finger nail；
Image analyzing unit analyzes the image of current violin, identifies profile and the fingerboard region of current violin, knows
Not Ding Wei violin four strings；
What storage unit, unit storage music score and syllable and the corresponding push action finger of note and bow were run
Virtual animated element；And operating unit, processing unit and display unit；
In use, user is started by operating unit control system, current performance song is selected；Processing unit is according to institute
The profile for the current violin stated and fingerboard region, four strings of the violin after positioning are shown on the display unit
Show, and sets the animated element display area；Processing unit transfer corresponding music score in storage unit and it is corresponding by
The virtual animated element for pressing work and bow operation, shows in the animated element display area；
User guides according to the virtual animated element, completes the performance of current melody, i.e. learning experience process.
In playing procedure, the processing unit obtains playing music purpose audio by sound collection unit, and analysis obtains
The audio curve stored in audio curve, with storage unit compares, and provides the scoring that currently used person plays level.
As preferred embodiment, the image acquisition units, image analyzing unit, operating unit and processing unit
And display unit is integrated on intelligent terminal or intelligent helmet/glasses.
As preferred embodiment, in use process, what the processing unit was collected by image acquisition units
Image tracks the position of fortune the bow track and nail of currently used person, especially presses the position of the nail of string, and described
The virtual animated element stored in storage unit is compared；
When the fortune bow track and nail position that determine user and virtual animated element close in threshold range, determine
Currently used person completes the correct performance of current note or syllable, and provides prompt.
As preferred embodiment, the string and bow surface are equipped with the equidistant mark that axis extends.
As preferred embodiment, user holds qin hand finger nail and is respectively equipped with the distinctive mark mutually distinguished.
Further, the distinctive mark is integrated on an auxiliary glove；
The glove fingertip position is equipped with electro photoluminescence patch；When the processing unit provides finger according to current note
Corresponding virtual animated element when, the electro photoluminescence patch for controlling corresponding finger is powered.
As preferred embodiment, the fixture of fixed intelligent terminal is equipped with above the tight boom plate of violin.
By using above-mentioned technical proposal, the violin auxiliary proposed by the present invention based on AR augmented reality and performance evaluation
Learning experience system, using existing hardware device, such as the equipment such as intelligent terminal (mobile phone) or AR glasses, based on maturation
Recognizer and locating and displaying technology realize the virtual auxiliary to violin user, greatly reduce violin
Threshold, technology maturation is reliable, be fully compatible on a large scale promote and use.
Detailed description of the invention
For the clearer technical solution for illustrating the embodiment of the present invention or the prior art, to embodiment or will show below
There is attached drawing needed in technical description to do one simply to introduce, it should be apparent that, the accompanying drawings in the following description is only
Some embodiments of the present invention without creative efforts, may be used also for those of ordinary skill in the art
To obtain other drawings based on these drawings.
Fig. 1 is system module figure of the invention
Specific embodiment
To keep the purposes, technical schemes and advantages of the embodiment of the present invention clearer, below with reference to the embodiment of the present invention
In attached drawing, technical solution in the embodiment of the present invention carries out clear and complete description:
A kind of violin assisted learning experiencing system based on AR augmented reality and performance evaluation as shown in Figure 1:, it is main to wrap
Image acquisition units are included, in use, which acquires the video image that currently used person holds violin fingerboard in real time,
Simultaneously for subsequent video identification, the video image also includes to hold qin hand finger nail.
Image analyzing unit analyzes the image of current violin, identifies profile and the fingerboard region of current violin, knows
Not Ding Wei violin four strings, that is, specific image-region is divided, as subsequent algorithm foundation.It was run subsequent
Cheng Zhong, image analyzing unit string, profile and fingerboard region described in work shadow always.
Storage unit, it is minimum in unit storage music score and a collection of impressions of seals by famous seal to play unit syllable and note is corresponding by pressing
Make the position of finger and the virtual animated element of bow operation, such as when a performance unit series of sound is needed by next string, simultaneously
Bow is needed when running full bow on a string, and virtual animation display indicates the animated element of index finger, for example, with number (such as
Index finger sets label 1) or it is different from the figures of other fingers, the animated element moved downward, while in a string of setting
Fortune bow angular regions show the animation of bow figure and the traffic direction with special angle.It is also possible to provide corresponding
The information such as text prompt or voice prompting.
In use, user is started by operating unit control system, selects current performance song；Processing unit
According to the profile of the current violin and fingerboard region, four strings of the violin after positioning, in the display list
It is shown in member.
The animated element display area delimited on the display region.Meanwhile processing unit is transferred in storage unit
The virtual animated element of corresponding music score and corresponding push action and bow operation, in the animated element display area
Display；User guides according to the virtual animated element, completes the performance of current melody, i.e. learning experience process.
In order to expand corresponding function, in playing procedure, the processing unit is drilled by sound collection unit acquisition
The audio of song is played, analysis obtains audio curve, compares with the audio curve stored in storage unit, provide currently used person and drill
Play the scoring of level.
As optional embodiment, sound collection unit can be used the earpiece of smart phone, or use integrate it is sound
Sound acquires the AR glasses of equipment or accesses existing equipment using Wireless/wired sound collection equipment.Scoring algorithm can use
The similar algorithm such as singing marking algorithm in existing similar KTV can be realized, the purpose in encouragement, and acquisition key sound is
It can.
Correspondingly, may also set up the evaluation criterion of different difficulty, for the user of different levels, different stage is set
Algorithm increases the sustainability operation of system.
In order to utilize the technical equipment of existing maturation as far as possible, increases device integration, reduce user and wear
Number of articles, as preferred embodiment, the image acquisition units, image analyzing unit, operating unit and processing are single
Member and display unit are integrated on intelligent terminal or intelligent helmet/glasses.As the core of system, replacement operation core and phase
The scheme of this patent proposition can be realized in the algorithm answered.
Whether the operation in order to identify currently used person is correct, as preferred embodiment, the processing list
The image that member is collected by image acquisition units is tracked the position of fortune the bow track and nail of currently used person, is especially pressed
The position of the nail of lower string is compared with the virtual animated element stored in the storage unit.
More feasible embodiment, when the fortune for determining user bends track and nail position and virtual animated element in threshold
When being worth close in range, in display interface, it can be understood as virtual animated element and finger nail appear in similar region substantially,
It can significantly be overlapped, that is, can determine that currently used person according to system suggestion, complete current note or the correct of syllable is drilled
It plays, and provides the prompt correctly played.
Herein, the double hit prompt mechanism in other game can be used for reference, when being correctly completed the performance of a note
The prompt for providing+1 provides the prompt of+N when being continuously correctly completed performance, and provide corresponding encouragement word or class
Like prompt.
Further, it is contemplated that the color of bow is closer to violin body color, leads to recognizer excessively
Complexity, as preferred embodiment, the string and bow surface are equipped with the equidistant mark that axis extends.Optional interval
The black and white color lump of setting identifies convenient for algorithm, reduces the identification runing time of algorithm.
Likewise, user, which holds qin hand finger nail, is respectively equipped with the distinctive mark mutually distinguished.Such as to respectively correspond
1/2/3/4 number is respectively set in index finger, middle finger, the third finger and the little finger of toe of 1/2/3/4 string, identifies convenient for computer.
As preferred embodiment, the distinctive mark is integrated in position corresponding with nail on an auxiliary glove.
In order to the various stimulations of user and guidance, as preferred embodiment, the processing unit
When providing the corresponding virtual animated element of finger according to current note, the electro photoluminescence patch for controlling corresponding finger is powered.
Above the tight boom plate of violin, it is equipped with the fixture of fixed intelligent terminal close to the position of micro actuator, facilitates fixed mobile phone
Etc. equipment.
The foregoing is only a preferred embodiment of the present invention, but scope of protection of the present invention is not limited thereto,
Anyone skilled in the art in the technical scope disclosed by the present invention, according to the technique and scheme of the present invention and its
Inventive concept is subject to equivalent substitution or change, should be covered by the protection scope of the present invention.
Claims (5)
1. a kind of violin assisted learning experiencing system based on AR augmented reality and performance evaluation, characterized by comprising:
Image acquisition units acquire the video image that currently used person holds violin fingerboard in real time, and video image also includes to hold
Qin hand finger nail；
Image analyzing unit analyzes the image of current violin, identifies profile and the fingerboard region of current violin, and identification is fixed
Four strings of position violin；
Storage unit, unit storage music score and syllable and the corresponding push action finger of note and bow run virtual
Animated element；And operating unit, processing unit and display unit；
In use, user is started by operating unit control system, current performance song is selected；Processing unit is according to described
Currently the profile of violin and fingerboard region, four strings of the violin after positioning are shown on the display unit, and
Set animated element display area；Processing unit transfers corresponding music score in storage unit and corresponding push action and bow
The virtual animated element of operation, shows in the animated element display area；
In use process, the image that the processing unit is collected by image acquisition units tracks the fortune of currently used person
The position for bending track and nail, especially presses the position of the nail of string, virtual dynamic with storing in the storage unit
Element is drawn to be compared；
When the fortune bow track and nail position that determine user and virtual animated element close in threshold range, determine current
User completes the correct performance of current note or syllable, and provides prompt；
User guides according to the virtual animated element, completes the performance of current melody；
In playing procedure, the processing unit obtains playing music purpose audio by sound collection unit, and analysis obtains audio
The audio curve stored in curve, with storage unit compares, and provides the scoring that currently used person plays level.
2. the violin assisted learning experiencing system according to claim 1 based on AR augmented reality and performance evaluation,
It is further characterized in that image acquisition units, image analyzing unit, operating unit and the processing unit and display unit are integrated
On intelligent terminal or intelligent helmet/glasses.
3. the violin assisted learning experiencing system according to claim 1 based on AR augmented reality and performance evaluation,
It is further characterized in that the string and bow surface are equipped with the equidistant mark that axis extends.
4. the violin assisted learning experiencing system according to claim 1 based on AR augmented reality and performance evaluation,
It is further characterized in that user holds qin hand finger nail and is respectively equipped with the distinctive mark mutually distinguished.
5. the violin assisted learning experiencing system according to claim 1 based on AR augmented reality and performance evaluation,
It is further characterized in that above the tight boom plate of violin, the fixture of fixed intelligent terminal is equipped with close to the position of micro actuator.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610985563.XA CN106373456B (en) | 2016-11-09 | 2016-11-09 | Based on AR augmented reality and play the violin assisted learning experiencing system evaluated |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610985563.XA CN106373456B (en) | 2016-11-09 | 2016-11-09 | Based on AR augmented reality and play the violin assisted learning experiencing system evaluated |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106373456A CN106373456A (en) | 2017-02-01 |
| CN106373456B true CN106373456B (en) | 2019-03-05 |
Family
ID=57893825
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610985563.XA Active CN106373456B (en) | 2016-11-09 | 2016-11-09 | Based on AR augmented reality and play the violin assisted learning experiencing system evaluated |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106373456B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109697918B (en) * | 2018-12-29 | 2021-04-27 | 深圳市掌网科技股份有限公司 | Percussion instrument experience system based on augmented reality |
| CN109493686B (en) * | 2018-12-29 | 2021-03-30 | 深圳市掌网科技股份有限公司 | Piano training system and method based on mixed reality |
| CN110689866A (en) * | 2019-09-18 | 2020-01-14 | 江西昕光年智能科技有限公司 | Violin auxiliary teaching method and system based on augmented reality |
| CN110910712B (en) * | 2019-12-06 | 2021-06-04 | 中国美术学院 | Zheng auxiliary teaching system and method based on AR |
| CN112150583B (en) * | 2020-09-02 | 2024-07-23 | 广东小天才科技有限公司 | Spoken language pronunciation assessment method and terminal equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101504832A (en) * | 2009-03-24 | 2009-08-12 | 北京理工大学 | Virtual performance system based on hand motion sensing |
| CN102737534A (en) * | 2011-04-13 | 2012-10-17 | 南京大学 | Method for realizing unmarked augmented reality piano teaching system |
| CN103794106A (en) * | 2014-03-11 | 2014-05-14 | 哈尔滨工业大学 | Assisted teaching system for musical instrument playing |
| CN104217625A (en) * | 2014-07-31 | 2014-12-17 | 合肥工业大学 | Augmented reality based assisted piano learning system |
| CN105632263A (en) * | 2016-03-29 | 2016-06-01 | 罗昆 | Augmented reality-based music enlightenment learning device and method |
-
2016
- 2016-11-09 CN CN201610985563.XA patent/CN106373456B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101504832A (en) * | 2009-03-24 | 2009-08-12 | 北京理工大学 | Virtual performance system based on hand motion sensing |
| CN102737534A (en) * | 2011-04-13 | 2012-10-17 | 南京大学 | Method for realizing unmarked augmented reality piano teaching system |
| CN103794106A (en) * | 2014-03-11 | 2014-05-14 | 哈尔滨工业大学 | Assisted teaching system for musical instrument playing |
| CN104217625A (en) * | 2014-07-31 | 2014-12-17 | 合肥工业大学 | Augmented reality based assisted piano learning system |
| CN105632263A (en) * | 2016-03-29 | 2016-06-01 | 罗昆 | Augmented reality-based music enlightenment learning device and method |
Non-Patent Citations (2)
| Title |
|---|
| Music Education using Augmented Reality with a Head Mounted;Jonathan Chow等;《Proceedings of the Fourteenth Australasian User Interface Conference (AUIC2013)》;20131231;第73-79页 |
| 基于增强现实的交互式古琴学习辅助***;张映雪;《中国优秀硕士学位论文全文数据库（电子期刊）》;20160831;第12页-第37页 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106373456A (en) | 2017-02-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106340215B (en) | Musical instrument assisted learning experiencing system based on AR augmented reality and self-adapting estimation | |
| CN106527718B (en) | Musical instrument auxiliary based on AR augmented reality is played and experiencing system is played by association | |
| CN106355974B (en) | Violin assisted learning experiencing system based on AR augmented reality | |
| CN106373456B (en) | Based on AR augmented reality and play the violin assisted learning experiencing system evaluated | |
| US8793118B2 (en) | Adaptive multimodal communication assist system | |
| US9524652B2 (en) | System and method for learning to play a musical instrument | |
| US20150332601A1 (en) | Piano Learning System | |
| CN106373440B (en) | Based on AR augmented reality and the violin assisted learning experiencing system that exchanges immediately | |
| CN107481581B (en) | Computer-aided method and computer system for piano teaching | |
| CN101604486A (en) | Musical instrument playing and practicing method based on speech recognition technology of computer | |
| CN110689866A (en) | Violin auxiliary teaching method and system based on augmented reality | |
| US11580868B2 (en) | AR-based supplementary teaching system for guzheng and method thereof | |
| CN106356046A (en) | Method, device and system for evaluating musical instrument playing | |
| US20130098225A1 (en) | Method of improving vocal performance with embouchure functions | |
| US20230230493A1 (en) | Information Processing Method, Information Processing System, and Recording Medium | |
| CN109785710B (en) | Piano exercise assistor | |
| CN110164241A (en) | A kind of guitar teaching method and system | |
| KR20180130432A (en) | Method for Performing Korean Charactor Sound of Drum Music Editing and Apparatus for Converting Music Performance File Composd of Korean Charactor Sound of Drum Music Editing Thereof | |
| CN108091207A (en) | A kind of piano for blind | |
| JP2011221472A (en) | Guitar with image display device | |
| CN108922315A (en) | A kind of guitar auxiliary practice device | |
| KR101131823B1 (en) | Apparatus and method for studying of chinese language | |
| Cosentino et al. | Music conductor gesture recognition by using inertial measurement system for human-robot musical interaction | |
| CN114783222B (en) | Musical instrument auxiliary teaching method and system based on AR technology | |
| KR102634347B1 (en) | Apparatus for training and testing to cognitive function using user interaction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 116023 No. 11, B block, 32 Torch Road, Dalian high tech Industrial Park, Liaoning, China. Applicant after: Fast creation technology (Dalian) Co., Ltd. Address before: 116000 Pioneer Building B 1101, No. 32 Torch Road, Dalian High-tech Zone, Liaoning Province Applicant before: Dalian Wensente Software Technology Co., Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |