CN114708367A - WebGL-based sign language digital human driving and real-time rendering system - Google Patents
WebGL-based sign language digital human driving and real-time rendering system Download PDFInfo
- Publication number
- CN114708367A CN114708367A CN202210313014.3A CN202210313014A CN114708367A CN 114708367 A CN114708367 A CN 114708367A CN 202210313014 A CN202210313014 A CN 202210313014A CN 114708367 A CN114708367 A CN 114708367A
- Authority
- CN
- China
- Prior art keywords
- sign language
- rendering
- real
- web
- animation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000009877 rendering Methods 0.000 title claims abstract description 46
- 238000013519 translation Methods 0.000 claims abstract description 19
- 208000032041 Hearing impaired Diseases 0.000 claims abstract description 13
- 230000004044 response Effects 0.000 claims abstract description 13
- 238000000034 method Methods 0.000 claims description 14
- 238000013461 design Methods 0.000 claims description 12
- 230000009471 action Effects 0.000 claims description 7
- 238000005286 illumination Methods 0.000 claims description 6
- 238000004422 calculation algorithm Methods 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 5
- 230000007246 mechanism Effects 0.000 claims description 4
- 230000002452 interceptive effect Effects 0.000 claims description 3
- 230000003993 interaction Effects 0.000 claims description 2
- 238000012986 modification Methods 0.000 claims description 2
- 230000004048 modification Effects 0.000 claims description 2
- 239000003086 colorant Substances 0.000 claims 2
- 210000000988 bone and bone Anatomy 0.000 claims 1
- 238000004519 manufacturing process Methods 0.000 claims 1
- 230000009467 reduction Effects 0.000 claims 1
- 238000012545 processing Methods 0.000 abstract description 5
- 238000012821 model calculation Methods 0.000 abstract description 2
- 230000000007 visual effect Effects 0.000 abstract 1
- 230000000694 effects Effects 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 230000033001 locomotion Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000005056 compaction Methods 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 230000003997 social interaction Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/20—3D [Three Dimensional] animation
- G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention discloses a sign language digital human driving and real-time rendering system based on WebGL, relates to the fields of three-dimensional model calculation animation, graph rendering, high concurrent response of a server, real-time task processing and the like, and provides a system for rendering a virtual sign language digital human by using WebGL at a web end, acquiring animation data through a network request and driving the virtual sign language digital human to translate Chinese sign language in real time. The system mainly comprises a web front end, a rendering module, a cloud service module, an animation driving module and the like. The invention can provide online, real-time and accurate visual Chinese sign language translation service for the hearing-impaired people at the web end, provides a new way for the hearing-impaired people to acquire information at the web end, and fills the technical gap that online Chinese sign language translation is not provided for the hearing-impaired people at the web end.
Description
Technical Field
The invention relates to the fields of three-dimensional model calculation animation, graphic rendering, high concurrent response of a server, real-time task processing and the like, in particular to a system for rendering a virtual sign language digital person by using WebGL, acquiring animation data through a network request and driving the virtual sign language digital person to perform sign language translation in real time in a browser supporting 3D rendering.
Background
With the gradual development of the internet, the proportion of people to obtain information from the mobile terminal and the web terminal is greater and greater, and the mobile terminal and the web terminal also become important channels for hearing-impaired people to obtain information. For the hearing-impaired people with low cultural degree, it is very difficult to obtain beneficial information from characters. At present, the application of Chinese sign language translation service for the hearing-impaired people is not provided at the web end, so that the hearing-impaired people cannot acquire correct, beneficial and instant information from social media or other channels, and the social interaction, employment, work and family communication of the people are influenced to a great extent.
Aiming at the problems, the invention provides a sign language digital human driving and real-time rendering system based on WebGL, which can provide online, real-time and accurate Chinese sign language translation service for hearing-impaired people at a web end.
Disclosure of Invention
Technical deficiencies to be solved
Aiming at the defects of the prior art and the application, the invention provides a sign language digital man driving and real-time rendering system based on WebGL, which can perform high-definition sign language digital man rendering at a Web end, realize on-line and real-time Chinese sign language translation through a network request according to characters required to be translated by a user, provide a new mode for hearing-impaired people to acquire information at the Web end, and fill the technical blank of providing on-line Chinese sign language translation for the hearing-impaired people at the Web end.
(II) technical scheme
The invention provides a sign language digital human driving and real-time rendering system based on WebGL, which is used for realizing on-line real-time sign language translation service displayed through three-dimensional animation of sign language digital human at a web end under various system platforms. In order to realize a sign language digital human driving and real-time rendering system based on WebGL, the invention provides the following technical scheme, which mainly comprises the following parts:
(1) web front end
The Web front-end module is used for system embedding, system UI interface display, user interaction, network request and three-dimensional scene display.
(2) Rendering module
The rendering module renders a virtual sign language digital person on a web end by using a programmable rendering pipeline based on WebGL, and renders a more vivid virtual sign language animation picture under the action of combining various illumination models and multiple light sources.
(3) Cloud service module
The cloud service module can respond to the high-concurrency request of the web front end, process each request in real time, generate corresponding animation data according to the translation text of the request and return the animation data to the web front end for processing.
(4) Animation driving module
After receiving animation data of cloud service, the Web front end preprocesses the animation data to obtain correct animation driving data, and then uses the animation driving data to carry out animation driving on the sign language digital human by combining with a calculation animation algorithm.
Preferably, the rendering module includes: third party libraries that support WebGL, such as ThreeJS, BabylonJS, etc., can be used.
Preferably, the rendering module includes: sign language digital human model loading and texture loading, compiling and running of a programmable rendering pipeline, using various illumination models in a three-dimensional scene, using various types and a plurality of light sources in the three-dimensional scene, and using different shadow effects in the three-dimensional scene.
Preferably, the cloud service module includes: responses to high concurrent network requests; load balancing is carried out on the high concurrent network requests, and the requests are dispatched according to the sub-service load state;
preferably, the animation data includes: animation data metadata, encryption information of animation data, confusion information of animation data, and animation driving data;
preferably, the animation module includes: loading and calculating the weight of the skeleton skin of the virtual character model, driving the virtual sign language digital human joints to move through animation data, driving the character skin to move through joint movement information, and outputting animation interpolation smoothing and animation.
(III) advantageous effects
Compared with the prior art, the invention provides a sign language digital human driving and real-time rendering system based on WebGL, which has the following beneficial effects:
(1) lower hardware operation requirement and excellent cross-platform performance
The system has lower requirements on the performance of equipment hardware during operation and has good cross-platform performance. The method can be run on any browser supporting three-dimensional rendering only by installing a browser supporting WebGL on a terminal by a user, can simultaneously support the use of a PC (personal computer) end, a mobile end and a web end, has good cross-platform performance, and supports mainstream system platforms such as Windows, Linux, MacOS, Android and ios.
(2) Non-invasive design scheme applicable without modifying web page structure
The system adopts a non-invasive application design scheme, does not need to modify the website structure of an application website, can complete the embedding of the whole web front-end page on the application website only by introducing a single js file of the system, and provides complete user interactive operation.
(3) Providing high-fidelity digital human rendering effect on web end
The system provides a high-fidelity digital human rendering effect at a web end, and provides two different rendering modes of low definition and high definition in order to adapt to the hardware performance of different terminals. The low-definition mode is suitable for terminals with lower performance of hardware equipment, the high-definition mode is suitable for terminals with higher performance of hardware equipment,
(4) accurate digital human-computing animation driving mechanism
The system can drive the sign language digital person to make accurate and smooth sign language action by depending on the existing digital person calculation animation driving algorithm. Generating confusion and encryption animation data through a server background, then transmitting the animation data to a web end, carrying out operations such as data decryption, data cleaning, data analysis and recombination after the web end receives the data, and finally driving a sign language digital person to complete sign language actions by using the animation data in a three-dimensional scene.
(5) Real-time, accurate Chinese sign language translation
The system can provide real-time and accurate Chinese sign language translation for the user at the web end. The user can obtain real-time and accurate Chinese sign language translation only by sliding the mouse to select the text needing sign language translation at the web end, and the convenience of information acquisition of the people with hearing disabilities is greatly improved.
Drawings
Fig. 1 is a technical flowchart of a sign language digital human driver and real-time rendering system based on WebGL of the present invention.
Detailed Description
Hereinafter, an embodiment of a method of the WebGL-based sign language digital human-driven and real-time rendering system of the present invention will be described with reference to fig. 1.
The embodiments described herein are specific embodiments of the present invention, and are intended to be illustrative and exemplary of the concepts of the present invention and should not be construed as limiting the scope of the embodiments of the present invention. In addition to the embodiments described herein, other solutions that are obvious to one skilled in the art can be used based on the disclosure of the claims and the specification of the present application, and these solutions also include any obvious substitutions and modifications to the embodiments described herein.
The drawings in the present specification are schematic diagrams, which are included to assist in explaining the concepts of the present invention and schematically show the interrelationship between the various parts.
Referring to fig. 1, the method for a sign language digital human driver and real-time rendering system based on WebGL of the present invention is used for providing an online real-time chinese sign language translation service for hearing-impaired people on a web end, and comprises the following steps:
(1) step 1: the web front-end page of the sign language digital man-driving and real-time rendering system based on the WebGL adopts a low-coupling and non-intrusive technical scheme, and a front-end UI interface of the system can be displayed in a webpage and the system UI can be interactively operated as long as a single js file of the system is required to be introduced into the webpage needing to use the sign language digital man-driving and real-time rendering system based on the WebGL. A three-dimensional scene rendered by a sign language digital person based on WebGL is embedded in an interface of the web front end of the system, when the js file is introduced, the three-dimensional scene can be automatically created, and a sign language digital person model is loaded, so that the rendering of the scene is completed at the same time.
(2) And 2, step: after the web front-end interface is successfully established in the application webpage, a user can slide and select characters needing sign language translation through a mouse, and then the system sends the texts selected by the user to the remote scheduling server through a network request.
(3) And step 3: aiming at processing a single request, the remote dispatching server distributes the request to an idle sub-service for processing after receiving the transmitted network request, the sub-service generates 30 frames of animation data according to the requested translation text at the speed of 30 frames per second and returns the animation data to the remote dispatching server, and the remote dispatching server transmits the result animation data to a corresponding web end according to a request ID after receiving the result animation data. The sub-service can perform data compaction, data confusion and data encryption on the animation data in the process of generating the frame animation data, and ensures that the animation data is not easy to crack and occupies smaller network bandwidth in the network transmission process.
(4) And 4, step 4: after receiving the animation data byte stream returned by the remote scheduling service, the web side firstly decrypts the received animation data, then cleans the data confusion part, and finally analyzes and recombines the cleaned data to obtain the available animation data.
(5) And 5: and transmitting the analyzed animation data to an animation driving module, converting the animation data into joint driving data by the animation driving module, and interpolating motion information between frames to drive the sign language digital human to move. When more animation data byte streams are received from the server, the motion information can be interpolated between the animation data streams, thereby forming a continuous and smooth sign language translation action.
(6) Step 6: in the process of driving the sign language digital human to move by using the animation data, the three-dimensional rendering scene displays a vivid sign language digital human action picture, and sign language translation information is transmitted to the users of the hearing-impaired people.
Claims (7)
1. A sign language digital human-driven and real-time rendering system based on WebGL is used for providing online, real-time and accurate Chinese sign language translation for hearing-impaired people on a web end, and comprises:
designing and interactively operating a web front end UI of a sign language digital human-driven and real-time rendering system based on WebGL;
the web front end of the system can adopt a non-invasive design scheme without modifying a webpage source code, and the non-invasive design scheme comprises web front end UI interface embedding and user interactive operation response embedding;
the WebGL-based rendering system comprises a programmable rendering pipeline design, an illumination model used for rendering, illumination used for rendering, an illumination design, a three-dimensional rendering scene design and the like;
the animation system based on the WebGL comprises a skeleton skinning animation, an animation interpolation method, an animation driving mechanism and the like;
a high-concurrency network request response, request distribution and load balancing mechanism of the cloud service;
and (3) a real-time animation data generation algorithm of the cloud service.
2. The method of claim 1, wherein the web front end UI design of the WebGL based sign language digital human driven and real-time rendering system comprises: UI layout design, UI picture design, product design and the like of the web front end; the interactive design comprises two modes of adjustable bright colors and dark colors, the sign language speed of a sign language digital person can be adjusted, the sign language action of the sign language digital person in the previous sentence can be replayed, the low-definition mode and the high-definition mode of three-dimensional scene rendering can be adjusted, and words or sentences to be translated can be selected through sliding.
3. The method of claim 1, wherein the web-front-end non-intrusive scheme comprises: the web front-end page can be embedded in the application website or the web page only by referencing a single js file of the system without modifying the existing web page structure code of the application website or the web page. In addition to the embedded web front end interface, responses of various interaction operations for interacting with the web front end of the system are included, such as a response of a button click event, a response of a web interface draggable event on a web page, a response of a displayed and hidden web interface event, a response of a web interface color modification event, a response of a user sliding a mouse to select a word and sentence event, and the like.
4. The method of claim 1, wherein the WebGL-based rendering system comprises: loading a sign language digital human model file and a texture file, designing a programmable rendering pipeline for rendering the sign language digital human model, using a plurality of illumination models in the rendering pipeline, and using the positions, the intensities, the azimuth angles and the like of a plurality of light sources in a three-dimensional rendering scene.
5. The method of claim 1, wherein the WebGL-based animation system comprises: the method comprises the steps of manufacturing and optimizing a bone skin weighted value of a sign language digital human model, driving a sign language digital human to make continuous smooth action through joint animation data, and performing smooth interpolation among a plurality of frame animations.
6. The method of claim 1, wherein the high concurrency network request response, request distribution and load balancing mechanism of the cloud service comprises: the remote cloud server carries out instant response on the high-concurrency network request, generates a sub-service load state distribution data calculation task according to the animation data, and carries out data calculation and animation data generation in real time by the sub-service according to the distributed calculation task.
7. The method of claim 1, wherein the cloud service real-time animation data generation algorithm comprises: and generating animation data packets of continuous frames at the speed of 30 frames per second by using the existing algorithm, and carrying out data reduction, data encryption and data confusion on the data packets to finally generate the encrypted and confused animation data byte stream.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210313014.3A CN114708367A (en) | 2022-03-28 | 2022-03-28 | WebGL-based sign language digital human driving and real-time rendering system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210313014.3A CN114708367A (en) | 2022-03-28 | 2022-03-28 | WebGL-based sign language digital human driving and real-time rendering system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114708367A true CN114708367A (en) | 2022-07-05 |
Family
ID=82170640
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210313014.3A Pending CN114708367A (en) | 2022-03-28 | 2022-03-28 | WebGL-based sign language digital human driving and real-time rendering system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114708367A (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103281345A (en) * | 2013-02-16 | 2013-09-04 | 赞奇科技发展有限公司 | Three-dimensional digital model real-time rendering realizing method and cloud service system based on cloud computing architecture |
| CN103956167A (en) * | 2014-05-06 | 2014-07-30 | 北京邮电大学 | Visual sign language interpretation method and device based on Web |
| KR20150102148A (en) * | 2014-02-27 | 2015-09-07 | 김조웅 | Sign language interpreting system of based on web contents and method for sign language interpreting system |
| CN111798553A (en) * | 2020-07-13 | 2020-10-20 | 重庆大学 | WebGL-based three-dimensional model rendering method, system and related device |
| CN111882628A (en) * | 2020-08-05 | 2020-11-03 | 北京智湃科技有限公司 | Method for rendering real-time behaviors of 3D digital virtual human based on WebGL |
| CN112750184A (en) * | 2019-10-30 | 2021-05-04 | 阿里巴巴集团控股有限公司 | Data processing, action driving and man-machine interaction method and equipment |
| CN112991508A (en) * | 2021-03-30 | 2021-06-18 | 赛瓦软件(上海)有限公司 | WebGL-based 3D rendering system and method |
| CN113407034A (en) * | 2021-07-09 | 2021-09-17 | 呜啦啦(广州)科技有限公司 | Sign language inter-translation method and system |
| CN113438300A (en) * | 2021-06-22 | 2021-09-24 | 江苏科技大学 | Network-based accessible communication online communication system and method for hearing-impaired people and normal people |
| CN113780013A (en) * | 2021-07-30 | 2021-12-10 | 阿里巴巴（中国）有限公司 | Translation method, translation equipment and readable medium |
-
2022
- 2022-03-28 CN CN202210313014.3A patent/CN114708367A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103281345A (en) * | 2013-02-16 | 2013-09-04 | 赞奇科技发展有限公司 | Three-dimensional digital model real-time rendering realizing method and cloud service system based on cloud computing architecture |
| KR20150102148A (en) * | 2014-02-27 | 2015-09-07 | 김조웅 | Sign language interpreting system of based on web contents and method for sign language interpreting system |
| CN103956167A (en) * | 2014-05-06 | 2014-07-30 | 北京邮电大学 | Visual sign language interpretation method and device based on Web |
| CN112750184A (en) * | 2019-10-30 | 2021-05-04 | 阿里巴巴集团控股有限公司 | Data processing, action driving and man-machine interaction method and equipment |
| CN111798553A (en) * | 2020-07-13 | 2020-10-20 | 重庆大学 | WebGL-based three-dimensional model rendering method, system and related device |
| CN111882628A (en) * | 2020-08-05 | 2020-11-03 | 北京智湃科技有限公司 | Method for rendering real-time behaviors of 3D digital virtual human based on WebGL |
| CN112991508A (en) * | 2021-03-30 | 2021-06-18 | 赛瓦软件(上海)有限公司 | WebGL-based 3D rendering system and method |
| CN113438300A (en) * | 2021-06-22 | 2021-09-24 | 江苏科技大学 | Network-based accessible communication online communication system and method for hearing-impaired people and normal people |
| CN113407034A (en) * | 2021-07-09 | 2021-09-17 | 呜啦啦(广州)科技有限公司 | Sign language inter-translation method and system |
| CN113780013A (en) * | 2021-07-30 | 2021-12-10 | 阿里巴巴（中国）有限公司 | Translation method, translation equipment and readable medium |
Non-Patent Citations (1)
| Title |
|---|
| 高春鸣等: "一种新的运动捕获数据转换方法", 计算机工程, vol. 38, no. 2, 31 January 2012 (2012-01-31) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Elliott et al. | Linguistic modelling and language-processing technologies for Avatar-based sign language presentation | |
| Lu et al. | Virtual learning environment for medical education based on VRML and VTK | |
| Pratama et al. | Workflow in virtual reality tool development for AEC industry | |
| CN109767368A (en) | A kind of Virtual Chemical Experiment's teaching platform based on WebGL technology | |
| CN110691010B (en) | Cross-platform and cross-terminal VR/AR product information display system | |
| KR20230098068A (en) | Moving picture processing method, apparatus, electronic device and computer storage medium | |
| Grimstead et al. | RAVE: the resource‐aware visualization environment | |
| CN114708367A (en) | WebGL-based sign language digital human driving and real-time rendering system | |
| Adamo-Villani et al. | Toward the ideal signing avatar | |
| Elliott et al. | A framework for non-manual gestures in a synthetic signing system | |
| Lu et al. | A Cross‐Platform Web3D Monitoring System of the Three‐Machine Equipment in a Fully Mechanized Coalface Based on the Skeleton Model and Sensor Data | |
| Yin et al. | Remote three-dimensional reconstruction for medical images based on Browser/Server framework | |
| Ji et al. | Exploring traditional handicraft learning mode using WebAR technology | |
| Di Giacomo et al. | Adaptation of virtual human animation and representation for MPEG | |
| Sang | [Retracted] Interactive Innovation Research on Film Animation Based on Maya‐Unity Animation Simulation of Visual Sensor | |
| Leone et al. | LUCIA-webGL: a web based Italian MPEG-4 talking head | |
| CN113724399B (en) | Teaching knowledge point display method and system based on virtual reality | |
| Probierz et al. | Sign language interpreting-relationships between research in different areas-overview | |
| CN116309970A (en) | Method and device for generating virtual digital image for vehicle, electronic equipment and storage medium | |
| Naritomi et al. | Ketchup As You Like: Drawing Editor for Foods | |
| Jensen | Real-time synchronised 3D-perceptualisation | |
| Misjuns et al. | Creation of virtual reality experiences for the web: shader programming for artists | |
| Wang et al. | Augmented virtuality space: Enriching virtual design environments with reality | |
| Deléglise et al. | 2D/3D Web Transitions: Methods and Techniques. | |
| Grewe et al. | The Open Virtual Mirror Framework for enfacement illusions |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |