CN102737278A - 联合非线性随机投影、受限波尔兹曼机、以及基于批量的可并行优化来使用的深凸网络 - Google Patents
联合非线性随机投影、受限波尔兹曼机、以及基于批量的可并行优化来使用的深凸网络 Download PDFInfo
- Publication number
- CN102737278A CN102737278A CN2012100894880A CN201210089488A CN102737278A CN 102737278 A CN102737278 A CN 102737278A CN 2012100894880 A CN2012100894880 A CN 2012100894880A CN 201210089488 A CN201210089488 A CN 201210089488A CN 102737278 A CN102737278 A CN 102737278A
- Authority
- CN
- China
- Prior art keywords
- module
- unit
- weight matrix
- layer
- linear
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000005457 optimization Methods 0.000 title claims abstract description 18
- 238000012549 training Methods 0.000 claims abstract description 43
- 238000000034 method Methods 0.000 claims abstract description 33
- 239000010410 layer Substances 0.000 claims description 97
- 239000011159 matrix material Substances 0.000 claims description 45
- 230000009471 action Effects 0.000 claims description 7
- 238000013528 artificial neural network Methods 0.000 claims description 6
- 230000000712 assembly Effects 0.000 claims description 5
- 238000000429 assembly Methods 0.000 claims description 5
- 230000000930 thermomechanical effect Effects 0.000 claims description 3
- 239000002356 single layer Substances 0.000 claims description 2
- 230000007704 transition Effects 0.000 abstract 2
- 238000009877 rendering Methods 0.000 abstract 1
- 230000000717 retained effect Effects 0.000 abstract 1
- 230000013016 learning Effects 0.000 description 10
- 238000013500 data storage Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 5
- 238000011160 research Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 241001269238 Data Species 0.000 description 2
- 230000035045 associative learning Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000005236 sound signal Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/08—Speech classification or search
- G10L15/16—Speech classification or search using artificial neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
Description
Claims (10)
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/077,978 US8489529B2 (en) | 2011-03-31 | 2011-03-31 | Deep convex network with joint use of nonlinear random projection, Restricted Boltzmann Machine and batch-based parallelizable optimization |
US13/077,978 | 2011-03-31 |
Publications (2)
Publication Number | Publication Date |
---|---|
CN102737278A true CN102737278A (zh) | 2012-10-17 |
CN102737278B CN102737278B (zh) | 2019-02-12 |
Family
ID=46928584
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201210089488.0A Active CN102737278B (zh) | 2011-03-31 | 2012-03-30 | 联合非线性随机投影、受限波尔兹曼机、以及基于批量的可并行优化来使用的深凸网络 |
Country Status (2)
Country | Link |
---|---|
US (2) | US8489529B2 (zh) |
CN (1) | CN102737278B (zh) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103345656A (zh) * | 2013-07-17 | 2013-10-09 | 中国科学院自动化研究所 | 一种基于多任务深度神经网络的数据识别方法及装置 |
CN104143327A (zh) * | 2013-07-10 | 2014-11-12 | 腾讯科技(深圳)有限公司 | 一种声学模型训练方法和装置 |
US9508347B2 (en) | 2013-07-10 | 2016-11-29 | Tencent Technology (Shenzhen) Company Limited | Method and device for parallel processing in model training |
CN108763360A (zh) * | 2018-05-16 | 2018-11-06 | 北京旋极信息技术股份有限公司 | 一种分类方法及装置、计算机可读存储介质 |
TWI643138B (zh) * | 2016-10-03 | 2018-12-01 | 三菱電機股份有限公司 | Network construction device and network construction method |
Families Citing this family (41)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2012047857A2 (en) * | 2010-10-04 | 2012-04-12 | Mind Over Matter Ai, Llc. | Coupling of rational agents to quantum processes |
US8489529B2 (en) | 2011-03-31 | 2013-07-16 | Microsoft Corporation | Deep convex network with joint use of nonlinear random projection, Restricted Boltzmann Machine and batch-based parallelizable optimization |
US9235799B2 (en) | 2011-11-26 | 2016-01-12 | Microsoft Technology Licensing, Llc | Discriminative pretraining of deep neural networks |
CN103366782B (zh) * | 2012-04-06 | 2014-09-10 | 腾讯科技(深圳)有限公司 | 在虚拟形象上自动播放表情的方法和装置 |
US9141916B1 (en) | 2012-06-29 | 2015-09-22 | Google Inc. | Using embedding functions with a deep network |
US9418334B2 (en) * | 2012-12-06 | 2016-08-16 | Nuance Communications, Inc. | Hybrid pre-training of deep belief networks |
US9842585B2 (en) | 2013-03-11 | 2017-12-12 | Microsoft Technology Licensing, Llc | Multilingual deep neural network |
US9099083B2 (en) | 2013-03-13 | 2015-08-04 | Microsoft Technology Licensing, Llc | Kernel deep convex networks and end-to-end learning |
US11842270B1 (en) | 2013-05-28 | 2023-12-12 | Deepmind Technologies Limited | Learning abstractions using patterns of activations of a neural network hidden layer |
US9728184B2 (en) | 2013-06-18 | 2017-08-08 | Microsoft Technology Licensing, Llc | Restructuring deep neural network acoustic models |
US9589565B2 (en) | 2013-06-21 | 2017-03-07 | Microsoft Technology Licensing, Llc | Environmentally aware dialog policies and response generation |
US9311298B2 (en) | 2013-06-21 | 2016-04-12 | Microsoft Technology Licensing, Llc | Building conversational understanding systems using a toolset |
US10152676B1 (en) * | 2013-11-22 | 2018-12-11 | Amazon Technologies, Inc. | Distributed training of models using stochastic gradient descent |
US9324321B2 (en) | 2014-03-07 | 2016-04-26 | Microsoft Technology Licensing, Llc | Low-footprint adaptation and personalization for a deep neural network |
US9529794B2 (en) | 2014-03-27 | 2016-12-27 | Microsoft Technology Licensing, Llc | Flexible schema for language model customization |
US10579922B2 (en) | 2014-04-08 | 2020-03-03 | Microsoft Technology Licensing, Llc | Deep learning using alternating direction method of multipliers |
US9614724B2 (en) | 2014-04-21 | 2017-04-04 | Microsoft Technology Licensing, Llc | Session-based device configuration |
US9520127B2 (en) | 2014-04-29 | 2016-12-13 | Microsoft Technology Licensing, Llc | Shared hidden layer combination for speech recognition systems |
US9384334B2 (en) | 2014-05-12 | 2016-07-05 | Microsoft Technology Licensing, Llc | Content discovery in managed wireless distribution networks |
US10111099B2 (en) | 2014-05-12 | 2018-10-23 | Microsoft Technology Licensing, Llc | Distributing content in managed wireless distribution networks |
US9384335B2 (en) | 2014-05-12 | 2016-07-05 | Microsoft Technology Licensing, Llc | Content delivery prioritization in managed wireless distribution networks |
US9430667B2 (en) | 2014-05-12 | 2016-08-30 | Microsoft Technology Licensing, Llc | Managed wireless distribution network |
US9874914B2 (en) | 2014-05-19 | 2018-01-23 | Microsoft Technology Licensing, Llc | Power management contracts for accessory devices |
US10037202B2 (en) | 2014-06-03 | 2018-07-31 | Microsoft Technology Licensing, Llc | Techniques to isolating a portion of an online computing service |
US9367490B2 (en) | 2014-06-13 | 2016-06-14 | Microsoft Technology Licensing, Llc | Reversible connector for accessory devices |
US10432711B1 (en) * | 2014-09-15 | 2019-10-01 | Amazon Technologies, Inc. | Adaptive endpoint selection |
WO2016134183A1 (en) | 2015-02-19 | 2016-08-25 | Digital Reasoning Systems, Inc. | Systems and methods for neural language modeling |
CN105184303B (zh) * | 2015-04-23 | 2019-08-09 | 南京邮电大学 | 一种基于多模态深度学习的图像标注方法 |
US9734436B2 (en) * | 2015-06-05 | 2017-08-15 | At&T Intellectual Property I, L.P. | Hash codes for images |
EP3374932B1 (en) | 2016-02-03 | 2022-03-16 | Google LLC | Compressed recurrent neural network models |
US9858340B1 (en) | 2016-04-11 | 2018-01-02 | Digital Reasoning Systems, Inc. | Systems and methods for queryable graph representations of videos |
JP6714690B2 (ja) * | 2016-04-26 | 2020-06-24 | 株式会社日立製作所 | 情報処理システム、情報処理システムの運用方法、および機械学習演算器 |
US10338931B2 (en) | 2016-04-29 | 2019-07-02 | International Business Machines Corporation | Approximate synchronization for parallel deep learning |
US10417566B2 (en) * | 2016-05-22 | 2019-09-17 | Microsoft Technology Licensing, Llc | Self-learning technique for training a PDA component and a simulated user component |
JPWO2018011842A1 (ja) * | 2016-07-11 | 2019-04-25 | 株式会社Uei | 階層ネットワークを用いた演算処理システム |
US10685285B2 (en) * | 2016-11-23 | 2020-06-16 | Microsoft Technology Licensing, Llc | Mirror deep neural networks that regularize to linear networks |
US10373049B2 (en) * | 2016-12-20 | 2019-08-06 | Google Llc | Generating an output for a neural network output layer |
US20180197080A1 (en) * | 2017-01-11 | 2018-07-12 | International Business Machines Corporation | Learning apparatus and method for bidirectional learning of predictive model based on data sequence |
US10657426B2 (en) * | 2018-01-25 | 2020-05-19 | Samsung Electronics Co., Ltd. | Accelerating long short-term memory networks via selective pruning |
CN108416434B (zh) * | 2018-02-07 | 2021-06-04 | 复旦大学 | 针对神经网络的卷积层与全连接层进行加速的电路结构 |
CN109784356B (zh) * | 2018-07-18 | 2021-01-05 | 北京工业大学 | 基于Fisher判别分析的矩阵变量受限玻尔兹曼机图像分类方法 |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO1990010270A1 (en) * | 1989-02-28 | 1990-09-07 | E.I. Du Pont De Nemours And Company | Apparatus and method for controlling a process using a trained parallel distributed processing network |
US20050089215A1 (en) * | 2003-10-25 | 2005-04-28 | Carl Staelin | Image artifact reduction using a neural network |
CN1627251A (zh) * | 2003-12-09 | 2005-06-15 | 微软公司 | 使用图形处理单元加速并优化机器学习技术的处理 |
CN1643920A (zh) * | 2002-03-19 | 2005-07-20 | 皇家飞利浦电子股份有限公司 | 使用多个推荐得分的推荐*** |
WO2006042142A2 (en) * | 2004-10-07 | 2006-04-20 | Bernard Widrow | Cognitive memory and auto-associative neural network based pattern recognition and searching |
CN101071412A (zh) * | 2006-05-10 | 2007-11-14 | 何千军 | 一种基于自定义模型的神经网络分析***及方法 |
CN101266145A (zh) * | 2008-04-14 | 2008-09-17 | 西安理工大学 | 用多层感知机神经网络实现测量的气压高度计及测量方法 |
CN101939833A (zh) * | 2007-08-31 | 2011-01-05 | 恪纳腾公司 | 用于跨晶片区域预测半导体参数的装置和方法 |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5226092A (en) | 1991-06-28 | 1993-07-06 | Digital Equipment Corporation | Method and apparatus for learning in a neural network |
US5479579A (en) | 1992-09-04 | 1995-12-26 | The United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration | Cascaded VLSI neural network architecture for on-line learning |
US5572028A (en) | 1994-10-20 | 1996-11-05 | Saint-Gobain/Norton Industrial Ceramics Corporation | Multi-element dosimetry system using neural network |
DE60130742T2 (de) | 2001-05-28 | 2008-07-17 | Honda Research Institute Europe Gmbh | Mustererkennung mit hierarchischen Netzen |
US7233931B2 (en) | 2003-12-26 | 2007-06-19 | Lee Shih-Jong J | Feature regulation for hierarchical decision learning |
US8234228B2 (en) | 2008-02-07 | 2012-07-31 | Nec Laboratories America, Inc. | Method for training a learning machine having a deep multi-layered network with labeled and unlabeled training data |
US8457409B2 (en) | 2008-05-22 | 2013-06-04 | James Ting-Ho Lo | Cortex-like learning machine for temporal and hierarchical pattern recognition |
US8489529B2 (en) | 2011-03-31 | 2013-07-16 | Microsoft Corporation | Deep convex network with joint use of nonlinear random projection, Restricted Boltzmann Machine and batch-based parallelizable optimization |
-
2011
- 2011-03-31 US US13/077,978 patent/US8489529B2/en active Active
-
2012
- 2012-03-30 CN CN201210089488.0A patent/CN102737278B/zh active Active
-
2013
- 2013-06-17 US US13/919,106 patent/US9390371B2/en active Active
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO1990010270A1 (en) * | 1989-02-28 | 1990-09-07 | E.I. Du Pont De Nemours And Company | Apparatus and method for controlling a process using a trained parallel distributed processing network |
CN1643920A (zh) * | 2002-03-19 | 2005-07-20 | 皇家飞利浦电子股份有限公司 | 使用多个推荐得分的推荐*** |
US20050089215A1 (en) * | 2003-10-25 | 2005-04-28 | Carl Staelin | Image artifact reduction using a neural network |
CN1627251A (zh) * | 2003-12-09 | 2005-06-15 | 微软公司 | 使用图形处理单元加速并优化机器学习技术的处理 |
WO2006042142A2 (en) * | 2004-10-07 | 2006-04-20 | Bernard Widrow | Cognitive memory and auto-associative neural network based pattern recognition and searching |
CN101071412A (zh) * | 2006-05-10 | 2007-11-14 | 何千军 | 一种基于自定义模型的神经网络分析***及方法 |
CN101939833A (zh) * | 2007-08-31 | 2011-01-05 | 恪纳腾公司 | 用于跨晶片区域预测半导体参数的装置和方法 |
CN101266145A (zh) * | 2008-04-14 | 2008-09-17 | 西安理工大学 | 用多层感知机神经网络实现测量的气压高度计及测量方法 |
Cited By (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104143327A (zh) * | 2013-07-10 | 2014-11-12 | 腾讯科技(深圳)有限公司 | 一种声学模型训练方法和装置 |
WO2015003436A1 (en) * | 2013-07-10 | 2015-01-15 | Tencent Technology (Shenzhen) Company Limited | Method and device for parallel processing in model training |
CN104143327B (zh) * | 2013-07-10 | 2015-12-09 | 腾讯科技(深圳)有限公司 | 一种声学模型训练方法和装置 |
US9508347B2 (en) | 2013-07-10 | 2016-11-29 | Tencent Technology (Shenzhen) Company Limited | Method and device for parallel processing in model training |
CN103345656A (zh) * | 2013-07-17 | 2013-10-09 | 中国科学院自动化研究所 | 一种基于多任务深度神经网络的数据识别方法及装置 |
CN103345656B (zh) * | 2013-07-17 | 2016-01-20 | 中国科学院自动化研究所 | 一种基于多任务深度神经网络的数据识别方法及装置 |
TWI643138B (zh) * | 2016-10-03 | 2018-12-01 | 三菱電機股份有限公司 | Network construction device and network construction method |
US11373110B2 (en) | 2016-10-03 | 2022-06-28 | Mitsubishi Electric Corporation | Apparatus and network construction method for determining the number of elements in an intermediate layer of a neural network |
CN108763360A (zh) * | 2018-05-16 | 2018-11-06 | 北京旋极信息技术股份有限公司 | 一种分类方法及装置、计算机可读存储介质 |
Also Published As
Publication number | Publication date |
---|---|
US20120254086A1 (en) | 2012-10-04 |
US20130282634A1 (en) | 2013-10-24 |
CN102737278B (zh) | 2019-02-12 |
US8489529B2 (en) | 2013-07-16 |
US9390371B2 (en) | 2016-07-12 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN102737278A (zh) | 联合非线性随机投影、受限波尔兹曼机、以及基于批量的可并行优化来使用的深凸网络 | |
KR102071582B1 (ko) | 딥 뉴럴 네트워크(Deep Neural Network)를 이용하여 문장이 속하는 클래스(class)를 분류하는 방법 및 장치 | |
Su et al. | Convolutional tensor-train LSTM for spatio-temporal learning | |
Chien et al. | Tensor-factorized neural networks | |
US9292787B2 (en) | Computer-implemented deep tensor neural network | |
Sharma et al. | Era of deep neural networks: A review | |
CN109446430B (zh) | 产品推荐的方法、装置、计算机设备及可读存储介质 | |
Deng et al. | Deep convex net: A scalable architecture for speech pattern classification | |
CN102436811B (zh) | 用于语音识别的深度结构的全序列训练 | |
Le | A tutorial on deep learning part 1: Nonlinear classifiers and the backpropagation algorithm | |
EP3685316A1 (en) | Capsule neural networks | |
Pandey et al. | Attention gated tensor neural network architectures for speech emotion recognition | |
WO2018220700A1 (ja) | 新規学習データセット生成方法、新規学習データセット生成装置および生成された学習データセットを用いた学習方法 | |
Nasien et al. | Statistical learning theory and support vector machines | |
Gouda et al. | Speech recognition: keyword spotting through image recognition | |
Wang et al. | A new delay connection for long short-term memory networks | |
Rajagede et al. | Recognizing Arabic letter utterance using convolutional neural network | |
CN117349402A (zh) | 一种基于机器阅读理解的情绪原因对识别方法及*** | |
Tripathi et al. | When sub-band features meet attention mechanism while knowledge distillation for sound classification | |
Bao et al. | Tensor classification network | |
Aryal et al. | Using pre-trained models as feature extractor to classify video styles used in MOOC videos | |
De Bortoli et al. | A fast face recognition CNN obtained by distillation | |
Hallyal et al. | Optimized recognition of CAPTCHA through attention models | |
Cuevas-Tello et al. | A tutorial on deep neural networks for intelligent systems | |
Singh | Speaker emotion Recognition System using Artificial neural network classification method for brain-inspired application |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 1175279 Country of ref document: HK |
|
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
ASS | Succession or assignment of patent right |
Owner name: MICROSOFT TECHNOLOGY LICENSING LLC Free format text: FORMER OWNER: MICROSOFT CORP. Effective date: 20150728 |
|
C41 | Transfer of patent application or patent right or utility model | ||
TA01 | Transfer of patent application right |
Effective date of registration: 20150728 Address after: Washington State Applicant after: Micro soft technique license Co., Ltd Address before: Washington State Applicant before: Microsoft Corp. |
|
GR01 | Patent grant | ||
GR01 | Patent grant |