Large-Scale Deep Learning for Intelligent Computer Systems Jeff Dean Google Brain team in collaboration with many other teams

Growing Use of Deep Learning at Google # of directories containing model description files

Across many products/areas: Android Apps GMail Image Understanding Maps NLP Photos Robotics Speech Translation many research uses.. YouTube … many others ...

Outline Two generations of deep learning software systems: ● 1st generation: DistBelief [Dean et al., NIPS 2012] ● 2nd generation: TensorFlow (unpublished) An overview of how we use these in research and products Plus, ...a new approach for training (people, not models)

Google Brain project started in 2011, with a focus on pushing state-of-the-art in neural networks. Initial emphasis: ● use large datasets, and ● large amounts of computation to push boundaries of what is possible in perception and language understanding

Plenty of raw data ● ● ● ● ● ●

Text: trillions of words of English + other languages Visual data: billions of images and videos Audio: tens of thousands of hours of speech per day User activity: queries, marking messages spam, etc. Knowledge graph: billions of labelled relation triples ...

How can we build systems that truly understand this data?

Text Understanding “This movie should have NEVER been made. From the poorly done animation, to the beyond bad acting. I am not sure at what point the people behind this movie said "Ok, looks good! Lets do it!" I was in awe of how truly horrid this movie was.”

Turnaround Time and Effect on Research ● Minutes, Hours: ○

Interactive research! Instant gratification!

● 1-4 days ○ ○

Tolerable Interactivity replaced by running many experiments in parallel

● 1-4 weeks: ○ ○

High value experiments only Progress stalls

● >1 month ○

Don’t even try

Important Property of Neural Networks

Results get better with more data + bigger models + more computation (Better algorithms, new insights and improved techniques always help, too!)

How Can We Train Large, Powerful Models Quickly? ● Exploit many kinds of parallelism ○ Model parallelism ○ Data parallelism

Model Parallelism

Model Parallelism

Model Parallelism

Data Parallelism Parameter Servers

Model Replicas

...

Data

...

Data Parallelism Parameter Servers

p Model Replicas

...

Data

...

Data Parallelism Parameter Servers

∆p

p

Model Replicas

...

Data

...

Data Parallelism Parameter Servers

∆p

p’ = p + ∆p

p

Model Replicas

...

Data

...

Data Parallelism Parameter Servers

p’ = p + ∆p

p’ Model Replicas

...

Data

...

Data Parallelism Parameter Servers

∆p’

p’

Model Replicas

...

Data

...

Data Parallelism Parameter Servers

∆p’

p’’ = p’ + ∆p

p’

Model Replicas

...

Data

...

Data Parallelism Parameter Servers

∆p’

p’’ = p’ + ∆p

p’

Model Replicas

...

Data

...

Data Parallelism Choices Can do this synchronously: ● ● ●

N replicas eqivalent to an N times larger batch size Pro: No noise Con: Less fault tolerant (requires recovery if any single machine fails)

Can do this asynchronously: ● ●

Con: Noise in gradients Pro: Relatively fault tolerant (failure in model replica doesn’t block other replicas)

(Or hybrid: M asynchronous groups of N synchronous replicas)

Data Parallelism Considerations Want model computation time to be large relative to time to send/receive parameters over network Models with fewer parameters, that reuse each parameter multiple times in the computation ●

Mini-batches of size B reuse parameters B times

Certain model structures reuse parameter many times within each example: ● ●

Convolutional models tend to reuse hundreds or thousands of times per example (for different spatial positions) Recurrent models (LSTMs, RNNs) tend to reuse tens to hundreds of times (for unrolling through T time steps during training)

What are some ways that deep learning is having a significant impact at Google?

Sequence to Sequence Models Oriol Vinyals, Ilya Sutskever & Quoc Le started looking at how to map one sequence to another sequence: 0.1 0.5 1.0 0.0 2.4

The lazy brown fox jumped over the ...

Sequence to Sequence Model Deep LSTM

High dimensional representation of a sequence 0.1 0.5 1.0 0.0 2.4

The lazy brown fox jumped over the ...

Connect two, you get a machine translation system Le renard brun rapide saute par dessus ...

The quick brown fox jumped over the ...

It works well WMT’14

BLEU

State-of-the-art

37.0

Neural Translation Model

37.3

Sequence to Sequence Learning with Neural Networks Ilya Sutskever, Oriol Vinyals, Quoc V. Le (NIPS 2014) Addressing Rare Word Problems in Neural Translation Models (arxiv.org/abs/1410.8206) Thang Luong, Ilya Sutskever, Oriol Vinyals, Quoc V. Le, Wojciech Zaremba

… or a chatbot. I'm fine, thank you!

Hi Shiva, how are you?

It produces sensible conversations. User> hello , i have a problem with my machine BrainStop> hi BrainStop> this is shiva . BrainStop> how are you doing today ? User> hi shiva , how are you ? BrainStop> i ' m fine , thank you ! BrainStop> how may i assist you today ? User> i want to access using vpn BrainStop> currently the computer is connected to corp network ? User> no BrainStop> check this solution : BrainStop> https : / / INTERNAL_HOSTNAME_REDACTED / articles / connecting - to disconnecting - from - ssl - vpn # heading - 4 User> thanks , bye ! BrainStop> thank you .

… or a parser. n:(S.17 n:(S.17 n:(NP.11 p:NNP.53 n:) ...

Allen is locked in, regardless of his situ...

It works well Completely learned parser with no parsing-specific code State of the art results on WSJ 23 parsing task Grammar as a Foreign Language, Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton (to appear in NIPS 2015) http://arxiv.org/abs/1412.7449

… or something that can learn graph algorithms output: Convex Hull (or Delauney Triangulation) (or Travelling Salesman tour)

input: collection of points

Pointer Networks, Oriol Vinyals, Meire Fortunato, & Navdeep Jaitly (to appear in NIPS 2015)

Object Recognition Improvement Over Time Predicted Human Performance

“cat”

ImageNet Challenge Winners

Image Models

“cat”

Module with 6 separate = convolutional layers 24 layers deep

Going Deeper with Convolutions Szegedy et al. CVPR 2015

Good Fine-Grained Classification

Good Generalization

Both recognized as “meal”

Sensible Errors

Works in practice… for real users

Works in practice… for real users

Connect sequence and image models, you get a captioning system “A close up of a child holding a stuffed animal”

It works well (BLEU scores) Dataset

Previous SOTA

Show & Tell

Human

MS COCO

N/A

67

69

FLICKR

49

63

68

PASCAL (xfer learning)

25

59

68

SBU (weak label)

11

27

N/A

Show and Tell: A Neural Image Caption Generator, Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan (CVPR 2015)

TensorFlow: Second Generation Deep Learning System

Motivations DistBelief (1st system) was great for scalability Not as flexible as we wanted for research purposes Better understanding of problem space allowed us to make some dramatic simplifications

TensorFlow: Expressing High-Level ML Computations ● ●

Core in C++ ○ Very low overhead Different front ends for specifying/driving the computation ○ Python and C++ today, easy to add more

...

Python front end

C++ front end

Core TensorFlow Execution System CPU

GPU

Android

iOS

...

TensorFlow Example (Batch Logistic Regression) graph = tf.Graph() with graph.AsDefault(): examples = tf.constant(train_dataset) labels = tf.constant(train_labels) W = tf.Variable(tf.truncated_normal([image_size * image_size, num_labels])) b = tf.Variable(tf.zeros([num_labels]))

# Create new computation graph # Training data/labels

# Variables

logits = tf.mat_mul(examples, W) + b # Training computation loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits, labels)) optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss) prediction = tf.nn.softmax(logits)

# Optimizer to use # Predictions for training data

TensorFlow Example (Batch Logistic Regression) graph = tf.Graph() with graph.AsDefault(): examples = tf.constant(train_dataset) labels = tf.constant(train_labels) W = tf.Variable(tf.truncated_normal([image_size * image_size, num_labels])) b = tf.Variable(tf.zeros([num_labels]))

# Create new computation graph # Training data/labels

# Variables

logits = tf.mat_mul(examples, W) + b # Training computation loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits, labels)) optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss) prediction = tf.nn.softmax(logits) with tf.Session(graph=graph) as session: tf.InitializeAllVariables().Run() for step in xrange(num_steps): _, l, predictions = session.Run([optimizer, loss, prediction]) if (step % 100 == 0): print 'Loss at step', step, ':', l print 'Training accuracy: %.1f%%' % accuracy(predictions, labels)

# Optimizer to use # Predictions for training data

# Run & return 3 values

Computation is a dataflow graph

Graph of Nodes, also called Operations or ops.

biases

Add

weights MatMul examples

labels

Relu Xent

Computation is a dataflow graph

Edges are N-dimensional arrays: Tensors

biases

Add

weights MatMul examples

labels

with

s r o s ten

Relu Xent

Computation is a dataflow graph

'Biases' is a variable

e t a t ith s

w

Some ops compute gradients

−= updates biases

biases

...

learning rate

Add

...

Mul

−=

Computation is a dataflow graph

d

Device A

biases

...

d e t u b i r t is

Add

learning rate

Devices: Processes, Machines, GPUs, etc

...

Mul

Device B

−=

TensorFlow: Expressing High-Level ML Computations Automatically runs models on range of platforms:

from phones ...

to single machines (CPU and/or GPUs) …

to distributed systems of many 100s of GPU cards

What is in a name? ● Tensor: N-dimensional array ○ ○ ○

1-dimension: Vector 2-dimension: Matrix Represent many dimensional data flowing through the graph ■ e.g. Image represented as 3-d tensor rows, cols, color

● Flow: Computation based on data flow graphs ○

Lots of operations (nodes in the graph) applied to data flowing through

● Tensors flow through the graph → “TensorFlow” ○ ○

Edges represent the tensors (data) Nodes represent the processing

Flexible ● General computational infrastructure ○ Deep Learning support is a set of libraries on top of the core ○ Also useful for other machine learning algorithms ○ Possibly even for high performance computing (HPC) work ○ Abstracts away the underlying devices/computational hardware

Extensible ● Core system defines a number of standard operations and kernels (device-specific implementations of operations) ● Easy to define new operators and/or kernels

Deep Learning in TensorFlow ●

Typical neural net “layer” maps to one or more tensor operations ○



e.g. Hidden Layer: activations = Relu(weights * inputs + biases)

Library of operations specialized for Deep Learning ○

Dozens of high-level operations: 2D and 3D convolutions, Pooling, Softmax, ...



Standard losses e.g. CrossEntropy, L1, L2



Various optimizers e.g. Gradient Descent, AdaGrad, L-BFGS, ...



Auto Differentiation



Easy to experiment with (or combine!) a wide variety of different models: LSTMs, convolutional models, attention models, reinforcement learning, embedding models, Neural Turing Machine-like models, ...

No distinct Parameter Server subsystem ● Parameters are now just stateful nodes in the graph ● Data parallel training just a more complex graph update

model computation

update

model computation

parameters

update

model computation

Synchronous Variant update

add

gradient

model computation

gradient

model computation

parameters

gradient

model computation

Nurturing Great Researchers ●

We’re always looking for people with the potential to become excellent machine learning researchers



The resurgence of deep learning in the last few years has caused a surge of interest of people who want to learn more and conduct research in this area

Google Brain Residency Program New one year immersion program in deep learning research Learn to conduct deep learning research w/experts in our team ●

Fixed one-year employment with salary, benefits, ...



Goal after one year is to have conducted several research projects



Interesting problems, TensorFlow, and access to computational resources

Google Brain Residency Program Who should apply? ●

people with BSc or MSc, ideally in computer science, mathematics or statistics



completed coursework in calculus, linear algebra, and probability, or equiv.



programming experience



motivated, hard working, and have a strong interest in Deep Learning

Google Brain Residency Program

Program Application & Timeline

Google Brain Residency Program For more information:

g.co/brainresidency Contact us:

[email protected]

Questions?

Large-Scale Deep Learning for Intelligent ... - Research at Google

Android. Apps. GMail. Image Understanding. Maps. NLP. Photos. Robotics. Speech. Translation many research uses.. YouTube … many others . ... Page 10 ...

2MB Sizes 3 Downloads 403 Views

Recommend Documents

Learning with Deep Cascades - Research at Google
based on feature monomials of degree k, or polynomial functions of degree k, ... on finding the best trade-off between computational cost and classification accu-.

Tera-scale deep learning - Research at Google
The Trend of BigData .... Scaling up Deep Learning. Real data. Deep learning data ... Le, et al., Building high-‐level features using large-‐scale unsupervised ...

Deep Learning Methods for Efficient Large ... - Research at Google
Jul 26, 2017 - Google Cloud & YouTube-8M Video. Understanding Challenge ... GAP scores are from private leaderboard. Models. MoNN. LSTM GRU.

Large-Scale Deep Learning for Intelligent Computer Systems - WSDM
Page 10 ... Growing Use of Deep Learning at Google. Android. Apps drug discovery. Gmail. Image understanding. Maps. Natural language understanding.

Deep Learning in Speech Synthesis - Research at Google
Aug 31, 2013 - Heiga Zen. Deep Learning in Speech Synthesis. August 31st, 2013. 6 of 50 ..... w/ vs w/o grouping questions (e.g., vowel, fricative). − Grouping ...

Resurrecting the sigmoid in deep learning ... - Research at Google
Since error information backpropagates faithfully and isometrically through the network, this stronger requirement is called dynamical isometry [10]. A theoretical analysis of exact solutions to the nonlinear dynamics of learning in deep linear netwo

Development and Validation of a Deep Learning ... - Research at Google
Nov 29, 2016 - CR1/DGi/CR2, and Topcon NW using 45° fields of view. ..... A, Model performance on the tuning set (24 360 images) as a function of number.

UNSUPERVISED CONTEXT LEARNING FOR ... - Research at Google
grams. If an n-gram doesn't appear very often in the training ... for training effective biasing models using far less data than ..... We also described how to auto-.

Multiframe Deep Neural Networks for Acoustic ... - Research at Google
windows going up to 400 ms. Given this very long temporal context, it is tempting to wonder whether one can run neural networks at a lower frame rate than the ...

DEEP MIXTURE DENSITY NETWORKS FOR ... - Research at Google
Statistical parametric speech synthesis (SPSS) using deep neural net- works (DNNs) has .... is the set of input/output pairs in the training data, N is the number ... The speech analysis conditions and model topologies were similar to those used ...

Deep Shot: A Framework for Migrating Tasks ... - Research at Google
contact's information with a native Android application. We make ... needed to return a page [10]. ... mobile operating systems such as Apple iOS and Android.

Deep Boosting - Proceedings of Machine Learning Research
We give new data-dependent learning bounds for convex ensembles. These guarantees are expressed in terms of the Rademacher complexities of the sub-families. Hk and the mixture weight assigned to each Hk, in ad- dition to the familiar margin terms and

Maths Research - Deep Learning Website.pdf
... Cumbria www.cumbria.ac.uk/LED ). Andy Ash (Deep Learning Teaching School Alliance http://www.deeplearningtsa.co.uk/ ). With seven teacher researchers: Lucy Evans, Ann Kirk, Rosie Ross, Paula Spenceley, Vicky. Stout, Adam Vasco and Keri Williams.

Deep Neural Networks for Small Footprint Text ... - Research at Google
dimensional log filterbank energy features extracted from a given frame, together .... [13] B. Yegnanarayana and S.P. Kishore, “AANN: an alternative to. GMM for ...

Unsupervised deep clustering for semantic ... - Research at Google
Experiments: Cifar. We also tried contrastive loss : Hadsell et al.Since the task is hard, no obvious clusters were formed. Two classes from Cifar 10. Evaluation process uses the labels for visualization (above). The figures show accuracy per learned

Unsupervised deep clustering for semantic ... - Research at Google
You can extract moving objects which will be entities. We won't know their class but will discover semantic affiliation. The goal is to (learn to) detect them in out-of-sample images. Unsupervised! Clearly all these apply to weakly supervised or semi

Deep Boosting - Proceedings of Machine Learning Research
ysis, with performance guarantees in terms of the margins ... In many successful applications of AdaBoost, H is reduced .... Our proof technique exploits standard tools used to de- ..... {0,..., 9}, fold i was used for testing, fold i +1(mod 10).

Dynamic Model Selection for Hierarchical Deep ... - Research at Google
Figure 2: An illustration of the equivalence between single layers ... assignments as Bernoulli random variables and draw a dif- ..... lowed by 50% Dropout.

permutation grouping: intelligent hash function ... - Research at Google
The combination of MinHash-based signatures and Locality-. Sensitive Hashing ... of hash groups (out of L) matched the probe is the best match. Assuming ..... [4] Cohen, E. et. al (2001) Finding interesting associations without support pruning.

Intelligent Email: Reply and Attachment Prediction - Research at Google
email overload, reply prediction, attachment prediction. INTRODUCTION. Numerous ..... html text alternatives and forwarded messages, not actual at- tachments.

Phrase Clustering for Discriminative Learning - Research at Google
Aug 7, 2009 - data, even if the data set is a relatively large one. (e.g., the Penn Treebank). While the labeled data is generally very costly to obtain, there is a ...

Learning Acoustic Frame Labeling for Speech ... - Research at Google
learning technique for sequence labeling using RNNs where the ..... aI. (j) CTC + sMBR bidirectional LSTM with phone labels. Fig. 1: Label posteriors estimated ...

Theoretical Foundations for Learning Kernels in ... - Research at Google
an unsupervised procedure, mostly used for exploratory data analysis or visual ... work is in analysing the learning kernel problem in the context of coupled ...