Lecture 4: Word Window Classification and Neural Networks

By: Stanford University School of Engineering

[MUSIC] Stanford University. It's getting real today. So, let's talk about a little bit of the overview today. So, we'll really get you into the background for classification. And then, we'll do some interesting things with updating these word vectors that we so far have learned in an unsupervised way.

We'll update them with some real supervision signals such as sentiment and other things. Then, we'll look at the first real model that is actually useful and you might wanna use in practice. Well, other than, of course, the word vectors, but one sort of downstream task which is window classification and we'll really also clear up some of the confusion around the cross entropy error and how it connects with the softmax. And then, we'll introduce the famous neural network, our most basic LEGO block that we may start to call deep to get to the actual title of this class. Deep learning in NLP. And then, we'll actually introduce another loss function, the max margin loss and take our first steps into the direction of backprop.

So, this lecture will be, I think very helpful for problem set one. We'll go into a lot of the math that you'll need probably for number two in the problem set. So, I hope it'll be very useful and I'm excited for you cuz at the end of this lecture, you'll feel hopefully a lot better about the magic of deep learning.

All right, are there any organizational questions around problem sets or programming sessions with the TAs? No, we're all good? Awesome, thanks to the TAs for clearing up everything. Cool, so let's be very careful about our notation today because that is one of the main things that a lot of people trip up over as we go through very complex chain-rules and so on. So, let's start at the beginning and say, all right, we have usually a training dataset of some input X and some output Y. X could be in the simplest case, words in isolation, just a single word vector.

Lecture 4: Word Window Classification and Neural Networks

It's not something you would usually do in practice. But it'll be easy for us to learn that way. So we'll start with that but then, we'll move to context windows today. And then eventually, we'll use the same basic building blocks that we introduce today for sentences and documents and then complex interactions for everything. Now, the output in the simplest case it's just a single label.

It's just a positive or a negative kind of sentence. It could be the named entities of certain words in their context. It can also be other words, so in machine translation, for instance, you might wanna output eventually a sequence of other words as our yi and we'll get to that in a couple weeks. And, yeah, basically they have multiword sequences as potential outputs. All right, so what is the intuition for classification? In the standard machine learning case, so not yet the deep learning world, we usually just, for something as simple logistic regression, basically want to define and learn a simple decision boundary where we say everything to the left of this or in one direction is in one class and the other one, all the other things in the other class. And so, in general machine learning, we assume our inputs, the Xs are kinda fixed, they're just set and we'll only train the W parameter, which is our softmax weights. So, we'll compute the probability of Y, given the input X with this kind of input. And so, one notational comment here is for the whole dataset, we often subscript with i but then, when I drop the i we're just looking at a single example of x and y.

Eventually, we're going to overload at the subscript a little bit and look at the indices of certain vector so, if you get confused, just raise your hand and ask. I'll try to make it clear which one is which. Now, let's dive into the softmax. We mentioned it before but we wanna really carefully define and recall the notation here cuz we'll go and take derivatives with respect to all of these parameters. So, we can tease apart two steps here for computing this probability of y given x. The first thing is, we'll take the y'th row of W and multiply that row with x. And so again this notation here, when we have Wy.

And that means we'll have, we're taking the y'th row of this matrix. And then, multiplying it here with x. Now if we do that multiple times for all c from one to our classes.

So let's say, this is 1, 2, 3, the 4th row and multiply each of these. So then we get four numbers here. And these are unnormalized scores. And then, we'll basically, pipe this vector through the softmax to compute a probability distribution that sums to one. All right, that's our step one. Any questions around that? Cuz it's just gonna keep on going from here. All right, great.

And, I get that sometimes in general from previous sort of surveys, it seems to be that 15% of the class are usually bored when we go through all of these, like all of these derivatives. 15% are super overwhelmed and then the majority of people are like, okay, it's a good speed, I'm learning something, I'm getting it, and you're making progress. So, sorry for the 30% for whom this is too slow or too fast.

You can probably just skim through the lecture slides or speed it up if you're watching online. If you're super familiar with taking super complex derivatives and if it's a little overwhelming, then definitely come to all the office hours. We have an awesome set of TAs who will help you. All right, now we, let's look at a single example of an x and y that we wanna predict. In general, we want our model to essentially maximize the probability of the correct class. We wanted to output the right class at the end by taking the argmax of that output. And maximizing probability is the same as maximizing log probability, it's the same as minimizing the negative of that log probability and that is often our objective function. So, why do we call this the cross-entropy error? Well, we can define the cross-entropy in the abstract in general as follows.

So let's assume we have the ground truth or gold or target probability distribution, we use those three terms interchangeably. Basically, what the ideal target in our training dataset, the y and we'll assume that, that is one at the right class and zero everywhere else. So if we have for instance, five classes here and it's the center class. Its the third class and this would be one and all the other, numbers would be zero. So, if we define this as p in our computed probability, that our softmax outputs as q then we would define here the cross-entropy is basically this sum over all the classes. And in our case, p here is just one-hot vector that's really only 1 in one location and 0 everywhere else. So, all these other terms are basically gone. And we end up with just log of q and that's exactly the log of what our softmax outputs, all right? And then, there are some nice connections to Kullback-Leibler divergence and so on.

I used to talk about it but we don't have that much time today. So and you can also if you're familiar of this in stats, you can see this as trying to minimize the Kullback-Leibler divergence between these two distributions. But really, this is all you need to know for the purpose of this class.

So this is for one element of your training data set. Now, of course, in general, you have lots of training examples. So we have our overall objective function we often denote with J, over all our parameters theta. And we basically sum these negative log probabilities of the correct classes that we index here, a sub-index with yi. And basically we want to minimize this whole sum.

So that's our cross-entropy error that we're trying to minimize, and we'll take lots of derivatives off in a lot of the next couple of hours. All right, any questions so far? So this is the general ML case where we assume our inputs here are fixed. Yes, it's a single number. So we are not multiplying a vector here, so p(c) is the probability for that class, so that's one single number. Great question. So the cross entropy, a single number, our main objective that we're trying to minimize, or our error that we're trying to minimize.

Now, whenever you write this F subscript Y here, we don't want to forget that F is really also a function of X, our inputs, right? It's sort of an intermediate step and it's very important for us to play around with this notation. So we can also rewrite this as W y, that row, times x, and we can write out that whole sum. And that can often be helpful as you are trying to take derivatives of one element at a time to eventually see the bigger picture of the whole matrix notation. All right, so often we'll write f here in terms of this matrix notation. So this is our f, this is our W, and this is our x. So just standard matrix multiplication with a vector.

All right, now most of the time we'll just talk about this first part of the objective function but it's a bit of a simplification because in all your real applications you will also have this regularization term here. As part of your overall objective function. And in many cases, this theta here for instance, if it's the W matrix of our standard logistic regression, we'll essentially just try this part of the objective function. We'll try to encourage the model to keep all the weights as small as possible and as close as possible to zero. You can kind of assume if you want as a Bayesian that you can have a prior, a Gaussian distributed prior that says ideally all these are small numbers. Often times if you don't have this regularization term your numbers will blow up and it will start to overfit more and more. And in fact, this kind of plot is something that you will very often see in your projects and even in the problem sets. And when I took my very first statistical learning class, the professor said, this is the number one plot to remember.

So, I don't know if it's that important, but it is very, very important for all our applications. And it's basically a pretty abstract plot. You can think of the x-axis as a variety of different things. For instance, how powerful your model is. How many deep layers you'll have or how many parameters you'll have. Or how many dimensions each word vector has. Or how long you trained a model for. You'll see the same kind of pattern with a lot of different, x-axis and then the y-axis here is essentially your error.

Or your objective function that you're trying to optimize and minimize. And what you often observe is, the more powerful your model gets, the better you are on lowering your training error, the better you can fit these x-i, y-i pairs. But at some point you'll actually start to over-fit, and then your test error, or your validation or development set error, will go up again. We'll go into a little bit more details on how to avoid all of that throughout this course and in the project advice and so on.

But this is a pretty fundamental thing and just keep in mind that for a lot of the implementations, and your projects you will want this regularization parameter. But really it's the same one for almost all the objective functions so we're going to chop it and mostly focus on actually fitting our dataset. All right, any questions around regularization? So basically, you can think of this in terms of if you really care about one specific number, then you can adjust all your parameters such that it will exactly go to those different points. And if you force it to not do that, it will kind of be a little smoother. And be less likely to fit exactly those points and hence often generalize slightly better. And we'll go through a couple of examples of what this will look like soon. All right, now as I mentioned in general machine learning, we'll only optimize the W here, the parameters of our Softmax classifier. And hence our updates and gradients will only be pretty small, so in many cases we only have you know a handful of classes and maybe our word vectors are hundred so if we have three classes and 100 dimensional word vectors we're trying to classify, we'd only have 300 parameters.

Now, in deep learning, we have these amazing word vectors. And we actually will want to learn not just the Softmax but also the word vectors. We can back propagate into them and we'll talk about how to do that today. Hint, it's going to be taking derivatives. But the problem is when we update word vectors, conceptually as you are thinking through this, you have to realize this is very, very large. And now all of the sudden have a very large set of parameters, right? Let's say your word vectors are 300 dimensional you have, you know 10,000 words in your vocabulary. All of the sudden you have an immensely large set of parameters so on this kind of plot you're going to be very likely to overfit. And so before we dive into all this optimization, I want you to get a little bit of an intuition of what it means to update word vectors.

So let's go through a very simple example where we might want to classify single words. Again, it's not something we'll do very often, but let's say you want to classify single words as positive or negative. And let's say in our training data set we have the word TV and telly and say you know this is movie reviews and if you say this movie is better suited for TV. It's not a very positive thing to say about a movie that's just coming out into movie theaters. And so we would assume that in the beginning telly, TV, and television are actually all close by in the vector space. We learn something with word2vec or glove vectors and we train these word vectors on a very, very large corpus and it learned all these three words appear often in a similar context, so they are close by in the vector space.

And now we're going to train but, our smaller sentiment data set only includes in the training set, the X-i Y-i as TV and telly and not television. So now what happens as we train these word vectors? Well, they will start to move around. We'll project sentiment into them and so you now might see telly and TV, that's a British dataset, so like to move somewhere else into the vector space.

But television actually stays where it was in the beginning. And now when we want to test it, we would actually now misclassify this word because it's never been moved. And so what does that mean? The take home message here will be that if you have only a very small training dataset. That will allow you especially with these deep models to overfit very quickly, you do not want to train your word vectors. You want to keep them fixed, you pre-trained them with nice Glove or word2vec models on a very large corpus or you just downloaded them from the cloud website and you want to keep them fixed, cuz otherwise you will not generalize as well. However, if you have a very large dataset it may be better to train them in a way we're going to describe in the next couple of slides. So, an example for where you do that is, for instance, machine translation where you might have many hundreds of Megabytes or Gigabytes of training data and you don't really need to do much with the word vectors other than initialize them randomly, and then train them as part of your overall objective.

All right, any questions around generalization capabilities of word vectors? All right, it might still be magical how we're training this, so that's what we're gonna describe now. So, we rarely ever really classify single words. Really what we wanna do is classify words in their context. And there are a lot of fun and interesting. Issues that arise in context really that's where language begins and grammar and the connection to meaning and so on.

So here, a couple of fun examples of where context is really necessary. So for instance, we have some words that actually auto-antonyms, so they mean their own opposite. So for instance to sanction can mean to permit or to punish. And it really depends on the context for you to understand which one is meant, or to seed can mean to place seeds or to remove seeds. So without the context, we wouldn't really understand the meaning of these words. And in one of the examples that you'll see a lot, which is named entity recognition, let's say we wanna find locations or people names, we wanna identify is this the location or not. You may also have things like Paris, which could be Paris in France or Paris Hilton.

And you might have Paris staying in Paris and you still wanna understand which one is which. Or if you wanna use deep learning for financial trading and you see Hathaway, you wanna make sure that if it's just a positive movie review from Anne Hathaway. You're not all the sudden buying stocks from Berkshire Hathaway, right? And so, there are a lot of issues that are fun and interesting and complex that arise in context. And so, let's now carefully walk through this first useful model, which is Window classification. So, we'll use as our first motivating example here 4-class named entity recognition, where we basically wanna identify a person or location or organization or none of the above for every single word in a large corpus. And there are lots of different possibilities that exist. But we'll basically look at the following model. Which is actually quite a reasonable model.

And also one that started in 2008. So the first beginning by Collobert and Weston, a great paper, to do the first kind of useful state of the art Text classification and word classification context. So, what we wanna do is basically train a softmax classifier by assigning a label to the center word and then concatenating all the words in a window around that word. So, let's take for example this subphrase here from a longer sentence.

We basically wanna classify the center word here which is Paris, in the context of this window. And we'll define the window length as 2. 2 being 2 words to the left and 2 words to the right of the current center word that we're trying to classify. All right, so what we will do is we'll define our new x for this whole window as the concatenation of these five word vectors. And just in general throughout all of this lecture all my vectors are going to be column vectors. Sadly in number two of the problem set, they're row vectors. Sorry for that.

Eventually, all these programming frameworks they're actually row-wise first and so it's faster in the low-level optimization to use row vectors. For a lot of the math it's actually I find it simpler to think of them as column vectors so. We're very clear in the problem set but don't get tripped up on that.

So basically, we'll define this here as one five D dimensional column vector. So, we have T dimensional word vectors, we have five of them and we stack them up in one column, all right. Now, the simplest window classifier that we could think of is to now just put the softmax on top of this concatenation of five word vectors and we'll define this, our x here. Our inputs is just the x of the entire window for this concatenation. And we have the softmax on top of that. And so, this is the same notation that we used before. We're introducing here y hat, with sadly the subscript y for the correct current class.

It's tough, I went through [LAUGH] several iterations, it's tough to have like prefect notation that works through the entire lecture always. But you'll see why soon. So, our overall objective here is, again, this whole sum over all these probabilities that we have, or negative log of those. So now, the question is, how do we update these word vectors x here? One x is a window, and x is now deep inside the softmax.

All right, well, the short answer is we'll take a lot of derivatives. But the long answer is, you're gonna have to do that a lot in problem set one and maybe in the midterm. So, let's be a little more helpful, and actually go through some of the steps and give you some hints. So some of this, you'll actually have to do in your problem set, so I'm not gonna go through all the details. But I'll give you a couple of hints along the way and then you can know if you're hitting those and then you'll see if you're on the right track. So, step one, always very carefully define your variables, their dimensionality and everything.

So, y hat will define as the softmax probability of the vector. So, the normalized scores or the probabilities for all the different classes that we have. So, in our case we have four. Then we have the target distribution. Again, that will be a one hot vector where it's all zeroes except at the ground truth index of the class y, where it's one. And we'll define our f here as f of x again, which is this matrix multiplication. Which is going to be a C dimensional vector where capital C is the number of classes that we have, all right. So, that was step one.

Carefully define all of your variables and keep track of their dimensionality. It's very easy when you implement this and you multiply two things, and they have wrong dimensionality, and you can't actually legally multiply them, you know you have a bug. And you can do this also in a lot of your equations. You'd be surprised. In the midterm, you're nervous. But maybe at the end you have some time.

And you could totally grade it by yourself in the first pass, by just making sure that all your dimensionality of your matrix and vector multiplications are correct. All right, the second tip is the chain rule, we went over this before, but I heard there's a little bit of confusion still in the office hours. So, let's define this carefully for a simple example and then we'll go and give you a couple more hints also for more complex example. So again, if you have something very simple, such as a function y, which you can defined here as f of u and u can be defined as g of x as in the whole function, y of x, can be described as f of g of x, then you would basically multiply dy, u times the udx. And so very concretely here, this is sort of high school level, but we'll define it properly in order to show the chain rule. So here, you can basically define u as g(x), which is just the inside in the parentheses here, so x cubed + 7. It can have y as a function of f(u), where we use 5 times u, just replacing the inside definition here. So it's very simple, just replacing things.

And now, we can take the derivative with respect to u and we can take the derivative with respect to x(u). And then we just multiply these two terms, and we plug in u again. So in that sense, we all know, in theory, the chain rule. But, now we're gonna have the softmax, and we're gonna have lots of matrices and so on. So, we have to be very, very careful about our notation. And we also have to be careful about understanding, which parameters appear inside what other higher level elements. So, f for instance is a function of x.

So, if you're trying to take a derivative with respect to x, of this overall soft max you're gonna have to sum over all of the different classes inside which x appears. And you'll see here, this first application, but not just of fy again this is just a subscript the y element of the effector which is the function of x, but also multiply it then here by this. So, when you write this out, another tip that can be helpful is for this softmax part of he derivative is to actually think of two cases. One where c = y, the correct class, and one where it's basically all the other incorrect classes.

And as you write this out, you will observe and come up with something like this. So, don't just write that as your thing you have to put in your problems, the steps on how to get there. Bur, basically at some point you observe this kinda pattern when you now try to look at all the derivatives with respect to all the elements of f. And now, when you have this you realize ,okay at the correct class we're actually subtracting one here, and all the incorrect classes, you will not do anything. Now, the problem is when you implement this, it kind of looks like a bunch of if statements. If y equals the correct class for my training set, then, subtract 1, that's not gonna be very efficient. Also, you're gonna go insane if you try to actually write down equations for more complex neural network architectures ever. And so, instead, what we wanna do is always try to vectorize a lot of our notation, as well as our implementation.

And so, what this means here, in this case, is you can actually observe that, well, this 1 is exactly 1, where t, our hot to target distribution, also happens to be 1. And so, what you're gonna wanna do, is basically describe this as y(hat)- t, so it's the same thing as this. And don't worry if you don't understand how we got there, cuz that's part of your problem set. You have to, at some point, see this equation while you're taking those derivatives. And now, the very first baby step towards back-propagation is actually to define this term, in terms of a simpler single variable and we'll call this delta. We'll get good, we'll become good friends with deltas because they are sort of our error signals.

Now, the last couple of tips. Tip number six. When you start with this chain rule, you might want to sometimes use explicit sums, before and look at all the partial derivatives. And if you do that a couple of times at some point you see a pattern, and then you try to think of how to extrapolate from those patterns of single partial derivatives, into vector and matrix notation. So, for example, you'll see something like this here, in at some point in your derivation.

S,o the overall derivative with respect to x of our overall objective function for one element, for one element from our training set x and y is this sum. And it turns out when you think about this for a while, you take here this row vector but then you transpose it, and becomes an inner product, well if you do that multiple times for all the C's and you wanna get in the end a whole vector out, it turns out you can actually just re-write the sum as W transpose* the delta. So, this is one error signal here that we got from our softmax, and we multiply the transpose of our softmax weights with this. And again, if some of these are not clear and you're confused, write them out into full sum, and then you'll see that it's really just re-write this in vector notation. All right, now what is the dimensionality of the window vector gradient? So in the end, we have this derivative of the overall cost here for one element of our training set with respect to x. But x is a window. All right, so each say we have a window of five words. And each word is d-dimensional.

Now, what should be the dimensionality of this derivative of this gradient? That's right, it's five times the dimensionality. And that's another really good way, and one of the reasons we make you implement this from scratch, if you have any kinda parameter, and you have a gradient for that parameter, and they're not the same dimensionality, you'll also know you screwed up and there's some mistake or bug in either your code or your map. So, it's very simple debugging skill. And way to check your own equations. So, the final derivative with respect to this window is now this five vector because we had five d-dimensional vectors that we concatenated.

Now, of course the tricky bit is, you actually wanna update your word vectors and not the whole window, right? The window is just this intermediate step also. So really, what you wanna do is update and take derivatives with respect to each of the elements of your word vectors. And so it turns out, very simply, that can be done by just splitting that error that you've got on the gradient overall, at the whole window and that's just basically the concatenation of the reduced of all the different word vectors. And those you can use to update your word vectors, as you train the whole system. All right, any questions? Is there a mathematical what? Is there a mathematical notation for the word vector t, other than it's just variable t? Or that seems like a fine notation. You can see this as a probability distribution, that is very peaked.

>> Yeah. >> That's all, there's nothing else to it. Just a single vector with all zeroes, except in one location. >> So I'll just write that down? >> You can write that up, yeah. You can always just write out and it's also something very important.

You always wanna define everything, so that you make sure that the TAs know that you're thinking about the right thing, as you're writing out your derivatives, you write out the dimensionality, you define them properly, you can use dot, dot, dot if it's a larger dimensional vector. You can just define t as your target distribution [INAUDIBLE] >> The question is, do we still have two vectors for each word? Great question, no. We essentially, when we did glove and word2vec, and had these two u's and v's, for all subsequent lectures from now on, we'll just assume we have the sum of u and v and that's our single vector x, for each word.

So, the question is does this gradient appear in lots of other windows and it does. So, if you, the answer is yes. If you have the word "in," that vector here and the gradients will appear in all the windows that have the word "in" inside of them.

And same with museums and so on. And so as you do stochastic gradient descent you look at one window at a time, you update it, then you go to the next window, you update it and so on. Great questions. All right.

Now, let's look at how we update these concatenated word vectors. So basically, as we're training this, if we train it for instance with sentiment we'll push all the positive words in one direction and the other words in other direction. If we train it, for named entity recognition and eventually our model can learn that seeing something like in as the word just before the center word, would be indicative for that center word to be a location. So now what's missing for training this full window model? Well mainly the gradient of J with respect to the softmax weights W. And so we basically will take similar steps. We'll write down all the partial derivatives with respect to Wij first and so on. And then we have our full gradient for this entire model. And again, this will be very sparse, and you're gonna wanna have some clever ways of implementing these word vector updates.

So you don't send a bunch of zeros around at every single window, Cuz each window will only have a few words. So in fact, it's so important for your code in the problem set to think carefully through your matrix implementations, that it's worth to spend two or three slides on this. So there are essentially two very expensive operations in the softmax. The matrix multiplication and the exponent. Actually later in the lecture, we'll find a way to deal with the exponent.

But the matrix multiplication can also be implemented much more efficiently. So you might be tempted in the beginning to think this is probability for this class and this is the probability for that class. And so implemented a for loop of all my different classes and then I'll take derivatives or matrix multiplications one row at a time.

And that is going to be very, very inefficient. So let's go through some very simple Python code here to show you what I mean. So essentially, always looping over these word vectors instead of concatenating everything into one large matrix. And then multiplying these is always going to be more efficient. So let's assume we have 500 windows that we want to classify, and let's assume each window has a dimensionality of 300. These are reasonable numbers, and let's assume we have five classes in our softmax. And so at some point during the computation, we now have two options.

So W here are weights for the softmax. It's gonna be C many rows and d many columns. Now the word vectors here that you concatenated for each window. We can either have the list of a bunch of separate word vectors, or we can have one large matrix that's going to be d times n. So d many rows and n many windows. So we have 500 windows, so we have 500 columns here in this 1 matrix. And now essentially, we can multiply the W here for each vector separately, or we can do this one matrix multiplication entirely. And you literally have a 12x speed difference.

And sadly with these larger models, one iteration or something might take a day, eventually for more complex models large data sets. So the difference is between literally 12 days or 1 day of you iterating and making your deadlines and everything. So it's super important, and now sometimes people are tripped up by what does it mean to multiply and do this here. Essentially, it's the same thing that we've done here for one softmax, but what we did is we actually concatenated.

A lot of different input vectors x, and so we'll get a lot of different unnormalized scores out at the end. And then we can tease them apart again for them. So you have here, c times t dimensional matrix for the d dimensional input. So using the same notation, yeah, dimensional of each window times d times n matrix to get a c times n matrix. So these are all the probabilities here for your N many training samples. Any questions around that? So it's super important, all your code will be way too slow if you don't do this.

And so this is very much an implementation trick. And so in most of the equations, we're not gonna actually go there cuz that makes everything more complicated. And the equations look at only a singular example at a time, but in the end you're gonna wanna vectorize all your code. Yeah, matrices are your friend, use them as much as you can.

Also in many cases, especially for this problem set where you really understand the nuts and bolts of how to train and optimize your models. You will come across a lot of different choices. It's like, I could implement it this way or that way. And you can go to your TA and ask, should I implement this way or that way? But you can also just use time it as your magic Python and just let, make a very informed decision and gain intuition yourself. And just basically wanna speed test a lot of different options that you have in your code a lot of the time. All right, so this is was just a pure softmax, and now the softmax alone is not play powerful. Because it really only gets with this linear decision boundaries in your original space. If you have very, very little training data that could be okay, and you kind of used a not so powerful model almost as an abstract regularizer.

But with more data, it's actually quite limiting. So if we have here a bunch of words and we don't wanna update our word vectors, softmax would only give us this linear decision boundary which is kind of lame. And it would be way better if we could correctly classify these points here as well. And so basically, this is one of the many motivations for using neural networks. Cuz neural networks will give us much more complex decision boundaries and allow us to fit much more complex functions to our training data.

And you could be snarky and actually rename neural networks which sounds really cool. It's just general function approximators. Just wouldn't have quite the same ring to it, but it's essentially what they are. So let's define how we get from the symbol of logistic regression to a neural network and beyond, and deep neural nets. So let's demystify the whole thing by starting, defining again some of the terminology.

And we can have more fun with the math, and then one and a half lectures from now. We can just basically use all of these Lego blocks. So bear with me, this is going to be tough. And try to concentrate and ask questions if you have any, cuz we'll keep building now a pretty awesome large model that's really useful. So we'll have inputs, we'll have a bias unit, we'll have an activation function and output for each single neuron in our larger neuron network. So let's define a single neuron first. Basically, you can see it as a binary logistic regression unit.

We're going to have inside, again a set of weights that we have in a product with our input. So we have the input x here to this neuron. And in the end, we're going to add a bias term. So we have an always on feature, and that kind of defines how likely should this neuron fire. And by firing, I mean have a very high probability that's close to one. For being on.

And f here is always, from now on, going to be this element wise function. In our case here the sigmoid that just squashes whatever this sum gives us in our product plus the bias term and basically just squashes it to be between 0 and 1. All right, so this is the definition of the single neuron. Now if we feed a vector of inputs through all this different little logistic regression functions and neurons, we get this output.

And now the main difference between just predicting directly a softmax and standard machine learning and deep learning is that we'll actually not force this to give directly the output. But they will themselves be inputs to yet another neuron. And it's a loss function on top of that neuron such as cross entropy that will now govern what these intermediate hidden neurons.

Or in the hidden layer what they will actually try to achieve. And the model can decide itself what it should represent, how it should transform this input inside these hidden units here in order to give us a lower error at the final output. And it's really just this concatenation of these hidden neurons, these little binary logistic regression units that will allow us to build very deep neural network architectures.

Now again, for sanity's sake, we're going to have to use matrix notation cuz all of this can be very simply described in terms of matrix multiplication. So a1 here is where going to be the final activation of the first neuron, a2 in second neuron and so on. So instead of writing out the inner product here, or writing even this as an inner product plus the bias term we're going to use matrix notation.

And it's very important now to pay attention to this intermediate variables that we'll define because we'll see these over and over again as we use a chain rule to take derivatives. So we'll define z here as W times x plus the bias vector. So we'll basically have here as many bias terms and this vector has the same dimensionality as the number of neurons that we have in this layer. And W will have number of rows for the number of neurons that we have times number of columns for the input dimensionality of x. And then, whenever we write a of f(z), what that means here is that we'll actually apply f element wise. So f(z) when z is a vector is just f(z1), f(z2) and f(z3).

And now you might ask, well, why do we have all this added complexity here with this sigmoid function. Later on we can actually have other kinds of so called non linearities. This f function and it turns out that if we don't have the non-linearities in between and we will just stack a couple of this linear layers together it wouldn't add a very different function.

In fact it would be continuing to just be a single linear function. And intuitively as you have more hidden neurons, you can fit more and more complex functions. So this is like a decision boundary in a three dimensional space, you can think of it also in terms of simple regression. If you had just a single hidden neuron, you kinda see here almost an inverted sigmoid. If you have three hidden neurons, you could fit this kind of more complex functions and with ten neurons, each neuron can start to essentially, over fit and try to be very good at fitting exactly one point. All right, now let's revisit our single window classifier and instead of slapping a softmax directly onto the word vectors we're now going to have an intermediate hidden layer between the word vectors and the output. And that's when we really start to gain an accuracy and expressive power. So let's define a single layer neural network.

We have our input x that will be again, our window, the concatenation of multiple word vectors. We'll define z and we'll define a as element wise on the areas a and z. And now, we can use this neural activation vector a as input to our final classification layer. The default that we've had so far was the softmax, but let's not rederive the softmax. We've done it multiple times now, you'll do it again in a problem set and introduce an even simpler one and walk through all the glory details of that simple classifier. And that will be a simple, unnormalized score. And this case here, this will essentially be the right mechanism for various simple binary classification problems, where you don't even care that much about this probability z is 0.8. You really just cares like, is it one, is it in this class, or is it not? And so we'll define the objective function for this new output layer in a second.

Well, let's first understand the feed-forward process. And well feed-forward process is what you will end up using a test time and for each element also in training before you can take derivative. Always be feed-forward and then backward to take the derivatives.

So what we wanna do here is for example, take basically each window and then score it. And say if the score is high we want to train the model such that it would assign high scores to windows where the center word is a named entity location. Such as Paris, or London, or Germany, or Stanford, or something like that. Now we will often use and you'll see a in a lot of papers this kind of graph, so it's good to get used to it. There are various other kinds, and we'll try to introduce them slowly throughout the lecture but this is the most common one.

So we'll define bottom up, what each of these layers will do and then we'll take the derivatives and learn how to optimize it. Now x window here is the concatenation of all our word vectors. So let's hear, and I'll ask you a question in a second, let's try to figure out the dimensionality here of all our parameters so that you're, I know you're with me. So let's say each of our word vectors here is four dimensional and we have five of these word vectors in each window that are concatenated. So x is a 20 dimensional vector.

And again, we'll define it as column vectors. And then lets say we have in our first hidden layer, lets say we have eight units here. So you want an eight unit hidden layer as our intermediate representation. And then our final scores just again a simple single number. Now what's the dimensionality of our W given what I just said? 20 dimensional input, eight hidden units. 20 rows and eight columns. We have one more transfer, [LAUGH] that's right. So it's going to be eight rows and 20 columns, right? And you can always whenever you're unsure and you have something like this then this will have some n times d.

And then multiply this and then this will have, this will always be d, and so these two always have to be the same, right? So all right, now what's the main intuition behind this extra layer, especially for NLP? Well, that will allow us to learn non-linear interactions between these different input words. Whereas before, we could only say well if in appears in this location, always increase the probability that the next word is a location. Now we can learn things and patterns like, if in is in the second position, increase the probability of this being the location only if museum is also the first vector. So we can learn interactions between these different inputs. And now we'll eventually make our model more accurate. Great question. So do I have a second W there.

So the second layer here the scores are unnormalized, so it'll just be U and because we just have a single U, this will just be a single column vector and we'll transpose that to get our inner product to get a single number out for the score. Sorry, yeah, so the question was do we have a second W vector. So yeah, that's in some sense our second matrix, but because we only have one hidden neuron in that layer, we only need a single vector. Wonderful. All right, so, now let's define the max-margin loss. It's actually a super powerful loss function often is even more robust than the cross entropy error in softmax, and is quite powerful and useful. So let's define here two examples. Basically, you want to give a high score to windows, where the center word is a location.

And we wanna give low scores to corrupt or incorrect windows where the center word is not a named entity location. So museum is technically a location, but it's not a named entity location. And so the idea for this training objective of max-margin is to essentially try to make the score of the true windows larger than the ones of the corrupt windows smaller or lower. Until they're good enough.

And we define good enough as being different by the value of one. And this one here is a margin. You can often see it as a hyperparameter too and set it to m and try different ones but in many cases one works fine. This is continuous and we'll be able to use SGD. So now what's the intuition behind the softmax, sorry the max-margin loss here? If you have for instance a very simple data set and you have here a couple of training samples. And here you have the other class c, what a standard softmax may give you is a decision boundary that looks like this. It's like perfectly separates the two.

It's a very simple training example. Most standard softmax classifiers will be able to perfectly separate these two classes. And again, this is just for illustration in two dimensions.

These are much higher dimensional problems and so on. But a lot of the intuition carries through. So now here we have our decision boundary and this is the softmax. Now, the problem is maybe that was your training data set. But your test set, actually, might include some other ones that are quite similar to those stuff you saw at training, but a little different.

And now this kind of decision boundary is not very robust. In contrast to this, what the max margin loss will attempt to do is to try to increase the margin between the closest points of your training data set. So if you have a couple of points here and you have different points here. We'll try to maximize the distance between the closest points here, and essentially be more robust. So then if at test time you have some things that are kinda similar, but not quite there, you're more likely to also correctly classify them. So it's a really great lost or objective function. Now in our case here when we say a sc for one corrupt window.

In many cases in practice we're actually going to have a sum over multiple of these. And you can think of this similar to the skip-gram model where we sample randomly a couple of corrupt examples. So you really only need for this kind of training a bunch of true examples of this is a location in this context. And then all the other windows where you don't have that as your training data are essentially part of your negative class. All right, any questions around the max-margin objective function? We're gonna take a lot of derivatives of it now.

That's right, is the corrupt window just a negative class? Yes, that's exactly right. So you can think of any other window that doesn't have as its center location just as the other class. All right, now how do we optimize this? We're going to take very similar steps to what we've done with cross entropy, but now we actually have this hidden layer and we'll take our second to last step towards the full back-propagation algorithm which we'll cover in the next lecture. So let's assume our cost J here is larger than 0. So what does that mean? In the very beginning you will initialize all your parameters here again. Either randomly or maybe you'll initialize your word vectors to be reasonable.

But they're not gonna be quite perfect at learning in this context in the window what is location and what isn't. And so in the beginning all your scores are likely going to be low cuz all our parameters, U and W and b have been initialized to small, random numbers. And so I'm unlikely going to be great at distinguishing the window with a correct location at center versus one that is corrupt. And so basically, we will be in this regime. After a while of training, eventually you're gonna get better and better.

And then intuitively if your score here for instance of the good window is five and one of the corrupt is just two, then you'll see 1- 5 + 2 is less than 0 so you just basically have 0 loss on those elements. And that's another great property of this objective function which is over time you can start ignoring more and more of your training set cuz it's good enough. It will assign 0 cost as in 0 error to these examples and so you can start to focus on your objective function only on the things that the model still has trouble to distinguish. All right, so let's in the very beginning assume most of our examples will J will be larger than 0 for them.

And so what we're gonna have to do now is take derivatives with respect to all the parameters of our model. And so what are those? Those are U, W, b and our word vectors x. So we always start from the top and then we go down because we'll start to reuse different elements and just the simple combination of taking derivatives and reusing variables is going to lead us to back propagation. So derivative of s with respect to U. Well, what was s? s was just u transpose times a and so we all know that derivative of that is just a. So that was easy, first element, first derivative super straight forward. Now it's important when we take the next derivative to also be aware of all our definitions. How we define these functions that we're taking derivatives off.

So s is basically U transpose a, a was f(z) and z was just Wx + b. All right, it's very important to just keep track. That's like almost 80% of the work.

Now, let's take the derivative like I said, first partial of only one element of W to gain intuitions. And then we can put it back together and have a more complex matrix notation. So we'll observe for Wij that it will actually only appear in the ith activation of our hidden layer. So for example, let's say we have a very simple input with a three dimensional x. And we have two hidden units, and this one final score U.

Then we'll observe that if we take the derivative with respect to W23. So the second row and the third column of W, well that actually only is needed in a2. You can compute a1 without using W23. So what does that mean? That means if we take the derivative of weight Wij, we really only need to look at the ith element of the vector a. And hence, we don't need to look at this whole inner product. So what's the next step? Well as we're taking derivatives with W, we need to be again aware of where does W appear and all the other parameters are essentially constant. So U here is not something we're taking a derivative off.

So what we can do is just take it out, just as like a single number, right. We'll just get it outside, put the derivative inside here. And now, we just need to very carefully define our ai. So a subscript i, so that's where Wij appears. Now, ai was this function, and we defined it as f of zi.

So why don't we just write this carefully out, and now this is first application of the chain rule with derivative of ai with respect to zi, and then zi with respect to Wij. So this is single application of the chain rule. And then end of it it looks kind of overwhelming, but each step is very clear. And each step is simple, we're really writing out all the glory details. So application of the chain rule, now we're going to define ai. Well ai is just f of zi, and f was just an element y function on a single number zi. So we can just rewrite ai with its definition of f of zi, and we keep this one intact, all right? And now derivative of f, we can just for now assume is f prime. Just a single number, take derivative.

We'll just define this as f prime for now. It's also just a single number, so no harm done. Now we're still in this part here, where we basically wanna take the derivative of zi with respect to Wij. Well let's define what zi was, zi was just here. The W of the ith row times x plus the ith element of b. So let's just replace zi with it's definition.

Any questions so far? All right, good or not? So we have our f prime and we have now the derivative with respect to Wij of just this inner product here. And we can again, very carefully write out well, the inner product is just this row times this column vector. That's just the sum, and now when we take the derivative with respect to Wij, all the other Ws are constants.

They fall out, and so basically it's only the xk, the only one that actually appears in the sum with Wij is xj and so basically this derivative is just Xj. All right, so now we have this whole expressions of just taking carefully chain rule multiplications definitions of all our terms and so on. And now basically, what we're gonna want to do is simplify this a little bit, cuz we might want to reuse different parts.

And so we can define, this first term here actually happens to only use subindices i. And it doesn't use any other subindex. So we'll just define Uif prime of zi for all the different is as delta i. At first notational simplicity and xj is our local input signal. And one thing that's very helpful for you to do is actually look at also the derivative of the logistic function here. Which can be very conveniently computed in terms of the original values.

And remember f of z here, or f of zi of each element is always just a single number. And we've already computed it during forward propagation. So we wanna ideally use hidden activation functions that are very fast to compute. And here, we don't need to compute another exponent or anything.

We're not gonna recompute f of zi cuz we already did that in the forward propagation step. All right, now we have the partial derivative here with respect to one element of W. But of course, we wanna have the whole gradient for the whole matrix. So now the question is, with the definitions of this delta i for all the different elements of i of this matrix and xj for all the different elements of the input.

What would be a good way of trying to combine all of these different elements to get a single gradient for the whole matrix W, if we have two vectors. That's right. So essentially, we can use delta times x transpose, namely the outer product to get all the combinations of all elements i and all elements j. And so this again might seem like a little bit like magic.

But if you just think again of the definition of the outer product here. And you write it out in terms of all the indices, you'll see that turns out to be exactly what we would want in one very nice, very simple equation. So we can kind of think of this delta term actually as the responsibility of the error signal that's now arriving from our overall loss into this layer of W. And that will eventually lead us to flow graphs.

And that will eventually lead us to you not having to actually go through all this misery of taking all these derivatives. And being able to abstract it away with software packages. But this is really the nuts and bolts of how this works, yeah? Yeah, the question is, this outer product will get all the elements of i and j? And that's right.

So when we have delta times x transposed. Then now we have basically here, x is usually this vector. So now let's take the right notation.

So we wanna have derivative with respect to W. W was a, 2x3 dimension matrix for example, 2x3. We should be very careful of our notation. 2x3.

So now, the derivative of j with respect to our w has to, in the end, also be a 2x3 matrix. And if we have delta times x transposed, then that means we'll have to have a two-dimensional delta, which is exactly the dimensions that are coming in. [INAUDIBLE] Signal that I mentions that we have for the number of hidden units that we have. Times this one dimensional, basically row vector times xt which is a 1 x 3 dimensional vector that we transpose. And so, what does that mean? Well, that's basically multiplying now, standard matrix multiplication.

You should write that. So now the last term that we haven't taken derivatives of off the [INAUDIBLE], is our bi and it'll eventually be very similar. We're going to go through it. We can pull Ui out, we're going to take f prime, assume that's the same. So now, this is our delta i.

We'll observe something very similar. These are very similar steps for bi. But in the end, we're going to just end up with this term and that's just going to be one. And so, the derivative of our bi element here, is just delta i and we can again use all the elements of delta, to have the entire gradient for the update of b.

Any questions? Excellent, so this is essentially, almost back-propagation. We’ve so far only taken derivatives and using the chain rule. And first thing, when I went through this, this is like a lot of the magic of deep learning, is just becoming a lot clear. We’ve just taken derivatives, we have an objective function and then we update based on our derivatives, all the parameters of these large functions.

Now the main remaining trick, is to re-use derivatives that we've computed for the higher layers in computing derivatives for the lower layers. It's very much an efficiency trick. You could not use it and it would just be very, very inefficient to do. But this is the main insight of why we re-named taking derivatives as back propagation. So what is the last derivatives that we need to take? For this model, well again, it's in terms of our word vectors.

So let's go through all of those. Basically, we'll have to take the derivative of the score with respect to every single element of our word vectors. Where again, we concatenated all of them into a single window. And now, the problem here is that each word vector actually appears in both of these terms. And both hidden units use all of the elements of the input here.

So we can't just look at a single element. We'll really have to sum over, both of the activation units in the simple case here, where we just have two hidden units and three dimensional inputs. Keeps it a little simpler, and there's less notation.

So then, we basically start with this. I have to take derivatives with respect to both of the activations. And now, we're just going to go through similar kinds of steps. We have s. We defined s as u transpose times our activation.

That was just Ui then ai was just f of w and so on. Now, what we'll observe as we're going through all these similar steps again is that, we'll actually see the same term here reused from before. It's Ui x F prime of Zi. This is exactly the same. That we've seen here. F prime of Zi. And what that means is, we can reuse that same delta. And that's really one of the big insights.

Fairly trivial but very exciting, cuz it makes it a lot faster. But, what's still different now, is that of course we have to take the the derivative with respect. To each of these, to this inner product here in Xj, where we basically dumped the bias term, cuz that's just a constant, when we were taking this derivative. And so, this one here again, Xj is just inner product, it's the jth element of this matrix W that's the relevant one for this inner product, let me take the derivative. So now we have this sum here, and now comes again this tricky bit of trying to simplify this sum into something simpler in terms of matrix products. And again, the reason we're getting towards back propagation is that we're reusing here these previous error signals, and elements of the derivative. Now, the simplest, the first thing we'll observe here as we're doing this sum, is that sum is actually also a simple inner product, where we now take the jth column.

So this again, this dot notation when the dot is after the first, and next we take the row, here we take the column. So it's a column vector. But then of course we transpose it, so it's a simple inner product for getting us a single number. Just the derivative of this element of the word vectors and the word window. Yes. Great question. So once we have the derivatives for all these different variables, what's the sequence in which we update them, and there's really no sequence we update them all in parallel.

We just take one step in all the elements that we now had a variable in or have seen that parameter in. And the complexity there, is in standard machine learning you'll see in many models just like standard logistic regression, you see all your parameters like your W in all the examples. And ours, it's a little more complex, because most words you won't see in a specific window and so, you only update the words that you see in that window. And if you assumed all the other ones, you'd just have very, very large, quite sparse updates, and that's not very RAM efficient, great question. So now we have this simple multiplication here and the sum is just is just inner product. So far so simple, and we have our D dimension vector which I mentioned, is two dimensions.

We have the sum over two elements. So, so far so good. Now, really, we would like to get the full gradient here with respect to all XJs for J equals one to three and its simple case, or five D if we have a five word large window. So now the question is, how do we combine this single element here.

Into a vector that eventually gives us all the different gradients for all the xij. And j equals 1 to however long our window is Is anybody follow along this closely? That's right. W transposed delta. Well done. So basically our final derivative and final gradient here for. Our score s with respect to the entire window, is just W transpose times delta. Super simple very fast to implement, I can easily think about how to vectorize this again by concatenating multiple deltas from multiple Windows and so on. And it can be very efficiently, like implemented and derived.

All right, now the error message is delta that arrives at this hidden layer, has of course the same dimensionality as its hidden layer because we're updating all the windows. And now from the previous slides we also know that when we update a window, it really means we now cut up that final gradient here into the different chunks for each specific word in that window, and that's how we update our first large neural network. So let's put all of this together again. So, our full objective function here was this max and I started out with saying let's assume it's larger than zero so you have this identity here. So this is simple indicator fu

Views: 31 178 Likes: 117 Dislikes: 1
100% Likes
0% Dislikes
How to express opposing ideas in English: despite, although, nevertheless,…

Hello. My name is Emma, and in today's video, I am going to teach you a very good and useful trick on how to write essays, how to sound better when you speak, how to do better in…

Views: 662 737 By: Learn English with Emma [engVid]
Jane Steele by Lyndsay Faye

April marks the 200th anniversary of the birth of Charlotte Bronte - she of course the author of one of my favourite books of all time Jane Eyre. You have your Austenite’s your Hufflepuffs…

Views: 1 255 By: ThePoptimist
A glimpse of teenage life in ancient Rome - Ray Laurence

Translator: tom carter Reviewer: Bedirhan Cinar It's March the 17th in A.D. 73. We're visiting ancient Rome to watch the Liberalia, an annual festival that celebrates the liberty…

Views: 6 365 647 By: TED-Ed
Book Club for Kids- The “Write” Stuff Family Research & Literacy Day

>> All right you guys. Thank you for coming out. I'm Kitty Felde and we're going to be taping what's going to end up being a live show which means you're the most important…

Views: 744 By: US National Archives