# GRADED FUNCTION: lstm_forwarddef lstm_forward(x, a0, parameters):\"\"\"Implement the forward propagation of the recurrent neural network using an LSTM-cell described in Figure (4).Arguments:x -- Input data for every time-step, of shape (n_x, m, T_x).a0 -- Initial hidden state, of shape (n_a, m)parameters -- python dictionary containing:Wf -- Weight matrix of the forget gate, numpy array of shape (n_a, n_a + n_x)bf -- Bias of the forget gate, numpy array of shape (n_a, 1)Wi -- Weight matrix of the update gate, numpy array of shape (n_a, n_a + n_x)bi -- Bias of the update gate, numpy array of shape (n_a, 1)56cWc -- Weight matrix of the first \"tanh\", numpy array of shape (n_a, n_a + n_x)bc -- Bias of the first \"tanh\", numpy array of shape (n_a, 1)Wo -- Weight matrix of the output gate, numpy array of shape (n_a, n_a + n_x)bo -- Bias of the output gate, numpy array of shape (n_a, 1)Wy -- Weight matrix relating the hidden-state to the output, numpy array of shape (n_y, n_a)by -- Bias relating the hidden-state to the output, numpy array of shape (n_y, 1)Returns:a -- Hidden states for every time-step, numpy array of shape (n_a, m, T_x)y -- Predictions for every time-step, numpy array of shape (n_y, m, T_x)c -- The value of the cell state, numpy array of shape (n_a, m, T_x)caches -- tuple of values needed for the backward pass, contains (list of all the caches, x)\"\"\"# Initialize \"caches\", which will track the list of all the cachescaches = []### START CODE HERE ###Wy = parameters[\'Wy\'] # saving parameters[\'Wy\'] in a local variable in case students use Wy instead of parameters[\'Wy\']# Retrieve dimensions from shapes of x and parameters[\'Wy\'] (≈2 lines)n_x, m, T_x = x.shapen_y, n_a = Wy.shape56c# initialize \"a\", \"c\" and \"y\" with zeros (≈3 lines)a = np.zeros((n_a, m, T_x))c = np.zeros((n_a, m, T_x))y = np.zeros((n_y, m, T_x))# Initialize a_next and c_next (≈2 lines)a_next = a0.copy()c_next = np.zeros((n_a, m))# loop over all time-stepsfor t in range(T_x):# Get the 2D slice \'xt\' from the 3D input \'x\' at time step \'t\'xt = x[:, :, t]# Update next hidden state, next memory state, compute the prediction, get the cache (≈1 line)a_next, c_next, yt, cache = lstm_cell_forward(xt, a_next, c_next, parameters)# Save the value of the new \"next\" hidden state in a (≈1 line)a[:,:,t] = a_next# Save the value of the next cell state (≈1 line)c[:,:,t]  = c_next# Save the value of the prediction in y (≈1 line)y[:,:,t] = yt# Append the cache into caches (≈1 line)caches.append(cache)### END CODE HERE #### store values needed for backward propagation in cachecaches = (caches, x)return a, y, c, caches

np.random.seed(1)x_tmp = np.random.randn(3,10,7)a0_tmp = np.random.randn(5,10)parameters_tmp = {}parameters_tmp[\'Wf\'] = np.random.randn(5, 5+3)parameters_tmp[\'bf\'] = np.random.randn(5,1)parameters_tmp[\'Wi\'] = np.random.rand56cn(5, 5+3)parameters_tmp[\'bi\']= np.random.randn(5,1)parameters_tmp[\'Wo\'] = np.random.randn(5, 5+3)parameters_tmp[\'bo\'] = np.random.randn(5,1)parameters_tmp[\'Wc\'] = np.random.randn(5, 5+3)parameters_tmp[\'bc\'] = np.random.randn(5,1)parameters_tmp[\'Wy\'] = np.random.randn(2,5)parameters_tmp[\'by\'] = np.random.randn(2,1)a_tmp, y_tmp, c_tmp, caches_tmp = lstm_forward(x_tmp, a0_tmp, parameters_tmp)print(\"a[4][3][6] = \", a_tmp[4][3][6])print(\"a.shape = \", a_tmp.shape)print(\"y[1][4][3] =\", y_tmp[1][4][3])print(\"y.shape = \", y_tmp.shape)print(\"caches[1][1][1] =\\n\", caches_tmp[1][1][1])print(\"c[1][2][1]\", c_tmp[1][2][1])print(\"len(caches) = \", len(caches_tmp))

a[4][3][6] =  0.172117767533a.shape =  (5, 10, 7)y[1][4][3] = 0.95087346185y.shape =  (2, 10, 7)caches[1][1][1] =[ 0.82797464  0.23009474  0.76201118 -0.22232814 -0.20075807  0.186561390.41005165]c[1][2][1] -0.855544916718len(caches) =  2

Congratulations! You have now implemented the forward passes for the basic RNN and the LSTM. When using a deep learning framework, implementing the forward pass is sufficient to build systems that achieve great performance.

The rest of this notebook is optional, and will not be graded.

3 - Backpropagation in recurrent neural networks25ec(OPTIONAL / UNGRADED)

In modern deep learning frameworks, you only have to implement the forward pass, and the framework takes care of the backward pass, so most deep learning engineers do not need to bother with the details of the backward pass. If however you are an expert in calculus and want to see the details of backprop in RNNs, you can work through this optional portion of the notebook.

When in an earlier course you implemented a simple (fully connected) neural network, you used backpropagation to compute the derivatives with respect to the cost to update the parameters. Similarly, in recurrent neural networks you can calculate the derivatives with respect to the cost in order to update the parameters. The backprop equations are quite complicated and we did not derive them in lecture. However, we will briefly present them below.

Note that this notebook does not implement the backward path from the Loss \'J\' backwards to \'a\'. This would have included the dense layer and softmax which are a part of the forward path. This is assumed to be calculated elsewhere and the result passed to rnn_backward in \'da\'. It is further assumed that loss has been adjusted for batch size (m) and division by the number of examples is not required here.

This section is optional and ungraded. It is more difficult and has fewer details regarding its implementation. This section only implements key elements of the full path.

3.1 - Basic RNN backward pass

We will start by computing the backward pass for the basic RNN-cell and then in the following sections, iterate through the cells.

Figure 6 : RNN-cell\'s backward pass. Just like in a fully-connected neural network, the derivative of the cost function \\(J\\) backpropagates through the time steps of the RNN by following the chain-rule from calculus. Internal to the cell, the chain-rule is also used to calculate \\((\\frac{\\partial J}{\\partial W_{ax}},\\frac{\\partial J}{\\partial W_{aa}},\\frac{\\partial J}{\\partial b})\\) to update the parameters \\((W_{ax}, W_{aa}, b_a)\\). The operation can utilize the cached results from the forward path.

Recall from lecture, the shorthand for the partial derivative of cost relative to a variable is dVariable. For example, \\(\\frac{\\partial J}{\\partial W_{ax}}\\) is \\(dW_{ax}\\). This will be used throughout the remaining sections.

Figure 7 : This implementation of rnn_cell_backward does not include the output dense layer and softmax which are included in rnn_cell_forward.

\\(da_{next}\\) is \\(\\frac{\\partial{J}}{\\partial a^{\\langle t \\rangle}}\\) and includes loss from previous stages and current stage output logic. The addition shown in green will be part of your implementation of rnn_backward.

Equations

To compute the rnn_cell_backward you can utilize the following equations. It is a good exercise to derive them by hand. Here, \\(*\\) denotes element-wise multiplication while the absence of a symbol indicates matrix multiplication.

\\[\\begin{align}\\displaystyle a^{\\langle t \\rangle} &= \\tanh(W_{ax} x^{\\langle t \\rangle} + W_{aa} a^{\\langle t-1 \\rangle} + b_{a})\\tag{-} \\\\[8pt]\\displaystyle \\frac{\\partial \\tanh(x)} {\\partial x} &= 1 - \\tanh^2(x) \\tag{-} \\\\[8pt]\\displaystyle {dW_{ax}} &= (da_{next} * ( 1-\\tanh^2(W_{ax}x^{\\langle t \\rangle}+W_{aa} a^{\\langle t-1 \\rangle} + b_{a}) )) x^{\\langle t \\rangle T}\\tag{1} \\\\[8pt]\\displaystyle dW_{aa} &= (da_{next} * ( 1-\\tanh^2(W_{ax}x^{\\langle t \\rangle}+W_{aa} a^{\\langle t-1 \\rangle} + b_{a}) )) a^{\\langle t-1 \\rangle T}\\tag{2} \\\\[8pt]\\displaystyle db_a& = \\sum_{batch}( da_{next} * ( 1-\\tanh^2(W_{ax}x^{\\langle t \\rangle}+W_{aa} a^{\\langle t-1 \\rangle} + b_{a}) ))\\tag{3} \\\\[8pt]\\displaystyle dx^{\\langle t \\rangle} &= { W_{ax}}^T (da_{next} * ( 1-\\tanh^2(W_{ax}x^{\\langle t \\rangle}+W_{aa} a^{\\langle t-1 \\rangle} + b_{a}) ))\\tag{4} \\\\[8pt]\\displaystyle da_{prev} &= { W_{aa}}^T(da_{next} * ( 1-\\tanh^2(W_{ax}x^{\\langle t \\rangle}+W_{aa} a^{\\langle t-1 \\rangle} + b_{a}) ))\\tag{5}\\end{align}\\]

Implementing rnn_cell_backward

The results can be computed directly by implementing the equations above. However, the above can optionally be simplified by computing \'dz\' and utlilizing the chain rule.
This can be further simplified by noting that \\(\\tanh(W_{ax}x^{\\langle t \\rangle}+W_{aa} a^{\\langle t-1 \\rangle} + b_{a})\\) was computed and saved in the forward pass.

To calculate dba, the \'batch\' above is a sum across all \'m\' examples (axis= 1). Note that you should use the keepdims = True option.

It may be worthwhile to review Course 1 Derivatives with a computational graph through Backpropagation Intuition, which decompose the calculation into steps using the chain rule.
Matrix vector derivatives are described here, though the equations above incorporate the required transformations.

Note rnn_cell_backward does not include the calculation of loss from \\(y \\langle t \\rangle\\), this is incorporated into the incoming da_next. This is a slight mismatch with rnn_cell_forward which includes a dense layer and softmax.

Note: in the code:
\\(\\displaystyle dx^{\\langle t \\rangle}\\) is represented by dxt,
\\(\\displaystyle d W_{ax}\\) is represented by dWax,
\\(\\displaystyle da_{prev}\\) is represented by da_prev,
\\(\\displaystyle dW_{aa}\\) is represented by dWaa,
\\(\\displaystyle db_{a}\\) is represented by dba,
dz is not derived above but can optionally be derived by students to simplify the repeated calculations.

def rnn_cell_backward(da_next, cache):\"\"\"Implements the backward pass for the RNN-cell (single time-step).Arguments:da_next -- Gradient of loss with respect to next hidden statecache -- python dictionary containing useful values (output of rnn_cell_forward())Returns:gradients -- python dictionary containing:dx -- Gradients of input data, of shape (n_x, m)da_prev -- Gradients of previous hidden state, of shape (n_a, m)dWax -- Gradients of input-to-hidden weights, of shape (n_a, n_x)dWaa -- Gradients of hidden-to-hidden weights, of shape (n_a, n_a)dba -- Gradients of bias vector, of shape (n_a, 1)\"\"\"# Retrieve values from cache(a_next, a_prev, xt, parameters) = cache# Retrieve values from parametersWax = parameters[\"Wax\"]Waa = parameters[\"Waa\"]Wya = parameters[\"Wya\"]ba = parameters[\"ba\"]by = parameters[\"by\"]### START CODE HERE #### compute the gradient of the loss with respect to z (optional) (≈1 line)dz = 1 - a_next ** 2# compute the gradient of the loss with respect to Wax (≈2 lines)dxt = np.dot(Wax.T, da_next * dz)dWax = np.dot(da_next * dz, xt.T)# compute the gradient with respect to Waa (≈2 lines)da_prev = np.dot(Waa.T, da_next * dz)dWaa = np.dot(da_next * dz, a_prev.T)# compute the gradient with respect to b (≈1 line)dba = np.sum(da_next * dz, axis=1, keepdims=True)### END CODE HERE #### Store the gradients in a python dictionarygradients = {\"dxt\": dxt, \"da_prev\": da_prev, \"dWax\": dWax, \"dWaa\": dWaa, \"dba\": dba}return gradients

np.random.seed(1)xt_tmp = np.random.randn(3,10)a_prev_tmp = np.random.randn(5,10)parameters_tmp = {}parameters_tmp[\'Wax\'] = np.random.randn(5,3)parameters_tmp[\'Waa\'] = np.random.randn(5,5)parameters_tmp[\'Wya\'] = np.random.randn(2,5)parameters_tmp[\'ba\'] = np.random.randn(5,1)parameters_tmp[\'by\'] = np.random.randn(2,1)a_next_tmp, yt_tmp, cache_tmp = rnn_cell_forward(xt_tmp, a_prev_tmp, parameters_tmp)da_next_tmp = np.random.randn(5,10)gradients_tmp = rnn_cell_backward(da_next_tmp, cache_tmp)print(\"gradients[\\\"dxt\\\"][1][2] =\", gradients_tmp[\"dxt\"][1][2])print(\"gradients[\\\"dxt\\\"].shape =\", gradients_tmp[\"dxt\"].shape)print(\"gradients[\\\"da_prev\\\"][2][3] =\", gradients_tmp[\"da_prev\"][2][3])print(\"gradients[\\\"da_prev\\\"].shape =\", gradients_tmp[\"da_prev\"].shape)print(\"gradients[\\\"dWax\\\"][3][1] =\", gradien15b0ts_tmp[\"dWax\"][3][1])print(\"gradients[\\\"dWax\\\"].shape =\", gradients_tmp[\"dWax\"].shape)print(\"gradients[\\\"dWaa\\\"][1][2] =\", gradients_tmp[\"dWaa\"][1][2])print(\"gradients[\\\"dWaa\\\"].shape =\", gradients_tmp[\"dWaa\"].shape)print(\"gradients[\\\"dba\\\"][4] =\", gradients_tmp[\"dba\"][4])print(\"gradients[\\\"dba\\\"].shape =\", gradients_tmp[\"dba\"].shape)

gradients[\"dxt\"][1][2] = -1.3872130506gradients[\"dxt\"].shape = (3, 10)gradients[\"da_prev\"][2][3] = -0.152399493774gradients[\"da_prev\"].shape = (5, 10)gradients[\"dWax\"][3][1] = 0.410772824935gradients[\"dWax\"].shape = (5, 3)gradients[\"dWaa\"][1][2] = 1.15034506685gradients[\"dWaa\"].shape = (5, 5)gradients[\"dba\"][4] = [ 0.20023491]gradients[\"dba\"].shape = (5, 1)

Backward pass through the RNN

Computing the gradients of the cost with respect to \\(a^{\\langle t \\rangle}\\) at every time-step \\(t\\) is useful because it is what helps the gradient backpropagate to the previous RNN-cell. To do so, you need to iterate through all the time steps starting at the end, and at each step, you increment the overall \\(db_a\\), \\(dW_{aa}\\), \\(dW_{ax}\\) and you store \\(dx\\).

Instructions:

Implement the

rnn_backward

function. Initialize the return variables with zeros first and then loop through all the time steps while calling the

rnn_cell_backward

at each time timestep, update the other variables accordingly.

• Note that this notebook does not implement the backward path from the Loss \'J\' backwards to \'a\'.This would have included the dense layer and softmax which are a part of the forward path.
• This is assumed to be calculated elsewhere and the result passed to rnn_backward in \'da\'.
• You must combine this with the loss from the previous stages when calling rnn_cell_backward (see figure 7 above).

Therefore, division by the number of examples is not required here.

def rnn_backward(da, caches):\"\"\"Implement the backward pass for a RNN over an entire sequence of input data.Arguments:da -- Upstream gradients of all hidden states, of shape (n_a, m, T_x)caches -- tuple containing information from the forward pass (rnn_forward)Returns:gradients -- python dictionary containing:dx -- Gradient w.r.t. the input data, numpy-array of shape (n_x, m, T_x)da0 -- Gradient w.r.t the initial hidden state, numpy-array of shape (n_a, m)dWax -- Gradient w.r.t the input\'s weight matrix, numpy-array of shape (n_a, n_x)dWaa -- Gradient w.r.t the hidden state\'s weight matrix, numpy-arrayof shape (n_a, n_a)dba -- Gradient w.r.t the bias, of shape (n_a, 1)\"\"\"### START CODE HERE #### Retrieve values from the first cache (t=1) of caches (≈2 lines)(caches, x) = caches(a1, a0, x1, parameters) = caches[0]# Retrieve dimensions from da\'s and x1\'s shapes (≈2 lines)n_a, m, T_x = da.shapen_x, m = x1.shape# initialize the gradients with the right sizes (≈6 lines)dx = np.zeros((n_x, m, T_x))dWax = np.zeros((n_a, n_x))dWaa = np.zeros((n_a, n_a))dba = np.zeros((n_a, 1))da0 = np.zeros((n_a, m))da_prevt = np.zeros((n_a, m))# Loop through all the time stepsfor t in reversed(range(T_x)):# Compute gradients at time step t.# Remember to sum gradients from the output path (da) and the previous timesteps (da_prevt) (≈1 line)gradients = rnn_cell_backward(da[:, :, t] + da_prevt, caches[t])# Retrieve derivatives from gradients (≈ 1 line)dxt, da_prevt, dWaxt, dWaat, dbat = gradients[\"dxt\"], gradients[\"da_prev\"], gradients[\"dWax\"], gradients[\"dWaa\"], gradients[\"dba\"]# Increment global derivatives w.r.t parameters by adding their derivative at time-step t (≈4 lines)dx[:, :, t] = dxtdWax += dWaxtdWaa += dWaatdba += dbat# Set da0 to the gradient of a which has been backpropagated through all time-steps (≈1 line)da0 = da_prevt### END CODE HERE #### Store the gradients in a python dictionarygradients = {\"dx\": dx, \"da0\": da0, \"dWax\": dWax, \"dWaa\": dWaa,\"dba\": dba}return gradients

np.random.seed(1)x_tmp = np.random.randn(3,10,4)a0_tmp = np.random.randn(5,10)parameters_tmp = {}parameters_tmp[\'Wax\'] = np.random.randn(5,3)parameters_tmp[\'Waa\'] = np.random.randn(5,5)parameters_tmp[\'Wya\'] = np.random.randn(2,5)parameters_tmp[\'ba\'] = np.random.randn(5,1)parameters_tmp[\'by\'] = np.random.randn(2,1)a_tmp, y_tmp, caches_tmp = rnn_forward(x_tmp, a0_tmp, parameters_tmp)da_tmp = np.random.randn(5, 10, 4)gradients_tmp = rnn_backward(da_tmp, caches_tmp)print(\"gradients[\\\"dx\\\"][1][2] =\", gradients_tmp[\"dx\"][1][2])print(\"gradients[\\\"dx\\\"].shape =\", gradients_tmp[\"dx\"].shape)print(\"gradients[\\\"da0\\\"][2][3] =\", gradients_tmp[\"da0\"][2][3])print(\"gradients[\\\"da0\\\"].sad0hape =\", gradients_tmp[\"da0\"].shape)print(\"gradients[\\\"dWax\\\"][3][1] =\", gradients_tmp[\"dWax\"][3][1])print(\"gradients[\\\"dWax\\\"].shape =\", gradients_tmp[\"dWax\"].shape)print(\"gradients[\\\"dWaa\\\"][1][2] =\", gradients_tmp[\"dWaa\"][1][2])print(\"gradients[\\\"dWaa\\\"].shape =\", gradients_tmp[\"dWaa\"].shape)print(\"gradients[\\\"dba\\\"][4] =\", gradients_tmp[\"dba\"][4])print(\"gradients[\\\"dba\\\"].shape =\", gradients_tmp[\"dba\"].shape)

gradients[\"dx\"][1][2] = [-2.07101689 -0.59255627  0.02466855  0.01483317]gradients[\"dx\"].shape = (3, 10, 4)gradients[\"da0\"][2][3] = -0.314942375127gradients[\"da0\"].shape = (5, 10)gradients[\"dWax\"][3][1] = 11.2641044965gradients[\"dWax\"].shape = (5, 3)gradients[\"dWaa\"][1][2] = 2.30333312658gradients[\"dWaa\"].shape = (5, 5)gradients[\"dba\"][4] = [-0.74747722]gradients[\"dba\"].shape = (5, 1)

3.2 - LSTM backward pass

3.2.1 One Step backward

The LSTM backward pass is slightly more complicated than the forward pass.

Figure 8: lstm_cell_backward. Note the output functions, while part of the lstm_cell_forward, are not included in lstm_cell_backward

The equations for the LSTM backward pass are provided below. (If you enjoy calculus exercises feel free to try deriving these from scratch yourself.)

3.2.2 gate derivatives

Note the location of the gate derivatives (\\(\\gamma\\)..) between the dense layer and the activation function (see graphic above). This is convenient for computing parameter derivatives in the next step.

\\[\\begin{align}d\\gamma_o^{\\langle t \\rangle} &= da_{next}*\\tanh(c_{next}) * \\Gamma_o^{\\langle t \\rangle}*\\left(1-\\Gamma_o^{\\langle t \\rangle}\\right)\\tag{7} \\\\[8pt]dp\\widetilde{c}^{\\langle t \\rangle} &= \\left(dc_{next}*\\Gamma_u^{\\langle t \\rangle}+ \\Gamma_o^{\\langle t \\rangle}* (1-\\tanh^2(c_{next})) * \\Gamma_u^{\\langle t \\rangle} * da_{next} \\right) * \\left(1-\\left(\\widetilde c^{\\langle t \\rangle}\\right)^2\\right) \\tag{8} \\\\[8pt]d\\gamma_u^{\\langle t \\rangle} &= \\left(dc_{next}*\\widetilde{c}^{\\langle t \\rangle} + \\Gamma_o^{\\langle t \\rangle}* (1-\\tanh^2(c_{next})) * \\widetilde{c}^{\\langle t \\rangle} * da_{next}\\right)*\\Gamma_u^{\\langle t \\rangle}*\\left(1-\\Gamma_u^{\\langle t \\rangle}\\right)\\tag{9} \\\\[8pt]d\\gamma_f^{\\langle t \\rangle} &= \\left(dc_{next}* c_{prev} + \\Gamma_o^{\\langle t \\rangle} * (1-\\tanh^2(c_{next})) * c_{prev} * da_{next}\\right)*\\Gamma_f^{\\langle t \\rangle}*\\left(1-\\Gamma_f^{\\langle t \\rangle}\\right56c)\\tag{10}\\end{align}\\]

3.2.3 parameter derivatives

\\[dW_f = d\\gamma_f^{\\langle t \\rangle} \\begin{bmatrix} a_{prev} \\\\ x_t\\end{bmatrix}^T \\tag{11}\\]

\\[dW_u = d\\gamma_u^{\\langle t \\rangle} \\begin{bmatrix} a_{prev} \\\\ x_t\\end{bmatrix}^T \\tag{12}\\]

\\[dW_c = dp\\widetilde c^{\\langle t \\rangle} \\begin{bmatrix} a_{prev} \\\\ x_t\\end{bmatrix}^T \\tag{13}\\]

\\[dW_o = d\\gamma_o^{\\langle t \\rangle} \\begin{bmatrix} a_{prev} \\\\ x_t\\end{bmatrix}^T \\tag{14}\\]

To calculate \\(db_f, db_u, db_c, db_o\\) you just need to sum across all \'m\' examples (axis= 1) on \\(d\\gamma_f^{\\langle t \\rangle}, d\\gamma_u^{\\langle t \\rangle}, dp\\widetilde c^{\\langle t \\rangle}, d\\gamma_o^{\\langle t \\rangle}\\) respectively. Note that you should have the

keepdims = True

option.

\\[\\displaystyle db_f = \\sum_{batch}d\\gamma_f^{\\langle t \\rangle}\\tag{15}\\]

\\[\\displaystyle db_u = \\sum_{batch}d\\gamma_u^{\\langle t \\rangle}\\tag{16}\\]

\\[\\displaystyle db_c = \\sum_{batch}d\\gamma_cad8^{\\langle t \\rangle}\\tag{17}\\]

\\[\\displaystyle db_o = \\sum_{batch}d\\gamma_o^{\\langle t \\rangle}\\tag{18}\\]

Finally, you will compute the derivative with respect to the previous hidden state, previous memory state, and input.

\\[da_{prev} = W_f^T d\\gamma_f^{\\langle t \\rangle} + W_u^T d\\gamma_u^{\\langle t \\rangle}+ W_c^T dp\\widetilde c^{\\langle t \\rangle} + W_o^T d\\gamma_o^{\\langle t \\rangle} \\tag{19}\\]

Here, to account for concatenation, the weights for equations 19 are the first n_a, (i.e. \\(W_f = W_f[:,:n_a]\\) etc...)

\\[dc_{prev} = dc_{next}*\\Gamma_f^{\\langle t \\rangle} + \\Gamma_o^{\\langle t \\rangle} * (1- \\tanh^2(c_{next}))*\\Gamma_f^{\\langle t \\rangle}*da_{next} \\tag{20}\\]

\\[dx^{\\langle t \\rangle} = W_f^T d\\gamma_f^{\\langle t \\rangle} + W_u^T d\\gamma_u^{\\langle t \\rangle}+ W_c^T dp\\widetilde c^{\\langle t \\rangle} + W_o^T d\\gamma_o^{\\langle t \\rangle}\\tag{21}\\]

where the weights for equation 21 are from n_a to the end, (i.e. \\(W_f = W_f[:,n_a:]\\) etc...)

Exercise: Implement

lstm_cell_backward

by implementing equations \\(7-21\\) below.

Note: In the code:

\\(d\\gamma_o^{\\langle t \\rangle}\\) is represented by

dot

,
\\(dp\\widetilde{c}^{\\langle t \\rangle}\\) is represented by

dcct

,
\\(d\\gamma_u^{\\langle t \\rangle}\\) is represented by

dit

,
\\(d\\gamma_f^{\\langle t \\rangle}\\) is represented by

dft

def lstm_cell_backward(da_next, dc_next, cache):\"\"\"Implement the backward pass for the LSTM-cell (single time-step).Arguments:da_next -- Gradients of next hidden state, of shape (n_a, m)dc_next -- Gradients of next cell state, of shape (n_a, m)cache -- cache storing information from the forward passReturns:gradients -- python dictionary containing:dxt -- Gradient of input data at time-step t, of shape (n_x, m)da_prev -- Gradient w.r.t. the previous hidden state, numpy array of shape (n_a, m)dc_prev -- Gradient w.r.t. the previous memory state, of shape (n_a, m, T_x)dWf -- Gradient w.r.t. the weight matrix of the forget gate, numpy array of shape (n_a, n_a + n_x)3ff8dWi -- Gradient w.r.t. the weight matrix of the update gate, numpy array of shape (n_a, n_a + n_x)dWc -- Gradient w.r.t. the weight matrix of the memory gate, numpy array of shape (n_a, n_a + n_x)dWo -- Gradient w.r.t. the weight matrix of the output gate, numpy array of shape (n_a, n_a + n_x)dbf -- Gradient w.r.t. biases of the forget gate, of shape (n_a, 1)dbi -- Gradient w.r.t. biases of the update gate, of shape (n_a, 1)dbc -- Gradient w.r.t. biases of the memory gate, of shape (n_a, 1)dbo -- Gradient w.r.t. biases of the output gate, of shape (n_a, 1)\"\"\"# Retrieve information from \"cache\"(a_next, c_next, a_prev, c_prev, ft, it, cct, ot, xt, parameters) = cache### START CODE HERE #### Retrieve dimensions from xt\'s and a_next\'s shape (≈2 lines)n_x, m = xt.shapen_a, m = a_next.shape# Compute gates related derivatives, you can find their values can be found by looking carefully at equations (7) to (10) (≈4 lines)dot = da_next * np.tanh(c_next) * ot * (1 - ot)dcct = (dc_next * it + ot * (1 - np.square(np.tanh(c_next))) * it * da_next) * (1 - cct ** 2)dit = (dc_next * cct + ot * (1 - np.square(np.tanh(c_next))) * cct * da_next) * it * (1 - it)dft = (dc_next * c_prev + ot * (1 - np.square(np.tanh(c_next))) * c_prev * da_next) * ft * (1 - ft)# Compute parameters related derivatives. Use equations (11)-(18) (≈8 lines)dWf = np.dot(dft, np.concatenate((a_prev, xt)).T)dWi = np.dot(dit, np.concatenate((a_prev, xt)).T)dWc = np.dot(dcct, np.concatenate((a_prev, xt)).T)dWo = np.dot(dot, np.concatenate((a_prev, xt)).T)dbf = np.sum(dft, axis=1, keepdims=True)dbi = np.sum(dit, axis=1, keepdims=True)dbc = np.sum(dcct, axis=1, keepdims=True)dbo = np.sum(dot, axis=1, keepdims=True)# Compute derivatives w.r.t previous hidden state, previous memory state and input. Use equations (19)-(21). (≈3 lines)da_prev = np.dot(parameters[\'Wf\'][:, :n_a].T, dft) + np.dot(parameters[\'Wi\'][:, :n_a].T, dit) + np.dot(parameters[\'Wc\'][:, :n_a].T, dcct) + np.dot(parameters[\'Wo\'][:, :n_a].T, dot)dc_prev = dc_next * ft + ot * (1 - np.tanh(c_next) ** 2) * ft * da_nextdxt = np.dot(parameters[\'Wf\'][:, n_a:].T, dft) + np.dot(parameters[\'Wi\'][:, n_a:].T, dit) + np.dot(parameters[\'Wc\'][:, n_a:].T, dcct) + np.dot(parameters[\'Wo\'][:, n_a:].T, dot)### END CODE HERE #### Save gradients in dictionarygradients = {\"dxt\": dxt, \"da_prev\": da_prev, \"dc_prev\": dc_prev, \"dWf\": dWf,\"dbf\": dbf, \"dWi\": dWi,\"dbi\": dbi,\"dWc\": dWc,\"dbc\": dbc, \"dWo\": dWo,\"dbo\": dbo}return gradients

np.random.seed(1)xt_tmp = np.random.randn(3,10)a_prev_tmp = np.random.randn(5,10)c_prev_tmp = np.random.randn(5,10)parameters_tmp = {}parameters_tmp[\'Wf\'] = np.random.randn(5, 5+3)parameters_tmp[\'bf\'] = np.random.randn(5,1)parameters_tmp[\'Wi\'] = np.random.randn(5, 5+3)parameters_tmp[\'bi\'] = np.random.randn(5,1)parameters_tmp[\'Wo\'] = np.random.randn(5, 5+3)parameters_tmp[\'bo\'] = np.random.randn(5,1)parameters_tmp[\'Wc\'] = np.random.randn(5, 5+3)parameters_tmp[\'bc\'] = np.random.randn(5,1)parameters_tmp[\'Wy\'] = np.random.randn(2,5)parameters_tmp[\'by\'] = np.random.randn(2,1)a_next_tmp, c_next_tmp, yt_tmp, cache_tmp = lstm_cell_forward(xt_tmp, a_prev_tmp, c_prev_tmp, parameters_tmp)da_next_tmp = np.random.randn(5,10)dc_next_tmp = np.random.randn(5,10)gradients_tmp = lstm_cell_backward(da_next_tmp, dc_next_tmp, cache_tmp)print(\"gradients[\\\"dxt\\\"][1][2] =\", gradients_tmp[\"dxt\"][1][2])print(\"gradients[\\\"dxt\\\"].shape =\", gradients_tmp[\"dxt\"].shape)print(\"gradients[\\\"da_prev\\\"][2][3] =\", gradients_tmp[\"da_prev\"][2][3])print(\"gradients[\\\"da_prev\\\"].shape =\", gradients_tmp[\"da_prev\"].shape)print(\"gradients[\\\"dc_prev\\\"][2][3] =\", gradients_tmp[\"dc_prev\"][2][3])print(\"gradients[\\\"dc_prev\\\"].shape =\", gradients_tmp[\"dc_prev\"].shape)print(\"gradients[\\\"dWf\\\"][3][1] =\", gradients_tmp[\"dWf\"][3][1])print(\"gradients[\\\"dWf\\\"].shape =\", gradients_tmp[\"dWf\"].shape)print(\"gradients[\\\"dWi\\\"][1][2] =\", gradients_tmp[\"dWi\"][1][2])print(\"gradients[\\\"dWi\\\"].shape =\", gradients_tmp[\"dWi\"].shape)print(\"gradients[\\\"dWc\\\"][3][1] =\", gradients_tmp[\"dWc\"][3][1])print(\"gradients[\\\"dWc\\\"].shape =\", gradients_tmp[\"dWc\"].shape)print(\"gradients[\\\"dWo\\\"][1][2] =\", gradients_tmp[\"dWo\"][1][2])print(\"gradients[\\\"dWo\\\"].shape =\", gradients_tmp[\"dWo\"].shape)print(\"gradients[\\\"dbf\\\"][4] =\", gradients_tmp[\"dbf\"][4])print(\"gradients[\\\"dbf\\\"].shape =\", gradients_tmp[\"dbf\"].shape)print(\"gradients[\\\"dbi\\\"][4] =\", gradients_tmp[\"dbi\"][4])print(\"gradients[\\\"dbi\\\"].shape =\", gradients_tmp[\"dbi\"].shape)print(\"gradients[\\\"dbc\\\"][4] =\", gradients_tmp[\"dbc\"][4])print(\"gradients[\\\"dbc\\\"].shape =\", gradients_tmp[\"dbc\"].shape)print(\"gradients[\\\"dbo\\\"][4] =\", gradients_tmp[\"dbo\"][4])print(\"gradients[\\\"dbo\\\"].shape =\", gradients_tmp[\"dbo\"].shape)

gradients[\"dxt\"][1][2] = 3.23055911511gradients[\"dxt\"].shape = (3, 10)gradients[\"da_prev\"][2][3] = -0.0639621419711gradients[\"da_prev\"].shape = (5, 10)gradients[\"dc_prev\"][2][3] = 0.797522038797gradients[\"dc_prev\"].shape = (5, 10)gradients[\"dWf\"][3][1] = -0.147954838164gradients[\"dWf\"].shape = (5, 8)gradients[\"dWi\"][1][2] = 1.05749805523gradients[\"dWi\"].shape = (5, 8)gradients[\"dWc\"][3][1] = 2.30456216369gradients[\"dWc\"].shape = (5, 8)gradients[\"dWo\"][1][2] = 0.331311595289gradients[\"dWo\"].shape = (5, 8)gradients[\"dbf\"][4] = [ 0.18864637]gradients[\"dbf\"].shape = (5, 1)gradients[\"dbi\"][4] = [-0.40142491]gradients[\"dbi\"].shape = (5, 1)gradients[\"dbc\"][4] = [ 0.25587763]gradients[\"dbc\"].shape = (5, 1)gradients[\"dbo\"][4] = [ 0.13893342]gradients[\"dbo\"].shape = (5, 1)

3.3 Backward pass through the LSTM RNN

This part is very similar to the

rnn_backward

function you implemented above. You will first create variables of the same dimension as your return variables. You will then iterate over all the time steps starting from the end and call the one step function you implemented for LSTM at each iteration. You will then update the parameters by summing them individually. Finally return a dictionary with the new gradients.

Instructions: Implement the

lstm_backward

function. Create a for loop starting from \\(T_x\\) and going backward. For each step call

lstm_cell_backward

and update the your old gradients by adding the new gradients to them. Note that

dxt

is not updated but is stored.

def lstm_backward(da, caches):\"\"\"Implement the backward pass for the RNN with LSTM-cell (over a whole sequence).Arguments:da -- Gradients w.r.t the hidden states, numpy-array of shape (n_a, m, T_x)caches -- cache storing information from the forward pass (lstm_forward)Returns:gradients -- python dictionary containing:dx -- Gradient of inputs, of shape (n_x, m, T_x)da0 -- Gradient w.r.t. the previous hidden state, numpy array of shape (n_a, m)dWf -- Gradient w.r.t. the weight matrix of the forget gate, numpy array of shape (n_a, n_a + n_x)dWi -- Gradient w.r.t. the weight matrix of the update gate, numpy array of shape (n_a, n_a + n_x)dWc -- Gradient w.r.t. the weight matrix of the memory gate, numpy array of shape (n_a, n_a + n_x)dWo -- Gradient w.r.t. the weight matrix of the save gate, numpy array of shape (n_a, n_a + n_x)dbf -- Gradient w.r.t. biases of the forget gate, of shape (n_a, 1)dbi -- Gradient w.r.t. biases of the update gate, of shape (n_a, 1)dbc -- Gradient w.r.t. biases of the memory gate, of shape (n_a, 1)dbo -- Gradient w.r.t. biases of the save gate, of shape (n_a, 1)\"\"\"# Retrieve values from the first cache (t=1) of caches.(caches, x) = caches(a1, c1, a0, c0, f1, i1, cc1, o1, x1, parameters) = caches[0]### START CODE HERE #### Retrieve dimensions from da\'s and x1\'s shapes (≈2 lines)n_a, m, T_x = da.shapen_x, m = x1.shape# initialize the gradients with the right sizes (≈12 lines)dx = np.zeros((n_x, m, T_x))da0 = np.zeros((n_a, m))da_prevt = np.zeros((n_a, m))dc_prevt = np.zeros((n_a, m))dWf = np.zeros((n_a, n_a + n_x))dWi = np.zeros((n_a, n_a + n_x))dWc = np.zeros((n_a, n_a + n_x))dWo = np.zeros((n_a, n_a + n_x))dbf = np.zeros((n_a, 1))dbi = np.zeros((n_a, 1))dbc = np.zeros((n_a, 1))dbo = np.zeros((n_a, 1))# loop back over the whole sequencefor t in reversed(range(T_x)):# Compute all gradients using lstm_cell_backwardgradients = lstm_cell_backward(da[:, :, t] + da_prevt, dc_prevt, caches[t])# Store or add the gradient to the parameters\' previous step\'s gradientda_prevt = gradients[\"da_prev\"]dc_prevt = gradients[\"dc_prev\"]dx[:,:,t] = gradients[\"dxt\"]dWf += gradients[\"dWf\"]dWi += gradients[\"dWi\"]dWc += gradients[\"dWc\"]dWo += gradients[\"dWo\"]dbf += gradients[\"dbf\"]dbi += gradients[\"dbi\"]dbc += gradients[\"dbc\"]dbo += gradients[\"dbo\"]# Set the first activation\'s gradient to the backpropagated gradient da_prev.da0 = da_prevt### END CODE HERE #### Store the gradients in a python dictionarygradients = {\"dx\": dx, \"da0\": da0, \"dWf\": dWf,\"dbf\": dbf, \"dWi\": dWi,\"dbi\": dbi,\"dWc\": dWc,\"dbc\": dbc, \"dWo\": dWo,\"dbo\": dbo}return gradients

np.random.seed(1)x_tmp = np.random.randn(3,10,7)a0_tmp = np.random.randn(5,10)parameters_tmp = {}parameters_tmp[\'Wf\'] = np.random.randn(5, 5+3)parameters_tmp[\'bf\'] = np.random.randn(5,1)parameters_tmp[\'Wi\'] = np.random.randn(5, 5+3)parameters_tmp[\'bi\'] = np.random.randn(5,1)parameters_tmp[\'Wo\'] = np.random.randn(5, 5+3)parameters_tmp[\'bo\'] = np.random.randn(5,1)parameters_tmp[\'Wc\'] = np.random.randn(5, 5+3)parameters_tmp[\'bc\'] = np.random.randn(5,1)parameters_tmp[\'Wy\'] = np.zeros((2,5))       # unused, but needed for lstm_forwardparameters_tmp[\'by\'] = np.zeros((2,1))       # unused, but needed for lstm_forwarda_tmp, y_tmp, c_tmp, caches_tmp = lstm_forward(x_tmp, a0_tmp, parameters_tmp)da_tmp = np.random.randn(5, 10, 4)gradients_tmp = lstm_backward(da_tmp, caches_tmp)print(\"gradients[\\\"dx\\\"][1][2] =\", gradients_tmp[\"dx\"][1][2])print(\"gradients[\\\"dx\\\"].shape =\", gradients_tmp[\"dx\"].shape)print(\"gradients[\\\"da0\\\"][2][3] =\", gradients_tmp[\"da0\"][2][3])print(\"gradients[\\\"da0\\\"].shape =\", gradients_tmp[\"da0\"].shape)print(\"gradients[\\\"dWf\\\"][3][1] =\", gradients_tmp[\"dWf\"][3][1])print(\"gradients[\\\"dWf\\\"].shape =\", gradients_tmp[\"dWf\"].shape)print(\"gradients[\\\"dWi\\\"][1][2] =\", gradients_tmp[\"dWi\"][1][2])print(\"gradients[\\\"dWi\\\"].shape =\", gradients_tmp[\"dWi\"].shape)print(\"gradients[\\\"dWc\\\"][3][1] =\", gradients_tmp[\"dWc\"][3][1])print(\"gradients[\\\"dWc\\\"].shape =\", gradients_tmp[\"dWc\"].shape)print(\"gradients[\\\"dWo\\\"][1][2] =\", gradients_tmp[\"dWo\"][1][2])print(\"gradients[\\\"dWo\\\"].shape =\", gradients_tmp[\"dWo\"].shape)print(\"gradients[\\\"dbf\\\"][4] =\", gradients_tmp[\"dbf\"][4])print(\"gradients[\\\"dbf\\\"].shape =\", gradients_tmp[\"dbf\"].shape)print(\"gradients[\\\"dbi\\\"][4] =\", gradients_tmp[\"dbi\"][4])print(\"gradients[\\\"dbi\\\"].shape =\", gradients_tmp[\"dbi\"].shape)print(\"gradients[\\\"dbc\\\"][4] =\", gradients_tmp[\"dbc\"][4])print(\"gradients[\\\"dbc\\\"].shape =\", gradients_tmp[\"dbc\"].shape)print(\"gradients[\\\"dbo\\\"][4] =\", gradients_tmp[\"dbo\"][4])print(\"gradients[\\\"dbo\\\"].shape =\", gradients_tmp[\"dbo\"].shape)

gradients[\"dx\"][1][2] = [ 0.00218254  0.28205375 -0.48292508 -0.43281115]gradients[\"dx\"].shape = (3, 10, 4)gradients[\"da0\"][2][3] = 0.312770310257gradients[\"da0\"].shape = (5, 10)gradients[\"dWf\"][3][1] = -0.0809802310938gradients[\"dWf\"].shape = (5, 8)gradients[\"dWi\"][1][2] = 0.40512433093gradients[\"dWi\"].shape = (5, 8)gradients[\"dWc\"][3][1] = -0.0793746735512gradients[\"dWc\"].shape = (5, 8)gradients[\"dWo\"][1][2] = 0.038948775763gradients[\"dWo\"].shape = (5, 8)gradients[\"dbf\"][4] = [-0.15745657]gradients[\"dbf\"].shape = (5, 1)gradients[\"dbi\"][4] = [-0.50848333]gradients[\"dbi\"].shape = (5, 1)gradients[\"dbc\"][4] = [-0.42510818]gradients[\"dbc\"].shape = (5, 1)gradients[\"dbo\"][4] = [-0.17958196]gradients[\"dbo\"].shape = (5, 1)

Congratulations !

Congratulations on completing this assignment. You now understand how recurrent neural networks work!

Let\'s go on to the next exercise, where you\'ll use an RNN to build a character-level language model.