RNN

Modelling Sequence Data §

Some task where the input is of sequential nature:

• speech recognition
• machine translation
• language modeling
• sentiment analysis
• video data

How should we choose our model? Again, FCN are universal, but the size of the model can grow too much.

If our sequence is long $x_1,\dots ,x_T$ the FCN would become prohibitively large: Idea input one $x_i$ at a time, re-use the same weights! With this idea we don’t care about the sequence length $T$.

To memorize prviously information, we employ a recurrent mechanism.

$$h^t=f_W(h^{t-1},x^t),h^{-1}\text{ given }$$

we can unfold this formula $t-1$ times.

A simple RNN model:

$$h^t=\tanh (W_{hh}{h}^{t-1}+W_{xh}{x}^t),o^t=softmax(W_{ho}{h}^t)$$

Training an RNN §

It’s like we arte backpropagating through time. Just unfold the network! The loss is just the sum:

$$L=\sum _{i=1}^{T}L(o^i,y^i)$$

for simplicity let’s assume that there is no bias and that the activation function is the identity. Then

$$h^i=W_{hh}{h}^{i-1}+W_{xh}{x}^i$$ $$o^i=W_{ho}{h}^i$$

Let $s\le t$, then

$$\frac{\partial L^t}{\partial h^s}=\frac{\partial L^t}{\partial h^t}\prod _{s<k\le t}\frac{\partial h^k}{\partial h^{k+1}}$$

but in our simple case

$$\frac{\partial h^k}{\partial h^{k+1}}=W_{hh}$$

so that

$$=\frac{\partial L^t}{\partial h^t}{W}_{hh}^l$$

things could get ugly if $l$ is big: if the greatest eigenvalue norm is greater than $1$ we get exploding gradient, if it’s less than $1$ vanishing gradient.

LSTM §

Long-Short-Term-Memory
Past input contribute less and less to the loss. We need to add memory!

Main ideas

• a new hidden state $c^t$ called cell state, with the ability to store long-term information.
• LSTM can read, write and erase information from a cell.
• Gates are defined to get the ability to select information, they are also vector of length $n$.
• At each time step $t$ the gates can be open (1) or close (0), or somewhere in between.

Forget Gate §

Controls what is kept vs forgotten from previous cell state.

$$f^t=\sigma (W_f{h}^{t-1}+U_f{x}^t+b_f)$$

Input Gate §

Controls what parts of the new input are written to the cell.

$$i^t=\sigma (W_i{h}^{t-1}+U_i{x}^t+b_i)$$

Output Gate §

Controls what parts of the cell state is trasnfered to the hidden state.

$$o^t=\sigma (W_o{h}^{t-1}+U_o{x}^t+b_o)$$

New Cell content §

We update the content of the cell and the hidden state usins the gates

$${\tilde{c}}^t=\tanh (W_c{h}^{t-1}+U_c{x}^t+b_c)$$

forget and write

$$c^t=f^t\odot c^{t-1}+i^t\odot {\tilde{c}}^t$$

the new hidden state uses the output gate

$$h^t=o^t\odot \tanh c^t$$

GRU §

Gated Recurrent Units It doesn’t use a cell state, but it has gates!

Reset Gate §

controls what parts of previous hidden state are used to compute new content

$$r^t=\sigma (W_r{h}^{t-1}+U_r{x}^t+b_r)$$

Update Gate §

controls what parts of previous hidden state are used to compute new content

$$u^t=\sigma (W_u{h}^{t-1}+U_u{x}^t+b_u)$$

We use the gates very similarly to create a new hidden state:

$${\tilde{h}}^t=\tanh (W_h(r^t\odot h^{t-1})+U_h{x}^t+b_h)$$ $$h^t=(1-u^t)\odot h^{t-1}+u^t\odot {\tilde{h}}^t$$