[深度学习论文笔记][Recurrent Neural Networks] Visualizing and Understanding Recurrent Networks
Karpathy et al.的研究团队发表在《arXiv预印本》上，并详细描述了循环神经网络的可视化与理解过程（参考文献: 79）。
1 RNN
RNN has form
Where W changes across layers while being maintained across time steps. ⃗x represents the input originating from the preceding layer.
It was noted that the back-propagation behaviors in an RNN led to either the disappearance of gradients or their instability.
2 LSTM
The exploding gradient issue is addressed through a technique known as gradient clipping. Additionally, Long Short-Term Memory networks (LSTMs) were developed to tackle the vanishing gradient problem by maintaining an internal memory structure. Furthermore, in addition to these features, LSTMs incorporate specialized mechanisms that enable them to process sequential data effectively by managing information flow through their architecture using gates and control units.
The three vectors ⃗i, ⃗f, o⃗ can be conceptualized as simple on/off switches that control the updating and resetting of each memory cell and the exposure of their local states to.
the hidden vectors, respectively. The activations of these gates are determined by the sigmoid function and thereby lie smoothly between zero and one to ensure the model's differentiability.
The vector g, which spans from -1 to 1, plays a role in modifying memory contents additively. This additive interaction represents a key characteristic of LSTM architecture, as during backpropagation summation operations merely propagate gradients. This permits gradients on memory cells ⃗ c to flow backwards through time without interruption for extended durations, typically continuing until interrupted by multiplicative interactions involving an active forget gate. The behavior described in this section can be further understood by consulting Figure .
3 GRU
This is a simple alternative to LSTM.
The GRU interprets the computation of a candidate hidden vector h ̃ t and gradually transitions to it, which is controlled by the gate vector ⃗ z.