下图是一个基于字符级循环神经网络的语言模型，能够基于当前的输入与过去的输入序列，预测序列的下一个字符。循环神经网络引入了一个隐藏变量H H H H t H_{t} H t H H H t t t H t H_{t} H t X t X_{t} X t H t − 1 H_{t-1} H t − 1 H t H_{t} H t H t H_{t} H t
循环神经网络有很多不同的构造方法，这里使用常见的一种。假设X t ∈ R n × d \boldsymbol{X}_t \in \mathbb{R}^{n \times d} X t ∈ R n × d t t t H t ∈ R n × h \boldsymbol{H}_t \in \mathbb{R}^{n \times h} H t ∈ R n × h
H t = ϕ ( X t W x h + H t − 1 W h h + b h ) . \boldsymbol{H}_t = \phi(\boldsymbol{X}_t \boldsymbol{W}_{xh} + \boldsymbol{H}_{t-1} \boldsymbol{W}_{hh} + \boldsymbol{b}_h).
H t = ϕ ( X t W x h + H t − 1 W h h + b h ) .
其中，W x h ∈ R d × h \boldsymbol{W}_{xh} \in \mathbb{R}^{d \times h} W x h ∈ R d × h W h h ∈ R h × h \boldsymbol{W}_{hh} \in \mathbb{R}^{h \times h} W h h ∈ R h × h b h ∈ R 1 × h \boldsymbol{b}_{h} \in \mathbb{R}^{1 \times h} b h ∈ R 1 × h ϕ \phi ϕ H t − 1 W h h \boldsymbol{H}_{t-1} \boldsymbol{W}_{hh} H t − 1 W h h H t H_{t} H t H t H_{t} H t H t − 1 H_{t-1} H t − 1
在时间步t t t
O t = H t W h q + b q . \boldsymbol{O}_t = \boldsymbol{H}_t \boldsymbol{W}_{hq} + \boldsymbol{b}_q.
O t = H t W h q + b q .
其中W h q ∈ R h × q \boldsymbol{W}_{hq} \in \mathbb{R}^{h \times q} W h q ∈ R h × q b q ∈ R 1 × q \boldsymbol{b}_q \in \mathbb{R}^{1 \times q} b q ∈ R 1 × q
注意：
即便在不同时间步，循环神经网络也始终使用这些模型参数。因此，循环神经网络模型参数的数量不随时间步的增加而增长；
批量训练的过程中，参数是以批为单位更新的，每个批次内模型的参数都是一样的；
RNN通过不断循环使用同一组参数来应对不同长度的序列，即网络参数数量与输入序列长度无关；
隐藏状态H t H_{t} H t H 1 , . . . , H t − 1 H_{1},...,H_{t-1} H 1 , . . . , H t − 1
导入上一篇 文章中对Jay歌词数据预处理后的语料数据。
deeplearning_03.py view raw 1 2 3 4 5 6 7 8 import torchimport torch.nn as nnimport timeimport mathimport deeplearning_02 as dl_2device = torch.device('cuda' if torch.cuda.is_available() else 'cpu' ) (corpus_indices, char_to_idx, idx_to_char, vocab_size) = dl_2.load_data_jay_lyrics()
将字符表示成向量，才能送进神经网络进行矢量运算，这里采用one-hot编码方式。
假设词典大小是N N N 0 0 0 N − 1 N-1 N − 1 N N N i i i i i i 1 1 1 0 0 0
deeplearning_03.py view raw 1 2 3 4 def one_hot (x, n_class, dtype=torch.float32) : result = torch.zeros(x.shape[0 ], n_class, dtype=dtype, device=x.device) result.scatter_(1 , x.long().view(-1 , 1 ), 1 ) return result
每次采样的小批量的形状是（batch_size, num_steps），将每个样本中的每个字符用one-hot编码后，会将这样的小批量变换成多个形状为（batch_size, 词典大小N N N t t t X t ∈ R n × d \boldsymbol{X}_t \in \mathbb{R}^{n \times d} X t ∈ R n × d n n n d d d
deeplearning_03.py view raw 1 2 def to_onehot (X, n_class) : return [one_hot(X[:, i], n_class) for i in range(X.shape[1 ])]
deeplearning_03.py view raw 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 num_inputs, num_hiddens, num_outputs = vocab_size, 256 , vocab_size def get_params () : def _one (shape) : param = torch.zeros(shape, device=device, dtype=torch.float32) nn.init.normal_(param, 0 , 0.01 ) return torch.nn.Parameter(param) W_xh = _one((num_inputs, num_hiddens)) W_hh = _one((num_hiddens, num_hiddens)) b_h = torch.nn.Parameter(torch.zeros(num_hiddens, device=device)) W_hq = _one((num_hiddens, num_outputs)) b_q = torch.nn.Parameter(torch.zeros(num_outputs, device=device)) return (W_xh, W_hh, b_h, W_hq, b_q)
num_inputs = d = 特征数 = one-hot向量长度 = 词典大小num_hiddens = h = 隐藏单元的个数（超参数）num_outputs = q = 输出个数 （= 分类类别数）
所以回顾上文RNN构造的数学表达式，会更好理解：
H t = ϕ ( X t W x h + H t − 1 W h h + b h ) . \boldsymbol{H}_t = \phi(\boldsymbol{X}_t \boldsymbol{W}_{xh} + \boldsymbol{H}_{t-1} \boldsymbol{W}_{hh} + \boldsymbol{b}_h).
H t = ϕ ( X t W x h + H t − 1 W h h + b h ) .
其中X t ∈ R n × d \boldsymbol{X}_t \in \mathbb{R}^{n \times d} X t ∈ R n × d H t ∈ R n × h \boldsymbol{H}_t \in \mathbb{R}^{n \times h} H t ∈ R n × h W x h ∈ R d × h \boldsymbol{W}_{xh} \in \mathbb{R}^{d \times h} W x h ∈ R d × h W h h ∈ R h × h \boldsymbol{W}_{hh} \in \mathbb{R}^{h \times h} W h h ∈ R h × h b h ∈ R 1 × h \boldsymbol{b}_{h} \in \mathbb{R}^{1 \times h} b h ∈ R 1 × h
O t = H t W h q + b q . \boldsymbol{O}_t = \boldsymbol{H}_t \boldsymbol{W}_{hq} + \boldsymbol{b}_q.
O t = H t W h q + b q .
其中W h q ∈ R h × q \boldsymbol{W}_{hq} \in \mathbb{R}^{h \times q} W h q ∈ R h × q b q ∈ R 1 × q \boldsymbol{b}_q \in \mathbb{R}^{1 \times q} b q ∈ R 1 × q
首先初始化隐藏状态，返回由一个形状为(批量大小, 隐藏单元个数)的值为0的NDArray组成的元组。使用元组是为了更便于处理隐藏状态含有多个NDArray的情况：
deeplearning_03.py view raw 1 2 def init_rnn_state (batch_size, num_hiddens, device) : return (torch.zeros((batch_size, num_hiddens), device=device), )
再定义RNN网络结构：
deeplearning_03.py view raw 1 2 3 4 5 6 7 8 9 10 11 def rnn (inputs, state, params) : W_xh, W_hh, b_h, W_hq, b_q = params H, = state outputs = [] for X in inputs: H = torch.tanh(torch.matmul(X, W_xh) + torch.matmul(H, W_hh) + b_h) Y = torch.matmul(H, W_hq) + b_q outputs.append(Y) return outputs, (H,)
最后记得检查下参数维度：
deeplearning_03.py view raw 1 2 3 4 5 state = init_rnn_state(X.shape[0 ], num_hiddens, ctx) inputs = to_onehot(X.as_in_context(ctx), vocab_size) params = get_params() outputs, state_new = rnn(inputs, state, params) print(len(outputs), outputs[0 ].shape, state_new[0 ].shape)
(5, (2, 1027), (2, 256))
为了在迭代模型参数优化计算效率，在训练过程中评价模型，先引入裁剪梯度和困惑度两个概念。
循环神经网络中较容易出现梯度衰减或梯度爆炸，这会导致网络几乎无法训练。裁剪梯度（clip gradient）是一种应对梯度爆炸的方法。假设把所有模型参数的梯度拼接成一个向量 g \boldsymbol{g} g θ \theta θ
min ( θ ∥ g ∥ , 1 ) g \min\left(\frac{\theta}{\|\boldsymbol{g}\|}, 1\right)\boldsymbol{g}
min ( ∥ g ∥ θ , 1 ) g
的L 2 L_2 L 2 θ \theta θ
deeplearning_03.py view raw 1 2 3 4 5 6 7 8 def grad_clipping (params, theta, device) : norm = torch.tensor([0.0 ], device=device) for param in params: norm += (param.grad.data ** 2 ).sum() norm = norm.sqrt().item() if norm > theta: for param in params: param.grad.data *= (theta / norm)
用于来评价语言模型的好坏，是对交叉熵损失函数做指数运算后得到的值。
最佳情况：模型总是把标签类别的概率预测为1，此时困惑度为1；
最坏情况：模型总是把标签类别的概率预测为0，此时困惑度为正无穷；
基线情况（随机分类模型）：模型总是预测所有类别的概率都相同，此时困惑度为类别个数。
可见，任何一个有效模型的困惑度必须小于类别个数。因此，在此处困惑度必须小于词典大小vocab_size。
引入上篇文章 中对时序数据采用随机采样和相邻采样方法。
注意：不同采样方法隐藏状态初始化不同detach_()的操作，每次迭代之后的输出是一个叶子节点，并且该叶子节点的requires_grad = True(从上面的计算图就可以看出)，也就意味着两次或者说多次的迭代，计算图一直都是连着的，因为没有遇到梯度计算的结束位置，这样将会一直持续到下一次隐藏状态的初始化。所以这将会导致计算图非常的大，进而导致计算开销非常大。detach_()出来，其实就是相当于每次迭代之后虽然是使用上一次迭代的隐藏状态，只不过我们希望重新开始，具体的操作就是把上一次的输出节点的参数requires_grad设置为False的叶子节点。
deeplearning_03.py view raw 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 def train_and_predict_rnn (rnn, get_params, init_rnn_state, num_hiddens, vocab_size, device, corpus_indices, idx_to_char, char_to_idx, is_random_iter, num_epochs, num_steps, lr, clipping_theta, batch_size, pred_period, pred_len, prefixes) : if is_random_iter: data_iter_fn = dl_2.data_iter_random else : data_iter_fn = dl_2.data_iter_consecutive params = get_params() loss = nn.CrossEntropyLoss() for epoch in range(num_epochs): if not is_random_iter: state = init_rnn_state(batch_size, num_hiddens, device) l_sum, n, start = 0.0 , 0 , time.time() data_iter = data_iter_fn(corpus_indices, batch_size, num_steps, device) for X, Y in data_iter: if is_random_iter: state = init_rnn_state(batch_size, num_hiddens, device) else : for s in state: s.detach_() inputs = to_onehot(X, vocab_size) (outputs, state) = rnn(inputs, state, params) outputs = torch.cat(outputs, dim=0 ) y = torch.flatten(Y.T) l = loss(outputs, y.long()) if params[0 ].grad is not None : for param in params: param.grad.data.zero_() l.backward() grad_clipping(params, clipping_theta, device) dl_2.sgd(params, lr, 1 ) l_sum += l.item() * y.shape[0 ] n += y.shape[0 ] if (epoch + 1 ) % pred_period == 0 : print('epoch %d, perplexity %f, time %.2f sec' % ( epoch + 1 , math.exp(l_sum / n), time.time() - start)) for prefix in prefixes: print(predict_rnn(prefix, pred_len, rnn, params, init_rnn_state, num_hiddens, vocab_size, device, idx_to_char, char_to_idx))
每过50个迭代周期pred_period便根据前缀“喜欢”和“分手”分别创作长度pred_len为50个字符（不考虑前缀长度）的一段歌词。
deeplearning_03.py view raw 1 2 num_epochs, num_steps, batch_size, lr, clipping_theta = 250 , 35 , 32 , 1e2 , 1e-2 pred_period, pred_len, prefixes = 50 , 50 , ['喜欢' , '分手' ]
基于随机采样的模型效果：
deeplearning_03.py view raw 1 2 3 4 5 train_and_predict_rnn(rnn, get_params, init_rnn_state, num_hiddens, vocab_size, device, corpus_indices, idx_to_char, char_to_idx, True , num_epochs, num_steps, lr, clipping_theta, batch_size, pred_period, pred_len, prefixes)
epoch 50, perplexity 70.843629, time 0.60 sec
基于相邻采样的模型效果：
deeplearning_03.py view raw 1 2 3 4 5 train_and_predict_rnn(rnn, get_params, init_rnn_state, num_hiddens, vocab_size, device, corpus_indices, idx_to_char, char_to_idx, False , num_epochs, num_steps, lr, clipping_theta, batch_size, pred_period, pred_len, prefixes)
epoch 50, perplexity 61.914999, time 0.61 sec
input_size - The number of expected features in the input xhidden_size – The number of features in the hidden state hnonlinearity – The non-linearity to use. Can be either ‘tanh’ or ‘relu’. Default: ‘tanh’batch_first – If True, then the input and output tensors are provided as (batch_size, num_steps, input_size). Default: False
这里的batch_first决定了输入的形状，默认为False，对应的输入形状是 (num_steps, batch_size, input_size)。
forward函数的参数为：
input of shape (num_steps, batch_size, input_size): tensor containing the features of the input sequence.h_0 of shape (num_layers * num_directions, batch_size, hidden_size): tensor containing the initial hidden state for each element in the batch. Defaults to zero if not provided. If the RNN is bidirectional, num_directions should be 2, else it should be 1.
forward函数的返回值是：
output of shape (num_steps, batch_size, num_directions * hidden_size): tensor containing the output features (h_t) from the last layer of the RNN, for each t.h_n of shape (num_layers * num_directions, batch_size, hidden_size): tensor containing the hidden state for t = num_steps.
deeplearning_03.py view raw 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 class RNNModel (nn.Module) : def __init__ (self, rnn_layer, vocab_size) : super(RNNModel, self).__init__() self.rnn = rnn_layer self.hidden_size = rnn_layer.hidden_size * (2 if rnn_layer.bidirectional else 1 ) self.vocab_size = vocab_size self.dense = nn.Linear(self.hidden_size, vocab_size) def forward (self, inputs, state) : X = to_onehot(inputs, vocab_size) X = torch.stack(X) hiddens, state = self.rnn(X, state) hiddens = hiddens.view(-1 , hiddens.shape[-1 ]) output = self.dense(hiddens) return output, state
deeplearning_03.py view raw 1 2 3 4 5 6 7 8 9 10 11 12 def predict_rnn_pytorch (prefix, num_chars, model, vocab_size, device, idx_to_char, char_to_idx) : state = None output = [char_to_idx[prefix[0 ]]] for t in range(num_chars + len(prefix) - 1 ): X = torch.tensor([output[-1 ]], device=device).view(1 , 1 ) (Y, state) = model(X, state) if t < len(prefix) - 1 : output.append(char_to_idx[prefix[t + 1 ]]) else : output.append(Y.argmax(dim=1 ).item()) return '' .join([idx_to_char[i] for i in output])
仅使用相邻采样
deeplearning_03.py view raw 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 def train_and_predict_rnn_pytorch (model, num_hiddens, vocab_size, device, corpus_indices, idx_to_char, char_to_idx, num_epochs, num_steps, lr, clipping_theta, batch_size, pred_period, pred_len, prefixes) : loss = nn.CrossEntropyLoss() optimizer = torch.optim.Adam(model.parameters(), lr=lr) model.to(device) for epoch in range(num_epochs): l_sum, n, start = 0.0 , 0 , time.time() data_iter = dl_2.data_iter_consecutive(corpus_indices, batch_size, num_steps, device) state = None for X, Y in data_iter: if state is not None : if isinstance (state, tuple): state[0 ].detach_() state[1 ].detach_() else : state.detach_() (output, state) = model(X, state) y = torch.flatten(Y.T) l = loss(output, y.long()) optimizer.zero_grad() l.backward() grad_clipping(model.parameters(), clipping_theta, device) optimizer.step() l_sum += l.item() * y.shape[0 ] n += y.shape[0 ] if (epoch + 1 ) % pred_period == 0 : print('epoch %d, perplexity %f, time %.2f sec' % ( epoch + 1 , math.exp(l_sum / n), time.time() - start)) for prefix in prefixes: print(predict_rnn_pytorch( prefix, pred_len, model, vocab_size, device, idx_to_char, char_to_idx))
deeplearning_03.py view raw 1 2 3 4 5 6 num_epochs, batch_size, lr, clipping_theta = 250 , 32 , 1e-3 , 1e-2 pred_period, pred_len, prefixes = 50 , 50 , ['喜欢' , '分手' ] train_and_predict_rnn_pytorch(model, num_hiddens, vocab_size, device, corpus_indices, idx_to_char, char_to_idx, num_epochs, num_steps, lr, clipping_theta, batch_size, pred_period, pred_len, prefixes)
epoch 50, perplexity 1.015603, time 0.37 sec
