前面倆小節已經講了經典的alex-net和vgg網絡，vgg-net在alex網絡的基礎上，測試了很多種加深網絡的方式，得到了vgg16和vgg19最后的結果還不錯，但是后來人們發現，在網絡深度到達一定程度后，繼續加深網絡，會有倆個問題，一個是太遠了，梯度消失，即數據分散在不再被激活的那個區域導致梯度為0消失了，這個可以通過norimalized核intermediate narmalization layers解決。二個是模型的準確率會迅速下滑，單并不是overfit造成的。作者提出了一個網絡結構，通過重復利用輸入來優化訓練，結果出奇的好。
一般的，總體分為6層，第一層為一個卷積層加一個pool層，最后為一個全鏈接fc層，中間四層的每一層都由多個residual block組成，然后每個residual block又由2個或3個卷積層加shortcut connections(捷徑，即加上了初始的輸入x)組成，這樣構成的深層次的卷積神經網絡。
中間weight layer一般由2層或者3層卷積組成，層與層之間加上batch norimalization以及relu（何凱名再后續的文章中有提到怎么處理比較好，可以看這篇博客Binbin Xu），最后一層加上x后再relu，最為輸出。 這里申明下，中間是4層，每層由多個residual block組成，每個block又由多個卷積層加上x（identity效果比較好，Binbin Xu identity好處很多，首先效果好，再者不會增加參數）
最右邊即為一個32層的resnet網絡，最上面一個卷積加池化，中間分別有3，4，6，3，這四層block，每個block由倆個卷積，即（3+4+6+3=16）×2=32，再加上最后一個fc層即34層的結構。 
發現復雜了可能效果還不好，所以就做了一個簡單的模型，原始為32*32的圖，padding了4位，然后再隨機crop出32*32的圖，接著便三個卷積層，分別為32*32×16，16*16×32，8*8×64，每層n個block，每個block倆個卷積層，再加上最后fc共6n+2層。說是110層效果最好，1000+層反而還不好了，可能是過擬合。
下面是部分實現代碼，來自于ry/tensorflow-resnet
# This is what they use for CIFAR-10 and 100.# See Section 4.2 in http://arxiv.org/abs/1512.03385def inference_small(x, is_training, num_blocks=3, # 6n+2 total weight layers will be used. use_bias=False, # defaults to using batch norm num_classes=10): c = Config() c['is_training'] = tf.convert_to_tensor(is_training, dtype='bool', name='is_training') c['use_bias'] = use_bias c['fc_units_out'] = num_classes c['num_blocks'] = num_blocks c['num_classes'] = num_classes inference_small_config(x, c)def inference_small_config(x, c): c['bottleneck'] = False c['ksize'] = 3 c['stride'] = 1 with tf.variable_scope('scale1'): c['conv_filters_out'] = 16 c['block_filters_internal'] = 16 c['stack_stride'] = 1 x = conv(x, c) x = bn(x, c) x = activation(x) x = stack(x, c) with tf.variable_scope('scale2'): c['block_filters_internal'] = 32 c['stack_stride'] = 2 x = stack(x, c) with tf.variable_scope('scale3'): c['block_filters_internal'] = 64 c['stack_stride'] = 2 x = stack(x, c) # post-net x = tf.reduce_mean(x, reduction_indices=[1, 2], name="avg_pool") if c['num_classes'] != None: with tf.variable_scope('fc'): x = fc(x, c) return x另一種部分實現代碼，參考自wenxinxu/resnet-in-tensorflow
def inference(input_tensor_batch, n, reuse): ''' The main function that defines the ResNet. total layers = 1 + 2n + 2n + 2n +1 = 6n + 2 :param input_tensor_batch: 4D tensor :param n: num_residual_blocks :param reuse: To build train graph, reuse=False. To build validation graph and share weights with train graph, resue=True :return: last layer in the network. Not softmax-ed ''' layers = [] with tf.variable_scope('conv0', reuse=reuse): conv0 = conv_bn_relu_layer(input_tensor_batch, [3, 3, 3, 16], 1) activation_summary(conv0) layers.append(conv0) for i in range(n): with tf.variable_scope('conv1_%d' %i, reuse=reuse): if i == 0: conv1 = residual_block(layers[-1], 16, first_block=True) else: conv1 = residual_block(layers[-1], 16) activation_summary(conv1) layers.append(conv1) for i in range(n): with tf.variable_scope('conv2_%d' %i, reuse=reuse): conv2 = residual_block(layers[-1], 32) activation_summary(conv2) layers.append(conv2) for i in range(n): with tf.variable_scope('conv3_%d' %i, reuse=reuse): conv3 = residual_block(layers[-1], 64) layers.append(conv3) assert conv3.get_shape().as_list()[1:] == [8, 8, 64] with tf.variable_scope('fc', reuse=reuse): in_channel = layers[-1].get_shape().as_list()[-1] bn_layer = batch_normalization_layer(layers[-1], in_channel) relu_layer = tf.nn.relu(bn_layer) global_pool = tf.reduce_mean(relu_layer, [1, 2]) assert global_pool.get_shape().as_list()[-1:] == [64] output = output_layer(global_pool, 10) layers.append(output) return layers[-1]層差原文tabble1 了例如50層的話，中間四層分別（3+4+6+3）=16個block，每個block3個卷積，1×1，3×3，1×1，共16×3=48層，加上下倆層就五十層了。 下面是部分實現代碼，來自于ry/tensorflow-resnet
def inference(x, is_training, num_classes=1000, num_blocks=[3, 4, 6, 3], # defaults to 50-layer network use_bias=False, # defaults to using batch norm bottleneck=True): c = Config() c['bottleneck'] = bottleneck c['is_training'] = tf.convert_to_tensor(is_training, dtype='bool', name='is_training') c['ksize'] = 3 c['stride'] = 1 c['use_bias'] = use_bias c['fc_units_out'] = num_classes c['num_blocks'] = num_blocks c['stack_stride'] = 2 with tf.variable_scope('scale1'): c['conv_filters_out'] = 64 c['ksize'] = 7 c['stride'] = 2 x = conv(x, c) x = bn(x, c) x = activation(x) with tf.variable_scope('scale2'): x = _max_pool(x, ksize=3, stride=2) c['num_blocks'] = num_blocks[0] c['stack_stride'] = 1 c['block_filters_internal'] = 64 x = stack(x, c) with tf.variable_scope('scale3'): c['num_blocks'] = num_blocks[1] c['block_filters_internal'] = 128 assert c['stack_stride'] == 2 x = stack(x, c) with tf.variable_scope('scale4'): c['num_blocks'] = num_blocks[2] c['block_filters_internal'] = 256 x = stack(x, c) with tf.variable_scope('scale5'): c['num_blocks'] = num_blocks[3] c['block_filters_internal'] = 512 x = stack(x, c) # post-net x = tf.reduce_mean(x, reduction_indices=[1, 2], name="avg_pool") if num_classes != None: with tf.variable_scope('fc'): x = fc(x, c) return x新聞熱點
疑難解答
