长春seo招聘,长春百度seo排名,平台网站怎么优化,重庆市建设领域农民工工资专户网站上篇我们讲解了rpn与fast rcnn的数据准备阶段#xff0c;接下来我们讲解rpn的整个训练过程。最后 讲解rpn训练完毕后rpn的生成。
我们顺着stage1_rpn_train.pt的内容讲解。 name: VGG_CNN_M_1024
layer {name: input-datatype: Pythontop: datatop: im_infotop: …上篇我们讲解了rpn与fast rcnn的数据准备阶段接下来我们讲解rpn的整个训练过程。最后 讲解rpn训练完毕后rpn的生成。
我们顺着stage1_rpn_train.pt的内容讲解。 name: VGG_CNN_M_1024
layer {name: input-datatype: Pythontop: datatop: im_infotop: gt_boxespython_param {module: roi_data_layer.layerlayer: RoIDataLayerparam_str: num_classes: 21}
}
layer {name: conv1type: Convolutionbottom: datatop: conv1param { lr_mult: 0 decay_mult: 0 }param { lr_mult: 0 decay_mult: 0 }convolution_param {num_output: 96kernel_size: 7 stride: 2}
}
layer {name: relu1type: ReLUbottom: conv1top: conv1
}
layer {name: norm1type: LRNbottom: conv1top: norm1lrn_param {local_size: 5alpha: 0.0005beta: 0.75k: 2}
}
layer {name: pool1type: Poolingbottom: norm1top: pool1pooling_param {pool: MAXkernel_size: 3 stride: 2}
}
layer {name: conv2type: Convolutionbottom: pool1top: conv2param { lr_mult: 1 }param { lr_mult: 2 }convolution_param {num_output: 256pad: 1 kernel_size: 5 stride: 2}
}
layer {name: relu2type: ReLUbottom: conv2top: conv2
}
layer {name: norm2type: LRNbottom: conv2top: norm2lrn_param {local_size: 5alpha: 0.0005beta: 0.75k: 2}
}
layer {name: pool2type: Poolingbottom: norm2top: pool2pooling_param {pool: MAXkernel_size: 3 stride: 2}
}
layer {name: conv3type: Convolutionbottom: pool2top: conv3param { lr_mult: 1 }param { lr_mult: 2 }convolution_param {num_output: 512pad: 1 kernel_size: 3}
}
layer {name: relu3type: ReLUbottom: conv3top: conv3
}
layer {name: conv4type: Convolutionbottom: conv3top: conv4param { lr_mult: 1 }param { lr_mult: 2 }convolution_param {num_output: 512pad: 1 kernel_size: 3}
}
layer {name: relu4type: ReLUbottom: conv4top: conv4
}
layer {name: conv5type: Convolutionbottom: conv4top: conv5param { lr_mult: 1 }param { lr_mult: 2 }convolution_param {num_output: 512pad: 1 kernel_size: 3}
}
layer {name: relu5type: ReLUbottom: conv5top: conv5
}# RPN layer {name: rpn_conv/3x3type: Convolutionbottom: conv5top: rpn/outputparam { lr_mult: 1.0 }param { lr_mult: 2.0 }convolution_param {num_output: 256kernel_size: 3 pad: 1 stride: 1weight_filler { type: gaussian std: 0.01 }bias_filler { type: constant value: 0 }}
}
layer {name: rpn_relu/3x3type: ReLUbottom: rpn/outputtop: rpn/output
}
layer {name: rpn_cls_scoretype: Convolutionbottom: rpn/outputtop: rpn_cls_scoreparam { lr_mult: 1.0 }param { lr_mult: 2.0 }convolution_param {num_output: 18 # 2(bg/fg) * 9(anchors)kernel_size: 1 pad: 0 stride: 1weight_filler { type: gaussian std: 0.01 }bias_filler { type: constant value: 0 }}
}
layer {name: rpn_bbox_predtype: Convolutionbottom: rpn/outputtop: rpn_bbox_predparam { lr_mult: 1.0 }param { lr_mult: 2.0 }convolution_param {num_output: 36 # 4 * 9(anchors)kernel_size: 1 pad: 0 stride: 1weight_filler { type: gaussian std: 0.01 }bias_filler { type: constant value: 0 }}
}
layer {bottom: rpn_cls_scoretop: rpn_cls_score_reshapename: rpn_cls_score_reshapetype: Reshapereshape_param { shape { dim: 0 dim: 2 dim: -1 dim: 0 } }
}
layer {name: rpn-datatype: Pythonbottom: rpn_cls_scorebottom: gt_boxesbottom: im_infobottom: datatop: rpn_labelstop: rpn_bbox_targetstop: rpn_bbox_inside_weightstop: rpn_bbox_outside_weightspython_param {module: rpn.anchor_target_layerlayer: AnchorTargetLayerparam_str: feat_stride: 16}
}
layer {name: rpn_loss_clstype: SoftmaxWithLossbottom: rpn_cls_score_reshapebottom: rpn_labelspropagate_down: 1propagate_down: 0top: rpn_cls_lossloss_weight: 1loss_param {ignore_label: -1normalize: true}
}
layer {name: rpn_loss_bboxtype: SmoothL1Lossbottom: rpn_bbox_predbottom: rpn_bbox_targetsbottom: rpn_bbox_inside_weightsbottom: rpn_bbox_outside_weightstop: rpn_loss_bboxloss_weight: 1smooth_l1_loss_param { sigma: 3.0 }
}# RCNN layer {name: dummy_roi_pool_conv5type: DummyDatatop: dummy_roi_pool_conv5dummy_data_param {shape { dim: 1 dim: 18432 }data_filler { type: gaussian std: 0.01 }}
}
layer {name: fc6type: InnerProductbottom: dummy_roi_pool_conv5top: fc6param { lr_mult: 0 decay_mult: 0 }param { lr_mult: 0 decay_mult: 0 }inner_product_param {num_output: 4096}
}
layer {name: fc7type: InnerProductbottom: fc6top: fc7param { lr_mult: 0 decay_mult: 0 }param { lr_mult: 0 decay_mult: 0 }inner_product_param {num_output: 1024}
}
layer {name: silence_fc7type: Silencebottom: fc7
}它的示意图如下 这里借用了
http://blog.csdn.net/zy1034092330/article/details/62044941里的图。上面Conv layers包含了五层卷积层。 接下来对于第五层卷积层进行了3*3的卷积操作输出了256个通道当然大小与卷积前的大小相同。
然后开始分别接入了cls层与regression层。对于cls层使用1*1的卷积操作输出了189*2 bg/fg个通道的feature map,大小不变。而对于regression层也使用1*1的卷积层输出了36(4*9)个通道的feature map大小不变。
对于cls层后又接了一个reshape层为什么要接这个层呢引用参考文献[1]的话其实只是为了便于softmax分类至于具体原因这就要从caffe的实现形式说起了。在caffe基本数据结构blob中以如下形式保存数据 blob[batch_size, channelheightwidth] 对应至上面的保存bg/fg anchors的矩阵其在caffe blob中的存储形式为[1, 2*9, H, W]。而在softmax分类时需要进行fg/bg二分类所以reshape layer会将其变为[1, 2, 9*H, W]大小即单独“腾空”出来一个维度以便softmax分类之后再reshape回复原状。
我们可以用python模拟一下看如下的程序 anp.array([[[1,2],[3,4]],[[5,6],[7,8]],[[9,10],[11,12]],[[13,14],[15,16]]])a
array([[[ 1, 2],[ 3, 4]],[[ 5, 6],[ 7, 8]],[[ 9, 10],[11, 12]],[[13, 14],[15, 16]]])a.shape
(4L, 2L, 2L)然后由于caffe中是行优先numpy也如此那么reshape一下的结果如下 ba.reshape(2,4,2)b
array([[[ 1, 2],[ 3, 4],[ 5, 6],[ 7, 8]],[[ 9, 10],[11, 12],[13, 14],[15, 16]]])从上面可以看出reshape是把相邻通道的矩阵移到它的下面了。这样就剩下两个大的矩阵了就可以相邻通道之间进行softmax了。
从中其实我们也能发现对于rpn每个点的18个输出通道前9个为背景的预测分数而后9个为前景的预测分数。 假定softmax昨晚后我们看看是否能够回到原先 b.reshape(4,2,2)
array([[[ 1, 2],[ 3, 4]],[[ 5, 6],[ 7, 8]],[[ 9, 10],[11, 12]],[[13, 14],[15, 16]]])果然又回到了原始的状态。 而对于regression呢不需要这样的操作那么他的36个通道是不是也是如上面18个通道那样呢即第一个9通道为dx,第二个为dy,第三个为dw第五个是dh。还是我们比较容易想到的那种,即第一个通道是第一个盒子的回归量(dx1,dy1,dw1,dh1),第二个为(dx2,dy2,dw,2,dh2).....。待后面查看对应的bbox_targets就知道了。先留个坑。
正如图上所示我们还需要准备一个层rpn-data。 layer {name: rpn-datatype: Pythonbottom: rpn_cls_scorebottom: gt_boxesbottom: im_infobottom: datatop: rpn_labelstop: rpn_bbox_targetstop: rpn_bbox_inside_weightstop: rpn_bbox_outside_weightspython_param {module: rpn.anchor_target_layerlayer: AnchorTargetLayerparam_str: feat_stride: 16}
}这一层输入四个量data,gt_boxes,im_info,rpn_cls_score,其中前三个是我们在前面说过的 data: 1*3*600*1000 gt_boxes: N*5, N为groundtruth box的个数每一行为(x1, y1, x2, y2, cls) 而且这里的gt_box是经过缩放的。 im_info 1*3 h,w,scale
rpn_cls_score是cls层输出的18通道shape可以看成是1*18*H*W.
输出为4个量rpn_labels 、rpn_bbox_targets回归目标、rpn_bbox_inside_weights内权重、rpn_bbox_outside_weights外权重。
通俗地来讲这一层产生了具体的anchor坐标并与groundtruth box进行了重叠度计算输出了kabel与回归目标。
接下来我们来看一下文件anchor_target_layer.py def setup(self, bottom, top):layer_params yaml.load(self.param_str_) #在第5个卷积层后的feature map上的每个点取anchor,尺度为8,16,32结合后面的feat_stride为16#再缩放回原来的图像大小正好尺度是(128,256,512),与paper一样。anchor_scales layer_params.get(scales, (8, 16, 32)) self._anchors generate_anchors(scalesnp.array(anchor_scales)) #产生feature map最左上角的那个点对应的anchorx1,y1,x2,y2# 尺度为原始图像的尺度可以看成是Im_info的宽和高尺度或者是600*1000。self._num_anchors self._anchors.shape[0] #9self._feat_stride layer_params[feat_stride] #16if DEBUG:print anchors:print self._anchorsprint anchor shapes:print np.hstack(( # 输出宽和高self._anchors[:, 2::4] - self._anchors[:, 0::4], #第2列减去第0列self._anchors[:, 3::4] - self._anchors[:, 1::4], #第3列减去第1列))self._counts cfg.EPSself._sums np.zeros((1, 4))self._squared_sums np.zeros((1, 4))self._fg_sum 0self._bg_sum 0self._count 0# allow boxes to sit over the edge by a small amount self._allowed_border layer_params.get(allowed_border, 0) height, width bottom[0].data.shape[-2:] #cls后的feature map的大小if DEBUG:print AnchorTargetLayer: height, height, width, widthA self._num_anchors # labelstop[0].reshape(1, 1, A * height, width) # 显然与rpn_cls_score_reshape保持相同的shape.# bbox_targetstop[1].reshape(1, A * 4, height, width) # bbox_inside_weightstop[2].reshape(1, A * 4, height, width)# bbox_outside_weightstop[3].reshape(1, A * 4, height, width)
setup设置了top输出的shape,并且做了一些准备工作。 接下来看forward函数。 def forward(self, bottom, top):# Algorithm:## for each (H, W) location i# generate 9 anchor boxes centered on cell i# apply predicted bbox deltas at cell i to each of the 9 anchors# filter out-of-image anchors# measure GT overlapassert bottom[0].data.shape[0] 1, \Only single item batches are supported # 仅仅支持一张图片# map of shape (..., H, W)height, width bottom[0].data.shape[-2:] # GT boxes (x1, y1, x2, y2, label)gt_boxes bottom[1].data # im_infoim_info bottom[2].data[0, :]if DEBUG:print print im_size: ({}, {}).format(im_info[0], im_info[1])print scale: {}.format(im_info[2])print height, width: ({}, {}).format(height, width)print rpn: gt_boxes.shape, gt_boxes.shapeprint rpn: gt_boxes, gt_boxes# 1. Generate proposals from bbox deltas and shifted anchorsshift_x np.arange(0, width) * self._feat_stride shift_y np.arange(0, height) * self._feat_stride shift_x, shift_y np.meshgrid(shift_x, shift_y)shifts np.vstack((shift_x.ravel(), shift_y.ravel(),shift_x.ravel(), shift_y.ravel())).transpose()# add A anchors (1, A, 4) to# cell K shifts (K, 1, 4) to get# shift anchors (K, A, 4)# reshape to (K*A, 4) shifted anchorsA self._num_anchorsK shifts.shape[0]all_anchors (self._anchors.reshape((1, A, 4)) shifts.reshape((1, K, 4)).transpose((1, 0, 2)))all_anchors all_anchors.reshape((K * A, 4))total_anchors int(K * A) # 根据左上角的anchor生成所有的anchor这里将所有的anchor按照行排列。行K*A(K height*width ,A9),列4且按照feature map按行优先这样排下来。# only keep anchors inside the image #取所有在图像内部的anchorinds_inside np.where((all_anchors[:, 0] -self._allowed_border) (all_anchors[:, 1] -self._allowed_border) (all_anchors[:, 2] im_info[1] self._allowed_border) # width(all_anchors[:, 3] im_info[0] self._allowed_border) # height)[0] if DEBUG:print total_anchors, total_anchorsprint inds_inside, len(inds_inside)# keep only inside anchorsanchors all_anchors[inds_inside, :]if DEBUG:print anchors.shape, anchors.shape# label: 1 is positive, 0 is negative, -1 is dont carelabels np.empty((len(inds_inside), ), dtypenp.float32)labels.fill(-1)# overlaps between the anchors and the gt boxes# overlaps (ex, gt)overlaps bbox_overlaps(np.ascontiguousarray(anchors, dtypenp.float),np.ascontiguousarray(gt_boxes, dtypenp.float))argmax_overlaps overlaps.argmax(axis1) #对于每一个anchor取其重叠度最大的ground truth的序号max_overlaps overlaps[np.arange(len(inds_inside)), argmax_overlaps] #生成max_overlaps,(为一列)即每个anchor对应的最大重叠度gt_argmax_overlaps overlaps.argmax(axis0) #对于每个类选择其对应的最大重叠度的anchor序号gt_max_overlaps overlaps[gt_argmax_overlaps, np.arange(overlaps.shape[1])] #生成gt_max_overlaps为一行即每类对应的最大重叠度gt_argmax_overlaps np.where(overlaps gt_max_overlaps)[0] #找到那些等于gt_max_overlaps的anchor,这些anchor将参与训练rpn# 找到所有overlaps中所有等于gt_max_overlaps的元素因为gt_max_overlaps对于每个非负类别只保留一个# anchor如果同一列有多个相等的最大IOU overlap值那么就需要把其他的几个值找到并在后面将它们# 的label设为1即认为它们是object毕竟在RPN的cls任务中只要认为它是否是个object即可即一个# 二分类问题。 (总结)# 如下设置了前景(1)、背景(0)以及不关心(-1)的anchor标签if not cfg.TRAIN.RPN_CLOBBER_POSITIVES:# assign bg labels first so that positive labels can clobber themlabels[max_overlaps cfg.TRAIN.RPN_NEGATIVE_OVERLAP] 0 #对于最大重叠度低于0.3的设为背景# fg label: for each gt, anchor with highest overlap labels[gt_argmax_overlaps] 1 # fg label: above threshold IOUlabels[max_overlaps cfg.TRAIN.RPN_POSITIVE_OVERLAP] 1 if cfg.TRAIN.RPN_CLOBBER_POSITIVES:# assign bg labels last so that negative labels can clobber positiveslabels[max_overlaps cfg.TRAIN.RPN_NEGATIVE_OVERLAP] 0# 取前景与背景的anchor各一半目前一批有256个anchor.# subsample positive labels if we have too manynum_fg int(cfg.TRAIN.RPN_FG_FRACTION * cfg.TRAIN.RPN_BATCHSIZE) #256*0.5128fg_inds np.where(labels 1)[0]if len(fg_inds) num_fg:disable_inds npr.choice(fg_inds, size(len(fg_inds) - num_fg), replaceFalse)labels[disable_inds] -1# subsample negative labels if we have too manynum_bg cfg.TRAIN.RPN_BATCHSIZE - np.sum(labels 1) #另一半256*0.5128bg_inds np.where(labels 0)[0]if len(bg_inds) num_bg:disable_inds npr.choice(bg_inds, size(len(bg_inds) - num_bg), replaceFalse)labels[disable_inds] -1#print was %s inds, disabling %s, now %s inds % (#len(bg_inds), len(disable_inds), np.sum(labels 0))#计算了所有在内部的anchor与对应的ground truth的回归量bbox_targets np.zeros((len(inds_inside), 4), dtypenp.float32)bbox_targets _compute_targets(anchors, gt_boxes[argmax_overlaps, :])#只有前景类内部权重才非0参与回归bbox_inside_weights np.zeros((len(inds_inside), 4), dtypenp.float32)bbox_inside_weights[labels 1, :] np.array(cfg.TRAIN.RPN_BBOX_INSIDE_WEIGHTS) #(1.0, 1.0, 1.0, 1.0)# Give the positive RPN examples weight of p * 1 / {num positives}# and give negatives a weight of (1 - p)/(num negative) # Set to -1.0 to use uniform example weightingbbox_outside_weights np.zeros((len(inds_inside), 4), dtypenp.float32)if cfg.TRAIN.RPN_POSITIVE_WEIGHT 0:# uniform weighting of examples (given non-uniform sampling)num_examples np.sum(labels 0)positive_weights np.ones((1, 4)) * 1.0 / num_examplesnegative_weights np.ones((1, 4)) * 1.0 / num_exampleselse:assert ((cfg.TRAIN.RPN_POSITIVE_WEIGHT 0) (cfg.TRAIN.RPN_POSITIVE_WEIGHT 1))positive_weights (cfg.TRAIN.RPN_POSITIVE_WEIGHT /np.sum(labels 1))negative_weights ((1.0 - cfg.TRAIN.RPN_POSITIVE_WEIGHT) /np.sum(labels 0))bbox_outside_weights[labels 1, :] positive_weights # 前景与背景anchor的外参数相同都是1/anchor个数bbox_outside_weights[labels 0, :] negative_weightsif DEBUG:self._sums bbox_targets[labels 1, :].sum(axis0)self._squared_sums (bbox_targets[labels 1, :] ** 2).sum(axis0)self._counts np.sum(labels 1)means self._sums / self._countsstds np.sqrt(self._squared_sums / self._counts - means ** 2)print means:print meansprint stdevs:print stds# map up to original set of anchors 生成全部anchor的数据将非0的数据填入。labels _unmap(labels, total_anchors, inds_inside, fill-1)bbox_targets _unmap(bbox_targets, total_anchors, inds_inside, fill0)bbox_inside_weights _unmap(bbox_inside_weights, total_anchors, inds_inside, fill0)bbox_outside_weights _unmap(bbox_outside_weights, total_anchors, inds_inside, fill0)if DEBUG:print rpn: max max_overlap, np.max(max_overlaps)print rpn: num_positive, np.sum(labels 1)print rpn: num_negative, np.sum(labels 0)self._fg_sum np.sum(labels 1)self._bg_sum np.sum(labels 0)self._count 1print rpn: num_positive avg, self._fg_sum / self._countprint rpn: num_negative avg, self._bg_sum / self._count# labels labels labels.reshape((1, height, width, A)).transpose(0, 3, 1, 2)labels labels.reshape((1, 1, A * height, width))top[0].reshape(*labels.shape)top[0].data[...] labels# bbox_targetsbbox_targets bbox_targets \.reshape((1, height, width, A * 4)).transpose(0, 3, 1, 2)top[1].reshape(*bbox_targets.shape)top[1].data[...] bbox_targets# bbox_inside_weightsbbox_inside_weights bbox_inside_weights \.reshape((1, height, width, A * 4)).transpose(0, 3, 1, 2)assert bbox_inside_weights.shape[2] heightassert bbox_inside_weights.shape[3] widthtop[2].reshape(*bbox_inside_weights.shape)top[2].data[...] bbox_inside_weights# bbox_outside_weightsbbox_outside_weights bbox_outside_weights \.reshape((1, height, width, A * 4)).transpose(0, 3, 1, 2)assert bbox_outside_weights.shape[2] heightassert bbox_outside_weights.shape[3] widthtop[3].reshape(*bbox_outside_weights.shape)top[3].data[...] bbox_outside_weights这里已经有详细的注释总的来说rpn_cls_score的作用就是告知第五层feature map的宽和高。便于决定生成多少个anchor. 而其他的bottom输入才最终决定top的输出。 首先这里生成了所有feature map各点对应的anchors。生成的方式很特别先考虑了左上角一个点的anchor生成考虑到feat_stride16所以这个点对应原始图像这里统一指缩放后image的(0,0,15,15)感受野。然后取其中心点生成比例为1:1,1:2,2:1尺度在128,256,512的9个anchor.然后考虑使用平移生成其他的anchor.
然后过滤掉那些不在图像内部的anchor. 对于剩下的anchor,计算与gt_boxes的重叠度再分别计算label,bbox_targets,bbox_inside_weights,bbox_outside_weights. 最后将内部的anchor的相关变量扩充到所有的anchor,只不过不在内部的为0即可。尤其值得说的是对于内部的anchor,bbox_targets都进行了运算。但是选取了256个anchor,前景与背景比例为1:1bbox_inside_weights中只有label1,即前景才进行了设置。正如论文所说对于回归项需要内部参数来约束bbox_inside_weights正好起到了这个作用。
我们统计一下top的shape:
rpn_labels (1, 1, 9 * height, width)
rpn_bbox_targets回归目标 (1, 36height, width)
rpn_bbox_inside_weights内权重(1, 36height, width)
rpn_bbox_outside_weights外权重(1, 36height, width) 回到stage1_rpn_train.pt接下里我们就可以利用rpn_cls_score_reshape与rpn_labels计算SoftmaxWithLoss输出rpn_cls_loss。
而regression可以利用rpn_bbox_predrpn_bbox_targetsrpn_bbox_inside_weightsrpn_bbox_outside_weights计算SmoothL1Loss输出rpn_loss_bbox。
回到我们之前有一个问题rpn_bbox_pred的shape怎么构造的。其实从rpn_bbox_targets的生成过程中可以推断出应该采用后一种即第一个盒子的回归量(dx1,dy1,dw1,dh1),第二个为(dx2,dy2,dw,2,dh2).....这样顺序着来。 其实怎么样认为都是从我们方便的角度出发。 至此我们完成了rpn的前向过程反向过程中只需注意AnchorTargetLayer不参与反向传播。因为它提供的都是源数据。
参考
1. http://blog.csdn.net/zy1034092330/article/details/62044941 2.
Faster RCNN anchor_target_layer.py
