Tensorflow Object Detection

前言

本文主要介紹如何利用官方庫tensorflow/models/research/objection
並通過faster rcnn resnet 101(以及其他)深度學習框架
訓練自己的資料集，並對訓練結果進行檢測和評估

準備工作

1. 準備自己的資料集

資料集檔案目錄如下

datas/
    datas/
        img/
        xml/
    disk_label_map.pbtxt

img/目錄下為資料集圖片

xml/目錄下為圖片對應的資訊

15_11_09_53_513.xml

<?xml version="1.0" encoding="utf-8"?>
<annotation>
    <folder>datas</folder>
    <filename>jpg</filename>
    <source>
        <database>Unknown</database>
    </source>
    <size>
        <width>564</width>
        <height
 
>430</height>
        <depth>3</depth>
    </size>
    <segmented>0</segmented>
    <object>
        <name>rect</name>
        <pose>Unspecified</pose>
        <truncated>0</truncated>
        <difficult>0</difficult>
        <bndbox
 
>
            <xmin>255</xmin>
            <ymin>47</ymin>
            <xmax>460</xmax>
            <ymax>170</ymax>
        </bndbox>
    </object>
    <object>
        <name>rice</name>
        <pose>Unspecified</pose>
        <truncated>0</truncated>
        <difficult>0</difficult>
        <bndbox>
            <xmin>81</xmin>
            <ymin>165</ymin>
            <xmax>246</xmax>
            <ymax>330</ymax>
        </bndbox>
    </object>
</annotation>

（其中object為檢測到的物體，name代表物體類別與disk_label_map.pbtxt中指定的一致，bndbox檢測到的區域）

disk_label_map.pbtxt

item {
  id: 1
  name: 'rice'
}

item {
  id: 2
  name: 'soup'
}

item {
  id: 3
  name: 'rect'
}

item {
  id: 4
  name: 'lcir'
}

item {
  id: 5
  name: 'ssquare'
}

item {
  id: 6
  name: 'msquare'
}

item {
  id: 7
  name: 'lsquare'
}

item {
  id: 8
  name: 'bsquare'
}

item {
  id: 9
  name: 'ellipse'
}

2.安裝tensorflow-gpu

$ sudo apt-get install python-virtualenv
$ virtualenv --system-site-packages tensorflow （在～目錄下建立獨立執行環境）
$ source ~/tensorflow/bin/activate (啟用tensorflow執行環境，以後每次執行該環境下的專案，都要啟用）
$ pip install --upgrade tensorflow-gpu

通過import tensorflow驗證安裝

3.下載tensorflow/models倉庫

$ git clone https://github.com/tensorflow/models.git

下載速度較慢，建議翻牆
之後把下載好的檔案解壓到~/tensorflow/目錄下

4.安裝object_detection專案

安裝依賴庫

$ sudo apt-get install protobuf-compiler
$ sudo pip install pillow
$ sudo pip install lxml
$ sudo pip install jupyter
$ sudo pip install matplotlib

編譯protobuf

# From tensorflow/models/research/
protoc object_detection/protos/*.proto --python_out=.

區域性執行時，把library加入PYTHONPATH

# From tensorflow/models/research/
export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim

測試安裝是否成功

python object_detection/builders/model_builder_test.py

5.下載faster rcnn resnet101 coco model

訓練工作

1.處理訓練集

對於訓練過程中影象畫素越大可能訓練神經網路引數消耗的CPU，佔用的記憶體就會越大，
一般4核，8G訓練500*500左右畫素大小的幾百張圖片比較適合

壓縮圖片大小：(將datas資料集複製一份命名為datas1放置於與datas同目錄下）
jpg_compression.py

# /home/user/Downloads/datas/jpg_compression.py
from PIL import Image
import os
import sys

# Define images type to detect
valid_file_type = ['.jpg','.jpeg']
# Define compression ratio
SIZE_normal = 1.0
SIZE_small = 1.5
SIZE_more_small = 2.0
SIZE_much_more_small = 3.0

def make_directory(directory):
    """Make dir"""
    os.makedirs(directory)


def directory_exists(directory):
    """If this dir exists"""
    if os.path.exists(directory):
        return True
    else:
        return False


def list_img_file(directory):
    """List all the files, choose and return jpg files"""
    old_list = os.listdir(directory)
    # print old_list
    new_list = []
    for filename in old_list:
        f, e = os.path.splitext(filename)
        if e in valid_file_type:
            new_list.append(filename)
        else:
            pass
    # print new_list
    return new_list


def print_help():
    print """
    This program helps compress many image files
    you can choose which scale you want to compress your img(jpg/etc)
    1) normal compress(4M to 1M around)
    2) small compress(4M to 500K around)
    3) smaller compress(4M to 300K around)
    4) much smaller compress(4M to ...)
    """


def compress(choose, src_dir, des_dir, file_list):
    """Compression Algorithm,img.thumbnail"""
    if choose == '1':
        scale = SIZE_normal
    if choose == '2':
        scale = SIZE_small
    if choose == '3':
        scale = SIZE_more_small
    if choose == '4':
        scale = SIZE_much_more_small
    for infile in file_list:
        filename = os.path.join(src_dir, infile)
        img = Image.open(filename)
        # size_of_file = os.path.getsize(infile)
        w, h = img.size
        img.thumbnail((int(w/scale), int(h/scale)))
        img.save(des_dir + '/' + infile)


if __name__ == "__main__":
    src_dir, des_dir = sys.argv[1], sys.argv[2]
    if directory_exists(src_dir):
        if not directory_exists(des_dir):
            make_directory(des_dir)
        # business logic
        file_list = list_img_file(src_dir)
        # print file_list
        if file_list:
            print_help()
            choose = raw_input("enter your choice:")
            compress(choose, src_dir, des_dir, file_list)
        else:
            pass
    else:
        print "source directory not exist!"

執行命令

python jpg_compression.py \
> --src_dir=/home/user/Downloads/datas1/datas/img/
> --des_dir=/home/user/Downloads/datas/datas/img/

根據壓縮圖片比例改變xml檔案內容

因為xml檔案有記錄對應影象畫素大小，以及檢測物體區域位置，所以要更改這些值
modify_xml.py(縮小的是三倍）

from PIL import Image
from xml.dom import minidom
import os
import sys


if __name__ == "__main__":
    src_dir, des_dir = sys.argv[1], sys.argv[2]
    file_list = os.listdir(src_dir)
    for file_name in file_list:
        xml_name = os.path.join(src_dir, file_name)
        with open(xml_name, 'r') as fh:
            dom = minidom.parse(fh)
            root = dom.documentElement
            # print root.nodeName
            sizeNode = root.getElementsByTagName('size')[0]
            # print size.nodeName
            widthNode = sizeNode.getElementsByTagName('width')[0]
            value = widthNode.childNodes[0].nodeValue.encode('gbk')
            value_int = int(value)/3
            value = str(value_int)
            value = value.decode('utf-8')
            widthNode.childNodes[0].nodeValue = value
            #print widthNode.childNodes[0].nodeValue
            heightNode = sizeNode.getElementsByTagName('height')[0]
            value = heightNode.childNodes[0].nodeValue.encode('gbk')
            value_int = int(value)/3
            value = str(value_int)
            value = value.decode('utf-8')
            heightNode.childNodes[0].nodeValue = value

            objectNodes = root.getElementsByTagName('object')
            for idx,subNode in enumerate(objectNodes):
                bndboxNode = subNode.getElementsByTagName('bndbox')[0]
                #print bndboxNode
                minxNode = bndboxNode.getElementsByTagName('xmin')[0]
                val = minxNode.childNodes[0].nodeValue.encode('gbk')
                val_int = int(val)/3
                val = str(val_int)
                val = val.decode('utf-8')
                minxNode.childNodes[0].nodeValue = val

                minyNode = bndboxNode.getElementsByTagName('ymin')[0]
                val = minyNode.childNodes[0].nodeValue.encode('gbk')
                val_int = int(val)/3
                val = str(val_int)
                val = val.decode('utf-8')
                minyNode.childNodes[0].nodeValue = val

                maxxNode = bndboxNode.getElementsByTagName('xmax')[0]
                val = maxxNode.childNodes[0].nodeValue.encode('gbk')
                val_int = int(val)/3
                val = str(val_int)
                val = val.decode('utf-8')
                maxxNode.childNodes[0].nodeValue = val

                maxyNode = bndboxNode.getElementsByTagName('ymax')[0]
                val = maxyNode.childNodes[0].nodeValue.encode('gbk')
                val_int = int(val)/3
                val = str(val_int)
                val = val.decode('utf-8')
                maxyNode.childNodes[0].nodeValue = val
                # print maxxNode.childNodes[0].nodeValue
                bndboxNode.replaceChild(bndboxNode.getElementsByTagName('xmin')[0], minxNode)
                bndboxNode.replaceChild(bndboxNode.getElementsByTagName('ymin')[0], minyNode)
                bndboxNode.replaceChild(bndboxNode.getElementsByTagName('xmax')[0], maxxNode)
                bndboxNode.replaceChild(bndboxNode.getElementsByTagName('ymax')[0], maxyNode)

                objectNodes[idx].replaceChild(objectNodes[idx].getElementsByTagName('bndbox')[0], bndboxNode)

                dom.documentElement.replaceChild(dom.documentElement.getElementsByTagName('object')[idx], objectNodes[idx])

            sizeNode.replaceChild(sizeNode.getElementsByTagName('width')[0], widthNode)
            sizeNode.replaceChild(sizeNode.getElementsByTagName('height')[0], heightNode)

            dom.documentElement.replaceChild(dom.documentElement.getElementsByTagName('size')[0], sizeNode)

            des_path = os.path.join(des_dir, file_name)
            # print des_path
            f = open(des_path, 'w')
            dom.writexml(f,encoding = 'utf-8' )
            f.close()  
            # print dom.documentElement.getElementsByTagName('size')[0].getElementsByTagName('height')[0].childNodes[0].nodeValue

執行檔案

python modify_xml.py \
> --src_dir=/home/user/Downloads/datas1/datas/xml/
> --des_dir=/home/user/Downloads/datas/datas/xml/

2.修改介面檔案

首先先熟悉一下這個object_detection專案需要修改的檔案的作用


eval.py可執行檔案用於測試評估訓練資料
train.py可執行檔案用於訓練給定的record檔案中的資料
export_inference_graph.py用於把訓練出的ckpt檔案轉換成pb檔案可供測試
samples/config/從中選出訓練所用的神經網路框架的配置檔案
dataset_tools/create…tf_record.py可執行檔案用於把資料集導成record檔案待訓練
object_detection_tutorial.ipynb在jupyter notebook裡執行用於測試，檢視圖片檢驗效果

create_disk_tf_record.py
轉化資料為train.tfrecords檔案

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import hashlib
import io
import logging
import os

from lxml import etree
import PIL.Image
import tensorflow as tf

from object_detection.utils import dataset_util
from object_detection.utils import label_map_util

flags = tf.app.flags
flags.DEFINE_string('data_dir', '', 'Root directory to dataset')
flags.DEFINE_string('images_dir', '', 'Path to images directory')
flags.DEFINE_string('annotations_dir', '', 'Path to annotations directory')
flags.DEFINE_string('output_path', '', 'Path to output TFRecord')
flags.DEFINE_string('label_map_path', '', 'Path to label map proto')
flags.DEFINE_boolean('ignore_difficult_instances', False, 'Whether to ignore difficult instances')

FLAGS = flags.FLAGS

def dict_to_tf_example(data,
                       dataset_directory,
                       image_directory,
                       label_map_dict,
                       ignore_difficult_instances=False):   
    """Convert XML derived dict to tf.Example proto.

    Notice that this function normalizes the bounding box coordinates provided
    by the raw data.

    Args:
        data: dict holding PASCAL XML fields for a single image (obtained by
          running dataset_util.recursive_parse_xml_to_dict)
        label_map_dict: A map from string label names to integers ids.
        ignore_difficult_instances: Whether to skip difficult instances in the
          dataset  (default: False).

    Returns:
        example: The converted tf.Example.

    Raises:
        ValueError: if the image pointed to by data['filename'] is not a valid JPEG
    """
    img_path = os.path.join(dataset_directory, image_directory, data['filename'])
    with tf.gfile.GFile(img_path, 'rb') as fid:
        encoded_jpg = fid.read()
    encoded_jpg_io = io.BytesIO(encoded_jpg)
    image = PIL.Image.open(encoded_jpg_io)
    if image.format != 'JPEG':
        raise ValueError('Image format not JPEG')
    key = hashlib.sha256(encoded_jpg).hexdigest()

    width = int(data['size']['width'])
    height = int(data['size']['height'])

    xmin = []
    ymin = []
    xmax = []
    ymax = []
    classes = []
    classes_text = []
    truncated = []
    poses = []
    difficult_obj = []
    for obj in data['object']:
        difficult = bool(int(obj['difficult']))
        if ignore_difficult_instances and difficult:
            continue

        difficult_obj.append(int(difficult))

        xmin.append(float(obj['bndbox']['xmin']) / width)
        ymin.append(float(obj['bndbox']['ymin']) / height)
        xmax.append(float(obj['bndbox']['xmax']) / width)
        ymax.append(float(obj['bndbox']['ymax']) / height)
        classes_text.append(obj['name'].encode('utf8'))
        classes.append(label_map_dict[obj['name']])
        truncated.append(int(obj['truncated']))
        poses.append(obj['pose'].encode('utf8'))

    example = tf.train.Example(features=tf.train.Features(feature={
        'image/height': dataset_util.int64_feature(height),
        'image/width': dataset_util.int64_feature(width),
        'image/filename': dataset_util.bytes_feature(
             data['filename'].encode('utf8')),
        'image/source_id': dataset_util.bytes_feature(
             data['filename'].encode('utf8')),
        'image/key/sha256': dataset_util.bytes_feature(key.encode('utf8')),
        'image/encoded': dataset_util.bytes_feature(encoded_jpg),
        'image/format': dataset_util.bytes_feature('jpeg'.encode('utf8')),
        'image/object/bbox/xmin': dataset_util.float_list_feature(xmin),
        'image/object/bbox/xmax': dataset_util.float_list_feature(xmax),
        'image/object/bbox/ymin': dataset_util.float_list_feature(ymin),
        'image/object/bbox/ymax': dataset_util.float_list_feature(ymax),
        'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
        'image/object/class/label': dataset_util.int64_list_feature(classes),
        'image/object/difficult': dataset_util.int64_list_feature(difficult_obj),
        'image/object/truncated': dataset_util.int64_list_feature(truncated),
        'image/object/view': dataset_util.bytes_list_feature(poses),
    }))
    return example

def main(_):
    data_dir = FLAGS.data_dir
    writer = tf.python_io.TFRecordWriter(FLAGS.output_path)

    label_map_dict = label_map_util.get_label_map_dict(FLAGS.label_map_path)
    logging.info('Reading from dataset.')
    images_dir = os.path.join(data_dir, FLAGS.images_dir)
    images_path = os.listdir(images_dir)
    annotations_dir = os.path.join(data_dir, FLAGS.annotations_dir)
    examples_list = [os.path.splitext(x)[0] for x in images_path]
    for idx, example in enumerate(examples_list):
        if idx % 10 == 0:
            logging.info('On image %d of %d', idx, len(examples_list))
        path = os.path.join(annotations_dir, example + '.xml')
        with tf.gfile.GFile(path, 'r') as fid:
            xml_str = fid.read()
        xml = etree.fromstring(xml_str)
        data = dataset_util.recursive_parse_xml_to_dict(xml)['annotation']

        tf_example = dict_to_tf_example(data, FLAGS.data_dir, FLAGS.images_dir, label_map_dict,
                                        FLAGS.ignore_difficult_instances)
        writer.write(tf_example.SerializeToString())

    writer.close()

 if __name__ == '__main__':
    tf.app.run()

通過tf.app來傳入外部引數
通過tf.train.example來把資料導成tf_example，然後序列化寫入tfrecords檔案
執行主要是5個引數

# From tensorflow/models/research/
python object_detection/dataset_tools/create_disk_tf_record.py \
> --data_dir=/home/icepoint/Downloads/datas/datas/ \
> --images_dir=img/ \
> --annotations_dir=xml/ \
> --output_path=/home/icepoint/Downloads/datas/train.tfrecords \
> --label_map_path=/home/icepoint/Downloads/datas/disk_label_map.pbtxt 

train.py
執行檔案(前面加以下指定的裝置，以防報錯)

$ CUDA_VISIBLE_DEVICE=0 python object_detection/train.py \
--logtostderr \
--train_dir=/home/icepoint/Downloads/datas/ \
--pipeline_config_path=/home/icepoint/tensorflow/models/research/object_detection/samples/configs/faster_rcnn_resnet101_pets.config 

指定訓練目錄，之後會把一系列訓練好的檔案存在那個目錄上
指定配置檔案
配置檔案
faster_rcnn_resnet101_pets.config

# Faster R-CNN with Resnet-101 (v1) configured for the Oxford-IIIT Pet Dataset.
# Users should configure the fine_tune_checkpoint field in the train config as
# well as the label_map_path and input_path fields in the train_input_reader and
# eval_input_reader. Search for "PATH_TO_BE_CONFIGURED" to find the fields that
# should be configured.
model {
  faster_rcnn {
    num_classes: 37
    image_resizer {
      keep_aspect_ratio_resizer {
        min_dimension: 600
        max_dimension: 1024
      }
    }
    feature_extractor {
      type: 'faster_rcnn_resnet101'
      first_stage_features_stride: 16
    }
    first_stage_anchor_generator {
      grid_anchor_generator {
        scales: [0.25, 0.5, 1.0, 2.0]
        aspect_ratios: [0.5, 1.0, 2.0]
        height_stride: 16
        width_stride: 16
      }
    }
    first_stage_box_predictor_conv_hyperparams {
      op: CONV
      regularizer {
        l2_regularizer {
          weight: 0.0
        }
      }
      initializer {
        truncated_normal_initializer {
          stddev: 0.01
        }
      }
    }
    first_stage_nms_score_threshold: 0.0
    first_stage_nms_iou_threshold: 0.7
    first_stage_max_proposals: 300
    first_stage_localization_loss_weight: 2.0
    first_stage_objectness_loss_weight: 1.0
    initial_crop_size: 14
    maxpool_kernel_size: 2
    maxpool_stride: 2
    second_stage_box_predictor {
      mask_rcnn_box_predictor {
        use_dropout: false
        dropout_keep_probability: 1.0
        fc_hyperparams {
          op: FC
          regularizer {
            l2_regularizer {
              weight: 0.0
            }
          }
          initializer {
            variance_scaling_initializer {
              factor: 1.0
              uniform: true
              mode: FAN_AVG
            }
          }
        }
      }
    }
    second_stage_post_processing {
      batch_non_max_suppression {
        score_threshold: 0.0
        iou_threshold: 0.6
        max_detections_per_class: 100
        max_total_detections: 300
      }
      score_converter: SOFTMAX
    }
    second_stage_localization_loss_weight: 2.0
    second_stage_classification_loss_weight: 1.0
  }
}

train_config: {
  batch_size: 1
  optimizer {
    momentum_optimizer: {
      learning_rate: {
        manual_step_learning_rate {
          initial_learning_rate: 0.0003
          schedule {
            step: 0
            learning_rate: .0003
          }
          schedule {
            step: 900000
            learning_rate: .00003
          }
          schedule {
            step: 1200000
            learning_rate: .000003
          }
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false
  }
  gradient_clipping_by_norm: 10.0
  fine_tune_checkpoint: "/home/icepoint/Downloads/faster_rcnn_resnet101_coco_2017_11_08/model.ckpt"
  from_detection_checkpoint: true
  # Note: The below line limits the training process to 200K steps, which we
  # empirically found to be sufficient enough to train the pets dataset. This
  # effectively bypasses the learning rate schedule (the learning rate will
  # never decay). Remove the below line to train indefinitely.
  num_steps: 200000
  data_augmentation_options {
    random_horizontal_flip {
    }
  }
}

train_input_reader: {
  tf_record_input_reader {
    input_path: "/home/icepoint/Downloads/datas/train.tfrecords"
  }
  label_map_path: "/home/icepoint/Downloads/datas/disk_label_map.pbtxt"
}

eval_config: {
  num_examples: 2000
  # Note: The below line limits the evaluation process to 10 evaluations.
  # Remove the below line to evaluate indefinitely.
  max_evals: 10
}

eval_input_reader: {
  tf_record_input_reader {
    input_path: "/home/icepoint/Downloads/datas/train.tfrecords"
  }
  label_map_path: "/home/icepoint/Downloads/datas/disk_label_map.pbtxt"
  shuffle: false
  num_readers: 1
}

需要修改一下train_config： fine_tune_checkpoint為下載的coco資料集中model.ckpt檔案，num_steps迭代次數
需要修改train_input_reader：input_path表示輸入train.tfrecords的檔案路徑，label_map_path表示類別檔案路徑
需要修改eval_input_reader：input_path與label_map_path

訓練過程中可能比較耗時，或者耗費資源

當自動儲存model.ckpt檔案時就可以終止訓練
訓練時訓練目錄下會有

export_inference_graph.py

轉換model.ckpt為pb檔案
首先需要把train_dir下的model.ckpt-xxx.*三個檔案+checkpoint檔案，複製到train_checkpoint_prefix目錄下
重新命名把model.ckpt-xxx的xxx去掉
修改checkpoint裡的路徑內容

/Downloads/datas/ckpt/

執行

python export_inference_graph \
    --input_type image_tensor \
    --pipeline_config_path /home/user/tensorflow/models/research/object_detection/samples/configs/faster_rcnn_resnet101_pets.config \
    --trained_checkpoint_prefix /home/user/Downloads/datas/ckpt/model.ckpt \
    --output_directory /home/user/Downloads/datas/ckpt/

Note:The expected output would be in the directory
path/to/exported_model_directory (which is created if it does not exist)
with contents:
- graph.pbtxt
- model.ckpt.data-00000-of-00001
- model.ckpt.info
- model.ckpt.meta
- frozen_inference_graph.pb
+ saved_model (a directory)

注意執行時可能會報錯：

 ValueError: Protocol message RewriterConfig has no "layout_optimizer" field.

推測可能是tensorflow臨時commit的bug
解決：開啟object_detection/exporter.py，將layout_optimizer字樣修改為optimize_tensor_layout字樣（函式名）即可
匯出後會生成frozen_inference_graph.pb用於資料檢測

測試工作

通過source, export PYTHONPATH開啟jupyter notebook
開啟object_detection_disk_dataset.ipynb
(具體參照object_detection/object_detection_tutorial.ipynb)

測試結果：