完整项目已上传至github:End-to-end-for-chinese-plate-recognition/License-plate-recognition at master · duanshengliu/End-to-end-for-chinese-plate-recognition · GitHub
整体思路:
1.利用u-net图像分割得到二值化图像
2.再使用cv2进行边缘检测获得车牌区域坐标,并将车牌图形矫正
3.利用卷积神经网络cnn进行车牌多标签端到端识别
环境:python3.11 tensorflow opencv4.6
1.车牌定位
首先贴一下图像分割的效果图:
我们可以通过图像分割算法对一张输入图片进行分割,分割后的图形其实是对原图中的区域进行的分类标注,例如这里我们可以将原图标注为2类,一类就是车牌区域,还有一类就是无关的背景区域。说到标注图形就不得不说labelme了,我们可以在cmd界面通过命令 pip install labelme 进行labelme库的安装,安装结束在cmd界面输入labelme即可打开lablem软件的标注界面如下
1. 点击OpenDir ,选择我们准备好的车辆数据集(注意:一定要先把图片全都resize为训练时所需的大小,再进行标注。我们知道图片数据的范围是0-255,背景为黑色0,车牌区域为255,我们需要的是标注好的图片即img_mask中值只有{0,255}这2种,如果我们不先resize,标注完再resize会导致一个大问题,就是数据的值并不是二类,会出现{0,1,10,248,251,255}等类似的多值问题,我在之前就遇到这样的问题,不得已又重新标注了300多张图)
2. 点击左上角File—>将Save Automatically勾选上,点击Change Output Dir选择保存路径,我这里是在桌面D:/desktop/下新建了一个文件夹命名为labelme,在labelm文件夹中新建了一个json文件夹用于保存我们标注的json数据,这里我们Change Output Dir的保存路径就选它,还新建了一个data文件夹用于存放后续转换的图片数据,而待标注图片在pic文件中,存放的都是resize好的512×512的图片,命名格式最好像我这样
3. 准备好上述一切就可以开始标注了,点击软件左侧的 这是画任意多边形的按钮,鼠标左键点击进行标注,最后双击鼠标左键会锁定标注区域,出现如下图界面,第一次标注需输入名称,后续标注就自动显示了,点击ok后标注的线条变为红色,同时json文件夹也会相应保存和pic名字对应的json文件
全部标注结束后,使用如下代码将json数据提取出来并保存到train_image和train_label文件夹中,u-net部分的数据集原作者一共标注了1200多张,最终效果很棒,达到了定位的效果
标注好的图片进行分类
把路径改为自己的路径
import os
import cv2
import numpy as np
#将json文件label转换为到data文件夹
n=1200#n为总共标注的图片数
for i in range(n):
os.system('labelme_json_to_dataset D:/desktop/labelme/json/%d.json -o D:/desktop/labelme/data/%d_json'%(i,i))
#dst_w=512
#dst_h=512
#dst_shape=(dst_w,dst_h,3)
train_image = 'D:/desktop/labelme/train_image/'
if not os.path.exists(train_image):
os.makedirs(train_image)
train_label = 'D:/desktop/labelme/train_label/'
if not os.path.exists(train_label):
os.makedirs(train_label)
for i in range(n):
print(i)
img=cv2.imread('D:/desktop/labelme/data/%d_json/img.png'%i)
label=cv2.imread('D:/desktop/labelme/data/%d_json/label.png'%i)
print(img.shape)
label=label/np.max(label[:,:,2])*255
label[:,:,0]=label[:,:,1]=label[:,:,2]
print(np.max(label[:,:,2]))
# cv2.imshow('l',label)
# cv2.waitKey(0)
print(set(label.ravel()))
cv2.imwrite(train_image+'%d.png'%i,img)
cv2.imwrite(train_label+'%d.png'%i,label)
这样一来,标注好的u-net训练图片就准备好了,分别在train_image和train_label文件夹中,一并放在unet_datasets文件夹内,如下图所示
接下来是u-net模型搭建和训练,使用tensorflow的keras实现,贴一下我训练u-net用的代码
def unet_train():
height = 512
width = 512
path = 'D:/desktop/unet_datasets/'
input_name = os.listdir(path + 'train_image')
n = len(input_name)
print(n)
X_train, y_train = [], []
for i in range(n):
print("正在读取第%d张图片" % i)
img = cv2.imread(path + 'train_image/%d.png' % i)
label = cv2.imread(path + 'train_label/%d.png' % i)
X_train.append(img)
y_train.append(label)
X_train = np.array(X_train)
y_train = np.array(y_train)
def Conv2d_BN(x, nb_filter, kernel_size, strides=(1, 1), padding='same'):
x = layers.Conv2D(nb_filter, kernel_size, strides=strides, padding=padding)(x)
x = layers.BatchNormalization(axis=3)(x)
x = layers.LeakyReLU(alpha=0.1)(x)
return x
def Conv2dT_BN(x, filters, kernel_size, strides=(2, 2), padding='same'):
x = layers.Conv2DTranspose(filters, kernel_size, strides=strides, padding=padding)(x)
x = layers.BatchNormalization(axis=3)(x)
x = layers.LeakyReLU(alpha=0.1)(x)
return x
inpt = layers.Input(shape=(height, width, 3))
conv1 = Conv2d_BN(inpt, 8, (3, 3))
conv1 = Conv2d_BN(conv1, 8, (3, 3))
pool1 = layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same')(conv1)
conv2 = Conv2d_BN(pool1, 16, (3, 3))
conv2 = Conv2d_BN(conv2, 16, (3, 3))
pool2 = layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same')(conv2)
conv3 = Conv2d_BN(pool2, 32, (3, 3))
conv3 = Conv2d_BN(conv3, 32, (3, 3))
pool3 = layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same')(conv3)
conv4 = Conv2d_BN(pool3, 64, (3, 3))
conv4 = Conv2d_BN(conv4, 64, (3, 3))
pool4 = layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same')(conv4)
conv5 = Conv2d_BN(pool4, 128, (3, 3))
conv5 = layers.Dropout(0.5)(conv5)
conv5 = Conv2d_BN(conv5, 128, (3, 3))
conv5 = layers.Dropout(0.5)(conv5)
convt1 = Conv2dT_BN(conv5, 64, (3, 3))
concat1 = layers.concatenate([conv4, convt1], axis=3)
concat1 = layers.Dropout(0.5)(concat1)
conv6 = Conv2d_BN(concat1, 64, (3, 3))
conv6 = Conv2d_BN(conv6, 64, (3, 3))
convt2 = Conv2dT_BN(conv6, 32, (3, 3))
concat2 = layers.concatenate([conv3, convt2], axis=3)
concat2 = layers.Dropout(0.5)(concat2)
conv7 = Conv2d_BN(concat2, 32, (3, 3))
conv7 = Conv2d_BN(conv7, 32, (3, 3))
convt3 = Conv2dT_BN(conv7, 16, (3, 3))
concat3 = layers.concatenate([conv2, convt3], axis=3)
concat3 = layers.Dropout(0.5)(concat3)
conv8 = Conv2d_BN(concat3, 16, (3, 3))
conv8 = Conv2d_BN(conv8, 16, (3, 3))
convt4 = Conv2dT_BN(conv8, 8, (3, 3))
concat4 = layers.concatenate([conv1, convt4], axis=3)
concat4 = layers.Dropout(0.5)(concat4)
conv9 = Conv2d_BN(concat4, 8, (3, 3))
conv9 = Conv2d_BN(conv9, 8, (3, 3))
conv9 = layers.Dropout(0.5)(conv9)
outpt = layers.Conv2D(filters=3, kernel_size=(1, 1), strides=(1, 1), padding='same', activation='relu')(conv9)
model = models.Model(inpt, outpt)
model.compile(optimizer='adam',
loss='mean_squared_error',
metrics=['accuracy'])
model.summary()
print(np.max(X_train))
print(np.max(y_train))
print(X_train.shape)
model.fit(X_train, y_train, epochs=100, batch_size=15)#epochs和batch_size看个人情况调整,batch_size不要过大,否则内存容易溢出
#我11G显存也只能设置15-20左右,我训练最终loss降低至250左右,acc约95%左右
model.save('unet.h5')
print('unet.h5保存成功!!!')
上述代码关键部分是要获取车牌四边形的四个顶点,一开始只使用cont中坐标到外接矩形四个端点的距离,发现对于倾斜度很高的车牌效果可能不佳,见下图,可以观察到,计算得到的4个黄色坐标中,左右有2个黄色点并不处在四边形的顶点位置,这样矫正效果大打折扣,同时也会影响后续的识别效果
发现上述问题后,我又想了个方法就是加入了上述的point_to_line_distance函数,即还计算坐标点到上下两条边的距离,并添加了权重,经过调整权重设置为0.975倍的点线距离,0.025点到端点距离时整体效果较佳,最终矫正效果如下图:
矫正效果大大改善后,识别率也将大大提高。
最终运行后上述代码后,提取的license文件夹中的车牌图如下:
2.车牌识别
我们的输入图片就是上述的宽240,高80的车牌图片,要实现车牌的端到端识别,显然是多标签分类问题,每张输入图片有7个标签,模型输出前的结构都是可以共享的,只需将输出修改为7个即可,7个输出对应了7个loss,总loss就是7个loss的和,使用keras可以很方便地实现,训练cnn的代码如下
def cnn_train():
char_dict = {"京": 0, "沪": 1, "津": 2, "渝": 3, "冀": 4, "晋": 5, "蒙": 6, "辽": 7, "吉": 8, "黑": 9, "苏": 10,
"浙": 11, "皖": 12, "闽": 13, "赣": 14, "鲁": 15, "豫": 16, "鄂": 17, "湘": 18, "粤": 19, "桂": 20,
"琼": 21, "川": 22, "贵": 23, "云": 24, "藏": 25, "陕": 26, "甘": 27, "青": 28, "宁": 29, "新": 30,
"0": 31, "1": 32, "2": 33, "3": 34, "4": 35, "5": 36, "6": 37, "7": 38, "8": 39, "9": 40,
"A": 41, "B": 42, "C": 43, "D": 44, "E": 45, "F": 46, "G": 47, "H": 48, "J": 49, "K": 50,
"L": 51, "M": 52, "N": 53, "P": 54, "Q": 55, "R": 56, "S": 57, "T": 58, "U": 59, "V": 60,
"W": 61, "X": 62, "Y": 63, "Z": 64}
# 读取数据集
path = 'home/cnn_datasets/' # 车牌号数据集路径(车牌图片宽240,高80)
pic_name = sorted(os.listdir(path))
n = len(pic_name)
X_train, y_train = [], []
for i in range(n):
print("正在读取第%d张图片" % i)
img = cv2.imdecode(np.fromfile(path + pic_name[i], dtype=np.uint8), -1) # cv2.imshow无法读取中文路径图片,改用此方式
label = [char_dict[name] for name in pic_name[i][0:7]] # 图片名前7位为车牌标签
X_train.append(img)
y_train.append(label)
X_train = np.array(X_train)
y_train = [np.array(y_train)[:, i] for i in range(7)] # y_train是长度为7的列表,其中每个都是shape为(n,)的ndarray,分别对应n张图片的第一个字符,第二个字符....第七个字符
# cnn模型
Input = layers.Input((80, 240, 3)) # 车牌图片shape(80,240,3)
x = Input
x = layers.Conv2D(filters=16, kernel_size=(3, 3), strides=1, padding='same', activation='relu')(x)
x = layers.MaxPool2D(pool_size=(2, 2), padding='same', strides=2)(x)
for i in range(3):
x = layers.Conv2D(filters=32 * 2 ** i, kernel_size=(3, 3), padding='valid', activation='relu')(x)
x = layers.Conv2D(filters=32 * 2 ** i, kernel_size=(3, 3), padding='valid', activation='relu')(x)
x = layers.MaxPool2D(pool_size=(2, 2), padding='same', strides=2)(x)
x = layers.Dropout(0.5)(x)
x = layers.Flatten()(x)
x = layers.Dropout(0.3)(x)
Output = [layers.Dense(65, activation='softmax', name='c%d' % (i + 1))(x) for i in range(7)] # 7个输出分别对应车牌7个字符,每个输出都为65个类别类概率
model = models.Model(inputs=Input, outputs=Output)
model.summary()
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy', # y_train未进行one-hot编码,所以loss选择sparse_categorical_crossentropy
metrics=['accuracy'])
# 模型训练
print("开始训练cnn")
model.fit(X_train, y_train, epochs=35) # 总loss为7个loss的和
model.save('cnn.h5')
print('cnn.h5保存成功!!!')
最终,训练集上准确率acc1(即车牌省份字符)为97%,其余字符均为99%左右,本地测试集准确率为97%,识别效果较佳。
最后放一下整体的效果图
转载于