rk3399使用阿里推理引擎MNN使用cpu和gpu进行benchmark，OpenCL效果不佳？

视频讲解

背景

MNN是阿里开源的推理引擎，今天测试一下在rk3399平台上的benchmark怎么样？
alibaba/MNN: MNN is a blazing fast, lightweight deep learning framework, battle-tested by business-critical use cases in Alibaba (github.com)

首先git clone

git clone git@github.com:alibaba/MNN.git

创建build目录

cd MNN
mkdir build
cd build

cmake配置

注意交叉编译器以及opencl库的使用方式，是使用系统opencl库还是使用wrap进行dlopen加载

cmake .. \
-DCMAKE_BUILD_TYPE=Release \
-DMNN_BUILD_DEMO=ON \
-DMNN_BUILD_BENCHMARK=true \
-DCMAKE_SYSTEM_NAME=Linux \
-DCMAKE_SYSTEM_VERSION=1 \
-DCMAKE_SYSTEM_PROCESSOR=aarch64 \
-DMNN_OPENCL=ON \
-DMNN_USE_SYSTEM_LIB=ON \
-DCMAKE_C_COMPILER=${cross_compile_toolchain}/bin/aarch64-linux-gnu-gcc \
-DCMAKE_CXX_COMPILER=${cross_compile_toolchain}/bin/aarch64-linux-gnu-g++

make -j32

部署

然后将build目录下的libMNN.so以及benchmark.out和上级目录下的benchmark的model放到一起，同时libMNN.so需要放到rk3399的lib目录下

sudo cp libMNN.so /lib
sudo cp -rf ../benchmark/model .

然后运行benchmark测试，第二个参数：loop测试次数，第4个参数：0代表使用cpu，3代表使用opencl

cpu测试

firefly@firefly:~/MNN$ sudo ./benchmark.out models/ 1 0 0clear
MNN benchmark
Forward type: CPU thread=4 precision=2 sparsity=0 sparseBlockOC=1 testQuantizedModel=0
--------> Benchmarking... loop = 1, warmup = 0
[-INFO-]: precision=2, use fp16 inference if your device supports and open MNN_ARM82=ON.
The device support i8sdot:0, support fp16:0, support i8mm: 0
[ - ] SqueezeNetV1.0.mnn          max =   86.128 ms  min =   86.128 ms  avg =   86.128 ms
[ - ] MobileNetV2_224.mnn         max =   42.041 ms  min =   42.041 ms  avg =   42.041 ms
[ - ] inception-v3.mnn            max =  505.111 ms  min =  505.111 ms  avg =  505.111 ms
[ - ] mobilenetV3.mnn             max =   13.533 ms  min =   13.533 ms  avg =   13.533 ms
[ - ] nasnet.mnn                  max =  145.489 ms  min =  145.489 ms  avg =  145.489 ms
[ - ] mobilenet-v1-1.0.mnn        max =   66.624 ms  min =   66.624 ms  avg =   66.624 ms
[ - ] squeezenetv1.1.mnn          max =   40.437 ms  min =   40.437 ms  avg =   40.437 ms
[ - ] resnet-v2-50.mnn            max =  308.836 ms  min =  308.836 ms  avg =  308.836 ms

gpu测试

firefly@firefly:~/MNN$ sudo ./benchmark.out models/ 1 0 3
MNN benchmark
Forward type: OpenCL thread=4 precision=2 sparsity=0 sparseBlockOC=1 testQuantizedModel=0
--------> Benchmarking... loop = 1, warmup = 0
[-INFO-]: precision=2, use fp16 inference if your device supports and open MNN_ARM82=ON.
The device support i8sdot:0, support fp16:0, support i8mm: 0
arm_release_ver of this libmali is 'r18p0-01rel0', rk_so_ver is '4'.[ - ] SqueezeNetV1.0.mnn          max =  159.619 ms  min =  159.619 ms  avg =  159.619 ms
[ - ] MobileNetV2_224.mnn         max =  126.671 ms  min =  126.671 ms  avg =  126.671 ms
[ - ] inception-v3.mnn            max =  800.436 ms  min =  800.436 ms  avg =  800.436 ms
[ - ] mobilenetV3.mnn             max =   61.661 ms  min =   61.661 ms  avg =   61.661 ms
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
[ - ] nasnet.mnn                  max =  140.189 ms  min =  140.189 ms  avg =  140.189 ms
[ - ] mobilenet-v1-1.0.mnn        max =   98.918 ms  min =   98.918 ms  avg =   98.918 ms
[ - ] squeezenetv1.1.mnn          max =  121.158 ms  min =  121.158 ms  avg =  121.158 ms
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
Map error scalePtrCL == nullptr
Map error biasPtrCL == nullptr
[ - ] resnet-v2-50.mnn            max =  428.075 ms  min =  428.075 ms  avg =  428.075 ms

结论

可以看到，gpu使用上很慢且存在算子的问题，实际上在rk3568上测试opencl很流畅且没有问题，这里留下问题，之后探究