UAP in Action: Running OpenCV on Raspberry Pi II

clip_image002Microsoft Windows 10 offers the promise to build applications once that will run on every device. Leveraging our contributions to OpenCV for Modern Windows , we have also worked with the product engineering team to apply this work to Win10. With this last branch (vs2015-samples) on the MS Open Tech GitHub repo, we allow developers to build OpenCV for ARM, in the Win10 contest.

Getting started

You will need the following pre-installed to run this demo:

· Windows 10 (I run the 10074 version)

· Visual Studio 2015 RC

· Visual Studio 2015 Tools for Windows 10

You can access these tools here and here

Once you have setup in place, you will be able to build Universal Applications for Win10. This means that rather than the case with Windows 8.1 (one for phone and one for Windows with shared source), you will now be able to build only one project that extends to potentially several targets. For more information about Windows 10 Universal Windows Platform functionality, have a look at this excellent MSDN article )

Setting Up Raspberry PI II with Windows 10

clip_image004Raspberry Pi II can run the Windows 10 IoT Core. Follow these steps to setup it up correctly

The archive Windows 10 IoT Core Insider Preview Image for Raspberry Pi 2 that you used to set up Raspberry Pi II also contains a very useful tool (Windows IoT Core Watcher) that will allows you to find your board on the network and connect to it through your browser. While it is optional for purposes of today’s demo, I highly recommend that you use it, as it is the only thing you need to deploy your OpenCV Application.

 

 

Building OpenCV Libraries for ARM

Before starting to code the apps, you will first need to build the OpenCV libraries and dlls for ARM. We (MS Open Tech) created a special branch for that purpose. At the time of this writing, this branch is not yet merged to the main OpenCV repo, but we plan to make the pull request soon…in the meantime, it can be accessed from our OpenCV repo.

So, I suggest that you clone the MS Open Tech GitHub repo.

As explained in the readme file, you will need to create the environment variable needed for Visual Studio 2015 and Winrt Sample.

You will then open the OpenCV solution itself located [vs2015\WS\10.0\ARM] and build it for ARM. The build process will generate all of the needed libs and dlls and place them in these folders [lib and bin in vs2015\WS\10.0\ARM\]. Note: This is the directory path you will need to reference to build your OpenCV application.

Build a Modern OpenCV Application

First, create a new solution with Visual Studio 2015, Universal App for Win10:

clip_image006

Add this path to the libs you just generated. You will need to reference the OpenCV dlls in your project (right click on the solution) and add [opencv_core300d.dll, opencv_imgproc300d.dll ….]

clip_image008

As the plan is to deploy your application on Raspberry Pi II, you should select the ARM Target and Remote Machine (we’ll go over how to configure it later):

clip_image009

For demo purposes, I have displayed a simple image on the screen. This image will be stored in the assets folder (for simplicity). You can add an image to the asset folder under your solution, and then add it in to asset folder of the solution (right click on asset folder in Visual studio).

 

Next, add three buttons and an image control on the XAML definition of your screen:

Then add the code within the first button click handler, to load the image:

cv::Mat image = cv::imread("Assets/grpPC1.jpg");

Lena = cv::Mat(image.rows, image.cols, CV_8UC4);

cv::cvtColor(image, Lena, CV_BGR2BGRA);

UpdateImage(Lena);

Lena is a cv::Mat declared in the header.

The UpdateImage is a method that will display the cv::Mat in the Image control

You will next add a second button to test the Canny filter with this code:

//Canny filter

cv::Mat result;

cv::Mat intermediateMat;

cv::Canny(Lena, intermediateMat, 80, 90);

cv::cvtColor(intermediateMat, result, CV_GRAY2BGRA);

UpdateImage(result);

Finally, you will add a third button to test the Face and Body detection using this code:

cv::String face_cascade_name = "Assets/haarcascade_frontalface_alt.xml";

cv::CascadeClassifier face_cascade;

cv::String body_cascade_name = "Assets/haarcascade_fullbody.xml";

cv::CascadeClassifier body_cascade;

void internalDetectObjects(cv::Mat& inputImg, std::vector<cv::Rect> & objectVector, std::vector<cv::Rect> & objectVectorBodies)

{

cv::Mat frame_gray;

cvtColor(inputImg, frame_gray, CV_BGR2GRAY);

cv::equalizeHist(frame_gray, frame_gray);

// Detect faces

face_cascade.detectMultiScale(frame_gray, objectVector, 1.1, 2, 0 | CV_HAAR_SCALE_IMAGE, cv::Size(30, 30));

//detect bodies

body_cascade.detectMultiScale(frame_gray, objectVectorBodies, 1.1, 2, 0 | CV_HAAR_SCALE_IMAGE, cv::Size(30, 300));

}

void RaspberryCV::MainPage::btn3_Click(Platform::Object^ sender, Windows::UI::Xaml::RoutedEventArgs^ e)

{

if (!face_cascade.load(face_cascade_name)) {

printf("Couldnt load Face detector '%s'\n", face_cascade_name);

exit(1);

}

if (!body_cascade.load(body_cascade_name)) {

printf("Couldnt load Body detector '%s'\n", body_cascade_name);

exit(1);

}

cv::Mat frame = cv::imread("Assets/grpPC1.jpg");

if (frame.empty())

return;

std::vector<cv::Rect> faces;

std::vector<cv::Rect> bodies;

internalDetectObjects(frame, faces, bodies);

for (unsigned int i = 0; i < faces.size(); i++)

{

auto face = faces[i];

cv::rectangle(Lena, face, cv::Scalar(0,0, 255, 255), 5);

}

for (unsigned int i = 0; i < bodies.size(); i++)

{

auto body = bodies[i];

cv::rectangle(Lena, body, cv::Scalar(0,0, 0, 255), 5);

}

UpdateImage(Lena);

}

All the code source outlined above is also available as sample in our repo: https://github.com/MSOpenTech/opencv/tree/vs2015-samples/samples/winrt_universal under the Raspberry folder and solution.

Deploying and Running the Application

Once you have built the solution, it will be time to deploy and run it on Raspberry PI II.

Boot Raspberry Pi II with a screen connected and a network. In the start screen, you will find the related IP address:

clip_image011

Next, you will configure Visual Studio to target Raspberry. Right click on the RaspberryCV Property Pages to set the IP address and select No when prompted whether to Require Authentication:

clip_image013

When you then click the Remote Machine Button, you should obtain the same result as presented below:

You will see it load and display the image:

clip_image015

 

Then apply the Canny filter:

clip_image017

 

And observe it detect the face and body within the image.

clip_image019

 

Try it for Yourself!

You can see with this first simple demo that we can enable OpenCV scenarios on Raspberry Pi II. At the time of this post, camera support is not yet available on Raspberry PI II running Windows 10, but it will be soon to enable a variety of additional scenarios, such as real time video frame analysis…

I encourage you to try this sample out for yourself, and I hope that you will share back about your experiences and/or contributions to enrich this sample.

Happy coding!

A bientôt