Thursday, October 31, 2019

How to do Real-time image segmentation with Mask RCNN?

Mask RCNN is Faster RCNN but with a mask, so Faster RCNN is an object detection algorithm that's pretty similar to Yolo, It's giving us bounding boxes, object labels, confident factor all those things we're used to seeing but we are also adding the mask so we are able to label all the pixels that belong to each object with a mask.

Real-time image segmentation

In this article, I am focusing on how to get set up using this GitHub repo that implements a Mask RCNN. So, we are gonna look at how to download install and test everything and get it running our system in real-time. So let's get started.

So now let's talk more about what Mask RCNN is? like I mentioned before Mask RCNN is taking Faster RCNN and building on top of it by adding a mask. Faster RCNN is doing the object detection, object localization, it's giving us bounding boxes, object labels, the confidence factors. So we are taking an object detection algorithm by adding a mask feature to it. and what mask does is it labels each pixel. Does this pixel belong to that object or not?

It was developed by the Facebook AI research group and they have written a paper on it. You can read that paper with this link. You can also see the 13minute overview from the creator on Youtube with this link.

Now how to download and install a particular implementation of Mask RCNN. So, the one we are gonna be using is from this group called matterport and it's called Mask RCNN and what's nice about this one is it's built using TensorFlow and Keras. So, you don't need to download any deep learning frameworks.

Overview on how to install 

Step 1: Create a conda virtual environment with python 3.6
Step 2: Install all required dependencies
Step 3: Clone the Mask_RCNN repository
Step 4: Install pycocotools
Step 5: Download the pre-trained weights
Step 6: Clone real-time image segmentation code
Step 7: Error solving
Step 8: Test it!

Step 1- Create a conda virtual environment

We will be using Anaconda with python 3.6.
run this command in cmd/anaconda prompt to create a virtual environment
conda create -n mrcnn python=3.6 pip

This is not a compulsory step but this is a good practice to make a virtual environment.

Step 2- Install all required dependencies

Open anaconda prompt and activate your environment by run activate mrcnn command.
We need to install these libraries numpy, scipy, cython, h5py, Pillow, scikit-image, tensorflow-gpu==1.8, keras, jupyter, spyder, opencv-python, matplotlib.

Dependencies installation

Step 3- Clone the Mask_RCNN repository

Now we are going to clone this repository by using git. We need to just run a command on anaconda prompt/command prompt. See the figure to follow the steps.

MaskRCNN installation

Now you are done with cloning.

Step 4- Install pycocotools

Now, We are going to install pycocotools for that we need to make sure Visual C++ 2017 build tools are installed so if you need to install those you can just follow this link. After successfully setup Visual C++ 2017, you need to install pycocotools by writing this command to anaconda prompt /command prompt.
pip install git+

pycocotool installation

Step 5- Download the Pre-trained weights

To download pre-trained weights for our model just go to this link and download mask_rcnn_coco.h5 file and we need to just place it in the main(Mask_RCNN) folder or directory.

Step 6- Clone real-time image segmentation code

To download real-time image segmentation code clone this repository and copy realtime_segment code then paste it into main(Mask_RCNN) directory.

Step 7- Error solving

After completion of all the steps when you run python you will get  an error of imgaug to deal with this error you need to do following steps:
I will use my bases environment in which  nothing is setup because in my mrcnn environment everything is already setup

  • Download the Shapely wheel file with this link.
  • Copy that wheel file from the Download folder and then paste it to the main(Mask_RCNN) folder.
  • Now run this command pip install Shapely-1.6.4.post2-cp36-cp36m-win_amd64.whl
imgaug installation

  • After this install imgaug by run this command pip install imgaug
  • After installation is completed you find opencv-python-headless is installed, it will create a problem while opening a camera. So you need to delete this.
imgaug installation

  • To delete opencv-python-headless you need to go to site-packages of your virtual environment and there you can find opencv-python-headless now you can easily delete this.
  • Now your main(Mask_RCNN) directory look like this:
Look of folder

  • After these all steps, you are all set to run python
  • If again you find an error then check the port of camera or you can post your query in the comment section.

Step 8- Test it!

Run python to test this. Then you find output like this.

Real-time image segmentation

I hope you have done all the steps correctly. For any query write your valuable comments in the comment section.