1 of 51

Introduction to OpenCV

by Aleksandr Rybnikov, TRA Robotics

arrybn@gmail.com

github.com/arrybn

​

2 of 51

You can view this slides on your device

2

3 of 51

Agenda

  • How OpenCV is organized
  • Basic operations: load, display, manipulate images
  • Simple object detection: color filtering, masks, contours
  • Camera calibration
  • From 2D to 3D
  • ArUCo markers
  • CV ideas
  • Deep Learning, even in OpenCV
  • Python tools
  • Install & build from source code
  • Books
  • Useful links

3

4 of 51

How OpenCV is organized

  • Two main repositories: opencv & opencv_contrib
  • Each repository consists of modules
  • Each module implements some functionality and depends on other modules
  • There are few basic modules

4

5 of 51

Before we start

  • Source code and all data for the slides are available here: https://github.com/arrybn/opencv_intro

5

6 of 51

Basic operations: load, display, manipulate images

Open images, open video, grab frames from a camera

6

import cv2

print(cv2.__version__)

​

img_color = cv2.imread('data/lena.png', cv2.IMREAD_COLOR)

img_gray = cv2.imread('data/lena.png', cv2.IMREAD_GRAYSCALE)

cap = cv2.VideoCapture('data/shuttle.mp4')

ret, frame = cap.read()

cap.release()

cap = cv2.VideoCapture(0)

ret, frame = cap.read()

cap.release()

7 of 51

Basic operations: load, display, manipulate images

Display images

7

cv2.imshow('color', img_color)

cv2.waitKey()

cv2.destroyWindow('color')

cv2.imshow('gray', img_gray)

cv2.waitKey(5000)

cv2.destroyAllWindows()

8 of 51

Basic operations: load, display, manipulate images

Display images

8

cap = cv2.VideoCapture('data/shuttle.mp4')

while True:

ret, frame = cap.read()

cv2.imshow('video', frame)

if cv2.waitKey(30) == ord('e'):

break

cv2.destroyAllWindows()

cap.release()

9 of 51

Basic operations: load, display, manipulate images

Basic draw functions, save results

9

img_color = cv2.circle(img_color, (256, 256),

10, (0, 255, 0), 5)

img_color = cv2.line(img_color, (0, 0),

(1024, 1024), (0, 0, 255), 3)

img_color = cv2.rectangle(img_color, (100, 100),

(250, 350), (255, 0, 0), 3)

cv2.imwrite('result.png', img_color)

10 of 51

Basic operations: load, display, manipulate images

Numpy: access to pixels, channels, ROIs

10

import numpy as np

print(img_color.shape)

print(img_gray.shape)

(512, 512, 3)�(512, 512)

11 of 51

Basic operations: load, display, manipulate images

Numpy: access to pixels, channels, ROIs

11

img_color[256, 256] = (0, 255, 0)

for i in range(100):

for j in range(100):

img_color[i, j] = (0, 255, 0)

cv2.imshow('color', img_color)

cv2.waitKey()

cv2.destroyWindow('color')

​

12 of 51

Basic operations: load, display, manipulate images

Numpy: access to pixels, channels, ROIs

12

img_color[10:90, 10:90] = 0

img_color[400:420, :] = 0

cv2.imshow('color', img_color)

cv2.waitKey()

cv2.destroyWindow('color')

13 of 51

Basic operations: load, display, manipulate images

Numpy: access to pixels, channels, ROIs

13

cv2.imshow('color', img_color[:, :, 0])

cv2.waitKey()

cv2.destroyWindow('color')

​

14 of 51

Basic operations: load, display, manipulate images

Numpy: some useful operations

14

img = img_gray

img[...] = 0 # actually change img_gray

img = np.copy(img_color)

img[...] = 0

black = np.zeros((100, 100, 3), dtype=np.uint8)

white = 255*np.ones((100, 100, 3), dtype=np.uint8)

blue = np.full((100, 100, 3), (255, 0, 0), np.uint8)

white_padded = np.pad(white, ((64,)*2, (64,)*2, (0,)*2), 'constant', constant_values=0)

15 of 51

Basic operations: load, display, manipulate images

Numpy: some useful operations

15

np.save('white.npy', white)

white_loaded = np.load('white.npy')

assert (white == white_loaded).all()

img_h = np.hstack((black, white))

img_v = np.vstack((black, white))

print(np.pi)

​

3.141592653589793

16 of 51

Simple object detection: color filtering, masks, contours

Conversions between color spaces and data types

16

img_bgr = cv2.imread('data/lena.png', cv2.IMREAD_COLOR)

img_rgb = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2RGB)

img_hsv = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2HSV)

img_gray = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2GRAY)

img_rgb_fp32 = img_rgb.astype(np.float32)

cv2.imshow('rgb float', img_rgb_fp32)

cv2.imshow('rgb float/255', img_rgb_fp32 / 255)

cv2.waitKey()

cv2.destroyAllWindows()

17 of 51

Simple object detection: color filtering, masks, contours

Conversions between color spaces and data types

17

18 of 51

Simple object detection: color filtering, masks, contours

Color filtering

18

duck_0 = cv2.imread('data/duck_2.jpg')

duck_0_hsv = cv2.cvtColor(duck_0, cv2.COLOR_BGR2HSV)

dark_yellow = np.array([15, 100, 100])

light_yellow = np.array([50, 255, 255])

mask = cv2.inRange(duck_0_hsv, dark_yellow, light_yellow)

duck_0_masked = np.copy(duck_0)

duck_0_masked[mask == 0] = 0

cv2.imshow('', duck_0_masked)

cv2.waitKey()

cv2.destroyAllWindows()

19 of 51

Simple object detection: color filtering, masks, contours

Color filtering

19

20 of 51

Simple object detection: color filtering, masks, contours

Color filtering

20

21 of 51

Simple object detection: color filtering, masks, contours

Morphology operations

21

kernel_open = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (10, 10))

kernel_close = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (50, 50))

mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel_open)

mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, kernel_close)

duck_0_masked = np.copy(duck_0)

duck_0_masked[mask == 0] = 0

cv2.imshow('', duck_0_masked)

cv2.waitKey()

cv2.destroyAllWindows()

22 of 51

Simple object detection: color filtering, masks, contours

Morphology operations

22

23 of 51

Simple object detection: color filtering, masks, contours

Contours

23

img, contours, hierarchy = cv2.findContours(mask_closed, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

contour_duck_index = max(range(len(contours)), key=lambda i: cv2.arcLength(contours[i], True))

hierarchy = hierarchy.squeeze()

contour_eye_index = -1

for i in range(len(hierarchy)):

if hierarchy[i][2] < 0 and hierarchy[i][3] == contour_duck_index:

contour_eye_index = i

break

duck_0_masked = cv2.drawContours(duck_0_masked, contours, contour_duck_index, (0, 255, 0), 4)

duck_0_masked = cv2.drawContours(duck_0_masked, contours, contour_eye_index, (0, 0, 255), 4)

cv2.imshow('', duck_0_masked)

cv2.waitKey()

cv2.destroyAllWindows()

24 of 51

Simple object detection: color filtering, masks, contours

Contours

24

25 of 51

Camera calibration

Main idea

25

26 of 51

Camera calibration

Main idea

We need to find:

  • Parameters of projection - camera matrix
  • Parameters of distortion - distortion coefficients

Let’s shoot an object with known geometry and try to find such camera matrix and distortion coefficients which minimize error between 2D object points in photos and reprojected 3D object points

26

27 of 51

Camera calibration

How to do

  1. Prepare calibration pattern. It should preserve the geometry during calibration
  2. Fix the zoom and don’t change it later
  3. Take a lot of photos of calibration pattern with different angles/positions in camera field of view: 100 is sufficient
  4. Find pattern points for all the images with subpixel accuracy. Exclude images where the points are incorrectly detected/refined
  5. Solve optimization problem
  6. Reprojection error should be ~0.3 px

27

28 of 51

Camera calibration

How to do

28

img = cv2.imread('data/chessboard.png')

pattern_size = (10, 7)

res, corners = cv2.findChessboardCorners(img, pattern_size)

criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 1e-3)

corners_subpix = cv2.cornerSubPix(cv2.cvtColor(img, cv2.COLOR_BGR2GRAY),

corners, (10, 10), (-1,-1), criteria)

for c in corners.squeeze():

img = cv2.circle(img, tuple(c), 10, (0, 0, 255))

cv2.drawChessboardCorners(img, pattern_size, corners_subpix, res)

29 of 51

Camera calibration

How to do

29

30 of 51

Camera calibration

How to do

30

pattern_points = np.zeros((np.prod(pattern_size), 3), np.float32)

pattern_points[:, :2] = np.indices(pattern_size).T.reshape(-1, 2)

rms, camera_matrix, dist_coefs, rvecs, tvecs = \

cv2.calibrateCamera([pattern_points], [corners_subpix], img[0].shape, None, None)

31 of 51

From 2D to 3D

When can we retrieve 3D information?

  • When we know the exact geometry of 3D object (actual size)
  • When we know distance to the object

31

32 of 51

From 2D to 3D

Undistortion of images

32

img = cv2.imread('data/chessboard.png')

camera_matrix = np.load('data/camera_mat.npy')

dist_coefs = np.load('data/dist_coefs.npy')

img_undistorted = cv2.undistort(img, camera_matrix, dist_coefs)

cv2.imshow('', img_undistorted)

cv2.waitKey()

cv2.destroyAllWindows()

33 of 51

From 2D to 3D

Undistortion of images

33

34 of 51

From 2D to 3D

How to reproject points from the image

34

ud_corners = cv2.undistortPoints(corners_subpix, camera_matrix, dist_coefs)

print(ud_corners[0])

ud_corners = np.c_[ud_corners.squeeze(), np.ones(len(ud_corners))]

img_pts, _ = cv2.projectPoints(ud_corners, (0, 0, 0), (0, 0, 0), camera_matrix, None)

for c in img_pts.squeeze().astype(np.float32):

cv2.circle(img_undistorted, tuple(c), 5, (0, 0, 255), 2)

cv2.imshow('', img_undistorted)

cv2.waitKey()

cv2.destroyAllWindows()

[[-0.16817029 -0.11604928]]

35 of 51

From 2D to 3D

How to reproject points from the image

35

36 of 51

From 2D to 3D

How to reproject points from the image

36

img_pts = ud_corners @ camera_matrix.T

img_pts /= img_pts[:, 2, None]

img_pts = img_pts[:, :2]

for c in img_pts.squeeze().astype(np.float32):

cv2.circle(img_undistorted, tuple(c), 5, (0, 255, 0), 2)

cv2.imshow('', img_undistorted)

cv2.waitKey()

cv2.destroyAllWindows()

37 of 51

From 2D to 3D

How to reproject points from the image

37

38 of 51

From 2D to 3D

Find 6-DOF position of the object

38

ret, rvec, tvec = \

cv2.solvePnP(pattern_points, corners_subpix, camera_matrix, dist_coefs)

39 of 51

ArUCo markers

Generate marker image

39

import cv2

import cv2.aruco as aruco # opencv_contrib

import numpy as np

aruco_dict = aruco.getPredefinedDictionary(aruco.DICT_6X6_1000)

marker_42 = aruco.drawMarker(aruco_dict, 42, 500)

marker_42 = np.pad(marker_42, 50, 'constant', constant_values=255)

cv2.imshow('', marker_42)

cv2.waitKey()

cv2.destroyAllWindows()

cv2.imwrite('data/marker_42_DICT_6X6_1000.png', marker_42)

40 of 51

ArUCo markers

Detection

40

marker_img = cv2.imread('data/marker_42_0.0078.png')

detect_params = aruco.DetectorParameters_create()

detect_params.cornerRefinementMethod = aruco.CORNER_REFINE_SUBPIX

detect_params.cornerRefinementWinSize = 15

corners, ids, _ = aruco.detectMarkers(marker_img, aruco_dict, None, None, detect_params)

detected_m_img = aruco.drawDetectedMarkers(marker_img, corners, ids, (0, 255, 0))

cv2.imshow('', detected_m_img)

cv2.waitKey()

cv2.destroyAllWindows()

​

41 of 51

ArUCo markers

Detection

41

42 of 51

ArUCo markers

Estimate pose and draw 3D axes

42

camera_matrix = np.load('data/camera_mat.npy')

dist_coefs = np.load('data/dist_coefs.npy')

rvecs, tvecs, _ = aruco.estimatePoseSingleMarkers(corners, 0.0078, camera_matrix, dist_coefs)

detected_m_img = aruco.drawAxis(detected_m_img, camera_matrix, dist_coefs, rvecs[0], tvecs[0], 0.005)

cv2.imshow('', detected_m_img)

cv2.waitKey()

cv2.destroyAllWindows()

43 of 51

ArUCo markers

Estimate pose and draw 3D axes

43

44 of 51

CV ideas

Detect the “duck”

  1. Apply color filter and tune thresholds

44

45 of 51

CV ideas

Estimate distance to the “duck”

  1. Roughly estimate a distance to the duck by knowing duck’s size

45

46 of 51

CV ideas

ArUCo

  • Surround the duck with ArUCo markers
    • Smaller region to find the duck - less false positives
    • Better distance estimation

46

47 of 51

Deep Learning, even in OpenCV

  • Special module in main repository: dnn
  • It’s possible to load and infer the models from: Caffe, Tensorflow, Torch, DarkNet, MXNet
  • Has a lot of samples: classification, segmentation, object detection, face detection
  • GPU isn’t used: hello, Raspberry Pi and mobile devices
  • Works faster than original frameworks (on CPU)
  • Works almost everywhere
  • Still developing
  • Has good support
  • Decent portion of functionality was added by me

47

48 of 51

Python tools

  • PyCharm (really good)
  • Jupyter Notebook

48

49 of 51

Books

  • Learning OpenCV 3 Computer Vision in C++ with the OpenCV Library
    • by Gary Bradski, Adrian Kaehler
  • OpenCV 3 Computer Vision with Python Cookbook
    • by Alexey Spizhevoy,‎ Aleksandr Rybnikov

49

50 of 51

Useful links

50

51 of 51

Questions & Discussion

51