Computer Vision

in FRC

By Max DeVos

What is Computer Vision?

• Taking an image (often using a camera)
• Manipulating it into usable data (pitch, yaw, height/distance)

​

2

Target

What isn’t Computer Vision?

• Taking an image
• Displaying it to a human for interpretation

​

3

Target

Driver

How Vision Processing Works

Acquiring, Transferring, and Using Data

The 3 Parts of Computer Vision Tracking

Transferring Vision Data

​

Moving data from the processor to a place where it can be used by the central robot controller to adjust actuators

​

Acquiring Vision Data

​

​

Using a processor to collect and manipulate camera input to get meaningful data and information

​

​

Using Vision Data

​

​

Using the Vision data to move actuators and solve engineering problems.

​

5

Getting Vision Data

The Process of Turning Camera Input into Usable Data

Cameras - Acquiring Vision Data

• Sensor for Vision Tracking
• Things to look for:

​

• USB Real-Time Transfer
• Aperture Priority
• High frame rate
• High Field of View (FOV)
• Mountability (can I get this on my robot)
• Affordability (You should have at least 2)
• Durability

7

Choosing a Camera Type - Acquiring Vision Data

Standard RGB

​

Pros:

• Easier to find
• More variety
• Often higher FOV
• Higher resolution

​

Cons:

• Doesn’t work as well with retro-reflective
• Can interfere with other teams’ wavelengths
• Harder to filter and use for vision tracking
• May have to be turned off for light rules

​

Example:

Microsoft Lifecam

HD-3000

​

Infrared (IR)

​

Pros:

• Works better for Vision tracking
• Doesn’t interfere with other teams’ light
• Doesn’t have to be turned off for light rules
• Drastically reduces mis-detections

​

Cons:

• Not as many of them (hard to find)
• Sometimes have smaller FOV
• Often aren’t USB

​

​

Example:�Infrared Camera Module

(NoIR) V2

8

The Vision Processor- Acquiring Vision Data

• The part of the system that runs the program that creates the usable data from the camera images
• Runs the program on startup
• Program written in

C++, Python, or Java

9

Vision Processor Options - Acquiring Vision Data

• Many available offboard processors

10

• Recommended
• NVidia Jetson Tegra K1
• NVidia Jetson Tegra X1
• NVidia Jetson Tegra X2

• Other Options
• Raspberry Pi
• BeagleBoard/BeagleBone
• Intel Galileo
• A10 Linux Dev Board

The Vision Processing Program

The Software Itself

The Program Pipeline - Acquiring Vision Data

Program runs a script on loop to handle new images.

12

1

3

4

2

Step 1 - “Input”

• Latest still frame is taken from the camera and input into the program
• Handled by the the VideoCapture object

Python

​

​

​

C++

13

Step 2 - “Image Manipulation”

14

Image is blurred and converted into easily filterable format (HSV).

Blur

​

Color Conversion

​

Blur - Step 2 “Image Manipulation”

• Blurring appears counter-intuitive
• Eliminates imperfections and oddities from an image
• Result: more accurate filtering and detection

Python

​

C++

15

Color Conversion - Step 2 “Image Manipulation”

• Convert type of the image to allow for a more intuitive filtering process
• Important to understand types of image color storage

16

Color Storage Formats - Step 2 “Image Manipulation”

BGR (RGB)

​

Classic images - contain a different amount of blue, green, or red to create any color. Terrible for filtering because of 3 different hue values.

Note: For some reason OpenCV refers to this format by default as BGR. Your IDE may suggest an RGB enum, but do not use it.

​

​

​

​

​

​

​

​

HSV

​

A coloring format optimized for filtering.

Like BGR, has 3 data points, but instead of different amounts of base colors includes one value (H) for the hue range, (S) for saturation (difference from white), and V (difference from black).

​

​

​

Mask Image (Boolean Image)

​

Pixels are activated or inactivated.

Used for contour detection and math. Often referred to as a 2-channel image/mask.

​

17

Color Conversion- Step 2 “Image Manipulation”

• Convert your image from unfriendly BGR format to filter-friendly HSV format
• Use OpenCV function cvtColor.

Python

​

C++

18

Step 3 - “Filtering”

19

Geometric Filtering

Program filters image using both color and geometry to determine a set of pixel coordinates of the target

Color Filtering

​

Contour

Detect

Color Conversion- Step 3 “Filtering”

• Convert your image from HSV to an Boolean Mask using upper and lower color filters that are tuned to the field
• Use OpenCV function inRange.

Python

​

​

C++

20

Contour Detection - Step 3 “Filtering”

• Convert white “active” pixels into blobs of coordinate data
• OpenCV has a function to do this called findContours.

Python

​

C++

21

Geometric Filtering - Step 3 “Filtering”

• The final step filtering is Geometric filtering
• Removes any mis-detections based on shape or size

Types of geometric filters:

• Area
• Perimeter
• Aspect Ratio
• Location on screen
• Location relative to other shapes
• etc.

22

Geometric Filtering (cont.) - Step 3 “Filtering”

Example: peg rectangles in Steamworks

23

Geometric Filtering (cont.) - Step 3 “Filtering”

Process:

• For loops, filtering out contours that are too small or oddly shaped
• Sorting functions to remove small,rectangular mis-detections
• Final, large scale sorting to determine final target(s) of out of remaining contours

Ex. Implementation:

24

All Contours Array

Large Contours Array

For: Larger than 100px²

For: Aspect Ratio ~ 2/5

Correct Shape

Sorting: Largest Two

Final Contour(s)

Step 4 - Calculations

25

Pixel points are used to run calculations to compare with previous data and discard invalid points.

Geometric Math

Comparison

​

Relative Tuning - Geometric Calculations

Absolute Tuning:

Using complex math and real world measurements to find exact pose of target, and then tuning robot mechanisms to that data.

​

Relative Tuning:

Using simple math and geometric properties to find relative position of target, and then tuning robot mechanisms to that data.

• Coordinate point data is used to interpret information about your target
• Use of relative tuning can make the math aspect of this simple.

​

26

Solving Math Problems - Geometric Calculations

• Finding actual center of a target: the X system
• Draw lines from opposing corners and find point of intersection
• Solves perspective problem by cancelling corners

​

27

Note:

​

To determine which corner is which

• Sort points by slope from the origin
• Take most extreme two and least extreme two, sort those by their X or Y value to ensure correct corners

Important Math Functions - Geometric Calculations

Yaw

​

​

Pitch

28

• Finding Yaw/Pitch Offset to Target
• Used for alignment to target around an axis

TargetX = The X value of the center of your target

TotalX = Total Width of screen

This will return a value -1 to 1 of how much the robot needs to move/rotate

-1 is left, 1 is right

TargetY = The Y value of the center of your target

TotalY = Total Height of screen

This will return a value -1 to 1 of how much the robot needs to move

1 is up, -1 is down

Comparison- Step 4 “Calculations”

Due to mis-detections, it’s important to verify sensibility of data. This can be done through comparing current point to previous data.

29

Store

If the data point is good, store it in the place of the previous data for future comparisons.

Timestamp

Use your language’s time system to assign a variable to the time of the frame for latency comparison

Compare

Compare your current point to your most recent data to verify that it is possible. If not, disregard that point and wait for next data point.

Transferring Vision Data

Moving Vision Data from the Processor To Your Robot Controller

Methods of Communication - Moving Vision Data

Serial Communication

​

• Pros
• Faster
• Cons
• Unstructured / no built in support for multiple values
• Potentially hard to debug

​

Network Communication

​

• Pros
• Easier
• Built in support for multiple values
• Cons
• Slower
• Requires a radio port

31

Network Communication - Moving Vision Data

• NetworkTables Library
• Built into WPILib
• Can be compiled independently
• PyNetworkTables for Python
• System of sending multiple values over radio

• Network Configuration
• Use RoboRIO as Server
• Use Coprocessor as Client
• Send data over radio

32

Using Vision Data

Moving Vision Data from the Processor to the Robot Controller

Using Vision Data to Actuate Actuators

• Vision data will be used to move the robot
• The values you have available:
• Yaw Offset (-1 to 1)
• Pitch Offset (-1 to 1)
• Average height of contour for distance

34

Relative Tuning - Using Vision Data

Relative Tuning:

• Using simple math and geometric properties to find relative position of target
• Tuning robot mechanisms to that data.

Use relative tuning to find a function that tells the robot specifically how to behave based on sensor data (in this case, Vision data)

​

Example:

Sensor Data: Height of contour for distance

Feedback: RPM of flywheel

Goal: Adjust RPM of flywheel to match distance

​

35

Relative Tuning - How To

1. Create a data table
2. Use a dynamic variable as your X axis (height of target contour)
3. Use your robot adjustment for your Y axis (flywheel RPM to get to goal)
4. Repeat a lot of times at different X values
5. Plot the data & create a line of best fit
6. Use that equation in your programming for interpreting vision data.

36

What is Computer Vision?

• Taking an image (often using a camera)
• Manipulating it into usable data (pitch, yaw, height/distance)

​

37

Target

Thanks!

Any questions?

You can email us at programming@kingtec2169.com

Or me personally at devosmaxwell@gmail.com

Or see me after the presentation :)

Presentation can be found on https://kingtec2169.com/resources/documents/

​