1 of 20

Raspberry Swimmer

A Real-Time Computer-Vision Airship

1

2 of 20

Background & Motivation

  • Previous experience with RPi & Arduino microcontrollers to make (arguably) useful things: Rep-Rap Printer, Solar Array Phone Charger, EMG Device
  • My background is in rocketry and aerospace, so a natural challenge was something semi-autonomous that flies
  • Wanted to make something purely for entertainment and enjoyment
  • Currently work on power systems, so practice with batteries is good

​

“Find a platform looking for a problem to solve”

      • Professor Sloss

2

3 of 20

Description & Goals

  • Modify Air Swimmer toy (Aka “Shark Blimp”) to semi-autonomously navigate an environment
    • Implement onboard real-time Computer Vision system for motion, object, and facial recognition
    • Secondary implementation of distance detection/terrain mapping
    • Utilize commercial motors
  • Tetherless power supply (battery powered)
  • Lightweight system

3

Air Swimmer Toy (it’s about 4’ long)

4 of 20

Raspberry Swimmer

​

Aka AutoShark

4

5 of 20

Electronic Hardware Overview

  • Raspberry Pi 4B (4 GB), running Bullseye Legacy Release
  • Raspberry Pi Camera 2
    • 8 MP IMX219 sensor, connects via CSI
  • L293D Mini Motor Driver Shield
    • Quadruple high-current half-H driver for inductive loads (motors)
  • 2x Micro DC motors
    • Probably rated for 3-5V DC
  • Li-Ion Battery HAT for RPi, boosts to 5V
    • Two-way fast-charging support SW6106, including 2x 14500 Li-Ion cells (3.7V)
  • Some 28 & 26 AWG
    • Not moving more than ~200 mA

5

6 of 20

High-Level Design

6

Ballast Motor

Fin Motor

RPi Camera 2

L293D Mini Shield

RPi 4B

Battery HAT + Li-Ion

7 of 20

System Block Diagram

7

High Level Functional Block Diagram of state machine

8 of 20

Initial Hardware Characterization

*ESD precautions important throughout the entirety of project*

  • 5V rail on RPi more like 5.5V on PS, 4.9 on battery
    • Verify charge enable on battery on PS
  • 5V rail on RPi found to supply ~200 mA, enable PWM on motors to slowly ramp voltage supplied to ensure DC motors aren’t destroyed
    • Testing on spare DC motor rated for 12V DC
  • Weight of all hardware: 135 g
  • Camera connection & pwr on validation
  • Soldering connections on original toy & continuity tests

8

9 of 20

Software Characterization

Camera Functionality

  • Libraries Installed: Picamera2
  • Boot RPi Camera 2 on Bookworm, establish functionality
  • Utilize camera in virtual environment, verify functionality for video and image capture

Object Recognition

  • Libraries installed: Opencv, tensorflow-lite
  • Utilize camera in virtual environment after versioning down OS (Bullseye), Python (3.11->3.8)
  • Object detection and retrieval retrieval on automated loop

9

10 of 20

Software Characterization (Cont.)

Gesture Recognition

  • Libraries Installed: CVzone, mediapipe
  • MediaPipe Hands includes a pre-trained palm detection and hand landmark model for gesture recognition
  • Establish basic functionality and scripted PWM control to motors

Facial Recognition

  • Libraries Installed: Included in tensorflow-lite
  • Trained top layers and fine tuned tensorflow’s FaceNet model to my own face
  • Establish functionality in real-time state machine

10

11 of 20

Interesting Things

  • RPi Bookworm seems to have re-done much of the libcamera library in latest releases (renamed to rpicam-apps) and support for Opencv library on 64-bit seems to remain an open issue. Libcamera is a Bullseye-era replacement for Raspistill, older support for
    • Libcamera is a hard dependency for platform agnostic support of MIPI CSI cameras, but I couldn’t get it to detect camera or process h.264
    • Workaround: version down to Bullseye and enable legacy camera stack in the config
    • Unsure if Opencv works better with 32-bit Bookworm/Bullseye
  • Opencv and mediapipe dependencies were incompatible with onboard python (3.11), versioned down to 3.8
  • Was able to get 200 mA from the Pi while running, which is much more than I expected
  • Facial recognition is the heaviest process running, probably because of the inefficiency of my top-layer training

11

12 of 20

Wiring Diagram

  • Raspberry Pi Pins out to IC on Driver board
  • Supplies 5V to Vss (internal logic), 5V to Vs (Internal H Bridges)
  • GPIO pins to ENA/B for PWM control on both motors
  • Two motors powered and controlled by board

12

13 of 20

Assembly

  • Attached to plastic COTs chassis mount, tried to use as little mass as possible
  • Some duct tape involved
  • Two additional ~27” Helium balloons
    • About 134 L He, lifts ~175g
  • Assembly weighs 136g, not including tape

13

RPi Holder (2x)

L293D Holder (2x)

Zip Tie passthrough

14 of 20

Software Implementation

  • CV-enabled state machine
    • Flying in autonomous mode with basic object detection
    • Checking on an interval for facial recognition match, moves to commanded mode
    • Checks on an interval for gestures, refreshes after execution for pre-set loop
  • “Real-time” – in reality, operates with significant lag (up to about 3s, an eternity)
    • State machine-like execution of recognition procedures improves event handling and lag significantly
  • Next steps for software implementation
    • Run at higher power
    • Train model to handle blurrier or less dense images (processing faster)
    • Crank down confidence intervals
    • More sophisticated event handling and queuing of events

14

15 of 20

Demonstration

15

16 of 20

Demo (Ballast)

16

17 of 20

Demo (POV & Turn)

17

18 of 20

Lessons Learned

  • Identify version control issues with hardware/software/libraries before speccing components, whenever possible
    • RPi Camera difficulties with library
  • Spec hardware with plenty of modularity but also flexibility (and scalability, when possible)
    • Motors on toy turned out to be DC, not servos
  • Avoid procuring poorly documented hardware, whenever possible
    • L293D Mini Driver board documentation seems to have been wiped from the entire internet
  • Leverage existing libraries and code, but be quick to write from scratch when difficulties arise
  • Proof-of-concept as the best pathway to performance
  • Avoid procuring hardware until a project goal and high-level requirements are ideated
    • Everything was very heavy, but ironically, my balloons were oversized.

18

19 of 20

References

Cvzone. (n.d.). Cvzone/cvzone: This is a computer vision package that makes its easy to run image processing and AI functions. at the core it uses opencv and Mediapipe libraries. GitHub. https://github.com/cvzone/cvzone

GitHub. (n.d.). https://github.com/freedomwebtech/raspbianlegacy/blob/main/hands.py

MediaPipe. layout: forward target: https://developers.google.com/mediapipe/solutions/vision/hand_landmarker title: Hands parent: MediaPipe Legacy Solutions nav_order: 4 - MediaPipe v0.7.5 documentation. (n.d.). https://mediapipe.readthedocs.io/en/latest/solutions/hands.html

Opencv. (n.d.). Add support for libcamera · ISSUE #21653 · opencv/opencv. GitHub. https://github.com/opencv/opencv/issues/21653

Staff, L. E. (2022, July 21). In-depth: Control DC motors with L293D Motor Driver IC & Arduino. Last Minute Engineers. https://lastminuteengineers.com/l293d-dc-motor-arduino-tutorial/

Tim. (2023, February 16). Face recognition with Raspberry Pi and opencv - tutorial australia. Core Electronics. https://core-electronics.com.au/guides/face-identify-raspberry-pi/#Where

YouTube. (2022, March 15). How to install tensorflow 2 and opencv on a raspberry pi. YouTube. https://www.youtube.com/watch?v=vekblEk6UPc

​

*Special thanks to UW Makespace in McCarty, The Mill

19

20 of 20

Thank you!

Questions?

20