Principles of Robotics 0618 final report

MYGO: an LLM-Controlled Robotic Dog that Mimics Real-World Dog Behavior

Group5

E94136180 黃　旺

F94101054 余奎甫

recap

Raspi 4B

Python API

Linux kernel API (C)

recap－testing flowchart

Raspi 4B +ubuntu

Tiger VNC

SSH

MAC

Firmware

Input from laptop

single arm 3DOF

3DP

UDP

Testing - Laptop keyboard input

Pygame keyboard listening + UDP repo

OUTPUT

1. switch walk/ stop mode
2. w/a/s/d -> move
3. jump
4. pitch
5. Tilt

​

10 KEYBOARD input type

​

Firmware

StanfordQuadruped API

Receive UDP

Inverse

kinematics

Pigpio

Angle2PWM -> PWM2DUTYCYCLE

​

revise

revise

Raspi 4B +ubuntu

Tiger VNC

SSH

MAC

Hardware

6V

SG90 1.8kg-cm*3

Main goal：mimic real world dog behavior

• LLM might not be the best solution but it can provide most subtle and real-life reflection

Real behavior?

• Limbs
• Head
• Voice
• Tail
• Emotion

​

External & internal environment

Our AI Dog Algorithm

Architecture

BLIP2-2.7b

Gemma-2-2b

Kaggle P100

ESP32-CAM

Real world

Thoughts

Actions

Sit()

Move_Left()

Bark()

Set_emotion()

Cloud AI Server

Memory

control

command

Raspi 4B

Image2text

Demo1 esp32cam->server->image2text

Demo2 esp32 cam POST to server

BLIP2-2.7b

Gemma-2-2b

Kaggle P100

Game environment

Thoughts

Actions

Sit()

Move_Left()

Bark()

Set_emotion()

Cloud AI Server

Memory

agent

command

Demo3 Robot dog simulator

AI agent

Architecture

BLIP2-2.7b

Gemma-2-2b

Kaggle P100

ESP32-CAM

Real world

Thoughts

Actions

Sit()

Move_Left()

Bark()

Set_emotion()

Cloud AI Server

Memory

control

command

硬件、控制來不及完善

Thanks!