AugPy:�Augmenting the Reality with Python
October, 2019
Central Idea
Augmented Reality is Computer Vision at its core
Outline
Augmented Reality
AR: Defining Characteristics
Source: Azuma, R.T (1997), A Survey of Augmented Reality Presence, 6(4), 355 - 385
Pokémon Go (Source: Niantic)
Doing proper AR in real world is chaotic…
Lots of varying parameters to handle:
Source: Blogpost by Andreas Jakl
The Marker Faced…
AR can be of two types:
Marker-less
(Source: Google ARCore docs)
Marker-based
(Source: Vuforia)
So what makes AR possible?
while(condition):
update tracking + environment data,
poll input for object placement/interactions
update placed virtual objects
Update understanding
AR requires tracking to really work well. Involves:
Known as Pose Estimation in XR speak.
A common coordinate system is important!
Update understanding
Shouldn’t there be some reference points to do all this?
YES!
Keypoints are the key here:
distinctive locations in images - corners, blobs or T-junctions.
Together, they describe features of the environment.
Properties:
Update understanding
Some SLAM algorithms used to identify the key-points to track a feature reliably:
Update understanding
FAST Algorithm, the basis of BRISK
For BRISK:
Keypoint detection:
At least 9 pixels should be brighter or darker than p
Update understanding
FAST Algorithm, the basis of BRISK
For BRISK:
Keypoint detection:
At least 9 pixels should be brighter or darker than p
Keypoint description:
Create binary string with 512 bits, with comparison results
(already handled for us by OpenCV)
Update understanding
Reference Image
Update understanding
Processing…
Update understanding
Tracking Points, aka Trackables
Another view
LIDAR point cloud (source: LizardTech )
Update understanding
Source: Bruce Thomas, COMP4010, University of South Australia
So what makes AR possible?
while(condition):
update tracking + environment data,
poll input for object placement/interactions
perform error correction
update placed virtual objects
Current AR libraries…
Many libraries exist for Augmented Reality:
And so are the problems!
An interesting paper reference:
“Perceptual issues in augmented reality revisited” by E. Kruijff, J. E. Swan II, and S. Feiner, IEEE International Symposium on Mixed and Augmented Reality, Oct 2010.
A ray of hope…
The recent release of ARKit by Apple is a step closer towards solving these problems
Hon’ble mention: 6D.ai
Image Source: Apple Developers
Problems
Some important ones:
Solution?
This is where Python can be helpful!
Pose Estimation
Image Source: Microsoft.
Sensors throw lots of data.
Process it well to understand the position better!
Object Segmentation
Image source: Niantic
Segregate objects to interact individually
Depth Estimation
Source: Google AI Blog
Understand depth from a given image
Future of AugPy?
Up next?
We have some basics in place, need to build an ecosystem
To-Do
Extensions
ML on Edge – make lightweight models
Deploy to AR headsets (HoloLens)
Federated Learning
???
Some references
References:
(in no particular order)
References:
(in no particular order)
Thank you
Contributions/Suggestions welcome!
Feel free to reach out: @itsmutnuri