1 of 27

Building Camera based Intelligent Applications

Nabarun Pal

2 of 27

About Me

nabarun.in�github.com/palnabarun

Final Year Undergrad @ IITR
Software Engineer Intern @ rorodata
Systems Engineer @ Team Robocon IITR
Love my kindle and hacking through things

3 of 27

Overview

Introduction
Use Cases
Components of the pipeline
Bottlenecks & Challenges
Our approach

Camera Agent
Cloud Architecture
Firefly
Dashboard

Code Samples
Ending notes

4 of 27

What are camera based applications?

5 of 27

What makes them intelligent?

6 of 27

Use Cases

Surveillance
Agricultural yield estimation
Object classification
Barcode recognition on conveyor

7 of 27

How would you build them?

8 of 27

Components of the pipeline

Data Ingestion
Data Storage
Machine Learning
Visualization

9 of 27

Bottlenecks and Challenges

What tools to use?
How to capture data?
Where to store?
Where should my ML code run?
How should I visualize the output?

There are several key decisions that a data scientist needs to make in order to make such a camera based intelligent applications. So this is an outline of how you would go on through a traditional approach.

- One needs to decide what framework would be best for his/her purpose.

- the method through which image/video needs to be captured, be it opencv, pygame or something else

- Then when we have the data comes the question of storing it. You have choices between AWS, Google Cloud Platform or other cloud providers.

- in answering this question, there may be issues with dependency management. take for example, opencv - you can install it using conda but then you wont be able use it with libraries not available with conda and installed with pip

- and if it has not been a turn off for you till now, you have to think about visualizing all the work you have done

10 of 27

Typical Timeline

Takes 2-3 weeks for building an application
Of that, only 2 days may be spent on actual machine learning
Majority of the time is spent in software engineering rather than ML

11 of 27

Time to Market!

12 of 27

Our Approach

13 of 27

Camera Agent

Program running on end device
Connects to pre-configured server
Captures and uploads pictures/videos
Remotely configurable by the server

14 of 27

Cloud Architecture

Server to get data from the agents and store them
Admin Dashboard to control the capture frequency, type of data captured, see logs and uploads from registered devices
Storage provider can be changed
Provides a client library for fetching data

15 of 27

Firefly

Framework to deploy Python functions as REST web service
Very easy and flexible to use
Used to deploy ML models and pre-processing functions
Easily deployable using WSGI servers like gunicorn
Open Source !

16 of 27

Dashboard

Variety of visualization options
Data provided by the client library or a firefly function running Machine Learning Code
Run anywhere
Plotting and charting using Dash by plot.ly or any other visualization tool

17 of 27

Talk is cheap.

Show me the code.

Linus Torvalds

18 of 27

19 of 27

20 of 27

21 of 27

22 of 27

23 of 27

24 of 27

Ending Notes

End to end application development in 3-4 days
Majority of time spent on ML
Robust infrastructure
Versatile data storage options
Scalability

25 of 27

Links

26 of 27

Thank You!

Have a nice day.

https://goo.gl/3iaiCn�@nabar0n�https://nabarun.in

27 of 27

Questions

@nabar0n, //nabarun.in