1 of 45

REACHER

AI & Computer-Vision Based Teacher Assistant Robot

PROJECT PRESENTATION

By:

AbdulRehman Muhammad Younis

Razib Sarkar

Sherif Moussa

Yazeed Omer Eldigair

2 of 45

Introduction�

  • Problem Statement:
    • Risky contact between students and faculty in face-to-face assessments.
    • Reacher would function as a teaching assistant by:
    • Interactions with students: communication with the robot, relevant and personalized content retrieval
    • Exam proctoring: cheating detection, in-door classroom navigation, exam submission paper scanning

​

  • Motivation:
    • Combating the spread of COVID-19
    • Expansion of the involvement of AI in education
    • Creating a solution that is marketable

​

​

3 of 45

Overview�

4 of 45

Hardware Design Base

Base:

Yujin Kobuki TurtleBot2

+ Laptop running Linux OS

Extension rods

Original

Modified

Making the frame more stable

5 of 45

Hardware Design�

Logitech C922

Orbecc Astra 3D Scanner

Slamtec RPI Lidar

ASUS Tablet

6000mAh Li-Poly Battery

�Components

6 of 45

Subsystem:�Cheating Detection

​

- Overview

7 of 45

Subsystem:�Cheating Detection

Head movement

Talking on the phone

Texting

Gazing

Unauthorized

equipment or material

Talking

​

- Actions

8 of 45

Subsystem:�Cheating Detection

CNN for feature extraction

LSTM

Input Video Frames

​

Convolutional neural network

LSTM Network

​

LSTM

LSTM

Output

​

No cheating detected

Cheating detected:

    • Talking
    • Texting
    • Suspicious Head movement
    • etc.

​

– Deep Learning Network

9 of 45

Subsystem:�Cheating Detection

Partitioning the Data

Database obtained

  • 600 clips of suspicious behavior and cheating activities

​

  • 13 different labels of cheating events

​

​

​

​

– Collected Data

10 of 45

Subsystem:�Cheating Detection

Validation accuracy: 93.10%

​

Number of Epochs: 30

​

Iterations per Epoch: 32

​

Hardware resource: GPU

​

​

​

�– Training

11 of 45

Subsystem:�Cheating Detection

Testing Scenarios

Scenario 1

  • Data from the same subjects has already been fed as training data to the algorithm

​

​

​

​

​

Scenario 3

  • Subjects that have not been encountered by the algorithm are introduced

​

​

​

​

​

​

Scenario 2

  • Subjects from the training data are used but with a different background

​

​

​

​

​

� – Testing Plan

12 of 45

Subsystem:�Cheating Detection

Scenario 1

Scenario 2

�– Test Results

13 of 45

Subsystem:�Cheating Detection

Scenario 3

�– Test Results

14 of 45

Subsystem:�Navigation

​

– Overview

15 of 45

Subsystem:�Navigation

  • Needs to:
    • Follow a predefined path
    • Avoid collisions
  • Two sensors working in tandem:
    • 3D Sensor for Collision Avoidance
    • 360 Lidar sensor for Mapping and Localization

​

ROS provides useful packages built specifically for indoor navigation

which deals with 3 key issues:

    • Mapping
    • Localization
    • Navigation

​

​

​

– Implementation

16 of 45

Subsystem:�Navigation

ROS Navigation Stack Overview

​

– Implementation

17 of 45

Subsystem:�Navigation

ROS Navigation Stack Recovery Behaviors

​

– Implementation

18 of 45

Subsystem:�Vision Based Scanning

​

– Overview

19 of 45

Subsystem:�Vision Based Scanning

1

2

Processing Server

  • Camera records students flipping through exam
  • Uploads to processing server

Pages have page number QR codes on them

Frames captured by camera

​

– Setup

20 of 45

Subsystem:�Vision Based Scanning

Video detected

File opened

Scanning entire file

One frame at a time

Video frames

– Implementation

21 of 45

Subsystem:�Vision Based Scanning

Video detected

File opened

Scanning entire file

One frame at a time

Video frames

– Implementation

22 of 45

Subsystem:�Vision Based Scanning

1

QR Code detected

Original Frame

Operation to detect QR Code

Calculated basic

Crop region

ROI extracted

– Implementation

23 of 45

Subsystem:�Vision Based Scanning

Cropped Frame

Grayscale

Blurred

Canny filter

Opened

Hough Transform

Perspective

Warp

– Implementation

24 of 45

Subsystem:�Vision Based Scanning

Handwritten

Student ID extracted

Sharpness of frame calculated

– Implementation

25 of 45

Subsystem:�Vision Based Scanning

Sharpness of frame calculated

Page 1

Handwritten

Student ID extracted

– Implementation

26 of 45

Subsystem:�Vision Based Scanning

Page 1

Page 2

Page 3

=1

Document 1

Handwritten

Student ID extracted

Sharpness of frame calculated

– Implementation

27 of 45

Subsystem:�Vision Based Scanning

Page 1

Page 2

Page 3

Document 1

– Implementation

28 of 45

Subsystem:�Vision Based Scanning

Page 1

Page 2

Page 3

Document 1

Best frames selected

for each page

Adaptive Thresholding

7469701

Classification by

Image classification model

– Implementation

29 of 45

Subsystem:�Vision Based Scanning

Document 1

Document 2

PDF

PDF

exam_2021065.mp4_7469701.pdf

exam_2021065.mp4_1059221.pdf

– Implementation

30 of 45

Subsystem:�Human-Robot Voice Interaction

​

– Overview

31 of 45

Subsystem:�Human-Robot Voice Interaction

Miss Rate

Raspberry Pi 3 CPU Usage

Benchmark ROC Curve for word “Jarvis”

​

– WW Engine

32 of 45

Subsystem:�Human-Robot Voice Interaction

Start Flow:

​

– Implementation

33 of 45

Subsystem:�Content Recommendation

​

– Overview

34 of 45

Subsystem:�Content Recommendation

​

– Content

  • Each topic has 3 levels of complexities with content based on that level, some content is repeated.
  • Topic dictionary containing name of topic and details of topic is retrieved from DialogFlow.
  • Level of complexity is retrieved from recommender.
  • TinyDB database queried using name of topic, details of topic, and level of complexity to retrieve content.

​

35 of 45

Subsystem:�Content Recommendation

​

– Face Recognition

Capture a frame

Extract features

Compare to faces in database

Identify person

Training Data: Shreif

Training Data: Abad

36 of 45

Subsystem:�Content Recommendation

​

– Face Recognition

Existing Student Profile

No Student Profile

37 of 45

Subsystem:�Content Recommendation

​

– Face Recognition

Unknown face detected

New user ID generated

New face stored in tinydb

Errors in Detection

38 of 45

Subsystem:�Content Recommendation

​

– Content Personalization

  • Level of complexity (called rating here) each user preferred for some topics.

​

  • These ratings are used to recommend the preferred level of complexity for a topic for the user.

​

39 of 45

Subsystem:�Content Recommendation

​

– Content Personalization

Existing Student Profile

No Student Profile

40 of 45

Future Recommendations�

    • Bias of cheating detection algorithm towards gazing actions
    • An abundant amount of collected data is required to combat the struggle of new examinee faces and environments
    • Pre-techniques such as background removal can enhance results

​

    • Image scanning system is limited by the type of paper
    • The 80% accuracy for detecting handwritten IDs requires improvement

​

    • Our recommender system can be further enriched by utilizing demographic information such as age groups and gender.

41 of 45

Conclusion�

    • Bias of cheating detection algorithm towards gazing actions
    • An abundant amount of collected data is required to combat the struggle of new examinee faces and environments
    • Pre-techniques such as background removal can enhance results
    • Image scanning system is limited by the type of paper
    • The 80% accuracy for detecting handwritten IDs requires improvement
    • Our recommender system can be further enriched by utilizing demographic information such as age groups and gender.

42 of 45

Conclusion�

REACHER is a multi-purpose robot that features various teacher assistant functionalities:

    • It utilizes a cheating detection algorithm for online examinations that is yet to be trained on in-person examinations
    • Autonomously navigate a classroom during an examination while avoiding fixed and dynamic objects.
    • Contactless-ly scan paper-based exam submissions for grading
    • Interact with humans via a voice-based chatbot
    • Provide personalized academic content for students

43 of 45

Last but not least…

44 of 45

�Food Delivery!��Thank you

45 of 45

References

  • P. Sharma, "Comprehensive Guide to build a Recommendation Engine from scratch (in Python)," Analytics Vidhya, 21 June 2018. [Online]. Available: https://www.analyticsvidhya.com/blog/2018/06/comprehensive-guide-recommendation-engine-python/?#. [Accessed April 2021].
  • Koren, Yehuda, Robert Bell and Chris Volinsky. “Matrix Factorization Techniques for Recommender Systems.” Computer Volume: 42, Issue: 8 (2009): 30-37.
  • S. Rendle, "Factorization Machines," in Proceedings of the 10th IEEE International Conference on Data Mining (ICDM), 2010.
  • A.Rozario,“Faq:Howwasgmatheldusingaiproctors?canitdetectcheating?”2020.[Online].Available:https://www.thequint.com/news/education/what-is-an-artificial-intelligence-proctored-test-can-it-detect-cheating-faq#read-more
  • “Respondus.” [Online]. Available: https://web.respondus.com/
  • Knowledge, “How respondus review priority is determined,” 2019.[Online]. Available: https://community.brightspace.com/seu/s/article/000007486
  • M. Shamqoli and H. Khosravi, “Border detection of document imagesscanned from large books,” in2013 8th Iranian Conference on MachineVision and Image Processing (MVIP), 2013, pp. 84–88.
  • O. Boudraa, W. K. Hidouci, and D. Michelucci, “An improved skewangle detection and correction technique for historical scanned doc-uments using morphological skeleton and progressive probabilistichough transform,” in2017 5th International Conference on ElectricalEngineering - Boumerdes (ICEE-B), 2017, pp. 1–6
  • Pech-Pacheco et al, “Diatom autofocusing in brightfield microscopy: a comparative study.”, in 2000 ICPR.

​

​