Team 77
A Data Capture System For Usability Research
April 9th 2025
The Client:
WHEEL
Sanjay Ramnauth
Lara Askovic
Ruby Sun
Harry Zhang
Carson Ding
Alan He
Team Leader
Project Manager
Multimodality
Quality Assurance
Meeting Coordinator
Communication Liaison
Disclaimer
This design has been prepared by first year engineering and architect students at the University of Toronto and does not present a Professional Engineering or Architecture design. A Professional Engineer or Architect has not reviewed this presentation for technical accuracy or adequacy. The recommendations of this design may not be implemented in any way unless reviewed and approved by a licensed Professional Engineer or Architect where such review and approval is required by professional or legal standards, it being understood that it is the responsibility of the recipient of the design to assess whether such a requirement exists.
The Presentation may not be reproduced, in whole or in part, without this Disclaimer.
© Engineering Strategies and Practice, University of Toronto, 2025
Permission to reproduce material from this presentation may be requested by email to your team’s contact person, with a CC to Engineering Strategies and Practice at esp@engineering.utoronto.ca
2
Our Client Conducts Usability Research
3
Usability testing is the process of evaluating how well users interact with a product or system.
A typical set up of the mobile testing lab. [6]
04
03
02
01
Start-up Companies
UofT Department of MIE
Women’s College Hospital
Neighbours in 800 Bay St.
Stakeholders
4
The Current Usability Lab is Underutilized
Observation Room
Evaluation Room
Functions
5
Top Objectives
Maximize Coverage of Visual Elements
Reduce
Time to Synchronize
Collect Data
Collect Visual Data
Collect Audio Data
Synchronize
Data
Efficient Data Collection is a Usability Lab’s Top Priority
The Livestreamer is a Modular and Portable System
6
Price List
Logitech Webcam $299.99[7]
Tobii Eye Tracker $396.98[8]
NeuroSky Mobile EEG Kit $198.81[9]
Ray-Ban Meta Wayfarer $369[10]
Wellue 12-Lead Holter Monitor
$809.6[11]
The Livestreamer is a Modular, Portable, and Synchronized System
7
Price List
Deity V-Mic D3 Pro
$239.00[12]
Pyannote AI
Free
OBS
Free
Dell 27” 4K UHD Monitor
$395.49[2]
Total: $2708.87 +Tax
AUDIO
SYNCH
Open Broadcaster Software (OBS)
Sync data from the moment of capture
source:[4]
Boom Microphone & Pyannote AI
Audio Recording
Sound diarization
source:[13]
8
Capturing Wide Visual Data is the Most Important Feature for a Usability Lab
source:[5]
source:[6]
VS
The MCS accurately measures the effectiveness of visual data captured.
WEIGHTING
9
The Livestreamer Achieves a High MCS
10
14 electrodes
- Ideal Coverage Area
- Actual Coverage Area
12 electrodes
≈ 0.857
Biometric Sensors
Eye Tracker
- Ideal Coverage Area
- Actual Coverage Area
Cameras
Webcam: 0.958
Glasses: 0.755
0.945
0.565
Dell monitor [2].
[14], [15], [16].
The Livestreamer Achieves a High MCS
11
0.360
0.900
12
The Livestreamer can capture data most meaningful to finding deeper insights in behaviours.
Integration with Rest of System Should be Considered
Before implementation, consider:
13
References
[1]“Tobii Eye Tracking - Bay Cities,” Bay Cities, Apr. 19, 2022. https://www.bay-cities.com/services/concept-and-design/tobii-eye-tracking/ (accessed Apr. 07, 2025).
[2]“Dell 27 4K UHD Monitor - S2721QS | Dell Canada,” Dell, Apr. 02, 2025. https://www.dell.com/en-ca/shop/dell-27-4k-uhd-monitor-s2721qs/apd/210-axlg/monitors-monitor-accessories
[3]“pyannote AI | Viva Technology ★,” Vivatechnology, 2025. https://vivatechnology.com/partners/pyannote-ai (accessed Apr. 07, 2025).
[4]“OBS Studio,” Wikipedia, Aug. 10, 2021. https://en.wikipedia.org/wiki/OBS_Studio
[5]J. Roose, “How to Conduct Usability Testing in Six Steps,” Toptal Design Blog, 2017. https://www.toptal.com/designers/ux/how-to-conduct-usability-testing-in-6-steps
[6]R. Hart, “Rory Hart,” Rory Hart, Feb. 23, 2015. https://www.thisisrory.com/blog/2015/2/11/hart-method-how-to-set-up-a-full-360-mobile-usability-lab (accessed Apr. 08, 2025).
[7]“Logitech,” Logitech, 2025. https://www.logitech.com/en-ca/shop/p/mx-brio-4k-webcam (accessed Apr. 07, 2025).
[8]“Tobii Eye Tracker 5 - The Next Generation of Head and Eye Tracking,” Tobii Gaming. https://gaming.tobii.com/product/eye-tracker-5/
[9]“EEG Headsets | NeuroSky Store,” Neurosky.com, 2017. https://store.neurosky.com/
[10]“Shop Ray-Ban Meta AI Glasses & Sunglasses,” Meta.com, 2022. https://www.meta.com/ca/ai-glasses/shop-all/ (accessed Apr. 07, 2025).
[11]“12-Lead Holter Monitor,” Wellue, 2021. https://getwellue.com/products/12-lead-holter-monitor (accessed Apr. 07, 2025).
[12]“Deity Directional Microphone,” Amazon. https://www.amazon.ca/Deity-Microphones-Directional-Microphone-Sensitivity
[13]“Deity V-Mic D3,” Broadway Camera, 2025. https://bccamera.com/deity-v-mic-d3/ (accessed Apr. 07, 2025).
[14] P. R. Ahmed, S. Akter, T. S. Pias, and M. A. Adnan, “Optimal EEG Electrode Set for Emotion Recognition From Brain Signals: An Empirical Quest,” arXiv.org, 2023. https://arxiv.org/abs/2311.17204 (accessed Apr. 07, 2025).
[15] J. Zhang and P. Chen, “Selection of Optimal EEG Electrodes for Human Emotion Recognition,” IFAC-PapersOnLine, vol. 53, no. 2, pp. 10229–10235, 2020, doi: https://doi.org/10.1016/j.ifacol.2020.12.2753.
[16] J. Montoya-Martínez, J. Vanthornhout, A. Bertrand, and T. Francart, “Effect of number and placement of EEG electrodes on measurement of neural tracking of speech,” PLoS ONE, vol. 16, no. 2, p. e0246769, Feb. 2021, doi: https://doi.org/10.1371/journal.pone.0246769.
[17] “Capturing a Human Factors Usability Study: AV Technology,” Research Collective, Jun. 15, 2023. https://research-collective.com/capturing-a-human-factors-usability-study-av-technology/
14
Appendix A: Calculation of Camera Views
15
MX Brio Webcam
Ray-Ban Smart Glasses
From Logitech’s website [link]:
From a product brochure [link]:
Appendix B: Calculation of MCS of Webcam
16
Calculate ideal area
Select captured area
Calculate captured area
0.071m2
0.068m2
Appendix C: Calculation of MCS of Cameras
17
0.011m2
0.193m2
0.023m2
0.023m2
0.017m2
0.009m2
Monitor coverage approximated using primitive shapes
Hand coverage approximated using surface area
Appendix D: Calculation of MCS for Typical Study
The standard MCS Score was developed using a usability test set-up that includes a PTZ Camera and a webcam to capture visual user data. Using the formula created above this means that N = 1 since only camera components are apparent in the design. W = 0.6 for cameras and C = 75% for the PTZOptics Camera and 45% for the Webcam and thus we take their average. Calculating, this results in a minimum value of 0.36 for MCS Score.
18
Study adapted from [17].
Appendix E: Full List of Objectives
19
Rank | Objective (Design Should…) | Metric | Goal | Justification |
O.1 | Maximize coverage of a test’s visual elements | Multimodal coverage score (MCS) (Appendix H) | ≥ 0.36 | Visual data quality depends on both the types of data collected, as well as how well each type is collected. Typical settings usually have a score of 0.36 (Appendix H). |
O.2 | Reduce time to synchronize footage | Ratio of processing time to time recorded | < 5:1 | The current ratio is around 5:1 for the client, which they stated is time-consuming (Appendix A). |
O.3 | Capture high-quality videos | Frames per second | ≥ 60 | Testing requires determining the duration of a behaviour, which is done more precisely with more frames. [11] (Appendix A). |
O.4 | Have remote operability of observation tools | Number of devices in each unique video stream that can be controlled without physical contact in observation room | ≥ 1 from each | Interrupting a study to modify a tool may change participant behaviour, introducing biases in results [12]. |
O.5 | Improve separation of speakers in audio tracks | Diarization error rate (DER) of processed data (Appendix I) | < 15% | The current state of the art automated speaker diarization model achieves an average DER of around 9% [13]. Many standard models fall under 15% DER. |
O.6 | Maximize length of stored data | Minimum length in hours of footage capable of being stored from any video and audio stream | ≥ 10 | The typical usability test takes 60-90 minutes per user, with 5 users being the standard amount for testing (a total of 7.5 hours per test) [14][15]. |
O.7 | Make data capture components removable | Percentage of components removable and installable elsewhere without structural damage | ≥ 60 % | Increased portable components allow more accurate data in external locations (Appendix A). |
O.8 | Make data capture components easy to move | Weight of data capture component along with peripherals that must be moved for its function (ie. wires) | < 10kg | Objects above 10kg will legally require employer training [16]. |