Mobile Base Sinhala �Book Reader For �Visually Impaired Individuals
Project ID : 2023-198 |
Supervisor : Prof. Koliya Pulasinghe |
We are….
Jayathunga T.M.
IT20146238
Semini J.P.D.L
IT20241346
Godakanda P.G.S.
IT20129712
Bhagya H.D.M.
IT20254520
Prof. Koliya
Pulasinghe
Ms. Poorna Panduwawala
We are Pleased to present Mobile Base Sinhala Book Reader for Visually Impaired Individuals that will ease out the process of Reading Children Story Books for Blind Students.
1
2
4
3
Text-to-Speech (TTS) Synthesizer
Image Detection
Object Detection & Voice Navigation
Optical Character Recognition(OCR)
Problem….
How can the accessibility of the book reader be enhanced for visually impaired individuals?
What are the user requirements and preferences for a mobile base Sinhala book reader for visually impaired individuals?
What are the existing technologies and tools available for creating a mobile-based Sinhala book reader for visually impaired individuals?
The Solution
Mobile Base Sinhala Book Reader Application
Ensure accurate Sinhala text-to-speech (TTS) capabilities
The app must support the Sinhala language.
The app must support the Sinhala Voice Navigation System.
Integrate voice recognition for hands-free navigation and control.
Provide alt text for images and illustrations to describe visual content.
Solved…..
Background
Visually impaired children can read Sinhala story books independently without external help.
Commercialization…
Overview
Child open the app using Voice command
Phone has been placed on the phone holder above the book
App gives the vocalized feedback to the child choose the option
When placed the book under the camera it starts reading line by line through camera
If Child uses book reading option, it opens the camera and load the models
Two Options are available as Book reading and voice navigation
Overview
The audio book, stores the results at the same time while reading the book for later review
When choose navigation option it opens the camera and loads navigation models interacts with TTS
App detects the objects surround the child and identify them in Sinhala
In book reading Child can use voice commands to stop or continuing the reading
And when there are images its basically describe what kind of image it is.
System Overview Diagram.
Mobile Application
App Design
Demo-Research Area PP-2
Proof of Fronted Application Working
Demo-Research Area PP-2
Proof of Backend API of Project
Demo-Research Area PP-2
Proof of Model Integrated
Text-to-Speech�(TTS)�Synthesizer
1
Sri Lanka Institute of Information Technology
Jayathunga T.M. – IT20146238
09-07-2023
Jayathunga T.M.
IT20146238
Background
The main purpose of using Text to speech (TTS) technology is to give a blind person the ability to access the printed text of a Sinhala book. This allows them to easily listen to the valuable content of Sinhala books. TTS technology allows the written text in a Sinhala book to be read out loud in a natural-sounding voice, which makes it easier for visually impaired people to understand the content.
Research Gap
Without the speech of an outside person, Sinhala text converts into clear speech and makes it easy for visually impaired students to here.
Given visually impaired students the opportunity to listen to the audiobook at the end of each book.
Research Problem
How can a Sinhala text-to-speech conversion system accurately handle the nuances of the language?
How to evaluate the effectiveness of the mobile-based Sinhala book reader in enhancing the reading experience of visually impaired individuals?
How to create an audiobook for each book?
Objectives
Developing a highly accurate Sinhala text-to-speech conversion system. Evaluating the effectiveness of a mobile-based Sinhala book reader and integrating the Sinhala TTS into it. Assessing the Sinhala TTS conversion system for visually impaired individuals. Develop an audiobook for each story books.
Sri Lanka Institute of Information Technology
Jayathunga T.M. – IT20146238
09-07-2023
METHODOLOGY
Sri Lanka Institute of Information Technology
Jayathunga T.M. – IT20146238
09-07-2023
Methodology
The text-to-speech application's architecture has been carefully designed by analyzing existing literature, as shown in Image. This thorough investigation forms the foundation for future evaluations, working seamlessly with the recommendations in the system design guide to ensure a consistently reliable performance. The architectural framework is meticulously structured around four pivotal components, each of which serves an indispensable role in ensuring the seamless and consistent functionality of the system.
The architecture of the text-to-speech application was meticulously designed by analyzing existing literature, as displayed in the accompanying image. This thorough investigation forms the foundation for future evaluations, working in conjunction with the recommendations in the system design guide to ensure consistently reliable performance. The architectural framework is carefully organized around four pivotal components, each of which plays an indispensable role in guaranteeing the seamless and consistent functionality of the system.
�
Seq2Seq
Proofed of Model Create
Import Libraries and Load the Model Dataset
Sri Lanka Institute of Information Technology
Jayathunga T.M. – IT20146238
09-07-2023
Training Model Dataset
Backend Response
Sri Lanka Institute of Information Technology
Jayathunga T.M. – IT20146238
09-07-2023
TTS Response
Create Audio Book
Sri Lanka Institute of Information Technology
Jayathunga T.M. – IT20146238
09-07-2023
Optical Character Recognition (OCR)
2
Sri Lanka Institute of Information Technology
Semini J.P.D.L. – IT20241346
09-07-2023
Semini J.P.D.L.
IT20241346
Background
Optical character recognition is the process of classification of optical patterns contained in a digital image. This turns images of typed of printed text into machine-encoded text that might be electronic or mechanical. This allows users to scan and recognize text from images or documents using the camera of their smartphone. With the help of OCR technology, the app can convert recognized text into spoken words, allowing visually impaired individuals to access and understand the content.
Research
Gap
One possible research gap in the OCR component is the need for a more accurate and efficient OCR engine specifically designed for the Sinhala language. This is specifically for the blind children who are unable to read, and this kind of existing OCR programs are designed for English and industrial use, so this application is mainly for the Sinhala language.
Research
Problem
Most visually impaired children in Sri Lanka doesn't have a proper platform to capture the texts they need and read it.
Identify the limitations of that visually impaired people while using a smart phone and their knowledge about the technology.
Objectives
Developing a reliable and accurate OCR system for identifying the Sinhala texts include in the children story books and send it to the TTS. Then integrate and evaluate the Sinhala OCR system for visually impaired individuals
Sri Lanka Institute of Information Technology
Semini J.P.D.L. – IT20241346
09-07-2023
Seq2Seq
Proofed of Model Create
Import Libraries and Load the Model Dataset
Sri Lanka Institute of Information Technology
Semini J.P.D.L. – IT20241346
09-07-2023
Training Model Dataset
METHODOLOGY
Sri Lanka Institute of Information Technology
Semini J.P.D.L. – IT20241346
09-07-2023
Developing a Sinhala OCR (Optical Character Recognition) system is a complex task that involves various stages of data collection, preprocessing, training, and evaluation for defining the scope of Sinhala OCR project. Determine the specific use cases and requirements, such as recognizing printed text .Overall here, the text is recognized from scanned printed images and gives the output of that recognized text through TTS.
Sri Lanka Institute of Information Technology
Semini J.P.D.L. – IT20241346
09-07-2023
OCR Response
Backend Response
Object Detection�&�Sinhala Voice Navigation
3
Sri Lanka Institute of Information Technology
Godakanda P.G.S. – IT20129712
09-07-2023
Godakanda P.G.S.
IT20129712
Background
Blind students encounter substantial hurdles in navigating their surroundings independently. In many countries, there is an absence of comprehensive assistive technologies in local languages, including Sinhala. These technologies typically rely on English for object recognition and navigation instructions. This linguistic barrier exacerbates the accessibility gap for Sinhala-speaking blind individuals.
Research
Gap
The primary issue at hand is the absence of a Sinhala-specific object detection and navigation system tailored to the needs of blind students. Existing solutions predominantly cater to English-speaking users, rendering them ineffective for Sinhala-speaking individuals who require navigation assistance in their native language.
Research
Problem
The overarching objective is to create a specialized object detection and navigation component for blind students, employing the Sinhala language. This component aims to empower visually impaired individuals by providing them with real-time information about their surroundings, object identification capabilities, and navigational instructions, all presented in Sinhala.
Objectives
The current research landscape lacks comprehensive Sinhala-language solutions for object detection and navigation for the visually impaired. The dearth of localized tools places Sinhala-speaking blind students at a significant disadvantage, hindering their ability to effectively perceive and interact with the environment. Bridging this gap is essential for promoting inclusivity and independence among this marginalized group.
Sri Lanka Institute of Information Technology
Godakanda P.G.S. – IT20129712
09-07-2023
METHODOLOGY
The methodology involves a comprehensive approach to address the research objective. Data collection comprises gathering a diverse dataset of objects with labels and recording corresponding audio descriptions. Model development utilizes computer vision techniques to train an object detection model with Sinhala labels. A voice-based navigation system is developed in Sinhala, incorporating text-to-speech technology. Integration into a user-friendly app interface follows, with usability testing and iterative refinement based on user feedback. Accessibility standards are upheld, and continuous improvement ensures the app remains up-to-date and effective. User education resources support blind students in utilizing the app for enhanced navigation and object recognition.
Sri Lanka Institute of Information Technology
Godakanda P.G.S. – IT20129712
09-07-2023
Proofed of Model Create
Import Libraries and Load the Model Dataset
Sri Lanka Institute of Information Technology
Godakanda P.G.S. – IT20129712
09-07-2023
Backend Response
Sri Lanka Institute of Information Technology
Godakanda P.G.S. – IT20129712
09-07-2023
Object Detection Response
Navigation Command Response
Image Detection
4
Sri Lanka Institute of Information Technology
Bhagya H.D.M. – IT20254520
09-07-2023
Background
Access to educational material for visually impaired students is a pressing concern. In Sri Lanka, a deficiency in Sinhala language image description technology hinders inclusive access to student books.
Research Gap
This study aims to address the accessibility gap by developing an image detection system capable of providing comprehensive and culturally relevant descriptions in Sinhala within a student book reading app, catering specifically to visually impaired students.
Research Problem
Current solutions primarily cater to English-language educational materials, leaving a substantial void in Sinhala student book accessibility. This research addresses this gap by proposing an image description framework tailored to Sinhala educational texts.
Objectives
The lack of a Sinhala image description tool within student book applications poses a significant hurdle for visually impaired students in Sri Lanka to access and comprehend local educational materials effectively. This research endeavors to mitigate this issue through innovative technology.
Sri Lanka Institute of Information Technology
Bhagya H.D.M. – IT20254520
09-07-2023
METHODOLOGY
This research employs a multi-phase approach, combining computer vision techniques with natural language processing. It involves dataset collection, training
deep learning models, and
integrating the image
description system into
the Sinhala student book
reading app, followed by
user testing and iterative
refinement
Sri Lanka Institute of Information Technology
Bhagya H.D.M. – IT20254520
09-07-2023
Sri Lanka Institute of Information Technology
Bhagya H.D.M. – IT20254520
09-07-2023
Proofed of Model Create
Backend Response
Sri Lanka Institute of Information Technology
Bhagya H.D.M. – IT20254520
09-07-2023
Task | Progress Presentation 1 | Progress Presentation 2 |
Gathering Requirements | | |
Complete Figma UI Design | | |
Gathering Data | | |
Complete Frontend Design | | |
Data Set Training | | |
Creating Models | | |
Models Integration | | |
Crete Backend API | | |
Completed Integration Mobile Application | | |
Work Breakdown
Jayathunga T.M.
IT20146238
Semini J.P.D.L.
IT20241346
Bhagya H.D.M.
IT20254520
Godakanda P.G.S.
IT20129712
Tools & Technologies to be used
Android Studio, Vs Code, TensorFlow, Google API, Flutter, Cloud, Dart, Postman, Npm
Python, HTML, CSS
Business Model
INITIAL
�
FURTHER | | |
|
| |
FUTURE | | |
|
| |