Data Wrangler extension for WSO2 Machine Learner
Overview
What is WSO2 machine learner?
Machine learning [1] is about building models from sample data. So that these models can be used to analyse big data. Machine learning is a key component of analysing big data. WSO2 is in need of integrating machine learning into their platform. This can be used in various WSO2 products.
What is Wrangler?
Wrangler [2] can be used to transform and clean input test data. According to this transformation, Transformation scripts can be generated either in python or JavaScript. These scripts can be used to analyse big data.
What is Jaggery?
Web applications need various web technologies. Front-end needs HTML, CSS, JavaScript. Back-end deals with POST, JSON, XML. Server side logic is in java, PHP. With all these technologies mismatches occur. Jaggery [3] is a Completely JavaScript based way to eliminate with mismatches in web development.
The idea behind the project
Project is Data Wrangler extension for WSO2 Machine Learner. Idea of the project is create the Wrangler in the machine learner. In here Jaggery UI will be used instead of Wrangler UI. Jaggery is connected to Wrangler. When an end-user connects to Jaggery and Transforms & cleans the data, Scripts are generated from Wrangler. Those scripts are converted into java and send to the back-end. These converted scripts can be used to transform big data efficiently. Wrangler, Jaggery and back-end all belong to machine learner. In this project Wrangler is also integrated into machine learner as an extension.
Design and Implementation Plan
Technologies used in this project
Java/HTML/JavaScript
Apache Spark
Jaggery
Machine Learner Architecture
All the part implemented in should be added to the above architecture. Where each part is relevant in the above structure is explain below.
- Implementing Jaggery UI
- Selecting subset of operations from Wrangler and Design customized UI.
- According to the design implementing the user interface using Jaggery.
- This should be added to the top of the above structure.
2.) Connect Jaggery UI to Wrangler
Wrangler has user interface and also back-end for scripts generating either in python or JavaScript. Jaggery UI should be usable as Wrangler UI for selected subset of operations. Wrangler libraries can be used to perform tasks on data cleaning and transformations for Jaggery UI. In order to this Jaggery UI should be connected to the Wrangler libraries. Next part is to get the generated JavaScript for selected operations using Jaggery by the user. In here Jaggery UI should be connected to the back-end of the Wrangler. Operations which is selected by the end user using Jaggery UI set as the input for the Wrangler back-end and get the generated scripts in JavaScript. In order to do this part Wrangler API can be used. Otherwise study how Wrangler works and create an API for done above connections. Wrangler should be in between machine learner UI and machine learner REST API.
3.) Conversion
This is the core part of the project. Get the scripts generated from the Wrangler. The API is necessary for the getting those scripts. Develop the API for doing that. Use Apache Spark transformation libraries and mapping the JavaScript functions of generated scripts to Java code. The generated scripts have a particular format. Understanding the format and Developing a tool for compiling the generated scripts into java.
4.) Developing the Back-End
The back-end should be in the machine learner core shown in the above diagram. Developing the back-end in the machine learner core. Include those converted scripts into the back-end.
final structure of all the parts included can be shown as below.
Deliverables
- The implementation of the extension.
- Documentation and tests.
- Final documentation.
Timeline
Pre-work
- Get an idea about Data Wrangler API.
- Literature survey of Jaggery and sample codes.
- Literature survey of Apache Spark Transformation.
- Literature survey of WSO2 machine learner.
May 11 — May 24
- Implementing user interface using HTML/Jaggery.
May 25 — June 14
- Connect Jaggery UI to Wrangler.
June 15 — June 25
- Developing API for getting the generated scripts.
June 26 — July 3
- Mid-term evaluation.
July 4 — July 26
- Implement a tool for converting the generated scripts into java.
July 27 — August 2
- Developing the back-end for including converted scripts.
August 3 — August 9
- Implementing test cases.
- Testing and bug fixing.
August 10 — August 16
- Refining the code.
- Preparing the documentation.
August 17 — August 23
- Final test.
- Final evaluation.
Community Engagement
I’ve been discussing about this project since 7th of March 2015. My initial understanding of the project was misled. The community helped me understand what it really is and recommended where to read more about the Wrangler API and Jaggery. This helped me study Wrangler’s source code and understand more clearly about Wrangler back-end and also the format of the generated scripts of it. Then I downloaded the source code of the Jaggery and build some programmes with it. I was stuck with the conversion part. The community told me to use Apache Spark transformation. Therefore I downloaded the source code of apache spark. I had some problem with building apache spark. The community solved that problem too. In essence, the community helped me gain a much better perspective on the purpose of and how to do the project.
About Me
I am Tharinda Ehelepola from University Of Peradeniya, Faculty of engineering. I am currently studying as a 3rd year student at department of computer engineering. I have knowledge in Java, OOP concepts, HTML, PHP and JavaScripts. I have already done projects in Java and also in PHP. I also have knowledge about data structures and algorithms.
Contact information
Name : U.W.T.D Ehelepola
Address : Dombawela, Mahawela, Matale, Sri Lanka
Institution : Faculty of Engineering, University of Peradeniya, Sri Lanka
E-mail : Tharinda221@gmail.com
T.P : (+94)71 35 39 164
LinkedIn : TharindaEhelepola