| [Online Peer Evaluation & Peer Feedback System]_ |
 Decision Analysis
This document records the analysis of various problems encountered, questions investigated, choices made, strategies devised, etc. When a problem requires detailed analysis, we present the analysis as a separate technical report.
The format can be PROBLEM/SOLUTION, QUESTION/ANSWER, GOAL|CHALLENGE|TASK/STRATEGY, CHOICE/RATIONALE etc.
Table of Contents
[ Functionality]__________________________________________________
Functionality: Related research
QUESTION: What are the findings from related research?
QUESTION: How does TEAMMATES address typical drawbacks of peer evaluations?
QUESTION: How do existing alternatives compare to TEAMMATES?
Functionality: Value for instructors
GOAL: Make the app support related learning outcomes
GOAL: Reduce extra work required by the app from instructors
GOAL: Support administrative goals of instructors
Functionality: Value for students
PROBLEM: Students consider peer evaluations a ‘time waster’
PROBLEM: Self-serving bias of peer-evaluations
PROBLEM: Students delay negative feedback until the end of the course
PROBLEM: Account claimed by wrong person
PROBLEM: Instructor wants to keep a record of original values before moderation
[ Implementation]______________________________________________
PROBLEM: Logs get messy and less useful
PROBLEM: Code comments are not effective
Implementation: Error handling
PROBLEM: GAE doesn’t support assertions
PROBLEM: GAE can throw unchecked exceptions
PROBLEM: Exception handling can become chaotic if done in an ad hoc way
Implementation: Data validation
QUESTION: Which characters to be allowed in person names?
GOAL: Make TEAMMATES thread-safe
Implementation: Tools and technologies
[ UI ]_______________________________________________________________
CHALLENGE: Checking tablet experience from a PC
GOAL: Support popular mobile devices
PROBLEM: Some browser might have JavaScript disabled
QUESTION: Which browsers to support?
CHALLENGE: Handling unsupported browsers
TASK: Testing for browser compatibility
PROBLEM: UI inconsistencies across different browsers.
PROBLEM: All-in-one style sheets
GOAL: A ‘Google quality’ look-and-feel
CHOICE: Client-side scripting technology
[ Storage]_________________________________________________________
CHALLENGE: Hide ‘eventual consistency’ from users
PROBLEM: Edit operations are not persisted until PersistenceManager is closed.
TASK: Use transaction control for our database operations
CHALLENGE: Implement schema changes over legacy data
[ Testing ]_________________________________________________________
PROBLEM: Cost of setting up test data
PROBLEM: Checking for emails takes time
PROBLEM: Not possible to run an arbitrary selection of tests
PROBLEM: Test suite consumes free quota too soon
PROBLEM: Failures due to ‘eventual consistency’ nature of datastore
PROBLEM: Random minor variations in page source (browser quirks)
PROBLEM: Tests are sensitive to dynamic data
PROBLEM: Dev server testing is not representative of live system
PROBLEM: Tests can affect normal operations
PROBLEM: Tests require real Google accounts
CHALLENGE: Increase test coverage
CHALLENGE: Increase testing frequency
CHALLENGE: Automated testing of a Web UI
CHALLENGE: Testing for UI appearance, not just content
CHALLENGE: Testing with dynamic UI elements
TASK: Testing static web pages
PROBLEM: Browser abstraction class is too big
Testing: Tools and technologies
CHOICE: Javascript testing tool
PROBLEM: Chrome/IE Web driver processes do not die after the testing
[ Documentation]_______________________________________________
[Usability]______________________________________________________
CHALLENGE: Make the app self-guiding
[ Security]_______________________________________________________
CHALLENGE: DoS attack protection
TASK: Prevent ‘abusive overuse’ of functionality
[ Scalability & Performance ]__________________________________
[ Administration]_______________________________________________
CHALLENGE: Monitoring activities
CHALLENGE: Â Monitoring for errors
CHALLENGE: Â Monitoring performance
TASK: Devise a way to take the system offline
Administration: Troubleshooting
PROBLEM: Difficult to locate entities
TASK: Backup/restore from within the same app
TASK: Backup/restore from one app to another
TASK: Download/upload between GAE and local disk
Administration: Project statistics
TASK: Calculate data entity count
[ Process]________________________________________________________
CHALLENGE: Define a suitable workflow
GOAL: Make the project set up easier
TASK: Define a suitable repository architecture and branching model
PROBLEM: Not enough manpower to guide many developers
PROBLEM: Novice contributors could mess up the repo structure.
[ Marketing]_____________________________________________________
TASK: Increase visibility to search engines (Search Engine Optimization)
TASK: Utilize direct marketing
TASK: Utilize indirect marketing
GOAL: Create a high brand value
[ Holding bay ]___________________________________________________
This section is about the functionality of the system.
ANSWER: none yet
TODO: Find related papers and summarize findings as a separate literature review.
What are the typical drawbacks of using peer evaluations in education? How does TEAMMATES address those?
E.g.,
Instructors are our primary adopters as they make the decision to use or not to use the app. Therefore, it is important that the app provides enough value to make them adopt the app.
STRATEGIES: Identify what are the typical learning outcomes that the app can contribute toward. Analyze how the app can support these outcomes.
TODO:
STRATEGIES: Identify extra work required and try to eliminate them, make them easier, make them optional.
TODO:
STRATEGIES: Identify typical administrative goals and support them.
TODO:
[TODO]
Students may try to use the system to pretend they are doing more than what they are actually doing.
[TODO]
Students hold back negative feedback until the course is over.
[TODO]
Situation: The key was sent to the wrong person and the account was claimed by that person.
Resolution: Delete the student and add again. This will generate a new key.
Resolution: Just download the report before moderating values.
This contains a list of feature ideas and thoughts about the app that may be useful in the future.
                       Â
                        Â
Other thoughts       Â
                       Â
GAE allows apps to log messages at different levels (info, warn, etc.). Logs can become messy if developers logged information at random logging levels.
Solutions [TODO]:
Contributors: Kenny Tan
We would like our logs to cover all past versions.
Solution: Define guidelines for writing comments.
Status: Some guidelines are contained in DevMan and Java style guide, but may be not enough.
TODO: Define more elaborate guidelines
Assertions are useful to verify programmer assumptions. However, GAE does not support normal java assertions. There used to be a workaround for this that doesn’t seem to work anymore.
Solution: We implemented our own class Assumptions.java to emulate assertions. This class is based on org.junit.Assert class and has methods such as assertEquals, assertNotNull etc. Here’s an example of it being used.
However, unlike normal assertions, we do not have a way to turn these off.
Contributors: Wang Sha
GAE can throw some unchecked exceptions such as DeadlineExceededException.
Solution: ControllerServlet is the main point at which we catch and handle exceptions. It has a ‘catch all’ block that displays a generic error message for all unexpected errors. [Contributors: Wang Sha]
TODO: Find out more about all unchecked exceptions thrown by GAE and make sure we handle them appropriately
Problem: DeadlineExceededException kills the instance before sending the alert email to admin.
Solution: A cron job checks the GAE log every few minutes and email admin if there are severe errors. [Contributors: Shawn Lee]
Solution: TEAMMATES have policies for what kind of exception to throw for which situation. These are documented in the DevMan.
Contributors: Wang Sha
TODO:
Answer: It is very hard to determine what should be allowed in a name, since names can vary widely across the world. We have narrowed it down to a few characters that would have a higher probability of appearing in names.
Acceptable characters: alphanumeric characters, fullstops and commas (Smith, John D.), hyphens, apostrophes (Hal’ Jacob-Smith), rounded brackets (John Smith(Jr)), slashes (Tom S/O Tommy)
Also acceptable are whitespaces and tabs within names.
We decided not to allow all characters in names, because this may result in SQL injection (or a variant of it based on how GAE stores data).
Since V4.55, we allow any character in the name. It is not nice to say someone’s name is ‘invalid’ because it contained a non-alphanumeric character.  [Contributors: Chun Teck]
Since V4.78, we allow non-English characters in names. [Contributors: Theong Siang]
[TODO]
Possibly useful:
Current choice: Google App Engine
Alternatives: AWS, Azure, Heroku, Engine Yard, Digital Ocean
Rationale:
Current choice: Java
Alternatives: GAE supports only Java, Python, and Go. PHP support is in beta.
Rationale: Java was chosen because it is more widely used than Python.
Current choice: Eclipse
Alternatives: Jetbrains IDEA
Rationale: Eclipse is Free and widely used.       Â
Current choice: Standard JavaEE with Servlets and JSP
Alternatives: Struts, Clojure, Spring, ...
Rationale: We do not want to bring in extra frameworks unless necessary. Some frameworks may not play well with GAE, especially those using dependency injection.
TODO:
Current choice: JSON
Alternatives: XML
Rationale: more natural and less verbose syntax.
                Â
Current choice: GSON
Alternatives:
Rationale:Â Â Â Â Â Â Â Â
Strategy:
Status: Seems to be working fine on iPad. ‘Quick tap’ can produce hover over text.
TODO: Expand to other tablets.
[TODO]
Strategy: Identify which phones to support. Find what needs to be done to support them. Implement those changes and test.
Current status: seems to be working on standard mobile browsers, although the user experience is not optimal.
Solution: Although only some of functionality requires JavaScript, we decided it is simpler to block browsers that don’t have JavaScript enabled from accessing TEAMMATES.
Status: We detect if JavaScript is disabled and show how to enable it.
Answer: Because TEAMMATES uses HTML5, only HTML5 browsers are supported. Following statistics from several websites, http://www.w3counter.com/globalstats.php, http://gs.statcounter.com, we have decided to support the following browsers.
Internet Explorer
 IE 9 and above. IE 8 does not support HTML5. Although IE8 is still holds a significant percentage for browser usage, we believe that our target audience (students, faculty staff) would very unlikely be using IE8.
Chrome
Recent three Chrome releases.
Firefox
Recent three Firefox releases.
Safari
Safari 4 and above. Safari is the default browser for MAC OS users. While some users may choose to download other browsers, a large percentage of MAC users would use Safari. Safari 4 was released in 2009 and it supports HTML5.
Contributors: Chun Teck
The first step of handling unsupported browsers would be to first detect the browser in use. The code for this can be found in webapp/js/checkBrowserVersion.js. This code is adapted from http://www.javascripter.net/faq/browsern.htm
Once an unsupported browser has been detected, we have to decide what to do. One alternative is to restrict access to the website until the user upgrades or downloads a supported web browser. However, this would greatly reduce traffic to the website, especially deterring new potential users. A few random tryouts showed that TEAMMATES is still usable in most popular browsers.
Therefore, rather than blocking access from unsupported Browsers, we have decided to just include a warning/message if the user is using an unsupported browser. This message is included in the homepage, such that users would know if they are using an unsupported browser. The warning is posted right at the top of the webpage, so that they do not miss it.
BrowserShots (http://browsershots.org/) is used for testing. {TODO: put in a screenshot}
Contributor: Chun Teck
Strategies:
Status: Tests are runnable for Firefox, Chrome, IE. However, FF is the one used most because it is fastest. There is no procedure for ensuring tests pass on all Browsers.
Contributions: Chun Teck
Different Browsers can render the same web page slightly differently. Â The reason for this is that the default CSS styles of each browser are different. For example, one browser may set the default value for line-height differently from another browser. Another browser may set the default value for the margin differently. This means that while the web-page may look nice and fine in one browser, the style and layout for the same web-page may become skewed in another browser.
e.g. Visual elements (border styles) are rendered differently (source: impressivewebs.com)
Solution: CSS resets
The solution to this problem is to introduce a CSS Reset into the website. The aim of a CSS Reset stylesheet is to remove all the inconsistent default styles that a browser may specify. Once these default styles are removed, the styles for the web application can then be introduced. This ensures that all the styles are the same throughout regardless of which browser is being used to render the web-page. The most widely used CSS Reset stylesheet Eric Meyer’s Reset Stylesheet, which is the same stylesheet that is used in TEAMMATES.
Since we migrated to bootstrap, there is no need for CSS resets anymore.
Contributors: Chun Teck
Previously, the CSS for the entire web application was consolidated into 2 very large stylesheets. Every web-page in the application would include these 2 stylesheets in their HTML document. Because all the styles from the entire web application are in these 2 stylesheets, they are very large and difficult to maintain. As a result, there are several inconsistencies within the 2 stylesheets, with some styles being defined twice, and styles from one stylesheet being overridden from the other stylesheet.
Solution: Split up the stylesheets.
We created one common stylesheet, called common.css, which is used by every web-page in the application. The CSS Reset stylesheet will be part of this common stylesheet. This stylesheet will hold the styles for the general theme of the web application. Next, we created individual stylesheets for each web-page, which will only contain styles that are particular for that specific webpage. In this way, the CSS is more structured, and it is much easier to sort and edit the different styles if required. It is also easy to change the entire theme of the website, since all the styles are consolidated in the common stylesheet.
One downside of this solution is that it increases the number of requests a browser has to make to render a page.
After moving to bootstrap, we are back to using a single custom css file for the entire application.
Contributors: Chun Teck
The theme, color scheme, fonts etc. should fit together nicely and the whole look and feel should be as good as that of products done by Google, Apple, Microsoft etc.
Strategy: to use the Bootstrap UI framework
Alternatives: HTML5, XHTML5
Choice: HTML5
Rationale:
Further information can be found in our HTML Style Guide
Contributors: Chun Teck
Not yet using CSS3. Refer to our ‘CSS standard’ for more details
Choice: Bootstrap
Choice: JQuery
{TODO: add a separate technical paper}
Current choice: JDO
Alternatives:
Rationale:
[TODO: add technical report written by Shawn]
Edit entity operations do not persist in dev server, unless we close the persistent manager. In a production server, the PersistenceManager gets closed at the end of the request.
Status: We close the PersistenceManager after each edit operation.
[TODO]: Analyze further and figure out whether this is the best way to tackle this situation.
We would like to use GAE’s transaction control facilities to our best advantage.
{TODO: add technical paper to elaborate this}
Since TM is currently being used by users, any changes to schema needs to be implemented on existing legacy data as well. One example of such a schema change is the one we did when adding support for multiple instructors.
Options:
Current choice: We used option 2b. Adopting a new framework (i.e option 1) is considered too drastic. Option 2b wasn’t necessary because our dataset was small.
{TODO: add technical paper to elaborate this. Move the references to the technical paper}
References:
When testing a feature deep within the system, the test needs to start from home page and navigate to page being tested.
Solution: Make each page reachable directly.
Status: Most pages are reachable directly. But still needs to login first.
[Solution?] Use a separate set of data for UI tests and keep them persistent?
This can be a problem when running tests in parallel and the server times out due to high load.
Current status: No longer a problem. A stress test showed that the GAE can respond to high load pretty quickly.
For example, trying to create an entity immediately after deleting it can cause EntityAlreadyExistsException because the delete operation is taking time to persist.
Current status: Some ‘retry’ mechanisms are in place to minimize this problem. More can be done.
In our current UI tests, we compare the HTML code that we expect to be in a certain page against the actual HTML code received by the browser. This task is complicated by the fact that different browsers spit out the HTML is slightly different ways.
Options:
Current status: Tried option 1. It worked, but it was difficult to debug when a test failed. Now using option 2, which seems to be working pretty good so far.
Contributors: Chun Teck
e.g. dates
Partial solution: use wildcard ‘*’ to exclude those parts of the html from the comparison
(i) sending emails
(ii) tests involving other GAE services that are not present on dev server
(i) Risk of live data getting wiped out
This is a problem because all developers have to share the password for test accounts.
Current status: Sharing of accounts is no longer necessary. Developers can use test accounts they created themselves. Still, each developer need 4-5 Google accounts to run the tests against a production server..
For Java: EclEmma.
For JavaScript: JSCoverage.
Contributors: Chua Hong Jing, Cheung Yew Kong Benjamin, Loke Yan Hao
Current solution: Current test coverage (Jun 2014) is around 80-90%. This is good, but can be improved further.
Challenges: The high cost of testing can reduce the test frequency.
Current solutions:
Our UI consists of web pages. Automating the testing of such a UI is not as easy as automating API testing.
Solution: Use selenium to automate browser actions.
 We need to automate comparing how a web page actually looks against how it is expected to look.
Solution: Create a master page (a mashup of all pages) that can be used for an eye ball test. A screenshot is given below.
TODO: Look for ways to automate this.
Contributors: Chun Teck.
                Â
AJAX-based UI features, hover over actions etc. makes testing harder. For example, if we move the mouse during a testing session, it can cause a tool tip to pop up, which dynamically modifies the page source (using JavaScript), which makes the test fail because the actual page source is now different from the expected page source.
Solutions:
Ensure the correctness of the static web site, especially, links.
Solution: [TODO]
The class we use to abstract the Browser (currently called BrowswerInstance.java) is getting too big and unwieldy.
Solution: [DONE] Apply the Page Object Pattern.
Bibliography:
Current choice: TestNG
Alternatives: Arquillian, JUnit
Rationale:
JUnit was our initial choice. We decided to move to TestNG after Milestone V4.53 because
The downside is,
Current choice: QUnit
Alternatives: Jasmin
Rationale:
Current choice:Selenium
Alternatives:
Rationale: One of the most commonly used library to do UI test
Current choice: NekoHTML
Alternatives: Â Â Â Â Â Â Â Â
Rationale: Already used by Selenium.
There is an issue regarding the Chrome Driver found here: http://code.google.com/p/selenium/issues/detail?id=3378Â
In order to kill the driver, we have to use the WebDriver API, calling the function driver.quit(). However, this only works for the IE driver, and not for the Chrome Driver, as found in the issue mentioned above. Additionally, killing the driver process will also close all browser windows that are linked to the driver. As a result, we would not be able to see the test results at the end of the test if we program the driver to quit after the tests.
Due to the way we write our test suites, we do not have a handle to the driver, and we are unable to get the driver to quit at the end of the test suite.
Solution: We will have to manually kill the chrome driver and IE driver processes by ourselves, using the task manager. Â
Contributor: Chun Teck
Choices: HTML, GoogleDoc, PowerPoint, Wiki
Current choice:
Generating javadoc, dependency diagrams.
Choices: javadoc tool, DoxyGen
Current choice: none
CHALLENGE: Injection attack protection
E.g., creating classes with too many students etc.
We have done some experiments to measure TM scalability up to 100K users and found the performance to be reasonable, but can be improved further.
Also see:
We have identified some possible scalability/performance bottlenecks and attempted to remove some of them.
Also see:
Possible tools: Â AppStatÂ
The admin needs to know which users are performing what kind of activities at a given time.
Current solution: Have an admin page that reports activities based on log messages.
TODO: Make the activity log more informative and applicable over past versions.
Contributors: Wang Sha, Chun Teck
While critical errors are captured in the application log file, the admin cannot be monitoring the log all the time.
Solution:
Sometimes, we need to make the application ‘unavailable’ to users to carry out maintenance activities. In such cases, we need a way to put the app in ‘maintenance mode’ easily and restore access easily after the work is done.
Option 1: GAE allows deploying multiple versions of the app. We can deploy one version that redirects all requests to a ‘under maintenance’ page. When we want to put the app in maintenance mode, we simply make that version the default.
Option 2: ?
Status: Option 1 used (see Issue 428).
Todo: Have a weekly maintenance window e.g., do not allow any deadlines in that period.
Contributors: Chun Teck
When a user contacts the support email, the system admin needs to locate entities (Students, Courses etc.) quickly.
Solution: We have implemented an entity search facility for system admins. This facility currently uses Google’s text search API. [for more info, see this technical report]
Contributors: Wang Sha
GAE provides ways to backup and restore data.
Currently, we have configured the system to take weekly backups (see Issues  307 343).
In addition, admin can take a backup anytime. Admin can also restore data from an existing backup anytime.
These backups are put in the blobstore, which has a free quota of 2Gb. The other option is to put the data in the cloud store. Cloud store has a free trial period.
TODO: Find a way to delete older backups automatically. Currently, this has to be done manually. Â
This ability is useful for replicating the data in another app for experimentation purposes. This ability is supposedly provided by GAE (see here). We are yet to try it out.
TODO: Try it out.
We do not have an urgent need for doing this at the moment. One reason for doing this is to save storage space for backups. i.e. backup to local disk instead of blob store or cloud storage. However, it is unlikely to be scalable.
Currently calculated by ohloh.net as given here.
It would be useful to know the current number of users, courses etc.
GAE now provides an API for accessing datastore statistics. In future, we hope to make use of this API to generate required statistics.
References:
The workflow should suit these characteristics of our project.
Status: Current workflow is defined in DevMan
TODO: Study the workflows of established OSS projects and adopt practices applicable to our project.
Solutions:
Current choice:
See devman for more details on the current repository architecture. Here are the salient features.
Alternatives:
Solutions:
TODO: Study how bigger OSS projects manage large developer communities and adopt practices applicable to our project.
Solution: Contributors do not have permission to push to the main repo. Only reviewed and tested code is pushed to the main repo.
Current choice: Previously GoogleCode, now GitHub
Alternatives: BitBucket, SourceForge, GoogleCode
Pros: Free and widely-used.
Cons: The issue tracker is not as powerful as GoogleCode
Current choice: previously Hg, now Git
Alternatives: SVN, Hg
Pros: Widely used and powerful.
Cons : Steep learning curve.
Alternatives:
Current choice: No special tools. Just use Eclipse to build and deploy.
Rationale: We experimented with Maven for a while. We found it to be too restrictive and an unnecessary complication. GAE plugin for Eclipse seems to be adequate for our purposes at this point.
Future work: Study more build/integration tools such as CruiseControl
Contributors: James Dam Tuan Long
Current choice: GitHub (previously: GoogleCode, Rietveld).
Contributors: James Dam Tuan Long, Steve Ng, Jie Sheng
GAE apps are not indexed by Google?
Create a blog?
Upload video to YouTube?
Tools: Google Webmaster Tools
{TODO: add technical report}
Contributors: Chun Teck
Advertising
Contacting educators?
Post in education related forums?
Enter competitions?
Get reviewed?
Area for holding items for future processing. These will be absorbed into the document at a later time.
Discarded
--- end of document ---