[Online Peer Evaluation & Peer Feedback System]_ 

 Decision Analysis

This document records the analysis of various problems encountered, questions investigated, choices made, strategies devised, etc. When a problem requires detailed analysis, we present the analysis as a separate technical report.

The format can be PROBLEM/SOLUTION, QUESTION/ANSWER, GOAL|CHALLENGE|TASK/STRATEGY, CHOICE/RATIONALE etc.

Table of Contents

[ Functionality]__________________________________________________

Functionality: Related research

QUESTION: What are the findings from related research?

QUESTION: How does TEAMMATES address typical drawbacks of peer evaluations?

QUESTION: How do existing alternatives compare to TEAMMATES?

Functionality: Value for instructors

GOAL: Make the app support related learning outcomes

GOAL: Reduce extra work required by the app from instructors

GOAL: Support administrative goals of instructors

Functionality: Value for students

PROBLEM: Students consider peer evaluations a ‘time waster’

PROBLEM: Self-serving bias of peer-evaluations

PROBLEM: Students delay negative feedback until the end of the course

Functionality: workarounds

PROBLEM: Account claimed by wrong person

PROBLEM: Instructor wants to keep a record of original values before moderation

Functionality: Ideas

[ Implementation]______________________________________________

Implementation: Logging

PROBLEM: Logs get messy and less useful

Implementation: Comments

PROBLEM: Code comments are not effective

Implementation: Error handling

PROBLEM: GAE doesn’t support assertions

PROBLEM: GAE can throw unchecked exceptions

PROBLEM: Exception handling can become chaotic if done in an ad hoc way

Implementation: Data validation

QUESTION: Which characters to be allowed in person names?

Implementation: Parallelism

GOAL: Make TEAMMATES thread-safe

Implementation: Tools and technologies

CHOICE: Application platform

CHOICE: Programming language

CHOICE: IDE

CHOICE: Web App framework

CHOICE: Data encoding

[ UI ]_______________________________________________________________

UI: Device compatibility

GOAL: Support popular tablets

CHALLENGE: Checking tablet experience from a PC

GOAL: Support popular mobile devices

UI: JavaScript support

PROBLEM: Some browser might have JavaScript disabled

UI: Browser compatibility

QUESTION: Which browsers to support?

CHALLENGE: Handling unsupported browsers

TASK: Testing for browser compatibility

PROBLEM: UI inconsistencies across different browsers.

UI: File structure

PROBLEM: All-in-one style sheets

UI: Look & feel

GOAL: A ‘Google quality’ look-and-feel

UI: Tools and technologies

CHOICE: HTML version

CHOICE: CSS version

CHOICE: CSS framework

CHOICE: Client-side scripting technology

[ Storage]_________________________________________________________

Storage: Database model

CHOICE: Storage model

Storage: Consistency

CHALLENGE: Hide ‘eventual consistency’ from users

PROBLEM: Edit operations are not persisted until PersistenceManager is closed.

Storage: Transaction control

TASK: Use transaction control for our database operations

Storage: Schema

CHALLENGE: Implement schema changes over legacy data

[ Testing ]_________________________________________________________

Testing: Time cost

PROBLEM: Page loading delays

PROBLEM: Cost of setting up test data

PROBLEM: Checking for emails takes time

PROBLEM: Not possible to run an arbitrary selection of tests

Testing: Money cost

PROBLEM: Test suite consumes free quota too soon

Testing: Non-determinism

PROBLEM: Timeouts

PROBLEM: Failures due to ‘eventual consistency’ nature of datastore

PROBLEM: Random minor variations in page source (browser quirks)

Testing: Fragility

PROBLEM: Tests are sensitive to dynamic data

PROBLEM: Dev server testing is not representative of live system

PROBLEM: Tests can affect normal operations

PROBLEM: Tests require real Google accounts

Testing: Coverage

TASK: Measure test coverage

CHALLENGE: Increase test coverage

CHALLENGE: Increase testing frequency

CHALLENGE: Automated testing of a Web UI

CHALLENGE: Testing for UI appearance, not just content

CHALLENGE: Testing with dynamic UI elements

TASK: Testing static web pages

Testing: Design

PROBLEM: Browser abstraction class is too big

Testing: Tools and technologies

CHOICE: Java testing tool

CHOICE: Javascript testing tool

CHOICE: UI testing tool

CHOICE: HTML parser

Testing: Miscellaneous

PROBLEM: Chrome/IE Web driver processes do not die after the testing

[ Documentation]_______________________________________________

Documentation

CHOICE: Format

CHOICE: Document generation

[Usability]______________________________________________________

CHALLENGE: Make the app self-guiding

[ Security]_______________________________________________________

CHALLENGE: Data protection

CHALLENGE: DoS attack protection

TASK: Prevent ‘abusive overuse’ of functionality

[ Scalability & Performance ]__________________________________

S&P: Measuring

S&P: Improving

S&P: Tools and technologies

[ Administration]_______________________________________________

Administration: monitoring

CHALLENGE: Monitoring activities

CHALLENGE:  Monitoring for errors

CHALLENGE:  Monitoring performance

Administration: System status

TASK: Devise a way to take the system offline

Administration: Troubleshooting

PROBLEM: Difficult to locate entities

Administration: Managing data

TASK: Backup/restore from within the same app

TASK: Backup/restore from one app to another

TASK: Download/upload between GAE and local disk

Administration: Project statistics

TASK: Calculate LOC count

TASK: Calculate data entity count

[ Process]________________________________________________________

Process: workflow

CHALLENGE: Define a suitable workflow

GOAL: Make the project set up easier

TASK: Define a suitable repository architecture and branching model

Process: Community management

PROBLEM: Not enough manpower to guide many developers

PROBLEM: Novice contributors could mess up the repo structure.

Process: Tools

CHOICE: Project hosting site

CHOICE: Revision control tool

CHOICE: Build tool

CHOICE: Code review tool

[ Marketing]_____________________________________________________

Marketing

TASK: Increase visibility to search engines (Search Engine Optimization)

TASK: Utilize direct marketing

TASK: Utilize indirect marketing

GOAL: Create a high brand value

[ Holding bay ]___________________________________________________

[ Functionality]__________________________________________________

Functionality: Related research

This section is about the functionality of the system.

QUESTION: What are the findings from related research?

ANSWER: none yet

TODO: Find related papers and summarize findings as a separate literature review.

QUESTION: How does TEAMMATES address typical drawbacks of feedback management tools?

What are the typical drawbacks of using peer evaluations in education? How does TEAMMATES address those?

QUESTION: How do existing alternatives compare to TEAMMATES?

E.g.,

Functionality: Value for instructors

Instructors are our primary adopters as they make the decision to use or not to use the app. Therefore, it is important that the app provides enough value to make them adopt the app.

GOAL: Make the app support related learning outcomes 

STRATEGIES: Identify what are the typical learning outcomes that the app can contribute toward. Analyze how the app can support these outcomes.

TODO:

GOAL: Reduce extra work required by the app from instructors

STRATEGIES: Identify extra work required and try to eliminate them, make them easier, make them optional.

TODO:

GOAL: Support administrative goals of instructors

STRATEGIES: Identify typical administrative goals and support them.

TODO:

Functionality: Value for students

PROBLEM: Students consider peer feedback a ‘time waster’

[TODO]

PROBLEM: Self-serving bias of peer-evaluations  

Students may try to use the system to pretend they are doing more than what they are actually doing.

[TODO]

PROBLEM: Students hold back negative feedback until the end of the course

Students hold back negative feedback until the course is over.

[TODO]

Functionality: workarounds

PROBLEM: Account claimed by wrong person

Situation: The key was sent to the wrong person and the account was claimed by that person.

Resolution: Delete the student and add again. This will generate a new key.

PROBLEM: Instructor wants to keep a record of original values before moderation

Resolution: Just download the report before moderating values.

Functionality: Ideas        

This contains a list of feature ideas and thoughts about the app that may be useful in the future.

                        

                         

Other thoughts        

                        

[ Implementation]______________________________________________

Implementation: Logging

PROBLEM: Logs get messy and less useful

GAE allows apps to log messages at different levels (info, warn, etc.). Logs can become messy if developers logged information at random logging levels.

Solutions [TODO]:

Contributors: Kenny Tan

PROBLEM: GAE logs are tied to an app version

We would like our logs to cover all past versions.

Implementation: Comments

PROBLEM: Code comments are not effective

Solution: Define guidelines for writing comments.

Status: Some guidelines are contained in DevMan and Java style guide, but may be not enough.

TODO: Define more elaborate guidelines

Implementation: Error handling

PROBLEM: GAE doesn’t support assertions

Assertions are useful to verify programmer assumptions. However, GAE does not support normal java assertions. There used to be a workaround for this that doesn’t seem to work anymore.

Solution: We implemented our own class Assumptions.java to emulate assertions. This class is based on org.junit.Assert class and has methods such as assertEquals, assertNotNull etc. Here’s an example of it being used.

However, unlike normal assertions, we do not have a way to turn these off.

Contributors: Wang Sha

PROBLEM: GAE can throw unchecked exceptions

GAE can throw some unchecked exceptions such as DeadlineExceededException.

Solution: ControllerServlet is the main point at which we catch and handle exceptions. It has a ‘catch all’ block that displays a generic error message for all unexpected errors. [Contributors: Wang Sha]

TODO: Find out more about all unchecked exceptions thrown by GAE and make sure we handle them appropriately

Problem: DeadlineExceededException kills the instance before sending the alert email to admin.

Solution: A cron job checks the GAE log every few minutes and email admin if there are severe errors. [Contributors: Shawn Lee]

PROBLEM: Exception handling can become chaotic if done in an ad hoc way

Solution: TEAMMATES have policies for what kind of exception to throw for which situation. These are documented in the DevMan.

Contributors: Wang Sha

TODO:

Implementation: Data validation

QUESTION: Which characters to be allowed in person names?

Answer: It is very hard to determine what should be allowed in a name, since names can vary widely across the world. We have narrowed it down to a few characters that would have a higher probability of appearing in names.

Acceptable characters: alphanumeric characters, fullstops and commas (Smith, John D.), hyphens, apostrophes (Hal’ Jacob-Smith), rounded brackets (John Smith(Jr)), slashes (Tom S/O Tommy)

Also acceptable are whitespaces and tabs within names.

We decided not to allow all characters in names, because this may result in SQL injection (or a variant of it based on how GAE stores data).

Since V4.55, we allow any character in the name. It is not nice to say someone’s name is ‘invalid’ because it contained a non-alphanumeric character.  [Contributors: Chun Teck]

Since V4.78, we allow non-English characters in names. [Contributors: Theong Siang]

Implementation: Parallelism

GOAL: Make TEAMMATES thread-safe

[TODO]

Possibly useful:

Implementation: Tools and technologies

CHOICE: Application platform

Current choice: Google App Engine

Alternatives: AWS, Azure, Heroku, Engine Yard, Digital Ocean

Rationale:

CHOICE: Programming language

Current choice: Java

Alternatives: GAE supports only Java, Python, and Go. PHP support is in beta.

Rationale: Java was chosen because it is more widely used than Python.

CHOICE: IDE

Current choice: Eclipse

Alternatives: Jetbrains IDEA

Rationale: Eclipse is Free and widely used.        

CHOICE: Web App framework

Current choice: Standard JavaEE with Servlets and JSP

Alternatives: Struts, Clojure, Spring, ...

Rationale: We do not want to bring in extra frameworks unless necessary. Some frameworks may not play well with GAE, especially those using dependency injection.

TODO:

CHOICE: Data encoding

Current choice: JSON

Alternatives: XML

Rationale: more natural and less verbose syntax.

                 

Current choice: GSON

Alternatives:

Rationale:        

[ UI ]_______________________________________________________________

UI: Device compatibility

GOAL: Support popular tablets

Strategy:

Status: Seems to be working fine on iPad. ‘Quick tap’ can produce hover over text.

TODO: Expand to other tablets.

CHALLENGE: Checking tablet experience from a PC

[TODO]

GOAL: Support popular mobile devices

Strategy: Identify which phones to support. Find what needs to be done to support them. Implement those changes and test.

Current status: seems to be working on standard mobile browsers, although the user experience is not optimal.

UI: JavaScript support

PROBLEM: Some browser might have JavaScript disabled

Solution: Although only some of functionality requires JavaScript, we decided it is simpler to block browsers that don’t have JavaScript enabled from accessing TEAMMATES.

Status: We detect if JavaScript is disabled and show how to enable it.

UI: Browser compatibility

QUESTION: Which browsers to support?

Answer: Because TEAMMATES uses HTML5, only HTML5 browsers are supported. Following statistics from several websites, http://www.w3counter.com/globalstats.php, http://gs.statcounter.com, we have decided to support the following browsers.

Internet Explorer

 IE 9 and above. IE 8 does not support HTML5. Although IE8 is still holds a significant percentage for browser usage, we believe that our target audience (students, faculty staff) would very unlikely be using IE8.

Chrome

Recent three Chrome releases.

Firefox

Recent three Firefox releases.

Safari

Safari 4 and above. Safari is the default browser for MAC OS users. While some users may choose to download other browsers, a large percentage of MAC users would use Safari. Safari 4 was released in 2009 and it supports HTML5.

Contributors: Chun Teck

CHALLENGE: Handling unsupported browsers

The first step of handling unsupported browsers would be to first detect the browser in use. The code for this can be found in webapp/js/checkBrowserVersion.js. This code is adapted from http://www.javascripter.net/faq/browsern.htm

Once an unsupported browser has been detected, we have to decide what to do. One alternative is to restrict access to the website until the user upgrades or downloads a supported web browser. However, this would greatly reduce traffic to the website, especially deterring new potential users. A few random tryouts showed that TEAMMATES is still usable in most popular browsers.

Therefore, rather than blocking access from unsupported Browsers, we have decided to just include a warning/message if the user is using an unsupported browser. This message is included in the homepage, such that users would know if they are using an unsupported browser. The warning is posted right at the top of the webpage, so that they do not miss it.

BrowserShots (http://browsershots.org/) is used for testing. {TODO: put in a screenshot}

Contributor: Chun Teck

TASK: Testing for browser compatibility

Strategies:

Status: Tests are runnable for Firefox, Chrome, IE. However, FF is the one used most because it is fastest. There is no procedure for ensuring tests pass on all Browsers.

Contributions: Chun Teck

PROBLEM: UI inconsistencies across different browsers.

Different Browsers can render the same web page slightly differently.  The reason for this is that the default CSS styles of each browser are different. For example, one browser may set the default value for line-height differently from another browser. Another browser may set the default value for the margin differently. This means that while the web-page may look nice and fine in one browser, the style and layout for the same web-page may become skewed in another browser.

e.g. Visual elements (border styles) are rendered differently (source: impressivewebs.com)

Solution: CSS resets

The solution to this problem is to introduce a CSS Reset into the website. The aim of a CSS Reset stylesheet is to remove all the inconsistent default styles that a browser may specify. Once these default styles are removed, the styles for the web application can then be introduced. This ensures that all the styles are the same throughout regardless of which browser is being used to render the web-page. The most widely used CSS Reset stylesheet Eric Meyer’s Reset Stylesheet, which is the same stylesheet that is used in TEAMMATES.

Since we migrated to bootstrap, there is no need for CSS resets anymore.

Contributors: Chun Teck

UI: File structure

PROBLEM: All-in-one style sheets

Previously, the CSS for the entire web application was consolidated into 2 very large stylesheets. Every web-page in the application would include these 2 stylesheets in their HTML document. Because all the styles from the entire web application are in these 2 stylesheets, they are very large and difficult to maintain. As a result, there are several inconsistencies within the 2 stylesheets, with some styles being defined twice, and styles from one stylesheet being overridden from the other stylesheet.

Solution: Split up the stylesheets.

We created one common stylesheet, called common.css, which is used by every web-page in the application. The CSS Reset stylesheet will be part of this common stylesheet. This stylesheet will hold the styles for the general theme of the web application. Next, we created individual stylesheets for each web-page, which will only contain styles that are particular for that specific webpage. In this way, the CSS is more structured, and it is much easier to sort and edit the different styles if required. It is also easy to change the entire theme of the website, since all the styles are consolidated in the common stylesheet.

One downside of this solution is that it increases the number of requests a browser has to make to render a page.

After moving to bootstrap, we are back to using a single custom css file for the entire application.

Contributors: Chun Teck

UI: Look & feel

GOAL: A ‘Google quality’ look-and-feel

The theme, color scheme, fonts etc. should fit together nicely and the whole look and feel should be as good as that of products done by Google, Apple, Microsoft etc.

Strategy: to use the Bootstrap UI framework

UI: Tools and technologies

CHOICE: HTML version

Alternatives: HTML5, XHTML5

Choice: HTML5

Rationale:

Further information can be found in our HTML Style Guide

Contributors: Chun Teck

CHOICE: CSS version

Not yet using CSS3. Refer to our ‘CSS standard’ for more details

CHOICE: CSS framework

Choice: Bootstrap

CHOICE: Client-side scripting technology

Choice: JQuery

[ Storage]_________________________________________________________

Storage: Database model

CHOICE: Storage model

{TODO: add a separate technical paper}

Current choice: JDO

Alternatives:

Rationale:

Storage: Consistency

CHALLENGE: Hide ‘eventual consistency’ from users

[TODO: add technical report written by Shawn]

PROBLEM: Edit operations are not persisted until PersistenceManager is closed.

Edit entity operations do not persist in dev server, unless we close the persistent manager. In a production server, the PersistenceManager gets closed at the end of the request.

Status: We close the PersistenceManager after each edit operation.

[TODO]: Analyze further and figure out whether this is the best way to tackle this situation.

Storage: Transaction control

TASK: Use transaction control for our database operations

We would like to use GAE’s transaction control facilities to our best advantage.

{TODO: add technical paper to elaborate this}

Storage: Schema

CHALLENGE: Implement schema changes over legacy data

Since TM is currently being used by users, any changes to schema needs to be implemented on existing legacy data as well. One example of such a schema change is the one we did when adding support for multiple instructors.

Options:

  1. Use Objectify framework that supports data migration.
  2. Write code to create new entities and copy all properties from original entities.
  1. Batch mode (in a single request)
  2. Use remote API (one operation at a time)
  3. Use MapReduce

Current choice: We used option 2b. Adopting a new framework (i.e option 1) is considered too drastic. Option 2b wasn’t necessary because our dataset was small.

{TODO: add technical paper to elaborate this. Move the references to the technical paper}

References:

[ Testing ]_________________________________________________________

Testing: Time cost

PROBLEM: Page loading delays

When testing a feature deep within the system, the test needs to start from home page and navigate to page being tested.

Solution: Make each page reachable directly.

Status: Most pages are reachable directly. But still needs to login first.

PROBLEM: Cost of setting up test data

[Solution?] Use a separate set of data for UI tests and keep them persistent?

PROBLEM: Checking for emails takes time

PROBLEM: Not possible to run an arbitrary selection of tests

Testing: Money cost

PROBLEM: Test suite consumes free quota too soon

Testing: Non-determinism

PROBLEM: Timeouts

This can be a problem when running tests in parallel and the server times out due to high load.

Current status: No longer a problem. A stress test showed that the GAE can respond to high load pretty quickly.

PROBLEM: Failures due to ‘eventual consistency’ nature of datastore

For example, trying to create an entity immediately after deleting it can cause EntityAlreadyExistsException because the delete operation is taking time to persist.

Current status: Some ‘retry’ mechanisms are in place to minimize this problem. More can be done.

PROBLEM: Random minor variations in page source (browser quirks)

In our current UI tests, we compare the HTML code that we expect to be in a certain page against the actual HTML code received by the browser. This task is complicated by the fact that different browsers spit out the HTML is slightly different ways.

Options:

  1. Do a logical comparison of HTML DOM trees instead of a string match.
  2. Transform the HTML to a standard format before comparing.

Current status: Tried option 1. It worked, but it was difficult to debug when a test failed. Now using option 2, which seems to be working pretty good so far.

Contributors: Chun Teck

Testing: Fragility

PROBLEM: Tests are sensitive to dynamic data

e.g. dates

Partial solution: use wildcard ‘*’ to exclude those parts of the html from the comparison

PROBLEM: Dev server testing is not representative of live system

(i) sending emails

(ii) tests involving other GAE services that are not present on dev server

PROBLEM: Tests can affect normal operations

(i) Risk of live data getting wiped out

PROBLEM: Tests require real Google accounts

This is a problem because all developers have to share the password for test accounts.

Current status: Sharing of accounts is no longer necessary. Developers can use test accounts they created themselves. Still, each developer need 4-5 Google accounts to run the tests against a production server..

Testing: Coverage

TASK: Measure test coverage

For Java: EclEmma.

For JavaScript: JSCoverage.

Contributors: Chua Hong Jing, Cheung Yew Kong Benjamin, Loke Yan Hao

CHALLENGE: Increase test coverage

Current solution: Current test coverage (Jun 2014) is around 80-90%. This is good, but can be improved further.

CHALLENGE: Increase testing frequency

Challenges: The high cost of testing can reduce the test frequency.

Current solutions:

CHALLENGE: Automated testing of a Web UI

Our UI consists of web pages. Automating the testing of such a UI is not as easy as automating API testing.

Solution: Use selenium to automate browser actions.

CHALLENGE: Testing for UI appearance, not just content

 We need to automate comparing how a web page actually looks against how it is expected to look.

Solution: Create a master page (a mashup of all pages) that can be used for an eye ball test. A screenshot is given below.

TODO: Look for ways to automate this.

Contributors: Chun Teck.

                 

CHALLENGE: Testing with dynamic UI elements

AJAX-based UI features, hover over actions etc. makes testing harder. For example, if we move the mouse during a testing session, it can cause a tool tip to pop up, which dynamically modifies the page source (using JavaScript), which makes the test fail because the actual page source is now different from the expected page source.

Solutions:

  1. Minimize AJAX-based UI features.
  2. Find a way to prevent tooltip text interfering with tests. [DONE: tooltip text are now ignored by tests].

TASK: Testing static web pages

Ensure the correctness of the static web site, especially, links.

Solution: [TODO]

Testing: Design

PROBLEM: Browser abstraction class is too big

The class we use to abstract the Browser (currently called BrowswerInstance.java) is getting too big and unwieldy.

Solution: [DONE] Apply the Page Object Pattern.

Bibliography:

Testing: Tools and technologies

CHOICE: Java testing tool

Current choice: TestNG

Alternatives: Arquillian, JUnit

Rationale:

JUnit was our initial choice. We decided to move to TestNG after Milestone V4.53 because

The downside is,

CHOICE: Javascript testing tool

Current choice: QUnit

Alternatives: Jasmin

Rationale:

CHOICE: UI testing tool

Current choice:Selenium

Alternatives:

Rationale: One of the most commonly used library to do UI test

CHOICE: HTML parser

Current choice: NekoHTML

Alternatives:         

Rationale: Already used by Selenium.

Testing: Miscellaneous

PROBLEM: Chrome/IE Web driver processes do not die after the testing

There is an issue regarding the Chrome Driver found here: http://code.google.com/p/selenium/issues/detail?id=3378 

In order to kill the driver, we have to use the WebDriver API, calling the function driver.quit(). However, this only works for the IE driver, and not for the Chrome Driver, as found in the issue mentioned above. Additionally, killing the driver process will also close all browser windows that are linked to the driver. As a result, we would not be able to see the test results at the end of the test if we program the driver to quit after the tests.

Due to the way we write our test suites, we do not have a handle to the driver, and we are unable to get the driver to quit at the end of the test suite.

Solution: We will have to manually kill the chrome driver and IE driver processes by ourselves, using the task manager.  

Contributor: Chun Teck

[ Documentation]_______________________________________________

Documentation

CHOICE: Format

Choices: HTML, GoogleDoc, PowerPoint, Wiki

Current choice:

CHOICE: Document generation

Generating javadoc, dependency diagrams.

Choices: javadoc tool, DoxyGen

Current choice: none

[Usability]______________________________________________________

CHALLENGE: Make the app self-guiding

[ Security]_______________________________________________________

CHALLENGE: Data protection

CHALLENGE: DoS attack protection

CHALLENGE: Injection attack protection

TASK: Prevent ‘abusive overuse’ of functionality

E.g., creating classes with too many students etc.

[ Scalability & Performance ]__________________________________

S&P: Measuring

We have done some experiments to measure TM scalability up to 100K users and found the performance to be reasonable, but can be improved further.

Also see:

S&P: Improving

We have identified some possible scalability/performance bottlenecks and attempted to remove some of them.

Also see:

S&P: Tools and technologies

Possible tools:  AppStat 

[ Administration]_______________________________________________

Administration: monitoring

CHALLENGE: Monitoring activities

The admin needs to know which users are performing what kind of activities at a given time.

Current solution: Have an admin page that reports activities based on log messages.

TODO: Make the activity log more informative and applicable over past versions.

Contributors: Wang Sha, Chun Teck

CHALLENGE:  Monitoring for errors

While critical errors are captured in the application log file, the admin cannot be monitoring the log all the time.

Solution:

CHALLENGE:  Monitoring performance

Administration: System status

TASK: Devise a way to take the system offline

Sometimes, we need to make the application ‘unavailable’ to users to carry out maintenance activities. In such cases, we need a way to put the app in ‘maintenance mode’ easily and restore access easily after the work is done.

Option 1: GAE allows deploying multiple versions of the app. We can deploy one version that redirects all requests to a ‘under maintenance’ page. When we want to put the app in maintenance mode, we simply make that version the default.

Option 2: ?

Status: Option 1 used (see Issue 428).

Todo: Have a weekly maintenance window e.g., do not allow any deadlines in that period.

Contributors: Chun Teck

Administration: Troubleshooting

PROBLEM: Difficult to locate entities

When a user contacts the support email, the system admin needs to locate entities (Students, Courses etc.) quickly.

Solution: We have implemented an entity search facility for system admins. This facility currently uses Google’s text search API. [for more info, see this technical report]

Contributors: Wang Sha

Administration: Managing data

TASK: Backup/restore from within the same app

GAE provides ways to backup and restore data.

Currently, we have configured the system to take weekly backups (see Issues  307  343).

In addition, admin can take a backup anytime. Admin can also restore data from an existing backup anytime.

These backups are put in the blobstore, which has a free quota of 2Gb. The other option is to put the data in the cloud store. Cloud store has a free trial period.

TODO: Find a way to delete older backups automatically. Currently, this has to be done manually.   

TASK: Backup/restore from one app to another

This ability is useful for replicating the data in another app for experimentation purposes. This ability is supposedly provided by GAE (see here). We are yet to try it out.

TODO: Try it out.

TASK: Download/upload between GAE and local disk

We do not have an urgent need for doing this at the moment. One reason for doing this is to save storage space for backups. i.e. backup to local disk instead of blob store or cloud storage. However, it is unlikely to be scalable.

Administration: Project statistics

TASK: Calculate LOC count

Currently calculated by ohloh.net as given here.

TASK: Calculate data entity count

It would be useful to know the current number of users, courses etc.

GAE now provides an API for accessing datastore statistics. In future, we hope to make use of this API to generate required statistics.

[ Process]________________________________________________________

Process: workflow

References:

CHALLENGE: Define a suitable workflow

The workflow should suit these characteristics of our project.

Status: Current workflow is defined in DevMan

TODO: Study the workflows of established OSS projects and adopt practices applicable to our project.

GOAL: Make the project set up easier

Solutions:

TASK: Define a suitable repository architecture and branching model

Current choice:

See devman for more details on the current repository architecture. Here are the salient features.

Alternatives:

Process: Community management

PROBLEM: Not enough manpower to guide many developers

Solutions:

TODO: Study how bigger OSS projects manage large developer communities and adopt practices applicable to our project.

PROBLEM: Novice contributors could mess up the repo structure.

Solution: Contributors do not have permission to push to the main repo. Only reviewed and tested code is pushed to the main repo.

Process: Tools

CHOICE: Project hosting site

Current choice: Previously GoogleCode, now GitHub

Alternatives: BitBucket, SourceForge, GoogleCode

Pros: Free and widely-used.

Cons: The issue tracker is not as powerful as GoogleCode

CHOICE: Revision control tool

Current choice: previously Hg, now Git

Alternatives: SVN, Hg

Pros: Widely used and powerful.

Cons : Steep learning curve.

CHOICE: Build tool        

Alternatives:

Current choice: No special tools. Just use Eclipse to build and deploy.

Rationale: We experimented with Maven for a while. We found it to be too restrictive and an unnecessary complication. GAE plugin for Eclipse seems to be adequate for our purposes at this point.

Future work: Study more build/integration tools such as CruiseControl

Contributors: James Dam Tuan Long

CHOICE: Code review tool

Current choice: GitHub (previously: GoogleCode, Rietveld).

Contributors: James Dam Tuan Long, Steve Ng, Jie Sheng

[ Marketing]_____________________________________________________

Marketing

TASK: Increase visibility to search engines (Search Engine Optimization)

GAE apps are not indexed by Google?

Create a blog?

Upload video to YouTube?

Tools: Google Webmaster Tools

{TODO: add technical report}

Contributors: Chun Teck

TASK: Utilize direct marketing

Advertising

Contacting educators?

Post in education related forums?

TASK: Utilize indirect marketing

Enter competitions?

Get reviewed?

GOAL: Create a high brand value

[ Holding bay ]___________________________________________________

Area for holding items for future processing. These will be absorbed into the document at a later time.

Discarded

--- end of document ---