Project Acronym: Total ReCal

Version: 0.1

Contact: Joss Winn jwinn@lincoln.ac.uk

Date: 28-07-2010

JISC Project Plan

Overview of Project

1. Background

Building on a university-wide initiative to improve collaborative, undergraduate research, this student-driven project will discuss, document and develop API plugins for a number of common corporate applications in the HE sector. The plugins will expose space-time data in an open, standardised format that can then be queried and aggregated by a student-centred calendaring service, which will also be developed during the course of the project.

The work undertaken by the project will improve the student experience by providing end-users with a cutting-edge, centrally supported calendaring service driven by existing aggregate services at the University of Lincoln. The plugins, full documentation and further libraries and code examples for the service will be offered to the JISC community for use by their own institutions.

2. Aims and Objectives

The Total ReCal project is part of the Online Services Team’s ‘Labs’ environment[1], where new ideas around web services are prototyped. Fundamental to all Labs work, is the exposure of data via APIs[2], allowing the querying and aggregation of data by innovative new services that are both planned and yet to be planned. This is intended to overcome a problem at the university: many of our existing systems exist as discrete silos of data and functionality, lacking the necessary interfaces for them to be incorporated into an SOA. This project, in providing a service wrapper to some of these systems, will offer a valuable opportunity to test the effectiveness and assess the challenges of this approach to leveraging our legacy systems into our future architecture.

3. Overall Approach

A specific problem that the university faces is the aggregation, integration and publishing of 'space-time data'; that is, data relating to the use of space (i.e. room bookings, geo-spatial location data) and time (i.e. timetables, event schedules, library book returns).  

This project will address this problem by developing plugins for existing university systems that expose useful data which can then be aggregated into new web-based services.

The proposed plugins will provide APIs for Blackboard, WordPress, Google Apps and SirsiDynix Horizon (HIP library portal) and we will endeavour to include a plugin for SharePoint during the project time, too.

These plugins will provide data (RSS, JSON, XML) over a REST interface relating to coursework hand-in dates/times (Blackboard), blogs by location, faculty and date/time (WordPress), calendar feeds for timetables and events (Google Calendar), and library related information such as book return dates (Horizon Information Portal).

We will then develop a web-based calendaring service that aggregates and integrates this data into a single end-user application. The development of this service will be informed by gathering 'user stories' and iterative feedback.

Having undertaken some preliminary research, we also intend to use a NoSQL database (i.e. MongoDB), as the underlying database for the service. As part of the project, we will undertake an appraisal of NoSQL databases over relational databases for use by web services in HE.

Each of these aspects of the project will be fully documented, both in development via the project blog and formally at the end of the project. Plugin code will be available via a public version controlled source code repository. Code examples and libraries for the calendaring service will also be discussed and made available via the blog, wiki and code repository.

4. Project Outputs

  1. Documented User Stories for a university-wide calendaring service
  2. Evaluation of MongoDB and other NoSQL databases for use in HE web services.
  3. Engagement with HE developer community through regular blog posts on all aspects of development (i.e. continual evaluation)
  4. Formal documentation for all code released
  5. API plugins for Blackboard, SharePoint, Horizon (HIP), WordPress, Google Apps
  6. Code examples from the aggregated calendaring service, including an iCal aggregation library, MongoDB library and integration examples for OAuth, ADFS, LDAP and SAML authentication protocols.
  7. Regular reflections on web service development in HE from student points of view.

5. Project Outcomes

  1. Experimentation and practical and rapid integration of disparate institutional administrative and related academic systems
  2. Efficiency savings through the extension of existing systems
  3. Enhancing institutional agility through the development of APIs and standardising of data for immediate and future use
  4. Improving the student and staff experience through the opening up and aggregation of disparate data
  5. Improving access to and exploitation of corporate and student data
  6. Identification and discussion of new ideas and technologies and investigate best practice i.e. through the use of a NoSQL database and OAuth single-sign on integration.

6. Stakeholder Analysis

Stakeholder

Interest / stake

Importance

Students/staff

Cutting-edge, user-driven, calendaring web service, which aggregates private and public space-time data into a personal, accessible application.

High

University of Lincoln

The development of an aggregated calendaring service through community-wide engagement. A documented example of 'student as producer' cross-departmental collaborative Research & Development. A 'refresh' of existing ICT services (Blackboard, HIP, SharePoint) through plugin extensions. Improved student experience, through a student-led and student-driven project.

High

Other HEIs

Documented and well maintained plugins which extend and innovate on applications in use across the sector. Sharing code and experience provides opportunities for efficiency savings as the useful lifetime of existing applications is extended.

Medium

Other developers

Re-usable, documented code which provides 'drop-in' APIs for popular software used by HEIs. Discussion around NoSQL databases, the use of OAuth in education, the use of Google Apps for Education. Code libraries for iCal, MongoDB and various authentication methods.

Medium

JISC

Value for money. Project Outcomes (above) are realised. A public-facing calendaring application is a showcase for JISC and helps drive innovation within the sector.

High

7. Risk Analysis

Nick and/or Alex are unable to work due to illness, etc. This risk is somewhat mitigated by having two developers working closely together on the project and developing from the same code base. Furthermore, their code will be well documented so that other staff developers should be able to contribute to the project if required. Probability=1 Severity=3  Score=3

Project objectives are too ambitious. We have intentionally avoided a commitment to delivering a SharePoint plugin for this reason and the development of discreet plugins mitigates this risk further. The method of development proposed for the project follows current practice and, based on experience, we are quite confident that we can meet the project deliverables. Probability=2 Severity=2 Score=4

Lack of Stakeholder engagement. Engagement with students and staff at the university is essential and peer-review from the JISC community is especially welcome to ensure that the project delivers useful and relevant outputs for re-use by other institutions. We will continue to build on an already established reputation for JISC community engagement and work with internal communications colleagues to ensure that our project is given a high profile. Through iterative releases and regular blogging, users should feel engaged and invested in the project as much as the project team. Probability=2 Severity=2 Score=4

8. Standards

Name of standard or specification

Version

Notes

HTML

5

Chosen over XHTML 1.0 / 1.1 for its improved feature range, its existing use in the CWD[3] and its backwards-compatibility. It also supports much better semantic markup than previous versions, so we can embed meaning directly within any web pages with a minimum of additional markup. HTML pages are used to provide a browser-friendly view to data.

XML

Data from the API can be requested in an XML format.

HTTP

The de-facto standard for requesting and delivering HTML data (as in all our web pages), making API calls and delivering the results of those calls. Using HTTP/1.1 as the nominal standard, although HTTP/1.0 is also supported by the servers where requested.

CSS

3

We're using CSS since it allows us to cleanly separate content from presentation (particularly relevant given our planned seamless support of mobile devices, and our drastically improved CWD web accessibility policy). CSS3 in specific is purely used to add additional visual polish in some cases, as it cleanly degrades on unsupported browsers.

UTF-8

All data exchange is carried out in the UTF-8 character set, since this is the generally supported standard for exchanging non-ASCII characters such as accented characters which may appear in event names. UTF-8 support is mandated by IETF for all internet protocols (in this case HTTP/1.1)

JSON

Data from the API can be requested in the JSON format.

BSON

Data is stored internally within MongoDB as BSON, which is simply a binary serialisation of a JSON object.

CSV

Data from the API can be requested in the CSV format.

iCalendar

Data from the API can be requested in the iCalendar format. In addition, Total ReCal will accept iCalendar events as inputs to the parsing engine (for example allowing external calendars to be accepted). iCalendar is the de-facto standard for exchange of calendar information, and is supported by clients including Google Calendar, Apple's iCal, Microsoft Outlook and many more (including mobile devices). Furthermore, it is a basic part of the CalDAV standard which, although not planned for inclusion in Total ReCal, will leave the avenue open for further development in the future.

9. Technical Development

On the whole, an Agile approach[4] will be taken but using Cowboy Coding[5] during the rapid prototyping phases and then re-architecting code in line with proper Agile practices once we're settled on specifications.

10. Intellectual Property Rights

2.5.1 Following the licensing of the CodeIgniter framework[6] and MongoDB[7], all code will be made available under an Apache style or AGPL v3.0 license. All documentation will be licensed under a CC-BY license. We will consult with OSSWatch[8] on the most appropriate open source license to use.

Project Resources

11. Project Partners

There are no formal external project partners, although links with other universities will be developed over the course of the project.

12. Project Management

The project will be reflexive and open with regular discussion of our work via the project blog and Twitter. Agile development methods such as pair programming, iterative releases every two-three weeks, and daily 'stand up' meetings are already established within the Labs environment of the Online Services team. Alex and Nick are trusted to meet user requirements with minimal oversight from their Manager.

A group of users, including both students and staff will be formed to gather stories/requirements through face-to-face meetings and a UserVoice site.[9]

Bit Bucket will be used as a source code repository, wiki and issue tracker.[10] The development of the project will be regularly discussed with ICT colleagues and to the Student Experience Committee.  A mailing list will be set up for other developers to discuss issues raised by the project in detail.

12.1 Team members

Alex Bilbie (Developer): Alex is a second year computing student and web developer. He currently works part-time for the ICT department and is working on a number of Labs projects which form the basis for this proposal. Previously, he worked on the JISC-funded JISCPress Rapid Innovation project.

Nick Jackson (Developer): Nick is a recently graduated third year computing student, pending post-graduate student and web developer. He currently works part-time for the ICT department and is working alongside Alex on a number of Labs projects.

Tim Simmonds (Online Service Manager): Tim has worked for the university for over 20 years and is in charge of all online services managed by the ICT department. He also manages Alex and Nick's work for ICT and will ensure that work on the project is appropriately represented at relevant meetings and co-ordinate the assistance of other ICT colleagues when integrating the project deliverables into university-wide services.

Paul Stainthorp (Electronic Resources Librarian): Paul works in the Library, where he manages all of the University's electronic library resources and systems, including the Lincoln open-access repository (EPrints). He is a regular contributor to the 'Mashed Library' series of events. He has a background in multimedia production, lecturing and library systems development. He will contribute to the work on HIP data aggregation.

Joss Winn (Project Manager): Joss is Technology Officer in the Centre for Educational Research and Development. He previously managed JISCPress, a JISC-funded Rapid Innovation project and ChemistryFM, a JISC/HEA-funded OER project, and was project officer on the JISC-funded LIROLEM repository project. He has several years experience in managing software development projects as well as having LPI Linux System Administration Certification.

13. Programme Support

We have contacted OSSWatch for advice on IPR and licensing.

14. Budget

See separate project proposal document

Detailed Project Planning

15. Work Plan

The work packages below list the main areas of work over the six months of the project. Within these work packages are a number of objectives, which are detailed in this project plan. Alex and Nick will work 15hrs/week each on this project, although this time is not ring-fenced as their work on TotalReCal overlaps with other work they are currently doing in their team. Source code for the project will be publicly hosted and stable releases announced on the project blog. Although development is not strictly tied to any one methodology, at all times it will be iterative with monthly (or more) point releases of code. The project will be run entirely openly, with reflections, announcements, code and documentation, all being published through public channels.

Following the initiation of the project, work will be centred around plugin development and database evaluation. A number of blog posts will be written reflecting on the choice of NoSQL[11] databases. The gathering of user stories[12] will occur informally and formally throughout the project. The greater involvement of students will occur from the beginning of the academic year (Sept/Oct). Documentation will be written throughout the project as required. Stakeholders/community engagement will likewise take place throughout the project, as outlined in the dissemination plan below. It is anticipated that work on the central integrated calendaring service will begin in September, around the time that student engagement will begin and this will help feed into the requirements for the service.

Work will continue iteratively until completion in January, when all code, code examples, documentation and user stories will be completed and an accurate and useful record of the project outcomes recorded.

Workpackages

July

August

September

October

November

Dec/Jan

1. Project initiation

X

2. Gather user stories & feedback

X

X

X

X

X

X

3. Database evaluation

X

X

4. Plugin development

X

X

X

X

5. Integrated service development

X

X

X

X

6. Documentation

X

X

X

X

X

X

7. Community engagement

X

X

X

X

X

X

8. Evaluation and project close

X

16. Evaluation Plan

A number of milestones are shown in our workplan, which act as marker points for evaluation. We will also solicit, both formally and informally, feedback from student and staff user groups. Code will be available at all times in a public source code repository and we intend to make contact with other web services teams in the programme and sector for peer review.

Documentation is a key output from this project and this will be openly drafted on the project wiki and available for review at any time. Both documentation and code will be points of focus during project team meetings during the course of the project.

We will solicit comment from all stakeholders during the project, addressing questions around usability, ease of re-use and sanity of code, as well as ensuring the wider objectives of improving efficiency and longevity of existing systems and improving the staff and student experience.

In addition, formal surveys and actively soliciting comment via the project blog, will contribute to the overall evaluation and review of the project.

17. Quality Plan

Documented User Stories for a university-wide calendaring service

‘User Stories’ will be collected from all Stakeholder groups, during face-to-face meetings and through the use of online tools such as surveys, project blog, Twitter and UserVoice. These will be collated, analysed and reported on via our blog.

Evaluation of MongoDB and other NoSQL databases for use in HE web services.

A blog post will be written that discusses our research around NoSQL databases and details the reasons for choosing this type of database. The post will reflect not only on the advantages (and disadvantages) of Lincoln adopting the use of NoSQL databases, but also the wider community.

Engagement with HE developer community through regular blog posts on all aspects of development (i.e. continual evaluation)

A series of blog posts are outlined in the workplan. This is a minimal set and we anticipate many more as we reflect on the project and update our Stakeholders on the progress of the project.

Formal documentation for all code released

All code will be documented appropriately as comments in the sourcecode and on our wiki. This will be publicly accessible for review throughout the course of the project.

API plugins for Blackboard, SharePoint, Horizon (HIP), WordPress, Google Apps

A number of checks can be made concerning the quality of the plugin code. Does it run without logging errors? Does it meet user requirements? Is it well documented? Does it provide the interoperability required for the calendaring web service? Is the code efficient and well optimised? Quality assurance checks that address these questions are likely to include face-to-face meetings with Stakeholders as well automated unit tests.

Code examples from the aggregated calendaring service, including an iCal aggregation library, MongoDB library and integration examples for OAuth, ADFS, LDAP and SAML authentication protocols.

As with the plugin code, a number of checks can be made to the quality of the code through the use of a source code repository and integrated issue tracker.

Regular reflections on web service development in HE from student points of view.

Both developers on this project are students and can therefore bring the ‘student voice’ to the project and encourage students in their own networks to contribute their views, too.

18. Dissemination Plan

Every aspect of the project will take place in public channels allowing anyone to watch the project evolve and participate through testing and feedback. Stakeholders such as JISC Programme Managers and HEI staff will be actively encouraged to offer feedback and participate in testing the project iteratively.

General feedback will be solicited throughout the project via the project blog and Twitter. User stories and feedback will be gathered early in workpackage two and then iteratively throughout the project in workpackage four. Public scrutiny of the code will be possible at any time via the public repository. Workpackage seven will document the outcomes of the project on the blog and wiki. In the JISCPress project, we demonstrated very active engagement by working in this way.

In addition to this, we will arrange for a press release to be issued at the start and end of the project, highlighting the objectives and achievements of the project respectively.

Early in the project, we will endeavour to contact other institutions inside and outside the Flexible Service Delivery Programme, which might be willing to help test the plugins against their own services.

At relevant points throughout the six months, Alex and Nick will disseminate information about the project across their social networks to solicit the views of other students on the progress of the project.

Mid-way during the project, an article will be written for the Staff Magazine, to inform all staff at the university of Lincoln about the project and seek their feedback.

At the end of the project, a workshop will be arranged for university staff in Lincoln and elsewhere, who wish to learn more about the project outputs and how they might affect or be used in their own work.

At least one member of the project team will attend the funding programme’s meetings and conferences and share the work of the project.

At the end of the project, a video will be made where the project team discuss their work and the relevance of it to the wider HE community.

19. Exit and Sustainability Plans

Our proposal is to develop and document a number of open source API plugins for use with popular ICT applications used within HE and elsewhere. In addition, we propose to work closely with students and staff so as to deliver a calendaring service that meets user needs. It is intended that this application will be implemented for all staff and students at the university in early 2011, thereby ensuring its sustainability within the University of Lincoln.

Through the development of plugins, this project will also provide a cost-effective way to extend the use, and therefore life, of existing applications. By releasing our work to the public, other institutions can similarly benefit through the addition of APIs to otherwise often closed or poorly understood proprietary products.

We will provide a number of ways to engage with the HE community throughout the project (blog, Twitter, UserVoice, mailing list) so as to ensure that our work is widely known and understood, encouraging other developers to use and contribute to future releases of the code.

Similarly, our work uses a popular development framework (CodeIgniter) and database layer (MongoDB) and we will ensure that developers in these communities are aware of our work and how it might benefit them.


[1]http://lost.lincoln.ac.uk

[2]http://nucleus.blogs.lincoln.ac.uk

[3]http://thecwd.blogs.lincoln.ac.uk

[4]https://secure.wikimedia.org/wikipedia/en/wiki/Agile_software_development

[5]https://secure.wikimedia.org/wikipedia/en/wiki/Agile_software_development#Contrasted_with_.22cowboy_coding.22

[6]http://codeigniter.com/

[7]http://www.mongodb.org/

[8]http://www.oss-watch.ac.uk/

[9]http://totalrecal.uservoice.com/

[10]http://bitbucket.org/alexbilbie/totalrecal/

[11]https://secure.wikimedia.org/wikipedia/en/wiki/NoSQL

[12]https://secure.wikimedia.org/wikipedia/en/wiki/User_story