Creating Offline Web Applications With Dojo Offline

by Brad Neuberg (SitePen), September 23rd, 2007

This tutorial steps you through creating offline web applications using Dojo Offline.

What is Dojo Offline?

Dojo Offline is an open-source toolkit that makes it easy to create sophisticated, offline web applications. It sits on top of Google Gears, a plugin from Google that helps extend web browsers with new functionality. Dojo Offline makes working with Google Gears easier; extends it with important functionality; creates a higher-level API than Google Gears provides; and exposes developer productivity features. In particular, Dojo Offline provides the following functionality:


Dojo Offline is built to work with the 0.9 release of Dojo, and will not work with older versions of Dojo, such as 0.4. It also requires the Google Gears plugin to function; if users do not have it installed Dojo Offline will prompt users to download it.

Using Dojo Offline

Let's dive in and start using Dojo Offline.

Download Dojo Offline SDK

First download the Dojo SDK and unzip it.

SDK Layout

When you unzip the SDK, you will find the following directories:



Dojo Offline is an optional Dojo extension, and is therefore located in the dojox directory.


If you are looking to track down Dojo Offline's source code, most of it is in dojox/off/. The Dojo SQL layer is in dojox/_sql/, while Dojo Storage is in dojox/storage/. An autogenerated, JavaDoc-like API is available. When looking at the API docs or source code, many advanced options are available to deeply customize Dojo Offline; you can almost always safely ignore these unless you are an advanced user, and they usually say "For advanced usage; most developers can ignore this" in their documentation.

Demos

Dojo Offline has three main demos, a Hello World example, a more complicated web-based editor named Moxie that includes an example server-side written in Java, and a demo of Dojo Offline's SQL cryptography. You can play with hosted versions of the Hello World example here; a hosted version of the Moxie editor here; and the SQL cryptography demo here. 


If you want to study the demo examples' source code, the Hello World example is located in dojox/off/demos/helloworld/, while the Moxie editor is located in dojox/off/demos/editor/. You can see the SQL cryptography demo source in dojox/_sql/demos/customers/customers.html.


The Hello World example has no server-side requirement; Moxie, however, includes a full Java server-side that you can use as a template and scaffolding. Running the server-side of Moxie is simple. Make sure you have Java 1.5+ installed, and then just type:

java -jar editor.jar

while inside the directory dojox/off/demos/editor/server/, and the Moxie server-side will start running, with an embedded web-server (Jetty) and relational database (Derby) already set up for you.


In a web-browser, you can now go to the following URL:

http://localhost:8000/dojox/off/demos/editor/editor.html

to run Moxie against the local server you just started.


For more details on the server-side portion of Moxie and how to build see the README file at dojox/off/demos/editor/server/README.

Creating an Offline Application

Now that you have awareness of the SDK, it's file layout, and the provided demos, let's get down to illustrating what you need to do to create an offline-aware application using Dojo Offline.

Bring in Dojo and Dojo Offline

For our example source code we will pretend that you are creating your application with the Dojo Offline SDK in a subdirectory named offline-sdk.

First, bring in Dojo and Dojo Offline:
<script type="text/javascript" src="offline-sdk/dojo/dojo.js" djConfig="isDebug: true"></script>
<script type="text/javascript" src="offline-sdk/dojox/off/offline.js"></script>
isDebug is a useful flag that when set to true will print out debug messages, and when set to false will hide them. In your own code you can add console.debug("some message"); to have these printed out to help with debugging. If you are using Firefox in conjunction with Firebug then these messages will print out to the Firebug console; otherwise they will print on to the web page itself, such as in Internet Explorer.

Notice that we do not bring in Dijit; Dojo Offline has no dependencies on Dijit, the Dojo Widget system, helping to keep your code size smaller. We only include it in the SDK because Moxie, the example editor, uses the Dijit rich text editor.

Next, bring in Dojo and Dojo Offline's style sheets:
<style type="text/css">
	@import "offline-sdk/dojo/resources/dojo.css";
			
	/* Bring in the CSS for the default Dojo Offline UI widget */
	@import "offline-sdk/dojox/off/resources/offline-widget.css";
</style>

Dojo Offline Widget

Dojo Offline includes a default offline widget that does much of the hard work of providing a good offline UI to your end-users. It updates the user with online and offline feedback; provides sync messages; and delivers help and instructions on using your application offline. If Dojo Offline did not provide these you would have to roll them yourself, so providing a default UI makes developing offline applications easier.

Getting this widget is as easy as adding a bit of HTML to your page, with a special, predefined ID:
<!-- 
	Place the Dojo Offline widget here; it will be automagically
	filled into any element with ID "dot-widget".
-->
<div id="dot-widget"></div>
When the page loads, Dojo Offline will automatically find an element with the ID dot-widget and automagically embed the offline widget.

The offline widget is a small, self-contained unit that provides feedback to the user, giving you a bunch of great functionality for free.

Here is a screenshot of Moxie, with the offline widget circled:

For more screenshots of the offline widget and a full description of what it gives you for free, see here. It is recommended that you use the offline widget in your own offline applications unless you are an advanced developer; information on customizing its look and feel, including dropping it, can be found here.

JavaScript

Let's take a look at our JavaScript now. All you have to do is set your application name:
// set our application name
dojox.off.ui.appName = "Example Application";

The Offline Widget will use your application name to customize it's user-interface; that is how the offline widget, for example, can insert "Learn How to Use Hello World Offline" into its instructions without you having to manually edit the offline widget's HTML.

Slurp! Bringing Your Application's Files Offline

For your application to be able to work offline, you must make sure that all of your JavaScript, HTML, CSS, etc. are available even when away from the network. While the browser may have these in its cache, you can't depend on this. The underlying Google Gears plugin that Dojo Offline uses makes it possible to specify what files you want to have available when offline, such as your JavaScript.

It can be tedious, however, to build up a full list of all your files, especially if you have many images and supporting libraries and are doing rapid development. Dojo Offline provides a single method, named slurp(), that will slurp the page and automatically figure out all of the resources you need to have available offline:
// automatically "slurp" the page and
// capture the resources we need offline
dojox.off.files.slurp();
The slurp() method is awesome; it will automatically scan your page and quickly figure out all your JavaScript and CSS files; grab any IMG tags in your source; and even grab any background URLs you might have in your attached CSS. The only thing it doesn't do is look at inline styles or scan your JavaScript for dynamic additions.

You can also manually add files if you want or need to; more information here.

slurp() has a nice debug method if you want to see all the files that have been slurped; make sure that Dojo's isDebug is true, and call dojox.off.files.printURLs() after the page and Dojo Offline are finished loading (more details on how to know when Dojo Offline is done loading in the next section).

During development, as you hit your offline application, it will always be pulled from the Google Gears file cache first, even if you are online. This can sometimes make development tricky and tedious. If you just made a change locally, hitting your web application, even if you are offline, will cause the older version to be pulled in first.

To make this process easier, I have created a bookmarklet that you can drag to your links toolbar in your browser. View this page to get the bookmarklets for Firefox and Internet Explorer.

These bookmarklets clear the Google Gears cache, removing all of the files that slurp() added offline. Press it during development when you have made a new change that you want to show up; you must press it when you are at your web application (i.e. it won't work if you just press it when you are at a different or blank page). Then, refresh the page to see your change.

Knowing When Dojo Offline Can Be Used

Dojo Offline can't be used until the page is finished loading and Dojo Offline itself is finished initializing (such as the offline widget finishing being placed into the page). You should wait until the page and Dojo Offline are finished loading, and then your application can start doing its own custom thing:
var myApp = {
	initialize: function(){
		alert("Dojo Offline and the page are finished loading!");
	}
}

// Wait until Dojo Offline is ready
// before we initialize ourselves. When this gets called the page
// is also finished loading.
dojo.connect(dojox.off.ui, "onLoad", myApp, "initialize");
			
// tell Dojo Offline we are ready for it to initialize itself now
// that we have finished configuring it for our application
dojox.off.initialize();
myApp is some object that will hold all of the methods for our application; myApp.initialize() is the method in particular that would initialize our application on page and Dojo Offline load in some way. We use dojo.connect to connect to the dojox.off.ui.onLoad event; when this fires, myApp.initialize() is called. At this point we could begin to manipulate the DOM on the page, since it is loaded, or do further actions specific to your application; in this case we just print an alert message.

The final call to dojox.off.initialize() is there to tell Dojo Offline that we are done configuring it, and that it can now initialize itself; we don't want Dojo Offline to initialize itself before we have set our appName and called slurp(), for example, so this final call tells Dojo Offline that we are finished doing configuration.

Let's look at all of our code so far in this tutorial:
<html>
	<head>

		<script type="text/javascript" src="offline-sdk/dojo/dojo.js" djConfig="isDebug: true"></script>
		<script type="text/javascript" src="offline-sdk/dojox/off/offline.js"></script>

		<style type="text/css">
			@import "offline-sdk/dojo/resources/dojo.css";
			
			/* Bring in the CSS for the default 
			   Dojo Offline UI widget */
			@import "offline-sdk/dojox/off/resources/offline-widget.css";
		</style>

		<script>
			// set our application name
			dojox.off.ui.appName = "Example Application";

			// automatically "slurp" the page and
			// capture the resources we need offline
			dojox.off.files.slurp();

			var myApp = {
				initialize: function(){
					alert("Dojo Offline and the page are finished loading!");
				}
			}

			// Wait until Dojo Offline is ready
			// before we initialize ourselves. When this gets called the page
			// is also finished loading.
			dojo.connect(dojox.off.ui, "onLoad", myApp, "initialize");
			
			// tell Dojo Offline we are ready for it to initialize itself now
			// that we have finished configuring it for our application
			dojox.off.initialize();
		</script>

	</head>
	
	<body>

		<!-- 
			Place the Dojo Offline widget here; it will be automagically
			filled into any element with ID "dot-widget".
		-->
		<div id="dot-widget"></div>	
		
	</body>
</html>

Important: I've noticed over and over in the frameworks I have created, such as Dojo Storage and the Really Simple History library, that developers get confused about an important point. Notice that configuring Dojo Offline must be done before the page loads: 

<script>
	// set our application name
	dojox.off.ui.appName = "Example Application";

	// automatically "slurp" the page and
	// capture the resources we need offline
	dojox.off.files.slurp();

	// ...

	// Wait until Dojo Offline is ready
	// before we initialize ourselves. When this gets called the page
	// is also finished loading.
	dojo.connect(dojox.off.ui, "onLoad", myApp, "initialize");
	
	// tell Dojo Offline we are ready for it to initialize itself now
	// that we have finished configuring it for our application
	dojox.off.initialize();
</script>

Notice that our calls to things like dojox.off.ui.appName and the dojo.connect to dojox.off.ui are at the top-level of the script tag done before the page loads; if you were to put these into a function and call that after the page has loaded, Dojo Offline will not work: 

<script>
// this is wrong -- don't do this because it will not work
window.onload = function(){
	// set our application name
	dojox.off.ui.appName = "Example Application";

	// automatically "slurp" the page and
	// capture the resources we need offline
	dojox.off.files.slurp();

	// etc.
}
</script>

At this point we have the shell of an offline application using Dojo Offline and Google Gears. Let's delve into common issues that come up with offline applications now, such as toggling your user-interface in different ways if you are on- or off-line; knowing the network status; syncing; and more.

Network Status

In the background Dojo Offline is checking to make sure that your web application is available on the network (for technical details on what it is doing see here). If your web application disappears or appears within a few seconds (five to thirty seconds), Dojo Offline will detect this and automatically inform your application so that you can move on- or offline, such as enabling or disabling UI elements that might not be available offline. Dojo Offline fires an event when the network status changes, which you can easily subscribe to using Dojo Connect: 

dojo.connect(dojox.off, "onNetwork", function(status){
   if(status == "online"){
      alert("We are online!");
   }else if(status == "offline"){
      alert("We are offline!");
   }
});
By default, the Dojo Offline UI widget also subscribes to these events, handling most of the work of informing the user when they are on- or off-line so you don't have to:


As you will see in more detail when we get to syncing later on in this tutorial, Dojo Offline also does an automatic sync when the network reappears:
Another useful property is dojox.off.isOnline. At any time in your code you can check dojox.off.isOnline to see if you are on- or off-line and change your applications behavior appropriately. This will be true if we are online and the application is available on the network, and false otherwise.

A common pattern in offline applications is the need to have a UI element do different things when you are on- or off-line. For example, you could imagine a SEARCH button that when you are online will shoot off a network request to the server to do a search, while when offline will do a SQL search on the local Google Gears datastore. A common way to handle this is to use Dojo Connect and the dojox.off.isOnline property:
var searchButton = dojo.byId("searchButton");

dojox.connect(searchButton, "onclick", function(evt){
   if(dojox.off.isOnline){ searchOnline(); }
   else{ searchOffline(); }
});
This can be a nice pattern to separate on- and off-line code and easily toggle between them without littering lower level code with dojox.off.isOnline network checks; just do it at the dojo.connect level instead, which is cleaner.

Storing Offline Data

An offline application is useless if it does not have access to data. Dojo Offline provides two ways to store data, depending on your needs. The first is Dojo Storage, which provides a simple, persistent hash table abstraction; and Dojo SQL, which provides SQL based access to your data.

Dojo Storage

Dojo Storage contains a very simple hash table abstraction, providing methods such as put() and get(), which you can use to quickly store data without having to delve into SQL. Under the covers it saves this data into the Google Gears persistent storage system.


Saving data is easy. You can store simple strings:


dojox.storage.put("someKey", "someValue");


or more complicated JavaScript objects:


var car = {type: "Nissan", color: "white", price: 20000, optional: "air-conditioner, stereo"};
dojox.storage.put("complexKey", car);

JavaScript objects are automatically saved as they are. Note, however, that browser objects will not get serialized, such as references to the DOM or an XMLHttpRequest object.

Also note that key names can only be letters, numbers, and the under score character. Spaces are not allowed.

Loading data is just as easy; if you stored a JavaScript object, it will be returned to you as a JavaScript object: 

  var value = dojox.storage.get("someKey");

  var car = dojox.storage.get("complexKey");

If the object is not found, null is returned.


Further info and advanced usage on Dojo Storage can be found here.

Dojo SQL

Dojo Storage can be great, but some times you need the full power of a relational data store. Enter Dojo SQL.


Dojo SQL is an easy to use, thin layer over Google Gear's relational storage layer. Executing SQL is easy: 

var results = dojox.sql("SELECT * FROM DOCUMENTS");

Results are returned as ordinary JavaScript objects, unlike Google Gears, which makes working with SQL much easier. For example, if you have created a table named DOCUMENTS: 

dojox.sql("CREATE TABLE IF NOT EXISTS DOCUMENTS ("
			+ "fileName		TEXT NOT NULL PRIMARY KEY UNIQUE, "
			+ "content		TEXT NOT NULL) ");
that has five rows in it (i.e. five documents), and you call dojox.sql("SELECT * FROM DOCUMENTS"), an array will be returned that has five rows, one for each document. Dojo SQL creates an object for each row, automatically taking each of the column names, such as fileName and content, and using those as the object literals:
var results = dojox.sql("SELECT * FROM DOCUMENTS");

// document 1
alert("The first documents file name is " + results[0].fileName + " and it's content is " + results[0].content);

// document 2
alert("The second documents file name is " + results[1].fileName + " and it's content is " + results[1].content);
Inserting data is just as easy:
dojox.sql("INSERT INTO DOCUMENTS (fileName, content) VALUES (?, ?)", fileName,
contents);
Simply put a question mark for each parameter in the SQL, and then provide the actual variables that will fill them in as variable length arguments at the end. You can use this for any SQL statement where you want to provide a parameter:

dojox.sql("SELECT * FROM DOCUMENTS WHERE fileName = ?", someFileName);


For a full list of SQL statements that can be executed by SQLite, see here. The SQL to do full-text indexing can be found here.

Dojo SQL automatically handles opening and closing the database; for advanced usage information on how to manually do this yourself, which you normally shouldn't need to do, see here.

Encrypting Offline Data

A common issue with offline web applications is that they must download data to be stored on the local machine. If the machine is a laptop, sensitive data could be stored that would be at risk if the laptop is stolen. For example, imagine you are dealing with an application that is downloading student records with social security numbers; you don't want to have these stored in the clear if the laptop is stolen.

To address this use-case, Dojo SQL includes a cool feature that makes it easy to encrypt and decrypt specific columns of data. For example, imagine we have a CUSTOMERS table with three columns, a last name, a first name, and a social security number:
dojox.sql("CREATE TABLE CUSTOMERS ("
					+ "last_name TEXT, "
					+ "first_name TEXT, "
					+ "social_security TEXT"
				+ ")"
	);
For the first and last names, we don't care if they are stored in the clear. However, for the social security column we would like to encrypt it.

Dojo SQL adds two new SQL keywords that make this easy, ENCRYPT() and DECRYPT(). Let's see how to use them.

To encrypt the SS number, we would do the following:
var password = "foobar";
dojox.sql("INSERT INTO CUSTOMERS VALUES (?, ?, ENCRYPT(?))", "Neuberg", "Brad", "555-34-8962",
		password,
		function(results, error, errorMsg){
			if(error){ alert(errorMsg); return; }
		});
In the example above, we provide our three INSERT parameters as usual; however, for the last one, the social security number, we put the ENCRYPT(?) keyword around it. After providing these as variable arguments at the end, we provide a password. The password is used for encryption rather than a key -- it is passed into the underlying cryptographic system; you should not store this password as a cookie or into Dojo Storage or Dojo SQL. Instead, the user should be prompted to enter it when they start using your application. If you store it then a laptop thief could simply use it to unlock the local data store.

The final argument is a callback. If you use the ENCRYPT or DECRYPT keywords then the call to dojox.sql becomes asychronous. This is because under the covers we spawn Google Gears Worker Pool threads to do the actual cryptography in such a way that the browser doesn't screech to a halt. When they are done the callback will be called. If are doing decryption, as you will see below, then the results will have a normal Dojo SQL JavaScript results object, but with the values decrypted. If there was an error then error will be true and errorMsg will have a reason for the error.

Decryption is just as simple; simply put the DECRYPT keyword around the result columns you want decrypted:
var password = "foobar";
dojox.sql("SELECT last_name, first_name, DECRYPT(social_security) FROM CUSTOMERS",
		password,
		function(results, error, errorMsg){
			if(error){ alert(errorMsg); return; }

			// go through decrypted results
			alert("First customer's info: " 
					+ results[0].first_name + " "
					+ results[0].last_name ", " 
					+ results[0].social_security);
		});

In this example we simply print out the decrypted results for the first row. If the password is wrong then the decrypted results will still be decrypted; there is no way to detect an incorrect password attempt programatically.

You can combine several columns at once into a single ENCRYPT statement, such as ENCRYPT(?, ?, ?, ?), and you can do the same with DECRYPT statements, such as DECRYPT(first_name, last_name, social_security).

Under the covers Dojo SQL's cryptography is powered by 256-bit AES, using the passphrase you provide to derive the key. Specifically, we use the JavaScript AES implementation given here if you would like to study how it works; special thanks to Chris Veness for contributing the AES encryption code to Dojo.

Loading Offline Data On Page Load

Whether you use Dojo Storage or Dojo SQL, there is an important scenario you need to think about. If the user loads your application while offline, you must initialize your application using your stored data rather than making a network call. You should do this after Dojo Offline has loaded:

initialize: function(){
	// initialize our UI using offline data if necessary
	var documents = null;
	if(dojox.off.isOnline){	
		// make a network call to get our data
		// ...
	}else{ // we are offline
		documents = dojox.storage.get("documents");
	}
}

In the example above, our initialize() method is called when Dojo Offline is done loading. Our UI must then get a list of documents to make available; if we are online (dojox.off.isOnline), then we just make a network call. If we are offline, then we just load our documents from offline storage, in this case Dojo Storage.


We cover how to get your downloaded data for offline use in the next section, covering syncing.

Syncing

As soon as you start creating offline applications syncing becomes an issue. However, if you are not careful, trying to figure out syncing can become an endless black hole that causes your project to spiral and fail. Syncing can either be relatively easy or infinitely complex, depending on how you approach the issue. In order to help, Dojo Offline includes both a syncing framework, named Dojo Sync, as well as a set of syncing guidelines to help you avoid sync land mines. We've thought through many of the hard issues and created a path for you to navigate through the sync process.


Lets begin with the guidelines.

Sync Guidelines

First, user's view syncing as a bug, not a feature. They don't want to toy with sync interfaces -- they just want syncing to happen automatically.


Along these lines, user's don't understand merge interfaces. Developers are pretty comfortable with merging files, such as using Subversion or CVS. Users, however, don't care and don't understand merge interfaces. If you are popping up a complicated diff and merge UI as you do syncing, then you have failed.


Therefore, Dojo Sync recommends that rather than showing merge interfaces when syncing, the client- or server-side should automatically make a best guess, without UI intervention by the user. If a document was modified locally while offline, and also modified on the server by someone else, for example, they should be automatically merged without user intervention. If there is a conflict, you should make a best decision on which to keep based on your application. You could choose to keep the newest one, for example, and could also make a backup in the document's history list of the other change in case the user needs to retrieve something from it. Do merging automatically, and decide the best way your application can support this.


With all this said, users still need basic UI feedback that syncing is occurring and where in the sync process they are. Syncing should not be completely invisible; just as a browser has a basic network throbber to show you network activity is happening without showing the entire intricacy of the process, syncing is the same way. Users want to know that syncing is occurring; see from a high-level where the syncing is at (uploading, downloading, etc.); whether syncing succeeded or failed; and want the option of finding out why syncing failed or any automatic merges that might have happened. Even though users don't want to do merging, they some times want the option to look at a sync log to see what was merged. This should be done in a way that doesn't clutter the main sync UI (i.e. it should be a sync log link, for example, that would display a pop up window with a sync log of what was merged and what conflicted, and how the server did automatic merging).


Finally, user's don't want to have to manually always kick off syncing -- they just want the application to work correctly offline and automatically start syncing at correct times. In light of this, an application should always sync when it first loads and the network is present, bringing down a subset of data to make available offline. Users rarely know when they will be offline, so if the application simply always syncs then there should be some set of data available offline when the user loses the network. Likewise, when the network reappears, the application should automatically sync any local changes back to the server, with the user not needing to kick this off.

Dojo Sync Overview

Let's look at working with Dojo Sync and how it implements the guidelines laid out above.

The first thing to know about Dojo Sync is that it works in conjunction with the default Dojo Offline UI widget you embedded into your page earlier, doing much of the hard work of communicating the sync process to the user in a lightweight way. When you embed the offline widget you get a sync user-interface for free.

Second, syncing happens automatically, without the user having to initiate it. Dojo Sync automatically kicks off syncing when the web application first loads and we are on the network, and also if we are offline and we detect that network has come back online.

When syncing starts, three things happen in the following order:
  1. Dojo Sync downloads and caches our web user-interface -- any files that dojox.off.files.slurp() detected are now brought offline
  2. We upload any data that was changed or created while offline
  3. We download data to make available offline

As you will see in a bit, Dojo Sync does much of the work for you, and provides some specific places you hook into to help it do its job.

Because Dojo Sync hooks into the offline widget, the user-interface for these three steps is completely automated. Let's quickly see what this default sync UI looks like from an end-user's perspective, since this can help you as a programmer understand how to tie into Dojo Sync.

First, Dojo Sync downloads our web user-interface to make it available offline:
Next, our web application uploads any changes that were made while offline:
Finally, we download data to make it available offline:
If everything is successful, the user is notified:
If there was an error, the user is also notified:
The user can find more details about the error by pressing the details link; this will cause a pop up window to appear with details on why the sync error occurred:


Now that you have an overview of Dojo Sync and what its user-interface looks like, let's see how your application code actually hooks in.

Hooking Into Dojo Sync

As stated before, Dojo Sync does much of the heavy lifting for you around user-interface and syncing. Your application only has to hook into two parts of Dojo Sync to help out: downloading data to make available when offline, and uploading any changes that were done while away from the network. Downloading is the simplest.

Sync Downloading

During the syncing process, Dojo Sync fires sync events that you can subscribe to:
dojo.connect(dojox.off.sync, "onSync", this, function(type){
});

There are many different onSync events, given by the type variable, but you only need to be concerned with a few unless you are doing something unusual (you can see a full list of the onSync events here).


When the sync process reaches the downloading stage, onSync will fire with type equal to the string download. You should listen for the download event and proceed to download your data. In the example code below, when the sync framework wants us to download, we make a network call to some web service that fetches new documents, for example:

dojo.connect(dojox.off.sync, "onSync", this, function(type){
	if(type == "download"){
		// download new documents
		dojo.xhrGet({
			url:		"/documents/download",
			handleAs:	"javascript",
			load:		function(data){
				dojox.storage.put("documents", data);
				dojox.off.sync.finishedDownloading();
			},
			error:		function(err){
				err = err.message||err;
				var message = "Unable to download our documents from server: "
								+ err;
				dojox.off.sync.finishedDownloading(false, message);
			}
		});	
	}
});

In the example code above, we first subscribe to onSync events. When a sync event fires, if it is the download event, we then proceed to make an XMLHttpRequest call to some web service that downloads new documents, located at "/documents/download", which returns JavaScript JSON as its results. We use Dojo's easy to use dojo.xhrGet() method to do so. xhrGet() can take an error handler and a load handler, as well as allowing us to specify how the return results should be handled. In the example above, our handleAs: "javascript" parameter says that the results will just be handed to us as a JavaScript object.


Let's look at the load() method first. If the web service returned correctly, we simply take the data returned, which would be our list of documents, and store it right into Dojo Storage with a put() call; we could also have used Dojo SQL here and iterated over the results to write into a relational store if we wanted.


The next line is important:


dojox.off.sync.finishedDownloading()


You must call this method when you are done downloading. Since downloading data tends to be an asychronous process that involves talking to the network, Dojo Sync has no way to know when it can continue the syncing process after downloading is complete. When you call dojox.off.sync.finishedDownloading() from your network callback, it tells Dojo Sync to continue. If you do not, Dojo Sync will get stuck at the downloading phase and your user-interface will just show "Downloading new data..." forever.


Let's look at the error callback now. If an error occurs during downloading, you must also call dojox.off.sync.finishedDownloading(), but with two arguments. The first should be false to indicate that downloading was unsuccessful; the second argument should be the reason that downloading failed, which will be reported in the sync log at the end of the sync session.

Sync Finished

We will see how to handle sync uploading in the next section, but first there is one more sync event that you will find useful in your code. When syncing is finished a sync event is fired with type "finished":

dojo.connect(dojox.off.sync, "onSync", this, function(type){
	if(type == "download"){
		// download data
		// ...
	}else if(type == "finished"){
		// refresh our UI when we are finished syncing
		var documents = dojox.storage.get("documents");
		dojo.forEach(documents, function(i){ 
			// update the UI list of documents with this new document
			// ...
		});
	}
});

In the example code above, when the finished event is fired, we load all of our new documents from Dojo Storage and then use each one to update our user-interface. The finished event can be useful for updating your UI once syncing is done.


Note that this event is fired whether an error ocurred or not during syncing; check dojox.off.sync.successful and dojox.off.sync.error for sync details. These are boolean properties that will tell you whether an error occurred or not.

Sync Uploading

Sync uploading is the trickiest part of syncing in general. Before we can tackle that, let's take a quick look at how Dojo Sync recommends you record user actions when they are done offline.

Offline User Actions

What do you do when a user is offline, but starts doing user actions that would normally cause a network call or which updates data?


Dojo Sync adopts what is known as a transactional model to solve this problem. When a user works offline, creating a new business contact or modifying an existing one, for example, we create a record of this action and update our local data. These actions are added to an action log.

For example, imagine that a user is working offline with a web application that has sales contacts in it. The user first modifies the contact record for "Brad Neuberg," changing the phone number. Since we are not on the network we can't write this change to the server; instead, we create an action object to indicate that "Brad Neuberg" has been updated, and store it in an action log. We also update our local offline data with the new phone number. The user next creates a new contact, named "John Doe"; again, we create an action object to represent this offline action, and store the new contact.

When we go back online, all we have to do is replay the action log, which will simply execute each of the actions while online in the exact same order they were performed when offline. When we replay actions while online, all we are doing is essentially "simulating" that the user just did them right then, but very fast. The great thing is you can call existing web services that you might have to handle these actions once online.

In the example above, when the network first appears and we start syncing and reach the Sync Upload phase, we first replay the user action of updating the "Brad Neuberg" contact. Replaying this action executes doing what we normally do while online, which is to call some network service to update some contact. In our code example above, we might do an XMLHttpRequest call to POST the updated contact data to "/contacts/Neuberg/Brad". We then "replay" the user action to create the "John Doe" contact, which might do an HTTP POST to "/contacts/Doe/John" with the new contact.

As recommended in the Sync Guidelines section above, these web-services should do automatic merge and conflict detection so that the user is not inundated with complicated merge UIs. If you insist on having a merge UI, see the Advanced Syncing section.

Let's see what all of this looks like from a code level now.

If the user is offline and does some action, we create action objects and add them to our action log:
// some new client added by sales person
var client = {lastName: “Doe”, firstName: "John", phoneNumber: “555-222-1212”);

// create an Action object to represent this action --
// these can have anything you want in them -- as you will
// see later, they should have enough data to help you replay them
// when we go back online
var action = {name: "new contact", data: client};

// add this to our offline action log
dojox.off.sync.actions.add(action);

// persist the contact into offline storage
dojox.storage.put("John Doe", client);

In the example code above, we first have an object that represents the new client that was created, John Doe. Next, we create an action object to represent this offline action; the action object can have any values you want -- it must simply have enough data for you to be able to replay this action when we go back online. In general, you will have an action name, such as "new contact", and some data that goes along with this action, which in this case is the newly created client.


Next, we add this action to our offline action log. The offline action log is automatically persisted, and records any actions that were done while offline, in the order in which they were performed by the user.

Finally, we persist this new or updated action into local offline storage; even though we can't communicate on the network, we want to keep our local data up to date with any offline changes that were made by the user.

Replaying

When the network reappears, Dojo Sync kicks off and starts automatic syncing. After refreshing the offline UI, Dojo Sync then attempts to upload any changes that were made while offline. Basically, Dojo Sync tries to replay the action log you built up while the user was offline, executing each one one at a time.


However, Dojo Sync doesn't know how to replay actions -- thats the job of your application. You register yourself to know when replaying has occurred, and Dojo Sync calls your replay function for each action, simply handing it the action object you created earlier while offline. You can now use this action object to execute the action, but this time while online.


Let's build this code segment up one piece at a time. First, we register to know when replaying has occurred:

dojo.connect(dojox.off.sync.actions, "onReplay", this, 
		function(action, actionLog){
		}
);

The function we give will be called over and over for each action that was added to the offline log. The offline log is just a persistent array, or list of actions in the order they were done by the user and added by you to the action log. This function is given each of the actions, one at a time, as the first argument action, and a reference to the action log itself, actionLog. Your code should then look at the action and try to replay it:

dojo.connect(dojox.off.sync.actions, "onReplay", this, 
			function(action, actionLog){
				if(action.name == "new contact"){
				}
			}
);

In this case we just check the action.name property, which we set earlier when the user actually performed this action. If it is the value "new contact", then we try to simply create this new contact:

dojo.connect(dojox.off.sync.actions, "onReplay", this, 
	function(action, actionLog){
		if(action.name == "new contact"){
			var contact = action.data;
			
			// create this new contact on the network
			dojo.xhrPost({
				url:		"/contact/" + contact.lastName + "/"
							+ contact.firstName,
				content:	{ "content": dojo.toJson(contact) },
				error:    	function(err){
							var msg = "Unable to create contact " 
								+  contact.firstName + " " 
								+ contact.lastName + ": " + err;
							actionLog.haltReplay(msg);
					  	}
				},
				load:		function(data){
							actionLog.continueReplay();
						}
			});
		}
	}
);

There's alot going on in the code block above. First, if we see that the action was to create a new contact, "new contact", then we actually shoot off an HTTP POST to create this new contact. We get the contact to create from the data:


var contact = action.data;


And then do the actual HTTP POST using this data. Our URL becomes the first and last name of the contact, or /contact/Doe/John in this case. Dojo's xhrPost() method takes the content of the post as an attribute named content, which we provide. We transform our contact into a JSON string to easily send to the server with the dojo.toJson(contact) method call.

xhrPost() calls error() or load() based on whether things were successful or not.

Notice that we call something called actionLog.haltReplay(msg) and actionLog.continueReplay(). Since most of the things we will do while replaying are asychronous, we must tell Dojo Sync when to continue replaying and whether an error occurred. If the action was replayed successfully, you should call actionLog.continueReplay(), such as we do above in the load() method. If an error occurred, you should tell Dojo Sync to halt replaying by calling actionLog.haltReplay(msg), providing a message string on what the error was -- this error message will be written into the sync log for users to see if they want.

Remember that these web-services should do automatic merge and conflict detection so that the user is not inundated with complicated merge UIs.

One final property that is useful is dojox.off.sync.actions.isReplaying. Sometimes you want to share the same network code whether you are online or replaying an offline action, since you don't want to have to rewrite the network call. In this case in your load or error callbacks you can see if replaying is occurring, and if so tell the action log to continue replaying or to halt. If you are not replaying, you would just handle the load or error as normal.

Speeding Up Syncing: version.js

During syncing, we always refresh the list of offline files. This is somewhat due to limitations in the underlying Google Gears APIs we use (for the technically inclined, we use Google Gear's ResourceStore instead of it's ManagedResourceStore). This can slow down syncing considerably, however, since every time we sync we also pull down the offline files.


Dojo Offline has an optional feature to speed this up. Your application can have an optional version.js file bundled in the same directory as it. During syncing, Dojo Offline will read this file to see if the version of the application has changed. If it has then we go ahead and refresh all the offline files; if not, then we skip this step.


The version.js file is very simple; it should simply have the version inside of it:


"07-12-2007.9"


This can be a string or number; it does not matter. We simply look to see if the value has changed. We also force an offline file refresh if you are in debug mode (i.e. djConfig.isDebug is true), or if you just debugged the prior time the page was loaded.


Simply place this file in the same directory as your main HTML file.


This is all your should need to know to work with Dojo Sync. For further details and advanced usage see here.

Further Resources

The following resources are available to help you while developing with Dojo Offline. In addition, this tutorial has extensive reference information below this section in an Addendum that provides further technical information and advanced usage of the Dojo Offline library.

Addendum

The following sections provide further information and advanced usage information useful for curious developers. You can safely ignore these.

More Info: Offline Widget Walkthrough

Let's take a look at what the offline widget looks like, to see all the stuff it does for you without lifting a finger - you can skip this section if you want, but it can be helpful to see what the offline widget looks like and what it does.

Let's peek at the offline widget a bit more, showing you its various states. If the user is offline, the offline widget will look like the following:

When the user goes back online, Dojo Offline automatically detects that the network and application have reappeared, and begins syncing:

When done, the offline widget provides automatic feedback to the end-user:

If the user does not have Google Gears installed, then we provide a prominent Learn How link:

If the user clicks the Learn How link, a popup-window will appear that provides them instructions on how to install Google Gears and use your application offline:

Notice that your application name is inserted into this Learn How page; the Learn How page automatically inserts your application's name, defined by setting dojox.off.appName, into this page, making it customized for your end-users. You do not have to edit it's HTML source.

The Learn How page also provides an easy hyperlink for your end-users to run your application when offline. When offline, a user would open their browser as normal and type your web application's URL into the location field, such as www.mywebapp.com. However, Google Gears is sensitive to how a user types this; for example, if they type mywebapp.com instead of www.mywebapp.com, they might not be able to pull up the application offline and will get an error. Also, if your web app has a long name, such as http://mywebapp.com/foobar/something then this is tedious and error-prone for an end user to type when offline. To solve this, the Learn How page automatically provides a hyperlink to your web application for a user to drag to their desktop or browser link toolbar:


When offline, they can double-click this link to open a web browser and easily access your app when offline. Dojo Offline automatically figures out what the URL of this link should be (instructions on how to change this default URL are available here).

If Google Gears is already installed, perhaps from working with another offline web application such as Google Reader, the offline widget still has the Learn How link at the bottom of the widget:

When clicked, though, the Learn How page automatically adapts to show less instructions, basically just providing the user with some minimal instructions on how to use things offline and provides the same draggable hyperlink:


Again, all of this is the kind of fit-and-finish that is hard to get right that Dojo Offline provides out of the box - it all comes for free.

If you want to customize the look and feel of the offline widget then learn more here.

More Info: Manually Adding Offline Files

Manually adding offline files is easy; you can even mix this with the slurp() method if you want, so you can slurp the page for most of your resources, then perhaps manually add a few that weren't picked up because they are dynamically brought in through JavaScript.

To add a single file:
dojox.off.files.cache("images/some_image_added_by_javascript.png");
To add many files at once:
dojox.off.files.cache([
                       "images/file1.png",
                       "images/file2.png",
                       "scripts/some_dynamic_script_tag.js"
                       ]);

You can call cache() as many times as you want; duplicates are automatically removed, and new cached additions are simply added to the end of our internal cached file list.

More Info: How Network Checking Works

Dojo Offline has a timer that every fifteen seconds is checking to make sure that your web application is available. By default, it looks for a simple static text file on your web server bundled with the Dojo Offline framework, located at dojox/off/network_check.txt. This file is very short, with just the number 1 in it. We use a static text file rather than hitting your main web page since the main web application is probably dynamically generated and hitting it every 15 seconds would impose scalability limits, while a static text file is simple to serve.

You shouldn't need to change this part of the framework, but if you need to the interval for checking can be set on dojox.off.NET_CHECK and the availability URL can be changed from the network_check.txt file to your own custom page by setting dojox.off.availabilityURL. If you want to completely disable this feature, you can set dojox.off.doNetChecking to false.

Advanced Usage

This section describes various advanced usage scenarios that most users will never need to use; you can safely ignore these unless you are doing various strange, advanced things.

Advanced Usage: Learn How URL

On the Learn How page is a URL that a user can drag to their desktop to run your web application when offline. This URL is set automatically to the main page of your web app, but you can manually set it for unusual situations by assigning a URL to dojox.off.runLink and change the title of the link by setting dojox.off.runLinkTitle; note that runLinkTitle should be normal text and not HTML.

Advanced Usage: Change The Dojo Offline Widget's Look and Feel

If you want to customize the look and feel of the offline widget you can. When we automagically embed the widget on page load, we actually grab a template file located at dojox/off/resources/offline-widget.html. This file simply has a bunch of HTML elements that we inline into the page. Every element has an ID, such as dot-widget-network-indicator, that you can attach to with CSS to change. There is also a CSS file with defaults in dojox/off/resources/offline-widget.css.

You have several options for customization. The easiest is to just create new CSS rules for the element IDs in the offline-widget.html file; look up the name of the ID that you want to change, and then create a CSS rule. More advanced is to cut and paste the HTML from the offline-widget.html file right into your HTML page, getting rid of the dot-widget automagic DIV you used in the past to have the template file inserted. Then, you can selectively drop any elements you want from the HTML, including adding new ones -- Dojo Offline is designed to be able to work even if you have dropped parts of the template.

If you want to change the Offline Widget's image icons for some reason, such as for branding or color palette reasons, the following properties can be set through JavaScript to a new URL:


For example, if you wanted to change the green online ball to a different icon, you would do the following:
dojox.off.ui.onlineImagePath = "images/myonlineball.png";
If you want to brand the Learn How page in some special way, its HTML is at dojox/off/resources/learnhow.html and its CSS is inside dojox/off/resources/offline-widget.css, near the bottom of the file.

To completely throw the Offline Widget away, you can simply set dojox.off.ui.autoEmbed to false. Note that while Dojo Offline is designed to not use the offline widget, dropping the offline widget and doing everything manually is currently not well tested yet. It is recommended that you use the offline widget in general.

Advanced Usage: Dojo Storage

This section provides further details on aspects of Dojo Storage that can help with advanced usage.

In addition to the standard put() and get(), there are some other methods on dojox.storage you might find useful:



Some readers of this tutorial might have used Dojo Storage in the past, before the new Google Gears and Dojo Offline integration. Some important notes. First, Dojo Offline provides a new Dojo Storage Provider named GearsStorageProvider, which stores it's hash table data into Google Gears. Second, we do not bundle any of the other storage providers with the Dojo Offline build, such as the Flash and Firefox storage providers, since they are not needed when used in conjunction with Google Gears and take up valuable space. Finally, you will notice that we did not give a callback to the put() example above like in the past; when using Dojo Storage in conjunction with Dojo Offline and Google Gears, put() calls happen immediately and won't be denied, so you do not need to provide a callback. This makes doing put() calls much easier.

You can safely ignore knowing this, but it can be useful for some applications. The default namespace, if none is specified for the various Dojo Storage methods that take an optional namespace, is in dojox.storage.DEFAULT_NAMESPACE, which has the value default. The underlying Google Gears database is used to store the Dojo Storage values; the table name is in dojox.storage.TABLE_NAME (it is actually defined in the dojox/storage/GearsStorageProvider.js file), and it's value is __DOJO_STORAGE. Knowing these values can help with debugging in some cases so you can peer at them in the underlying Google Gears data store if necessary.

Advanced Usage: Dojo SQL Open/Close

The underlying Google Gears SQL data store must be opened before SQL can be executed, and should be closed when it is not used, though it will recover if you do not close it. Dojo SQL automatically does this open/close routine to simplify your code. When you execute a SQL statement with Dojo SQL, it will automatically open the database, execute your SQL, and close it. If you are working on performance sensitive code, where you have a for loop for example that is rapidly executing hundreds of inserts, this behavior could slow your app down considerably. In this case, you can just manually call dojox.sql.open() and dojox.sql.close() yourself, and Dojo SQL will automatically detect this and leave the database connection alone and open:

var bigData = getBigData(); // get an array that has a bunch of data in it

dojox.sql.open();

for(var i = 0; i < bigData.length; i++){
    dojox.sql("INSERT INTO MYTABLE (LAST_NAME, FIRST_NAME) VALUES (?, ?)", bigData[i].lastName, bigData[i].firstName);
}

dojox.sql.close();

Google Gears also supports having multiple named relational data stores for a single application; Dojo SQL does not currently expose this functionality, and stores its dataset as a Google Gears relational data store under the name given by dojox.sql.dbName, which is currently PersistentStorage. Note that if you have multiple applications running on the same web site they will create their tables in the same data store, the PersistentStorage one, which can create conflicts; this is a known issue that will be fixed in the future.

Advanced Usage: Dojo Sync

The Dojo Sync onSync event fires more than just the download and finished events touched on in the tutorial above. Here is the full list of event types if you want to do deep customization, such as updating your UI to display the progress of syncing (note that the default Dojo Offline UI widget does this for you if you choose to pull that in). Most of these are only appropriate for advanced usage and can be safely ignored:


If you want to pop up a merge UI, you can create your own and choose to pop it up during your custom onReplay() callback during replaying.

If you don't want to use the transactional action log system for syncing, register for the upload event and simply do uploading however you want.

If you want to completely override the sync process in general and roll your own sync infrastructure, override dojox.off.sync.synchronize() with your own logic; make sure to refresh the offline files as a minimum, though. Rolling your own sync infrastructure is not well supported, however, and is not recommended.