Mission Net Application: Documentation

Mission Net is a solution for sending messages to people via mobile phone text messages (SMS) and/or email. It was developed especially as a way to get important information (warnings about community unrest etc.) to members of our organization in an unstable country where Internet access is limited, but nearly everyone has a mobile phone. A key aim of the application is that it should be accessible by SMS and email. Users should not only receive messages but should be able to request information, broadcast messages, and change their personal details without having to use the Internet. A second driving aim is that the entire application resides online, so that there is no need for a local server or operator.

See the Features section below for a more complete description.


Important Note: this application is in development, and so is this documentation. Furthermore, this is my first experience in making an app open source, so I’ll probably be making some mistakes along the way. Anyone is welcome to try out the system (which would involve installing and configuring it)[1] but you should use caution if you’re considering it for production. It is not yet an easy-to-install, ready-to-use application, at least if you have no experience putting a Rails application into production. That said, we are currently using it as a real-life application for rapid communication via SMS and email among nearly 200 users in Jos, Nigeria, where the main purpose is dissemination of security information. To date it has only been used quite lightly, so we don’t know how it will perform under a heavier load.

There is not yet a demonstration or sandbox site, but I hope to be able to add one soon. There are some screenshots throughout this document.

If you are interested in setting up a demo site, let me know.

Note that “Mission Net” is a provisional name for this application … it might conflict with some other app, or we might find a better name in the future …


Contents

Contents

Features

Using Mission Net

Accessing Mission Net via Web Browser

Send Messages

News Updates

Members/Users

List View

Form View (for creating/editing individuals)

Departure and Arrival Dates

Groups

Access Mission Net by Mobile Phone SMS Messages

Access Mission Net by Email

Bugs & problems

Comments and Requests

To Do List

(A Few) Technical Details

Setup (A Sketch)

Installation

Customization

Preparing IronWorker

Testing

Deploying to a Server

Heroku

Other Servers

Maintenance tasks

Under the Hood

Authentication

Authorization

Features

Using Mission Net

Accessing Mission Net via Web Browser

If you have Internet access, this is the easiest way to use or manage the system.

Login via your Facebook or Google account. The email address returned by those must match one of the email addresses you have on the database. (For example, if your email address for logging in to Facebook is myaddress@littlefoot.com, then you can log in to Mission Net if your primary or secondary email is that address.)

Use the menu on the right to view, add or edit members, groups, or messages. Other tables such as locations and system settings can be accessed by users with appropriate permissions.

Send Messages

To send a message, click on the “Create new message” link in the main menu. Within the new message form, fill in the applicable fields.

An individual will not receive more than one message, no matter how many groups he or she belongs to, so there is no problem with selecting overlapping groups.

You may optionally add keywords (under advanced options) or type in a longer message in the Email box, which could be useful later if you want to follow up or search for the message later.

News Updates

Members/Users

(Users are simply members when they’re accessing the system. At this point, there is no separate list of “users” as opposed to people listed in the database. )

List View

The Wife Field

To link a couple, click the Wife field in the husbands row.[3] Choose from the possibilities, which represent all the unmarried people with the same last name. (In the event you need to link a couple with a different last names, start by giving the same last name, link them, and then change the last names to the way they should be.)

Form View (for creating/editing individuals)

Departure and Arrival Dates

The departure and arrival dates recorded for each member, besides being possibly useful information in themselves, are used to determine when someone is “in-country” (being “in-country” just means that the person will be included in group email and SMS broadcasts).[4] (The system looks through all the records early each morning and updates their “in-country” fields. You can use the dates in any way that makes sense:

The main advantage of this system is that you can define the dates in advance, and the system will know each day who is leaving or arriving. Otherwise to really keep up-to-date you would have to manually select and unselect the in-country field on every day that someone comes or goes.

The system will not change the in-country status when something does not make sense. For example, if on the one hand you list someone as being in the country, while on the other you show their departure date as in the past, the system won’t change anything. Perhaps in the future it will warn you, or perhaps not!

Groups

Groups have two functions. First, groups are used to select message recipients. Create any groups you need, select the members for each one, then use the groups for sending messages. (Presently, you can not assign individuals to receive messages, only groups.)

Second, groups determine the roles (privilege levels) of users on the system. When viewing the list of groups, you can select which roles apply to which groups. Note that the roles have an order (none < limited < member < moderator < administrator); if you select one level for a group, the lower levels will automatically be included.

Groups do not need to include a role. For example, if you have a set of people who should receive messages but not be able to log on to the system at all, don’t give that group any role. If you need to allow some of those people to have higher privileges, either create a new group for them or add them to an existing group that contains the role you want. A person’s effective role is the highest role in any of the groups they belong to. For example, if someone belongs to group A which has only a “limited” role, and also to group B which has a moderator role, her effective role will be moderator.

NB: All the data in the database, including groups and members, is shared. Thus in a situation where more than one organization is using the same installation, users need to be careful, cooperative and considerate about how they define and use groups. For example, if three organizations share the database, “staff” might not be a good group name.

Here are the system’s rules about roles:

[The privileges of different roles can be adjusted through the CanCan interface. The file to change is app/models/ability.rb.]

Access Mission Net by Mobile Phone SMS Messages

One of the most important reasons for the existence of this system is the need to be operate without an Internet connection, which will never be available to everyone and will sometimes be available to no one. Therefore, one goal is to make as much functionality as possible available for ordinary mobile phones using simple SMS text messaging.

Users communicate with the system by sending an SMS containing a command and possibly options and a message to store or forward. The case (upper or lower) of commands (and options??) does not matter. Spaces are used to separate the elements, which is why a group abbreviation cannot contain a space (only the first part would be seen).

Currently, these commands are available:

Command

Action

Examples

d group message

deliver to group (currently, a single request can only target a single group)

d alerts Warning that there could be trouble tomorrow so be careful.

groups

send the user a list of groups marked as “primary”

groups

help

send list of commands

help

info name

send information on the requested directory entry, based on name fields

info Gordon

info Jack

info Jack Gordon

report message

posts message as a news update

report Heavy traffic near stadium.

updates keyword limit

(keyword and limit are optional)

send current updates. If a keyword is given, search for messages containing it. If limit is given, return no more than that number of messages.

updates

updates traffic

updates traffic 3

updates 3

!msg-id reply-message

confirm receipt of message with number msg-id. The optional reply-message (up to one line of text) will be stored in the database under the original message. Currently, the reply-message is not delivered to the sender of the original message.

!21

!21 Thanks for the info.

!21 All safe at home (e.g. in response to a message requesting that everyone reply with their location and status)

If the system receives a message from a member and it does not contain one of these commands, then

[More explanation: The system is not designed to handle two-way traffic, for two reasons. First, it will generally cost more to send a message to the system (international SMS) than directly to another member. Second, unlike email, SMS messages have no “Reply-to” field. There is no way to know who an incoming message is intended for unless it contains a reference to an outgoing message, as in the “confirm receipt” message (see table above). Furthermore, a forwarded message no longer contains the original phone number, so the recipient will not know who it is from unless the information is added into the body of the message.]

Access Mission Net by Email

[Need to expand this a bit but here are the basics.]

You can configure an incoming email address. You must be sending from one of your registered addresses.

Subject line is not needed

Body can be like the SMS commands except that updates and groups are not yet implemented. Command “d” sends SMS to groups, “email” sends email to groups, and “d+email” or “email+d” sends both email and SMS. Differences from the “d” command sent by SMS: (1) more than one group can be specified, with spaces between them (2) a colon must follow the last (or only) group. So you can have

d alerts: Blah blah blah   Sends SMS to alerts group

d+email alerts directors: Blah blah blah   Sends SMS & email to alerts & directors

email members: Message to everyone... Sends email (not SMS) to members group

The colon is needed so that the program can distinguish the group names from the start of the message.

You should get a confirmation message saying that your broadcast message has been sent.

Updating the directory by email

You can use email to update the information listed in the database (directory). Simply include lines like

change Adam Wolf 0808-888-8888 awolf@sample.com

The only formatting requirements are that the name be the first thing after the change command and that the phone number cannot contain spaces. The system will do its best to find the matching name even if you give only the first or last name. The phone number will be normalized when stored. [To date, only a single change command should be sent in one email].

When your email reaches the system, it will be checked for consistency. If the name identifies a unique person and you have the required privileges to update the person’s data, the system will return to you a verification email describing the changes to be made. If those are correct, simply reply to the email (just click “reply” and send the message).[5]

If more than one person matches the name, or if there are other problems, the system will email an explanation back to you.

Reports

Select the Reports menu item to generate concise, printable listings of information. The only report type available so far is the directory, a list of email addresses and phone numbers. You can view the whole list on the Web browser, which is more convenient than using the normal Members view, or you can download a printable (PDF) version. Select which groups you want to include and whether you want a list sorted by location, by name, or both reports.


Bugs & problems

Just add a description of the issue you’re having

Comments and Requests

To Do List

These are in no particular order

Function & usability

Stability & Generalizability

These features are required or useful for setting up new installations.

(A Few) Technical Details

Framework: Ruby on Rails version 3.2, with ActiveScaffold.

Database: PostgreSQL (The app is also configured to include Redis (in-memory DB), but this is not much, currently only for caching the current user role.)

Source repository: http://github.com/MikeBlyth/mission_net

SMS Gateways already configured: Clickatell, Twilio

Setup (A Sketch)

Installation

This is currently just my guess at how to install this application on a new site. You can use if as a guideline, but someone with experience in Rails should be around to help!

Just a thought: You might be able to deploy directly from the source (on github) to a server (such as Heroku), but I don’t have any experience doing that and there may still be some customization needed, which will be easier to do on your local machine than on a server.

I’m using Linux Ubuntu 10.04 for development. You can follow most of the same instructions on Windows, and probably Mac, but Windows is much slower with Rails development than Linux, in my experience (though this was with Rails ver 3.0; 3.2 may be faster).

  1. Start with a fresh installation of Ubuntu (other distros may work also).
  2. Start at the command line (open a shell prompt)
  3. Install Git source manager
  4. Install Ruby (I’m using 1.9.2p0, but newer versions will probably work also. See also Three Ways of Installing Ruby
  5. Install Rails (just a matter of “gem install rails”)
  6. Create the source folder for this application:
  1. go to the folder into which you want to create the application folder. Your home folder is fine. The following step will create a new folder “mission_database”.
  2. git clone git://github.com/MikeBlyth/mission_database.git
  3. If you want to use a folder name other than “mission_database,” add that name after the clone command above, e.g.
    git clone git://github.com/MikeBlyth/mission_database.git myfolder
  1. Go into the project folder you just created, i.e.
  2. Install all the dependencies used by the application: bundle install
  3. Install the PostgreSQL database (Heroku currently requires using PostgreSQL. If you’re using a different host or if they’ve changed their policy, use MySQL or whatever database you prefer; see the Rails documentation for how to do this).
  4. Edit the database definitions found in config/database.yml.example, and save as the new file config/database.yml (see Configuring a Database in the Rails Guides)
  5. Initialize the database: rake db:setup
    This creates the database and seeds it with initial data.
  6. Run the local server: rails server
    This process stays in the foreground by default, so you can see the log, so you may want to put it into its own instance of the shell (command line window).
  7. Open a browser and point it to “http://localhost:3000”. You should get a page that offers to set up an administrator, since your database of members is empty. Enter and save the information.
  8. Now you should be logged in (or able to log in) to your local system.

Customization

Most of the parameters that need customization are available from the running program itself, but currently (and possibly while you’re following these instructions) there are still some that need to be changed in the source files.

  1. Database definition: see above (editing the database.yml file)
  2. Edit config/application_settings.yml, if it exists, and enter your own organization’s name, administrator email, etc.
  3. The file config/site_settings.yml, if it exists, contains gateway and service settings, some email addresses, etc. You can edit it and enter the values you wish in the “default:” lines, or you can simply edit the same data from the web interface by selecting Settings (or something like it!) from the main menu. These settings include:
  1. SendGrid username and password if you’re using SendGrid for your email (but note: the app is not currently set up to use SendGrid so you’ll have to modify the code if that hasn’t been done by the time you’re reading this).
  2. Credentials for SMS gateway providers such as Twilio and Clickatell. You do not need both.
  3. Twilio Background job provider (“DelayedJob” on Heroku, or “IronWorker”) [currently only used with Twilio outgoing SMS, since Clickatell is already fast, needing only a single API call to send up to 300 messages].
  4. Notifications: a list of email addresses, comma separated, to which various notifications will be sent (changes of address, new locations, etc.). This might not be implemented yet.
  5. Parameters for how many items to keep in the log, messages, and sent-messages folders.
  1. Edit config/initializers/private_parameters.rb.example and fill in the parameters you need. These include:
  1. OmniAuth authentication. This is the system that lets you piggyback on Google, FaceBook, or other servers for authentication. That means the users can log in without having to know another password just for this system. See http://www.omniauth.org/ for information.
    This Mission Net application is already configured to use Google and FaceBook for authentication, so you just have to get API keys for those and plug them into the proper places in the private_paremeters file. You can add other “strategies” as well but will have to follow the instructions at the OmniAuth site.
    The
    List of Strategies describes which “sites” (loosely speaking) can be used, such as Facebook and Google, and links to instructions on using them. The current setup for Google uses their OAuth2, and requires you to get an API Key at https://code.google.com/apis/console/. The info on getting a key for FaceBook is at http://developers.facebook.com/docs/authentication. Of course, these sites change from time to time so you may have to hunt for the instructions.
  2. Heroku API Key. This is needed if your app is deployed on Heroku and you want to use DelayedJob to run background tasks like sending out SMS messages. All you need is to check on your accounts page to copy the key. This is the easiest option to set up.
  3. IronWorker credentials. The other built-in option for running background tasks is IronWorker. Simply go to that site, sign up for a free account, and enter the credentials into this file. (You will still have to save your “worker” to your IronWorker account, see below).
  4. SMTP outgoing email account information. Get this from your email provider. The online Rails Guides has a section on configuring this for a Gmail account.

Maintenance tasks

A number of maintenance tasks are included in the package (under lib/tasks)  and can be run on a fixed schedule or as needed. These are run with the rake command. They include

To run one of these commands manually, simply run them from the command line in the application folder, e.g.

rake unduplicate_group_members

Preparing IronWorker

If you are going to use IronWorker to do background jobs, you need to upload the “workers” first. Currently only an SMS sender (twilio_multi_worker) exists but in the future an emailer may be added. If voice is used, this should also be done in the background.

The iron_worker program on your local computer should have been installed already as it is part of the Gemfile which you ran with bundle. You still need to install your credentials, though, so follow the instructions on the Iron.io site. Currently, this requires creating a file named iron.json which looks like

{

        "token": "YOUR TOKEN",

        "project_id": "YOUR PROJECT ID"

 }

and placing it in either in your home folder or in the folder that contains the workers.

The program code for workers is found in lib/workers. To upload twilio_multi_worker, just type

iron_worker upload twilio_multi_worker

Internationalization (I18n)

Mission Net incorporates Rails’ I18n API, making it is easy to add a new locale, which would include mainly translations of messages in a new language but also the date and time formats. Simply copy the config/locales/en.yml file to one representing the new locale (e.g. es.yml for Spanish), and update all the strings with the proper translations. See the French file (fr.yml) as an example. Presently, there is also a separate fr_datetime.yml file from a third party. If you are adding a new language, you might find such a package for that language so that you don’t have to add every detail yourself.

Testing

Rspec is used for running a suite of tests on the app. The tests are not perfect and don’t cover all the code, but they’re better than nothing. To run them, type

rspec spec

on the command line in the application’s main folder.

Deploying to a Server

Heroku

  1. Get an account on Heroku and follow their instructions for deploying your app. Once you’ve gotten the account and installed the Heroku local application, getting the application running on Heroku should be as simple as
  1. git push heroku master
  2. heroku run rake db:setup
  3. heroku ps:restart

However, you’ll have to consult the Heroku documentation to get up-to-date instructions.

Other Servers

I don’t know about these, so you’ll have to do it yourself.

Under the Hood

A disorganized hodge-podge of bits of information about the program as they arise …

Authentication

OpenAuthorization providers Facebook and Google are used. Others can be added. Authentication is based on email addresses. When the user clicks on “Connect using your Facebook account,” the authenticator connects to Facebook and verifies that the user is logged in (or can log in) to Facebook, and returns an email address. If the application finds a matching email address in the database of members, it assigns the owner of that address to be the current user. If more than one person shares an email address, the one with the highest privilege is assigned to be the user.

Authorization

This whole area may need to be reworked, but this is the current setup.

NB: there is some confusion because of the mixed terminology I started with and have not yet systematized. “Member” can mean (a) the model representing the people who are listed in the database or (b) a role (privilege level). So a member in sense (a) might not have the role (b) of member, but might have a limited role or no role. That is, there can be entries in the database for people (or entities) that have no privileges to access the database. Another confusing area is that “users” (those accessing the database) are actually members (in sense (a)). There is no separate model for application users as opposed to members (a) contained in the database.

Presently, though, the system uses 4 pre-defined roles or levels of authorization: administrator, moderator, member, and limited. These are expressed in the code in symbol form, e.g. :administrator, :limited, and so on. They are ranked in the order given, so that administrator will have all the privileges of the levels it under it and so on. [To do: Currently ranking occurs in several places: Moderator.recalc_highest_role, Moderator.roles_include?, and Ability.initialize, possibly others! It would be good to formalize a single location for ranking, whether by creating a separate class for roles or a single method to compare them.]

Important methods include

The privileges of different roles can be adjusted through the CanCan interface. The file to change is app/models/ability.rb.

Simplified diagram of how incoming and outgoing SMS messages are processed. Click on image link to see zoomable image.


[1] The application uses Rails 3.2; source repository: http://github.com/MikeBlyth/mission_net

[2] Authorizations are currently hard-coded into the program, using the pre-defined groups Administrators, Moderators, Members, and Limited (with “no privileges” being the default absence of any of the others). More fine-grained authorization controls may be added later.

[3] A spouse field would be more straightforward conceptually, but as far as I could see, it would be easier to implement it this way. For one thing, it’s not necessary to specify everyone’s sex. For another, a one-way relationship (husband has a wife, wife belongs to a husband) is simpler than each member of the couple having a spouse.

[4] Naturally, you can use “in-country” to mean something other than that; whatever you call it, messages will not be delivered unless the person has that field selected (ticked). You can change the displayed name of the column in app/controllers/members_controller.rb by adding the line “config.columns[:in_country].label = 'On location' “ (or your desired label) to the active_scaffold configuration section at the top of the file.

[5] Besides serving as a verification step, this back-and-forth email prevents any third party from attacking the database by spoofing an authorized email. Only the person controlling the valid email account will receive the verification email, and the encrypted key prevents anyone else from exploiting that email.